First word response method, device and equipment for large language model

Abstract

The embodiment of the invention discloses a first word response method, device and equipment for a large language model, relates to the technical field of artificial intelligence, and is used for solving the problem of low interaction naturalness caused by high first word delay in a serial processing mode of traditional voice interaction. Performing streaming speech recognition on the speech stream; receiving an incremental text fragment output by streaming speech recognition, and dynamically updating a key value cache of the large language model based on the incremental text fragment; wherein the streaming speech recognition and the dynamic updating of the key value cache are synchronously carried out; continuously updating the key value cache until the voice input of the user is finished, so as to obtain a pre-filled key value cache when the voice input of the user is finished; and generating a response first word aiming at the voice stream according to the pre-filled key value cache.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, and device for initial character response in a large language model. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have been widely used in intelligent voice interaction systems. In these systems, real-time performance, especially the delay between the end of a user's question and the first word of the LLM's response (i.e., the first-word response time), is a key factor determining the fluency and naturalness of voice interaction based on the LLM, directly impacting the user's immersion and satisfaction.

[0003] Traditional solutions employ a sequential processing flow for user interaction. The user inputs a question via voice, the system converts the speech to text using Automatic Speech Recognition (ASR), a large language model generates a text response, and finally, text-to-speech (TTS) outputs the reply corresponding to the user's voice input. However, this sequential process requires waiting for the user to completely stop speaking before it can begin, and the large language model only starts generating the first character of the text response after the ASR outputs the complete text. This results in the large language model's inference resources remaining idle and underutilized while the user is speaking.

[0004] As the parameter size and interactive context of large language models increase, the cumulative delays in each stage of the serial processing flow significantly increase the first-word response time of the large language model. This prolonged waiting time disrupts the continuity and naturalness of human-computer dialogue, severely impacting the user experience and practicality of large language models in voice interaction scenarios.

[0005] Therefore, there is a need for ways to build inference states during user voice input, reduce first-word latency, and improve the naturalness of interaction in large language models. Summary of the Invention

[0006] This specification provides one or more embodiments of a large language model first-word response method, apparatus, and device to address the following technical problem: the need to construct inference states during user voice input, reduce first-word latency, and improve the naturalness of large language model interaction.

[0007] To solve the above-mentioned technical problems, one or more embodiments of this specification are implemented as follows: This specification provides one or more embodiments of a large language model initial response method, including: During the continuous input of voice stream by the user, streaming speech recognition is performed on the voice stream; The system receives incremental text segments output by the streaming speech recognition and dynamically updates the key-value cache of the large language model based on the incremental text segments; wherein the streaming speech recognition and the dynamic update of the key-value cache are performed synchronously. The key-value cache is continuously updated until the user's voice input ends, so as to obtain a pre-filled key-value cache when the user's voice input ends; Based on the pre-filled key-value cache, generate the first character of the response for the speech stream.

[0008] This specification provides one or more embodiments of a large language model initial response device, comprising: The recognition module is used to perform streaming speech recognition on the voice stream as the user continuously inputs the voice stream. An incremental update module is used to receive incremental text segments output by the streaming speech recognition and dynamically update the key-value cache of the large language model based on the incremental text segments; wherein, the streaming speech recognition and the dynamic update of the key-value cache are performed synchronously. The pre-filling module is used to continuously update the key value cache until the user's voice input ends, so as to obtain the pre-filled key value cache when the user's voice input ends; The generation module is used to generate the first character of the response for the speech stream based on the pre-filled key-value cache.

[0009] This specification provides one or more embodiments of a large language model initial response device, comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: During the continuous input of voice stream by the user, streaming speech recognition is performed on the voice stream; The system receives incremental text segments output by the streaming speech recognition and dynamically updates the key-value cache of the large language model based on the incremental text segments; wherein the streaming speech recognition and the dynamic update of the key-value cache are performed synchronously. The key-value cache is continuously updated until the user's voice input ends, so as to obtain a pre-filled key-value cache when the user's voice input ends; Based on the pre-filled key-value cache, generate the first character of the response for the speech stream.

[0010] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: By moving the streaming speech recognition and key-value cache update of the large language model from the point after the user finishes speaking to the point during the speaking process, the pre-computation part of the large language model, which was originally concentrated after the end of the speech, is distributed and hidden within the speaking time of the user's input speech stream. When the user finishes speaking, the system already has a complete pre-filled key-value cache and can immediately generate the first character of the response, avoiding the accumulated latency of serial processing in traditional solutions and reducing the first character delay time. The parallel execution of streaming speech recognition and cache update enables listening and understanding simultaneously, providing the response content as soon as the user finishes speaking, reducing the user's perceived latency and making voice interaction closer to natural conversation. Continuously updating the key-value cache until the end of the user's voice input ensures that the pre-filled key-value cache obtained at the end of the user's voice input fully covers the complete speech semantics, improving the first character response speed while ensuring response accuracy, achieving an immediate response effect. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This specification provides a flowchart illustrating a method for a large language model initial response method in an embodiment. Figure 2 This specification provides a schematic diagram of a system architecture for the first character response of a large language model in an application scenario, as illustrated in the embodiments of this specification. Figure 3 A schematic diagram illustrating the workflow of a large language model's first-character response in an application scenario provided in this specification's embodiments; Figure 4 This diagram illustrates a dynamic key-value cache management scenario provided in the embodiments of this specification. Figure 5 This is a schematic diagram illustrating the comparison of delay optimization effects provided in the embodiments of this specification; Figure 6 A schematic diagram of the structure of a large language model initial response device provided in the embodiments of this specification; Figure 7 This is a schematic diagram of the structure of a large language model initial response device provided in the embodiments of this specification. Detailed Implementation

[0012] This specification provides an embodiment of a method, apparatus, and device for initial character response in a large language model.

[0013] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0014] In the context of the background technology, with the rapid development of artificial intelligence technology, large language models have become the core engine of natural intelligent dialogue. Intelligent voice interaction systems based on large language models have been widely used in mobile terminal scenarios such as smartphones, smart speakers, in-vehicle assistants, and wearable devices. When users ask questions to intelligent voice interaction systems with large language models using natural language and receive dialogue responses, the real-time performance, especially the time required from the end of voice input to the generation of the first output character by the large language model—that is, the Time to First Token (TTFT)—is a key indicator for measuring the response speed of real-time interactive systems.

[0015] Current large language model-based voice interaction methods generally employ a serial processing flow. In this flow, the voice interaction system waits for the user's voice input to completely finish and for the automatic speech recognition module to output the complete text before sending the text to the large language model for context encoding to generate a text response. Then, through text-to-speech conversion, the system outputs the response content corresponding to the user's voice input. During the user's speech input phase, the computational resources of the large language model remain idle and are not effectively utilized, resulting in wasted computational resources and first-word delays.

[0016] Furthermore, as the parameter scale of large language models continues to increase and the supported context length expands, the computational overhead of the large language model in the response generation stage increases significantly, and the first-word response time becomes more delayed. This delay is particularly prominent in high-frequency interaction scenarios such as navigation guidance and real-time question answering, causing users to perceive lag or slow response, thus affecting the continuity and immersion of human-computer interaction. More specifically, in a certain in-vehicle scenario, when a user issues a voice command, traditional solutions require waiting until the user finishes speaking and the ASR confirms the input before initiating the large language model's inference. During this period, even if the user's previous voice command has already triggered some semantic understanding, it is impossible to prepare a response in advance, resulting in a waste of processing time and computing resources.

[0017] To address the issue of initial character delay caused by the cumulative delays in various stages of the serial processing, which significantly increases the overall response time, this paper proposes several optimization methods for large language models. These methods are presented in a more intuitive and illustrative manner.

[0018] One approach optimizes response time by using speculative decoding. This method employs a smaller, faster draft model to generate a batch of initial character predictions, which are then validated and corrected in parallel by the larger language model, thus reducing response time. However, speculative decoding primarily optimizes the output generation / decoding stage, as the computational bottleneck in the pre-padding stage remains, limiting its improvement on first-character response latency. Furthermore, it requires training and maintaining a dedicated draft model, increasing system complexity and resource consumption. The prediction accuracy of the draft model directly impacts the generation quality, introducing an additional trade-off between quality and speed.

[0019] In another approach, prediction-based optimization begins generating candidate responses as the user speaks, using speculative decoding techniques to prepare possible output content in advance and mitigate output latency. However, predicting response content before the user finishes speaking makes accuracy difficult to guarantee and may result in a large amount of unnecessary computation. Furthermore, pre-compiling multiple possible response directions but only using one leads to wasted computational resources. Additionally, if the user changes the topic or interrupts the conversation, the pre-generated content becomes completely invalid, requiring the system to restart. The pre-generated content may also mismatch with the user's complete input, affecting the coherence of the dialogue.

[0020] Traditional key-value caching optimization improves the inference efficiency of large language models by optimizing the storage, compression, and reuse mechanisms of key-value pairs, such as through dynamic allocation and cache compression. However, this approach only optimizes the storage and reuse of already computed key-value pairs and cannot address the latency issues of initial computation. The key-value cache size grows linearly with the sequence length, resulting in significant memory consumption for long inputs. Furthermore, this optimization primarily targets batch processing scenarios and offers limited improvement to the latency of single real-time interactions. It assumes that the input is static and complete, failing to handle the dynamic characteristics of streaming speech input.

[0021] Incremental decoding optimization methods, such as those used in machine translation, translate sentence by sentence by considering a broader context to maintain the coherence and consistency of the entire text. However, this approach primarily focuses on output quality and coherence, rather than latency optimization. It typically processes sentences or paragraphs at a time, failing to achieve fine-grained character-level optimization. Furthermore, the design does not consider the needs of real-time interaction and cannot handle the dynamic characteristics of voice input. It also still requires waiting for complete input before processing begins, and the first-word delay issue persists.

[0022] Therefore, to address the aforementioned issues and accelerate the first-word response of the large language model, this application proposes a method for first-word response of the large language model. This method can simultaneously perform streaming speech recognition and incremental construction of the large language model's key-value cache during the user's continuous input of speech. When the user finishes speaking, the voice interaction system containing the large language model already holds a complete pre-filled key-value cache, which can immediately generate the first word of the response based on the pre-filled key-value cache, achieving the effect of getting a response as soon as the user finishes speaking without additional encoding delay. This scheme makes full use of the idle time of the user's speech, distributing and hiding the high-latency large language model pre-filling calculation, which was originally concentrated after the user spoke, into the idle time of the user's speech, completing the state warm-up of the large language model, significantly reducing the first-word response time, and achieving a more natural dialogue rhythm. At the same time, by dynamically updating the key-value cache of the large language model through incremental text fragments, the key-value pair cache state that is strictly consistent with the current accurate input context of the large language model is maintained in real time, ensuring accuracy and robustness under long voice interaction. Based on this overall idea, the scheme of this application will be described in detail below.

[0023] Figure 1 This diagram illustrates a flowchart of a large language model initial response method provided in one or more embodiments of this specification. This method can be applied to various application areas with low-latency responses, such as mobile terminals with intelligent voice interaction systems, including smartphones, smart speakers, in-vehicle assistants, and wearable devices; other applications such as intelligent customer service systems requiring natural and fluent dialogue experiences for increasingly faster intelligent voice assistants; online education platforms requiring low-latency teacher-student dialogue in audio-interactive learning systems; and voice-based medical record entry and medical consultation systems with high real-time requirements. The process can be executed by the computing device (such as a server) of the intelligent voice interaction system containing the large language model. Certain input parameters or intermediate results in the process can be manually adjusted to help improve accuracy.

[0024] The following detailed description of each step of the method, with reference to specific embodiments, illustrates the method, which includes the following steps: S102: During the continuous input of voice stream by the user, perform streaming speech recognition on the voice stream.

[0025] Streaming Speech Recognition (hereinafter referred to as Streaming ASR) is a speech recognition technology that can output partial recognition results in real time during the user's speech. It refers to a speech recognition mode in which, during the process when the user's speech input has not ended, the system can incrementally and in real time convert the received speech segments into text and continuously output intermediate recognition results. Different from traditional non-streaming ASR that needs to wait for the complete speech input before decoding, streaming speech recognition can continuously obtain incremental text segments during the process of the user's continuous input of the speech stream, providing a real-time input basis for the dynamic update of the key-value cache of the subsequent large language model.

[0026] Among them, the incremental text segment is a partial recognition result output periodically after the streaming speech recognition of the speech stream during the process when the user's speech stream input has not ended. Its content may contain one or more tokens. And each incremental text segment represents the complete context text up to the current moment. Subsequent incremental text segments may expand, correct, or replace some content in the previous segments. In addition, each incremental text segment is generated in chronological order and gradually constitutes the complete text corresponding to the user's input speech stream.

[0027] More intuitively, in an application scenario such as Figure 2 As shown, during the process of the user's continuous input of the speech stream, after detecting speech activity based on voice activity detection, the speech stream after audio stream caching and preprocessing is subjected to streaming speech recognition, and partial recognition results are output at fixed time intervals. For example: at t = 100 ms, "guide" is recognized and output; at t = 200 ms, "navigate to nearby" is recognized and output; at t = 300 ms, "navigate to nearby diesel" is recognized and output; at t = 400 ms, "navigate to nearby diesel gas station" is recognized and output. In addition, it should also be noted that during the process of the user's continuous input of the speech stream, the speech stream after audio stream caching and preprocessing is cached.

[0028] In an embodiment of this specification, in the real-time speech interaction scenario, the user may have discontinuous speech input due to environmental interference, thinking pauses, or being interrupted by others. For example, when the user pauses halfway through speaking or resumes speech input after being interrupted, there is discontinuous speech input. In this case, if the streaming speech recognition continuously receives and processes speech segments, it will cause noise or irrelevant speech generated during the interruption to be misrecognized as valid input, making the context contain invalid information. Moreover, the key-value cache of the large language model is built based on incomplete or incorrect incremental text, affecting the accuracy of subsequent first-word responses. At the same time, the processing of this invalid speech segment will also increase the computational overhead of the large language model and waste computing resources. Based on this, during the process of the user's input of the speech stream, the streaming speech recognition of the speech stream specifically includes: Detect the start and stop events of the user's input voice stream. The start event is the trigger signal that indicates the user has begun valid voice input; in one method, such as... Figure 2 As shown, the audio input signal can be monitored in real time using a Voice Activity Detection (VAD) module. When continuous voice activity is detected and its duration exceeds a preset threshold, it is determined that the user has started inputting voice data. Alternatively, in one approach, if a user wake-up word is detected in the audio input signal, the start event of the user's voice input stream is determined based on the keyword recognition result of that wake-up word, indicating that the user's voice input stream has started. An interruption event refers to an audio input signal where the user's voice input is actively interrupted or passively interrupted, and can be based on, for example... Figure 2 The interrupt handling module shown executes the following actions: in one mode, it detects that the continuous silence duration exceeds a preset interruption threshold via VAD detection. This preset interruption threshold can be dynamically set based on the application scenario. In another mode, it detects an external cancellation signal from the user opening the cancel button. In yet another mode, it detects a cancellation command appearing in a newly added text segment in the voice stream. In yet another mode, it determines that an interruption event has been detected in the user's input voice stream based on the audio signal pattern and the VAD detection result.

[0029] In response to a start event, streaming speech recognition is initiated on the speech stream, converting received speech segments into text incrementally in real time and continuously outputting incremental text segments. Alternatively, in response to an interruption event, streaming speech recognition on the speech stream is paused, the current context after the pause is saved, and unconfirmed incremental text segments are cleared to avoid wasting computational resources and ensure that the next round of voice interaction can start normally.

[0030] The pause refers to stopping the transmission of audio frames to the streaming speech recognition system and saving the current recognition state. The unconfirmed incremental text segment refers to the portion of incremental text that is in the intermediate state of the streaming speech recognition system and has not yet been output when the interruption event occurs.

[0031] The process of handling interruption events in a certain application scenario includes the following mechanism code: class InterruptionHandler: def __init__(self, preprocessor): self.preprocessor = preprocessor self.interrupt_threshold = 0.5 # Interrupt detection threshold def detect_interruption(self, audio_signal, vad_result): "Detecting user interruption behavior" # Interruption in detection based on audio features and VAD results if self.is_interruption_pattern(audio_signal, vad_result): return True return False def handle_interruption(self): Handling user interruptions # 1. Pause the current pre-calculation self.preprocessor.pause_processing() # 2. Save the current state current_state = self.preprocessor.save_state() # 3. Prepare to process new input self.preprocessor.reset_for_new_input() return current_state This process performs streaming speech recognition only during valid speech, avoiding the problem of misprocessing interfering information. Responding to interruption events clears unconfirmed incremental text fragments, ensuring that subsequent interactions avoid initial response errors caused by historical remnants. Furthermore, the different responses to start and interruption events allow users to naturally pause, correct their statements, or cancel operations, making the interaction more consistent with users' daily conversational habits.

[0032] S104: Receive the incremental text fragments output by the streaming speech recognition, and dynamically update the key-value cache of the large language model based on the incremental text fragments; wherein, the streaming speech recognition and the dynamic update of the key-value cache are performed synchronously.

[0033] Key-Value Cache (KV Cache) is an intermediate state cache maintained during the autoregressive decoding process of a large language model. It is used to cache the key tensors and value tensors in the attention mechanism to avoid repeatedly calculating the key tensors and value tensors in the already processed context.

[0034] Specifically, during the continuous input of the user's speech stream, whenever the streaming speech recognition outputs an incremental text segment, forward inference of the large language model is performed based on the word sequence corresponding to that incremental text segment to generate a corresponding key-value tensor. This key-value tensor is then integrated into the currently maintained key-value cache in a way that maintains context consistency, such as appending or replacing, to achieve incremental updates to the cache. By distributing and hiding the time-consuming cache construction work during the user input process, the latency caused by serial waiting is fundamentally eliminated.

[0035] The dynamic update synchronization of streaming speech recognition and key-value caching advances and overlaps the traditional large language model context state construction work, which is performed after the user's voice input ends, into the process of user speaking and speech recognition, thereby achieving a lower latency for first-word response.

[0036] In other words, during the stage of outputting incremental text segments by streaming speech recognition, the dynamic update of the key-value cache for the incremental text segments is processed synchronously, so that the streaming speech recognition and the dynamic update process of the key-value cache overlap on the time axis. Since the dynamic update of the key-value cache is executed in parallel with streaming speech recognition, it is not necessary to wait for the user's speech to completely end. The complete context cache state can be gradually built during the user's speech, which solves the resource idle problem of large language models in traditional serial processing flow and avoids the cumulative delay effect of the delay of each stage in traditional serial architecture.

[0037] Specifically, in one embodiment, during the streaming speech recognition process in step S102 above, the key-value cache of the large language model may be dynamically updated based on incremental text fragments due to the user's natural continuation of speech. It may also be necessary to correct previous recognition results due to unclear pronunciation, environmental noise, or insufficient confidence in the language model. In such cases, directly appending to the cache would lead to semantic confusion in the context of the large language model's inference process. To address this problem, such as... Figure 3 The workflow shown in this embodiment dynamically updates the key-value cache of the large language model based on incremental text fragments, specifically including: In response to the semantic relationship between the incremental text fragment and the existing recognition results, the corresponding update operation is performed to realize the key-value cache update of the large language model.

[0038] The existing recognition results consist of the historical text comprised of all received and confirmed incremental text segments. Semantic relationships are determined by comparing the word sequences of the current incremental text segment with those of the existing recognition results. These relationships can be categorized as semantic continuation of the historical text (where the current text is a semantic continuation of the historical text), continuous expansion without content conflict, or content correction where the current text modifies or replaces parts of the historical text. The incremental text segments output by streaming speech recognition exhibit dynamic evolution; they may naturally continue the user's semantics or change the recognized historical text due to corrections. If a uniform key-value cache dynamic update method is used for all cases, it could lead to computational waste or semantic inconsistencies. Therefore, in response to the semantic relationship between the incremental text segment and the existing recognition results, update operations corresponding to these semantic relationships are performed—that is, update operations corresponding to continuous expansion or content correction—to update the key-value cache of the large language model, ensuring low-latency speech interaction while maintaining semantic consistency. Figure 3 As shown, in a certain scenario, the semantic relationship between the incremental text fragment and the existing recognition result can be determined by whether a correction to the streaming speech recognition result is detected. If so, the update operation corresponding to the content correction is executed, the number of rollback tokens is calculated, the key-value cache state is rolled back based on the number of rollback tokens, and the corrected text is reprocessed. If no correction to the streaming speech recognition result is detected, the update operation corresponding to the continuous expansion is executed, the incremental text fragment is segmented, and the key-value pairs of the newly added token sequence are calculated, thereby realizing the incremental update of the key-value cache.

[0039] Specifically, in response to the semantic relationship between the incremental text fragment and the existing recognition results, the corresponding update operation is performed, including the following process: The contextual consistency state between the incremental text fragment and the existing recognition result is obtained. This contextual consistency state can be determined by judging whether the current incremental text fragment uses the existing recognition result as a prefix. Specifically, if the starting part of the incremental text fragment is completely consistent with the existing recognition result, it is determined to be a continuous expansion state; otherwise, it is determined to be a content correction state, and the cache rollback position is further determined based on descending order of common prefixes. Alternatively, the existing recognition result and the incremental text fragment can be segmented separately to obtain a first word sequence and a second word sequence; the editing operation sequence between the two can be calculated using a sequence comparison algorithm. If the editing operation sequence only contains append operations, it is determined to be a continuous expansion state; if it contains replacement or deletion operations, it is determined to be a content correction state.

[0040] If the contextual consistency state is continuous expansion, then the key-value cache of the large language model is updated based on the key-value tensor corresponding to the incremental text fragment. That is, only for the newly added lexical subsequence, i.e., the part of the current lexical sequence that exceeds the historical length, is the target of large language forward inference, generating the corresponding key-value tensor to update the key-value cache of the large language model.

[0041] If the context consistency state is content correction, then in response to the correction position of the content correction, a partial reconstruction of the key-value cache of the large language model is performed. That is, a rollback operation is performed on the key-value cache according to the correction position, discarding all cached data after that position, obtaining the corrected key-value cache, regenerating the corrected key-value tensor from the correction position, and performing a partial reconstruction of the key-value cache of the large language model to ensure that the reconstructed key-value cache is consistent with the semantics of the input speech stream.

[0042] Based on the above approach, by adaptively matching different update operations according to the contextual consistency between incremental text fragments and existing recognition results, the computational waste and semantic inconsistency problems caused by a uniform cache update method are avoided. In the case of continuous expansion, only newly added content is processed, improving the update speed of the key-value cache. In the case of content correction, semantic conflicts are eliminated through local reconstruction, reducing the error rate of the first character response.

[0043] In other embodiments, the semantic relationship is not limited to a context-consistent state, but may also include intention. Figure 1 At least one of the following: consistency, semantic similarity, referential dependency, ASR confidence evolution trend, syntactic integrity, or information gain.

[0044] Furthermore, based on the key-value tensors corresponding to the incremental text fragments, the key-value cache of the large language model is updated, specifically including: When the streaming speech recognition module outputs a new incremental text segment, it first concatenates this segment with the processed historical text already used to build the key-value cache—that is, the historical text corresponding to the existing recognition results—to form the current text representing the complete semantic context up to the current moment. It should be noted that this concatenation is not a simple string join, but rather refers to constructing a complete context representation based on semantic alignment. Since each incremental text segment output by the streaming speech recognition module is itself a relatively optimal complete transcription result up to the current moment, the current text is essentially equivalent to the complete sentence expressed by that incremental text segment. However, its construction process explicitly associates with the historical processing state to facilitate accurate positioning in subsequent caching operations.

[0045] By segmenting the current text, the current word sequence corresponding to the current text is obtained, along with the historical word sequence corresponding to the historical text. Through a word-by-word comparison between the current word sequence and the historical word sequence, the differences between them are determined, and the newly added word sequence is extracted. It can be understood that the newly added word sequence is a continuous word sequence in the current word sequence that exceeds the length of the historical word sequence and has not been corrected; it represents the semantic content newly added to the user's input speech stream.

[0046] Based on the newly added lexical subsequence, the encoder of the large language model performs forward inference to generate a key-value tensor corresponding to the subsequence. The key-value tensor includes the key vector and value vector calculated based on the newly added lexical units in the attention mechanism layer of the large language model. This key-value tensor is then integrated into the end of the current key-value cache according to the lexical order, thereby completing the incremental expansion of the key-value cache. By only calculating the key-value tensor and updating the cache for the newly added lexical subsequence, the repeated encoding of processed historical context is avoided, improving the computational efficiency and response speed in the streaming voice interaction process, while ensuring that the key-value cache always remains consistent with the currently updated speech semantics.

[0047] Furthermore, since intermediate results of streaming speech recognition often contain uncertainty, subsequent segments may correct the preceding text (e.g., changing "book a flight to City A" to "book a flight to City B"). If each incremental segment is directly appended to the key-value cache after word segmentation, the cache will contain contexts of both 'City A' and 'City B', causing logical inconsistencies in the output of the large language model. To address this, the above embodiment judges the contextual consistency between the incremental text segment and the existing recognition results. In the case of content correction, in response to the correction location, a local reconstruction of the key-value cache of the large language model is performed. This local reconstruction specifically includes the following process: The corrected text is obtained by updating the historical text corresponding to the existing recognition results based on the content correction. Specifically, the historical text corresponding to the existing recognition results is updated based on the current incremental text fragment to obtain the corrected complete context text. By comparing the word sequence of the corrected text with that of the historical text, the starting position of the content change, i.e., the correction position, is determined. Based on this correction position, the key-value cache is rolled back, discarding all key-value data from the correction position to the end of the cache, thus obtaining the corrected key-value cache. That is, as follows... Figure 4 As shown, in a certain scenario, a correction request can be generated based on the correction position and the correction text, thereby calculating the number of tokens that need to be rolled back and rolling back the key-value cache. After the rollback is completed, the request is reprocessed, the corrected text is processed, the corrected token sequence is obtained, and the key-value cache is updated based on the corrected token sequence.

[0048] This process ensures that the corrected key-value cache no longer contains the corrected error context information. Based on the corrected text, a subsequence of lexical terms from the corrected position to the end of the text is extracted, and forward inference of a large language model is performed on this subsequence to generate a corrected key-value tensor from the corrected position. The corrected key-value tensor is then integrated into the corrected key-value cache according to the lexical order, thereby completing the local reconstruction of the key-value cache.

[0049] Based on this process, when the speech recognition result of the streaming speech recognition of the user's continuous input speech stream is corrected, the key-value data associated with the erroneous semantics in the cache is accurately cleared, and the key-value cache is reconstructed with the corrected content. This effectively avoids the output logic chaos of the large language model caused by cache errors, while avoiding the computational overhead of rebuilding the key-value cache, thus balancing the accuracy and efficiency of the voice interaction process.

[0050] In a certain application scenario, dynamically updating the key-value cache of a large language model based on the incremental text fragments can be achieved using the following code mechanism: class IncrementalPreprocessor: def __init__(self, tokenizer, model): self.tokenizer = tokenizer self.model = model self.kv_cache = KVCache() self.processed_text = "" self.processed_tokens = [] def process_asr_fragment(self, new_text, confidence): """Processing new text fragments from ASR output""" # 1. Text Concatenation and Standardization full_text = self.processed_text + new_text # 2. Incremental word segmentation new_tokens = self.tokenizer.encode(full_text) added_tokens = new_tokens[len(self.processed_tokens):] # 3. Calculate the key-value pairs of the newly added token. if added_tokens: new_kv = self.model.compute_kv_incremental( added_tokens, existing_cache=self.kv_cache ) # 4. Update cache status self.kv_cache.append(new_kv) self.processed_tokens.extend(added_tokens) self.processed_text = full_text def handle_asr_correction(self, corrected_text, correction_start): """Handling ASR Corrections""" # 1. Calculate the number of tokens that need to be rolled back. rollback_tokens = self.calculate_rollback_tokens( corrected_text, correction_start ) # 2. Rollback KV cache state self.kv_cache.rollback(rollback_tokens) # 3. Reprocess the corrected text self.reprocess_from_correction(corrected_text, correction_start) Furthermore, in the process of streaming speech recognition, different users have significantly different speaking habits, such as speaking speed, pauses, and correction tendencies. The confidence level of the speech recognition results also dynamically changes with factors such as environmental noise and speech clarity. Using a fixed key-value cache update method may lead to erroneous intermediate results being written to the cache in high-noise or low-confidence scenarios. For users with correction tendencies, repeated cache rebuilding also wastes computational resources. Therefore, in one or more embodiments of this specification, the dynamic update of the key-value cache after obtaining a new text segment further includes: Based on the speech stream, user speaking patterns are identified. The update parameters of the key-value cache are adjusted according to the confidence level of incremental text segments and the user speaking patterns. These update parameters include at least: update trigger conditions and processing scale.

[0051] User speaking patterns refer to the behavioral characteristics exhibited by users in voice interactions, including but not limited to: average speech rate, distribution of pause duration within sentences, frequency of historical corrections, and common sentence structures. This pattern can be obtained through offline modeling of historical interaction data or online incremental learning. The confidence level of incremental text fragments, such as... Figure 4 The figure shows the numerical indicators output when streaming speech recognition outputs incremental text segments. These indicators represent the reliability of the streaming speech recognition output content and guide the dynamic adjustment of the pre-computation strategy, i.e., the key-value cache update parameters. The update parameters are adjustable configuration items that control the key-value cache update behavior, such as determining the update trigger conditions when to start the cache update, and determining the processing scale of the update granularity, such as performing only local reconstruction or updating new words.

[0052] In this process, by acquiring the confidence level of the current incremental text segment in real time and combining it with the identified user speaking patterns, the update parameters of the key-value cache are dynamically adjusted to achieve personalized and robust cache management. Specifically, adjusting the update parameters of the key-value cache based on the confidence level of the incremental text segment and the user speaking pattern can be achieved by first classifying users into several speaking patterns based on their behavioral characteristics during voice interaction, such as fluent mode, hesitant mode, and slow mode. Each user speaking pattern corresponds to a set of default update parameters. For example, a lower confidence trigger threshold is set for fluent users, allowing for rapid cache submission to reduce latency. A higher confidence threshold is set for hesitant users, and cache snapshots are enabled. When the confidence levels of two consecutive segments are both higher than the threshold, the temporary cache is persisted. At this point, based on the confidence level of the current incremental text segment and the identified user speaking patterns, it can be determined whether the confidence level has reached the corresponding confidence threshold for the user speaking pattern, thereby dynamically adjusting the update parameters of the key-value cache. In another feasible approach, multi-dimensional behavioral indicators of user speech patterns are extracted from historical interaction data, including but not limited to: average number of corrections per unit time, frequency of speech pauses, sentence completion rate, and the confidence stability of historical recognition results. These indicators are normalized and weighted to form a user reliability score corresponding to the user's speech pattern, where a higher value indicates more stable user voice input. Simultaneously, the real-time confidence score of the current incremental text segment output by the streaming speech recognition module is obtained. Based on the real-time confidence score and the user reliability score, a comprehensive update confidence score is obtained. The update parameters of the corresponding key-value cache are matched based on this comprehensive update confidence score, thereby achieving smooth adaptation to new users or scenarios with sudden behavioral changes, effectively balancing response speed and semantic accuracy, and improving robustness and user experience in complex real-world environments.

[0053] Furthermore, in one embodiment, to address the issue of continuous high-bandwidth storage resource consumption caused by large language model key-value caching in long speech input scenarios, the following process is also included: In streaming speech recognition, a hierarchical storage management system is implemented for key-value pairs. This system comprises a three-tiered cache structure: the first cache layer stores key-value pairs of recently processed terms, supporting fast access and modification; the second cache layer provides a compressed representation of historical context to conserve memory; and the third cache layer stores summary information of long-term dialogue history, supporting contextual continuity.

[0054] Tiered storage management includes: such as Figure 4 As shown, during the word segmentation of incremental text fragments to obtain newly added word subsequences, the key vector and value vector of these new word subsequences are calculated to update the key-value cache. During incremental key-value cache updates, the cache size is checked. If the available storage capacity of the first cache layer is insufficient, the key-value cache exceeding the available storage capacity of the first cache layer (i.e., the earlier generated portion) undergoes lossless compression processing, such as numerical precision conversion or sparse encoding, and is then migrated to the second cache layer. When the user's voice input ends, the key-value caches of the first and second cache layers are merged. Specifically, the compressed key-value data is read from the second cache layer, decompressed, and then concatenated with the key-value cache in the first cache layer based on time order to obtain a complete pre-filled key-value cache. Based on this pre-filled cache, the large language model directly performs first-character decoding to generate the first character of the response. After generating the first character of the response, in order to achieve persistent storage, a semantic summary is generated based on the complete text corresponding to the speech stream, and the semantic summary is persisted to the third cache layer to support the reuse of context in subsequent voice interactions rather than the generation of the response in the current round.

[0055] By employing a tiered storage management approach, when supporting ultra-long voice input on memory-constrained devices, some key-value data generated earlier can be compressed and migrated to the second cache layer, avoiding interaction interruptions due to cache overflow. Furthermore, lossless compression and tiered migration ensure the integrity of the cache before first character generation and controllable response latency. By generating semantic summaries based on the complete text corresponding to the voice stream and persisting these summaries to the third cache layer, the overhead of cross-turn context reconstruction is reduced, improving the coherence of multi-turn dialogues.

[0056] In a certain application scenario described in this manual, dynamic management of key-value caching can be implemented based on the following code mechanism: class DynamicKVCache: def __init__(self, max_sequence_length=2048): self.keys = [] # Store the key tensor self.values ​​= [] # Store the Value tensor self.positions = [] # Store position information self.max_length = max_sequence_length def append(self, new_keys, new_values): """Incrementally add new key-value pairs""" self.keys.extend(new_keys) self.values.extend(new_values) self.positions.extend(range(len(self.keys)-len(new_keys), len(self.keys))) # Memory management: Compress when the length limit is exceeded if len(self.keys)>self.max_length: self.compress_cache() def rollback(self, num_tokens): """Roll back a specified number of tokens""" if num_tokens>0: self.keys = self.keys[:-num_tokens] self.values ​​= self.values[:-num_tokens] self.positions = self.positions[:-num_tokens] def compress_cache(self): """Cache compression strategy""" # Implement compression algorithms such as sliding window or importance sampling # Retain recent and important key-value pairs Pass Based on this design, the key-value cache state synchronized with the speech semantics can be efficiently maintained during the user's continuous speaking. It supports both rapid appending during smooth expansion and accurate rollback during recognition correction. It also has memory management capabilities to cope with long speech input, providing basic support for achieving low-latency and high-accuracy large-scale speech model interaction.

[0057] S106: Continuously update the key-value cache until the user's voice input ends, so as to obtain the pre-filled key-value cache when the user's voice input ends.

[0058] As described above, to achieve low-latency voice interaction, incremental updates to the key-value cache of the large language model are initiated during the user's speech. In one embodiment of this specification, in order to advance and distribute the context encoding task, which was originally concentrated after the speech ended, within the user's voice input time window, the key-value cache is continuously updated until the user's voice input ends, thereby obtaining a complete pre-filled key-value cache for first-character generation at the moment the speech ends.

[0059] Specifically, upon detecting the start of the user's input speech stream, incremental updates of streaming speech recognition and key-value cache are simultaneously initiated. Each time streaming speech recognition outputs an incremental text segment, the segment is segmented, and then semantic alignment and contextual consistency are assessed. Based on the assessment results, corresponding cache operations are performed: if it's a continuous expansion, the key-value tensor corresponding to the newly added word is appended; if it's content correction, the affected portion of the cache is rolled back and partially rebuilt. These update operations are repeatedly performed in real-time during the user's speech, enabling continuous updates to the key-value cache based on each incremental text segment as the user continuously inputs the speech stream.

[0060] If an end-of-stream event is detected during this process, the key-value cache is incrementally updated based on the complete text from the streaming speech recognition, resulting in a complete key-value cache. It's important to note that upon detecting an end-of-stream event, the streaming speech recognition outputs the complete text, which is then compared to the text with the updated key-value cache. If a tail difference exists, only the differing portion undergoes an additional forward computation, and the corresponding key-value cache is appended or replaced, ensuring strict alignment between the cache and the complete text input. At this point, the key-value cache completely covers the entire text context corresponding to the user's speech, eliminating the need for any additional context encoding. This complete key-value cache is the pre-filled key-value cache. Since the autoregressive decoding process of the large language model relies on this cache to calculate attention weights, having a pre-filled cache means that the first-word generation stage can proceed directly, reducing the time interval from the end of the speech to the first-word response.

[0061] S108: Generate the first character of the response for the speech stream based on the pre-filled key-value cache.

[0062] After obtaining the pre-filled key-value cache based on S106, this cache contains the key and value vector representations of the text context corresponding to the user's input speech stream in each attention mechanism layer of the large language model. Based on this pre-filled key-value cache, the autoregressive decoding of the large language model can be directly initiated, using a specific starting word as the initial input to the decoder. The large language model then uses this starting word to calculate its query vector and interacts with the key and value vectors of all historical contexts in the pre-filled key-value cache through attention, obtaining a context-aware representation that incorporates complete user semantics. This representation, after processing by subsequent layers of the model, is fed into the language model head, outputting a probability distribution covering the entire vocabulary. Words with higher probabilities are selected from this distribution, or words are selected through a sampling strategy to obtain the first character of the response. Since the pre-filled key-value cache has been pre-constructed, no further context encoding of the complete text corresponding to the user's input speech stream is required during this process; only a single-step decoding forward propagation is needed, thus completing the process in a relatively short time. Unlike traditional methods that require hundreds of milliseconds to encode the complete text after the speech ends before generating the first character, this application hides the encoding overhead of the large language model during the user's speech by listening and caching simultaneously. This significantly reduces the latency of first character generation and makes the user experience more natural and smooth.

[0063] like Figure 3 As shown, after generating the first character response, the large language model will continue to perform autoregressive generation to complete the generation of the response text. At this time, text-to-speech will be started to convert the response text into a speech response, thereby outputting the speech response to the user, clearing the cache state and preparing for the next round of interaction.

[0064] Furthermore, following the embodiment of S106 above, that is, based on continuously updating the key value cache until the user's voice input ends, in order to further improve the timing accuracy and semantic rationality of the first character response, the method may also include the following process: Prosodic features are extracted from the speech stream, and the probability of an ending event is predicted based on these features. Prosodic features refer to the phonological attributes of language that transcend individual phonemes or words; they endow spoken language with rhythm, intonation, emotion, and structure, and are key suprasegmental components for conveying semantics, intention, and emotion. These prosodic features include at least fundamental frequency, energy, speech rate, pause duration, and intonation profile. In one method of predicting the probability of an ending event based on prosodic features, the prosodic features are aligned by time frames, with each frame containing at least two of the following: fundamental frequency, energy, speech rate change rate, and zero-crossing rate, forming a prosodic feature sequence. This prosodic feature sequence is then input into a pre-trained lightweight prosodic model to obtain the probability of the speech naturally ending at the current moment, i.e., the probability of the ending event. The lightweight prosodic model can be a single-layer bidirectional recurrent neural network, whose hidden states are global average pooled and then fed into a fully connected layer and a sigmoid activation function, or a transformer encoder with only 1-2 layers. Its output sequence is classified and labeled or average pooled along the time dimension, then linearly projected and fed into a sigmoid activation function to output the probability of the model's output ending event. In another method of predicting the probability of ending events based on prosodic features, events corresponding to each prosodic feature can be detected separately. For example, based on continuous first and second-order differences of the fundamental frequency trajectory, significant fundamental frequency resets or drops can be identified to detect pitch boundary events; by calculating the short-time energy envelope, continuous energy decreases within a local time period can be identified to detect energy decay events, etc. If each detected event corresponds to a preset score for the ending event, the probability of the ending event is determined.

[0065] The semantic completeness of incremental text segments output by streaming speech recognition is determined. One method is to assess the semantic completeness of the incremental text segment based on its semantic similarity to the dialogue semantic summary, which is generated based on the context of the incremental text segment. Another method involves sequentially checking whether the incremental text segment contains a complete subject-verb-object structure; detecting the presence of unclosed clauses, dangling interrogative words, or omitted components; and then comprehensively evaluating the results to determine its semantic completeness.

[0066] The confidence score of a complete query in the user's input speech stream is obtained by weighted summation of the occurrence probability and semantic completeness. If the confidence score exceeds a preset threshold, the current speech stream is considered a complete query. Alternatively, a complete query can be determined when both the occurrence probability and semantic completeness are greater than their respective preset thresholds. This reduces invalid pre-generation, avoids resource waste, and improves robustness by preventing pre-generation if the user continues speaking since the threshold has not been reached.

[0067] When a complete query is determined, a forward decoding operation of the large language model can be performed in advance, based on the current key-value cache, to generate a predicted score vector for the first character of the response, even before the user's voice is completely muted. In response to the end event of the user's input speech stream, instead of recalculating the first character of the response, the highest-scoring word is selected as the first character of the response and output. This process determines whether a complete query is constituted based on the probability of the end event and semantic completeness, thus deciding whether to pre-generate the first character of the response. This avoids premature responses when the sentence is incomplete and allows for immediate responses as soon as the user's voice ends, significantly reducing perceptual latency and improving the naturalness of the interaction while ensuring the accuracy of the first character response.

[0068] Furthermore, in one embodiment, the scheme can be optimized for parallel computing, such as parallel execution of streaming speech recognition, text preprocessing, and key-value calculation. The parallel computing capabilities of the Graphics Processing Unit (GPU) are utilized to accelerate key-value pair calculations, achieving GPU acceleration. Memory pooling and zero-copy techniques are employed to reduce memory allocation overhead, achieving memory optimization.

[0069] like Figure 5 The diagram illustrates a comparison between the large language model's first-character response method and the traditional method in a specific application scenario. As can be seen, in the traditional method, when the user begins speaking, user input begins at time t0, and the system waits for the user to finish speaking. During this process, the user's input speech stream is not processed. Only after the user finishes speaking (time t1) does the speech recognition process begin within the time interval t1-t2. After recognition is complete, the large language model's pre-filling calculation begins based on the speech recognition result within the time interval t2-t3. At time t3, the first character is generated and output as the first-character response. It is evident that the large language model's pre-filling calculation is the main source of latency in this process, with the total latency (t3-t1) typically exceeding 800ms.

[0070] In this application, when the user begins speaking, user input begins at time t0. Streaming speech recognition is initiated during this input process for incremental computation. This means that streaming speech recognition starts immediately after t0 and the key-value cache of the recognized incremental text segments is computed in parallel. As the user continues speaking, background computation is performed during this time, continuously updating the key-value cache. At time t1, when the user finishes input, a pre-filled cache is obtained. Processing is then completed within a short period t1-t4 using this pre-filled cache, quickly generating the first character. The total latency t4-t1 is typically less than 200ms. Therefore, traditional centralized computation leads to significant latency peaks. In contrast, this application's computation, distributed throughout the user's speaking time, hides this latency. Streaming speech recognition and key-value cache computation are executed in parallel, with most pre-filling computation completed during the user's speech, effectively improving the first character response speed and achieving a natural interactive effect that closely resembles human conversation, where the interactive result is obtained immediately after speaking.

[0071] In addition to other solutions addressing initial character generation latency, there are hardware acceleration solutions, model architecture optimization solutions, system architecture optimization solutions, network and deployment optimization solutions, and algorithm-level alternatives. Specifically, hardware acceleration solutions primarily improve speed by enhancing the underlying computing hardware. For example, dedicated AI chip optimization accelerates computation by customizing matrix multiplication units, on-chip high-bandwidth memory, and pipeline architectures for the Transformer architecture, but its development cycle is long, costly, and lacks versatility. GPU cluster parallelization attempts to utilize multiple GPUs to process a single request in parallel, but faces problems such as high communication overhead, low resource utilization, and excessive cost, making it unsuitable for real-time interactive scenarios. Regarding model architecture optimization solutions, lightweight model design accelerates performance by reducing model size and using efficient attention mechanisms, but significantly sacrifices model capabilities and dialogue quality. Mixed-precision computation uses low-precision formats for inference, achieving limited acceleration but potentially affecting output quality and being complex to implement. System architecture optimization solutions improve performance at the software system design level, with pre-warming caching strategies pre-compute KV caches of common dialogue templates for reuse, but its template coverage is limited and storage overhead is enormous. The hierarchical inference architecture uses large and small models to work together, dynamically selecting based on complexity, resulting in an exceptionally complex system with inconsistent quality. Network and deployment optimization schemes focus on reducing data transmission latency, with edge computing deployment placing the model near the network edge close to the user; however, the limited computing power of edge nodes cannot support large models. Predictive loading predicts input based on user behavior and prepares it in advance, but its prediction accuracy is limited and privacy issues arise. Algorithm-level alternatives include non-autoregressive generation, which attempts to output multiple words in parallel at once, but its generation quality is generally inferior to mainstream autoregressive methods and training is complex. Retrieval-enhanced generation combines a retrieval system to obtain content from a pre-set response library, which can speed up the process but limits the originality and complex reasoning capabilities of the responses. As shown in Table 1 below, which compares the latency optimization effects of various schemes, it can be seen that the input-side pre-computation method in this application significantly improves the first-character response speed compared to other methods.

[0072] Table 1. Comparison of Delay Optimization Effects of Different Schemes

[0073] Based on the same idea, one or more embodiments of this specification also provide apparatus and devices corresponding to the above methods, such as... Figure 6 , Figure 7 As shown.

[0074] Figure 6 This specification provides a schematic diagram of the structure of a large language model initial response device according to one or more embodiments, the device comprising: The recognition module 602 is used to perform streaming speech recognition on the voice stream during the process of continuous input by the user. The incremental update module 604 is used to receive incremental text segments output by the streaming speech recognition and dynamically update the key-value cache of the large language model based on the incremental text segments; wherein the streaming speech recognition and the dynamic update of the key-value cache are performed synchronously. The pre-filling module 606 is used to continuously update the key value cache until the user's voice input ends, so as to obtain the pre-filled key value cache when the user's voice input ends; The generation module 608 is used to generate the first character of the response for the voice stream based on the pre-filled key value cache.

[0075] Optionally, the incremental update module 604 includes: In response to the semantic relationship between the incremental text fragment and the existing recognition results, the corresponding update operation is performed to update the key-value cache of the large language model.

[0076] Optionally, the incremental update module 604 includes: Obtain the contextual consistency status between the incremental text fragment and the existing recognition result; If the context consistency state is continuous expansion, then the key-value cache of the large language model is updated based on the key-value tensor corresponding to the incremental text fragment; If the context consistency state is content correction, then in response to the correction position of the content correction, the key-value cache of the large language model is partially reconstructed.

[0077] Optionally, the incremental update module 604 includes: Based on the comparison between the current word sequence and the historical word sequence corresponding to the current text, the newly added word subsequence is determined; the current text is obtained by concatenating the incremental text fragment with the processed historical text. Based on the newly added lexical subsequences, corresponding key-value tensors are generated, and the key-value tensors are sequentially integrated into the current key-value cache.

[0078] Optionally, the incremental update module 604 includes: correcting and updating the historical text corresponding to the existing recognition result according to the content to obtain the corrected text; Based on the correction position, the key-value cache is rolled back to obtain the corrected key-value cache; Based on the corrected text, a correction key-value tensor starting from the corrected position is generated, and the correction key-value tensor is sequentially integrated into the correction key-value cache.

[0079] Optionally, the device further includes: an adjustment module 610; The adjustment module 610 is used to identify the user's speaking pattern based on the voice stream; Based on the confidence level of the incremental text fragment and the user's speaking pattern, the update parameters of the key-value cache are adjusted; wherein the update parameters include at least: update trigger conditions and processing scale.

[0080] Optionally, the device further includes: a dynamic key-value cache management module 612; The dynamic key-value cache management module 612 is used to perform hierarchical storage management of the key-value cache during the streaming speech recognition process; The hierarchical storage management includes: when the available storage capacity of the first cache layer is insufficient, performing lossless compression on the key-value cache that exceeds the available storage capacity, and migrating the compressed key-value cache to the second cache layer; When the user's voice input ends, the key-value caches of the first cache layer and the second cache layer are merged to obtain a pre-filled key-value cache to generate the first character of the response; After generating the first character of the response, a semantic summary is generated based on the complete text corresponding to the speech stream, and the semantic summary is persisted to the third cache layer to support context reuse for subsequent voice interactions.

[0081] Optionally, the identification module 602 includes: Detect the start and interruption events of the user's input voice stream; In response to the start event, streaming speech recognition of the speech stream is initiated to obtain incremental text segments; In response to the interruption event, streaming speech recognition of the speech stream is paused, and unconfirmed incremental text segments are cleared.

[0082] Optionally, the pre-filling module 606 specifically includes: As the user continues to input the voice stream, the key-value cache is continuously updated based on each incremental text segment; If the end event of the user input voice stream is detected, the key-value cache is incrementally updated based on the complete text of the streaming speech recognition to obtain the complete key-value cache; The complete key-value cache is used as a pre-populated key-value cache.

[0083] Optionally, the device further includes: a pre-generation module 614; The pre-generation module 614 is used to extract the prosodic features of the speech stream and predict the probability of the occurrence of the ending event based on the prosodic features. Obtain the semantic completeness of the incremental text fragment; Based on the occurrence probability and the semantic completeness, predict whether the user input voice stream constitutes a complete query; If so, then generate a predicted score vector for the first character of the response based on the current key-value cache; In response to the end event of the user input voice stream, the first character of the response is selected and output based on the predicted score vector.

[0084] Figure 7 A schematic diagram of a large language model initial response device provided for one or more embodiments of this specification, the device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: During the continuous input of voice stream by the user, streaming speech recognition is performed on the voice stream; The system receives incremental text segments output by the streaming speech recognition and dynamically updates the key-value cache of the large language model based on the incremental text segments; wherein the streaming speech recognition and the dynamic update of the key-value cache are performed synchronously. The key-value cache is continuously updated until the user's voice input ends, so as to obtain a pre-filled key-value cache when the user's voice input ends; Based on the pre-filled key-value cache, generate the first character of the response for the speech stream.

[0085] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog are commonly used. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0086] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0087] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0088] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0089] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0093] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0094] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0095] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0096] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0097] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0098] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0099] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0100] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for initial character response in a large language model, the method comprising: During the continuous input of voice stream by the user, streaming speech recognition is performed on the voice stream; The system receives incremental text segments output by the streaming speech recognition and dynamically updates the key-value cache of the large language model based on the incremental text segments; wherein the streaming speech recognition and the dynamic update of the key-value cache are performed synchronously. The key-value cache is continuously updated until the user's voice input ends, so as to obtain a pre-filled key-value cache when the user's voice input ends; Based on the pre-filled key-value cache, generate the first character of the response for the speech stream.

2. The method as described in claim 1, wherein dynamically updating the key-value cache of the large language model based on the incremental text fragments specifically includes: In response to the semantic relationship between the incremental text fragment and the existing recognition results, the corresponding update operation is performed to update the key-value cache of the large language model.

3. The method of claim 2, wherein, in response to the semantic relationship between the incremental text fragment and the existing recognition result, a corresponding update operation is performed, including: Obtain the contextual consistency status between the incremental text fragment and the existing recognition result; If the context consistency state is continuous expansion, then the key-value cache of the large language model is updated based on the key-value tensor corresponding to the incremental text fragment; If the context consistency state is content correction, then in response to the correction position of the content correction, the key-value cache of the large language model is partially reconstructed.

4. The method as described in claim 3, wherein updating the key-value cache of the large language model based on the key-value tensor corresponding to the incremental text fragment specifically includes: Based on the comparison between the current word sequence and the historical word sequence corresponding to the current text, the newly added word subsequence is determined; the current text is obtained by concatenating the incremental text fragment with the processed historical text. Based on the newly added lexical subsequences, corresponding key-value tensors are generated, and the key-value tensors are sequentially integrated into the current key-value cache.

5. The method as described in claim 3, wherein, in response to the correction location of the content modification, a local reconstruction of the key-value cache of the large language model is performed, specifically including: Based on the aforementioned content, the historical text corresponding to the existing recognition results is corrected and updated to obtain the corrected text; Based on the correction position, the key-value cache is rolled back to obtain the corrected key-value cache; Based on the corrected text, a correction key-value tensor starting from the corrected position is generated, and the correction key-value tensor is sequentially integrated into the correction key-value cache.

6. The method of claim 1, wherein the dynamic updating of the key-value cache further includes: Based on the audio stream, identify the user's speaking pattern; Based on the confidence level of the incremental text fragment and the user's speaking pattern, the update parameters of the key-value cache are adjusted; wherein the update parameters include at least: update trigger conditions and processing scale.

7. The method of claim 1, further comprising: In the streaming speech recognition process, the key-value cache is managed in a hierarchical manner. The hierarchical storage management includes: when the available storage capacity of the first cache layer is insufficient, performing lossless compression on the key-value cache that exceeds the available storage capacity, and migrating the compressed key-value cache to the second cache layer; When the user's voice input ends, the key-value caches of the first cache layer and the second cache layer are merged to obtain a pre-filled key-value cache to generate the first character of the response; After generating the first character of the response, a semantic summary is generated based on the complete text corresponding to the speech stream, and the semantic summary is persisted to the third cache layer to support context reuse for subsequent voice interactions.

8. The method as described in claim 1, wherein during the process of user inputting a voice stream, streaming speech recognition is performed on the voice stream, specifically including: Detect the start and interruption events of the user's input voice stream; In response to the start event, streaming speech recognition of the speech stream is initiated to obtain incremental text segments; In response to the interruption event, streaming speech recognition of the speech stream is paused, and unconfirmed incremental text segments are cleared.

9. The method as described in claim 1, wherein the key-value cache is continuously updated until the user's voice input ends, to obtain a pre-filled key-value cache, specifically includes: As the user continues to input the voice stream, the key-value cache is continuously updated based on each incremental text segment; If the end event of the user input voice stream is detected, the key-value cache is incrementally updated based on the complete text of the streaming speech recognition to obtain the complete key-value cache; The complete key-value cache is used as a pre-populated key-value cache.

10. The method of claim 9, further comprising: Extract the prosodic features of the speech stream, and predict the probability of the occurrence of the ending event based on the prosodic features; Obtain the semantic completeness of the incremental text fragment; Based on the occurrence probability and the semantic completeness, predict whether the user input voice stream constitutes a complete query; If so, then generate a predicted score vector for the first character of the response based on the current key-value cache; In response to the end event of the user input voice stream, the first character of the response is selected and output based on the predicted score vector.

11. A large language model initial character response device, comprising: The recognition module is used to perform streaming speech recognition on the voice stream as the user continuously inputs the voice stream. An incremental update module is used to receive incremental text segments output by the streaming speech recognition and dynamically update the key-value cache of the large language model based on the incremental text segments; wherein, the streaming speech recognition and the dynamic update of the key-value cache are performed synchronously. The pre-filling module is used to continuously update the key value cache until the user's voice input ends, so as to obtain the pre-filled key value cache when the user's voice input ends; The generation module is used to generate the first character of the response for the speech stream based on the pre-filled key-value cache.

12. The apparatus of claim 11, wherein the incremental update module comprises: In response to the semantic relationship between the incremental text fragment and the existing recognition results, the corresponding update operation is performed to update the key-value cache of the large language model.

13. The apparatus of claim 12, wherein the incremental update module comprises: Obtain the contextual consistency status between the incremental text fragment and the existing recognition result; If the context consistency state is continuous expansion, then the key-value cache of the large language model is updated based on the key-value tensor corresponding to the incremental text fragment; If the context consistency state is content correction, then in response to the correction position of the content correction, the key-value cache of the large language model is partially reconstructed.

14. A large language model initial response device, comprising: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: During the continuous input of voice stream by the user, streaming speech recognition is performed on the voice stream; The system receives incremental text segments output by the streaming speech recognition and dynamically updates the key-value cache of the large language model based on the incremental text segments; wherein the streaming speech recognition and the dynamic update of the key-value cache are performed synchronously. The key-value cache is continuously updated until the user's voice input ends, so as to obtain a pre-filled key-value cache when the user's voice input ends; Based on the pre-filled key-value cache, generate the first character of the response for the speech stream.

Images