VAD-Tolerant and Semantically Coherent Real-Time Voice Interaction System and Method

Abstract

This invention discloses a real-time voice interaction system and method with VAD (Voice Activity Detection) fault tolerance and semantic coherence. Addressing the shortcomings of existing solutions, such as lack of fault tolerance for VAD missed and false detections, semantic fragmentation due to fixed segmentation, and incoherent cross-segment transcription, the front end dynamically segments the speech stream into speech segments with overlapping buffers by combining speech activity detection and semantic integrity prediction, and caches the original audio in a sliding window. The back end uses word-level confidence from ASR (Automatic Speech Recognition) output to determine VAD missed detections. When a missed detection occurs, the audio is retrieved back, merged into an extended segment, and re-transcribed. After dynamic time warping and optimal merging, incremental correction is performed, thus forming a two-way closed loop where downstream confidence feedback drives upstream VAD decision-making. The overlapping buffer also serves as cross-segment context and backtracking material. This invention can automatically compensate for VAD errors, ensure semantic coherence across segments, and improve recognition accuracy and fault tolerance in complex acoustic scenes with low latency.

Description

Technical Field

[0001] This invention relates to the field of real-time audio and video interaction and speech recognition technology, specifically to a real-time voice interaction system and method with VAD fault tolerance and semantic coherence. Background Technology

[0002] AI digital avatars are virtual characters driven by artificial intelligence technology that can interact with users in real time via voice. They are widely used in scenarios such as intelligent customer service, virtual humans, and remote medical consultations. Early solutions generally adopted a serial processing mode of "full voice recording → complete transcription → model generation → full response", which has problems such as high interaction latency, low efficiency in processing long voice messages, and poor human-like interaction effects.

[0003] To reduce latency, existing technologies are increasingly adopting streaming processing solutions that combine speech segmentation with Voice Activity Detection (VAD). VAD detects speech activity and segments the user's speech stream into segments, which are then fed one by one into Automatic Speech Recognition (ASR), Large Language Modeling (LLM), and Text-to-Speech (TTS) models for streaming processing. However, this approach still faces two major challenges in practical engineering implementation:

[0004] First, VAD lacks a system-level fault tolerance mechanism for missed and false detections. In scenarios with weak speech, background noise, rapid word linking, and prolonged speech endings, VAD is prone to problems such as missed speech detection, false silence detection, and false noise detection. Existing solutions only optimize the features and upgrade the model of the VAD algorithm itself, which cannot completely eliminate detection errors and lacks closed-loop compensation methods after errors occur. Once VAD determines an abnormality, it will cause the loss of speech segments and semantic breaks, ultimately leading to incorrect content output by ASR and LLM.

[0005] Secondly, fixed-duration segmentation and cross-segment transcription result in semantic inconsistency. Mainstream segmentation schemes use fixed-duration segmentation of the speech stream, which easily splits complete semantic units into two speech segments. Due to the lack of context at the segment boundaries, the ASR recognition accuracy drops significantly. Furthermore, existing streaming ASR only performs simple text concatenation on the segment results and does not pass the decoder context state between adjacent segments. Cross-segment transcription is prone to problems such as word truncation, repetition, semantic confusion, and errors in the recognition of technical terms.

[0006] At a deeper level, the aforementioned pain points stem from the fact that existing technologies generally follow a unidirectional serial processing paradigm of "VAD detection → speech segmentation → ASR transcription → text concatenation." This paradigm assumes that speech activity detection must be accurate at the signal source in one go. Therefore, long-term improvements in this field have focused on improving the detection accuracy of the VAD model itself, such as optimizing acoustic features, increasing model capacity, and introducing attention mechanisms. However, the unidirectional serial paradigm has inherent and insurmountable flaws: once a VAD misses or falsely detects, the error propagates irreversibly downstream along the data stream, and ASR and LLM can only passively accept the erroneous input; while the downstream ASR stage actually contains rich semantic information and word-level confidence, which could be used to correct the upstream VAD judgment, but cannot be reused under the unidirectional paradigm. Therefore, simply improving the accuracy of the VAD source cannot fundamentally solve the problems of fault tolerance and semantic coherence in complex acoustic scenarios. Summary of the Invention

[0007] To address the shortcomings of existing technologies, such as insufficient reliability of VAD detection, semantic breaks caused by fixed-duration segmentation, inconsistent cross-segment transcription, and lack of system fault tolerance, this invention provides a real-time voice interaction system and method with VAD fault tolerance and semantic coherence. Under the premise of ensuring low interaction latency, it achieves self-detection and self-compensation for VAD missed detection, semantically driven dynamic segmentation, and semantic coherence assurance for cross-segment transcription, comprehensively improving the recognition accuracy, semantic integrity, and system stability in complex acoustic scenarios.

[0008] The real-time voice interaction system with VAD fault tolerance and semantic coherence provided by the present invention includes a front-end acquisition module, a back-end processing module and an AI model module. The AI ​​model module includes an ASR model, which is used to transcribe speech segments in real time and output word-level confidence scores. The front-end acquisition module includes a multi-level VAD detection unit. The front-end acquisition module combines the voice activity detection signal output by the multi-level VAD detection unit with the semantic integrity prediction result to dynamically divide the user's voice stream into voice segments. After attaching an overlapping buffer to the end of each voice segment, the segment is sent to the back-end processing module in real time. The recent original audio is continuously cached in the sliding window voice buffer. The backend processing module obtains the confidence scores of consecutive words at the end of the segment and the difference in confidence scores between adjacent segments from the word-level confidence scores output by the ASR model, and uses this information to reversely determine whether the multi-level VAD detection unit has missed detections. When the backend processing module determines a missed detection, it controls the back-end processing module to extract the missed audio from the sliding window speech buffer, merge it with the adjacent speech segments into an extended segment, and trigger the ASR model to rewrite the extended segment. This forms a two-way closed loop between the multi-level VAD detection unit, the ASR model, and the backend processing module, where downstream transcription confidence feedback drives the upstream multi-level VAD detection unit to make detection decisions. The overlapping buffer serves both as a shared context for transcribing adjacent segments and as material for backtracking missed audio, thus compensating for missed audio without expanding the scope of audio backtracking.

[0009] Furthermore, the back-end processing module includes an ASR confidence analyzer, a missed detection engine, and a fragmented retransmission scheduler; The ASR confidence analyzer calculates the word-level confidence of each speech segment and the confidence difference ΔC between adjacent segments, where ΔC is the absolute value of the difference between the arithmetic mean of the word-level confidence of two adjacent segments. The missed detection engine uses a strategy that triggers when any one of the conditions is met to jointly determine missed detections. These conditions include: The word-level confidence scores of several consecutive words at the end of the segment are all lower than the first threshold; The ΔC exceeds the second threshold, and the average confidence level of the current fragment is lower than that of the previous fragment; The lightweight language model determines that the probability of semantic coherence between adjacent segments is below the third threshold; Dependency parsing determines that the part of speech at the end of a segment is not the final part of speech; After determining a missed detection, the segmented retransmission scheduler extracts the audio of the missed detection interval from the sliding window speech buffer, merges it with the adjacent segments to form an extended segment, and triggers the ASR model to rewrite. The backend processing module aligns the extended segmented transcription result with the original segmented transcription result according to the word sequence using dynamic time normalization. For overlapping words, words with higher word-level confidence are retained, and words that only exist in the extended segmented transcription result are retained, resulting in the deduplicated corrected transcription text. The corrected transcribed text is input into the LLM model of the AI ​​model module, and the LLM model generates updated response text only for the positional increments affected by the correction. The TTS model of the AI ​​model module will discard audio segments with sequence numbers greater than the corrected segment and that have not yet been played, and then regenerate them.

[0010] Furthermore, the front-end acquisition module includes a semantically aware dynamic segmentation unit, which is configured with a semantic integrity prediction model. The semantic-aware dynamic segmentation unit triggers segmentation when any of the following conditions are met: the semantic integrity score reaches a preset threshold; the end of speech is detected; or the collection time reaches the maximum segmentation time. The duration of the overlapping buffer at the end of each speech segment is 200 to 500 milliseconds; The backend processing module uses the longest common subsequence to perform word-level alignment on the transcribed word sequences of adjacent overlapping regions. Common words are selected based on word-level confidence, and non-common words are discarded based on their region, outputting continuous transcribed text without repetition.

[0011] Furthermore, the multi-level VAD detection unit consists of three layers: frame-level VAD, segment-level VAD, and semantic-level VAD. The frame-level VAD is determined based on audio energy, zero-crossing rate, and spectral characteristics; The fragment-level VAD is determined based on a deep learning model; The semantic-level VAD is determined based on the ASR transcription result and semantic coherence; The three-layer detection results are dynamically weighted and fused to output the final judgment; When the semantic-level VAD conflicts with the frame-level VAD or the segment-level VAD, the result of the semantic-level VAD shall prevail, and real-time error correction of the current speech segment shall be triggered.

[0012] Furthermore, the dynamic weighted fusion adaptively allocates the weights of the three-layer detection based on the environmental signal-to-noise ratio: When the ambient signal-to-noise ratio is greater than the first signal-to-noise ratio threshold, the weight of the frame-level VAD is increased to prioritize reducing latency. When the environmental signal-to-noise ratio is between the first and second signal-to-noise ratio thresholds, the weights of the three layers are evenly distributed. When the environmental signal-to-noise ratio is lower than the second signal-to-noise ratio threshold, or when multi-person dialogue features are detected, the weight of the semantic-level VAD is increased to prioritize ensuring recognition accuracy.

[0013] Furthermore, the back-end processing module also includes a closed-loop adaptive parameter tuning unit; The closed-loop adaptive parameter tuning unit maintains a sliding decision window of fixed length and counts the false trigger rate and false detection rate within the window that have been verified by downstream. The closed-loop adaptive parameter tuning unit takes the tuning cost, which is a weighted average of the false trigger rate and the false detection rate, as the optimization target, performs a bounded online update on the first threshold, and constrains the update result within a preset range. The closed-loop adaptive parameter tuning unit maintains its own sliding decision window and first threshold for different acoustic scenarios, and tunes them independently.

[0014] Furthermore, the front-end acquisition module also includes a session-level acoustic profiling unit; The conversation-level acoustic profiling unit accumulates a conversation-level acoustic profile using an exponentially weighted moving average during a single conversation. The conversation-level acoustic profile includes a noise feature signature and a background signal-to-noise ratio baseline. The conversation-level acoustic profiling unit outputs conversation-level weight priors based on the conversation-level acoustic profiling. The final weight of the dynamic weighted fusion is obtained by combining the instantaneous signal-to-noise ratio weighting result with the prior weighting of the session-level weights; During the warm-up phase at the start of a session, only the session-level acoustic profile is accumulated, without applying the session-level weighted prior.

[0015] Furthermore, the backend processing module includes a cross-shard context bridging and consistency verification unit; The cross-segment context bridging and consistency verification unit transmits the decoder hidden state and language model context state of the ASR model between adjacent speech segments. The cross-segment context bridging and consistency verification unit calculates the confidence gating coefficient based on the average confidence level of the last word at the end of the previous segment. The cross-shard context bridging and consistency verification unit weights the transmitted state using the confidence gating coefficient, so that the lower the confidence at the end of the previous shard, the stronger the transmitted state is suppressed, thereby preventing erroneous contexts in the preceding shards from propagating to subsequent shards.

[0016] Furthermore, the fragmented retransmission scheduler executes the backtracking of missed audio using a multi-level strategy with increasing cost: Level 1: Only backtrack the preset short-duration audio at the end of the missed segment, merge and rewrite it; If the confidence level of the first-level re-transcription is still insufficient, proceed to the second level, backtrack and merge complete adjacent segments, and add extended audio for re-transcription; If the second level still fails to meet the standard, proceed to the third level, using high-confidence words as anchors to extend the backtracking interval forward until the anchors are hit; The cumulative number of backtracking attempts is limited to a preset upper limit. Once the upper limit is reached, the current optimal transcription is output, and uncertain regions are marked.

[0017] A real-time voice interaction method for AI digital avatars that combines VAD fault tolerance and cross-slice semantic coherence, applied to the system of any one of claims 1 to 9, comprising: The front-end acquisition module combines the voice activity detection signal and semantic integrity prediction output by the multi-level VAD detection unit to dynamically divide the user's voice stream into voice segments with overlapping buffers and send them in real time, while caching recent original audio in a sliding window voice buffer. The backend processing module calls the ASR model to transcribe speech segments and obtains word-level confidence scores. The back-end processing module uses the confidence level at the end of the segment and the difference in confidence level between adjacent segments to reversely determine whether the multi-level VAD detection unit has missed detection. When a missed detection is determined, the missed audio is retrieved from the sliding window audio buffer, merged with the adjacent segments into an extended segment and rewritten. The rewritten results are then merged in the best order and input into the LLM model to generate an incremental response, which is then streamed back through the TTS model. Thus, the downstream transcription confidence feedback drives the upstream multi-level VAD detection unit and speech segmentation, forming a real-time fault-tolerant two-way closed-loop interaction.

[0018] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention provides a real-time voice interaction system and method with VAD fault tolerance and semantic coherence, and constructs a two-way closed-loop error correction mechanism between VAD and ASR, which can automatically identify and compensate for VAD missed detection and false detection, and significantly improve the system's fault tolerance and stability in scenarios such as complex noise, weak speech, and rapid connected speech.

[0019] 2. This invention provides a real-time voice interaction system and method with VAD fault tolerance and semantic coherence. It adopts semantic-driven dynamic segmentation combined with overlapping buffers and cross-segment context bridging to solve the semantic breaks and cross-segment recognition errors caused by fixed-duration segmentation. The accuracy of cross-segment semantics and professional terminology recognition is significantly improved.

[0020] 3. This invention provides a real-time voice interaction system and method with VAD fault tolerance and semantic coherence. All fault tolerance optimization operations are triggered only on demand in abnormal scenarios, and there is almost no additional overhead in normal interaction scenarios. The end-to-end interaction latency can be stably controlled within 1 to 2 seconds, and it can be directly embedded into existing streaming voice interaction systems.

[0021] 4. This invention provides a real-time voice interaction system and method with VAD fault tolerance and semantic coherence. Closed-loop adaptive tuning and session-level acoustic profiling enable the system to achieve threshold self-convergence and VAD intra-session self-evolution without retraining the model. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a block diagram of the overall system architecture of the present invention; Figure 2 This is a diagram illustrating the two-way closed-loop principle of downstream ASR confidence feedback driving upstream VAD decision-making. Figure 3 This is a schematic diagram of the three-layer decision-making and dynamic weighted fusion of multi-level VAD; Figure 4 This is a closed-loop flowchart for VAD missed detection compensation based on ASR confidence feedback; Figure 5 This is a schematic diagram illustrating the dual purpose of semantically aware dynamic fragmentation and overlapping buffers; Figure 6 This is a schematic diagram of cross-slice context bridging and confidence gating; Figure 7 This is a schematic diagram of the online tuning process for closed-loop adaptive parameters; Figure 8 This is a schematic diagram of the conversation-level acoustic environment profiling construction and VAD self-evolution process; Figure 9 This is a flowchart of a three-level strategy for tiered and progressive compensation for missed detections. 100 - Front-end acquisition module; 110 - Multi-level VAD detection unit; 111 - Frame-level VAD; 112 - Segment-level VAD; 113 - Semantic-level VAD; 120 - Sliding window speech buffer; 130 - Semantic-aware dynamic segmentation unit; 140 - Conversation-level acoustic profiling unit; 200 - Back-end processing module; 210 - ASR confidence analyzer; 220 - Missed detection engine; 230 - Segment retransmission scheduler; 240 - Cross-segment context bridging and consistency verification unit; 250 - Closed-loop adaptive parameter tuning unit; 300 - AI model module; 310 - ASR model; 320 - LLM model; 330 - TTS model. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0025] This invention can be deployed on servers, edge computing devices, or cloud clusters. The front-end acquisition module is compatible with acquisition devices such as computers, mobile phones, and smart terminals, and supports mainstream operating systems such as Windows, Linux, Android, and iOS. The ASR, LLM, and TTS models can be deployed on GPU / CPU clusters, while the lightweight semantic integrity prediction model and VAD unit can run on the terminal side, achieving end-to-cloud collaboration. The front-end and back-end are configured with shard / send segmented transmission interfaces, shard / end voice end signal interfaces, and AUDIO_SHARD audio return interfaces. After system power-on, the front-end loads the frame-level VAD, segment-level VAD models, and semantic integrity prediction model, and starts the sliding window voice buffer. As a specific implementation, the buffering time of the sliding window voice buffer is no less than 5 seconds. The back-end loads the ASR, LLM, and TTS models, and initializes the ASR confidence analyzer, missed detection engine, cross-shard context bridging, and consistency verification unit, opening the aforementioned communication interfaces.

[0026] like Figure 1 As shown, the system of the present invention includes a front-end acquisition module 100, a back-end processing module 200, and an AI model module 300. The front-end acquisition module 100 integrates a multi-level VAD detection unit 110 (including frame-level VAD 111, segment-level VAD 112, and semantic-level VAD 113), a sliding window speech buffer 120, a semantic-aware dynamic segmentation unit 130, and a conversation-level acoustic profiling unit 140. The back-end processing module 200 integrates an ASR confidence analyzer 210, a missed detection engine 220, a segmented retransmission scheduler 230, a cross-segment context bridging and consistency verification unit 240, and a closed-loop adaptive parameter tuning unit 250. The AI ​​model module 300 includes an ASR model 310, an LLM model 320, and a TTS model 330. The front end sends the speech segments to the back end in real time via shard / send. The back end then transfers the text to the AI ​​model and sends the synthesized audio back to the front end for playback via AUDIO_SHARD. Unlike the existing one-way link, the back end extracts evidence from the word-level confidence output by the ASR model 310, which drives the front end VAD and the segmentation unit in reverse, forming a two-way closed loop that runs through the entire link.

[0027] like Figure 2As shown, the forward data flow is as follows: user voice stream passes through voice activity detection, semantic-aware dynamic segmentation, ASR transcription, LLM and TTS generation, and audio playback; the reverse closed loop is as follows: ASR confidence analyzer 210 calculates word-level confidence of the transcription results, the missed detection engine 220 determines missed detections based on this, and the segmentation retransmission scheduler 230, after determining a missed detection, backtracks the missed audio, re-segments and re-transcribes it, and feeds the results back upstream. This invention does not pursue absolute accuracy at the VAD source, but rather corrects upstream missed detections through a reverse closed loop using semantic and confidence evidence from downstream ASR. This is the core difference between this invention and existing unidirectional serial schemes.

[0028] like Figure 3 As shown, the multi-level VAD detection unit 110 includes three layers: frame-level VAD 111, based on a 10-30ms time scale, relies on audio energy, zero-crossing rate, and spectral features for judgment, offering low latency and high sensitivity; segment-level VAD 112, based on a 100-300ms time scale, employs a deep learning model for judgment, achieving high accuracy; and semantic-level VAD 113, based on a 1-3 second time scale, combines ASR transcription results with semantic coherence judgment, offering the highest reliability. The results from the three layers are dynamically weighted and fused to output the final judgment. When there is a conflict between the semantic-level and frame-level or segment-level judgments, the semantic-level result takes precedence and triggers real-time error correction for the current segment.

[0029] The dynamic weighted fusion adaptively allocates weights based on the environmental signal-to-noise ratio (SNR): when the SNR is greater than 20dB (quiet environment), the frame-level, segment-level, and semantic-level weights wframe, wseg, and wsem are 0.5, 0.3, and 0.2 respectively, prioritizing low latency; when the SNR is between 5 and 20dB (medium noise), they are 0.2, 0.4, and 0.4 respectively, balancing latency and accuracy; when the SNR is less than 5dB (high noise) or multi-person dialogue features are detected, they are 0.1, 0.3, and 0.6 respectively, prioritizing accuracy.

[0030] As a specific implementation, the semantic-level VAD 113 determines whether speech has ended through the following three joint methods: First, sentence-end punctuation prediction, using a three-layer feedforward neural network, taking the word embedding concatenation vectors of the current word and the two words before and after it as input, and outputting the sentence-end punctuation probability Ppunc. When the probability Ppunc of a period, question mark, or exclamation mark exceeds the threshold Tpunc (default 0.7), the sentence is determined to have ended. Second, sentence integrity determination, inputting the current transcribed text into the semantic integrity prediction model, and determining semantic integrity when its output semantic integrity score Scomplete exceeds a preset threshold (default 0.85). Third, semantic pause detection, determining the semantic-level speech to have ended when the pause duration between adjacent words exceeds Tpause (default 800 milliseconds) and at least one of the above two methods is true.

[0031] As a specific implementation, when the determination results of the frame-level VAD 111 and the segment-level VAD 112 are inconsistent, the difference in their determination confidence scores Δcf = |cf frame cf seg |, if Δcf is less than the conflict threshold T conflict (Default 0.2), then the frame-level weight is reduced to 0.1 and the fragment-level weight is increased to 0.5; when the semantic-level VAD 113 conflicts with the first two layers, the result of the semantic-level VAD 113 is used as the final judgment basis, and the conflict event is recorded for subsequent model adaptive optimization.

[0032] As a specific implementation method, the real-time error correction handles three types of errors respectively: when the lower-level VAD misjudges the speech to be over, it triggers speech continuation and continues to collect audio; when the lower-level VAD misjudges silence as speech, it truncates redundant invalid audio; when the VAD misjudges noise as speech, it discards the current speech segment and records the noise feature for model adaptive optimization.

[0033] like Figure 4 As shown, the closed-loop process for missed detection compensation is as follows: The ASR model 310 transcribes the speech segments in real time and outputs the word-level confidence score for each word. The word-level confidence score is the posterior probability of each word output by the ASR decoder, with a value ranging from 0 to 1. The ASR confidence analyzer 210 calculates the confidence scores of consecutive words at the end of the segment and the difference in confidence scores between adjacent segments, ΔC, where ΔC = |C avg (Si) C avg (Si 1) As a specific implementation method, suppose the i-th segment contains N words, and their word-level confidence scores are c1, c2, ..., c... N Then C avg (Si) = (c1 + c2 + … + c N ) / N, which is the arithmetic mean of the word-level confidence scores of all words in the segment. The missed detection engine 220 adopts a strategy of triggering if any one of the conditions is met, and combines four rules to jointly determine missed detections: ① The last N consecutive segments tail Words (take N) tail The word-level confidence scores of all terms with a value of 3 are lower than the first threshold Tc (default 0.6), i.e., min(c N 2, c N 1, c N ) <Tc; ② The difference ΔC exceeds the second threshold TΔ (default 0.15), and the mean confidence score of the current fragment is lower than that of the previous fragment, i.e., C avg(Si) <C avg (Si 1); ③ The lightweight language model determines that the semantic coherence probability Pcoherence between adjacent segments is lower than the third threshold Tsem (default 0.5). As a specific implementation method, the lightweight language model adopts a six-layer DistilBERT model with approximately 66 million parameters. It is pre-trained and fine-tuned using medical dialogue corpus. The input and output semantic coherence probability Pcoherence is the concatenated and transcribed text of two adjacent segments. ④ Dependency parsing determines that the part of speech at the end of the segment is a preposition, conjunction, auxiliary word, or other non-ending part of speech.

[0034] After a missed detection is determined, the segmented retransmission scheduler 230 backtracks to extract the audio of the missed detection interval from the sliding window speech buffer 120, merges it with the adjacent segments to form an extended segment, and triggers the ASR model 310 to rewrite it. Then, the extended segment transcription result is aligned with the original segment transcription result by word sequence using dynamic time warping (DTW). The word-level confidence of the two overlapping word positions is compared and the higher one is retained. Words that only exist in the extended segment transcription result are retained to obtain the deduplicated corrected transcription text. Finally, the corrected text is input into the LLM model 320. The LLM only recalculates the positions affected by the correction and incrementally generates the updated response. The TTS model 330 discards audio segments with sequence numbers greater than the corrected segment that have not yet been played and regenerates them. The front end seamlessly switches to play the corrected audio.

[0035] As a specific implementation, the correction is transmitted between units in JSON format as a correction instruction. The fields of the correction instruction include: cmd_type (instruction type, value CORRECTION), shard_id (shard identifier), orig_text (text before correction), corrected_text (text after correction), correction_type (value one of SUBSTITUTION, INSERTION, DELETION, or MERGE), correction_span (containing the start word index start_word_idx and the end word index end_word_idx), confidence_before, confidence_after, and timestamp; the LLM model is 320-dimensional. The system protects the text buffer and the generated word sequence. Upon receiving a correction instruction, it locates the correction interval, replaces the text within the correction_span range with corrected_text, and recalculates the self-attention weights only for the affected positions from the replacement point. It also incrementally generates new word sequences to replace the original subsequent words. At the same time, it marks audio segments that have been sent to TTS but have not yet been played as pending disposal. The TTS model 330 maintains an audio playback queue. Each audio segment carries a sequence number seq_no and a status flag. The status flag can be PENDING, PLAYING, PLAYED, or STALE. Upon receiving a correction instruction, it marks segments with a sequence number greater than the corrected segment and in the PENDING state as STALE, stops playing the currently affected segment, and waits for the LLM model 320 to regenerate before synthesizing new audio segments and inserting them into the playback queue.

[0036] End-to-end example: A user's voice input is "I've been feeling dizzy and nauseous lately." The front-end segments this into S1 = "I've been feeling dizzy lately" (semantic integrity score 0.92, triggering segmentation) and S2 = "and nauseous." ASR initially transcribes S2 as "and also feels like vomiting," where the confidence of "heart" is 0.35 (lower than Tc=0.6), triggering a missed detection: Rule ① passes, Rule ② passes (ΔC=0.45>0.15), Rule ③ passes (semantic coherence probability 0.32<0.5), and Rule ④ passes ("vomit" is a verb, not a final part of speech). Based on the combined results, a missed detection is determined. The fragmented retransmission scheduler extracts the last 200ms of S1 and merges it with S2 to form an extended fragment, which is then rewritten as "and nauseous and want to vomit". The confidence of each word is higher than 0.85. After DTW alignment and confidence-based merging, the final output is "and nauseous and want to vomit". Based on this, the system sends a correction instruction to the LLM and instructs the TTS to be discarded and regenerated, so that the patient finally hears a coherent and accurate response.

[0037] like Figure 5 As shown, the semantic-aware dynamic segmentation unit 130 is configured with a semantic integrity prediction model. Segmentation is triggered when any one of the following three conditions is met: the semantic integrity score reaches a preset threshold, the VAD detects the end of the speech, or the acquisition time reaches the maximum segmentation time (1~3 seconds). An overlap buffer of 200~500ms is added to the end of each speech segment. As a specific implementation, the semantic integrity prediction model employs a 4-layer bidirectional LSTM network with 256 hidden layer dimensions. Its input is a concatenated vector of three features: first, audio acoustic features, composed of 80-dimensional MFCC features and 3-dimensional energy features, linearly mapped to 128 dimensions; second, word embedding sequences from the initial ASR transcription results, using 300-dimensional pre-trained word vectors, linearly mapped to 128 dimensions; and third, sentence integrity probability features based on an n-gram language model, taking the inverse of perplexity and linearly mapped to 64 dimensions. The three features are concatenated to a 320-dimensional vector, encoded by a bidirectional LSTM, and the output of the last hidden layer is taken, then passed through a fully connected layer and activated by a sigmoid function to output a semantic integrity score Scomplete ranging from 0 to 1. The model is trained using manually annotated medical dialogue speech segments with approximately 100,000 training data points. The training objective is to obtain a binary cross-entropy loss L = [y·log(Scomplete) + (1 y)·log(1 Scomplete)], where y is the annotation label, the model has approximately 8 million parameters, and the single-frame inference latency is less than 5 milliseconds.

[0038] The overlapping buffer serves a dual purpose: firstly, it acts as a shared context for transcribing adjacent segments. The backend uses the Longest Common Subsequence (LCS) to perform word-level alignment of the transcribed word sequences in the overlapping areas of adjacent segments, selecting common words based on confidence and discarding non-common words based on their location, outputting continuous transcribed text without repetition. Secondly, it serves as real-time backtracking material for VAD (Voice over Detection) omission compensation, enabling omission compensation to be completed with extremely low additional latency without extensive audio backtracking. It is this coupling between the segment design and the compensation mechanism that allows this invention to significantly reduce compensation latency while ensuring semantic coherence.

[0039] As a specific implementation method, the deduplication and merging adopts an algorithm based on word-level alignment and confidence optimization, including: performing ASR transcription on the overlapping regions of two adjacent segments to obtain the word sequence W. i With W i+1 The longest common subsequence algorithm is used to analyze W. i With W i+1 Perform word-level alignment to identify common word subsequences and their respective non-common words; for each word position in the common word subsequence, compare its position in W. i With W i+1The word-level confidence score is used to retain words with higher confidence scores. For non-common words that exist only in one sequence, if they are located in a non-overlapping region, they are retained; if they are located in an overlapping region and their confidence score is below the overlap threshold T, they are retained. overlap If the value is 0.5 (default), it will be discarded; after merging, the output will be continuous transcribed text without repetition.

[0040] like Figure 6 As shown, the cross-segment context bridging and consistency verification unit 240 transmits the ASR decoder hidden state and language model context state between adjacent speech segments, ensuring that the decoding of the subsequent segment inherits the context of the previous segment. To prevent erroneous context in the preceding segment from propagating to subsequent segments, this invention introduces a confidence gating coefficient α when transmitting context, according to α = sigmoid(k·(C tail Calculate C0), where C tail C0 is the mean of the word-level confidence scores at the end of the previous segment, C0 is the gating center (default 0.6), and k is the gating steepness (default 10). When C... tail When α is high, it approaches 1, and the context is fully inherited; when C tail When α is low, it approaches 0, and the propagation of context is suppressed, thus preventing the backward propagation of erroneous context. Unit 240 also sequentially performs multiple checks on overlapping area edit distance, referential resolution, logical coherence, named entities, and technical terms. After detecting errors, it generates correction instructions and synchronizes them to the LLM.

[0041] like Figure 7 As shown, the closed-loop adaptive parameter tuning unit 250 maintains a sliding decision window with a fixed length of M (default 50), continuously records the most recent M VAD judgment results verified by downstream, and calculates the false trigger rate R within the window. false With the false negative rate R miss With tuning cost J = λ1·R false + λ2·R miss (Default λ1=0.4, λ2=0.6) is the optimization target. The first threshold Tc used by the missed detection engine 220 is updated online in a bounded manner: Tc ← clip(Tc ± η, [0.45, 0.75]), where η is the update step size (default 0.01) and clip is the interval truncation function. When the missed detection rate within the window is too high, Tc is lowered to trigger compensation more aggressively; when the false trigger rate is too high, Tc is raised to suppress false triggers. The system maintains its own sliding window and threshold for different acoustic scenarios and tunes them independently, so that the threshold can adaptively converge during the operation process without repeated manual parameter tuning.

[0042] like Figure 8As shown, the conversation-level acoustic profiling unit 140 continuously accumulates a conversation-level acoustic environment profile Φ using an exponentially weighted moving average (EWMA) during a single conversation. The profile Φ includes noise signature features, a background signal-to-noise ratio baseline, speaker count estimation, and average speech rate. The noise signature features are calculated according to Nsig ← (1 β)·Nsig + β·nt online update (default β=0.1). The scene classifier outputs the session-level weight prior wprior based on the image Φ. The final weight of the multi-level VAD fusion is composed of the instantaneous signal-to-noise ratio weighted result winst and the session-level weight prior wprior according to w = γ·wprior + (1 The γ)·winst combination (default γ=0.6) accumulates images during the initial Nwarm frames (default 30 frames) warm-up phase at the start of the session without imposing priors, thus avoiding cold start errors. This mechanism allows the VAD to evolve automatically within a single session as it accumulates knowledge of the acoustic environment, and this process is entirely online without requiring model retraining.

[0043] like Figure 9 As shown, the fragment retransmission scheduler 230 performs backtracking of missed audio using a multi-level strategy with increasing cost: the first level only backtracks the last 200ms of audio from the missed fragment and merges and rewrites it; if the confidence level of the first-level rewrite is still insufficient, the second level is entered, backtracking and merging complete adjacent fragments and adding approximately 500ms of extended audio for rewriting; if the second level is still insufficient, the third level is entered, using high-confidence words as anchors (default 0.85) to extend the backtracking interval forward until the anchor is hit; the cumulative number of backtrackings is limited to a preset upper limit K (default 3). After reaching the upper limit, the current optimal transcription is output and the uncertain region is marked to avoid infinite compensation loops. This strategy minimizes compensation overhead while ensuring compensation effectiveness.

[0044] The overall real-time voice interaction method of the present invention may include the following steps: Step 1: The user initiates a voice interaction, the front end starts three-layer VAD detection, and simultaneously enables sliding window audio caching and semantic integrity prediction, entering the voice acquisition state; Step 2: The front end dynamically divides the voice into segments according to three rules: semantic integrity, VAD signal and maximum duration. Each segment is attached with an overlapping buffer and sent to the back end in real time via the shard / send interface. Step 3: The backend receives voice segments and calls the ASR model for transcription. At the same time, the ASR confidence analyzer and the missed detection engine run synchronously to monitor VAD missed detections. Step 4: Perform cross-segment transfer of ASR decoder state and language model state, and perform consistency verification and automatic correction on overlapping areas, text semantics and technical terms. Step 5: If a VAD is missed, perform audio backtracking, segment merging, rewriting, and text correction; if no VAD is missed, directly transfer the text segments. Step 6: Stream the verified text segments into the LLM model to incrementally generate responses, and simultaneously send them into the TTS model to generate audio segments; Step 7: The backend sends audio segments back to the frontend in real time via the AUDIO_SHARD interface and plays them one by one, achieving low-latency interaction of speaking and replying simultaneously. Step 8: When the three-layer VAD fusion determines that the speech is finished, the front end sends a shard / end signal. The back end completes the remaining segment processing and final verification, and then outputs the final audio to end this interaction.

[0045] As a specific implementation, the parameters of this invention can take the following values: maximum duration of speech segmentation is 1 to 3 seconds, overlap buffer duration is 200 to 500 milliseconds, and sliding window audio buffer duration is 5 seconds; frame-level detection timescale is 10 to 30 milliseconds, segment-level detection is 100 to 300 milliseconds, and semantic-level detection is 1 to 3 seconds; the word-level confidence lower limit, i.e., the first threshold Tc, defaults to 0.6, the adjacent segment confidence difference threshold TΔ defaults to 0.15, and the semantic integrity score threshold defaults to 0.85. The above parameter values ​​are merely examples and can be adjusted within a reasonable range according to the specific deployment scenario.

[0046] To verify the effectiveness of this invention, the system of this invention was compared and ablation tested with a baseline system without the aforementioned fault-tolerant optimizations under various acoustic environments (including quiet, moderate noise, and high noise scenarios). The results show that this invention can significantly reduce the word error rate and improve the recognition accuracy of cross-segment semantics and technical terms in complex scenarios such as weak speech, background noise, and rapid connected speech. Furthermore, with the synergistic effect of multiple mechanisms, the overall system performance is superior to the simple superposition of the individual mechanisms, demonstrating synergistic gain. In addition, since all fault-tolerant optimization operations are triggered only on demand in abnormal scenarios, the additional processing overhead introduced in normal interaction scenarios is minimal, and the end-to-end interaction latency remains within a range that meets the requirements of real-time interaction.

[0047] The above-described specific embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Various modifications, substitutions, and improvements made by those skilled in the art to the technical solutions of the present invention based on the provided textual description and drawings, without departing from the design concept and spirit of the present invention, should all fall within the scope of protection of the present invention. The scope of protection of the present invention is determined by the claims.

Claims

1. A VAD-tolerant and semantically coherent real-time voice interaction system, characterized in that, It includes a front-end acquisition module (100), a back-end processing module (200) and an AI model module (300), wherein the AI ​​model module (300) contains an ASR model (310). The ASR model (310) is used to transcribe speech segments in real time and output word-level confidence scores; The front-end acquisition module (100) includes a multi-level VAD detection unit (110). The front-end acquisition module (100) combines the voice activity detection signal output by the multi-level VAD detection unit (110) with the semantic integrity prediction result to dynamically divide the user's voice stream into voice segments. After attaching an overlapping buffer to the end of each voice segment, the segments are sent to the back-end processing module (200) in real time. The recent original audio is continuously cached in the sliding window voice buffer (120). The back-end processing module (200) obtains the confidence of consecutive words at the end of the segment and the difference in confidence of adjacent segments from the word-level confidence output by the ASR model (310), and determines in reverse whether the multi-level VAD detection unit (110) has missed detection. When the back-end processing module (200) determines a missed detection, it controls the back-end processing module (200) to extract the missed audio from the sliding window speech buffer (120), merge it with the adjacent speech segments into an extended segment, and trigger the ASR model (310) to rewrite the extended segment. Thus, a two-way closed loop is formed between the multi-level VAD detection unit (110), the ASR model (310), and the back-end processing module (200), whereby the downstream transcription confidence feedback drives the upstream multi-level VAD detection unit (110) to make detection decisions. The overlapping buffer serves both as a shared context for transcribing adjacent segments and as material for backtracking missed audio, thus compensating for missed audio without expanding the scope of audio backtracking.

2. The VAD-tolerant and semantically coherent real-time voice interaction system according to claim 1, characterized in that: The back-end processing module (200) includes an ASR confidence analyzer (210), a missed detection engine (220), and a fragmented retransmission scheduler (230). The ASR confidence analyzer (210) calculates the word-level confidence of each speech segment and the confidence difference ΔC between adjacent segments, where ΔC is the absolute value of the difference between the arithmetic mean of the word-level confidence of two adjacent segments; The missed detection engine (220) uses a strategy that triggers when any one of the conditions is met to jointly determine missed detections. The conditions include: The word-level confidence scores of several consecutive words at the end of the segment are all lower than the first threshold; The ΔC exceeds the second threshold, and the average confidence level of the current fragment is lower than that of the previous fragment; The lightweight language model determines that the probability of semantic coherence between adjacent segments is below the third threshold; Dependency parsing determines that the part of speech at the end of a segment is not the final part of speech; After determining that a missed detection has been detected, the segmented retransmission scheduler (230) extracts the audio of the missed detection interval from the sliding window audio buffer (120), merges it with the adjacent segments into an extended segment, and triggers the ASR model (310) to rewrite it. The back-end processing module (200) aligns the extended segmented transcription result with the original segmented transcription result according to the word sequence using dynamic time normalization. For overlapping words, words with higher word-level confidence are retained, and words that only exist in the extended segmented transcription result are retained, resulting in the deduplicated corrected transcription text. The corrected transcribed text is input into the LLM model (320) of the AI ​​model module (300), and the LLM model (320) generates an updated response text only for the positional increment affected by the correction; The TTS model (330) of the AI ​​model module (300) will discard audio segments with serial numbers greater than the corrected segment and that have not yet been played and then regenerate them.

3. The VAD-tolerant and semantically coherent real-time voice interaction system according to claim 1, characterized in that: The front-end acquisition module (100) includes a semantic-aware dynamic segmentation unit (130), which is configured with a semantic integrity prediction model. The semantically aware dynamic fragmentation unit (130) triggers fragmentation when any of the following conditions are met: The semantic integrity score reaches the preset threshold; The end of the voice recording was detected; or the recording duration reached the maximum segment duration. The duration of the overlapping buffer at the end of each speech segment is 200 to 500 milliseconds; The back-end processing module (200) uses the longest common subsequence to perform word-level alignment on the transcribing word sequences of adjacent overlapping regions. It selects the best common words based on word-level confidence and discards non-common words based on their region, outputting non-repeating cross-regional continuous transcribing text.

4. The VAD-tolerant and semantically coherent real-time voice interaction system according to claim 1, characterized in that: The multi-level VAD detection unit (110) consists of three layers: frame-level VAD (111), segment-level VAD (112), and semantic-level VAD (113). The frame-level VAD (111) is determined based on audio energy, zero-crossing rate, and spectral characteristics; The fragment-level VAD(112) is determined based on a deep learning model; The semantic-level VAD (113) is determined based on the ASR transcription result and semantic coherence. The three-layer detection results are dynamically weighted and fused to output the final judgment; When the semantic-level VAD (113) conflicts with the frame-level VAD (111) or the segment-level VAD (112), the result of the semantic-level VAD (113) shall prevail, and real-time error correction of the current speech segment shall be triggered.

5. The VAD-tolerant and semantically coherent real-time voice interaction system according to claim 4, characterized in that, The dynamic weighted fusion adaptively allocates the weights of the three detection layers according to the environmental signal-to-noise ratio: When the environmental signal-to-noise ratio is greater than the first signal-to-noise ratio threshold, the weight of the frame-level VAD (111) is increased to prioritize reducing latency; When the environmental signal-to-noise ratio is between the first and second signal-to-noise ratio thresholds, the weights of the three layers are evenly distributed. When the environmental signal-to-noise ratio is lower than the second signal-to-noise ratio threshold, or when multi-person dialogue features are detected, the weight of the semantic-level VAD (113) is increased to prioritize the recognition accuracy.

6. The VAD-tolerant and semantically coherent real-time voice interaction system according to claim 2, characterized in that: The back-end processing module (200) also includes a closed-loop adaptive parameter tuning unit (250); The closed-loop adaptive parameter tuning unit (250) maintains a sliding decision window with a fixed length and counts the false trigger rate and false detection rate within the window that have been verified by downstream. The closed-loop adaptive parameter tuning unit (250) takes the tuning cost, which is composed of the false trigger rate and the missed detection rate, as the optimization target, performs a bounded online update on the first threshold, and constrains the update result within a preset range. The closed-loop adaptive parameter tuning unit (250) maintains its own sliding decision window and first threshold for different acoustic scenarios and tunes them independently.

7. The VAD-tolerant and semantically coherent real-time voice interaction system according to claim 4, characterized in that: The front-end acquisition module (100) also includes a conversation-level acoustic profiling unit (140). The conversation-level acoustic profiling unit (140) accumulates a conversation-level acoustic profile in an exponentially weighted moving average manner during a conversation. The conversation-level acoustic profile includes a noise feature signature and a background signal-to-noise ratio baseline. The conversation-level acoustic profiling unit (140) outputs conversation-level weight priors based on the conversation-level acoustic profiling; The final weight of the dynamic weighted fusion is obtained by combining the instantaneous signal-to-noise ratio weighting result with the prior weighting of the session-level weights; During the warm-up phase at the start of a session, only the session-level acoustic profile is accumulated, without applying the session-level weighted prior.

8. The VAD-tolerant and semantically coherent real-time voice interaction system according to claim 1, characterized in that: The backend processing module (200) includes a cross-shard context bridging and consistency verification unit (240). The cross-segment context bridging and consistency verification unit (240) transmits the decoder hidden state and language model context state of the ASR model (310) between adjacent speech segments; The cross-segment context bridging and consistency verification unit (240) calculates the confidence gating coefficient based on the mean of the word-level confidence at the end of the previous segment; The cross-shard context bridging and consistency verification unit (240) weights the transmitted state with the confidence gating coefficient, so that the lower the confidence at the end of the previous shard, the stronger the transmitted state is suppressed, thereby preventing the erroneous context in the preceding shard from propagating to the subsequent shard.

9. The VAD-tolerant and semantically coherent real-time voice interaction system according to claim 2, characterized in that, The segmented retransmission scheduler (230) performs backtracking of missed audio using a multi-level strategy with increasing cost: Level 1: Only backtrack the preset short-duration audio at the end of the missed segment, merge and rewrite it; If the confidence level of the first-level re-transcription is still insufficient, proceed to the second level, backtrack and merge complete adjacent segments, and add extended audio for re-transcription; If the second level still fails to meet the standard, proceed to the third level, using high-confidence words as anchors to extend the backtracking interval forward until the anchors are hit; The cumulative number of backtracking attempts is limited to a preset upper limit. Once the upper limit is reached, the current optimal transcription is output, and uncertain regions are marked.

10. A VAD-tolerant and semantically coherent real-time voice interaction method, applied to any one of claims 1 to 9, characterized in that, include: The front-end acquisition module (100) combines the voice activity detection signal and semantic integrity prediction output by the multi-level VAD detection unit (110) to dynamically divide the user's voice stream into voice segments with overlapping buffers and send them in real time, while caching the recent original audio in the sliding window voice buffer (120). The backend processing module (200) calls the ASR model (310) to transcribe speech segments and obtain word-level confidence scores; The back-end processing module (200) determines whether the multi-level VAD detection unit (110) has missed detection based on the confidence level at the end of the segment and the difference in confidence level between adjacent segments. When a missed detection is determined, the missed audio is retrieved from the sliding window audio buffer (120), merged with the adjacent segments into an extended segment and rewritten. The rewritten results are merged in the best order and then input into the LLM model (320) to generate an incremental response. The audio is then streamed back through the TTS model (330). Thus, the downstream transcription confidence feedback drives the upstream multi-level VAD detection unit (110) and speech segmentation to form a real-time fault-tolerant two-way closed-loop interaction.

Images