Low-delay streaming voice interaction system with interruption processing function

Abstract

The invention provides a low-delay streaming voice interaction system with an interruption processing function, and relates to the technical field of artificial intelligence. Necessary preprocessing and acoustic feature extraction are performed through a real-time acoustic processing module, and distortion and complex environmental noise introduced by an interaction channel are resisted through a robustness enhancement technology; the streaming acoustic decoding module performs acoustic modeling, language model application and decoding in real time in parallel, and outputs an ultra-low-delay text transfer result stream; the real-time acoustic processing module is combined with a signal processing technology and is responsible for detecting user voice activity in a high-precision and ultra-low-delay manner, and particularly judging the real-time voice activity state of a user through the user voice activity in an AI voice playing period; an efficient and low-delay bidirectional streaming network transmission mode is adopted between modules of the system and between the modules and a communication platform, and it is ensured that audio streams, acoustic feature streams, text streams and control signals can be transmitted and processed in real time with extremely low end-to-end delay.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and in particular to a low-latency streaming voice interaction system with an interruption processing function. Background Art

[0002] As a core AI technology for human-computer interaction, speech recognition (ASR) has been widely applied in various fields. Traditional ASR systems consist of acoustic models, pronunciation dictionaries, and language models. Deep learning is driving the development of end-to-end (E2E) ASR models, integrating multiple models and improving performance. However, current applications in intelligent interaction scenarios still face numerous challenges: Robustness issues in complex interactive channel environments: Networks are subject to channel influences such as bandwidth limitations, coding distortion, and packet loss. User interaction environments are diverse and uncontrollable, and they are subject to background noise, reverberation, echo, and leakage. This significantly reduces the recognition accuracy of existing ASR models.

[0003] Ultra-low latency bottlenecks for real-time interaction: Natural and smooth human-computer dialogue requires ASR systems to have extremely low end-to-end latency. Traditional batch or streaming processing introduces noticeable pauses in the conversation, impacting the user experience. Intelligent interaction places strict demands on first-word latency and real-time text output.

[0004] Complex multi-speaker conversation processing and user interruption detection challenges: AI interactions involve two speakers, the AI ​​and the user. The system must accurately distinguish between voices and support user interruptions at any time. Existing ASR systems struggle to accurately detect user voices when the AI ​​is playing back speech, limiting the naturalness and efficiency of the interaction.

[0005] Inadequate adaptability to individual user differences: AI interaction users are diverse, with varying accents, speaking speeds, volume, and word usage. Existing ASR systems struggle to adapt quickly and in real time to specific user language characteristics, impacting recognition accuracy. Summary of the Invention

[0006] To solve the above problems, the present invention provides a low-latency streaming voice interaction system with interruption processing function, and proposes an innovative ASR technical solution. In particular, through a carefully designed interruption processing mechanism, it realizes the user voice recognition function with high robustness, ultra-low latency, real-time streaming processing and support for natural interruptions in AI intelligent interaction, thereby significantly improving the overall performance of the intelligent interaction system and the user interaction experience.

[0007] To achieve the above object, the technical solution adopted by the present invention is to provide a low-latency streaming voice interaction system with an interruption processing function, comprising: The voice input module is used to collect the user's original digital audio stream and send it to the real-time acoustic processing module; A real-time acoustic processing module, configured to pre-process the original digital audio stream and extract real-time acoustic features based on a deep learning model to obtain a real-time acoustic feature sequence stream; A streaming acoustic decoding module is used to construct a causal streaming architecture acoustic model based on Transducer through the real-time acoustic feature sequence stream and output a real-time text transcription result stream; A voice activity and interruption detection module is configured to determine the user's real-time voice activity status based on the user's original digital audio stream and the audio signal played in real time by the system. The real-time voice activity status includes whether the user is speaking or muted. When the voice activity and interruption detection module determines that the user is speaking, it outputs an interruption event signal and sends the interruption event signal to the dialogue management module. A dialogue management module is configured to perform intent recognition based on the real-time text transcription result stream and context to understand the user's intent; upon receiving the interruption event signal, the dialogue management module immediately sends a stop instruction to the voice output module to forcibly terminate the playback of the current voice interaction information; The voice output module is used to output a complete call transcription text with a timestamp and user identifier, and output real-time interactive voice based on the complete call transcription text through an audio player.

[0008] Preferably, the original digital audio stream is pulse code modulation (PCM) data with a sampling rate of 8 kHz or 16 kHz.

[0009] Preferably, the specific steps of extracting real-time acoustic features include: The continuously input raw digital audio stream is divided into short time frames of fixed duration, and the acoustic feature vector of each frame is extracted by sliding with a fixed step length smaller than the fixed duration.

[0010] Preferably, the deep learning model receives the noisy original digital audio stream of the user and the AI ​​playback audio reference signal as input, trains the deep learning model to predict and suppress the noise in the user's original digital audio stream and the leakage and echo components of the AI ​​audio, extracts the noise-free user voice signal or its corresponding acoustic features, and obtains a real-time acoustic feature sequence stream.

[0011] Preferably, the Transducer-based causal streaming architecture acoustic model is an ASR model, which only relies on information of all current and previous input frames or blocks when processing the current input frame or a small block of features.

[0012] More optimally, the streaming decoding algorithm used by the system achieves strict synchronization with the ASR model output frequency and low-latency output through the following collaborative strategies, including: The decoder in the ASR model receives the real-time probability stream output by the acoustic model and combines it with the streaming language model information to update the decoding status and search for the optimal text path in real time in a frame-synchronous or block-synchronous manner. The decoder adopts an aggressive decoding strategy with a small initial beam width. Once the confidence of a certain path reaches a preset threshold, it immediately outputs part of the text. The real-time text transcription result stream output during the decoding process is dynamically updated and corrected by the decoder until the user mutes the voice.

[0013] Preferably, the voice activity and interruption detection module includes a real-time voice activity detection submodule and an interruption processing submodule, wherein the user voice detection submodule is used to determine whether there is a voice stream input from the user end in the user audio; and the interruption processing submodule is used to output an interruption event signal; The voice activity detection submodule includes a dual-input neural network model: the dual-input neural network model is used to predict the probability of the presence of user voice, by suppressing or ignoring components in the user audio that are highly correlated with the AI ​​reference audio, and only responding to the user's input voice.

[0014] More preferably, the logic of the interruption processing submodule outputting the interruption event signal is as follows: When the system is playing voice, if the voice activity and interruption detection module detects user voice activity with high confidence, it immediately triggers a high-priority interruption event signal and sends it to the dialogue management module via a two-way streaming network transmission. The generation delay of the interruption event signal is less than 100 milliseconds, which is much faster than the time required for the streaming acoustic decoding module to output the first word, ensuring that the system responds in a timely manner and immediately stops outputting real-time interactive voice.

[0015] Preferably, the voice activity and interruption detection module is further configured to detect voice activity and identify whether the voice stream belongs to the target user, specifically in the following steps: By embedding the user voiceprint extracted in real time and comparing it with the pre-stored user voiceprint or the user voiceprint obtained through rapid learning in the early stage of intelligent interaction, the start and end timestamps of the user voice segment are accurately marked, and the user is identified; the speakers of the detected non-AI and non-main user voice segments are distinguished and different speaker IDs are assigned; when voice overlap is detected between different speakers on the user side, real-time voice separation technology is used to separate the overlapping user voice streams.

[0016] The beneficial effects of the present invention are: Excellent robustness and environmental adaptability: Utilizing real-time acoustic processing technology optimized for interactive audio characteristics and complex background noise (including AI leakage), combined with blind source separation and speech de-reverberation techniques, the system significantly improves speech recognition accuracy and stability in complex call environments, breaking through the performance bottleneck of traditional ASR.

[0017] Extremely low latency and real-time streaming capabilities: An optimized causal streaming ASR model architecture and efficient streaming decoding algorithms ensure extremely low end-to-end latency from user voice input to text output, reducing time-to-first-word (TTFW) to under 200 milliseconds. This technology supports real-time, continuous text streaming, eliminating pauses in conversation and improving responsiveness. Continuous optimization of the model architecture and inference technology reduces computing resource consumption and supports efficient deployment.

[0018] Support for natural and smooth user interruptions: The innovatively designed real-time voice activity detection and interruption processing module detects user voice activity with high sensitivity and ultra-low latency (<100 milliseconds) during AI playback, identifies interruptions, and promptly signals the dialogue management module. This interruption detection mechanism, which combines acoustic confidence, language model probability, lexical information, and prosodic feature analysis, enables the AI ​​system to instantly stop the current playback and process the user's speech, providing a smooth interactive experience that aligns with human habits.

[0019] Enhanced adaptability to individual user differences and dialects: The model, trained based on data on diverse accents, speaking speeds, and dialects, has the ability to quickly adapt online with a small number of samples, more accurately and robustly identifying user voices with different accents, speaking speeds, and common dialects, improving service capabilities for a wide range of user groups.

[0020] Improved Complex Conversation Handling: Accurately distinguishes AI from user voices, effectively handles user interruptions, and enhances the performance and stability of AI interaction systems in complex conversational scenarios (including interruptions and chat interruptions). This system also features sophisticated multi-speaker processing, capable of handling any number and overlap of user voices and overlapping portions, improving the accuracy and stability of phrase-level speaker differentiation.

[0021] High-efficiency integrated system and two-way streaming transmission: Robust acoustic processing, ultra-low-latency streaming ASR, high-precision interruption detection, and user adaptability are integrated into a unified real-time processing pipeline. An efficient two-way streaming network transmission mechanism is used to optimize system efficiency and data transmission latency, meeting the high concurrency and high real-time requirements of AI interactive systems.

[0022] Significantly improve user experience: This system provides high-precision, ultra-low latency voice interaction capabilities that support natural interruptions and adapt to individual user differences and dialects. This makes AI interactions more human, intelligent, and easy to use, significantly increasing end-user acceptance and satisfaction. It integrates non-verbal information recognition and sentiment analysis to provide richer information dimensions. End-to-end voice-to-intention or voice-to-action capabilities further reduce overall response latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a framework diagram of a low-latency streaming voice interaction system with an interruption processing function according to the present invention.

[0024] Figure 2 This is a framework diagram of the voice activity and interruption detection module of the present invention. DETAILED DESCRIPTION

[0025] See also Figure 1-2 As shown, a low-latency streaming voice interaction system with an interruption processing function is provided, including: The voice input module is used to collect the user's original digital audio stream and send it to the real-time acoustic processing module; A real-time acoustic processing module, configured to pre-process the original digital audio stream and extract real-time acoustic features based on a deep learning model to obtain a real-time acoustic feature sequence stream; The Voice Activity and Interruption Detection module receives raw user-side audio streams from the AI ​​interaction platform, performs necessary preprocessing and acoustic feature extraction, and implements a series of robustness enhancement techniques to effectively combat distortion and complex environmental noise introduced by the interactive input channel (including background noise and leakage / echo of AI-played voice). This module utilizes deep learning models and signal processing technologies pre-trained on large-scale telephone communication and noise data.

[0026] a) Interactive Input Channel Feature Compensation and Normalization: This system performs real-time analysis of user input audio and applies signal processing techniques to compensate for frequency response distortion and bandwidth limitations introduced by the interactive input channel. For 8kHz narrowband audio, super-resolution or bandwidth extension techniques are used to convert it into a wideband feature representation. Furthermore, real-time volume normalization is performed to account for differences in call volume between users.

[0027] b) Real-time Framing and Feature Extraction: The continuous input audio stream is segmented into short frames of fixed length (25 milliseconds). The acoustic feature vector (40-dimensional Mel filter bank features (MFE)) of each frame is extracted with a fixed step size (10 milliseconds). This process is designed to be strictly causal, using only information from the current frame and previous frames to ensure low latency and real-time processing.

[0028] c) Deep Learning Real-Time Noise Suppression and Speech Enhancement: This module utilizes a real-time speech enhancement or noise suppression algorithm based on deep neural networks. This algorithm employs a deep learning variant based on time-frequency masking or spectral subtraction. To address leakage or echo in AI-played speech, a dual-input single-output (DISO) or multiple-input single-output (MISO) model is employed. The model receives a noisy user audio stream and an AI-played audio reference signal (or its features) as input. The training objective is to predict and suppress the noise in the user audio and the leakage / echo components of the AI ​​audio, thereby extracting a purer user speech signal or its corresponding acoustic features. The enhancement process is designed with strict causal processing and low computational complexity to ensure real-time performance. The system is capable of further improving robustness in complex acoustic environments and far-field sound by integrating technologies such as blind source separation and speech de-reverberation.

[0029] d) Online Feature Normalization and Environment Adaptation: This approach uses real-time, online calculations of mean and variance to normalize acoustic features (online cepstral mean variance normalization (CMVN)), dynamically reducing variations in feature distribution caused by different call environments or speakers. This approach integrates real-time acoustic environment estimation methods based on statistical models or deep learning, and dynamically adjusts feature normalization parameters or employs environment-dependent feature representations based on the estimation results.

[0030] A streaming acoustic decoding module is used to construct a causal streaming architecture acoustic model based on Transducer through the real-time acoustic feature sequence stream, output a real-time text transcription result stream, and send the real-time text transcription result stream to the speech output module; The acoustic model of the Transducer-based causal streaming architecture is an ASR model (including an ASR modeling and decoding engine). After receiving the robustness-enhanced acoustic feature stream, the streaming acoustic decoding module performs acoustic modeling, language model application, and decoding in real time and in parallel, outputting an ultra-low-latency text transcription result stream. This engine is the ASR core of the present invention, and its design goal is to achieve extremely low first-letter (word) latency (TTFW <200 milliseconds, preferably <150 milliseconds) and continuous real-time text output to support smooth conversation turns. The core model of this engine is based on an end-to-end deep learning model pre-trained on large-scale speech and text data, and is deeply optimized for streaming and low latency.

[0031] a) Acoustic Model Based on Transducer or Other Causal Streaming Architectures: The core of the engine utilizes an end-to-end streaming ASR model, preferably one based on a causal streaming architecture using Transducers (RNN Transducers). This model is designed to be strictly causal. When processing the features of the current input frame or a small chunk, it relies only on information from the current and previous input frames or chunks, and never on future information. The acoustic model receives a stream of acoustic features and outputs a stream of probability distributions for the corresponding ASR modeling units (which can be characters, subwords, or words) in real time at a fixed frequency (after processing a certain number of frames or chunks).

[0032] b) Efficient streaming inference and computational graph optimization: The model's inference process runs on a specially optimized inference engine, fully leveraging the parallel computing capabilities of CPUs, GPUs, or dedicated AI accelerator chips. The model's computational graph has been deeply optimized to support efficient chunk-based processing, minimizing the processing latency of each input feature block. Model compression technologies such as pruning and quantization (FP16 or INT8 quantization) are applied to further improve inference speed and reduce computing resource consumption, making it suitable for high-concurrency AI interaction scenarios. The system continuously optimizes model architecture and inference technology to reduce the model's computational complexity and memory usage, supporting deployment in resource-constrained environments.

[0033] c) Ultra-low latency streaming decoding algorithm: This algorithm employs an efficient streaming decoding algorithm synchronized with the output frequency of the streaming acoustic model. It employs an improved variant of synchronous beam search or greedy decoding. The decoder receives the real-time probability stream output by the acoustic model and, combined with information from the streaming language model, updates the decoding state and searches for the optimal text path in real time, either frame-synchronized or block-synchronized. To achieve ultra-low time-to-flow (TTFW), an aggressive decoding strategy is employed, using a small initial beam width. Once the confidence level of a path reaches a preset threshold, partial text is output (an early exit mechanism). Lattice generation and backtracking are optimized to reduce latency. These optimizations aim to reduce the time-to-flow (TTFW) from the time the user stops speaking to the first word output to below 200 milliseconds, preferably close to 100 milliseconds. The decoding process continues, and the output text stream can be dynamically updated and corrected until the user mutes or the call ends.

[0034] d) Integrated or Collaborative Streaming Language Model with Personalization and Dialect Adaptation: ASR models typically include an integrated language model based on the Transducer's Prediction Network. This language model provides real-time lexical and grammatical constraints, helping the decoder select the most appropriate word sequence from ambiguous acoustic information. Language model inference is optimized to support low-latency streaming queries. To improve adaptability to individual user differences and dialects, the language model and / or acoustic model are trained on a large-scale corpus containing data on various accents, speaking rates, and dialects, enabling preliminary recognition capabilities for dialects and different accents. Initially, the system can use real-time user voiceprints or preliminary recognition results to identify user characteristics or possible dialect regions. It can then dynamically load or adjust language model or acoustic model parameters tailored to these characteristics or dialects (through adaptive or personalized technologies) to improve recognition accuracy. Furthermore, conversation context (such as historical conversation content, business keywords, and user tags in the CRM system) can be used to dynamically bias the language model or decoder, increasing the probability of recognizing specific words. The system has the ability to quickly adapt online to a small number of speakers, accents, and specific dialects, and can quickly improve recognition accuracy when facing new users.

[0035] A voice activity and interruption detection module is configured to determine the user's real-time voice activity status based on the user's original digital audio stream and the audio signal played in real time by the system. The real-time voice activity status includes whether the user is speaking or muted. When the voice activity and interruption detection module determines that the user is speaking, it outputs an interruption event signal and sends the interruption event signal to the dialogue management module. The Voice Activity and Interruption Detection module consists of a real-time voice activity detection submodule and an interruption handling submodule. This module is responsible for detecting user voice activity with high precision and ultra-low latency, particularly during AI playback (i.e., when the user interrupts the AI). This module utilizes a deep learning model pre-trained on large-scale data and combines it with signal processing techniques.

[0036] Acoustic model based on Transducer or other causal streaming architectures: The engine core utilizes an end-to-end streaming ASR model, preferably one based on a causal streaming architecture using Transducer (RNN Transducer). This model is designed to be strictly causal. When processing the features of the current input frame or a small chunk, it relies only on information from the current and previous input frames or chunks, and never on future information. The acoustic model receives a stream of acoustic features and outputs a stream of probability distributions for the corresponding ASR modeling units (which can be characters, subwords, or words) in real time at a fixed frequency (for every number of frames or chunks processed).

[0037] The model's inference process runs on a specially optimized inference engine, fully leveraging the parallel computing capabilities of CPUs, GPUs, or dedicated AI accelerator chips. The model's computational graph has been deeply optimized to support efficient chunk-based processing, minimizing the processing latency of each input feature block. Model compression techniques such as pruning and quantization (FP16 or INT8 quantization) further improve inference speed and reduce computing resource consumption, making it suitable for high-concurrency AI interaction scenarios. The system continuously optimizes the model architecture and inference technology to reduce the model's computational complexity and memory usage, supporting deployment in resource-constrained environments.

[0038] The decoder in the ASR model receives a real-time probability stream from the acoustic model and employs an efficient streaming decoding algorithm synchronized with the output frequency of the streaming acoustic model. It uses an improved variant of synchronous beam search or greedy decoding, combined with information from the streaming language model, to update the decoding state and search for the optimal text path in real time, either frame-synchronized or block-synchronized. To achieve ultra-low time-to-flow (TTFW), an aggressive decoding strategy is employed, using a small initial beam width. Once the confidence level of a path reaches a preset threshold, partial text is output (an early exit mechanism). Lattice generation and backtracking are optimized to reduce latency. These optimizations aim to reduce the time-to-flow (TTFW) from the time the user stops speaking to the first word output to below 200 milliseconds, preferably close to 100 milliseconds. The decoding process continues, and the output text stream is dynamically updated and corrected until the user mutes or the call ends.

[0039] The real-time text transcription result stream output during the decoding process is dynamically updated and corrected through the subsequent incoming acoustic features, language model constraints, and the continuous evolution of the decoder state until the user mutes.

[0040] The streaming decoding algorithm used by the system achieves strict synchronization with the ASR model output frequency and low-latency output through the following collaborative strategies: (1) Real-time incremental decoding mechanism: The decoder uses the frame or block output by the acoustic model as the basic processing unit and dynamically executes the following at each time step: Receive the acoustic probability vector of the current time step; Fusion streaming language model provides real-time context vectors; Expand the current active path set and perform pruning, retaining the top-K paths with the highest scores (where K ≤ 5).

[0041] (2) Confidence-driven instant submission strategy: Define the comprehensive index of path confidence: Where: P AM : The posterior probability of the current acoustic model output; P LM : Sequence score provided by the language model; T: confidence compensation factor based on time decay; α, β, γ: Represent the dynamically adjusted weight coefficients. When any path satisfies C>θcommit (θcommit∈[0.8,0.95]): Immediately submit the unoutput token sequence (such as subword or word fragment) corresponding to the path; Lock the prefix of the submitted text, and subsequent decoding will only optimize the unlocked suffix part.

[0042] (3) Dynamic backtracking correction mechanism for submitted text fragments: Cache the acoustic features and language model status of the most recent N frames (N=3~10); When the new input causes the historical path probability change rate |ΔP|>θrollback, the following actions are triggered: (4) Collaborative optimization with the interruption handling function.

[0043] Through the synergy of the above strategies: Achieve extremely low output latency of ≤200ms: high threshold θcommit ensures that the word can be committed before it ends; Support for real-time user interruptions: When an interruption event is detected, the path cache generated based on the old audio is immediately cleared; Ensuring final transcription accuracy: A dynamic backtracking mechanism compensates for early aggressive output errors.

[0044] Note: Specific parameters (including beam width K, commit threshold θcommitθcommit, number of backtracking frames N, etc.) need to be verified and tuned based on the target domain speech dataset; its engineering implementation can be adaptively optimized based on a general streaming decoding framework (such as the gradient search algorithm of RNN-T or the prefix beam search algorithm of CTC).

[0045] ASR models typically include an integrated language model based on the Transducer's Prediction Network. This language model provides real-time lexical and grammatical constraints, helping the decoder select the most appropriate word sequence from ambiguous acoustic information. The language model's inference is optimized to support low-latency streaming queries. To improve adaptability to individual user differences and dialects, the language model and / or acoustic model are trained on a large corpus containing data on various accents, speaking rates, and dialects, enabling preliminary recognition of dialects and different accents. Initially, the system can use real-time user voiceprints or preliminary recognition results to identify user characteristics or possible dialect regions. It can then dynamically load or adjust language model or acoustic model parameters tailored to these characteristics or dialects (through adaptive or personalized technologies) to improve recognition accuracy. Furthermore, conversation context (such as historical conversation content, business keywords, and user tags in the CRM system) can be used to dynamically bias the language model or decoder, increasing the probability of recognizing specific words. The system has the ability to quickly adapt online to a small number of speakers, accents, and specific dialects, and can quickly improve recognition accuracy when facing new users.

[0046] The interruption handling submodule is a key innovation of this invention, specifically designed for complex AI interaction scenarios. It is responsible for detecting user voice activity with high precision and ultra-low latency, particularly during AI playback (i.e., when a user interrupts the AI). This module utilizes a deep learning model pre-trained on large-scale data and combines it with signal processing techniques.

[0047] Input and output: Input: The original audio stream from the user side, and the audio signal currently being played by the AI ​​or its corresponding acoustic features as a reference.

[0048] Output: Real-time user voice activity status (user speaking / user muted). This detection is highly sensitive and robust, and can accurately judge even in background noise and overlap with AI voice.

[0049] An event signal indicating that the user is interrupting the AI. The trigger speed of this signal is designed to be much faster than the first word output of the ASR to ensure that the AI ​​can respond in time.

[0050] The start and end timestamps of the user's voice segment.

[0051] The system identifies and timestamps detected non-AI, non-primary user voice segments and can handle the presence of other human speakers on the user side and any overlap between them. It also offers more sophisticated multi-speaker processing capabilities, including speaker diarization (SD) for detected non-AI, non-primary user voice segments, assigning different speaker IDs. It also detects overlap between human speakers on the user side and attempts to separate overlapping user voice streams using real-time speech separation technology, or employs a joint recognition model optimized for multi-speaker scenarios. This multi-speaker processing capability can handle speech with any number of speakers and any overlap ratio.

[0052] The core technology and workflow of the interruption processing submodule: a) Highly Sensitive and Robust VAD: This system utilizes a voice activity detector (VAD) based on a deep learning model. This VAD model, trained on a large amount of noisy environment and phone audio data, accurately distinguishes user speech from background noise. This VAD model is optimized for high sensitivity to low-energy speech, weak signals at the onset of speech, and speech activity in noise, minimizing missed detections and delays.

[0053] b) Core interruption detection mechanism based on AI reference signals: This module receives both the user audio stream and the AI ​​playback audio reference signal. Its core innovation lies in determining whether non-AI-sourced voice activity, namely the user's own voice, is present in the user audio with high precision and ultra-low latency, even when AI playback causes AI voice leakage or echo in the user audio. This is achieved by employing a dual-input neural network model: the model receives a real-time feature sequence of the user audio and a real-time feature sequence of the AI ​​reference audio as input, and is trained to predict the probability of the user's voice being present. Through learning, the model effectively suppresses or ignores components in the user audio that are highly correlated with the AI ​​reference audio (i.e., AI voice leakage or echo), and responds only to voice components unrelated to the AI ​​reference (i.e., the user's voice). This is similar to implementing acoustic echo cancellation (AEC) or speech separation at the feature or model level, but the goal is to detect user voice activity rather than completely separate the speech.

[0054] c) Ultra-low-latency interrupt signal triggering logic: The interrupt signal is triggered based on the output of the real-time voice activity detection and interruption processing module. When the AI ​​system is playing audio and this module detects user voice activity with high confidence (configurable threshold), it immediately triggers a high-priority "user interruption event signal" and transmits it to the dialogue management module via a bidirectional streaming network. The interrupt signal generation latency (from the start of the user speaking to the signal being transmitted) is designed to be extremely low, less than 100 milliseconds, significantly faster than the time required for the ASR engine to output the first word, ensuring that the dialogue management system can react promptly and immediately stop AI playback. The interrupt signal is triggered based on the instantaneous output confidence of the VAD model or by combining the VAD output of several consecutive frames to balance sensitivity and false trigger rate. The system combines the acoustic confidence output of the ASR engine, language model probabilities, initially recognized vocabulary information, and even analysis of prosodic features such as speech rate and intonation to more intelligently and accurately determine the intention and effectiveness of user interruptions, reducing false positives and missed detections.

[0055] d) User voice segment identification and differentiation and multi-speaker processing: The module further processes the detected voice activity to identify whether it belongs to the target user. The user voiceprint embedding (Speaker Embedding) extracted in real time is used and compared with the pre-stored user voiceprint or the user voiceprint quickly learned at the beginning of the call to accurately mark the start and end timestamps of the user voice segment and identify the user. The system is expanded to handle other human speakers on the user side and the voice overlap between them. This includes speaker differentiation (SpeakerDiarization, SD) of the detected non-AI, non-main user voice segments and assigning different speaker IDs; as well as detecting the voice overlap between human speakers on the user side and using real-time speech separation technology to try to separate the overlapping user voice streams, or using a joint recognition model optimized for multi-speaker scenarios. This multi-speaker processing capability can handle speech with any number of speakers and any overlap ratio, and improve the accuracy and stability of speaker differentiation at the phrase level.

[0056] A dialogue management module is configured to perform intent recognition based on the real-time text transcription result stream and context to understand the user's intent; upon receiving the interruption event signal, the dialogue management module immediately sends a stop instruction to the voice output module to forcibly terminate the playback of the current voice interaction information; The voice output module is used to output a complete call transcription text with a timestamp and user identifier, and output real-time interactive voice based on the complete call transcription text through an audio player.

[0057] The ASR system of this invention operates as a standalone microservice or module, tightly integrated with the AI ​​interactive system's core components, including the communication platform (FreeSWITCH or other softswitch platforms), the TTS (Text-to-Speech) engine, and the dialogue manager / LLM (Large Language Model). The system utilizes efficient, low-latency, bidirectional streaming network transmission (based on WebSocket or gRPC) between each module and with the communication platform, ensuring real-time transmission and processing of audio streams, acoustic feature streams, text streams, and control signals (including critical interruption signals) with extremely low end-to-end latency.

[0058] Working mode (emphasis on interrupt processing and parallel processes): a) AI Playback and Reference: The Dialogue Management Module generates AI voices using the TTS engine based on business logic and plays them to the user. The AI-played audio signal (or a digital copy) is simultaneously transmitted via a bidirectional streaming network and serves as a critical reference input to the ASR system's "Real-time Voice Activity Detection and Interruption Processing Module."

[0059] b) User audio input and parallel processing: When a user speaks through a voice input device or communication platform, the original audio stream is captured by the communication platform and transmitted to the ASR system through a bidirectional streaming network. The original audio stream is sent to two modules in parallel: The data is sent to the "robustness enhanced real-time acoustic processing module" for noise reduction, enhancement and feature extraction to generate a robust acoustic feature stream.

[0060] The audio is sent to the "Real-time Voice Activity Detection and Interruption Processing Module" to perform user voice activity and interruption detection together with the AI ​​playback reference audio.

[0061] c) Ultra-low-latency detection of user activity and interruptions (core process): The "Real-time Voice Activity Detection and Interruption Processing Module" analyzes both raw user audio and AI reference audio in real time. Upon detecting user voice activity (particularly when the AI ​​is playing), and determining that the activity originates from the user (rather than AI leakage), the module immediately triggers a "user interruption event signal" with extremely low latency (<100 milliseconds) and transmits it to the Dialogue Management Module via a bidirectional streaming network. Simultaneously, the module continuously outputs user voice activity status and precise timestamp information.

[0062] d) ASR Streaming Transcription: Simultaneously, the "Ultra-Low Latency Causal Streaming ASR Modeling and Decoding Engine" processes the enhanced acoustic feature stream from the acoustic processing module in real time, streaming the user's speech transcript with ultra-low latency (<200 milliseconds after the user speaks the first word) and sending it to the Dialogue Management Module via a bidirectional streaming network. The ASR engine's streaming output is continuous, and the transcript is dynamically updated as the user speaks.

[0063] e) Rapid Response of the Dialogue Management Module: The Dialogue Management Module receives real-time text streams from the ASR engine to understand user intent. More importantly, the Dialogue Management Module monitors user interruption signals and voice activity status from the Real-Time Voice Activity Detection and Interruption Processing Module in real time. Receiving a user interruption signal is a top-priority event, and the Dialogue Management Module immediately sends a stop command to the TTS engine, forcibly terminating the current AI playback. The Dialogue Management Module then utilizes the received ASR text (even partial text after the user interruption) in context to perform intent recognition and business logic analysis. Due to the timely triggering of the interruption signal and the ultra-low latency of ASR, the Dialogue Management Module can react extremely quickly after the user interrupts and begins speaking, stopping AI playback and preparing for the next response.

[0064] f) Conversation turn switching: After the user mutes (the silence is detected by the interruption processing module or the ASR engine), the dialogue management module generates the next AI speech text based on the complete user speech text (or the speech text after the user interrupts), sends it to the TTS engine, and starts the next round of AI speech, driving the conversation to proceed smoothly.

[0065] g) Result Output and Enhanced Information: Ultimately, the system outputs a complete call transcript with timestamps and user IDs. It can further integrate the recognition of non-verbal events (laughter, crying, ambient sound types) or the speaker's emotional state, providing a richer information dimension for AI interactions and assisting in dialogue management and subsequent analysis. The system has the ability to integrate non-verbal information recognition and sentiment analysis. The system can explore closer integration of ASR with NLU / dialogue management, and even develop models that can directly predict user intent or desired actions end-to-end from speech signals (Speech-to-Intent or Speech-to-Action), further reducing overall response latency. The system has end-to-end Speech-to-Intent or Speech-to-Action capabilities.

[0066] The technical solutions of the present invention are further described below through specific examples, but the protection scope of the present invention is not limited to these examples.

[0067] Example 1: Natural user interruptions and quick responses in AI interactions Scenario: A bank's AI interactive system is playing a presentation about a new product to a customer. Upon hearing a point, the customer immediately raises a question and wishes to interrupt the AI's presentation to ask a question. The system can immediately detect the user's interruption, stop the AI ​​presentation, and quickly and accurately identify the user's question, enabling a smooth transition between conversations.

[0068] Implementation process: ① The AI ​​interactive system plays a preset text using the TTS engine. The audio signal played by the AI ​​is used as a reference input and transmitted to the ASR system's "Real-time Voice Activity Detection and Interruption Processing Module" via a two-way streaming network.

[0069] ② When a customer hears something, they immediately become suspicious and begin to speak: "Wait, interrupt me, is your rate fixed?" The original audio stream of the customer's initial speech is captured by the communication platform and transmitted to the ASR system via a bidirectional streaming network. This audio stream is then sent in parallel to the "Real-time Acoustic Processing Module with Enhanced Robustness" and the "Real-time Voice Activity Detection and Interruption Handling Module."

[0070] ③ In the "Real-time Acoustic Processing Module with Enhanced Robustness," the customer's voice audio undergoes real-time noise reduction, enhancement (suppressing background noise and AI-generated sound leakage), and feature extraction to generate an enhanced acoustic feature stream. This feature stream is transmitted via a bidirectional streaming network to the "Ultra-Low Latency Causal Streaming ASR Modeling and Decoding Engine," where streaming recognition begins. The ASR engine's model, trained on large-scale data, has preliminary adaptability to individual user differences and dialects, and boasts rapid online adaptation capabilities with small sample sizes.

[0071] ④ Simultaneously, the original audio stream from the customer's initial speech and the reference audio stream played by the AI ​​enter the "Real-time Acoustic Processing Module." This module's core dual-input neural network model analyzes these two audio streams in real time, accurately detecting any speech activity (i.e., the user's own voice) that doesn't match the AI ​​reference audio.

[0072] ⑤ Within a very short time after the user utters the first word "wait" (approximately 80 milliseconds after the user begins speaking), the "Real-time Voice Activity Detection and Interruption Processing Module" determines with high confidence that the user has interrupted, immediately triggering a "user interruption signal" and sending it to the Dialogue Management Module with the highest priority via a bidirectional streaming network. This interruption determination combines the acoustic confidence output of the ASR engine, the language model probability, and the initial vocabulary information recognized, improving accuracy.

[0073] ⑥ After the dialogue management module receives the "user interruption signal", it immediately sends a stop instruction to the TTS engine, and the AI playback is immediately forced to terminate, avoiding the AI from continuing to play invalidly.

[0074] ⑦ The ASR engine continuously performs low-latency streaming recognition on the user speech stream. Since the AI playback has stopped, the acoustic environment is further simplified. The ASR engine quickly processes the subsequent speech of the user and outputs a complete transcribed text stream with ultra-low latency (within about 200 milliseconds after the user finishes saying "吗?"): "等等,我打断一下,你们这个利率是固定的吗?". The transcribed text stream is quickly transmitted to the dialogue management module through bidirectional streaming network transmission.

[0075] ⑧ After the dialogue management module receives the accurate user question text, it performs intent recognition (accurately recognizes "asking whether the interest rate is fixed").

[0076] ⑨ The dialogue management module generates corresponding response text according to the recognized user intent, then sends it to the TTS engine, starts the next round of AI speech, and drives the dialogue to proceed smoothly and naturally.

[0077] Example 2: Robust Speech Recognition and Fast Response in a Noisy Environment (Including Dialects) Scenario: The customer interacts in a noisy outdoor environment, for example, through a phone or its App voice assistant, and the customer uses Mandarin with a dialect accent or directly uses a common dialect. The system of the present invention needs to accurately recognize the customer's response in this complex acoustic environment and language variant and quickly conduct the next round of dialogue.

[0078] Implementation process: ① The AI system plays the opening statement.

[0079] ② The customer responds in a noisy environment: "好的,请说。" (There is significant background noise in the user's speech and an obvious dialect accent) ③ The user speech stream enters the "robustness-enhanced real-time acoustic processing module" of the ASR system through bidirectional streaming network transmission. This module uses a deep learning noise suppression model trained on a large amount of noise data and telephone audio to effectively filter or reduce the influence of background noise, and at the same time compensate for telephone channel distortion, generating a cleaner and more robust acoustic feature. The system has the ability to further enhance the robustness in complex acoustic environments and far fields through technologies such as blind source separation and speech dereverberation.

[0080] ④ The enhanced acoustic feature stream is transmitted via a bidirectional streaming network to the "Ultra-Low Latency Causal Streaming ASR Modeling and Decoding Engine." This engine's core model has been extensively trained on a large corpus containing significant noise, telephone channel distortion, and multiple dialects (including those used by customers), resulting in high recognition robustness against such complex audio and language variations. Based on preliminary recognition results or acoustic features, the engine quickly identifies user characteristics or possible dialect regions and dynamically loads or adjusts language or acoustic model parameters specific to that dialect, further improving recognition accuracy. The system boasts deeper personalization and dialect adaptability, particularly rapid online adaptation to small sample sizes.

[0081] ⑤ The ASR engine outputs the transcribed text with ultra-low latency (within approximately 250 milliseconds after the user says "Speak"): "Yes, please speak." (The transcription accurately reflects the user's actual speech, even with dialect accents and background noise).

[0082] ⑥ After receiving the accurate transcription, the dialogue management module immediately understands the user's intent and quickly generates the next AI speech text, sending it to the TTS engine, driving the conversation to proceed smoothly and efficiently without being significantly affected by environmental noise and dialect.

[0083] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary engineering technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A low-latency streaming voice interaction system with interruption processing function, characterized in that: Includes the following connected in sequence: The voice input module is used to collect the user's original digital audio stream and send it to the real-time acoustic processing module; A real-time acoustic processing module, configured to pre-process the original digital audio stream and extract real-time acoustic features based on a deep learning model to obtain a real-time acoustic feature sequence stream; A streaming acoustic decoding module is used to construct a causal streaming architecture acoustic model based on Transducer through the real-time acoustic feature sequence stream and output a real-time text transcription result stream; A voice activity and interruption detection module is configured to determine the user's real-time voice activity status based on the user's original digital audio stream and the audio signal played in real time by the system. The real-time voice activity status includes whether the user is speaking or muted. When the voice activity and interruption detection module determines that the user is speaking, it outputs an interruption event signal and sends the interruption event signal to the dialogue management module. A dialogue management module is used to identify intent based on the real-time text transcription result stream and combine it with the context to understand the user's intent; When the dialogue management module receives the interruption event signal, it immediately sends a stop instruction to the voice output module to forcibly terminate the playback of the current voice interaction information; The voice output module is used to output a complete call transcription text with a timestamp and user identifier, and output real-time interactive voice based on the complete call transcription text through an audio player.

2. A low-latency streaming voice interaction system with interruption processing function according to claim 1, characterized in that: The original digital audio stream is pulse code modulation (PCM) data with a sampling rate of 8 kHz or 16 kHz.

3. A low-latency streaming voice interaction system with interruption processing function according to claim 1, characterized in that: The specific steps of the real-time acoustic feature extraction include: The continuously input raw digital audio stream is divided into short time frames of fixed duration, and the acoustic feature vector of each frame is extracted by sliding with a fixed step length smaller than the fixed duration.

4. A low-latency streaming voice interaction system with interruption processing function according to claim 1, characterized in that: The deep learning model receives the noisy user's original digital audio stream and the AI ​​playback audio reference signal as input, trains the deep learning model to predict and suppress the noise in the user's original digital audio stream and the leakage and echo components of the AI ​​audio, extracts the noise-free user voice signal or its corresponding acoustic features, and obtains a real-time acoustic feature sequence stream.

5. The low-latency streaming voice interaction system with interruption processing function according to claim 1, characterized in that: The Transducer-based causal streaming acoustic model is an ASR model. The streaming decoding algorithm used by the system achieves strict synchronization with the ASR model output frequency and low-latency output through the following collaborative strategies, specifically including: The decoder in the ASR model receives the real-time probability stream output by the acoustic model and combines it with the streaming language model information to update the decoding status and search for the optimal text path in real time in a frame-synchronous or block-synchronous manner. The decoder adopts an aggressive decoding strategy with a small initial beam width. Once the confidence of a certain path reaches a preset threshold, it immediately outputs part of the text. The real-time text transcription result stream output during the decoding process is dynamically updated and corrected by the decoder until the user mutes the voice.

6. A low-latency streaming voice interaction system with an interruption processing function according to claim 1, characterized in that: The voice activity and interruption detection module includes a real-time voice activity detection submodule and an interruption processing submodule. The user voice detection submodule is used to determine whether there is a voice stream input from the user end in the user audio; the interruption processing submodule is used to output an interruption event signal; The voice activity detection submodule includes a dual-input neural network model: the dual-input neural network model is used to predict the probability of the presence of user voice, by suppressing or ignoring components in the user audio that are highly correlated with the AI ​​reference audio, and only responding to the user's input voice.

7. A low-latency streaming voice interaction system with interruption processing function according to claim 6, characterized in that: The logic of the interruption processing submodule outputting the interruption event signal is as follows: When the system is playing voice, if the voice activity and interruption detection module detects user voice activity with high confidence, it immediately triggers a high-priority interruption event signal and sends it to the dialogue management module via a two-way streaming network transmission. The generation delay of the interruption event signal is less than 100 milliseconds, which is much faster than the time required for the streaming acoustic decoding module to output the first word, ensuring that the system responds in a timely manner and immediately stops outputting real-time interactive voice.

8. The low-latency streaming voice interaction system with interruption processing function according to claim 1, characterized in that: The voice activity and interruption detection module is also used to detect voice activity and identify whether the voice stream belongs to the target user. The specific steps are as follows: By embedding the user voiceprint extracted in real time and comparing it with the pre-stored user voiceprint or the user voiceprint obtained through rapid learning in the early stage of intelligent interaction, the start and end timestamps of the user voice segment are accurately marked, and the user is identified; the speakers of the detected non-AI and non-main user voice segments are distinguished and different speaker IDs are assigned; when voice overlap is detected between different speakers on the user side, real-time voice separation technology is used to separate the overlapping user voice streams.

Images