A low-delay streaming voice interaction system with break handling functionality

Abstract

The application provides a low-delay streaming voice interaction system with a breaking processing function, and relates to the technical field of artificial intelligence.The application carries out necessary preprocessing and acoustic feature extraction through a real-time acoustic processing module, and uses robustness enhancement technology to resist distortion and complex environmental noise introduced by an interaction channel.A streaming acoustic decoding module carries out acoustic modeling, language model application and decoding in real time and in parallel, and outputs an ultralow-delay text transcription result stream.The real-time acoustic processing module is responsible for high-precision and ultralow-delay detection of user voice activity, especially user voice activity during AI playback of voice, to determine the real-time voice activity state of the user.The system uses an efficient and low-delay bidirectional streaming network transmission mode between modules and between the system and a communication platform, so that audio streams, acoustic feature streams, text streams and control signals can be transmitted and processed in real time with extremely low end-to-end delay.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a low-delay streaming voice interaction system with interrupt processing function. BACKGROUND

[0002] As the core AI technology of human-computer interaction, speech recognition ASR has been widely applied in many fields. The traditional ASR system is composed of an acoustic model, a pronunciation dictionary, and a language model. Deep learning promotes the development of end-to-end E2E ASR models, integrates multiple models, and improves performance. However, the current intelligent interaction scene still faces many challenges when applied:

[0003] Robustness problem of complex interactive channel environment: There are bandwidth limitations, coding distortion, packet loss, and other channel effects in the network, and the user interaction environment is diverse and uncontrollable, with background noise, reverberation, echo, or leakage. This makes the recognition accuracy of existing ASR models significantly decrease.

[0004] Real-time interaction ultra-low delay bottleneck: Natural and smooth human-computer dialogue requires ASR systems to have extremely low end-to-end delay. Traditional batch processing or streaming processing will introduce obvious dialogue pauses, affecting user experience. Intelligent interaction has strict requirements for first-word delay and real-time text output.

[0005] Multi-speaker complex dialogue processing and user interrupt detection problem: AI interaction involves two speakers, AI and user. The system needs to accurately distinguish speech and support the user to interrupt AI at any time. Existing ASR systems have difficulty in accurately detecting user speech when AI plays voice, limiting the naturalness and efficiency of interaction.

[0006] Insufficient adaptability to individual differences: AI interaction user groups are diverse, with differences in accent, speech rate, volume, and word usage habits. Existing ASR systems are difficult to adapt to specific user language features in real time, affecting recognition accuracy. SUMMARY

[0007] To solve the above problems, the present application provides a low-delay streaming voice interaction system with interrupt processing function, and proposes an innovative ASR technical solution. In particular, through the careful design of the interrupt processing mechanism, the user speech recognition function with high robustness, ultra-low delay, real-time streaming processing, and support for natural interruption in AI intelligent interaction is realized, thereby significantly improving the overall performance of the intelligent interaction system and the user interaction experience.

[0008] To achieve the above purpose, the technical solution adopted by the present application is: a low-delay streaming voice interaction system with interrupt processing function is provided, which includes the following in sequence:

[0009] a voice input module, configured to collect a raw digital audio stream of a user and send the raw digital audio stream to a real-time acoustic processing module;

[0010] a real-time acoustic processing module, configured to pre-process the raw digital audio stream and perform real-time acoustic feature extraction based on a deep learning model to obtain a real-time acoustic feature sequence stream;

[0011] a streaming acoustic decoding module, configured to construct a Transducer-based causal streaming architecture acoustic model based on the real-time acoustic feature sequence stream to output a real-time text transcription result stream;

[0012] a voice activity and interruption detection module, configured to determine a real-time voice activity state of the user based on the raw digital audio stream of the user and an audio signal played by the system in real time, the real-time voice activity state including that the user is speaking and that the user is silent, and output an interruption event signal and send the interruption event signal to a dialogue management module when the voice activity and interruption detection module determines that the user is speaking;

[0013] a dialogue management module, configured to perform intent recognition based on the real-time text transcription result stream in combination with context to understand the user's intent, and send a stop instruction to a voice output module to forcibly terminate playing of current voice interaction information when the dialogue management module receives the interruption event signal;

[0014] a voice output module, configured to output complete call transcription text with a timestamp and a user identifier, and output real-time interactive voice based on the complete call transcription text through an audio player.

[0015] Preferably, the raw digital audio stream is pulse code modulation (PCM) data with a sampling rate of 8 kHz or 16 kHz.

[0016] Preferably, the specific steps of the real-time acoustic feature extraction include:

[0017] dividing the continuously input raw digital audio stream into short-time frames with a fixed time length, and extracting acoustic feature vectors of each frame through sliding with a fixed step length smaller than the fixed time length.

[0018] Preferably, the deep learning model receives a noisy raw digital audio stream of the user and an AI playback audio reference signal as input, trains the deep learning model to predict and suppress noise in the raw digital audio stream of the user and leakage and echo components of the AI audio, extracts a noise-free user voice signal or acoustic features corresponding to the noise-free user voice signal, and obtains a real-time acoustic feature sequence stream.

[0019] As preferred, the transducer-based causal streaming architecture acoustic model is an ASR model, which only relies on the information of the current and previous input frames or blocks when processing the current input frame or a small block of features.

[0020] More preferably, the streaming decoding algorithm adopted by the system achieves strict synchronization with the output frequency of the ASR model and low-latency output through the following cooperative strategies, including:

[0021] The decoder in the ASR model updates the decoding state and searches for the optimal text path in real time in a frame-synchronous or block-synchronous manner by receiving the real-time probability stream output by the acoustic model and combining the streaming language model information. The decoder adopts an aggressive decoding strategy, uses a small initial beam width, and outputs partial text as soon as the confidence of a certain path reaches a preset threshold. The real-time text transcription result stream output during the decoding process is dynamically updated and corrected by the decoder until the user is silent.

[0022] As preferred, the speech activity and interruption detection module includes a real-time speech activity detection submodule and an interruption processing submodule. The user speech detection submodule is used to determine whether there is a speech stream from the user input in the user audio. The interruption processing submodule is used to output an interruption event signal.

[0023] The speech activity detection submodule includes a dual-input neural network model, which is used to predict the existence probability of user speech and only responds to the input speech of the user by suppressing or ignoring the components in the user audio that are highly correlated with the AI reference audio.

[0024] More preferably, the logic of the interruption processing submodule outputting the interruption event signal is as follows:

[0025] When the system is playing speech, the speech activity and interruption detection module detects user speech activity with high confidence, and immediately triggers a very high priority interruption event signal. The speech activity and interruption detection module sends the interruption event signal to the dialogue management module through a bidirectional streaming network transmission. The generation delay of the interruption event signal is less than 100 milliseconds, which is much faster than the time required for the streaming acoustic decoding module to output the first word, ensuring that the system reacts in time and immediately stops outputting real-time interactive speech.

[0026] As preferred, the speech activity and interruption detection module is also used to detect speech activity and identify whether the speech stream belongs to the target user, with the following specific steps:

[0027] Through real-time extraction of user voiceprints, comparison with pre-stored or quickly learned user voiceprints at the beginning of intelligent interaction, accurate marking of the start and end timestamps of user voice segments, and user identification; and speaker differentiation of detected non-AI and non-main user voice segments, different speaker IDs are assigned; when there is voice overlap between different speakers on the user side, real-time voice separation technology is used to separate the overlapping user voice streams.

[0028] The beneficial effects of the present application are:

[0029] Excellent robustness and environmental adaptability: Real-time acoustic processing technology optimized for interactive audio characteristics and complex background noise (including AI leakage), combined with blind source separation, speech de-reverberation, and other technologies, significantly improves the voice recognition accuracy and stability of the system in complex communication environments, breaking through the performance bottleneck of traditional ASR.

[0030] Extremely low latency and real-time streaming capability: Optimized causal streaming ASR model architecture and efficient streaming decoding algorithm, with extremely low end-to-end latency from user voice input to text output, first word / character latency (TTFW) reduced to below 200 milliseconds, and support for real-time, continuous text stream output, eliminating conversation pauses and improving conversation response speed. At the same time, continuously optimize model architecture and inference technology to reduce computing resource consumption and support efficient deployment.

[0031] Support for natural and smooth user interruptions: Innovative real-time voice activity detection and interruption processing module can detect user voice activity and determine user interruption with high sensitivity and ultra-low latency (<100 milliseconds) while AI is playing voice, and send a signal to the conversation management module in a timely manner. Combined with acoustic confidence, language model probability, vocabulary information, and prosody feature analysis, the interruption judgment mechanism enables the AI system to stop the current playback and process user speech immediately, providing a smooth interaction experience that meets human habits.

[0032] Enhanced user individual differences and dialect adaptability: Models trained on diverse accents, speech rates, and dialect data have fast online adaptation capabilities for small samples, allowing more accurate and robust recognition of different accents, speech rates, and common dialects, improving service capabilities for a wide range of user groups.

[0033] Improved complex conversation processing capability: Accurate differentiation between AI and user voice, effective handling of user interruptions, and enhanced AI interaction system performance and stability in complex conversation scenarios (including interruptions and word stealing). It has fine multi-speaker processing capabilities and can handle any number and overlap ratio of user voice and overlapping parts, improving the accuracy and stability of short phrase-level speaker differentiation.

[0034] High-efficiency integrated system and bidirectional streaming: integrate robust acoustic processing, ultra-low latency streaming ASR, high-precision interruption detection, and user adaptability into a unified real-time processing pipeline, use high-efficiency bidirectional streaming network transmission mechanism, optimize system efficiency and data transmission delay, meet the high concurrency and high real-time requirements of AI interaction system.

[0035] Significantly improve user experience: provide high-precision, ultra-low latency, support natural interruption and user individual differences and dialect adaptation voice interaction capabilities, make AI interaction more humanized, intelligent and easy to use, greatly improve the acceptance and satisfaction of end users. Integrate non-verbal information recognition and sentiment analysis to provide more rich information dimensions. With end-to-end speech-to-intent or speech-to-action capabilities, further reduce overall response delay. BRIEF DESCRIPTION OF DRAWINGS

[0036] Fig. 1 Figure 1 is a low-latency streaming voice interaction system framework with interruption processing function according to the present application.

[0037] Fig. 2 Figure 2 is a voice activity and interruption detection module framework according to the present application. DETAILED DESCRIPTION

[0038] Please refer to Figs. 1-2 Figure 1 shows a low-latency streaming voice interaction system with interruption processing function, comprising:

[0039] a voice input module for collecting the user's original digital audio stream and sending it to the real-time acoustic processing module;

[0040] a real-time acoustic processing module for pre-processing the original digital audio stream and extracting real-time acoustic features based on a deep learning model to obtain a real-time acoustic feature sequence stream;

[0041] The voice activity and interruption detection module receives the user's original audio stream from the AI interaction platform, performs necessary preprocessing and acoustic feature extraction, and implements a series of robust enhancement techniques to effectively combat distortion introduced by the interaction input channel and complex environmental noise (including background noise and AI playback voice leakage / echo). The module uses a deep learning model and signal processing techniques based on large-scale telephone communication data and noise data pre-training.

[0042] a) Interaction input channel feature compensation and normalization: real-time analysis of user input audio, application of signal processing techniques to compensate for frequency response distortion and bandwidth limitations introduced by the interaction input channel. For 8kHz narrowband audio, use super-resolution or bandwidth expansion techniques for band-limited signals to convert them into wideband feature representations. At the same time, real-time volume normalization processing is performed to cope with the differences in user's speaking volume.

[0043] b) Real-time framing and feature extraction: The continuously incoming audio stream is segmented into short-time frames (25 ms) of fixed duration, and the acoustic feature vector (40-dimensional Mel-filterbank features MFE) of each frame is extracted with a fixed step (10 ms) sliding. This process is designed as a strict causal processing, only using the current frame and previous information, ensuring low latency and real-time processing.

[0044] c) Deep learning real-time noise suppression and speech enhancement: The module adopts a real-time speech enhancement or noise suppression algorithm based on deep neural networks. This algorithm uses deep learning variants based on time-frequency masking or spectral subtraction. To deal with the leakage or echo of AI playback speech, a dual-input single-output (DISO) or multi-input single-output (MISO) model is used: the model receives the noisy user audio stream and the AI playback audio reference signal (or its features) as input, and the training target is to predict and suppress the noise in the user audio and the leakage / echo component of the AI audio, thereby extracting a purer user speech signal or its corresponding acoustic features. The enhancement process is designed as a strict causal processing and low computational complexity, ensuring real-time processing. The system has the ability to further improve the robustness of complex acoustic environments and far-field, by integrating blind source separation (Blind Source Separation), speech dereverberation (Dereverberation) and other technologies.

[0045] d) Online feature normalization and environment adaptation: Apply real-time online calculation of mean and variance for acoustic feature normalization (online cepstral mean variance normalization CMVN), dynamically reduce the feature distribution changes caused by different conversation environments or speakers. Integrate real-time acoustic environment estimation methods based on statistical models or deep learning, and dynamically adjust the feature normalization parameters or use environment-dependent feature representations according to the estimation results.

[0046] a stream of real-time acoustic features, and a stream of real-time text transcription results are outputted, and the stream of real-time text transcription results is sent to a speech output module;

[0047] The Transducer-based causal streaming architecture acoustic model is an ASR model (including ASR modeling and decoding engine). After receiving robustly enhanced acoustic feature streams, the streaming acoustic decoding module performs acoustic modeling, language model application and decoding in real time and in parallel, and outputs an ultra-low delay text transcription result stream. The engine is the core of the ASR of the present application, and the design goal is to achieve extremely low first word (TTFW < 200 milliseconds, preferably < 150 milliseconds) and continuous real-time text output to support smooth dialogue turns. The core model of the engine is based on an end-to-end deep learning model pre-trained on large-scale speech and text data, and is deeply optimized for streaming and low latency.

[0048] a) Transducer or other causal streaming architecture-based acoustic model: the engine core uses an end-to-end streaming ASR model, preferably based on the Transducer (RNN Transducer) causal streaming processing architecture. The model is designed as a strict causal model, which only relies on the information of the current and previous input frames or chunks when processing the current input frame or a small chunk of features, and never relies on future information. The acoustic model receives acoustic feature streams and outputs a corresponding ASR modeling unit (which can be characters, subwords or words) probability distribution stream in real time at a fixed frequency (every certain number of frames or every chunk).

[0049] b) Efficient streaming inference and computation graph optimization: the inference process of the model runs on a specially optimized inference engine, fully utilizing the parallel computing capabilities of CPUs, GPUs or dedicated AI acceleration chips. The model computation graph is deeply optimized to support efficient chunk-based processing, and the processing delay of each input feature chunk is minimized. Model compression techniques such as pruning and quantization (FP16 or INT8 quantization) are applied to further improve inference speed and reduce computational resource consumption, making it suitable for high-concurrency AI interaction scenarios. The system continuously optimizes the model architecture and inference technology to reduce the computational complexity and memory usage of the model, supporting deployment in resource-constrained environments.

[0050] c) Ultra-Low Latency Streaming Decoding Algorithm: An efficient streaming decoding algorithm is adopted to synchronize with the output frequency of the streaming acoustic model. Improved Synchronous Beam Search or Greedy Decoding variants are used. The decoder receives the real-time probability stream output by the acoustic model, combines it with the streaming language model information, and updates the decoding state and searches for the optimal text path in a frame-synchronous or block-synchronous manner. To achieve ultra-low TTFW, a radical decoding strategy is used, with a small initial beam width, and once the confidence of a certain path reaches a preset threshold, the system outputs partial text (Early Exit mechanism); the lattice generation and backtracking process is optimized to reduce latency. These optimizations aim to reduce the delay from the user stopping speaking to the system outputting the first word (TTFW) to below 200 milliseconds, preferably close to 100 milliseconds. The decoding process continues, and the text stream output can be dynamically updated and corrected until the user is silent or the call ends.

[0051] d) Integrated or Collaborative Streaming Language Model with Personalization and Dialect Adaptation: The ASR model usually contains an integrated language model based on the Prediction Network of the Transducer. This language model provides real-time lexical and grammatical constraints to help the decoder choose the most reasonable word sequence in ambiguous acoustic information. The reasoning of the language model is optimized to support low-latency streaming queries. To improve the adaptability to individual differences and dialects of users, the language model and / or acoustic model are trained on large-scale corpora containing multiple accents, speech rates, and dialect data, enabling preliminary recognition of dialects and different accents. The system can use real-time extracted user voiceprints or preliminary recognition results to determine user characteristics or possible dialect regions at the beginning of the call, and dynamically load or adjust the language model or acoustic model parameters (through adaptive or personalized techniques) for that characteristic or dialect to improve recognition accuracy. In addition, dialog context information (historical dialog content, business keywords, user labels in the CRM system, etc.) can be used to dynamically bias the language model or decoder, increasing the recognition probability of specific words. The system has the ability to quickly adapt to new users in terms of few-shot speaker, accent, and specific dialect, quickly improving recognition accuracy.

[0052] A voice activity and interruption detection module is used to determine the real-time voice activity state of the user based on the user's original digital audio stream and the audio signal played by the system in real time, wherein the real-time voice activity state includes the user speaking and the user being silent, and when the voice activity and interruption detection module determines that the user is speaking, an interruption event signal is output and sent to the dialog management module.

[0053] The voice activity and interruption detection module includes a real-time voice activity detection submodule and an interruption processing submodule. The voice activity and interruption detection module is responsible for detecting user voice activity with high precision and ultra-low delay, especially during AI playback of voice (i.e., user interruption of AI). The module uses a deep learning model pre-trained based on large-scale data and combines signal processing techniques.

[0054] Acoustic model based on Transducer or other causal streaming architecture: The engine core uses an end-to-end streaming ASR model, preferably based on the Transducer (RNN Transducer) causal streaming architecture. This model is designed as a strict causal model that only relies on current and previous input frames or chunks of features when processing the current input frame or chunk, and never relies on future information. The acoustic model receives a stream of acoustic features and outputs a corresponding stream of probability distributions of ASR modeling units (which can be characters, subwords, or words) at a fixed frequency (every certain number of frames or every chunk).

[0055] The inference process of the model runs on a specially optimized inference engine, fully utilizing the parallel computing capabilities of CPUs, GPUs, or dedicated AI acceleration chips. The model computation graph is deeply optimized to support efficient chunk-based processing, and the processing delay of each input feature chunk is minimized. Model compression techniques such as pruning and quantization (FP16 or INT8 quantization) are applied to further improve inference speed and reduce computational resource consumption, making it suitable for high-concurrency AI interaction scenarios. The system continuously optimizes the model architecture and inference technology to reduce the computational complexity and memory usage of the model, supporting deployment in resource-constrained environments.

[0056] The decoder in the ASR model receives the real-time probability stream output by the acoustic model and uses a high-efficiency streaming decoding algorithm synchronized with the output frequency of the streaming acoustic model. Using an improved synchronous beam search (Synchronous Beam Search) or greedy decoding (Greedy Decoding) variant, combined with streaming language model information, the decoding state is updated in real time and the optimal text path is searched in a frame-synchronous or block-synchronous manner. To achieve ultra-low TTFW, a radical decoding strategy is used, a small initial beam width is used, and once the confidence of a certain path reaches a preset threshold, partial text is output (Early Exit mechanism); the lattice generation and backtracking process is optimized to reduce the waiting time. These optimizations aim to reduce the delay from the user stopping speaking to the system outputting the first word (TTFW) to less than 200 milliseconds, preferably close to 100 milliseconds. The decoding process continues, and the text stream output can be dynamically updated and corrected until the user is silent or the call ends.

[0057] The real-time text transcription result stream output during the decoding process is dynamically updated and corrected through the continuous evolution of the subsequent incoming acoustic features, language model constraints, and decoder state until the user is silent.

[0058] The streaming decoding algorithm used by the system achieves strict synchronization with the output frequency of the ASR model and low-latency output through the following collaborative strategies:

[0059] (1) Real-time incremental decoding mechanism: the decoder takes the frame or block output by the acoustic model as the basic processing unit and dynamically executes at each time step:

[0060] Receive the acoustic probability vector of the current time step;

[0061] Fuse the real-time context vector provided by the streaming language model;

[0062] Expand the current active path set and prune, retaining the top-K paths with the highest scores (where K≤5).

[0063] (2) Confidence-driven immediate submission strategy: define the path confidence comprehensive index:

[0064]

[0065] Where:

[0066] P AM : posterior probability output by the current acoustic model;

[0067] P LM : sequence score provided by the language model;

[0068] T: confidence compensation factor based on time decay;

[0069] α, β, γ: respectively represent the weight coefficients of dynamic adjustment. When any path satisfies C>θcommit(θcommit∈[0.8, 0.95] when:

[0070] Immediately commit the unoutput token sequence (such as subword or word fragment) corresponding to the path;

[0071] Lock the prefix of the committed text, and subsequent decoding only optimizes the unlocked suffix part.

[0072] (3) Dynamic rollback correction mechanism for committed text fragments:

[0073] Cache the acoustic features and language model states of the last N frames (N=3~10);

[0074] When new input causes the historical path probability change rate |ΔP|>θrollback, the following operations are triggered:

[0075]

[0076] (4) Cooperative optimization with the interrupt processing function.

[0077] Through the synergistic effect of the above strategies:

[0078] Achieve an extremely low output delay of ≤200ms: a high threshold θcommit ensures that words can be committed before they end;

[0079] Support real-time user interruption: when an interruption event is detected, immediately clear the path cache generated based on the old audio;

[0080] Guarantee the final transcription accuracy: the dynamic rollback mechanism compensates for early aggressive output errors.

[0081] Note: Specific parameters (including beam width K, commit threshold θcommitθcommit, rollback frame number N, etc.) need to be verified and optimized by the target domain speech data set; its engineering implementation can be based on the general streaming decoding framework (such as the gradient search algorithm of RNN-T or the prefix beam search algorithm of CTC) for adaptive optimization.

[0082] The ASR model usually contains an integrated language model inside, which is based on the Prediction Network of Transducer. This language model provides real-time lexical and grammatical constraints to help the decoder choose the most reasonable word sequence in ambiguous acoustic information. The inference of the language model is optimized to support low-latency streaming queries. To improve the adaptability to individual differences and dialects of users, the language model and / or acoustic model are trained on a large-scale corpus containing multiple accents, speech rates and dialect data, so that they have the ability to preliminarily identify dialects and different accents. The system can use the real-time extracted user voiceprint or preliminary identification result at the beginning of call establishment to determine the characteristics or possible dialect area of the user, and dynamically load or adjust the language model or acoustic model parameters (through adaptive or personalized technology) for this characteristic or dialect to improve recognition accuracy. In addition, the language model or decoder can be dynamically biased using the dialogue context information (historical dialogue content, business keywords, user labels in the CRM system, etc.) to improve the recognition probability of specific words. The system has the ability to quickly adapt to few-shot speakers, accents and specific dialects online, and can quickly improve recognition accuracy when facing new users.

[0083] The interrupt processing submodule is the key innovative module of the present application, designed specifically for complex interactive scenarios of AI interaction, responsible for detecting user voice activity with high precision and ultra-low latency, especially during AI playback of voice (i.e. user interrupting AI). This module uses a deep learning model pre-trained on large-scale data, combined with signal processing techniques.

[0084] Input and output:

[0085] Input: User-side raw audio stream, and the audio signal currently being played by AI or its corresponding acoustic features as reference.

[0086] Output:

[0087] Real-time user voice activity status (user is speaking / user is silent), this detection has high sensitivity and robustness, and can accurately determine even in the presence of background noise and overlapping with AI voice.

[0088] An event signal indicating that the user is interrupting the AI, the triggering speed of this signal is designed to be much faster than the first word output of ASR, to ensure that the AI can respond in time.

[0089] Start and end timestamps of the user's voice segment.

[0090] The detected non-AI, non-primary user speech segments are labeled and timestamped, and can handle other human speakers present on the user side and speech overlap between them, with more refined multi-speaker processing capabilities, including speaker diarization (SD) of the detected non-AI, non-primary user speech segments, assigning different speaker IDs; and detecting speech overlap between user-side human speakers, and using real-time speech separation techniques to attempt to separate overlapping user speech streams, or using joint recognition models optimized for multi-speaker scenarios. This multi-speaker processing capability can handle any number of speakers and any overlap ratio of speech.

[0091] The core technology and workflow of the interruption processing submodule:

[0092] a) High-sensitivity robust VAD: A voice activity detector (VAD) based on a deep learning model is used. This VAD model is trained on a large amount of noisy environment and telephone audio data, and can accurately distinguish between user speech and background noise. This VAD model is optimized to have extremely high sensitivity to low-energy speech, weak signals at the start of speech, and speech activity in noise, reducing missed detections and delays.

[0093] b) AI reference signal-based interruption detection core mechanism: This module receives both user audio stream and AI playback audio reference signal. The core innovation lies in how to accurately and with ultra-low latency determine whether there is "non-AI source" speech activity in the user audio, i.e. the user's own speech, even in the case of AI playback speech causing AI speech leakage or echo in the user audio. This is achieved by using a dual-input neural network model: the model receives real-time feature sequences of user audio and real-time feature sequences of AI reference audio as input, and the training goal is to predict the probability of user speech. Through learning, the model can effectively suppress or ignore the components in the user audio that are highly correlated with the AI reference audio (i.e. AI speech leakage or echo), and only respond to the speech components that are not related to the AI reference (i.e. user speech). This is similar to implementing acoustic echo cancellation (AEC) or speech separation at the feature or model level, but its goal is to detect user speech activity rather than completely separate speech.

[0094] c) Ultra-low latency trigger logic for interrupt signal: The trigger of the interrupt signal is based on the output of the real-time voice activity detection and interrupt handling module. When the AI system is playing voice, and the module detects user voice activity with high confidence (threshold can be set), the module immediately triggers a very high priority "user interrupt event signal" and sends it to the dialogue management module through the bidirectional streaming network transmission. The generation delay of the interrupt signal (from the user starting to speak to the signal being sent out) is designed to be very low, less than 100 milliseconds, much faster than the time required for the ASR engine to output the first word, ensuring that the dialogue management system can react in time and immediately stop the AI playing. The trigger of the interrupt signal is based on the instantaneous output confidence of the VAD model, or combined with the VAD output of several consecutive frames to balance sensitivity and false trigger rate. The system combines the acoustic confidence output by the ASR engine, the language model probability, the preliminary recognized word information, and even the analysis of the user's speech rhythm, intonation, etc. to more intelligently and accurately judge the user's interrupt intention and effectiveness, reducing false positives or false negatives.

[0095] d) User voice segment recognition and differentiation and multi-speaker processing: The module further processes the detected voice activity to identify whether it belongs to the target user. By using the real-time extracted user voiceprint embedding and comparing it with the pre-stored or quickly learned user voiceprint at the beginning of the call, the system accurately marks the start and end timestamps of the user voice segment and performs user identification. The system extends to handle other human speakers on the user side and their voice overlap. This includes speaker diarization (SD) of detected non-AI, non-primary user voice segments, assigning different speaker IDs; and detecting voice overlap between user-side human speakers and using real-time voice separation techniques to try to separate overlapping user voice streams, or using joint recognition models optimized for multi-speaker scenarios. This multi-speaker processing capability can handle any number of speakers and any overlap ratio of voice, and improve the accuracy and stability of speaker diarization at the phrase level.

[0096] The dialogue management module is configured to perform intent recognition and understand user intent based on the real-time text transcription result stream and context; and the dialogue management module is configured to send a stop instruction to the voice output module immediately upon receiving the interrupt event signal, to forcibly terminate playing of the current voice interaction information;

[0097] The voice output module is configured to output complete call transcription text with timestamps and user identification, and output real-time interactive voice based on the complete call transcription text through an audio player.

[0098] The ASR system of the present application is tightly integrated with the communication platform (FreeSWITCH or other softswitch platform), TTS (Text-to-Speech) engine, dialogue management / LLM (Large Language Model) and other core components of the AI interaction system as an independent microservice or module. The system modules and the communication platform use efficient, low-latency bidirectional streaming network transmission (based on WebSocket or gRPC) to ensure that audio streams, acoustic feature streams, text streams, and control signals (including critical interrupt signals) can be transmitted and processed in real time with extremely low end-to-end delay.

[0099] Working mode (emphasis on interrupt processing and parallel processes):

[0100] a) AI playback and reference: The dialogue management module generates AI speech through the TTS engine according to business logic and plays it to the user. The AI playback audio signal (or its digital copy) is transmitted synchronously through bidirectional streaming network transmission as an important reference input to the "real-time voice activity detection and interrupt processing module" of the ASR system.

[0101] b) User audio input and parallel processing: When the user speaks through a voice input device or communication platform, the original audio stream is captured by the communication platform and transmitted into the ASR system through bidirectional streaming network transmission. This original audio stream is sent to two modules in parallel:

[0102] To the "robust real-time acoustic processing module" for noise reduction, enhancement and feature extraction to generate robust acoustic feature streams.

[0103] To the "real-time voice activity detection and interrupt processing module" for user voice activity and interrupt detection together with AI playback reference audio.

[0104] c) Ultra-low latency detection of user activity and interrupts (core process): The "real-time voice activity detection and interrupt processing module" analyzes user original audio and AI reference audio in real time. Once user voice activity is detected (especially when AI is playing), and it is determined that the activity comes from the user (not AI leakage), the module immediately triggers a "user interrupt event signal" with extremely low latency (<100ms) and sends it to the dialogue management module through bidirectional streaming network transmission. At the same time, the module continuously outputs user voice activity status and accurate timestamp information.

[0105] d) ASR streaming transcription: At the same time, the "ultra-low latency causal streaming ASR modeling and decoding engine" processes the enhanced acoustic features stream from the acoustic processing module in real-time to output the transcription text of the user's speech with ultra-low latency (< 200 ms after the user finishes the first word) and sends it to the dialog management module through a bidirectional streaming network transmission. The streaming output of the ASR engine is continuous, and the text is dynamically updated as the user speaks.

[0106] e) Fast response of the dialog management module: The dialog management module receives the real-time text stream from the ASR engine for understanding the user's intent. More importantly, the dialog management module listens to the "user interrupt signal" and the voice activity status from the "real-time voice activity detection and interrupt handling module" in real-time. Upon receiving the "user interrupt signal", which is the highest priority event, the dialog management module immediately sends a stop instruction to the TTS engine to forcibly terminate the current AI playback. Subsequently, the dialog management module uses the received ASR text (even if it is partial text after the user interrupts) to identify the intent and make business logic judgments in combination with the context. Due to the timely triggering of the interrupt signal and the ultra-low latency characteristics of ASR, the dialog management module can react extremely quickly after the user interrupts and starts speaking, stopping the AI playback and preparing for the next round of response.

[0107] f) Dialog turn switching: After the user is silent (detected by the interrupt handling module or the ASR engine), the dialog management module generates the next AI speech text based on the complete user speech text (or the speech text after the user interrupts) and sends it to the TTS engine to start the next round of AI speech, driving the smooth progress of the conversation.

[0108] g) Result output and enhanced information: Finally, the system outputs the complete call transcription text with timestamps and user identification. Non-verbal event (laughter, crying, environmental sound type) recognition or speaker emotion state recognition can be further integrated to provide more rich information dimensions for AI interaction, assisting in dialog management and subsequent analysis. The system has the ability to integrate non-verbal information recognition and emotion analysis. The system can explore tighter integration of ASR and NLU / dialog management, or even develop models that can directly predict user intent or required actions from speech signals end-to-end (Speech-to-Intent or Speech-to-Action), further reducing the overall response delay. The system has the ability to end-to-end speech-to-intent (Speech-to-Intent) or speech-to-action (Speech-to-Action).

[0109] The technical solutions of the present application are further illustrated by specific embodiments, but the protection scope of the present application is not limited to these embodiments.

[0110] Embodiment 1: Natural interruption and quick response of user in AI interaction

[0111] Scenario: An AI interaction system of a bank is playing an introduction about a new product to a customer. The customer has a question immediately when hearing a certain point and wants to interrupt the AI explanation to ask a question. The invented system can immediately detect the user's interruption, stop the AI playing, and quickly and accurately identify the user's question, realizing smooth conversation switching.

[0112] Implementation process:

[0113] ① The AI interaction system plays a preset text through the TTS engine. The audio signal of AI playing is synchronized as a reference input, which is transmitted to the "real-time voice activity detection and interruption processing module" of the ASR system through a bidirectional streaming network.

[0114] ② The customer has a question immediately when hearing a certain point and starts to speak: "Wait, I interrupt you, is this interest rate fixed?" The original audio stream when the customer starts to speak is captured through the communication platform and transmitted into the ASR system through a bidirectional streaming network. The audio stream is sent to the "robust real-time acoustic processing module" and the "real-time voice activity detection and interruption processing module" in parallel.

[0115] ③ In the "robust real-time acoustic processing module", the customer's voice audio is subjected to real-time noise reduction, enhancement (background noise suppression and AI playing leakage suppression) and feature extraction to generate an enhanced acoustic feature stream. The feature stream is transmitted to the "ultra-low latency causal streaming ASR modeling and decoding engine" through a bidirectional streaming network to start streaming recognition. The model of the ASR engine is trained on a large scale of data, has preliminary adaptability to individual differences and dialects of users, and has fast online adaptability to few samples.

[0116] ④ At the same time, the original audio stream when the customer starts to speak and the reference audio stream of AI playing enter the "real-time acoustic processing module". The core dual-input neural network model of this module analyzes these two audio streams in real time and accurately detects that there is a voice activity in the user's audio that does not match the AI reference audio (i.e. the user's own voice).

[0117] ⑤ Within a very short time (about 80 milliseconds after the user starts to speak) when the user says the first word "wait", the "real-time voice activity detection and interruption processing module" judges with high confidence that the user interrupts, immediately triggers a "user interruption signal", and sends it to the dialogue management module through a bidirectional streaming network with the highest priority. The interruption judgment combines the acoustic confidence, language model probability and preliminary recognized word information output by the ASR engine, improving the accuracy.

[0118] ⑥ After receiving the "user interrupt signal", the dialogue management module immediately sends a stop instruction to the TTS engine, and the AI playback is forcibly terminated, avoiding the AI from continuing to play invalidly.

[0119] ⑦ The ASR engine continues to perform low-latency streaming recognition on the user's voice stream. Since the AI playback has stopped, the acoustic environment is further simplified. The ASR engine quickly processes the user's subsequent speech and outputs a complete transcription text stream: "Wait, I interrupt you, is this interest rate fixed?" in ultra-low latency (within about 200 milliseconds after the user says "Yes?"). The transcription text stream is quickly transmitted to the dialogue management module through the bidirectional streaming network transmission.

[0120] ⑧ After receiving the accurate user question text, the dialogue management module performs intent recognition (accurately identifies "inquiry about fixed interest rate").

[0121] ⑨ According to the identified user intent, the dialogue management module generates a corresponding reply text and sends it to the TTS engine, starting the next round of AI speech and driving the dialogue to flow smoothly and naturally.

[0122] Example 2: Robust speech recognition and fast response in noisy environment (including dialect)

[0123] Scenario: The customer interacts through a phone or App voice assistant in a noisy outdoor environment, and the customer uses Mandarin with a dialect accent or directly uses a common dialect. The system needs to accurately recognize the customer's response in this complex acoustic environment and language variation and quickly proceed to the next round of dialogue.

[0124] Implementation process:

[0125] ① The AI system plays an opening speech.

[0126] ② The customer responds in a noisy environment: "Yes, please speak." (The user's voice is mixed with a lot of background noise and has a clear dialect accent)

[0127] ③ The user's voice stream enters the "robust enhanced real-time acoustic processing module" of the ASR system through bidirectional streaming network transmission. This module uses a deep learning noise suppression model trained on massive noise data and telephone audio to effectively filter or reduce the impact of background noise, while compensating for telephone channel distortion to generate more pure and robust acoustic features. The system has the ability to further improve the robustness of complex acoustic environments and far-field, through techniques such as blind source separation and speech dereverberation.

[0128] IV. Enhanced acoustic features stream through a bidirectional streaming network into a "ultra-low latency causal streaming ASR modeling and decoding engine". The core model of this engine is fully trained on a large-scale corpus containing a large amount of noise, telephone channel distortion, and multiple dialects (including the dialect used by the customer), and has high recognition robustness to such complex audio and language variants. The engine quickly determines the user's characteristics or possible dialect region based on the preliminary recognition results or acoustic features, and dynamically loads or adjusts the language model or acoustic model parameters for this dialect to further improve recognition accuracy. The system has a deeper personalization and dialect adaptability, especially a fast online adaptation capability with a small amount of samples.

[0129] V. The ASR engine outputs the transcribed text with ultra-low latency (about 250 milliseconds after the user finishes speaking "said"): "OK, please say." (The transcription result accurately reflects the user's actual speech, even with a dialect accent and background noise).

[0130] VI. After receiving the accurate transcribed text, the dialogue management module immediately understands the user's intent and quickly generates the next AI speaking text, which is sent to the TTS engine to drive the dialogue to flow smoothly and efficiently, with little impact from environmental noise and dialect.

[0131] The above embodiments only describe the preferred embodiments of the present application, and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by ordinary engineering technicians in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. A low-latency streaming voice interaction system with a break processing function, characterized in that, Comprise sequentially connected: A voice input module for collecting the user's original digital audio stream and sending it to the real-time acoustic processing module; A real-time acoustic processing module for pre-processing the original digital audio stream and extracting real-time acoustic features based on a deep learning model to obtain a real-time acoustic feature sequence stream; A streaming acoustic decoding module for constructing a Transducer-based causal streaming architecture acoustic model based on the real-time acoustic feature sequence stream to output a real-time text transcription result stream; A voice activity and interruption detection module for determining the user's real-time voice activity state based on the user's original digital audio stream and the system's real-time played audio signal, the real-time voice activity state including the user speaking and the user being silent, when the voice activity and interruption detection module determines that the user is speaking, an interruption event signal is output and sent to the dialogue management module; A dialogue management module for intent recognition based on the real-time text transcription result stream combined with the context to understand the user's intent; The dialogue management module immediately sends a stop instruction to the voice output module when receiving the interruption event signal, forcing the termination of the current voice interaction information playback; A voice output module for outputting complete call transcription text with timestamps and user identifiers, and outputting real-time interactive voice based on the complete call transcription text through an audio player; The voice activity and interruption detection module includes a real-time voice activity detection submodule and an interruption processing submodule, the real-time voice activity detection submodule is used to determine whether there is a voice stream from the user's input in the user's audio; The interruption processing submodule is used to output an interruption event signal; The voice activity detection submodule includes a dual-input neural network model: the dual-input neural network model is used to predict the probability of user voice, and only responds to the user's input voice by suppressing or ignoring the components highly related to AI reference audio in the user's audio; The logic of the interruption processing submodule outputting the interruption event signal is as follows: When the system is playing voice, the voice activity and interruption detection module detects user voice activity through high confidence, the voice activity and interruption detection module immediately triggers a high-priority interruption event signal, and sends it to the dialogue management module through a bidirectional streaming network, the generation delay of the interruption event signal is less than 100 milliseconds, which is much faster than the time required for the streaming acoustic decoding module to output the first word, ensuring that the system reacts in time and immediately stops outputting real-time interactive voice.

2. The low-latency streaming voice interaction system with a break processing function according to claim 1, wherein, The original digital audio stream is 8kHz or 16kHz sampling rate pulse code modulation PCM data.

3. The low-latency streaming voice interaction system with a break processing function according to claim 1, wherein, The specific steps of real-time acoustic feature extraction include: Divide the continuously input original digital audio stream into short time frames of fixed length, and extract the acoustic feature vector of each frame by sliding with a fixed step length less than the fixed length.

4. The low-latency streaming voice interaction system with a break processing function according to claim 1, wherein, The deep learning model is trained to predict and suppress noise and AI audio leakage and echo components in the original digital audio stream of the user by receiving the original digital audio stream of the noisy user and the AI playback audio reference signal as input, and extract a noise-free user speech signal or its corresponding acoustic features, obtaining a real-time acoustic feature sequence stream.

5. The low-latency streaming voice interaction system with a break processing function according to claim 1, wherein, The Transducer-based causal streaming architecture acoustic model is an ASR model, and the streaming decoding algorithm used by the system realizes strict synchronization with the output frequency of the ASR model and low-delay output through the following cooperative strategies, specifically including: The decoder in the ASR model updates the decoding state and searches for the optimal text path in real time in a frame-synchronous or block-synchronous manner by receiving the real-time probability stream output by the acoustic model in combination with the streaming language model information, and the decoder uses an aggressive decoding strategy with a small initial beam width, and once the confidence of a certain path reaches a preset threshold, it immediately outputs partial text; the real-time text transcription result stream output during the decoding process is dynamically updated and corrected by the decoder until the user is silent.

6. The low-latency streaming voice interaction system with a break processing function according to claim 1, wherein, The voice activity and interruption detection module is also used to detect voice activity and identify whether the voice stream belongs to the target user, and the specific steps are as follows: By comparing the real-time extracted user voiceprint embedding with the pre-stored or quickly learned user voiceprint at the initial stage of intelligent interaction, the starting and ending time stamps of the user voice segment are accurately marked, and the user is identified; and the detected non-AI and non-main user voice segments are distinguished by speakers, and different speaker IDs are assigned; when there is voice overlap between different speakers on the user side, real-time voice separation technology is used to separate the overlapping user voice streams.

Images