Real-time full-duplex conversational speech-oriented streaming conversational state prediction method and system

Abstract

Embodiments of the present application provide a kind of real-time full duplex voice dialogue-oriented streaming dialogue state prediction method and system.The method comprises: the processed speech mark sequence is input to the dialogue state prediction model of multimodal full duplex, carries out streaming dialogue state prediction, adopts staggered prediction mechanism, the current speech feature, text recognition result and dialogue state mark are modeled on time dimension, for realizing that text recognition result explicitly participates in the judgment process of dialogue state prediction, obtains the full duplex dialogue state mark of streaming continuous prediction;The interactive state of user is judged in real time based on the full duplex dialogue state mark of continuous prediction, to determine the control behavior of downstream dialogue.The embodiments of the present application reduce the overall computing overhead and real-time inference pressure of system, more accurately identify complex interactive behaviors such as user sentence completion, pause and interruption, etc.Improve the overall response performance and user experience of full duplex voice interaction system.

Description

Technical Field

[0001] This invention relates to the field of intelligent voice, and more particularly to a method, system, electronic device, and storage medium for predicting the state of streaming dialogue for real-time full-duplex voice conversations. Background Technology

[0002] Spoken Dialogue Models (SDMs) traditionally operate in half-duplex mode, with input reception and response output strictly separated in time. In contrast, Full-Duplex Spoken Dialogue Systems (FD-SDS) can listen and speak simultaneously, supporting interruption handling, pause handling, and dynamic response, making them more suitable for user interaction.

[0003] In the field of existing full-duplex voice dialogue systems, based on different system modeling methods, they can be broadly divided into two categories: (1) End-to-end full-duplex voice dialogue model These methods simultaneously perform speech understanding, dialogue management, and speech generation using a single model, enabling the system to process user speech input and system speech output within the same framework. Specifically, end-to-end full-duplex speech dialogue models typically employ a unified neural network model to jointly model user speech input and system speech output. At each time step, these methods simultaneously receive the user-side speech stream and the system-side generated speech stream, predicting the next system output segment through joint representation learning within the model, thereby achieving two-stream speech interaction between the user and the system. Furthermore, some methods introduce dialogue state prediction mechanisms within the end-to-end framework, controlling the model's speech output behavior by predicting whether the system is currently in a "listening" or "responding" state.

[0004] (2) Modular full-duplex voice dialogue system These methods typically build upon traditional half-duplex voice dialogue systems by introducing an independent dialogue state control module to manage dialogue turns and respond to user voice messages, thereby supporting full-duplex voice interaction capabilities. Specifically, the system first detects user voice using Voice Activity Detection (VAD), then performs Automatic Speech Recognition (ASR) on the segmented speech segments, and finally determines whether the user's statement has ended and whether the system needs to respond based on the recognized text results (Turn Detection).

[0005] Both of these technologies attempt to address issues such as real-time interaction and interruption handling faced by voice dialogue systems in real-world human-computer interaction scenarios.

[0006] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology: End-to-end full-duplex models are heavily reliant on high-quality training data. Full-duplex voice interaction requires dialogue data with precise time annotations, but acquiring such data is costly, making model training difficult and prone to overfitting to specific scenarios. Furthermore, end-to-end models incur significant computational overhead during real-time deployment. Because the model continuously processes the speech stream and simultaneously generates the system's speech output, larger models are prone to high computational burdens and latency, impacting real-time interaction performance and limiting model scalability. In modular systems, dialogue state control typically relies on traditional speech activity detection methods. These methods primarily rely on acoustic features for judgment and lack the ability to understand semantic information. Therefore, they often struggle to accurately identify dialogue boundaries in complex interaction scenarios such as user pauses, interruptions, or incomplete semantics. Furthermore, existing dialogue control modules still employ non-streaming processing for speech recognition and turn detection, introducing additional response latency that increases with the duration of user input, thus affecting the fluency of human-computer dialogue. Summary of the Invention

[0007] In order to at least address the problem that existing full-duplex models tend to generate high computational burden and latency when the model size is large, thus affecting the real-time interactive performance and smoothness of the system.

[0008] In a first aspect, embodiments of the present invention provide a streaming dialogue state prediction method for real-time full-duplex voice dialogue, including: Continuously receive continuous speech signals from the user side, and convert the continuous speech signals into discrete speech tag sequences based on a speech tagger streaming method; The speech-text multimodal feature consistency representation processing of the speech-text sequence is performed using a multilayer perceptron; The processed speech-tagged sequence is input into a multimodal full-duplex dialogue state prediction model for streaming dialogue state prediction. - In each time step, historical and current speech features are determined based on the speech tag sequence to obtain the text recognition result of the speech segment in the current time step; - Based on the current speech features, historical text recognition results, and historical dialogue state information, predict the dialogue state marker used to describe the user's current speech interaction attributes; - An interleaved prediction mechanism is adopted to uniformly model the current speech features, text recognition results and dialogue state markers in the time dimension, so as to explicitly participate the text recognition results in the judgment process of dialogue state prediction and obtain full-duplex dialogue state markers for streaming continuous prediction. Based on the continuously predicted full-duplex dialogue state markers, the user's interaction state is determined in real time to determine the control behavior of the downstream dialogue.

[0009] Secondly, embodiments of the present invention provide a streaming dialogue state prediction system for real-time full-duplex voice dialogue, comprising: A voice input module is used to continuously receive continuous voice signals from the user side and convert the continuous voice signals into discrete voice tag sequences based on a voice tagger streaming method. The feature representation module is used to perform speech-text multimodal feature consistency representation processing on the speech tag sequence using a multilayer perceptron; The streaming state prediction module is used to input the processed speech tag sequence into a multimodal full-duplex dialogue state prediction model to perform streaming dialogue state prediction. In each time step, based on the speech tag sequence, the historical and current speech features are determined to obtain the text recognition result of the speech segment in the current time step. Based on the current speech features, historical text recognition results, and historical dialogue state information, a dialogue state tag describing the user's current speech interaction attributes is predicted. An interleaved prediction mechanism is adopted to uniformly model the current speech features, text recognition results, and dialogue state tags in the time dimension, so as to explicitly participate the text recognition results in the judgment process of dialogue state prediction and obtain streaming continuously predicted full-duplex dialogue state tags. The decision output module is used to make real-time judgments on the user's interaction state based on the continuously predicted full-duplex dialogue state markers, so as to determine the control behavior of the downstream dialogue.

[0010] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of a streaming dialogue state prediction method for real-time full-duplex voice dialogue according to any embodiment of the present invention.

[0011] Fourthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the streaming dialogue state prediction method for real-time full-duplex voice dialogue according to any embodiment of the present invention.

[0012] Fifthly, embodiments of the present invention provide a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the streaming dialogue state prediction method for real-time full-duplex voice dialogue according to any embodiment of the present invention.

[0013] The beneficial effects of this invention are as follows: At the system deployment level, by centralizing the real-time-critical dialogue state control function into a lightweight state prediction module, it runs continuously during inference, while the half-duplex voice dialogue model is only invoked when a response needs to be generated. This design significantly reduces the overall computational overhead and real-time inference pressure of the system, thereby reducing overall deployment costs while ensuring smooth interaction and controlling the cost increases brought about by model scaling. Regarding dialogue state modeling, the text-guided state prediction method introduces text information generated by speech recognition during streaming processing, enabling the model to explicitly utilize semantic information for user intent understanding and dialogue state judgment. Compared to methods relying solely on acoustic features, it has stronger discriminative capabilities at the semantic level, and can more accurately identify complex interactive behaviors such as user sentence completion, pauses, and interruptions. Furthermore, by combining end-to-end optimization strategies with a teacher-forced inference mechanism, this invention achieves efficient collaboration between streaming speech recognition and dialogue state prediction, ensuring recognition accuracy while achieving low-latency real-time dialogue management, significantly improving the overall response performance and user experience of the full-duplex voice interaction system. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a streaming dialogue state prediction method for real-time full-duplex voice dialogue provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the Plugin-FD architecture for a streaming dialogue state prediction method for real-time full-duplex voice dialogue provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of a three-stage hybrid training method for predicting the state of streaming dialogue for real-time full-duplex voice dialogue, provided by an embodiment of the present invention. Figure 4This is a schematic diagram of the state token design and streaming inference of Plugin-FD, a streaming dialogue state prediction method for real-time full-duplex voice dialogue provided by an embodiment of the present invention. Figure 5 This is a schematic diagram of the results of a full-duplex Chinese-English bilingual benchmark test set for a streaming dialogue state prediction method for real-time full-duplex voice dialogue provided by an embodiment of the present invention. In this diagram, the latency indicators are all in seconds, the best results are shown in bold, and the second best results are shown in underline. Figure 6 This is a schematic diagram comparing a streaming dialogue state prediction method for real-time full-duplex voice dialogue provided by an embodiment of the present invention with a non-streaming state prediction module on the bilingual Easy Turn test set. Figure 7 This is a schematic diagram comparing the latency of a streaming dialogue state prediction method for real-time full-duplex voice dialogue provided by an embodiment of the present invention with that of different streaming state prediction modules. Figure 8 This is a schematic diagram of the ablation results of a streaming dialogue state prediction method for real-time full-duplex voice dialogue provided by an embodiment of the present invention on the Easy Turn test set. Figure 9 This is a schematic diagram of the structure of a streaming dialogue state prediction system for real-time full-duplex voice dialogue provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of an embodiment of an electronic device for predicting the state of streaming dialogue for real-time full-duplex voice conversations, as provided in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] like Figure 1 The diagram shows a flowchart of a streaming dialogue state prediction method for real-time full-duplex voice dialogue according to an embodiment of the present invention, including the following steps: S11: Continuously receive continuous speech signals from the user side, and convert the continuous speech signals into discrete speech tag sequences based on a speech tagger streaming method; S12: Use a multilayer perceptron to perform speech-text multimodal feature consistency representation processing on the speech tag sequence; S13: Input the processed speech tag sequence into the multimodal full-duplex dialogue state prediction model to perform streaming dialogue state prediction. - In each time step, historical and current speech features are determined based on the speech tag sequence to obtain the text recognition result of the speech segment in the current time step; - Based on the current speech features, historical text recognition results, and historical dialogue state information, predict the dialogue state marker used to describe the user's current speech interaction attributes; - An interleaved prediction mechanism is adopted to uniformly model the current speech features, text recognition results and dialogue state markers in the time dimension, so as to explicitly participate the text recognition results in the judgment process of dialogue state prediction and obtain full-duplex dialogue state markers for streaming continuous prediction. S14: Based on the continuously predicted full-duplex dialogue state markers, the user's interaction state is judged in real time to determine the control behavior of the downstream dialogue.

[0018] This method introduces an independent streaming dialogue state prediction module, decoupling the turn management capabilities of full-duplex interaction (e.g., in full-duplex dialogue, determining whether the user is still speaking, whether the semantics are complete, and whether an interruption can be triggered for a response) from the voice dialogue system. This enables modular, plug-and-play capabilities. Specifically, the streaming dialogue state prediction method can be configured within the system module and integrated as an independent component into any half-duplex voice dialogue system, giving it full-duplex interaction capabilities. Under this architecture, the original half-duplex model does not need to rely on full-duplex dialogue data for training, thus eliminating the dependence on high-cost full-duplex data. It can utilize richer and more readily available or synthesized turn-based dialogue data for training, effectively alleviating the problem of training data scarcity and reducing the risk of catastrophic forgetting in multi-task or multi-scenario scenarios.

[0019] For step S11, the streaming dialogue state prediction of this method is implemented by a multimodal full-duplex dialogue state prediction model. This multimodal full-duplex dialogue state prediction model can be configured into various full / half-duplex voice dialogue systems. As long as a full / half-duplex device has a speaker and a microphone, it can use this method to achieve real-time full-duplex voice dialogue, realizing modular "plug and play".

[0020] During the voice input phase, the microphone continuously receives continuous voice signals from the user and discretizes the raw audio using a pre-trained voice tokenizer, streaming the continuous voice signal into a discrete sequence of voice tokens. This process is performed in a fixed time window, with each time slice corresponding to a short audio segment, ensuring that the system can continuously receive and process input voice with low latency. This method can employ the GLM-4-Voicetokenizer, a fundamental voice encoder for bilingual speech understanding, to tag large-scale voice data, for example, extracting audio tokens from the user's continuous spoken voice signal at a frequency of 12.5Hz. .

[0021] For step S12, the obtained speech tag sequence is input into the feature mapping module, and the speech tags are converted into a feature representation space consistent with the language model through a multilayer perceptron, thereby realizing the alignment between speech information and subsequent language modeling modules and providing a foundation for multimodal joint modeling.

[0022] For example, for streaming inference, this method encodes a 160ms target window for each time step using a 960ms left context (look-back) and a 40ms right context (look-ahead), resulting in a total receptive field of 1160ms and 15 extracted tokens. The tokens corresponding to the target region are aligned with the second-to-last and third-to-last tokens within the block. Then, the linear encoder projector... The embeddings are transformed into features A to ensure alignment with the embedding dimensions of LLMs, defined as follows: .

[0023] For step S13, the processed speech tag sequence is input into the multimodal full-duplex dialogue state prediction model.

[0024] As one implementation, the multimodal full-duplex dialogue state prediction model includes: VAD, ASR, and VAD-ASR-Turn Detection for text-guided turn detection within the same streaming model framework.

[0025] In this embodiment, the multimodal full-duplex dialogue state prediction model of this method can be called Plugin-FD (component-independent full-duplex), such as... Figure 2As shown, it integrates VAD (Voice-Aided Detection) for acquiring continuous speech signals from the user side, ASR (Automatic Speech Recognition) for text recognition, and VAD-ASR-Turn Detection for state prediction in text-guided turn detection into a single model framework, rather than a cascaded structure of VAD, ASR, and Turn Detection. Specifically, the model operates using interleaved audio tokens, text tokens, and state tokens, ensuring that textual information is provided when predicting state tokens. During training, VAD, ASR, and state prediction are all optimized end-to-end. During inference, a lightweight, state-of-the-art ASR model is provided with textual guidance enforced by the teacher.

[0026] As one implementation, the hybrid training of the multimodal full-duplex dialogue state prediction model by teacher-forced reasoning includes: The first stage involves constructing non-streaming automatic speech recognition data to pre-train the speech recognition module's recognition capabilities. In the second stage, timestamps are added to the non-streaming automatic speech recognition data to obtain streaming automatic speech recognition data, and the streaming automatic speech recognition data is used to perform streaming adaptation of the speech recognition module. In the third stage, VAD, ASR, and VAD-ASR-Turn Detection are jointly interleaved for prediction training, which is used to achieve streaming prediction in full-duplex speech dialogue while being guided by the semantics corresponding to the text recognition results.

[0027] In this implementation, the hybrid three-stage training combined with teacher-forced reasoning strategies is as follows: Figure 3 As shown, the process includes: non-streaming ASR pre-training, streaming ASR adaptation, and duplex state prediction fine-tuning. The first two stages focus on speech recognition capabilities, while the final stage optimizes the model end-to-end across VAD, ASR, and state prediction tasks. In the inference strategy, a teacher-forced ASR text is provided using a teacher model. The teacher only provides the ASR text during inference and does not directly participate in training.

[0028] Regarding the joint interleaved prediction training of the activity speech detection module, speech recognition module, and state prediction module in the third stage, this method innovatively introduces a joint ASR objective and designs an interleaved prediction paradigm: Each 160ms audio block corresponds to two audio tokens, and the t-th block... Based on history The model first predicts the ASR token sequence for the current block: in, This indicates a streaming ASR output aligned with block t. (In generating...) Next, the model predicts the dialogue state token: in, This represents the duplex state associated with the current block. This interleaved design provides explicit semantic guidance for state prediction while maintaining streaming reasoning.

[0029] In the above training, the loss from mixed training includes: in, This represents the complete target dialogue state tag sequence. Indicates the first A dialogue status marker, Indicates the prediction of the dialogue state flag. Cross-entropy loss, Used to map each dialogue state flag to its corresponding voice interaction attribute type. These are weighting coefficients differentiated by the type, used to balance the optimization process among different heterogeneous categories.

[0030] The above method enables hybrid pre-training of a multimodal full-duplex dialogue state prediction model. This hybrid pre-trained model is then used for streaming dialogue state prediction. The method employs a text-guided joint modeling approach, processing the speech input step-by-step for each time slice.

[0031] Specifically, at each time step, the multimodal full-duplex dialogue state prediction model first generates corresponding speech recognition text results based on current and historical speech features. This is done to characterize the semantic information of the current speech segment. Subsequently, by fusing current speech features, historical text information, and historical dialogue state information, it further predicts a dialogue state label to describe the user's current speech interaction attributes. This dialogue state describes the interaction attributes of the current user's speech, including whether there is valid semantics, whether it is an echoing behavior, and whether the semantics are complete, thereby achieving fine-grained modeling of the dialogue process.

[0032] As one implementation, the types used to describe the user's voice interaction attributes include: interaction attributes indicating the existence of valid semantics, interaction attributes indicating agreement behavior, and interaction attributes indicating whether the semantics are complete.

[0033] In this implementation, the method defines five dialogue state tokens to simulate the interactive dynamics in full-duplex spoken dialogue, such as... Figure 4 As shown: *<|user_idle|> indicates that the current audio block does not contain semantic content, such as silence or noise.

[0034] *<|user_nonidle|> indicates that the audio block contains semantics and is meaningful speech.

[0035] *<|user_backchannel|> indicates the user's echoing feedback behavior.

[0036] *<|user_complete|> indicates that the user's statement is semantically complete, and the system can respond to it.

[0037] *<|user_incomplete|> indicates that the user is pausing, but his / her utterance is semantically incomplete, so the system should wait.

[0038] Furthermore, this method employs an interleaved prediction mechanism to uniformly model speech features, text output, and state labels in the time dimension, enabling text information to explicitly participate in the state judgment process, thereby significantly improving the model's discriminative ability at the semantic level.

[0039] As one implementation, the multimodal full-duplex dialogue state prediction model is scalable, and the multimodal full-duplex dialogue state prediction model further includes: a scalable external speech recognition model, used to reduce the latency of dialogue state prediction.

[0040] In this implementation, to balance accuracy and real-time performance during actual inference, this method employs an inference strategy that combines external speech recognition results. Specifically, a lightweight speech recognition module provides high-precision text input for each time slice, thereby assisting the state prediction model in making more accurate semantic judgments. This design significantly improves the stability and accuracy of dialogue state prediction while maintaining low overall system latency. Specifically, a lightweight external ASR model (e.g., Paraformer or SenseVoice Small) can be used to provide teacher-mandated streaming ASR output for each block. This design retains the benefits of unified end-to-end training while ensuring more accurate and efficient real-time deployment. Full-duplex dialogue state labels are obtained through the above joint interleaved prediction.

[0041] For step S14, the multimodal full-duplex dialogue state prediction model of this method outputs a streaming, continuously predicted full-duplex dialogue state label. During the system output decision phase, the system uses the streaming, continuously predicted full-duplex dialogue state label to make real-time judgments on the user's voice interaction state and controls the behavior of the downstream dialogue system accordingly, for example... For example, when the dialogue state is marked as *<|user_incomplete|>, it indicates that the user is pausing, and the system should continue to wait for user input. If the dialogue state continues to *<|user_complete|>, although the user has not finished speaking, the semantics are complete, and the system can then take over. Based on the continuously predicted full-duplex dialogue state markers, the system can then decide whether to respond, continue waiting for user input, or handle user interruptions, thereby achieving natural and fluent full-duplex voice interaction.

[0042] Regarding latency, the multimodal full-duplex dialogue state prediction model of this method is applied to full / half-duplex voice dialogue systems, for example, where the fixed duration of an audio block is 160 milliseconds. It is assumed that the user's voice occurs within a block... Internal termination. Because the model operates in a streaming manner, it can only determine the absence of subsequent active speech after the next block arrives. In processing chunks... When the model fails to detect active speech, it considers the <|user_nonidle|> state to have ended, and then determines whether the preceding semantic content is complete (i.e., <|user_collect|> or <|user_incomplete|>). Since the actual endpoint of the user's speech may appear in a block... At any location within the region, under the assumption of uniform distribution, the theoretical average delay of Plugin-FD is: =80 milliseconds + 160 milliseconds = 240 milliseconds. This allows for full-duplex dialogue with higher semantic understanding capabilities to be provided for half-duplex voice dialogue systems, while ensuring low latency.

[0043] As can be seen from this implementation, at the system deployment level, this method centralizes the real-time-critical dialogue state control function into a lightweight state prediction module, allowing it to run continuously during inference, while the half-duplex voice dialogue model is only invoked when a response needs to be generated. This design significantly reduces the overall computational overhead and real-time inference pressure of the system, thereby reducing overall deployment costs while ensuring smooth interaction and controlling the cost increases brought about by model scaling. Regarding dialogue state modeling, the text-guided state prediction method introduces text information generated by speech recognition during streaming processing, enabling the model to explicitly utilize semantic information to understand user intent and determine dialogue state. Compared to methods relying solely on acoustic features, this method has stronger discriminative capabilities at the semantic level, more accurately identifying complex interactive behaviors such as user sentence completion, pauses, and interruptions. Furthermore, by combining end-to-end optimization strategies with a teacher-forced inference mechanism, this invention achieves efficient collaboration between streaming speech recognition and dialogue state prediction, ensuring recognition accuracy while achieving low-latency real-time dialogue management, significantly improving the overall response performance and user experience of the full-duplex voice interaction system.

[0044] Similarly, this method can be integrated into a hardware functional box. Users can plug and play this functional box with existing smart conversational devices in their homes (for example, these smart conversational devices may be half-duplex due to early purchase; the connection can be hardware-level, using a plug-and-play audio cable, or software-level, with the functional box wirelessly connecting to the smart conversational device). In use, the functional box receiving the user's voice performs full-duplex voice conversation streaming state prediction, integrates the state and input, and then controls the conversational behavior of the smart conversational device, which then outputs feedback to the user. In this way, the functional box with this method essentially bridges the gap between the user and existing half-duplex smart voice conversational devices, enabling full-duplex conversation.

[0045] Experiments illustrate this method. Specifically, for ASR training of Plugin-FD, large-scale Mandarin and English corpora were used. The Mandarin data included AISHELL-1, AISHELL-3, WenetSpeech, and subsets of CommonVoice CN, Emilia CN, and MAGICDATA from VoxBox, totaling approximately 47,000 hours. The English data included LibriSpeech, GigaSpeech, and subsets of CommonVoice EN and Emilia EN from VoxBox, totaling approximately 31,000 hours. For streaming ASR training, this method first obtains character-level or word-level alignment, then reorganizes the data in an interleaved, block-based format. Timestamps for the Mandarin corpus were generated using Paraformer, while timestamps for the English corpus were generated using WhisperX.

[0046] During the state prediction training phase, the Fisher dataset was used for training on English at the thousand-hour level. For Mandarin, due to the lack of a suitable open-source dataset, a ten-thousand-hour internal corpus constructed in the same format as Fisher was used. This method first performs alignment, filtering samples based on bi-ASR consistency for the Mandarin data. To improve robustness, this method globally adds Musan noise to the training data or adds ESC-50 noise to silent segments. State labels are annotated using Qwen2.5-72B-Instruct. Plugin-FD uses a pre-trained GLM-4-Voice tagger as the speech encoder and Qwen3-0.6B as the LLM backbone. The speech tagger remains frozen throughout training. During ASR pre-training, both the LLM and adapter layers were fully fine-tuned. In state prediction training, this method applied LoRA fine-tuning with rank r=32 to the LLM on the bilingual training set. All training was performed on an NVIDIA H20 GPU.

[0047] During inference, teacher-mandated ASR was applied to provide more accurate text guidance. For Mandarin, Paraformer was used as the ASR teacher, while for English, SenseVoice Small was employed. All evaluations were performed in a simulated online streaming inference setup on a single NVIDIA L20 GPU.

[0048] To evaluate the dialogue state control capabilities of Plugin-FD, this method builds a modular FD-SDS based on it, using Qwen2.5-7B-Instruct as the LLM and IndexTTS-1.52 as the TTS model. The system is evaluated against selected tasks from the FDB (Full-Duplex-Bench) series, including user pause handling, turn transition judgment, and user interruption in FDB v1, and user agreement feedback and user interruption in FDB v1.5. Furthermore, Freeze-Omni and this method's systems are evaluated on a Chinese test set to assess cross-language performance. Official metrics defined in FDB are used. For user pause handling, the takeover rate (TOR) measures the frequency with which the system takes over the speaking role when a user pauses but has not yet finished speaking; a lower value indicates better user pause management. For turn transitions, TOR represents the proportion of successful dialogue turn transitions, and RL (Response Latency) measures the delay between the end of a user's speech and the start of a system response. For user interruption v1, TOR assesses whether the system performs a rotation after the interruption, and RL measures the response latency after the interruption. For user echo feedback, RsR (Resume Rate) is reported, defined as the proportion of resumed behavior after an overlap event. For user interruption v1.5, this method reports RpR (Respond Rate), which represents the proportion of responding behavior. Additionally, SL (Stop Latency) measures the latency from the start of the user interruption to the system stopping speaking, and RL measures the latency from the end of the interruption to the subsequent system response. This method summarizes these metrics to obtain an overall metric for system performance: 1-TOR for pause handling is averaged with TOR, RsR, and RpR for other tasks; for overall latency, the average of all RL and SL values ​​across tasks is calculated. Paraformer is used for ASR when evaluating the Chinese subset. For duplex state prediction, this method further evaluates the model on the Easy Turn test set, using prediction accuracy (ACC) and inference latency as evaluation metrics for further comparison and ablation studies.

[0049] Regarding the experimental results, this method constructs a cascaded full-duplex dialogue system by integrating Plugin-FD (as the speech understanding and state management module) with Qwen2.5-7B-Instruct for response generation and IndexTTS-1.5 for speech synthesis. The entire system was evaluated on a Chinese-English bilingual full-duplex benchmark set, and the results are summarized in... Figure 5Overall, the Plugin-FD-based system demonstrated balanced performance across all evaluation dimensions, achieving the highest average scores in overall interaction timing and round control, without exhibiting abnormally low values ​​in any individual metric. Despite employing a cascaded architecture, it consistently maintained low latency, ranking second on average in English and first in Chinese, indicating that the streaming round transition judgment control module effectively supports real-time interaction.

[0050] End-to-end continuous output models (i.e., dGSM, PersonaPlex, and Moshi) achieve very high answer rates and low response latency. However, these models also exhibit high TOR in user pause handling, indicating that their strong turn-switching judgment performance is partly achieved by shifting action points to more aggressive reactive behaviors. In contrast, the Plugin-FD system achieves a TOR of 0.933 in dialogue turn-switching, approaching the best reporting performance, while maintaining a significantly lower probability of unwarranted interruptions in user pause handling compared to dGSM, PersonaPlex, and Moshi. Furthermore, our proposed system consistently outperforms Freeze Omni on all three relevant metrics across both Chinese and English test sets. Compared to Gemini, our proposed system improves the TOR of dialogue turn-switching by more than 40%. Although Gemini has a slightly lower TOR in user pause handling, its response latency is more than twice that of Plugin-FD, reflecting a different trade-off between conservatism and responsiveness.

[0051] In user interruption scenarios, v1 and v1.5 shared the same test data but used different evaluation protocols. Plugin-FD performed exceptionally well in both v1.TOR and v1.5 response rate, ranking highest in Chinese and second in English. Regarding latency, it achieved the lowest response latency under the v1.5 metric, with its stopping latency second only to GPT-4o in English. Freeze Omni exhibited a higher TOR under the Chinese v1 metric, but its response rate was only 0.44 and its recovery rate reached 0.47 under the v1.5 evaluation. Manual examination of a subset of the test audio samples revealed that the model occasionally failed to respond to user interruptions. This discrepancy suggests that the v1 evaluation method may incorrectly capture the latter part of the model's first-round response, leading to excessively high TOR and latency scores. Moshi and Gemini also observed similar metric behavior on the English test set.

[0052] Models with high recovery rates tend to exhibit longer stopping delays in interruption tests for user agreement, typically exceeding 1 second and in some cases exceeding 2 seconds. In contrast, GPT-4o has a stopping delay of 0.23 seconds but a recovery rate of 0.7, indicating a clear trade-off between agreement feedback processing and response to user interruptions. Plugin-FD, with a stopping delay of 0.45 seconds and a recovery rate of 0.74, represents a balance point without extreme bias towards either objective.

[0053] like Figure 6 As shown, our method was evaluated against non-streaming state prediction modules on the bilingual EasyTurn test set. The performance of the streaming Plugin-FD and open-source non-streaming modules was compared on the bilingual EasyTurn test set, comparing their performance in terms of latency and dialogue state prediction. Despite streaming modeling, a fair comparison was made with a cascaded baseline consisting of a state-of-the-art ASR model and a 7B-parameter TENTurn Detection, where neither Plugin-FD nor this baseline was trained on the EasyTurn training set. Plugin-FD's accuracy was only about 3% lower, which is within an acceptable range. This result demonstrates that even compared to large-scale non-streaming baselines, uniform text-guided streaming state judgment modeling does not affect the accuracy of turn transition judgment detection.

[0054] Plugin-FD's streaming design achieves consistently low latency and stable latency. In contrast, in real-world deployment scenarios, non-streaming methods must rely on an external VAD model to segment the audio before inference. This VAD-based truncation introduces additional latency, typically several hundred milliseconds (e.g., the latency of the VAD model used for comparison was reported as 500 milliseconds in Flexduo). Furthermore, this latency can increase with longer input times. Therefore, while non-streaming systems may appear to achieve higher accuracy, they produce larger and more unpredictable response latency in real-world applications.

[0055] To quantify the independent latency of Plugin-FD in a real-world dialogue system, this method measures the turn-by-turn transition judgment task on a full-duplex benchmark set. Specifically, the latency of the first data packet from the downstream LLM and TTS modules is recorded during inference. Then, the latency of Plugin-FD is estimated by subtracting the latency of the first data packet from the LLM and TTS modules from the call latency reported in the evaluation metrics, as shown in the figure. Figure 7 As shown.

[0056] Since the FlexDuo model and evaluation set are not publicly available, its latency cannot be evaluated under the same experimental settings. However, for the streaming module, latency typically varies only slightly across test sets, so figures from the FlexDuo paper are used for comparison. As shown in the table, in the streaming state prediction module, Plugin-FD achieved an average latency of 250 ms for both Chinese and English in a real deployment, significantly lower than methods based on FlexDuo and VAD. This improvement can be attributed to the small chunk size used in Plugin-FD and its lightweight components, particularly the backbone LLM. Notably, the actual latency (250 ms) is close to the theoretical latency (240 ms), confirming that teacher-forced inference does not introduce additional latency.

[0057] Ablation experiments were conducted on this method to examine the effectiveness of the training and inference strategies used in Plugin-FD. The results are as follows: Figure 8 As shown.

[0058] First, when the first and second stages of ASR pre-training are removed, the state prediction accuracy decreases significantly. This observation indicates that the model relies on its ASR capabilities when performing state prediction tasks. The ASR pre-training stage provides improved semantic representations, which in turn benefit supervised fine-tuning in the third stage and downstream inference.

[0059] Furthermore, prediction accuracy decreases when externally guided ASR is removed during inference. This result validates the effectiveness of our method. With very short chunk settings, externally provided ASR output offers more reliable textual supervision, thus stabilizing semantic interpretation. The performance degradation further confirms that the model does indeed utilize explicit textual semantic information to support streaming state prediction.

[0060] In summary, based on the overall framework and implementation of the dialogue state prediction module proposed in this paper, the constructed full-duplex voice dialogue system achieved excellent performance on the Chinese-English bilingual benchmark Full-Duplex-Bench. Specifically, it achieved superior performance among current full-duplex models in terms of dialogue turn control and system response latency, validating the effectiveness and stability of this scheme in real-time interactive scenarios.

[0061] Beyond its direct effects, this method achieves structural decoupling and modularity by independently designing and lightweighting the dialogue state prediction module, leading to a range of application potentials. Firstly, this design significantly reduces the system's reliance on full-duplex speech data, allowing the training of the voice dialogue model to depend more on highly structured, lower-cost turn-based data, thereby improving data utilization efficiency and lowering the training threshold. Secondly, this module can be flexibly integrated into different types of voice dialogue systems as a general-purpose component, contributing to improved overall system scalability and engineering reusability. Furthermore, this method, through text-guided dialogue state modeling, enables the system to more accurately understand user intent at the semantic level, thus supporting more granular interactive behavior modeling. For example, it can be further extended to support more interactive behaviors such as system-initiated agreement or interruption, providing a possible direction for achieving a more natural and human-like voice dialogue experience.

[0062] like Figure 9 The diagram shown is a schematic representation of a streaming dialogue state prediction system for real-time full-duplex voice dialogue provided in an embodiment of the present invention. This system can execute the streaming dialogue state prediction method for real-time full-duplex voice dialogue described in any of the above embodiments and is configured in a terminal.

[0063] This embodiment provides a streaming dialogue state prediction system 10 for real-time full-duplex voice dialogue, which includes: a voice input module 11, a feature representation module 12, a streaming state prediction module 13, and a decision output module 14.

[0064] The system includes a voice input module 11 for continuously receiving continuous voice signals from the user side and converting them into discrete voice tag sequences based on a voice tagger. A feature representation module 12 uses a multilayer perceptron to perform multimodal feature consistency representation processing on the voice tag sequences. A streaming state prediction module 13 inputs the processed voice tag sequences into a multimodal full-duplex dialogue state prediction model for streaming dialogue state prediction. At each time step, it determines historical and current voice features based on the voice tag sequences to obtain the text recognition result of the voice segment within the current time step. Based on the current speech features, historical text recognition results, and historical dialogue state information, a dialogue state marker describing the user's current speech interaction attributes is predicted. An interleaved prediction mechanism is adopted to uniformly model the current speech features, text recognition results, and dialogue state markers in the time dimension, so as to explicitly participate the text recognition results in the judgment process of dialogue state prediction, and obtain a streaming continuously predicted full-duplex dialogue state marker. The decision output module 14 is used to make a real-time judgment on the user's interaction state based on the continuously predicted full-duplex dialogue state marker, so as to determine the control behavior of the downstream dialogue.

[0065] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the streaming dialogue state prediction method for real-time full-duplex voice dialogue in any of the above method embodiments. In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows: Continuously receive continuous speech signals from the user side, and convert the continuous speech signals into discrete speech tag sequences based on a speech tagger streaming method; The speech-text multimodal feature consistency representation processing of the speech-text sequence is performed using a multilayer perceptron; The processed speech-tagged sequence is input into a multimodal full-duplex dialogue state prediction model for streaming dialogue state prediction. - In each time step, historical and current speech features are determined based on the speech tag sequence to obtain the text recognition result of the speech segment in the current time step; - Based on the current speech features, historical text recognition results, and historical dialogue state information, predict the dialogue state marker used to describe the user's current speech interaction attributes; - An interleaved prediction mechanism is adopted to uniformly model the current speech features, text recognition results and dialogue state markers in the time dimension, so as to explicitly participate the text recognition results in the judgment process of dialogue state prediction and obtain full-duplex dialogue state markers for streaming continuous prediction. Based on the continuously predicted full-duplex dialogue state markers, the user's interaction state is determined in real time to determine the control behavior of the downstream dialogue.

[0066] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, they perform the streaming dialogue state prediction method for real-time full-duplex voice dialogue in any of the above method embodiments.

[0067] Figure 10 This is a schematic diagram of the hardware structure of an electronic device for a streaming dialogue state prediction method for real-time full-duplex voice dialogue provided in another embodiment of this application, as shown below. Figure 10 As shown, the device includes: One or more processors 1010 and memory 1020, Figure 10 Taking a processor 1010 as an example, the device for a streaming dialogue state prediction method for real-time full-duplex voice dialogue may further include an input device 1030 and an output device 1040.

[0068] The processor 1010, memory 1020, input device 1030, and output device 1040 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0069] The memory 1020, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the streaming dialogue state prediction method for real-time full-duplex voice dialogue in the embodiments of this application. The processor 1010 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 1020, thereby implementing the streaming dialogue state prediction method for real-time full-duplex voice dialogue in the above-described method embodiments.

[0070] The memory 1020 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 1020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1020 may optionally include memory remotely located relative to the processor 1010, and these remote memories can be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0071] Input device 1030 can receive input numerical or character information. Output device 1040 may include display devices such as a display screen.

[0072] The one or more modules are stored in the memory 1020, and when executed by the one or more processors 1010, they execute the streaming dialogue state prediction method for real-time full-duplex voice dialogue in any of the above method embodiments.

[0073] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0074] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0075] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the streaming dialogue state prediction method for real-time full-duplex voice dialogue according to any embodiment of this invention.

[0076] The electronic devices described in this application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0077] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0078] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0079] (4) Other electronic devices with data processing functions.

[0080] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0081] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0082] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A streaming dialogue state prediction method for real-time full-duplex voice dialogue, comprising: Continuously receive continuous speech signals from the user side, and convert the continuous speech signals into discrete speech tag sequences based on a speech tagger streaming method; The speech-text multimodal feature consistency representation processing of the speech-text sequence is performed using a multilayer perceptron; The processed speech-tagged sequence is input into a multimodal full-duplex dialogue state prediction model for streaming dialogue state prediction. - In each time step, historical and current speech features are determined based on the speech tag sequence to obtain the text recognition result of the speech segment in the current time step; - Based on the current speech features, historical text recognition results, and historical dialogue state information, predict the dialogue state marker used to describe the user's current speech interaction attributes; - An interleaved prediction mechanism is adopted to uniformly model the current speech features, text recognition results and dialogue state markers in the time dimension, so as to explicitly participate the text recognition results in the judgment process of dialogue state prediction and obtain full-duplex dialogue state markers for streaming continuous prediction. Based on the continuously predicted full-duplex dialogue state markers, the user's interaction state is determined in real time to determine the control behavior of the downstream dialogue.

2. The streaming dialogue state prediction method according to claim 1, characterized in that, The multimodal full-duplex dialogue state prediction model includes: VAD, ASR, and VAD-ASR-Turn Detection, which are based on the same streaming model framework.

3. The streaming dialogue state prediction method according to claim 2, characterized in that, The hybrid training of the multimodal full-duplex dialogue state prediction model, which involves teacher-forced reasoning, includes: The first stage involves constructing non-streaming automatic speech recognition data to pre-train the speech recognition module's recognition capabilities. In the second stage, timestamps are added to the non-streaming automatic speech recognition data to obtain streaming automatic speech recognition data, and the streaming automatic speech recognition data is used to perform streaming adaptation of the speech recognition module. In the third stage, VAD, ASR, and VAD-ASR-Turn Detection are jointly interleaved for prediction training, which is used to achieve streaming prediction in full-duplex speech dialogue while being guided by the semantics corresponding to the text recognition results.

4. The streaming dialogue state prediction method according to claim 3, characterized in that, The loss in the hybrid training includes: in, This represents the complete target dialogue state tag sequence. Indicates the first A dialogue status marker, Indicates the prediction of the dialogue state flag. Cross-entropy loss, Used to map each dialogue state flag to its corresponding voice interaction attribute type. These are weighting coefficients differentiated by the type, used to balance the optimization process among different heterogeneous categories.

5. The streaming dialogue state prediction method according to claim 1, characterized in that, The types of attributes used to describe a user's voice interaction include: interaction attributes indicating the existence of valid semantics, interaction attributes indicating agreement behavior, and interaction attributes indicating whether the semantics are complete.

6. The streaming dialogue state prediction method according to claim 2, characterized in that, The multimodal full-duplex dialogue state prediction model is scalable, and the multimodal full-duplex dialogue state prediction model further includes: a scalable external speech recognition model, used to reduce the latency of dialogue state prediction.

7. A streaming dialogue state prediction system for real-time full-duplex voice dialogue, comprising: A voice input module is used to continuously receive continuous voice signals from the user side and convert the continuous voice signals into discrete voice tag sequences based on a voice tagger streaming method. The feature representation module is used to perform speech-text multimodal feature consistency representation processing on the speech tag sequence using a multilayer perceptron; The streaming state prediction module is used to input the processed speech tag sequence into a multimodal full-duplex dialogue state prediction model to perform streaming dialogue state prediction. In each time step, based on the speech tag sequence, the historical and current speech features are determined to obtain the text recognition result of the speech segment in the current time step. Based on the current speech features, historical text recognition results, and historical dialogue state information, a dialogue state tag describing the user's current speech interaction attributes is predicted. An interleaved prediction mechanism is adopted to uniformly model the current speech features, text recognition results, and dialogue state tags in the time dimension, so as to explicitly participate the text recognition results in the judgment process of dialogue state prediction and obtain streaming continuously predicted full-duplex dialogue state tags. The decision output module is used to make real-time judgments on the user's interaction state based on the continuously predicted full-duplex dialogue state markers, so as to determine the control behavior of the downstream dialogue.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-6.

9. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-6.

10. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-6.

Images