A full-duplex voice interaction control method and system based on a large voice understanding model

Abstract

The application discloses a full-duplex voice interaction control method and system based on a large voice understanding model. The method obtains user uplink voice stream, system broadcast state and conversation context information, and obtains user voice start event and stop event through voice activity detection. When the stop event is detected, a first voice segment is formed and input into the large voice understanding model together with the conversation context information; the model triggers transcription when it is judged that the real tail point is reached, and otherwise, the voice segment is kept and the system continues to wait. When the start event is detected during system broadcast, a second voice segment is formed and input into the large voice understanding model together with the system broadcast state and the conversation context information; the model triggers transcription after pausing or terminating the broadcast when it is judged that the effective interruption is formed, and otherwise, the broadcast is kept. The large voice understanding model directly understands the current voice and the context information, and the input does not include the transcription text of the current user statement, so that the tail point judgment and the interruption judgment are completed uniformly before complete transcription, and the false cut-off and the false interruption are reduced.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, speech recognition, speech activity detection, speech understanding, and human-computer speech interaction, specifically to a full-duplex speech interaction control method and system based on a large speech understanding model. Background Technology

[0002] With the development of intelligent customer service, in-vehicle assistants, meeting assistants, smart speakers, and personalized smart terminals, users expect to be able to naturally interject during system announcements and for the system to correctly understand their pauses, thoughts, and supplementary expressions. Full-duplex voice interaction has therefore become an important direction for voice interaction systems.

[0003] Existing full-duplex voice interaction solutions typically cascade modules such as speech recognition, voice activity detection, intent recognition, interruption control, buffer management, and resource management. These solutions often first determine if the user is speaking through voice activity detection, then generate text through speech recognition, and finally use an intent classification model or rule-based thresholds to determine whether the system should interrupt the playback.

[0004] The cascaded scheme described above can improve the response speed of the system when interrupted by the user to some extent, but errors between different modules will propagate step by step. The start or stop events of speech activity detection can only represent changes in speech activity at the acoustic level, and do not necessarily correspond to valid interruptions or true end points at the semantic level.

[0005] In end-point detection scenarios, existing systems typically use the preset duration of the user's voice cessation event output by the voice activity detection module as the basis for determining whether the user has reached the end point. For example, some systems assume that the user has finished speaking after detecting silence lasting for approximately 300 milliseconds or longer.

[0006] However, users often pause mid-sentence when performing tasks such as inquiries, ticket bookings, repair requests, and money transfers. The reasons for this pause might include thinking, searching for information, waiting for prompts, or self-correction. If the system judges whether the user has finished speaking based solely on the duration of silence, it might generate a response prematurely before the user has fully expressed themselves, leading to a degraded user experience and increased subsequent error correction costs.

[0007] In interruption detection scenarios, existing systems often use the detection of user speech, recognition of text length exceeding a threshold, or the output of a specific intent by an intent classification model as the basis for triggering an interruption. This approach struggles to distinguish whether the user truly intends to change the current interaction flow.

[0008] During system broadcasts, there may be simultaneous user echoes, environmental noise, background voices, coughs, laughter, sighs, short responses, or non-system-oriented speech. If the system interrupts broadcasts whenever it detects speech activity, the broadcasts will be frequently interrupted erroneously; if the system relies solely on simple intent classification, it may easily overlook the current broadcast content and the status of historical dialogue tasks.

[0009] Furthermore, speech recognition results may be unstable in short speech, overlapping speech, noisy environments, and colloquial expressions. If the existing system must first obtain the complete transcribed text of the current sentence before making end-point or interruption judgments, it will increase latency and pass transcription errors to the subsequent intent judgment module, resulting in unreliable control results.

[0010] Therefore, a technical solution is needed that can directly utilize the user's current voice and combine it with the system's broadcast status and historical context to make a comprehensive judgment, so as to reduce the complexity of the cascaded judgment link and at the same time improve the accuracy of end point detection and interruption detection. Summary of the Invention

[0011] This invention provides a full-duplex voice interaction control method and system based on a large-scale speech understanding model, aiming to solve the following technical problems in existing full-duplex voice interaction schemes.

[0012] First, the existing tail point detection relies on the duration of silence, which can easily misjudge a user's pause in the middle as the end of the expression.

[0013] Second, existing interruption detection relies on speech activity or text intent classification, which can easily misjudge echoes, background noise, or non-target speech as valid interruptions.

[0014] Third, the existing tail point detection and interruption detection are performed by multiple cascaded modules, which has problems with error propagation and response delay.

[0015] To address the aforementioned issues, the method of this invention includes: acquiring user uplink voice stream, system broadcast status, and session context information during full-duplex voice interaction; performing voice activity detection on the user uplink voice stream to obtain user voice start events and user voice stop events; forming a first voice segment for end-point judgment or a second voice segment for interruption judgment based on different control scenarios; inputting the corresponding voice segment, system broadcast status, and session context information into a large-scale voice understanding model to obtain structured control results; and executing control actions such as continue waiting, transcription, pause broadcasting, terminate broadcasting, or maintain broadcasting based on the structured control results.

[0016] In this invention, the first voice segment is used for end-point determination. The first voice segment is formed by the voice buffer between a user voice start event and a corresponding user voice stop event, and may include a short buffer after the user voice stop event, such as a voice buffer of about 100 milliseconds after the stop event.

[0017] In this invention, the second voice segment is used for interruption detection. The second voice segment is formed by a short voice window following the user's voice start event when the system is in voice broadcast mode, and the length of this short voice window can be set to at least 1 second.

[0018] When a user continues to speak during system announcements, the interruption detection window is not limited to a single, fixed-length window. The system can capture subsequent audio windows at preset intervals, such as once every second, and continuously determine whether a valid interruption has been constituted by combining cached user audio, system announcement status, and conversation context information.

[0019] In this invention, the input of the large-scale speech understanding model does not depend on the complete transcribed text of the current sentence. The large-scale speech understanding model directly receives the speech encoding representation of the first or second speech segment, obtains the corresponding speech encoding representation through the speech encoding unit, and then outputs a structured control result by combining the conversation context information and the system broadcast status.

[0020] When the speech understanding big model outputs the true tail point result, the tail point control module triggers the transcription interface to transcribe the first speech segment or the merged speech segment and writes the transcribed text into the session record.

[0021] When the speech understanding big model outputs a valid interruption result, the interruption control module first pauses or terminates the current speech playback according to the recommended control action; then, the transcription interface transcribes the user speech segment corresponding to the valid interruption and writes the transcribed text into the session record.

[0022] By using the above timing sequence, the control judgment of the present invention precedes the generation of the complete ASR text, which can reduce response latency and reduce the impact of current sentence transcription errors on end point judgment and interruption judgment.

[0023] The present invention also provides a system comprising a speech activity detection module, an echo processing module, a speech understanding module, a tail point control module, an interruption control module, a context management pool, a transcription interface, a response text generation interface, and a speech synthesis interface.

[0024] The echo processing module is used to suppress or cancel echoes in the user's uplink voice stream in full-duplex scenarios, thereby reducing the interference of downlink audio broadcasts on user speech detection and speech understanding. The echo processing module can be set as an optional module to enhance the recognizability of the user's uplink voice stream in full-duplex scenarios; in implementations without an echo processing module, the voice activity detection module and the speech understanding module can also directly process the user's uplink voice stream.

[0025] The speech understanding module deploys a large-scale speech understanding model. This large-scale speech understanding model includes a speech coding unit, a context coding unit, a fusion understanding unit, and a control output unit.

[0026] The speech coding unit is used to convert user speech segments into a speech coding representation. This speech coding unit can adopt a structure similar to the audio encoder in an end-to-end speech recognition model to perform audio framing, acoustic feature extraction, and sequence coding.

[0027] The context encoding unit is used to convert session context information, system broadcast status, and task prompts into text status representations. Session context information may include historical user input, historical system responses, current broadcast text, broadcast progress, task status, and information to be completed.

[0028] The fusion understanding unit is used to jointly model the speech coding representation and the text state representation. This joint modeling can be achieved through methods such as concatenation, adapter mapping, cross-attention, multimodal encoders, or cue template concatenation.

[0029] The structured control result includes at least a control identifier and a recommended control action corresponding to the current control scenario; wherein, in the tail point judgment scenario, the control identifier includes a tail point identifier; and in the interruption judgment scenario, the control identifier includes an interruption identifier. The structured control result may also include a confidence level, a cause field, or a status field.

[0030] Since the large speech understanding model directly receives the user's speech encoding representation and combines it with the conversation context to determine the end point, it does not need to rely on the complete transcribed text of the current sentence before determining the end point, thus reducing the impact of transcription errors on the end point determination.

[0031] Since the interruption judgment input includes the current system broadcast text and the broadcast progress, the speech understanding big data model can determine whether the user's speech is related to the current broadcast content, thereby reducing false interruptions caused by echoing sounds, background sounds, or non-target speech.

[0032] Since both tail point control and interruption control are based on the structured control results output by the same large speech understanding model or a large speech understanding model with shared parameters, the cascading error between speech recognition, intent classification, and rule thresholds is reduced.

[0033] Because effective control actions are executed before the complete ASR transcription, the system can respond to control requests more quickly when the user interrupts or pauses. Attached Figure Description

[0034] Figure 1 This is a schematic diagram illustrating the composition and module functions of the full-duplex voice interaction control system provided in an embodiment of the present invention.

[0035] Figure 2 This is a flowchart illustrating the shared processing steps, tail point detection branch, and interruption detection branch provided in the embodiments of the present invention. Detailed Implementation

[0036] The technical solution of the present invention will be further described below with reference to the accompanying drawings. The described embodiments are only for explaining the present invention and are not intended to limit the scope of protection of the present invention. Unless otherwise specified, the technical features in the following embodiments can be combined with each other.

[0037] like Figure 1 As shown, this embodiment describes the system composition according to the speech processing and control sequence. The system includes an echo processing module, a speech activity detection module, a speech segment construction module, a context management pool, a speech understanding module, an interactive control module, and transcription and response processing interfaces. The interactive control module may include an end-point control module and an interruption control module; the transcription and response processing interfaces may include a transcription interface, a response text generation interface, and a speech synthesis interface.

[0038] The voice activity detection module continuously monitors the user's uplink voice stream and outputs user voice start and stop events. The voice activity detection module can employ a neural network voice activity detection model, a detector combining energy thresholding and neural networks, or other detection methods capable of outputting voice start and stop events.

[0039] The role of the voice activity detection module is to provide candidate triggering opportunities, rather than directly determining whether the user has finished expressing themselves or whether the system broadcast needs to be interrupted.

[0040] In full-duplex scenarios, downlink audio broadcast by the system may enter the user's uplink voice stream through speakers, airborne transmission, or device sampling. To mitigate this impact, the system can incorporate an echo processing module to suppress, cancel, or cancel the broadcast reference signal in the uplink audio.

[0041] The echo processing module can receive downlink broadcast audio from the system as a reference signal and suppress echo components related to the reference signal in the user's uplink voice stream. The echo-processed audio can then be sent to the speech activity detection module and the speech understanding module.

[0042] The context management pool is used to maintain text and state information related to the current session, serving as a storage component in the system description. The context management pool can be implemented using a sliding window, queue, key-value cache, digest cache, or vectorized cache.

[0043] The context management pool can include historical user input, historical system responses, current system broadcast text, current system broadcast progress, dialogue task status, slots to be completed, voice segment identifiers to be merged, voice activity detection timestamps, historical end point judgment results, and historical interruption judgment results.

[0044] Once the true endpoint is confirmed, the system will write the user text output by the transcription interface into the context management pool as either historical user input or current round user input.

[0045] After the response text generation interface generates a system reply, the system writes the system reply into the context management pool as a historical system reply, and updates the current system broadcast text and broadcast progress during speech synthesis and broadcasting.

[0046] When the tail point determination result indicates that the true tail point has not been reached, the system does not immediately trigger the generation of response text. Instead, it saves the corresponding speech segment as a speech segment to be merged and records the identifier, start time, end time and waiting status of the speech segment to be merged in the context management pool.

[0047] When a subsequent user's voice segment arrives and is determined to be able to form a complete expression with the voice segment to be merged, the system will concatenate the preceding and following voice segments or use the combined input transcription interface to transcribe the text and write the transcribed text into the session record.

[0048] If the audio segment to be merged does not receive subsequent replenishment within the preset timeout period, the system can process the segment based on the secondary judgment result of the large-scale speech understanding model, business scenario strategies, or default rules. The preset timeout period can be set to approximately 2.5 seconds, or it can be adjusted according to the business scenario.

[0049] In the timeout handling method, if the secondary judgment result indicates that the speech segment can be used as a complete expression, the system can trigger transcription and enter response generation; if the secondary judgment result still indicates that the expression is incomplete, the system can continue to wait, prompt the user to supplement, or discard the segment to be merged.

[0050] Once a valid interruption is confirmed, the interruption control module can pause or terminate the original broadcast content based on the recommended control action. If the recommended control action is pause, the original broadcast content can be retained and resumed when subsequent conditions are met; if the recommended control action is stop, the original broadcast content can be terminated or removed from the broadcast queue.

[0051] In multi-turn dialogues, the context management pool can prune historical content according to the window length, and can also summarize earlier turns of the dialogue. The pruned or summarized context can still be used by the large speech understanding model to determine whether the current speech has reached the true end point or constitutes a valid interruption.

[0052] The large-scale speech understanding model includes a speech coding unit, a context coding unit, a fusion understanding unit, and a control output unit.

[0053] The speech coding unit receives a first or second speech segment and converts it into a speech-coded representation. The speech coding unit can first segment the original audio into frames, and then extract Mel spectrum, filter bank features, energy features, or other acoustic features.

[0054] The speech coding unit can also use the audio coding structure in end-to-end speech recognition models, self-supervised speech models, or multimodal speech models to convert acoustic features into continuous vector sequences or discrete speech token sequences.

[0055] In one implementation, the speech coding representation output by the speech coding unit is mapped to a latent space that can be received by a large language model via an adapter. The adapter may include a linear projection layer, a multilayer perceptron, a convolutional projection layer, an attention pooling layer, or a combination thereof.

[0056] The context encoding unit is used to organize task prompts, session context text, system broadcast status, and business status into a text status representation. Task prompts are used to define whether the current inference task is a tail point judgment task or an interruption judgment task.

[0057] In one fusion approach, the model input can be organized as “task prompt words + conversation context text + system broadcast status + voice token”, where the voice token is generated by the speech coding unit and the adapter.

[0058] In another fusion approach, speech coding representation and text state representation can be encoded separately first, and then jointly modeled through a cross-attention layer or a multimodal encoder.

[0059] The fusion understanding unit can be implemented using a general large language model or other sequence modeling networks. To avoid limiting to specific products or model versions, the implementation of this invention does not depend on a specific large language model; in actual implementation, a language model with appropriate parameter scales can be selected based on the deployed computing power.

[0060] The control output unit is used to generate state control tokens or structured control fields based on the output of the fusion understanding unit. For the end-point determination task, the control output unit can output state control tokens such as endpoint_true, endpoint_false, or endpoint_unknown.

[0061] For interruption judgment tasks, the control output unit can output status control tokens such as interrupt_true, interrupt_false, or interrupt_unknown.

[0062] The confidence level can be represented by the output probability, normalized probability, or calibrated probability of the corresponding state control token. The system can then execute the appropriate control strategy based on the confidence level and the recommended control action.

[0063] Structured control results can be represented using JSON, enumeration labels, state machine inputs, or fixed-field text. For example, structured control results may include endpoint, interrupt, confidence, action, and reason fields.

[0064] The `action` field can take the values ​​`wait`, `transcribe`, `pause`, `stop`, `keep`, or `ask_more`. `wait` indicates that the system will continue waiting for the user to speak further; `transcribe` indicates that transcription will be triggered; `pause` indicates that the playback will be paused; `stop` indicates that the playback will be terminated; `keep` indicates that the current playback will continue; and `ask_more` indicates that the system will prompt the user for additional information.

[0065] like Figure 2 As shown, this embodiment organizes tail point detection and interruption detection into a shared processing step, a scene branching step, and a re-merging output step. This method may include steps S1 to S9.

[0066] Step S1: Obtain the user's uplink voice stream, system broadcast status, and session context information during the full-duplex voice interaction process.

[0067] Step S2 involves performing echo processing on the user's uplink speech stream and detecting user speech start and stop events using the speech activity detection module. The current speech after echo processing retains its acoustic content and prosodic features, and a complete transcribed text of the current sentence is not generated before control judgment.

[0068] Step S3: Cache the current speech based on the speech activity detection results, and use the speech activity event as the trigger condition for end point candidate judgment or interruption candidate judgment.

[0069] Step S4: Determine candidate control scenarios based on system broadcast status and voice activity events. When the system is not in broadcast status or the current task requires determining whether the user has completed their expression, and a user voice stop event is detected, proceed to the end-point detection branch; when the system is in broadcast status and a user voice start event is detected, proceed to the interruption detection branch.

[0070] In step S5a, in the tail point detection branch, after the user's voice stops, wait for about 100 milliseconds and replenish the tail buffer to form the first voice segment for tail point determination.

[0071] In step S5b, in the interruption detection branch, after a user voice start event is detected during system broadcast, at least 1 second of the current voice is extracted to form a second voice segment for interruption judgment.

[0072] Step S6a: Input the first speech segment and conversation context information into the speech understanding model, and output the true end point, the non-true end point, or the low confidence state. For the non-true end point or the low confidence state, the system continues to wait for subsequent speech, merge the preceding and following speech segments, or perform a second judgment.

[0073] Step S6b: Input the second speech segment, system playback status, and conversation context information into the speech understanding model, and output a valid interruption, invalid interruption, or low confidence status. For invalid interruption or low confidence status, the system maintains playback or reduces the playback volume, and updates the speech window and makes another judgment when the user continues to speak.

[0074] Step S7: Combine the structured control results output by the tail point detection branch and the interruption detection branch, and execute control actions such as continue waiting, keep broadcasting, pause broadcasting, terminate broadcasting, or confirm entry into the transcription process based on the structured control results.

[0075] Step S8: For voice segments confirmed to have reached the true end point, trigger the transcription interface; for voice segments confirmed to constitute a valid interruption, first pause or terminate the current playback, then trigger the transcription interface. The user text output by the transcription interface is written to the session record.

[0076] Step S9 involves submitting the transcribed text and updated conversation information to the subsequent response text generation interface for processing, and then generating the system's response speech through the speech synthesis interface. Response text generation and speech synthesis are not considered major improvements of this invention.

[0077] In the tail point detection scenario, the system generates a first speech segment. The first speech segment is formed by the speech buffer between a user speech start event and the corresponding user speech stop event, and may include buffered audio approximately 100 milliseconds after the user speech stop event.

[0078] In one tail-point triggering method, when the voice activity detection module detects a user's voice stop event, the system waits for about 100 milliseconds to absorb detected jitter and short tail sounds; then the first voice segment and conversation context information are input into the large voice understanding model.

[0079] The first speech segment can include the user's speech content, pauses, trailing sounds, intonation, stress, and other prosodic features. The large-scale speech understanding model can then determine whether the user has truly completed their expression based on this speech information and the conversational context.

[0080] In this invention, a true end point refers to an interactive node where the user has completed the current round of expression in terms of semantics, task status, and speech expression, and the system can proceed to transcription, response generation, or subsequent processing.

[0081] True end points differ from simple VAD stops. A VAD stop only indicates that no user voice activity was detected acoustically for a period of time, while true end points require combining current speech, historical context, task status, and information to be completed for speech understanding and judgment.

[0082] When the speech understanding big model outputs endpoint_true and the confidence level reaches the preset threshold, the end point control module confirms that the true end point has been reached and triggers the transcription interface to transcribe the first speech segment or the merged speech segment.

[0083] When the speech understanding big model outputs endpoint_false, the tail point control module confirms that the true tail point has not been reached, saves the first speech segment as a speech segment to be merged, and continues to wait for subsequent speech from the user.

[0084] When the speech understanding model outputs "endpoint_unknown" or its confidence level is below a preset threshold, the end-point control module can perform a secondary judgment, extend the waiting time, or adopt a conservative strategy. The conservative strategy can prioritize waiting for the user's subsequent speech to reduce the risk of the user's pause being misinterpreted as the end of the expression.

[0085] In the tail point detection embodiment, the user says "Book me a ticket for tomorrow morning..." followed by a short pause. The voice activity detection module triggers a user voice stop event. The system generates a first voice segment approximately 100 milliseconds after the stop event and inputs the first voice segment and conversation context information into the large-scale voice understanding model.

[0086] The speech understanding model determines that the user has not yet reached the true end point based on speech prosody, trailing features, and the fields to be completed in the ticket booking task, and the system continues to wait. After the user completes "high-speed rail ticket to Shanghai", the system merges the preceding and following speech segments into a single round of input, and performs transcription and response generation after confirming the true end point.

[0087] In interruption detection scenarios, the system generates a second speech segment. The second speech segment is formed from a short speech window following the user's speech start event during system broadcasting.

[0088] In one interruption triggering method, when the system is broadcasting a voice message and the voice activity detection module detects the user's voice start event, the system extracts at least 1 second of audio after the user starts speaking as a second voice segment, and inputs the second voice segment, the system broadcast status, and the conversation context information into the large voice understanding model.

[0089] If the user continues speaking, the system can capture a new audio window every second, or continuously update the second audio segment using a sliding window. The system can re-trigger the interruption check after each window update.

[0090] Therefore, the interruption determination window of the present invention is not a fixed duration window that judges only once, but uses a short window as the smallest judgment unit and makes continuous judgments while the user continues to speak.

[0091] When the speech understanding big model outputs interrupt_true and the confidence level reaches the preset threshold, the interruption control module first pauses or terminates the current speech playback according to the recommended control action.

[0092] After pausing or terminating the current voice broadcast, the transcription interface transcribes the user's voice segment corresponding to the effective interruption and writes the transcribed text into the session record.

[0093] When the speech understanding model outputs interrupt_false, the interruption control module maintains the current speech playback and continues to listen to the user's speech.

[0094] When the speech understanding big model outputs interrupt_unknown or the confidence level is lower than the preset threshold, the interruption control module can continue broadcasting, reduce the broadcast volume and continue listening, delay the judgment, or trigger a secondary judgment.

[0095] Effective interruptions can include voices that negate the current broadcast content, such as "No, that's not it"; voices that change the task, such as "Stop checking the balance, check the credit card limit"; voices that request the broadcast to stop, such as "Stop for a moment"; voices that add key information, such as "I meant last month"; and voices that ask new questions, such as "How much is the handling fee?"

[0096] Ineffective interruptions can include echolalia such as “um” or “okay”; nonverbal sounds such as coughing, laughing, or sighing; background noise; user talking to themselves that is not directed at the system; and short audio clips that are irrelevant to the current task and have low confidence.

[0097] In the intelligent customer service embodiment, when the system announces "Checking your account balance, please wait," the user suddenly says, "Wait, check my credit card limit." The voice activity detection module triggers the user's voice start event, the system captures a second voice segment, and inputs the second voice segment, the current announcement content, and the conversation context information into the speech understanding big data model.

[0098] After the speech understanding big model outputs interrupt_true and pause or stop actions, the interruption control module pauses or terminates the current broadcast; then, the transcription interface generates user text and writes it into the session record, and the response text generation interface generates a credit card limit query response based on the new user intent.

[0099] In another embodiment, during the system's broadcast, the user makes acknowledgments such as "um" or "okay." The speech understanding big data model, combining the second speech segment, the current broadcast content, and the historical context, determines that it does not constitute a valid interruption, and the interruption control module maintains the current broadcast.

[0100] In another embodiment, background personnel speak during the system's broadcast. The speech understanding big data model can determine that the speech is not directed at the system based on the acoustic saliency of the current speech segment, speaker features, tone features, and conversational context, thus outputting an invalid interruption.

[0101] In an optional embodiment, to adapt the large speech understanding model to end-point detection and interruption detection tasks, training or fine-tuning samples can be constructed. Each sample includes a user speech segment or its acoustic features, conversation context information, system broadcast status, and a label, without using the transcribed text of the current sentence as model input.

[0102] Tags can include true tail point, non-true tail point, valid interruption, invalid interruption, confidence level, and recommended action.

[0103] The training samples can cover scenarios such as users continuing to express themselves after a pause, semantically complete pauses, correction systems during broadcasts, echoing sounds during broadcasts, background noise mixing, multiple people speaking, echo interference, and different speaking speeds.

[0104] The session context information in the training samples can come from real business logs, manually constructed dialogues, or simulated dialogues. To protect privacy, sensitive fields such as user identity, phone number, address, and account information can be anonymized before training.

[0105] For the tail point detection task, positive samples can be semantically complete speech segments that are suitable for transcription and response generation, while negative samples can be speech segments where the user pauses midway through their expression, prolongs a sound, pauses in thinking, waits for supplementary information, or where the task fields are not yet complete.

[0106] For interruption detection tasks, positive samples can be speech segments that correct the system, change the task, request to stop broadcasting, supplement key information, or raise new questions, while negative samples can be echoing sounds, background sounds, non-target speaker speech, or short speech segments without actual control intent.

[0107] During training or fine-tuning, tail point detection and interruption detection can be used as multi-task learning objectives. Loss functions can include classification cross-entropy, structured output loss, multi-task loss, or the generation loss of state-controlled tokens.

[0108] Confidence can be obtained based on the output probability of the target state control token. For example, in a tail-end task, the output probability of endpoint_true or endpoint_false can be used as the confidence of tail-end judgment; in an interrupt task, the output probability of interrupt_true or interrupt_false can be used as the confidence of interrupt judgment.

[0109] To avoid the model learning a single silence duration or a single volume threshold, the training samples can cover different speaking speeds, different dialect accents, different noise environments, and different terminal sampling rates.

[0110] The model can learn the prosody, pauses, intonation, stress, speaker emotion, and the relationship between the current speech and the context and task state, which helps to improve the stability of judgment in complex interactive scenarios.

[0111] In low-confidence scenarios, the system can adopt a conservative control strategy. For example, when the confidence of the end-point detection is low, it can continue to wait for a preset time or perform a secondary judgment; when the confidence of the interruption detection is low, it can maintain the current broadcast or reduce the broadcast volume and then continue listening.

[0112] The system can also set strategy parameters for different business scenarios. For example, in high-risk scenarios such as financial customer service or medical consultation, the effective interruption threshold can be increased and the tail point confirmation waiting time can be extended; in scenarios that emphasize real-time performance, such as in-vehicle assistants or smart speakers, the waiting time can be reduced and priority can be given to ensuring that user interruptions can take effect quickly.

[0113] When the speech understanding model outputs abnormally, the service becomes unavailable, or the confidence level remains insufficient, the system can fall back to a preset rule strategy. The fallback strategy may include fixed-duration silence tail point detection, keyword-triggered interruption, or manual intervention prompts. The fallback strategy is only used to improve system robustness and does not affect the technical solution of this invention to perform tail point detection and interruption detection through the speech understanding model under normal operating conditions.

[0114] The modules of this invention can be deployed on servers, edge devices, vehicle terminals, smart speakers, or customer service systems. Voice activity detection, large-scale voice understanding model inference, and conversation record updates can be performed using streaming processing; user speech transcription is performed after the actual end point or valid interruption is confirmed.

[0115] In the server deployment method, the client can transmit the uplink audio stream to the server, which will then perform echo processing, voice activity detection, large-scale voice understanding inference, and control action distribution.

[0116] In edge or terminal deployment, voice activity detection, echo processing, and some voice coding can be completed on the terminal, while large-scale voice understanding inference can be performed on edge nodes or local models to reduce network latency and improve privacy protection.

[0117] The response text generation module and speech synthesis module can use any suitable large language model, rule engine or speech synthesis service, and their specific implementation does not affect the core control flow of the present invention.

[0118] The above embodiments are merely preferred embodiments of the present invention. Those skilled in the art can substitute or adjust the module names, model types, input fields, output fields, and control thresholds without departing from the spirit and substance of the present invention; all such substitutions or adjustments should fall within the protection scope of the present invention.

Claims

1. A full-duplex voice interaction control method based on a large-scale speech understanding model, characterized in that, include: Acquire user uplink voice stream, system broadcast status, and session context information; Perform voice activity detection on the user's uplink voice stream to obtain user voice start and stop events; In response to the stop event, a first speech segment is formed for tail point detection. The first speech segment and the conversation context information are input into the speech understanding big model to obtain a first structured control result including tail point identification and recommended control actions. Based on this, transcription, retention of speech segment or continued waiting is triggered. When the system is in broadcast mode and the start event is detected, a second speech segment for interruption detection is formed. The second speech segment, the system broadcast state, and the session context information are input into the speech understanding model to obtain a second structured control result including an interruption flag and recommended control actions. Based on this, the broadcast is paused or terminated, triggering the transcription of the user speech segment corresponding to the effective interruption, or the broadcast is continued. The first and second structured control results are generated before the complete transcribed text of the current user's statement is obtained. The input of the speech understanding model does not include the transcribed text of the current user's statement.

2. The method according to claim 1, characterized in that, The session context information includes at least one of the following: historical user statements, historical system replies, current system broadcast text, current system broadcast progress, dialogue task status, business status, current interaction stage, and dialogue round information.

3. The method according to claim 1, characterized in that, The speech understanding big model includes a speech encoding unit, a context encoding unit, a fusion understanding unit, and a control output unit. The speech encoding unit is used to extract the speech encoding representation of the first speech segment or the second speech segment. The context encoding unit is used to extract the context encoding representation of the session context information and the system broadcast status. The fusion understanding unit is used to perform fusion reasoning on the speech encoding representation and the context encoding representation. The control output unit is used to output the end point identifier or the interruption identifier and the corresponding recommended control action.

4. The method according to claim 1, characterized in that, The first voice segment includes the user's voice before the user's voice stop event and the voice data within a preset duration after the user's voice stop event; the preset duration is determined based on the detection delay of the voice activity detection module, the voice sampling rate, or business scenario parameters; when the user's statement has not reached the true end point and subsequent user voice is detected within the preset waiting duration, the subsequent user voice is merged with the already retained first voice segment and the end point is detected again; when no subsequent user voice is detected after the preset waiting duration, transcription is triggered or waiting continues according to the preset timeout strategy.

5. The method according to claim 1, characterized in that, The second voice segment is formed when the system detects the user's voice start event while it is in voice broadcasting state, and its initial detection window is a preset short-term detection window; when the user continues to speak, the second voice segment is updated according to a preset time interval or a sliding window and interruption detection is performed again to distinguish between valid interruptions for the current system broadcast or the current session task and invalid interruptions formed by echoing sounds, background sounds, non-target speaker voices, or voices without control intent.

6. The method according to claim 1, characterized in that, The first structured control result and the second structured control result further include at least one of confidence level, judgment basis or status identifier; when the confidence level is lower than a preset threshold or the status identifier is an uncertain state, a control strategy of continuing to wait or supplementing voice segments is adopted for tail point detection, and a control strategy of maintaining system voice broadcast or continuing to collect user voice is adopted for interruption detection.

7. The method according to claim 1, characterized in that, Also includes: Before voice activity detection, echo cancellation or echo suppression is performed on the collected sound signal; the user's uplink voice stream is buffered in a streaming manner and the first or second voice segment is incrementally updated; After the transcription is triggered, the obtained transcribed text is written to the session record.

8. The method according to claim 7, characterized in that, Also includes: The session record is sent to the response generation model to generate response text, and the response text is sent to the speech synthesis module to generate the system's reply speech.

9. A full-duplex voice interaction control system based on a large-scale speech understanding model, characterized in that, include: The voice activity detection module is used to detect the user's uplink voice stream and output the user's voice start event and stop event; The context management module is used to provide session context information; The speech understanding module, including the speech understanding big model, is used to receive the first speech segment and conversation context information in the end-point detection scenario, receive the second speech segment, conversation context information and system broadcast status in the interruption detection scenario, and output structured control results without receiving the transcribed text of the current user's statement. The tail point control module is used to control the triggering of transcription, retention of speech segments, or continued waiting based on the structured control results; the interruption control module is used to control the pause, termination, or maintenance of system broadcasting based on the structured control results. The transcription module is used to transcribe the first speech segment after confirming that the true end point has been reached, or to transcribe the user speech segment corresponding to the effective interruption after pausing or terminating the system broadcast.

10. The system according to claim 9, characterized in that, The speech understanding model includes a speech encoding unit, a context encoding unit, a fusion understanding unit, and a control output unit. The speech encoding unit is used to convert the current speech segment into a speech encoding representation. The context encoding unit is used to convert the session context information and the system broadcast status into a context encoding representation. The fusion understanding unit is used to determine whether the user's statement has reached the true end point or whether the user's speech constitutes a valid interruption based on the speech encoding representation and the context encoding representation. The control output unit is used to output the corresponding structured control result.

11. The system according to claim 9, characterized in that, It also includes at least one of the following modules: an echo processing module for performing echo cancellation or echo suppression on the acquired sound signal before speech activity detection; and a response generation module for generating response text based on the transcribed text and the session record. A speech synthesis module is used to generate a system response speech based on the response text; wherein, the response generation module and the speech synthesis module are not necessary modules for generating the structured control result, and the structured control result output by the speech understanding module is used to control the full-duplex voice interaction process.

12. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 8.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images