Full-duplex voice interaction method and system

Abstract

The application relates to the technical field of intelligent voice interaction, and discloses a full-duplex voice interaction method based on interruption scoring, action decision and non-announced suffix re-planning. In the process of system announcement of a current voice reply, comprehensive observation quantities containing announced prefixes, non-announced suffixes and announcement progress are obtained in each control cycle, continuous interruption scores are calculated according to the observation quantities, and profit decisions are made among candidate actions such as continuous announcement, suspended announcement and switching reply in combination with residual reply values and time delay costs; when the selected action indicates switching reply, the announced prefixes are reserved, only the non-announced suffixes are locally re-planned, and the candidate suffixes are announced from the position of the adjacent playing pointer after being spliced with the announced prefixes. Therefore, the probability of false interruption can be reduced, the repeated announcement and re-generation time delay can be reduced, and the semantic coherence of the announced prefixes and new suffixes can be maintained.

Description

Technical Field

[0001] This application relates to the field of intelligent voice interaction technology, and in particular to a full-duplex voice interaction real-time control technology based on interruption scoring, action decision-making, and unpronounced suffix replanning. Background Technology

[0002] With the development of voice interaction technology, intelligent voice systems capable of speech recognition, natural language understanding, response generation, and voice broadcasting are widely used in various practical scenarios. For example, in in-vehicle voice assistants, the system often needs to broadcast navigation routes, traffic updates, or contact confirmation information to the driver while driving; in meeting assistants, the system needs to broadcast meeting summaries or minutes to participants; in telephone customer service and intelligent customer service, the system needs to broadcast business descriptions, processing procedures, or identity and order confirmation information to users; in addition, in companion robots and real-time voice agent systems, the system also needs to broadcast longer content during continuous dialogue with users. In these scenarios, users often need to interrupt the system's broadcasting to agree, correct, supplement explanations, or make new requests.

[0003] However, most existing voice interaction systems still operate in a near-half-duplex manner, meaning the system only begins recognizing, understanding, generating a response, and broadcasting after the user has finished speaking, and during the broadcast, it typically discourages or insufficiently supports the user to continue speaking. In the aforementioned scenarios, existing technologies suffer from the following problems: First, when a user interrupts during system broadcasting, the system often simply stops or ignores the interruption, making it difficult to handle the interruption appropriately; second, existing methods struggle to distinguish between brief agreement, correction, or supplementary explanations and genuine interruptions, easily leading to false interruptions; third, once the system begins broadcasting, it typically treats the entire response as an unmodifiable whole, lacking the ability to dynamically reprogram unbroadcast portions, resulting in high regeneration costs, response redundancy, and increased latency; fourth, in full-duplex scenarios, system broadcast audio can easily mix with microphone input, making incremental user speech comprehension difficult. These problems are particularly prominent in scenarios requiring user interruptions, such as navigation broadcasts, meeting summary broadcasts, and business explanation broadcasts. Therefore, there is an urgent need for a new technical solution to continuously listen to user input during system broadcasts and to handle user interruptions in a more natural, low-latency, and robust manner. Summary of the Invention

[0004] The purpose of this application is to provide a full-duplex voice interaction method to solve the problems of existing voice interaction systems, such as difficulty in properly handling user interruptions during broadcasting, misjudging short echoes or environmental noise as valid interruptions, and usually having to rewrite the entire reply when switching responses, resulting in repeated broadcasts and increased response latency.

[0005] This application provides a full-duplex voice interaction method based on interruption scoring, action decision-making, and replanning of unpronounced suffixes. Running on an electronic device, the method executes the following steps in a control cycle during the process of the system pronouncing the current voice response: Acquire comprehensive observations: Receive user-input voice in real time to obtain the user's incremental recognition prefix. And determine the current broadcast progress based on the voice broadcast status. Prefix already broadcast and unbroadcast suffix ,in The position of the played token corresponding to the current playback pointer. The total number of tokens in the current voice reply; Calculate the continuous interruption score: Based on the acoustic activity features of the user input speech, the completeness of the user incremental recognition prefix, and the degree of contextual conflict between the user incremental recognition prefix and the already played prefix, calculate the continuous interruption score at the current moment. ; Multi-action benefit decision: based on the continuous interruption score and the remaining response value of the unannounced suffix. The system calculates the latency cost of each candidate action, the comprehensive benefit of each candidate action in the candidate action set, and selects the current control action from the candidate action set based on the comprehensive benefit. Unreported suffix replanning: When the current control action indication is switched back, the already reported prefix is ​​retained, and based on the semantic state of the user incremental identification prefix, the already reported prefix, and the unreported suffix, local replanning is performed only on the unreported suffix to generate candidate suffixes. ; Output a new voice response: Concatenate the already broadcast prefix with the candidate suffix to obtain the total number of tokens. The system provides a new voice response and controls the system to broadcast the candidate suffix starting from the playback pointer position adjacent to the already broadcast prefix.

[0006] In a preferred embodiment, the continuous interruption scoring Calculate using the following formula: The meanings of each function in the formula are as follows: The discrete time points corresponding to the control cycle; Score for acoustic activities, User voice activity is determined by front-end voice activity detection or energy detection. This is the scaling factor. The threshold for voice activity. For the Sigmoid function; Score the text integrity. For the user incremental identification prefix Stability or integrity; Score for contextual conflict. For the previously broadcast prefix, The number of tokens that have been broadcast. It is a similarity function; For sustained scoring, Deadline User's continuous speaking time For reference duration, To obtain the minimum value; , , , For each score weight, and .

[0007] In a preferred embodiment, the candidate action set is: These represent continuing broadcast, lowering the volume, pausing broadcast, stopping broadcast, and switching to reply, respectively; for any candidate action... Its comprehensive benefits Calculate using the following formula: The meanings of each function in the formula are as follows: In response to incremental user input, the control benefits generated by the broadcast control, For action The responsiveness coefficient to incremental user input; The remaining response value, For the broadcast progress ( , (Total number of tokens in the current voice reply) The percentage of cases not reported. The importance coefficient of the remaining content is determined by the response content structure tag, broadcast type, or content type tag, where the content type tag includes security prompts, identity verification, or business verification. To retain the benefits of the current response on the system side. , The retention factor for residual income; As a cost of time delay, Its average time delay coefficient; The cost of overlapping and conflicting statements The conflict sensitivity coefficient, The user's voice activity level; To re-plan at a cost, For replanning cost coefficient; to These are the weighting coefficients, and .

[0008] In a preferred embodiment, when the selected current control action indicates a switchover response, the switching benefit is first calculated. and the remaining response value The unbroadcast suffix reprogramming is performed only if the following condition is met: The meanings of each function in the formula are as follows: To realize the potential benefits of implementing a switchover and rebalancing plan; The incremental importance of the new user input relative to the original unannounced suffix. The unbroadcast suffix, For the similarity function, The larger the value, the greater the difference between the new user input and the original unannounced suffix; , , These are the weighting coefficients; To switch the revenue threshold.

[0009] In a preferred embodiment, the local reprogramming employs a suffix reprogramming function. Generate the candidate suffixes and impose length constraints on them: The meanings of each function in the formula are as follows: That is, the candidate suffixes; For the current dialogue context; For the user incremental identification prefix; The previously broadcast prefix; The intent or slot semantic state of the unbroadcast suffix is ​​taken from the summary of the original unbroadcast suffix, the dialogue intent slot state, or the intermediate semantic representation of the original response; The total length of the new voice response. The maximum allowed length for reprogramming suffixes.

[0010] In a preferred embodiment, before outputting the new voice response, the candidate suffixes are... Relative to the previously broadcast prefix Calculate the consistency of connections Only New voice replies are only allowed at certain times: The meanings of each function in the formula are as follows: This is a non-redundant component, representing a candidate suffix that does not repeat the information slots that have already been conveyed in the broadcast prefix, obtained by text comparison or duplicate information detection; For factual non-contradictory items, it indicates that the candidate suffix does not negate the security prompts, identity verification or business verification conclusions that are still valid in the broadcast prefix, and is obtained by contradiction detection; For the reference consistency sub-item, it represents that the pronouns or referents in the candidate suffixes can be resolved in a consistent manner with the already broadcast prefix, and is obtained by coreference resolution; The slot consistency item represents that the task slots with already filled broadcast prefixes are consistent in the candidate suffixes (except for the slots for explicit error correction of the user incremental identification prefix), which is obtained by slot comparison; , , , Assigning weights to each item. To ensure consistency, a threshold is passed; specifically, when the candidate suffix negates a valid security prompt, identity verification, or business verification conclusion in the already broadcast prefix, the factual non-contradiction item is separated. Set to a failed state, or change the connection consistency. Determined to be below the aforementioned consistency threshold This prevents switching responses and keeps the previously broadcast prefix unchanged.

[0011] In a preferred embodiment, based on the connection consistency Calculate the semantic bridging cost and by switching net income As a criterion for executing a switchover response: If and only if , and If both conditions are met, execute the switching response and output the new voice response; The meanings of each function in the formula are as follows: For the semantic bridging cost, and Negative correlation; The number of boundary conflicts between candidate suffixes and already broadcast prefixes in terms of time or logical order is obtained by sequential comparison. For candidate suffix length or generation delay risk items; , , These are the weighting coefficients; To switch to net income, , , , , These are the switching revenue, remaining recovery value, switching revenue threshold, maximum length of the replanning suffix, and connection consistency pass threshold, respectively.

[0012] In a preferred embodiment, the multi-action benefit decision further includes action-level hysteresis: The action that yields the greatest overall benefit is designated as the candidate control action. The currently executed control action is recorded as And set an action hysteresis between the two; defined according to the intervention intensity of the current broadcast. And construct the modified return With adaptive switching margin : Let gain :exist The direction of the upgrade, when or conditions for forceful interruption At the time of its establishment, the order Among them, forced insertion is triggered without minimum dwell time; The direction of retreat, except In addition, the number of control cycles that the current action has lasted must also be considered. Only then can updates be performed; otherwise, the previous control cycle will remain unchanged. ; The meanings of each function in the formula are as follows: This refers to the actual actions executed in the previous control cycle. This is an indicator function (it takes a value of 1 if the condition is true, and 0 otherwise). To switch the penalty level, This is the last action transition time. The decay time constant, It is a natural exponential function; This is the margin base value. This is the variance sensitivity coefficient. For the most recent The sliding variance of continuous interruption scoring within a control period. This is the schedule coefficient; , , These are the thresholds for forced interruption, persistence, and stability, respectively. The minimum number of dwell periods in the reversing direction.

[0013] In a preferred embodiment, the unreported suffix replanning further includes tiered fallback: when the candidate suffix generation times out, , or At the same time, the already broadcast prefix is ​​not discarded, and hierarchical fallback is performed in sequence: Level 1, when the user incremental identification prefix is ​​not yet stable, the broadcast is paused and re-evaluated in the next control cycle; Level 2, when the interruption intensity drops and there is no obvious conflict between the original unbroadcast suffix and the current user input, the volume is reduced or the broadcast continues to continue the original unbroadcast suffix and the current switch is abandoned; Level 3, when the interruption intensity remains high and the candidate suffix fails the connection consistency check multiple times, the broadcast is stopped and the user enters the listening state, and the overall planning is carried out after the user's expression is complete.

[0014] This application provides a full-duplex voice interaction system, including at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, cause the at least one processor to perform the full-duplex voice interaction method described above.

[0015] This application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described full-duplex voice interaction method.

[0016] In this application, by using acoustic activity, user incremental identification prefix completeness, contextual conflict degree and persistence together for continuous interruption scoring, the probability of false interruption is reduced; by making benefit decisions among multiple candidate actions, appropriate broadcast control is adopted for different types of interruptions; by retaining the broadcast prefix and only performing local replanning on the unbroadcast suffix, the latency of repeated broadcasting and regeneration is reduced, and the semantic coherence of the switching response is improved while maintaining the continuity of the broadcast content.

[0017] The above technical solution operates in cycles during the process of the system broadcasting the current voice response. The technical effect it achieves is not a simple sum of the effects of various technical means, but rather it gradually emerges with the step-by-step coordination of observation, scoring, decision-making, replanning and verification.

[0018] Because the comprehensive observations incorporate user incremental identification prefixes, already broadcast prefixes, unbroadcast suffixes, and broadcast progress during the acquisition phase, the interruption score calculation is based on rich information such as "whether what the user says conflicts with the content being broadcast" and "whether they are close to finishing speaking," rather than degenerating into simple speech activity detection based solely on the presence or absence of sound. This lays the foundation for the reliability of subsequent judgments. Furthermore, acoustic activity, user incremental identification prefix integrity, contextual conflict level, and interruption persistence are all used in the continuous interruption score, allowing these four components to cross-check each other. Even if short echoes, environmental noise, or single-frame acoustic spikes increase the acoustic activity score, they will be suppressed due to lower text integrity and persistence scores, effectively reducing the probability of misjudging echoes, noise, or short interruptions as valid interruptions.

[0019] By incorporating continuous interruption scoring, the remaining response value derived from the broadcast progress, latency costs, speech overlap conflict costs, and replanning costs into the multi-action benefit comparison, the system can perform fine-grained broadcast control between continuing broadcast, lowering volume, pausing broadcast, stopping broadcast, and switching responses. It takes appropriate actions for different types of interruptions, rather than mechanically stopping broadcasting as soon as a user speaks. Furthermore, when instructing a switch response, the switching benefit is first compared with the remaining response value. Only when the switch is indeed relatively more valuable does it enter the suffix replanning path, thus avoiding unnecessary interruptions and interaction jitter caused by stopping and rewriting responses every time a user speaks.

[0020] By retaining the pre-announced prefixes that the user has already heard during switching, performing local replanning only on the unannounced suffixes, and imposing length constraints on candidate suffixes, the system can generate only suffixes that inherit from the prefixes and respond to the user's new intentions, without repeating the prefixes. Since the pre-announced prefixes have already entered the user's perception, retaining this part of the content and playing candidate suffixes from the adjacent playback pointer position can keep the voice output link continuous in physical playback order, and also limit the replanning range of the current response to the suffixes that have not yet been output. This reduces the repeated playback content and regeneration length while maintaining the continuity of the already played content, thereby reducing the switching response latency. To address the issues of prefix / suffix duplication, contradiction, and referential conflict arising from the "suffix-only planning" approach, a consistency check is implemented using four components: non-redundancy, factual non-contradiction, referential consistency, and slot consistency. This check verifies the self-consistency of candidate suffixes relative to already broadcast prefixes. Switching is prohibited when a candidate suffix negates a valid security message, identity verification, or business confirmation conclusion from the already broadcast prefix, thereby reducing the probability of semantic duplication, contradiction, unclear referentiality, or slot conflicts in prefixes / suffixes. After incorporating semantic bridging costs into the decision on net switching benefits, the switching benefit comparison and semantic consistency check work in tandem. The former filters out cases where switching is "not worthwhile," while the latter filters out cases where switching is "worthwhile but poorly connected." Together, they achieve a better balance between reducing switching latency and maintaining semantic coherence.

[0021] Furthermore, after maximizing benefits, separating candidate control actions from actual execution actions, and employing tiered intervention intensity, time-decreasing switching penalties, and adaptive switching margins based on interruption score fluctuations and broadcast progress, along with ensuring that upgrade directions respond instantly to actual interruptions and fallback directions maintain stability with minimal dwell time, reduces action jitter, volume jitter, and erroneous switching near the benefit threshold. This also facilitates timely responses to explicit error corrections or new requests. When candidate suffix generation times out, exceeds the allowed length, or connection verification fails, tiered fallback processing—pausing and waiting, resuming playback of the original suffix, or stopping and switching to listening—ensures the system maintains control link convergence even when candidate suffixes are temporarily unavailable, avoiding both complete rewrites and repeated oscillations of "switching-replanning-switching." In summary, these methods, interconnected in a causal order, collectively achieve a more stable, lower-latency, and semantically coherent full-duplex voice broadcast control compared to existing technologies.

[0022] The specification of this application contains numerous technical features distributed across various technical solutions. Listing all possible combinations of these technical features (i.e., technical solutions) would make the specification excessively lengthy. To avoid this problem, the various technical features disclosed in the above-described invention, the various technical features disclosed in the following embodiments and examples, and the various technical features disclosed in the accompanying drawings can be freely combined to form various new technical solutions (all of which should be considered as described in this specification), unless such a combination of technical features is technically infeasible. For example, one example discloses feature A+B+C, and another example discloses feature A+B+D+E. Features C and D are equivalent technical means that serve the same function, and technically only one needs to be used; they cannot be used simultaneously. Feature E can technically be combined with feature C. Therefore, the solution A+B+C+D should not be considered as described because it is technically infeasible, while the solution A+B+C+E should be considered as described. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the full-duplex voice interaction method according to the first embodiment of this application; Figure 2 This is a schematic diagram of the structure of a full-duplex voice interaction system according to the second embodiment of this application. Detailed Implementation

[0024] In the following description, many technical details are presented to help the reader better understand this application. However, those skilled in the art will understand that the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0025] Explanation of some concepts Full-duplex voice interaction refers to a voice interaction method in which the system continues to receive user input voice while broadcasting the current voice response, and executes broadcast control according to user input and broadcast status, rather than a near-half-duplex working mode where the system only begins to recognize, understand, generate and broadcast after the user has finished speaking.

[0026] The control cycle refers to the time unit in which an electronic device performs observation, scoring, decision-making, and broadcast control cycle by cycle. It can correspond to an audio frame, a time window composed of several audio frames, the update cycle of streaming speech recognition, the playback control cycle, or the polling cycle of a dialogue controller.

[0027] Comprehensive observations It refers to the observation set formed by incorporating user-side acoustic input features, user incremental identification prefixes, already broadcast prefixes, unbroadcast suffixes, current full-duplex interaction status, and broadcast progress within a certain control cycle of the system broadcasting the current voice response.

[0028] User incremental identification prefix This refers to the phased recognition result of the streaming speech recognition module based on the user's input speech output in the current control cycle.

[0029] Prefix already broadcast This refers to the portion of the response content that the system has output through the voice broadcast link and heard by the user up to the current control cycle. This is the position of the played token corresponding to the current playback pointer.

[0030] Unreported suffix This refers to the portion of the response content that the system has not yet output through the voice broadcast link up to the current control cycle. This represents the total number of tokens in the current voice response.

[0031] Broadcast progress This refers to the ratio of the number of tokens currently broadcast to the total number of tokens in the current voice response. In some implementations, it can be expressed as... This is used to represent the proportion of current responses that have been broadcast. This indicates the percentage of items not broadcast.

[0032] Continuous interruption score , refers to a continuous value calculated based on the acoustic activity characteristics of the user's input speech, the completeness of the user incremental identification prefix, the degree of contextual conflict between the user incremental identification prefix and the already broadcast prefix, and the persistence of the user's interruption, used to measure the probability that the current interruption constitutes an effective interruption. It is different from the binary determination that directly equates the detection of user speech with an effective interruption.

[0033] Remaining reply value This refers to the retention value of the currently unbroadcast suffix for the system to complete the current response content. It can be determined based on the broadcast progress and the importance coefficient of the remaining content. The importance coefficient of the remaining content can be determined by content type tags such as response content structure tags, broadcast type or security prompts, identity verification, and business verification.

[0034] Unreported suffix replanning refers to a process where, when the system needs to switch responses, the already reported prefix is ​​retained, and only the semantic state of the unreported suffix is ​​combined with the current dialogue context, user incremental identification prefix, already reported prefix, and unreported suffix is ​​combined to partially replan the unreported suffix to generate candidate suffixes. The candidate suffixes are then concatenated with the already reported prefixes before being output. This is different from rewriting the entire response.

[0035] Unreported suffix semantic state , refers to the information state used to represent the original unbroadcast suffix intent, slot, summary, or intermediate semantics, which can be taken from the summary of the original unbroadcast suffix, the dialogue intent slot state, or the intermediate semantic representation of the original response.

[0036] Consistency , refers to the measure of whether a candidate suffix is ​​consistent with the already broadcast prefix at the level of continuity. It is obtained by weighting four sub-items: non-redundancy, factual non-contradiction, referential consistency, and slot consistency. It is clearly different from topic similarity. That is, it is not measured by the degree of similarity between the candidate suffix and the original unbroadcast suffix in terms of content, but is used to judge whether the candidate suffix is ​​duplicated, contradictory, unclear in reference, or slot conflicted with the already broadcast prefix.

[0037] Switching Net Income It refers to the net benefit result obtained by considering the switching benefits, residual response value, and semantic bridging costs, used to determine whether to execute a switching response.

[0038] Action grading hysteresis refers to the separation of candidate control actions obtained from maximizing the overall benefit from the actual execution actions after multi-action benefit decision-making. A stabilization control mechanism is then used to suppress action back-and-forth derived from continuous scoring and multi-action benefit maximization in the benefit critical region by grading the intervention intensity of the current broadcast, applying switching penalties and adaptive switching margins.

[0039] Forced interruption refers to a situation where a user's interruption simultaneously exceeds the corresponding thresholds in interruption score, persistence score, and identification prefix stability, i.e., it satisfies... At this point, the system can bypass the action hysteresis threshold and minimum dwell time limit to improve the timeliness of response to explicit error corrections or new requests.

[0040] Tiered rollback refers to a rollback mechanism that handles situations such as candidate suffix generation timeout, length not meeting constraints, failure to pass connection consistency, or insufficient net switching benefit by pausing and waiting, resuming the original unbroadcasted suffix, or stopping and switching to monitoring.

[0041] Self-broadcast suppression refers to a processing method in full-duplex broadcast scenarios that estimates the system's broadcast leakage component based on the current broadcast content, playback volume, vocoder output status, and playback link delay, and then recovers the user's valid voice from the microphone mixed signal to improve the stability of the user incremental recognition prefix.

[0042] Summary of some innovative points In summary, the inventiveness of this application lies not in identifying user voice as an interruptible event in isolation, nor in immediately suspending system broadcasting upon detecting user voice, but in its focus on… Figure 1 The full-duplex broadcast control flow shown integrates user-input voice and user incremental recognition prefixes. Prefix already broadcast Unreported suffix Broadcast progress Current interaction state Incorporating into comprehensive observations (Step 100) limits the judgment of interruption primarily to the technical environment of the voice broadcast link and the already broadcast content. It is precisely because of this... It also carries the broadcast content and broadcast progress, and calculates a continuous interruption score based on this. Acoustic activity score Text integrity score Contextual conflict score With sustained scores Only then can they be mutually verified (step 200), thus preventing short echoes, ambient noise, or single-frame acoustic spikes from being raised. It will also be due to and The signal is suppressed due to its low level, thus differentiating it from the binary processing path that triggers a halt to playback upon detecting voice activity. Based on this, the system does not directly stop playback, but instead... Importance coefficient of remaining content and the proportion of those not broadcast Exported remaining reply value Delay cost Overlapping conflict costs With the cost of replanning Mapped together to the overall benefit of candidate actions (Step 300) A comparable runtime control selection is formed among continue broadcasting, lower volume, pause broadcasting, stop broadcasting, and switch to a response, rather than mechanically stopping broadcasting. Crucially, when the selected action indicates a switch to a response, this application does not use the path regenerated after the entire response is discarded, but instead... The corresponding playback pointer is the boundary, preserving what the user has already heard. Only combined semantic state of unbroadcast suffix Postfix reprogramming function Generate candidate suffixes (Step 600); It is precisely this technical concept of the coexistence of a broadcast prefix and a new suffix that, in turn, gives rise to technical contradictions such as duplication, contradiction, and referential conflict between the prefix and suffix. Therefore, this application uses a connection consistency weighted by four sub-items: non-redundancy, factual non-contradiction, referential consistency, and slot consistency. (Step 700) Verify the consistency of the candidate suffix with respect to the already broadcast prefix, and calculate the semantic bridging cost. Reverse inclusion switching net income The criteria ensure that even situations where a switch is worthwhile but poorly executed are blocked. Therefore, Figure 2 The voice interaction system shown is not a loosely pieced together VAD thresholds or semantic jump rules, but rather a closed-loop control concept that integrates observation, continuous interruption scoring, multi-action benefit decision-making, playback pointer boundary suffix replanning, semantic connection verification, and broadcast status update. The signal results generated in the previous stage become the input or constraint of the next stage. Each stage is organically unified and interlocked according to this causal order. The stable full-duplex broadcast control effect obtained by their collaboration cannot be predicted by simply adding up the above technical features.

[0043] Furthermore, the inventors of this application have discovered through long-term in-depth research that the reason why the above-mentioned problems have been difficult to be effectively solved in full-duplex voice interaction scenarios is that the existing technology has structural limitations in terms of the modeling method, action selection method, and response update method for interruption processing.

[0044] After a thorough analysis of the principles of existing barge-in and dialogue management solutions, the inventors discovered that most traditional solutions rely on rules or thresholds to control speech activity. They immediately pause or stop playback upon detecting user voice activity, fundamentally equating binarized voice activity detection with valid interruption determination. This directly results in the system's inability to distinguish between brief echoes, environmental noise, false single-frame detections, and genuine user corrections or new requests. Consequently, false interruptions frequently occur in scenarios requiring continuous playback, impairing the naturalness of the interaction. The inventors thus realized that whether a barge-in constitutes a valid interruption is not determined by a single acoustic event, but rather by simultaneously considering whether the user is actually speaking, whether the content has formed a stable intent, whether the content conflicts with the currently playing content, and whether the speaking is continuous.

[0045] Furthermore, through observation and research of numerous actual interaction processes, the inventors discovered that even when user interruptions are accurately identified, existing solutions still have limitations in the "how to handle interruptions after detection" stage. Existing methods typically only allow choices between stopping, resuming, or replaying the entire message, lacking a unified approach to compare the merits of "continuing to play the current content" versus "immediately switching to a new response." This stems from the failure to incorporate factors such as playback progress, the importance of remaining content, response latency, and speech overlap / conflict into the same comparative framework. More critically, through repeated analysis, the inventors found that once the system discards the entire response and regenerates it after detecting an interruption, the effective content already heard by the user is wasted, requiring the system to repeat the playback from the beginning. This not only creates response redundancy but also significantly increases latency. This is because existing solutions treat the entire response as an indivisible whole, failing to recognize that in most cases of interruptions, the only part that truly needs modification is the unplayed subsequent portion. Furthermore, during full-duplex broadcasting, the system's own broadcasting sound may leak into the microphone input, making the user incremental recognition results unstable and further affecting the reliability of interruption judgment and action selection.

[0046] Based on the above in-depth research, the inventors of this application propose an innovative technical solution: During the system's playback of the current voice response, user input is continuously received and real-time observations are constructed. Based on this, a continuous interruption score is calculated. Then, considering the remaining response value, latency cost, and conflict cost, a benefit decision is made among multiple actions such as continuing playback, lowering the volume, pausing, stopping, and switching responses. When a switch is necessary, the already played prefix is ​​retained, and only the unplayed suffix is ​​partially reprogrammed. The implementation process of this application is described in detail below through specific embodiments.

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0048] The technical solution of this application will be further described below with reference to specific embodiments. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this application. Without departing from the technical concept of this application, those skilled in the art can make adaptive adjustments to the implementation details according to the actual deployment method of the speech recognition module, speech broadcast module, dialogue management module, and electronic device. The technical names used in this embodiment are consistent with the terminology in the foregoing technical solutions, and the same technical features use the same technical names.

[0049] This embodiment provides a full-duplex voice interaction method based on interruption scoring, action decision-making, and replanning of unplayed suffixes. This method runs on an electronic device, which can be a smart voice assistant device, an in-vehicle voice terminal, a conference assistant terminal, a telephone customer service server, a companion robot, or a terminal device, edge computing device, or server device in a real-time voice agent system. The electronic device has at least or can invoke functional modules such as microphone input, streaming speech recognition (hereinafter referred to as streaming ASR), dialogue management, response generation, speech synthesis, and text-to-speech (hereinafter referred to as TTS) control. It should be noted that the focus of this embodiment is not on changing the training structure of the speech recognition model or response generation model, but rather on enabling the system to make relatively stable technical decisions between actions such as continuing playback, lowering volume, pausing playback, stopping playback, and switching responses through full-duplex control cycle-by-cycle control during the system's playback of the current voice response. When switching responses is required, the already played prefix is ​​retained, and only the unplayed suffix is ​​locally replanned. Therefore, this method operates during both the inference and control phases, making it easy to deploy in existing voice interaction systems.

[0050] In this embodiment, the system's current voice response can be represented as a sequence of tokens or a sequence of text segments corresponding to the token sequence. For ease of description, let's assume the system's current voice response is... It includes tokens from the first token to the second token. The response content for each token. During playback, the system can determine the current playback position by using the TTS playback buffer, playback pointer, text-to-audio alignment information, or token-to-audio frame alignment information. Each token. Therefore, the system's current voice response can be divided into a pre-broadcast prefix and a post-broadcast suffix: In formula 1, This indicates the system's current voice response; Indicates up to the discrete time. The "already broadcast" prefix has already been announced; Indicates up to the discrete time. Unannounced suffixes; This indicates the position of the played token corresponding to the current playback pointer; This indicates the total number of tokens in the current voice response; This indicates a sequence concatenation operation. It should be noted that... It is not required that the token-level TTS system directly provide the information. In some implementations, it can also be obtained from the audio playback timestamp and the text-to-audio alignment relationship, thus giving the distinction between the played prefix and the unplayed suffix a feasible physical source.

[0051] During system broadcasts, the electronic devices execute the following steps according to the control cycle. For example... Figure 1 As shown, the full-duplex voice interaction method of this application embodiment may include steps 100 to 900. The control cycle may correspond to an audio frame, a time window composed of several audio frames, a streaming ASR update cycle, a playback control cycle, or a polling cycle of a dialogue controller. This application does not limit its fixed duration, and it can be specifically calibrated according to the system's real-time performance and computing resources.

[0052] Step 100: Obtain comprehensive observations Specifically, during the system's broadcast of the current voice response, the electronic device continuously receives user input voice, obtains the user-side acoustic features from the microphone input, and sends the user input voice to the streaming ASR module to obtain the user incremental recognition prefix. Let the discrete time be... User-side acoustic input characteristics are User incremental identification prefix is Therefore, the comprehensive observations during this control period can be expressed as: (Formula 2) In formula 2, Representing discrete time Comprehensive observations; Indicates up to the discrete time. The user-side acoustic input feature sequence can be derived from the acoustic features, energy features, speech activity detection results, or speech confidence of the microphone input after front-end processing. This represents the user incremental identification prefix output by the streaming ASR at the current time. This indicates that the prefix has been broadcast; This indicates that the suffix was not broadcast. This indicates the current full-duplex interaction state, such as listening state, broadcasting state, broadcasting and listening state, paused state, or toggle state; This indicates the current broadcast progress. The broadcast progress can be determined using the following formula: In formula 3, This indicates the current progress of the voice response playback; This indicates the number of tokens that have been broadcast so far. This indicates the total number of tokens in the current voice response. Therefore, This can be used to represent the proportion of unplayed content. It is important to emphasize that this step incorporates user input, the prefix of what has been played, the suffix of what has not been played, the playback status, and the playback progress into a comprehensive observation. This is the basis for the subsequent interruption scoring not degenerating into simple voice activity detection (VAD). Because the observation carries both the content that has been played and the playback progress, the system can determine in subsequent steps whether "what the user said conflicts with the content being played" and "whether the user is close to finishing speaking," rather than simply reacting to "whether there is sound."

[0053] Furthermore, in full-duplex broadcast scenarios, the system's own broadcast audio may leak to the microphone input. To reduce the interference of the system's self-broadcast audio on the user's incremental identification prefix, in some implementations, the electronic device can perform self-broadcast suppression in or before step 100. Let the mixed signal received by the microphone be... ,but: In formula 4, Indicates the microphone at time Received mixed signal; Represents the actual voice components of the user; This indicates that the system is broadcasting a leak or echo component; This represents ambient noise. Electronic devices can estimate the system's self-broadcast components based on the current broadcast content, playback volume, vocoder output status, and playback link latency. In formula 5, This represents the estimated system broadcast leakage component; This represents the function for estimating the broadcast leakage. This indicates that the prefix has been broadcast; Indicates the current broadcast volume or vocoder output status; This indicates playback link delay or alignment parameters. Furthermore, valid user speech can be recovered based on estimated system-reported leakage components. In formula 6, This represents the effective speech estimate of the user after recovery; This represents the time-varying suppression coefficient. The time-varying suppression coefficient can be determined by the following formula: In formula 7, This represents the signal amplitude norm or energy norm; To prevent division by zero of small constants; This represents the minimum value function. Through self-broadcast suppression, user incremental identification prefixes are used. The stability can be improved, which helps to make subsequent continuous interruption scoring and action benefit decisions more stable. It should be noted that self-broadcast suppression is an optional enhancement path in this implementation method. Its role is to improve the quality of full-duplex input, rather than to replace subsequent interruption scoring, action benefit decisions and unbroadcast suffix replanning.

[0054] Step 200: Calculate the continuous interruption score. Specifically, the electronic device does not directly equate detecting a user's voice with an effective interruption. Instead, it calculates the continuous interruption score at the current moment based on the acoustic activity characteristics of the user's input speech, the completeness of the user's incremental recognition prefix, the degree of contextual conflict between the user's incremental recognition prefix and the previously played prefix, and the duration of the user's interruption. In one alternative implementation, the continuous interruption score is calculated using the following formula: In formula 8, This indicates the score for consecutive interruptions at the current moment; Indicates the score for acoustic activities; Indicates the text integrity score; Indicates the score for contextual conflict; Indicates a sustained score; , , , Let each represent a weighted score, and let them satisfy the following conditions: The aforementioned weights can be calibrated based on the noise level, streaming ASR stability, and broadcast type for different application scenarios, or they can be adaptively updated from the operation logs.

[0055] More specifically, the acoustic activity score can be calculated based on the user's voice activity: In formula 9, This indicates the user's voice activity level obtained from the front-end voice activity detection, energy detection, or voice confidence module. Indicates the scaling factor; Indicates the threshold for speech activity; This represents the Sigmoid function. This score is used to characterize whether there is valid speech activity in the microphone input that may originate from the user.

[0056] Text integrity scores can be calculated by assessing the stability or completeness of user-incremented prefix identification: In formula 10, Indicates user incremental identification prefix The stability or integrity of. For example, The text integrity score can be determined based on the number of backtracking attempts, prefix length, keyword integrity, semantic closure, or intent slot integrity for the same prefix in adjacent control cycles of streaming ASR. For example, short, abrupt responses such as "um" or "okay" may have a lower text integrity score; while inputs that clearly form a correction or new request, such as "not this contact," "change to home," or "transfer directly to human," may have a higher text integrity score.

[0057] Contextual conflict score can be determined based on the similarity between the user incrementally identified prefix and the already broadcast prefix: In formula 11, This represents the similarity function. The similarity function can be implemented using text edit distance normalized similarity, keyword overlap, slot consistency comparison results, semantic vector cosine similarity, or a combination of the above methods, thereby avoiding the black-boxing of key relationships. The lower the similarity, the more likely the user's current input is to have an intention to correct, replace, or add content to the already broadcasted content, resulting in a higher contextual conflict score.

[0058] The duration score can be calculated based on the user's continuous speaking time: In formula 12, This indicates the duration of the user's continuous voice communication up to the current moment; This indicates the reference duration required to determine a stable interruption; This represents the minimum value function. This score is used to reduce the probability of false handover responses caused by short bursts of noise, short bursts of noise, or false detections in a single frame.

[0059] As can be seen above, the four sub-items work together and verify each other: the acoustic activity score answers "whether a user actually made a sound," the text integrity score answers "whether what the user said forms a stable intention," the contextual conflict score answers "whether the content conflicts with the current broadcast," and the persistence score answers "whether the sound is persistent." It is precisely because of the combined effect of these four factors that even short echoes, environmental noise, and single-frame acoustic spikes can be effectively mitigated. Elevation can also be due to , The signal is suppressed due to its low level, thus distinguishing it from the binary scheme that treats "detecting a sound as an interruption." When Greater than the interruption trigger threshold At this point, the system enters the action benefit evaluation phase, rather than directly stopping the broadcast.

[0060] Step 300: Calculate the overall benefit of each candidate action. Specifically, the set of preset candidate actions for electronic devices This can include continuing broadcasting. Lower the volume Broadcast suspended Stop broadcasting and switch reply Among them, "Continue Playback" means the system will maintain the normal playback of the current voice reply; "Lower Volume" means the system will temporarily lower the playback volume without interrupting the current content to reduce speech overlap interference; "Pause Playback" means the system will temporarily stop playback and wait for user input to further stabilize; "Stop Playback" means the system will stop the current playback but will not immediately generate a new suffix; "Switch Response" means the system will stop the current playback and trigger the replanning of unplayed suffixes.

[0061] For any candidate action The electronic device can calculate the overall benefit of a candidate action based on continuous interruption scores, the remaining response value of unannounced suffixes, the latency cost of the candidate action, the speech overlap conflict cost, and the replanning cost. In formula 13, Indicates candidate actions At any moment The overall benefits; This represents the control benefits generated by responding to incremental user input for broadcast control. Specifically, it can be understood as the control benefits of reducing invalid broadcasts, re-identification, regeneration, and rebroadcast when responding to new user intentions in a timely manner. This indicates that the system has retained the benefits of completing the current response. This represents the latency cost introduced by the candidate action; This indicates the cost of conflict caused by overlap between system announcements and user speech; This indicates the cost of replanning when a candidate action triggers a replanning or rewriting process. to This represents the weight coefficients of each item, and it satisfies... The above items can be calculated on the same normalized scale to compare the overall benefits of different actions. It should be noted that... It represents the latency and resource costs required to respond to new user intentions, rather than user subjective satisfaction, thus making the revenue item fall on the technical aspects rather than the operational aspects.

[0062] In one implementation, the control benefit term in response to incremental user input can be determined by the following formula: In formula 14, Indicates candidate actions The responsiveness coefficient to incremental user input. Generally, switching to reply and pausing broadcasting have a higher responsiveness to incremental user input, while continuing broadcasting has a lower responsiveness. However, the specific coefficient can be calibrated according to the application scenario.

[0063] The remaining value of responses with unannounced suffixes can be determined using the following formula: In formula 15, This indicates the remaining value of replies for the currently unbroadcast suffix; This indicates the importance coefficient of the remaining content, which can be determined by the response content structure tag, broadcast type, or content type tag. For example, content types such as security prompts, identity verification, and business confirmation can have a higher importance coefficient. This indicates the percentage of content that has not yet been broadcast. This item is used to prevent the system from easily switching when the broadcast is nearing completion or when the remaining content has a high technical necessity.

[0064] Based on this, the retention benefit of the current reply content completed by the system can be determined by the following formula: (Formula 16) In formula 16, and This represents the residual income retention factor, and its value can be located in... These values ​​represent the degree to which lowering the volume and pausing the broadcast retain the remaining value of the original response. Continuing the broadcast usually retains the highest remaining value of the current response; stopping the broadcast or switching responses no longer directly retains the system-side benefits of the original unbroadcast suffix.

[0065] The latency cost, conflict cost, and replanning cost of the candidate action can be expressed as follows: In formula 17, Indicates candidate actions The corresponding average latency coefficient is generally the largest for handover response, because it may involve new understanding, new generation, and new broadcasting; in Formula 18, Indicates candidate actions Conflict sensitivity coefficient This indicates the user's voice activity level. The sensitivity coefficient to actions that conflict with continuing broadcasting is typically high, while the sensitivity coefficient to actions that require stopping or switching is typically low; in Formula 19, This represents the replanning cost coefficient. For actions like continuing to broadcast and lowering the volume, the replanning cost can be small or zero; for actions like switching to reply, the replanning cost is usually related to the length of the unbroadcast suffix or the proportion of unbroadcast actions.

[0066] Step 400: Select candidate control actions and perform action grading and hysteresis. Specifically, the electronic device first determines the candidate control action based on the overall benefit: (Formula 20) In formula 20, This represents the candidate control action obtained by maximizing the overall benefit within the current control period; Indicates comprehensive income The largest action selection operation.

[0067] It should be noted that continuous interruption scores, streaming ASR results, and acoustic activity scores may exhibit short-term fluctuations between adjacent control cycles. If each control cycle is executed directly... In the critical zone of revenue generation, actions may frequently change between continuing broadcasting, lowering the volume, pausing broadcasting, stopping broadcasting, and switching back, causing volume jitter, pause jitter, or erroneous switching. This needs to be distinguished from ordinary user interface debouncing: the jitter to be suppressed in this implementation is the repetitive action in the critical zone derived from the "continuous scoring + multi-action revenue maximization" architecture itself, rather than repeated triggering at the interactive control level. Therefore, in a preferred embodiment, the electronic device will select candidate control actions... With actual actions Separate the two and set action hysteresis between them to create a buffer between revenue decisions and playback control.

[0068] Specifically, the action intensity function can be defined according to the intervention intensity of the current broadcast. , so that: In formula 21, Indicates candidate actions The intensity of intervention in the current broadcast link. If If so, it means that the current candidate control action belongs to the upgrade direction relative to the actual action executed in the previous control cycle; if If it is, it indicates that it belongs to the direction of regression.

[0069] In some implementations, the electronic device incorporates a switching penalty that decays over time, resulting in a correction benefit: In formula 22, Indicates candidate actions Correction gains; This indicates a change in the penalty level; This indicates an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. This indicates the actual action performed in the previous control cycle; Indicates the time of the last action switch; Indicates the decay time constant; This represents the natural exponential function. This formula reduces motion jitter by overcoming the additional switching cost when switching actions continuously within a short period of time.

[0070] Furthermore, electronic devices can determine adaptive switching margins based on the sliding variance of continuous interruption scoring and broadcast progress: In formula 23, This represents the adaptive switching margin for the current control cycle; Indicates the margin base value; This represents the variance sensitivity coefficient; Indicates recent Continuous interruption scoring within each control cycle The sliding variance; This represents the progress coefficient. Therefore, when the interruption score fluctuates significantly or the current broadcast is nearing completion, the system raises the switching threshold, thereby reducing the probability of erroneous switching in the critical section or frequent replanning in the tail segment.

[0071] Electronic devices can calculate the gain of a candidate control action relative to the previous actually executed action: In formula 24, This represents the corrected benefit gain of the candidate control action relative to the previous actually executed action. If the candidate control action is the same as the previous actually executed action, the electronic device maintains the current action. If it belongs to the upgrade direction, then when At that time, the electronic device can update the actual action performed. Furthermore, when user input meets the forced interruption conditions, electronic devices can also bypass the hysteresis threshold and minimum dwell time limit to reduce response delays to explicit error corrections or new requests. The forced interruption conditions can be expressed as: In formula 25, This indicates the threshold for interrupting a conversation; Indicates the persistence threshold; Indicates the stability threshold; This represents a logical AND operation. If it belongs to the backoff direction, then unless the condition is met... In addition, it can also require the number of control cycles that the current action has lasted. Not less than the minimum number of stay periods This is to avoid the oscillation of immediately resuming after pausing or canceling after switching. Therefore, the upgrade direction responds instantly to actual call interruptions, while the rollback direction prioritizes stability with minimal dwell time; the combination of these two approaches makes the control both sensitive and oscillating.

[0072] Step 500: Determine if the unbroadcast suffix replanning has been triggered. Specifically, when the actual action is performed When instructing a switchover response, the electronic device does not immediately discard the entire current voice response. Instead, it first calculates the switchover benefit and compares it with the value of the remaining responses. It should be noted that this step is a preliminary filtering process before entering the un-played suffix replanning stage. The final decision on whether to execute the switchover response is based on the net switchover benefit criterion in step 700. The switchover benefit can be expressed as: In formula 26, This indicates the potential benefits of implementing a switchover and replanning strategy; , , Indicates the weighting coefficient; This represents the incremental importance of the new user input relative to the previously unannounced suffix. The incremental importance can be calculated using the following formula: In formula 27, This indicates the original, unannounced suffix. If the user's new input differs significantly from the original, unannounced suffix, then... A higher similarity indicates a greater need to re-plan the unreported suffixes. This similarity function can be implemented in the same way or differently as the similarity function in step 200, but its input-output relationship should remain clear, for example, it can be determined by slot matching, keyword overlap, or semantic vector cosine similarity.

[0073] In one implementation, subsequent unbroadcast suffix replanning is triggered only when the switching benefit is sufficiently advantageous relative to the remaining response value. In formula 28, This indicates the threshold for switching revenue. This criterion ensures that the system only enters the suffix replanning path when the user's input genuinely creates a significant need for replacement, correction, or addition of the currently unplayed suffix, rather than stopping and rewriting the response every time a user speaks, thus avoiding unnecessary interruptions and jitter.

[0074] Step 600: Generate candidate suffixes Specifically, when the handover benefit criterion is met, the electronic device retains the previously broadcast prefix. Based on the current dialogue context, the semantic state of the user-incremented prefix, the already played prefix, and the unplayed suffix, local replanning is performed only on the unplayed suffix. Candidate suffixes can be generated using the suffix replanning function: In formula 29, This indicates the candidate suffix obtained from reprogramming. This indicates a runtime postfix replanning function; Indicates the current dialogue context; This indicates the intent or slot semantic state that the suffix was not broadcast; This indicates the total length of the new voice response. The semantics can be derived from the summary of the original unplayed suffix, the intent slot state maintained by the dialogue manager, the intermediate semantic representation of the original complete response, or the semantic state extracted from the unplayed suffix through compression. This differs from traditional solutions: the closest existing technology, after an interruption, either resumes the original response or completely replaces it with a new response, in which case the already played prefix and the new response do not coexist; however, this implementation allows the prefix already heard by the user to coexist and be concatenated with the newly planned suffix. Therefore, only a suffix that inherits the prefix and responds to the user's new input needs to be generated, without repeating the prefix, thereby reducing the latency of repeated playback and regeneration.

[0075] In one feasible path, the suffix reprogramming function This can be implemented at runtime using rule templates, dialogue managers, lightweight generation modules, or trained language models. Electronic devices can access the current dialogue context. User incremental identification prefix Prefix already broadcast and the unreported suffix semantic state The organization takes the replanning input as input and provides constraints to the suffix replanning function: it must not repeat information slots that have already been conveyed in the already played prefix, it must not deny valid slots or confirmation conclusions in the already played prefix except for slots that the user incrementally identifies and explicitly corrects, and the output content is only used as a new suffix adjacent to the playback pointer.

[0076] To reduce generation delays and broadcast instability caused by excessively long candidate suffixes, electronic devices can impose length constraints on candidate suffixes: In formula 30, Indicates the length of the candidate suffix for replanning; This indicates the maximum allowed length of the reprogramming suffix. This length constraint can be calibrated based on the control cycle, TTS buffer capacity, target response latency, or broadcast scenario.

[0077] Step 700: Perform semantic coherence verification on candidate suffixes. Specifically, because this application adopts a technical approach of retaining the already broadcast prefix and only replanning the unbroadcast suffix, the candidate suffix and the already broadcast prefix will jointly constitute a new voice response at the user's perception level. Therefore, before outputting a new voice response, the electronic device can perform a connection consistency check on the candidate suffix relative to the already broadcast prefix. It should be noted that this connection consistency is clearly different from topic similarity: since the candidate suffix is ​​often used to carry over the user's new intention, its content may differ from the original unbroadcast suffix, so it is not measured by the degree of content similarity; this connection consistency is mainly used to determine whether the candidate suffix is ​​repetitive, contradictory, unclear in reference, or conflicting in slot with the already broadcast prefix, that is, to measure whether the connection is self-consistent. This distinction is precisely to address the problems of prefix and suffix repetition, contradiction, and referential conflict derived from the "only replanning suffix" architecture.

[0078] In one implementation, consistency of connection It can be calculated using the following formula: In formula 31, This indicates the consistency of the candidate suffix with respect to the already broadcast prefix; This indicates a non-redundant item, used to characterize whether the candidate suffix avoids repeating the information slots that have already been conveyed in the broadcast prefix. It can be obtained by text comparison or duplicate information detection. This indicates a factual non-contradictory item, used to characterize whether the candidate suffix does not negate the security prompts, identity verification, or business verification conclusions that are still valid in the broadcast prefix. It can be obtained by contradiction detection or rule comparison. This represents the coreference consistency item, used to characterize whether the pronouns or referents in the candidate suffixes can be resolved in consistency with the already broadcast prefixes, and can be obtained from the coreference resolution module; The slot consistency item is used to characterize whether the filled task slots in the broadcast prefix are consistent in the candidate suffixes, except for the slots for user incremental identification prefix explicit error correction; , , , Let each item represent its weight, and let them satisfy the following conditions: In Formula 32, each weight is used to normalize the combination of the four coherence consistency items. In some implementations, for scenarios such as financial customer service, in-vehicle safety announcements, and identity verification, the factual non-contradiction item can be appropriately increased. The weight of the non-redundant component can be appropriately increased for scenarios involving conference summaries or long text presentations. The weights mentioned above can be determined based on operation logs, manually labeled samples, or system test results.

[0079] When a candidate suffix negates a security prompt, identity verification, or business confirmation conclusion that is still valid in the broadcast prefix, the electronic device can present the factual non-contradictory items. Set to a failed state, or directly disable connection consistency. Determined to be below the consistency threshold This prevents switching responses and keeps the already broadcast prefix unchanged. For example, if the system has already broadcast "Your delivery address has been locked as your company address," but the candidate suffix is ​​generated as "Please confirm your delivery address first," then there is a conflict between the candidate suffix and the already broadcast prefix in terms of business confirmation. The system should not directly output the candidate suffix, but should instead proceed with a rollback or re-evaluation process.

[0080] Furthermore, electronic devices can calculate semantic bridging costs based on connectivity consistency: In Equation 33, Indicates the cost of semantic bridging; This indicates the amount of boundary conflict between candidate suffixes and already broadcast prefixes in terms of time or logical order. It can be obtained by sequence comparison or rule detection. It only carries the sequence boundary conflict; semantic layer conflicts have already been resolved. The four components bear the load, and the two components do not overlap; Indicates the candidate suffix length or generation delay risk item; , , This represents the weighting coefficients and can satisfy... The higher the semantic bridging cost, the higher the risk of connection between the candidate suffix and the already broadcast prefix, even though the candidate suffix may respond to incremental user input.

[0081] Based on this, electronic devices can calculate the net benefit of switching: In Equation 34, This represents the net benefit of the handover. By incorporating the semantic bridging cost into the net benefit of the handover, this implementation avoids forcibly outputting candidate suffixes that respond to user input but have poor coherence with the already broadcast prefix. In one implementation, the electronic device performs a handover response and outputs a new voice response only when the following conditions are met simultaneously: In Equation 35, This indicates that the consistency of the transition has passed the threshold. Therefore, the system does not arbitrarily replace the original suffix after generating candidate suffixes, but only executes a handover response if the net benefit of the handover, length constraint, and semantic coherence constraint are all satisfied. It is evident that the handover benefit comparison (step 500) and semantic coherence verification (step 700) work together: the former filters out cases where "handover is not worthwhile," and the latter filters out cases where "handover is worthwhile but the transition is poor." Together, they ensure a balance between low latency and semantic coherence.

[0082] Step 800: Output a new voice response or perform a tiered rollback. Specifically, when the criterion in step 700 is met, the electronic device will concatenate the already broadcast prefix with the candidate suffix to obtain a new voice response: In Equation 36, This indicates the new voice response after reprogramming. Since the previously announced prefix has already been heard by the user, the electronic device does not need to replay it during actual broadcast control. Instead, it controls the TTS broadcast buffer or playback pointer from the position adjacent to the already broadcast prefix (i.e., the first...). Starting from the position corresponding to each token, candidate suffixes are announced. In other words, the electronic device can clear or replace the playback buffer corresponding to previously unannounced suffixes, send the text or audio segment corresponding to the candidate suffixes into the TTS or playback buffer, and start the playback pointer from... The corresponding location will resume broadcasting.

[0083] When candidate suffix generation times out, candidate suffix length does not meet formula 30, or the coherence consistency is lower than 10, the following conditions apply: Or switch to net income Not greater than In this scenario, the electronic device does not discard the already broadcast prefix and performs a tiered fallback. In the first-level fallback, if the user incremental identification prefix is ​​not yet stable, or if a more reliable candidate suffix may be generated in the next control cycle, the electronic device pauses broadcasting and re-evaluates after reacquiring the user incremental identification prefix in the next control cycle. In the second-level fallback, if the interruption intensity decreases and there is no significant conflict between the previously unbroadcast suffix and the current user input, the electronic device lowers the volume or continues broadcasting to continue broadcasting the previously unbroadcast suffix and abandons the current handover. In the third-level fallback, if the interruption intensity remains high, the candidate suffix fails the connection consistency check multiple times, and there is a significant conflict between the previously unbroadcast suffix and the current user input, the electronic device stops broadcasting and enters a listening state, waiting for the user to complete their statement before proceeding with overall planning. Through this tiered fallback method, and by incorporating the fallback decision into the hysteresis, the system can maintain control link convergence when candidate suffixes are temporarily unavailable, avoiding repeated oscillations of "handover-replanning-re-handover."

[0084] Step 900: Update the full-duplex status and continue running. Specifically, the electronic device updates its system state based on the current actual action being performed, continuous interruption scoring, and broadcast progress. The state transition can be represented as: In Equation 37, Indicates the system state for the next control cycle; Represents the state transition function; This indicates the current system state. For example, when the system is in a broadcast and listen state and the actual action is to continue broadcasting, the next state can remain in the broadcast and listen state; when the actual action is to pause broadcasting, the next state can transition to the paused state; when the actual action is to switch to a response and the candidate suffix has passed verification and entered the playback buffer, the next state can transition to either the switch state or the broadcast and listen state; when the actual action is to stop broadcasting and wait for the user to complete their statement, the next state can transition to the listen state. Thus, the system forms a closed-loop control between broadcasting, listening, pausing, and switching.

[0085] Furthermore, to illustrate how the above method operates under different types of interruptions, several exemplary scenarios are given below. These examples are used to explain the control logic and are not intended to limit specific voice content, thresholds, or parameters.

[0086] In in-vehicle voice assistant scenarios, the system is broadcasting navigation or contact confirmation information. If the user only responds with short, curt "um" or "okay," the acoustic activity score may increase briefly, but the text integrity and persistence scores will be low, and the continuous interruption score will be unstable. The sliding variance and switching penalty in the action grading hysteresis will increase the switching threshold. The system tends to continue broadcasting or briefly lower the volume without triggering a switching response, thus avoiding the jitter of a "volume down-and-restore" cycle. If the user says "not this contact" or "go to the airport," the user's incremental recognition prefix usually has high stability and a high contextual conflict with the already broadcast prefix or the previously unbroadcast suffix. The continuous interruption score and incremental importance increase, and the system may choose to switch responses. While retaining the already broadcast safety prompts or confirmation prefixes, it will only re-plan the unbroadcast suffix.

[0087] In a meeting assistant scenario, the system is broadcasting a meeting summary. If a participant says "Yes" or "Um," the system avoids accidental switching by continuously interrupting the scoring and using action lag. If a participant says "It's not the finance department, it's the legal department," the user's incremental prefix recognition explicitly corrects a slot. The system allows the corrected slot to change when generating candidate suffixes, while requiring other already broadcast and valid slots to remain consistent. If the candidate suffix overlaps or contradicts the already broadcast prefix, the consistency check will be weakened. This prevents the direct output of the candidate suffix and initiates a fallback process.

[0088] In an intelligent customer service scenario, the system is broadcasting a business description. If a user says, "I understand, please transfer me to a human agent," the system can trigger a switch response by using a high score for continuous interruptions, a high incremental importance score, and a low retention value for the original unbroadcast suffix, generating candidate suffixes such as "Transferring you to a human agent." If a candidate suffix needs to negate the already broadcasted identity verification or business confirmation conclusion, the factual non-contradiction item will be reduced or set to a "disapproved" state, and the system will not directly concatenate the output, thereby reducing the semantic contradiction rate between prefixes and suffixes.

[0089] System embodiments, electronic device embodiments, and storage medium embodiments This embodiment also provides a full-duplex voice interaction system corresponding to the above method. For example... Figure 2 As shown, the system may include at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor. When the instructions are executed, the at least one processor performs the full-duplex voice interaction method described in any of the foregoing embodiments. In other words, the system may be configured to: acquire comprehensive observations cycle by cycle during the process of the system playing the current voice response, calculate a continuous interruption score, calculate the comprehensive benefit of each candidate action in the candidate action set, select the current control action, retain the already played prefix and perform local replanning only on the unplayed suffix when the current control action indicates a switch response, generate candidate suffixes, and control the system to play candidate suffixes starting from the playback pointer position adjacent to the already played prefix.

[0090] This embodiment also provides an electronic device. The electronic device may include a processor, a memory, an audio input interface, an audio output interface, and a software module for performing speech recognition, dialogue management, response generation, and voice broadcast control. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned full-duplex voice interaction method. The electronic device can execute all steps locally, or it can collaboratively execute some steps with a cloud server, edge node, or external speech recognition service; as long as it can perform continuous interruption scoring, multi-action benefit decision-making, and un-broadcast suffix replanning during the system's broadcast of the current voice response, it is an optional implementation of this embodiment.

[0091] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the above-described full-duplex voice interaction method. The computer-readable storage medium can be a read-only memory, random access memory, flash memory, solid-state drive, magnetic disk, optical disk, or other non-transitory medium capable of storing program instructions.

[0092] Technical effect The above embodiments have the following technical effects: First, by using acoustic activity, user incremental identification prefix completeness, contextual conflict degree, and persistence together for continuous interruption scoring, the probability of false interruptions caused by short echoes, environmental noise, or single acoustic activity spikes can be reduced. Second, by using continuous interruption scoring, remaining response value, latency cost, speech overlap conflict cost, and replanning cost together for multi-action benefit decision-making, the system can perform more granular playback control between continuing playback, reducing volume, pausing playback, stopping playback, and switching responses, rather than simply "stopping playback when user speech is detected." Third, by retaining already played prefixes and only performing local replanning on unplayed suffixes, the system can reduce repeated playback content and regeneration length, reducing switching response latency while maintaining the continuity of already played content. Furthermore, through congruence consistency verification and semantic bridging cost, the system can reduce the probability of duplication, contradiction, unclear referencing, or slot conflicts between candidate suffixes and already played prefixes. Furthermore, by employing action-level hysteresis, switching penalty, adaptive switching margin, and graded backoff mechanisms, the system can reduce action jitter near the critical score and maintain control flow convergence when candidate suffixes time out, exceed the limit, or fail to validate, thereby achieving a better overall balance in full-duplex voice broadcast control compared to existing technologies.

[0093] Other optional implementation methods and parameter calibration To facilitate implementation, several alternative implementation methods are further provided below. The specific processing methods and values ​​involved below are merely examples, and those skilled in the art can make adaptive adjustments according to the actual deployment situation.

[0094] In some implementations, the control cycle can be consistent with the incremental output cycle of the streaming ASR or the polling cycle of the playback controller, for example, equivalent to the duration of one or several audio frames; the playback pointer can be determined by the audio timestamp and text token alignment information in the TTS playback buffer; when a switching response is triggered, the system retains the playback record corresponding to the already played prefix, clears or replaces the playback buffer corresponding to the unplayed suffix, and adds the candidate suffix to the queue to be played. To ensure the feasibility of distinguishing between played prefixes and unplayed suffixes, the current voice response is empty or the total number of tokens is zero. At this time, the system does not execute the above division, but instead maintains the listening state or regenerates the response to be broadcast; parameters involving division , , , as well as All values ​​are positive.

[0095] In some implementations, the stability or completeness of the user incremental identification prefix. The determination can be based on a comprehensive assessment of whether the streaming ASR prefix has regressed in the most recent control periods, whether a complete keyword or intent slot has been formed, and whether the identification confidence level is higher than a reference threshold, and then normalized to... Interval. If the identification prefix changes multiple times in adjacent control cycles or contains only short adverbs, then Take the lower value; if the identification prefix remains stable and contains explicit error correction, new request, or slot replacement information, then... Take the higher value.

[0096] In some implementations, the similarity function This can be achieved by combining slot comparison with text similarity: first, extract the task slots from the user incremental recognition prefix and the already broadcast prefix or the unbroadcast suffix; then, calculate the normalized similarity based on whether the slots are consistent, whether the keywords overlap, and the cosine similarity of the semantic vectors; when the user input contains error correction or switching trigger words such as "not", "change to", "replace", "cancel", "directly", the similarity result can be reduced or the context conflict score can be increased to reflect the difference between the user incremental input and the already broadcast prefix or the original unbroadcast suffix.

[0097] In some implementations, the suffix reprogramming function It can be implemented using rule templates, a dialogue manager, a lightweight generation module, or a trained language model; the replanning input includes the current dialogue context, the user's incremental recognition prefix, the semantic state of the already played prefix and the unplayed suffix; the replanning constraints include not repeating the already played prefix, responding to the user's addition or correction intent, maintaining consistency of the already played valid slots except for the explicit correction slots, and the output length meeting the requirements. Consistency of connection In this context, non-redundant items can be obtained from duplicate information slot detection, factual non-contradictory items can be obtained from rule bases, business confirmation status, or natural language contradiction detection models, referential consistency items can be obtained from coreference resolution modules, and slot consistency items can be determined by comparing the slot table in the dialogue state with the candidate suffix slot extraction results. When a certain detection module cannot reliably judge, the corresponding item can be set to a conservative value, so that the system tends to pause and wait or continue playing the original unplayed suffix, rather than forcibly outputting candidate suffixes with connection risks.

[0098] In some implementations, when the combined or modified benefits of multiple candidate actions are the same or the difference is less than a preset small threshold, the electronic device can prioritize maintaining the actual action executed in the previous control cycle; if there is no previous actual action executed, then the candidate action with lower intervention intensity is selected first, thereby further reducing the repetition of actions near the benefit critical zone.

[0099] In some implementations, each normalized weight (including to , to , to , to , to The sum of all parameters (including threshold, window length, and duration) can be 1; , , , , , , , , , , , , , , , Calibration can be performed using offline labeled samples, runtime logs, or grayscale testing; sliding window length With minimum number of stay periods The length can be equivalent to several control cycles. In the absence of labeled data, empirical initial values ​​can be used first, and adjustments can be made during operation based on accidental switching, timeout rollback, and repeated user input.

[0100] In some implementations, the candidate suffix generation exceeds the system's allowed response time, or the candidate suffix length exceeds... The consistency of the connection is lower than that of the previous one. If the net benefit of switching does not meet the threshold, the system enters a first-level pause and wait state. If the user input intensity decreases in the next control cycle and there is no obvious conflict with the previously unbroadcast suffix, the system enters a second-level state to continue broadcasting the original suffix. If the candidate suffix fails the verification several times consecutively and the user input continues to stably express a new intention, the system enters a third-level state to stop broadcasting and switch to a listening state. The number of consecutive times can be determined according to the control cycle length, system response latency requirements, and application scenario.

[0101] In some implementations, system parameters can be calibrated using scenarios such as in-vehicle navigation, meeting summaries, and intelligent customer service. Calibration metrics may include false interruption rate, average switching response latency, number of repeated token broadcasts, number of action switching attempts, candidate suffix verification failure rate, and prefix-suffix semantic contradiction rate. These metrics are used to select weights, thresholds, and window lengths; their specific values ​​are subject to actual measurement and calibration. In the in-vehicle scenario, the broadcast prefix can be "We have confirmed the safety of your current route," the user incremental identification prefix can be "Change to the airport," and the candidate suffix can be "We are now replanning your route to the airport." In the customer service scenario, the broadcast prefix can be "Your identity information has been verified," the user incremental identification prefix can be "Transfer directly to a human agent," and the candidate suffix can be "We are transferring you to a human agent." The above scripts are only used to illustrate the connection method between prefix retention and suffix replanning; the actual scripts can be configured by the business system.

[0102] illustrate It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this application, if a reference is made to performing an action based on an element, it means performing the action at least based on that element, including two cases: performing the action only based on that element, and performing the action based on that element and other elements. Expressions such as "multiple," "repeatedly," and "various" include two, two times, two kinds, and more than two, more than two times, and more than two kinds.

[0103] All documents mentioned in this application are considered to be incorporated in their entirety into the disclosure of this application so that they can serve as a basis for modifications if necessary. Furthermore, it should be understood that after reading the foregoing disclosure of this application, those skilled in the art can make various alterations or modifications to this application, and these equivalent forms also fall within the scope of protection claimed in this application.

Claims

1. A full-duplex voice interaction method, running on an electronic device, characterized in that, During the system's broadcast of the current voice response, the following steps are executed periodically: Acquire comprehensive observations: Receive user-input voice in real time to obtain the user's incremental recognition prefix. And determine the current broadcast progress based on the voice broadcast status. Prefix already broadcast and unbroadcast suffix ,in The position of the played token corresponding to the current playback pointer. The total number of tokens in the current voice reply; Calculate the continuous interruption score: Based on the acoustic activity features of the user input speech, the completeness of the user incremental recognition prefix, and the degree of contextual conflict between the user incremental recognition prefix and the already played prefix, calculate the continuous interruption score at the current moment. ; Multi-action benefit decision: based on the continuous interruption score and the remaining response value of the unannounced suffix. The system calculates the latency cost of each candidate action, the comprehensive benefit of each candidate action in the candidate action set, and selects the current control action from the candidate action set based on the comprehensive benefit. Unreported suffix replanning: When the current control action indication is switched back, the already reported prefix is ​​retained, and based on the semantic state of the user incremental identification prefix, the already reported prefix, and the unreported suffix, local replanning is performed only on the unreported suffix to generate candidate suffixes. ; Output a new voice response: Concatenate the already broadcast prefix with the candidate suffix to obtain the total number of tokens. The system provides a new voice response and controls the system to broadcast the candidate suffix starting from the playback pointer position adjacent to the already broadcast prefix.

2. The method according to claim 1, characterized in that, The continuous interruption score Calculate using the following formula: The meanings of each function in the formula are as follows: The discrete time points corresponding to the control cycle; Score for acoustic activities, User voice activity level is determined by front-end voice activity detection or energy detection. This is the scaling factor. The threshold for voice activity. For the Sigmoid function; Score the text integrity. For the user incremental identification prefix Stability or integrity; Score for contextual conflict. For the previously broadcast prefix, The number of tokens that have been broadcast. For similarity functions; For sustained scoring, Deadline User's continuous speaking time For reference duration, To obtain the minimum value; , , , For each score weight, and .

3. The method according to claim 1, characterized in that, The candidate action set is These represent continuing broadcast, lowering the volume, pausing broadcast, stopping broadcast, and switching to reply, respectively; for any candidate action... Its comprehensive benefits Calculate using the following formula: The meanings of each function in the formula are as follows: In response to incremental user input, the control benefits generated by the broadcast control, For action The responsiveness coefficient to incremental user input; The remaining response value, For the broadcast progress ( , (Total number of tokens in the current voice reply) The percentage of cases not reported. The remaining content importance coefficient is determined by the response content structure tag, broadcast type, or content type tag, where the content type tag includes security prompts, identity verification, or business verification. To retain the benefits of the current response on the system side. , The retention factor for residual income; As a cost of time delay, Its average time delay coefficient; The cost of overlapping and conflicting statements The conflict sensitivity coefficient, To measure user voice activity; To re-plan at a cost, For replanning cost coefficient; to These are the weighting coefficients, and .

4. The method according to claim 3, characterized in that, When the selected current control action indicates a switchover, first calculate the switching benefit. and the remaining response value The unbroadcast suffix reprogramming is performed only if the following condition is met: The meanings of each function in the formula are as follows: To realize the potential benefits of implementing a switchover and rebalancing plan; The incremental importance of the new user input relative to the original unannounced suffix. The unbroadcast suffix, For similarity function, The larger the value, the greater the difference between the new user input and the original unannounced suffix; , , These are the weighting coefficients; To switch the revenue threshold.

5. The method according to claim 4, characterized in that, The local reprogramming uses a suffix reprogramming function. Generate the candidate suffixes and impose length constraints on them: The meanings of each function in the formula are as follows: That is, the candidate suffixes; For the current dialogue context; For the user incremental identification prefix; The previously broadcast prefix; The intent or slot semantic state of the unbroadcast suffix is ​​taken from the summary of the original unbroadcast suffix, the dialogue intent slot state, or the intermediate semantic representation of the original response; The total length of the new voice response. The maximum allowed length for reprogramming suffixes.

6. The method according to claim 5, characterized in that, Before outputting a new voice response, the candidate suffixes are... Relative to the previously broadcast prefix Calculate the consistency of connections Only New voice replies are only allowed at certain times: The meanings of each function in the formula are as follows: This is a non-redundant component, representing a candidate suffix that does not repeat the information slots that have already been conveyed in the broadcast prefix, obtained by text comparison or duplicate information detection; For factual non-contradictory items, it indicates that the candidate suffix does not negate the security prompts, identity verification or business verification conclusions that are still valid in the broadcast prefix, and is obtained by contradiction detection; For the reference consistency sub-item, it represents that the pronouns or referents in the candidate suffixes can be resolved in a consistent manner with the already broadcast prefix, and is obtained by coreference resolution; The slot consistency item represents that the task slots with already filled broadcast prefixes are consistent in the candidate suffixes (except for the slots for explicit error correction of the user incremental identification prefix), which is obtained by slot comparison; , , , Assigning weights to each item. To ensure consistency, a threshold is passed; specifically, when the candidate suffix negates a valid security prompt, identity verification, or business verification conclusion in the already broadcast prefix, the factual non-contradiction item is reclassified. Set to a failed state, or change the connection consistency. Determined to be below the aforementioned consistency threshold This prevents switching responses and keeps the previously broadcast prefix unchanged.

7. The method according to claim 6, characterized in that, Based on the aforementioned consistency Calculate the semantic bridging cost and by switching net income As a criterion for executing a switchover response: If and only if , and If both conditions are met, execute the switching response and output the new voice response; The meanings of each function in the formula are as follows: For the semantic bridging cost, and Negative correlation; The number of boundary conflicts between candidate suffixes and already broadcast prefixes in terms of time or logical order is obtained by sequential comparison. For candidate suffix length or generation delay risk items; , , These are the weighting coefficients; To switch to net income, , , , , These are the switching revenue, remaining recovery value, switching revenue threshold, maximum length of the replanning suffix, and connection consistency pass threshold, respectively.

8. The method according to claim 3, characterized in that, The multi-action revenue decision further includes action-level hysteresis: The action that yields the greatest overall benefit is designated as the candidate control action. The currently executed control action is recorded as And set an action hysteresis between the two; defined according to the intervention intensity of the current broadcast. And construct the modified return With adaptive switching margin : Let gain :exist The direction of the upgrade, when or conditions for forceful interruption At the time of its establishment, the order Among them, forced insertion is triggered without minimum dwell time; The direction of retreat, except In addition, the number of control cycles that the current action has lasted must also be considered. Only then can updates be performed; otherwise, the previous control cycle will remain unchanged. ; The meanings of each function in the formula are as follows: This refers to the actual actions executed in the previous control cycle. This is an indicator function (it takes a value of 1 if the condition is true, and 0 otherwise). To switch the penalty level, This is the last action transition time. The decay time constant, It is a natural exponential function; This is the margin base value. This is the variance sensitivity coefficient. For the most recent The sliding variance of continuous interruption scoring within a control period. This is the schedule coefficient; The user's ability to interrupt and continue to score points. The completeness of the incremental identification prefix for the user; , , These are the thresholds for forced interruption, persistence, and stability, respectively. The minimum number of dwell periods in the reversing direction.

9. The method according to claim 7, characterized in that, The unreported suffix replanning further includes tiered fallback: when the candidate suffix generation times out, , or At the same time, the already broadcast prefix is ​​not discarded, and hierarchical rollback is performed in sequence: Level 1, when the user incremental identification prefix is ​​not yet stable, the broadcast is paused and re-evaluated in the next control cycle; Level 2: When the interruption intensity drops and there is no obvious conflict between the original un-broadcast suffix and the current user input, the volume is reduced or the broadcast continues to continue broadcasting the original un-broadcast suffix and the current switch is abandoned. Level 3: When the interruption intensity remains high and the candidate suffix fails the connection consistency check multiple times, stop broadcasting and switch to listening mode, and make overall plans after the user's expression is complete.

10. A full-duplex voice interaction system, characterized in that, The method includes at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions, when executed, causing the at least one processor to perform the full-duplex voice interaction method according to any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the full-duplex voice interaction method according to any one of claims 1 to 9.

Images