Dialogue control method, model training method, dialogue system, and computing device

Abstract

The embodiment of the specification provides a dialogue control method, a model training method, a dialogue system and a computing device, wherein the dialogue control method comprises: in response to receiving an audio segment of a current turn of a target dialogue, obtaining a fixed prefix of the target dialogue; based on the fixed prefix and the audio segment, constructing a model input of the current turn, the model input comprising text tokens and audio tokens; calling a multi-modal model, incrementally performing forward calculation on the model input based on a key-value cache of the fixed prefix, and reading a prediction score corresponding to any decision reading point at the decision reading point, wherein the decision reading point is the position of the last audio token; based on the prediction score, determining a target action from a preset action set; based on the target action, performing turn state switching of the current turn. End-to-end turn control is realized, which guarantees the continuity of interaction and the timeliness of response, and takes into account the computing efficiency and decision accuracy.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to a dialogue control method, a model training method, a dialogue system, and a computing device. Background Technology

[0002] With the development of artificial intelligence and human-computer voice interaction technology, intelligent dialogue systems have been widely adopted in various application scenarios.

[0003] Currently, dialogue systems typically employ cascaded speech recognition and decision control modules, or multimodal models for end-to-end turn control. The cascaded architecture serially connects speech recognition and decision control, using the recognition results as the basis for decision-making to support the orderly flow of dialogue turns. The multimodal model, on the other hand, uniformly processes multimodal inputs such as audio and text to achieve end-to-end determination and switching of dialogue states. Both aim to ensure the continuity and timeliness of dialogue interaction.

[0004] However, cascaded architectures suffer from serial latency in speech recognition, and speech recognition errors in noisy backgrounds can propagate to the decision-making layer. When using multimodal models for high-frequency direct tuning, the interactive control requires online high-frequency calls (e.g., every 200ms), and direct calls would incur unbearable computational costs.

[0005] Therefore, how to effectively reduce the computing cost of high-frequency calls while achieving end-to-end call state switching has become an urgent problem to be solved. Summary of the Invention

[0006] In view of this, embodiments of this specification provide a dialogue control method. One or more embodiments of this specification also relate to a model training method, a dialogue system, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0007] According to a first aspect of the embodiments of this specification, a dialogue control method is provided, comprising: In response to receiving the audio segment of the current playback round of the target dialogue, obtain the fixed prefix of the target dialogue, wherein the key-value cache of the fixed prefix has been pre-written; Based on fixed prefixes and audio segments, construct the model input for the current broadcast round. The model input includes text words and audio words. The multimodal model is invoked, and incremental forward computation is performed on the model input based on the key-value cache with a fixed prefix. At any decision read point, the prediction score corresponding to that decision read point is read. The decision read point is the position of the last audio word in that beat. Based on the predicted score, the target action is determined from a preset action set; Based on the target action, switch the state of the current broadcast wheel.

[0008] According to a second aspect of the embodiments of this specification, a model training method is provided, comprising: Obtain the fixed prefix of the target dialogue sample, the sample audio segment, and the frame-level decision label corresponding to the sample audio segment; The sample model input is constructed based on the fixed prefix of the sample and the sample audio segments; The multimodal model is invoked, and the sample model input is forward computed based on the key-value cache with a fixed prefix of the sample. At any decision read point, the prediction score corresponding to that decision read point is read. Based on the predicted scores, training actions are determined from a pre-set action set; Determine the decision loss based on training actions and frame-level decision labels; Based on the decision loss, the multimodal model is trained to obtain a trained multimodal model, which is then used to execute the aforementioned dialogue control method.

[0009] According to a third aspect of the embodiments of this specification, a dialogue system is provided, including an online reasoning module, the online reasoning module comprising: The prefix acquisition unit is used to acquire the fixed prefix of the target dialogue in response to receiving the audio segment of the current playback round of the target dialogue, wherein the key-value cache of the fixed prefix has been pre-written; Input building units are used to construct the model input for the current broadcast round based on a fixed prefix and audio segments; The model invocation unit is used to invoke the multimodal model, perform incremental forward computation on the model input based on the key-value cache with a fixed prefix, and read the prediction score corresponding to any decision read point, where the decision read point is the position of the last audio word of the beat; An action determination unit is used to determine the target action from a preset action set based on the prediction score; The state switching unit is used to switch the state of the current broadcast wheel based on the target action.

[0010] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which implement the steps of the above method when executed by the processor.

[0011] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0012] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0013] In one embodiment of this specification, by pre-writing a fixed prefix of the target dialogue and reusing its key-value cache, when the audio segment of the current turn is received, only incremental forward computation needs to be performed on the model input constructed from the fixed prefix and the audio segment based on this cache. This avoids the repetitive computational cost caused by calculating all prefixes from scratch every time a beat is called, thereby effectively reducing the computational cost of high-frequency direct tuning of the multimodal model. By selecting the position of the last audio word of the current turn as the decision reading point to read the prediction score, the model outputs a decision only after fully perceiving the audio information of the beat, cutting off the transmission path of speech recognition serial delay and recognition errors in high-noise backgrounds to the decision layer in the cascaded architecture. Based on the prediction score, the target action is determined from the preset action set and the turn state is switched, realizing end-to-end turn control. While ensuring interaction continuity and response timeliness, it also takes into account computational efficiency and decision accuracy. Attached Figure Description

[0014] Figure 1 This is a flowchart of a dialogue control method provided in one embodiment of this specification; Figure 2 This is a flowchart illustrating a dialogue control method provided in one embodiment of this specification; Figure 3 This is a schematic diagram of the dialogue timeline in a dialogue control method provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the interaction control delay in a dialogue control method provided in one embodiment of this specification; Figure 5 This is a schematic diagram illustrating the updating of interactive personalities and / or output labels by a multimodal model in a dialogue control method provided in one embodiment of this specification; Figure 6 This is a flowchart illustrating a model training method provided in one embodiment of this specification; Figure 7 This is a schematic diagram of frame-level decision tag synthesis and multi-system mapping in a dialogue control method provided in one embodiment of this specification; Figure 8 This is a schematic diagram of a parallel tag track paradigm in a dialogue control method provided in one embodiment of this specification; Figure 9 This is a schematic diagram of a decision feedback paradigm in a dialogue control method provided in one embodiment of this specification; Figure 10This is a schematic diagram of the structure of a dialogue system provided in one embodiment of this specification; Figure 11 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0015] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below. The audio data and dialogue data involved in the embodiments of this specification all originate from legally authorized data collection channels, and their collection and use are within the scope of the corresponding authorization.

[0016] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0017] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0018] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0019] Automatic Speech Recognition (ASR): a technology that converts audio segments into text; streaming ASR refers to receiving the audio segment of the current playback round while progressively outputting text. In traditional cascaded systems, it is used to convert user speech into text input for large language models.

[0020] Text-to-Speech (TTS): a technology that converts text into speech; streaming TTS refers to synthesizing audio while receiving text. In this solution, it corresponds to generating a voice response or interruption response based on the target action for playback.

[0021] Cascaded speech system: A dialogue system composed of independent modules such as automatic speech recognition, large language model, and speech synthesis. In this scheme, a multimodal model is used to replace the cascaded control layer, and fixed prefixes and audio segments are used as inputs to perform turn-by-turn state switching.

[0022] Inner-monologue: Before generating a voice response, the model first generates a text prefix aligned with the audio to guide pronunciation and improve content quality. In this scheme, the fixed prefix is ​​constructed based on system prompts, historical dialogue text, and preset turn control tags, and its key-value cache has been pre-written.

[0023] Chunk / Prefill / Look-ahead: A chunk refers to a small segment of audio or audio word processed at one time in streaming processing; prefill refers to calculating and writing a known fixed prefix into a key-value buffer at once; look-ahead refers to allowing a small number of future frames to be referenced when encoding audio words at the current decision point, in order to improve audio encoding quality.

[0024] Half-duplex / Full-duplex: Half-duplex refers to the interaction being forcibly discretized into "you finish speaking, then I speak" rounds; full-duplex refers to the system being able to dynamically switch between listening and broadcasting states, enabling simultaneous listening and speaking, and being interrupted at any time. The interaction is treated as two parallel audio streams. This solution determines the target action by reading the predicted score at the decision read point to execute the full-duplex turn state switching.

[0025] Voice Activity Detection (VAD) / Endpointing: A traditional method for determining acoustic state (such as a sounding state or a silent state) based on acoustic energy or silence duration, often using silence thresholds to segment rounds; In this scheme, the judgment conditions of the corresponding model call gating are used to determine whether to construct the model input for the current broadcast round based on the audio segment.

[0026] Turn-taking / turn: The process and unit of transferring the right to speak between participants in a dialogue; in this scheme, it specifically refers to the switching between the listening state and the speaking state of the current speaking turn, which is executed based on the target action.

[0027] Backchannel: A brief response given by the listener without interrupting the speaker; it is a type of overlapping speech. In this scheme, the backchannel action corresponding to the preset listening state action set is used to keep the current broadcast round in the listening state and generate the corresponding backchannel speech for broadcast.

[0028] Interruption / Barge-in: One party intervenes during the other party's speech and attempts to take over the right to speak; in this solution, the active interruption or takeover action corresponding to the preset listening state action set is used to switch the current broadcast wheel from listening state to broadcast state, stop receiving audio segments, and generate the corresponding interruption reply voice or voice reply for broadcast.

[0029] When-to-stop / When-to-speak: These are two core judgments in full-duplex interactive control, namely "whether the speaker should stop and give way" and "whether to take over the speaking." In this scheme, the target action is determined from the preset action set based on the predicted score to execute the switching of the current broadcast turn's state.

[0030] Large Language Model (LLM) / Omni-modal Large Language Model: An autoregressive model based on a Transformer architecture and pre-trained on massive amounts of data; the Omni-modal Large Language Model can simultaneously receive multimodal inputs such as audio segments, already played text, and reference audio, and is used to call the multimodal model to perform incremental forward computation and read the predicted score at the decision read point.

[0031] Autoregressive / causal attention: The model is generated bit by bit, and the representation of any decision point depends only on the audio lexical units at that position and those preceding it, not on subsequent lexical units; causal masking is a common means of implementing this mechanism, used to support incremental forward computation of the model input based on key-value caching with fixed prefixes.

[0032] Linear attention: A type of structure that approximates standard self-attention with linear complexity, suitable for streaming long sequences; it is usually used in combination with a small number of full attention layers. In this scheme, it can be used to process key-value caching of long audio segments to reduce computational cost.

[0033] Supervised Fine-Tuning (SFT), Reinforcement Learning (RL), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO) are common paradigms for further training pre-trained multimodal models. In the model training method of this scheme, they can be used to train multimodal models based on decision loss to obtain a trained model.

[0034] Cross-Entropy Loss (CE) / Focal Loss / Class Imbalance: Commonly used losses for classification training and their improvements for rare classes; Class imbalance refers to a significant disparity in the number of samples in each class. In this scheme, the calculation of the corresponding decision loss includes type weighting, which assigns weights according to the frequency of occurrence of each action type in a unified decision vocabulary, and hard example modulation, which reduces the weight of easily classified samples based on prediction differences.

[0035] Forced alignment: Aligning the transcribed text and audio word by word in time to obtain word-level timestamps; in the model training method of this scheme, it is used as an alternative scheme to generate frame-level decision labels, and the preferred scheme is calculated by a deterministic function determined by multiple orthogonal objective axes and the current effective judgment rule.

[0036] Speaker diarization / binarization: a technique for determining "when and by whom" and can be separated into multiple channels; in this solution, it corresponds to the speaker gating in the two-step method, used to determine whether the audio segment comes from the caller, is in the near field, or is being spoken to by the bot, in order to avoid accidental triggering by bystanders / noise.

[0037] Data synthesis: This involves automatically generating training data to augment datasets when real data is scarce, expensive, or involves privacy concerns. In the model training method of this scheme, the process includes generating a preset dialogue script based on a preset behavior recipe, performing audio rendering to synthesize training seeds, calculating frame-level labels through a deterministic function, and segmenting sample audio segments and frame-level decision labels according to a preset window.

[0038] Token / Frame Rate: The discrete units processed by the model and their number per second; in this scheme, tokens include audio tokens, decision words, etc., the frame rate corresponds to the tag frame grid (e.g., 1 / 7 second ≈ 142.9 milliseconds) and is bound to the audio encoding rate, and the decision read point is the position of the last audio token of that beat.

[0039] Key-value cache (KV cache) / prefix cache: caches the calculated key-value pairs or the calculation results of shared prefixes to avoid repeated calculations, and is used to reduce the cost and latency of high-frequency, long context calls. In this scheme, the key-value cache of fixed prefixes has been pre-written, reused in incremental forward calculation, and reorganized and updated based on the switched talk state after the talk state switch.

[0040] Unnormalized scores (logits): The unnormalized scores output by the model on the unified decision vocabulary are used to obtain the probability distribution through softmax. In this scheme, the predicted scores corresponding to the decision read points are read at the decision read points and used to determine the target action from the preset action set.

[0041] Streaming / Incremental inference: An inference method in which the input arrives gradually over time, the model performs incremental calculations and outputs results in a timely manner; in this scheme, in response to receiving the audio segment of the current broadcast round, the multimodal model is invoked to perform incremental forward calculations on the model input based on a key-value cache with a fixed prefix.

[0042] In voice dialogue scenarios, to achieve a dialogue experience close to real interpersonal communication, the core lies in whether the dialogue control layer of the dialogue system can accurately and in real time determine the timing of speaking, stopping, and speaking, thereby executing the switching of the current turn's state. Traditional half-duplex solutions forcibly discretize the dialogue into turns, with external speech activity detection determining the turn boundaries based on acoustic energy or silence duration. This approach has the following limitations: First, the silence threshold must be statically balanced between early interruption and slow response, which cannot adapt to the ever-changing nature of real dialogue; second, it cannot express overlapping speech that does not interrupt, such as quiet acceptance; in addition, the model is in an idle state during the listening phase, only starting processing when the endpoint detection is triggered, naturally introducing latency.

[0043] The current network uses a cascaded speech system as its dialogue control layer, which consists of streaming automatic speech recognition, rule-based judgment, and independent large language models concatenated sequentially. This cascaded solution has significant bottlenecks: First, the latency is compounded by the first-word delay of automatic speech recognition, endpoint waiting, and the sequential calling of the large language model. Second, once recognition errors enter the text, they cannot be corrected downstream. In noisy or background noise environments, background speakers or noise are often misidentified as user speech, leading to false triggers or missed triggers, i.e., automatic speech recognition error propagation. Third, the round boundary is determined by a static silence threshold, requiring a static trade-off between early interruption and slow response.

[0044] Dialogue control schemes based on multimodal large models offer a possibility for overcoming the aforementioned bottlenecks. Multimodal large models can directly use audio segments as input for dialogue control judgments, thus eliminating reliance on real-time automatic speech recognition, removing serial latency and error propagation, and being more robust to background human voices. Furthermore, learnable semantic frame-level decisions can replace static silence thresholds. However, dialogue control requires high-frequency online calls; directly calling large-scale multimodal large models at high frequencies would be computationally prohibitive. Moreover, when the basic model is used for pure state control, improper information organization and decision-making methods can significantly degrade performance. For example, mixing historical dialogue text with the currently judged audio segment without structural isolation; or inserting the decision word corresponding to the target action determined by the previous decision point into the model input stream for attention calculation at subsequent positions, can easily cause the model to learn a shortcut by copying the previous label, leading to delayed flip points; or inserting preset turn control markers in the middle of the sequence to switch phases can cause all subsequent key-value caches to become invalid and result in judgment jitter.

[0045] Therefore, this specification provides a dialogue control method that, by pre-writing a fixed prefix of the target dialogue and reusing its key-value cache, upon receiving the audio segment of the current turn, only incremental forward computation needs to be performed on the model input constructed from the fixed prefix and the audio segment based on this cache, avoiding the repetitive computational cost of recalculating all prefixes from scratch every time a beat is called. By selecting the position of the last audio word of the current turn as the decision reading point to read the prediction score, the model outputs a decision only after fully perceiving the audio information of the beat, cutting off the transmission path of speech recognition serial delay and recognition errors in high-noise backgrounds to the decision layer in the cascaded architecture. Based on the prediction score, the target action is determined from the preset action set and the turn state is switched, realizing end-to-end turn control.

[0046] In one optional implementation, the dialogue control method of this specification embodiment can be deployed as an independent, pluggable interactive central control module, which only undertakes frame-level decisions on the timing of speech interruption and speech generation. It can be mounted on existing full-duplex voice dialogue systems or existing network voice customer service links, replacing their original cascaded control layers without affecting other components. For example, it can be deployed in the control layer of products such as intelligent customer service systems, intelligent outbound call systems, intelligent callback systems, or intelligent question-and-answer navigation systems. Alternatively, it can be deployed as an independent service in parallel with the main dialogue model and voice generation model of the aforementioned products, with the products calling the interactive central control module according to the decision rhythm and executing its output target action.

[0047] In related technologies, there are various full-duplex voice dialogue systems, which can be categorized according to the stage where the round-based control decision-making occurs. Some systems place the decision-making in an independent scheduler, some systems read the hidden state and make decisions in an external side-by-side module, and some systems embed round-based decisions within the speech-text word stream. The direct counterpart of the embodiments in this specification is an independent interactive central control module, which carries frame-level decisions and reads the prediction scores corresponding to the decision read points.

[0048] Existing interactive control module solutions suffer from the following shortcomings: First, some existing solutions employ a cascaded architecture, leading to issues such as latency aggregation, error propagation, and static thresholds. Some solutions directly call large models at high frequencies, resulting in prohibitive computational costs. Furthermore, some solutions have flawed decision-making mechanisms, such as the lack of explicit readable decision bits, semantic fragmentation requiring structural modifications, the existence of copy shortcuts and positional drift, or cache invalidation due to rewriting markers in the sequence. All of these are unsuitable for low-latency, low-cost, high-frequency, and noise-resistant online dialogue control layers. Second, some existing solutions adjust dialogue style via numerical knobs, which are non-natural language commands, non-combinable, and require manual adjustment. Alternatively, the output label system is fixed during training, with one set of weights locking one set of outputs, making online switching via natural language commands impossible. Third, the decision label granularity of existing training data routes is generally coarse, either limited to sentence level or lacking frame-level explicit labels, resulting in insufficient point density. Rare but crucial behaviors are not learned effectively due to class imbalance.

[0049] To address the shortcomings of existing solutions, the dialogue control method and model training method provided in this specification effectively solve the aforementioned problems by employing a scalable multimodal model, a parallel label trajectory paradigm, a command-switched output label system and dialogue personality, and a large-scale synthesis route for frame-level decision training data. Specific implementation details will be provided in subsequent embodiments.

[0050] This specification provides a dialogue control method that effectively reduces the computational cost of high-frequency calls while achieving end-to-end turn-by-turn state switching. It also relates to a model training method, a dialogue system, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail in the following embodiments.

[0051] See Figure 1 , Figure 1 A flowchart of a dialogue control method provided in one embodiment of this specification is shown, including: Step 102: In response to receiving the audio segment of the current broadcast round of the target dialogue, obtain the fixed prefix of the target dialogue, wherein the key-value cache of the fixed prefix has been pre-written.

[0052] The target dialogue is the dialogue currently undergoing turn-based state judgment and processing in the dialogue system, corresponding to a fixed prefix and its key-value cache. The current turn is the time segment in the target dialogue where turn-based state judgment and processing are currently underway. For example, it could be the period when the bot is broadcasting speech and the user might interrupt, or the period when the user is speaking and the bot is listening. For example, the current turn corresponds to one turn, and its boundary is defined by the turn-based state switching point.

[0053] The audio segment in the current broadcast round is user-side audio data received within the current broadcast round and sampled using a preset frame grid. In the embodiments of this specification, the audio segment is a single-channel audio stream (e.g., single-channel audio with a sampling rate of 16kHz), and it is entered into the model in time-segmented, beat-by-beat format. For example, the tag frame grid is 1 / 7 second ≈ 142.9 milliseconds, which is bound to the audio encoding rate of approximately 7 words / second, and each audio word corresponds to a decisionable position; the lower limit for determining the beat interval is one frame grid, and the upper limit is determined by the delay budget.

[0054] The dialogue system is a system capable of full-duplex voice interaction. It includes an online inference module, which performs frame-level decisions on when to initiate and stop speech, and can be integrated into existing full-duplex voice dialogue systems or existing voice customer service links, replacing the original cascaded control layer without affecting other components. The dialogue system is suitable for products requiring real-time voice interaction with users, such as intelligent customer service, intelligent outbound calling, intelligent callback, and intelligent question-and-answer navigation.

[0055] The fixed prefix for the target dialogue is an input portion that remains unchanged across multiple calls, consisting of system prompts, historical dialogue text, etc., and optionally includes preset turn control markers. The fixed prefix is ​​reused throughout the entire dialogue or window, and its corresponding historical context has several selectable forms, such as automatically recognized speech text from previous turns; or a temporal mixture of bot-produced text and user-side audio (this form does not undergo automatic speech recognition); or a short segment of user-side reference audio (only a short segment is used for speaker gating, approximately 1 second in length, with a maximum of approximately 1.5 seconds); all three can also be combined. The fixed prefix constitutes the fixed prefix area, followed by the live area (the current user-side audio; during the broadcast phase, bot text words are interspersed according to the broadcast progress).

[0056] The fixed prefix key-value cache is the result of encoding the fixed prefix region and caching the pre-computed key-value pairs to avoid repeated calculations of the fixed prefix in subsequent calls. In the embodiments of this specification, the fixed prefix key-value cache is reused as much as possible throughout the entire conversation, and each subsequent decision only needs to calculate the newly added part based on the cache. For example, at the beginning of the session, the fixed prefix is ​​prefilled once, and the calculation result is written to the key-value cache; thereafter, each decision directly reuses the cache.

[0057] To obtain the fixed prefix of the target dialogue, one possible approach is to read the pre-written fixed prefix from the cache; another possible approach is to construct the fixed prefix of the target dialogue based on system prompts, historical dialogue text, and preset turn control flags; yet another possible approach is to use system prompts, historical dialogue text, and preset turn control flags as a fixed prefix area, encode the fixed prefix area to obtain the fixed prefix of the target dialogue, and write the fixed prefix into a key-value cache; no specific method is specified here.

[0058] For example, taking a smart customer service call as an example: At the start of the conversation, the system injects a system prompt (including the selection of the output tag system, interaction profile, and judgment criteria), along with historical dialogue text (such as the user saying "wait a moment" in the previous round, and the opening text already played by the bot), which together form a fixed prefix. The system pre-fills this fixed prefix area, writing the key-value pairs into the key-value cache. Subsequently, throughout the entire playback round, whenever an audio segment of the current playback round is received, the fixed prefix and its key-value cache are directly retrieved without recalculation.

[0059] Step 104: Based on the fixed prefix and audio segment, construct the model input for the current broadcast round. The model input includes text words and audio words.

[0060] The model input for the current broadcast round is a single causal word sequence composed of a fixed prefix and the audio segment received in the current broadcast round. This sequence is used to input the multimodal model for incremental forward computation. The model input includes text words from the fixed prefix and audio words from the audio segment, and may also include previously broadcast text and reference audio. For example, multimodal words may arrive simultaneously in the same frame (such as one bot text word and two audio words), which are arranged in a fixed order in the sequence, and the training process is consistent with the online process.

[0061] Based on a fixed prefix and an audio segment, the model input for the current playback round is constructed. The model input includes text words and audio words. One possible approach is to concatenate the fixed prefix and the audio segment to obtain the model input for the current playback round. Another possible approach is to obtain the already played text and reference audio, and then concatenate the fixed prefix, the audio segment, the already played text, and the reference audio to obtain the model input for the current playback round. Yet another possible approach is to insert the already played text and the reference audio into the audio word sequence corresponding to the audio segment in chronological order, and then concatenate the fixed prefix and the audio segment after inserting the already played text and the reference audio in a fixed arrangement order to obtain the model input for the current playback round. No specific limitations are imposed here.

[0062] For example, when constructing the model input, starting from a fixed prefix pre-written into the key-value cache, the audio segments arriving in real-time during the current playback round (e.g., the audio item corresponding to "Hello, may I interrupt?" inserted by the user during the bot's playback of "Are you available to communicate today?") are appended to the end of the sequence. Only the existing content is appended during the construction process; no rewriting is performed, thus ensuring compatibility with the fixed prefix's key-value cache. If the window contains already played text, the already played text and the reference audio are inserted into the audio lexical sequence corresponding to the audio segment in chronological order, and then concatenated with the fixed prefix in a fixed arrangement order (e.g., bot text first, then audio) to obtain the model input for the current playback round.

[0063] Step 106: Call the multimodal model, perform incremental forward computation on the model input based on the key-value cache with a fixed prefix, and read the prediction score corresponding to any decision read point, where the decision read point is the position of the last audio word of the beat.

[0064] A multimodal model is an autoregressive model that simultaneously receives multimodal inputs such as audio and text to perform interactive control decisions. It can be pre-trained on massive datasets based on the Transformer architecture. In the embodiments described in this specification, the multimodal model is a model with controllable scale (e.g., the number of activation parameters is greater than 0.6B and not greater than 3B; multiple size ranges can be selected within this range for separate training and evaluation) to reduce the computational cost of high-frequency calls. The multimodal model can employ causal / block-causal encoding with a fixed lookahead, ensuring that appending subsequent audio does not change the already encoded lexical units, thereby guaranteeing bit-by-bit equivalence between training and online implementation.

[0065] Due to the causal attention mechanism, the representation of any decision read point depends only on the word at that position and the words preceding it. Therefore, the forward computation of the entire window sequence during the training phase is equivalent to the step-by-step incremental forward computation of the hidden state at the decision read point during the online phase, which satisfies the following causal equivalence theorem: h^{(1)}(a_k)=h^{(N)}(a_k), a_k∈P Where h^{(1)}(a_k) is the hidden state of the k-th incremental forward pass at decision read point a_k, and h^{(N)}(a_k) is the hidden state of the entire training window forward pass at a_k. This equivalence is uniquely constrained by the rule that "no additional tokens that will be attended by subsequent positions are mixed between any two read points", which ensures the consistency between training and online inference, and makes the decision distribution learned in training reproducible online as is.

[0066] Incremental forward computation involves the model input arriving gradually over time. The multimodal model, based on cached key-value pairs of fixed prefixes, only computes and outputs new input tokens in a timely manner, without recompiling the entire prefix from scratch. In the embodiments described in this specification, incremental forward computation is performed based on the key-value cache of fixed prefixes. The fixed prefixes are directly reused from the cache of previous audio, and only the newly added parts are computed.

[0067] The decision read point is the position of the last audio word in the current beat of the model input. In the embodiments of this specification, multiple words may arrive simultaneously within the same beat. The decision read point is the position of the last audio word in that beat. This decision read point reuses existing audio bits and does not insert any special positions.

[0068] The prediction score corresponding to the decision read point is the probability distribution output by the multimodal model at the decision read point on a unified decision vocabulary. In the embodiments of this specification, the prediction score corresponding to the decision read point is read at the decision read point and used to determine the target action from a preset action set, without generating or adding any lexical units.

[0069] A beat is a time unit for one processing step in streaming, corresponding to one decision beat. In the embodiments of this specification, the lower limit of the decision beat interval is equal to the tag frame grid (1 / 7 second ≈ 142.9 milliseconds), and the upper limit is determined by the latency budget; the larger the interval, the slower the response.

[0070] The last audio word of a beat is the last audio word in the sequence after the audio segment of a beat has been encoded. In the embodiments of this specification, the decision read point is the position of the last audio word of that beat, that is, the read point reuses an existing audio bit.

[0071] The predicted score for the last audio word of a beat is the unnormalized score of the multimodal model outputting the corresponding unified decision vocabulary at the position of the last audio word of the beat. In the embodiments of this specification, since the predicted score of the last audio word is pre-calculated in the pre-filled data, the additional decoding latency is approximately zero.

[0072] The multimodal model is invoked, and incremental forward computation is performed on the model input based on a key-value cache with a fixed prefix. At any decision readout point, the predicted score corresponding to that decision readout point is read. One possible approach is to invoke the multimodal model, perform incremental forward computation on the model input based on a key-value cache with a fixed prefix, and read the predicted score corresponding to that decision readout point at any decision readout point. Another possible approach is to read the predicted score at the last audio word of the beat during the incremental forward computation. Yet another possible approach is to use the multimodal model to perform forward propagation on the model input and obtain the predicted score at the decision readout point. No specific limitation is imposed here.

[0073] For example, the bot is broadcasting an opening statement (Phase A, the broadcast phase), and the user inserts "Hey, excuse me" at the 3-second mark. The system initiates decisions frame by frame according to the decision rhythm, appending the newly added audio item to the end of the sequence at each frame, and performing incremental forward computation on the model input based on a key-value cache with a fixed prefix. When processing the last audio word of the current frame, the system directly reads the logits at that position as the prediction score corresponding to that decision point, and then continues to append the next audio segment. The entire process does not generate or append any decision words, and the additional latency is approximately zero. Since the model directly uses speech as input, the serial delay of automatic speech recognition is eliminated, and the propagation path of recognition errors to the decision layer in high-noise backgrounds is cut off.

[0074] Step 108: Determine the target action from the preset action set based on the predicted score.

[0075] The preset action set is a pre-defined set of multiple selectable actions used to indicate the switching of the current talk wheel state. In the embodiments of this specification, the preset action set may correspond to the current talk wheel state, and may include actions such as continue talking, stop talking, continue listening, take over, quietly accept, and actively interrupt; each action is a word already existing in the unified decision word list, and may optionally occupy a single word unit.

[0076] The target action is an action selected from a preset action set based on a predicted score, used to indicate a change in the current turn state. In the embodiments of this specification, the target action is used to trigger a change in the turn state of the current broadcast turn, such as switching from broadcast state to listen state, or from listen state to broadcast state.

[0077] Based on the predicted score, the target action is determined from the preset action set. One possible method is to determine the target action from the preset action set based on the predicted score; another possible method is to select the action with the highest score from all actions corresponding to the predicted score as the target action; yet another possible method is to determine the target action from the preset action set based on the comparison result between the predicted score and the preset threshold; no limitation is made here.

[0078] For example, during the bot's announcement phase, if the user inserts a clear interruption message into the background, and the predicted score at the decision readout point is highest for "Stop Announcement," then the system determines the target action as stopping the announcement. During the user's statement phase, if the predicted score for "Continue Listening" is highest, then the target action is determined as continuing to listen. In this way, the model outputs a decision only after fully perceiving the audio information of the given moment, achieving end-to-end turn control.

[0079] Step 110: Based on the target action, perform a switch of the current talk wheel state.

[0080] The current turn state is the current speaking power state of the target dialogue, including the listening state and the speaking state. In this specification, turn state switching refers to the switching between the listening state and the speaking state of the current turn.

[0081] Based on the target action, the current talk wheel state is switched. One possible approach is to switch the current talk wheel state based on the target action; another possible approach is to switch the current talk wheel from one talk wheel state to another according to the target action instruction; yet another possible approach is to reorganize the fixed prefix of the target dialogue and update the key-value cache based on the already played text and historical audio corresponding to the switched talk wheel state after the state switch; no specific restrictions are imposed here.

[0082] For example, when the target action is to stop playback, the system switches the current playback cycle from playback mode to listening mode, immediately stops the current voice playback, and waits to receive the audio segment of the next playback cycle. When the target action is to continue playback, the system keeps the current playback cycle in playback mode and continues to generate the corresponding playback voice. When the target action is to take over, the system switches the current playback cycle from listening mode to playback mode and generates the corresponding voice response based on the target action for playback. When the conversation cycle state changes, the playback progress of this cycle and the part already spoken by the user are reduced to automatic speech recognition text at the conversation cycle boundary and enter a fixed prefix. The real-time audio buffer is cleared and reopened, and the fixed prefix is ​​reorganized and the key-value cache is updated for reuse in subsequent playback cycles.

[0083] In the embodiments of this specification, by pre-writing a fixed prefix of the target dialogue and reusing its key-value cache, when the audio segment of the current turn is received, only incremental forward computation needs to be performed on the model input constructed from the fixed prefix and the audio segment based on the cache. This avoids the repetitive computational cost caused by calculating all prefixes from scratch every time a beat is called, thereby effectively reducing the computational cost of high-frequency direct tuning of the multimodal model. By selecting the position of the last audio word of the current turn as the decision reading point to read the prediction score, the model outputs a decision only after fully perceiving the audio information of the beat, cutting off the transmission path of speech recognition serial delay and recognition errors in high-noise backgrounds to the decision layer in the cascaded architecture. Based on the prediction score, the target action is determined from the preset action set and the turn state is switched, realizing end-to-end turn control. While ensuring interaction continuity and response timeliness, it also takes into account computational efficiency and decision accuracy.

[0084] Optionally, before obtaining the fixed prefix of the target dialogue in step 102, the method further includes: Based on system prompts, historical dialogue text, and preset turn control flags, a fixed prefix is ​​constructed for the target dialogue; the fixed prefix is ​​then written to a key-value cache.

[0085] The system prompt is text injected at the beginning of the session to define the model's behavioral instructions. In this scheme, the system prompt is used to specify the output label system selection, interaction profile, and judgment criteria, and may optionally include historical context. In the embodiments of this specification, the system prompt constitutes the starting part of a fixed prefix area, which remains unchanged throughout the entire dialogue. For example, the system prompt includes "(i) output label system selection, (ii) interaction profile, (iii) judgment criteria", which can be combined and switched online without retraining.

[0086] The historical dialogue text is the text corresponding to the dialogue content that occurred before the current turn, such as the automatic speech recognition text of the previous turn and the text already played by the bot. In the embodiments of this specification, the historical dialogue text, system prompts, and preset turn control markers together constitute a fixed prefix area; the historical dialogue text has several selectable forms, such as: the automatic speech recognition text of the previous turn; or the bot's played text mixed with the user-side audio in sequence (this form does not undergo automatic speech recognition); or a short segment of user-side reference audio (only a short segment is taken for speaker gating, with a duration of about 1 second and an upper limit of about 1.5 seconds); all three can also be combined. Taking the automatic speech recognition text of the previous turn as an example, its content is the "wait a moment" said by the user in the previous turn and the opening text already played by the bot.

[0087] The preset turn control marker is a label or special mark used to distinguish control-related information of the dialogue system from the dialogue content. In the embodiments of this specification, the preset turn control marker is used to explicitly separate the "preceding background" (historical dialogue text) and the "real-time frame to be determined" (audio segment of the current turn) in the sequence, so that the two are structurally isolated. For example, a label text (such as "The following is background, not content to be determined") is inserted, and this marker is included in the preceding text area without loss. The preset turn control marker can also optionally be used to identify control information such as turn stage and playback progress.

[0088] For example, at the start of a conversation, the dialogue system first injects a system prompt, which includes the selection of an output tag system (such as selecting a unified decision vocabulary for six output actions), an interaction profile (such as a conservative profile of "better to be slow than to rush"), and a judgment criterion (such as judging the speaker first, then the semantic content). Then, it attaches the historical dialogue text, such as the user saying "wait a moment" in the previous round, or the opening text already played by the bot. Simultaneously, at the junction of the system prompt and the historical dialogue text, and between the historical dialogue text and the subsequent real-time audio, a preset turn control marker (such as "The following is background, not content to be judged") is inserted to explicitly separate the background from the real-time frame to be judged. The system prompt, the historical dialogue text, and the preset turn control marker combine to form a fixed prefix for the target dialogue. This fixed prefix area is pre-filled, and the key-value pairs are written to a key-value cache, which can then be directly retrieved and reused in subsequent steps.

[0089] In the embodiments described in this specification, a fixed prefix for the target dialogue is constructed based on system prompts, historical dialogue text, and preset turn control tags before obtaining the fixed prefix. This allows the fixed prefix area to uniformly carry behavioral instructions, historical context, and control tags, remaining unchanged throughout the entire dialogue. This enables the key-value cache to be directly obtained and reused during each decision, eliminating the need for repeated calculations and further reducing the computational cost of high-frequency calls. Simultaneously, the preset turn control tags explicitly separate the preceding background from the real-time frame to be determined in the sequence, helping the model avoid mistaking historical background content for the current content to be determined, thus balancing computational efficiency with decision accuracy.

[0090] Optionally, a fixed prefix for the target dialogue is constructed based on system prompts, historical dialogue text, and preset turn control markers, including: Use system prompts, historical dialogue texts, and preset turn control markers as a fixed prefix area; Encode the fixed prefix region to obtain the fixed prefix of the target dialogue; The fixed prefix area is a text region that serves as the starting part of the model input, composed of system prompts, historical dialogue text, and preset turn control markers. In the embodiments of this specification, the fixed prefix area forms the basis of the fixed prefix of the dialogue system, remains unchanged throughout the entire dialogue or window, and is used to reuse its key value cache across frames. Its corresponding historical context has several optional forms, such as the automatic speech recognition text of the historical turns, or the bot's broadcast text mixed with the user's audio in sequence, or a short segment of user-side reference audio (approximately 1 second in length, with a maximum of approximately 1.5 seconds). The three can also be combined. For example, the system prompt containing the output label system selection, interaction profile, and judgment criteria, the user's "Wait a moment" in the previous round, the opening text already broadcast by the bot, and the preset turn control markers used to explicitly separate the preceding background from the real-time frame to be judged (such as "The following is background, not content to be judged") are sequentially concatenated to form the fixed prefix area.

[0091] Key-value caching caches previously calculated key-value pairs or the results of shared prefix calculations, avoiding redundant calculations and reducing the cost and latency of high-frequency, long-context calls. In the embodiments of this specification, the key-value cache for fixed prefixes is reused as much as possible throughout the entire conversation. Subsequent decisions only need to calculate the newly added part based on this cache, and after the talk round state changes, it is reorganized and updated based on the changed talk round state. For example, after encoding the fixed prefix area, the resulting key-value pairs are written to the cache, and subsequent decisions directly reuse this cache without recalculating the entire prefix.

[0092] For example, taking a smart customer service call as an example: At the start of the conversation, firstly, the system prompts (including a unified decision vocabulary of six actions, a conservative "better slow than fast" approach, and a judgment standard of judging the speaker first and then the semantics), historical dialogue text (the user said "wait a moment" in the previous round, and the opening text already played by the bot), and preset turn control markers (such as "The following is background, not content to be judged") are used as a fixed prefix area. Next, this fixed prefix area is encoded (for example, through the embedding layer and self-attention layer of a multimodal model) to obtain the fixed prefix of the target dialogue (i.e., the encoded feature representation). Then, the key-value pairs of this fixed prefix are written to the key-value cache (i.e., a prefill is performed). Thereafter, whenever the audio segment of the current turn is received, the fixed prefix and the calculation result written to the key-value cache can be directly obtained, and incremental forward calculation is performed based on the cache, avoiding calculation from the beginning for each beat.

[0093] In the embodiments of this specification, the system prompts, historical dialogue text, and preset turn control markers are used as fixed prefix areas, and the fixed prefix areas are encoded to obtain the fixed prefix of the target dialogue. This ensures that the composition and encoding method of the fixed prefix are fixed and remain unchanged throughout the entire dialogue. In addition, by writing the fixed prefix into a key-value cache, when the audio segment of the current turn is received in a subsequent turn, it is not necessary to recalculate the fixed prefix. Instead, incremental forward calculation is performed based on the cached key-value pairs. This effectively reduces the computational cost of high-frequency direct tuning of the multimodal model, while ensuring the consistency and real-time performance of the decision.

[0094] Optionally, before step 104, the method further includes: Get the text that has been played and the reference audio. The text that has been played is the text that has been played in the current round, and the reference audio is the audio segment of the historical round used for speaker gating. Step 104 includes: By concatenating a fixed prefix, an audio segment, the previously played text, and a reference audio, the model input for the current playback round is obtained.

[0095] The already played text refers to the text content that the dialogue system has generated and played to the user during the playback phase. In the embodiments of this specification, the already played text is interspersed into the model input sequence according to the playback progress, enabling the model to perceive the already played content. Together with the current audio segment, it constitutes the input and is used to assist in the decision-making of the turn-by-turn state. For example, if the bot is playing "Are you available to communicate today?", this text is the already played text.

[0096] The reference audio is a short segment of audio data extracted from the user side, used for speaker gating judgment. In the embodiments of this specification, only a short segment of reference audio is taken, with a duration of approximately 1 second and an upper limit of approximately 1.5 seconds, to determine whether the current audio segment comes from the caller, is in the near field, or is spoken to the bot, in order to avoid accidental triggering by bystanders / noise. For example, the audio segment of the user saying "wait a moment" in the previous round, with a duration of approximately 1 second, can be used as reference audio.

[0097] For example, consider a typical intelligent customer service call: the bot is broadcasting the opening line, "Is today a convenient time to communicate?" (Phase A, broadcast phase). At this moment, the user inserts "Hello, may I interrupt?" in the background. The system obtains the broadcast text (i.e., "Is today a convenient time to communicate?") and the reference audio (i.e., the approximately 1-second audio clip from the user's previous "Wait a moment"). The system concatenates a fixed prefix, the current audio clip (the audio item corresponding to "Hello, may I interrupt?"), the broadcast text, and the reference audio to obtain the model input for the current broadcast round. Specifically, the broadcast text and the reference audio are inserted into the audio lexical sequence corresponding to the audio clip in chronological order, and then concatenated with the fixed prefix in a fixed arrangement order (e.g., bot text first, then audio) to obtain a single causal lexical sequence, which is used for subsequent incremental forward computation.

[0098] In the embodiments of this specification, the model input for the current turn is obtained by acquiring the already played text and reference audio, and concatenating a fixed prefix, audio segment, already played text and reference audio. This ensures that the model input includes not only the audio segment to be judged, but also the already played text and reference audio. As a result, the model can simultaneously perceive the already played content and the user's reference audio, which helps improve the accuracy of turn state switching. In particular, during the playback phase, it can make more reasonable decisions based on the already played content. At the same time, the reference audio helps the speaker to make gating judgments and avoid accidental triggering by bystanders / noise.

[0099] Optionally, the model input for the current playback round is obtained by concatenating a fixed prefix, an audio segment, the previously played text, and a reference audio, including: In chronological order, the broadcast text and reference audio are inserted into the audio word sequence corresponding to the audio segment; By splicing a fixed prefix, inserting the audio segment after the already broadcast text and reference audio, the model input for the current broadcast round is obtained according to a fixed arrangement order.

[0100] In chronological order, the broadcast text and reference audio are inserted into the audio word sequence corresponding to the audio segment. One possible method is to insert the broadcast text and reference audio into the corresponding time positions in the audio word sequence corresponding to the audio segment according to the generation timestamp of the broadcast text and the acquisition timestamp of the reference audio. Another possible method is to convert the broadcast text and reference audio into corresponding words and insert the words into the audio word sequence corresponding to the audio segment in chronological order. Yet another possible method is to alternately insert the broadcast text and reference audio into the audio word sequence corresponding to the audio segment in chronological order, so that the words in the sequence are arranged in chronological order. No limitation is made here.

[0101] Following a fixed arrangement order, the model input for the current playback round is obtained by concatenating a fixed prefix, inserting the already played text, and the audio segment after the reference audio. One possible approach is to use the fixed prefix as the starting part and the audio segment after the inserted already played text and reference audio as the subsequent part, concatenating them in a fixed arrangement order of first the fixed prefix, then the audio segment after the inserted already played text and reference audio. Another possible approach is to concatenate the fixed prefix, the already played text, the reference audio, and the audio segment sequentially into a single causal word sequence according to the fixed arrangement order, obtaining the model input for the current playback round. Yet another possible approach is to concatenate the fixed prefix and the audio segment after the inserted already played text and reference audio at the beginning and end according to the fixed arrangement order, obtaining the model input for the current playback round. No specific limitations are imposed here.

[0102] For example, consider a typical intelligent customer service call: the bot is broadcasting the opening line, "Is today a convenient time to communicate?" (Phase A, broadcast phase). At this moment, the user inserts "Hello, may I interrupt for a moment?" in the background. The system obtains the broadcast text (i.e., "Is today a convenient time to communicate?") and the reference audio (i.e., the approximately 1-second audio clip from the user's previous "Wait a moment"). When inserting the broadcast text and reference audio into the audio word sequence corresponding to the audio segment in chronological order, based on the timestamp, the reference audio (the previous "Wait a moment") occurred earliest, followed by the broadcast text ("Is today a convenient time to communicate?"), and the current audio segment ("Hello, may I interrupt for a moment") is the most recent. Therefore, the broadcast text and reference audio are inserted into the audio word sequence corresponding to the audio segment in chronological order, forming a sequence containing the reference audio word, the broadcast text word, and the current audio word. Then, following a fixed arrangement order (e.g., bot text first, then audio), the fixed prefix (containing system prompts, historical dialogue text, and preset turn control markers) is concatenated with the audio lexical sequence after inserting the already played text and reference audio. That is, the fixed prefix comes first, followed by the audio lexical sequence after inserting the already played text and reference audio, thus obtaining the model input for the current turn. This model input is a single causal lexical sequence, used for subsequent incremental forward computation based on the key-value cache of the fixed prefix.

[0103] In the embodiments of this specification, by inserting the broadcast text and reference audio into the audio word sequence corresponding to the audio segment in chronological order, and splicing the fixed prefix and the inserted audio segment in a fixed arrangement order, the model input sequence can accurately reflect the temporal sequence and structural relationship of the multimodal input, ensuring the consistency between training and online inference. This helps the model accurately read the predicted score at the decision read point based on the key-value cache with the fixed prefix in the incremental forward computation, thereby improving the decision accuracy of the talk turn state switching, while maintaining the reusability of the key-value cache and reducing the computational cost of high-frequency calls.

[0104] Optionally, step 106 includes: The multimodal model is invoked, and incremental forward computation is performed on the model input based on the key-value cache with a fixed prefix to obtain the predicted scores output by the multimodal model at each word position of the model input; under the constraints of the interactive personality and / or the current turn state, the predicted score corresponding to the decision read point is read at any decision read point.

[0105] Interaction personality refers to pre-defined parameters or instructions that characterize the interaction style preferences of a dialogue system. Examples include a conservative "better slow than aggressive" profile (preferring to minimize interruptions and prioritize user interaction), an aggressive "take-over" profile (preferring proactive intervention and timely interruption), or a "balanced" profile. In the embodiments described in this specification, interaction personality is used to apply a bias to the action direction score in the predicted score and / or determine the set of compliant actions corresponding to the current turn's state. This allows the same multimodal model to output different decisions under different interaction personalities, achieving configurability of interaction styles. Interaction personality can be specified by the interaction personality profile in the system prompts and can be switched online via preset switching instructions without retraining the model. For example, a debt collection scenario can be configured with a conservative "better slow than aggressive" profile, while a navigation scenario can be configured with a "timely response" profile.

[0106] The current turn state is the current speaking power state of the dialogue system, including the listening state and the broadcasting state. In the embodiments of this specification, the current turn state is used to determine the set of compliant actions (i.e., the set of actions allowed to be selected in the current state), and a mask is applied to the predicted scores of actions outside the compliant action set, thereby constraining the predicted scores at the decision read point to fall only within the range of compliant actions. For example, when the current turn state is broadcasting, the set of compliant actions includes the continue broadcasting action and the stop broadcasting action; when the turn state is listening, the set of compliant actions includes the continue listening action, the takeover action, the quiet acceptance action, and the active interruption action. The system determines the bias parameter β(persona) corresponding to the interaction personality through a preset switching instruction, and applies a logit bias to the predicted score during the decoding period, which satisfies the following personality level modulation formula: logit'_a = logit_a + β(persona) · bias_a,a∈{STOP, TAKE, WAIT, BC} STOP and TAKE correspond to the direction of interrupting / taking over, WAIT corresponds to the direction of yielding / waiting, and BC corresponds to the direction of accepting. The larger the β, the more inclined one is to interrupt / actively break the conversation; the smaller the β, the more inclined one is to listen / be conservative (e.g., conservative mode corresponds to β<1, and assertive mode corresponds to β>1). This bias is superimposed on the stage legal transition mask without changing the model weights, thus realizing the configurability of the interaction style.

[0107] For example, consider a typical intelligent customer service call: the system's interaction personality is configured as the conservative "better slow than interrupt" mode, and the current turn is in the broadcast state (the bot is broadcasting "Are you available to communicate today?"). The user inserts "Hello, may I interrupt?" into the background. When calling the multimodal model, under the constraints of the interaction personality and the current turn's state, the model input is first incrementally forward-computed based on a key-value cache with a fixed prefix. Specifically, the bias parameters corresponding to the interaction personality (positive bias for yielding direction and negative bias for interrupting direction) and the compliant action set (continue broadcasting, stop broadcasting) corresponding to the current turn's state (broadcasting state) are determined by preset switching instructions. Based on the bias parameters, a bias is applied to the predicted score of the action direction (e.g., a positive bias is applied to the "stop broadcasting" direction), and a mask (e.g., set to negative infinity) is applied to the predicted score of action directions outside the compliant action set (e.g., "take over", "actively interrupt", etc., in the listening state). Then, based on the predicted scores after applying bias and masking, the predicted score corresponding to any decision read point (i.e., the position of the last audio word in that beat) is read. Since the conservative mode applies a positive bias to the stop playback direction and the user's voice clearly indicates an intention to interrupt, the predicted score corresponding to "stop playback" at the decision read point is the highest, and the system determines the target action to be stop playback accordingly. In this way, the constraints of interaction style and turn state are encoded as bias and masking at the score level, achieving configurable interaction style and compliant action space constraints without changing the model weights or adding a new predictor.

[0108] In the embodiments described in this specification, by introducing constraints of interaction personality and / or the current turn state when invoking the multimodal model, a bias is applied to the score of action direction in the predicted score, and a mask is applied to the predicted score of action direction outside the compliant action set. This ensures that the predicted score read at the decision read point conforms to the current interaction style preference and falls within the compliant action range of the current turn state. In this way, the same multimodal model can switch interaction personality and output constraints online and composably via natural language commands, without needing to train a separate model for each set of outputs, or retraining or intrusively modifying the model structure. This ensures compliance in judgment while also taking into account flexibility and configurability.

[0109] Optionally, under the constraints of the interaction personality and the current turn state, at any decision point, the predicted score corresponding to that decision point is read, including: By pre-setting switching instructions, the bias parameters corresponding to the interactive personality and the set of compliant actions corresponding to the current voice wheel state are determined; At any decision readout point, obtain the prediction score output by the multimodal model; Based on the bias parameter, a bias is applied to the score of action direction in the predicted score, where action direction includes the direction of interruption, the direction of giving way, and the direction of acceptance. Based on the set of compliant actions, a mask is applied to the predicted scores of actions outside the set of compliant actions, and the predicted scores after applying the bias and mask are determined as the predicted scores corresponding to the decision read point.

[0110] The preset switching command is a command used to switch interaction personalities and / or output tag systems online, for example, injected in the form of natural language text or system prompts. In the embodiments of this specification, the preset switching command is used to dynamically determine the bias parameters corresponding to the interaction personality and the set of compliant actions corresponding to the current turn state during the conversation, without retraining the model weights, thus achieving configurable interaction style. For example, in a debt collection scenario, the preset switching command can be used to switch the interaction personality to the conservative "better slow than aggressive" mode.

[0111] The bias parameter corresponding to the interaction personality is a scalar or vector parameter determined based on the interaction personality, used to apply an offset to the scores of different action directions in the predicted score. In the embodiments of this specification, the bias parameter is used to apply a bias to the score of action direction in the predicted score, making actions in a specific direction easier or harder to select, thereby achieving differences in interaction style without changing the model weights. For example, the conservative mode corresponds to a positive bias for yielding and a negative bias for interrupting.

[0112] The set of compliant actions corresponding to the current speaking turn state is a set of actions that the model is allowed to choose in the current state, determined based on the current speaking turn state. In the embodiments of this specification, the set of compliant actions is used to constrain the decision space, applying a mask to the predicted scores of action directions outside the set of compliant actions, ensuring that the predicted scores read at the decision read point fall only within the range of compliant actions. For example, the set of compliant actions corresponding to the speaking state includes the action of continuing to speak and the action of stopping speaking.

[0113] In the predicted score, the action direction is grouped into categories corresponding to different interactive intentions, including the interruption direction, the yielding direction, and the acceptance direction. In the embodiments of this specification, the action direction is used to classify actions in a unified decision vocabulary in order to apply different biases and realize differentiated control of various actions by the interactive personality. For example, the six actions are divided into the interruption direction (taking over, actively interrupting), the yielding direction (stopping broadcasting), and the acceptance direction (continuing to broadcast, continuing to listen, and quietly accepting).

[0114] The "intercepting direction" corresponds to the action category of actively acquiring the right to speak from a listening state, including takeover actions and active interruption actions. In the embodiments of this specification, the intercepting direction is used to adjust the bias parameter under the constraint of the interaction personality. For example, in the conservative mode, the intercepting direction bias is negative to reduce the probability of accidental intercepting. For example, a user inserting interruptive speech when the bot is broadcasting corresponds to an intercepting direction.

[0115] The yielding direction corresponds to the action category of stopping speaking from the broadcasting state and relinquishing the right to speak, including the action of stopping broadcasting. In the embodiments of this specification, the yielding direction is used to adjust the bias parameter under the constraint of the interaction personality. For example, in the conservative mode, the yielding direction bias is positive to improve the yielding sensitivity. For example, if the bot detects a user's explicit intention to interrupt during broadcasting, the corresponding yielding direction is used.

[0116] The acceptance direction corresponds to the action category that maintains the current turn state and does not switch speaking rights, including the continue broadcast action, the continue listening action, and the quiet acceptance action. In the embodiments of this specification, the acceptance direction is used to maintain the current state under the constraints of the interaction personality, and its bias parameter can be set to zero or a fine-tuned value. For example, if the user only makes a humming echo during the bot's broadcast, it corresponds to the quiet acceptance in the acceptance direction.

[0117] The predicted scores for action directions are the probability distributions of each action direction on a unified decision vocabulary at the decision read point. In the embodiments of this specification, a bias is applied to the action direction scores, that is, a bias parameter is superimposed on the original predicted scores so that the predicted scores after applying the bias reflect the preferences of the interaction personality. For example, a positive value is superimposed on the logits of the stop broadcast direction.

[0118] By using preset switching instructions, the bias parameters corresponding to the interactive personality and the set of compliant actions corresponding to the current voice turn state are determined. One possible approach is to parse the interactive personality identifier in the preset switching instructions, query the preset bias parameter mapping table to obtain the bias parameters corresponding to the interactive personality, and then query the preset action set mapping table according to the current voice turn state to obtain the set of compliant actions. Another possible approach is to dynamically generate the bias parameters and set of compliant actions corresponding to the interactive personality based on the preset switching instructions. Yet another possible approach is to determine the bias parameters and set of compliant actions corresponding to the interactive personality based on the scene identifier carried in the preset switching instructions. No specific method is specified here.

[0119] Based on the bias parameter, a bias is applied to the scores of action directions in the predicted score. The action directions include the direction of interruption, the direction of giving way, and the direction of acceptance. One possible approach is to superimpose the bias values ​​of each action direction in the bias parameter onto the scores of the corresponding action directions in the predicted score. Another possible approach is to weight the scores of action directions in the predicted score according to the bias parameter. Yet another possible approach is to adjust the distribution of the scores of action directions in the predicted score based on the bias parameter. No specific method is specified here.

[0120] Based on the set of compliant actions, a mask is applied to the predicted scores of action directions outside the set of compliant actions. One possible approach is to set the predicted scores of action directions outside the set of compliant actions to negative infinity; another possible approach is to multiply the predicted scores of action directions outside the set of compliant actions by zero; yet another possible approach is to generate a mask vector based on the set of compliant actions and multiply the mask vector by the predicted scores; no specific restrictions are imposed here.

[0121] Based on the predicted scores after applying bias and masking, the predicted score corresponding to the decision read point is read. One possible approach is to obtain the predicted score after applying bias and masking at the decision read point as the predicted score corresponding to the decision read point. Another possible approach is to normalize the predicted scores after applying bias and masking and read the normalized probability distribution as the predicted score. Yet another possible approach is to select the score corresponding to the action with the highest score from the predicted scores after applying bias and masking. No specific method is specified here.

[0122] For example, consider a typical intelligent customer service call: the system's interaction personality is configured as the conservative "better slow than interrupt" mode, and the current turn is in the broadcast state (the bot is broadcasting "Are you available to communicate today?"). The user inserts "Hello, may I interrupt?" in the background. First, the system determines the corresponding bias parameters (+α for yielding direction, -β for interrupting direction, and 0 for accepting direction) and the compliant action set (continue broadcasting, stop broadcasting) corresponding to the current turn's state (broadcasting state) through a preset switching instruction (such as "interaction personality: better slow than interrupting" in the system prompt). Then, the multimodal model is invoked, and incremental forward calculation is performed on the model input based on a key-value cache with a fixed prefix. The original predicted score is read at the decision read point (i.e., the position of the last audio word in the beat). Next, a bias is applied to the action direction score in the predicted score based on the bias parameters: +α is added to the score for the stop broadcasting direction, and 0 is added to the score for the continue broadcasting direction. Simultaneously, based on the set of compliant actions, a mask is applied to the predicted scores of actions outside the compliant action set (takeover, active interruption, continued listening, and quiet acceptance), for example, setting it to negative infinity. Finally, based on the predicted scores after applying the bias and mask, the predicted score corresponding to the decision read point is read, where the score corresponding to "stop broadcasting" is the highest. Based on this, the system determines that the target action is to stop broadcasting and switches the current broadcast wheel from broadcasting state to listening state.

[0123] In the embodiments described in this specification, bias parameters and compliant action sets are determined by preset switching instructions. Based on the bias parameters, a bias is applied to the score of the action direction, and a mask is applied based on the compliant action set. This allows the constraints of interaction style and turn state to be encoded as operations at the score level. Without changing the model weights or adding a predictor, configurable interaction style and compliant action space constraints are achieved, improving the accuracy and flexibility of decision-making. At the same time, the reusability of key-value cache is maintained, and the computational cost of high-frequency calls is reduced.

[0124] Optionally, the method further includes: In response to the application scenario of switching target dialogue, update the interaction personality and / or output label, wherein the updated interaction personality and / or output label are used to constrain the prediction score output by the multimodal model at the decision read point; Based on the updated interactive personality and / or output labels, the multimodal model is invoked to perform the speech state switching for the current and subsequent speech rounds.

[0125] The application scenarios of a dialogue system refer to the service type or interactive environment it serves, such as intelligent debt collection, intelligent navigation, intelligent customer service, and intelligent question answering. In the embodiments of this specification, different application scenarios correspond to different interaction personalities and / or output label systems. By switching application scenarios, the interaction personalities and / or output labels can be dynamically updated, enabling the same multimodal model to adapt to different interaction needs without retraining the model weights. For example, a debt collection scenario corresponds to a conservative interaction personality of "better slow than fast," while a navigation scenario corresponds to an interaction personality of "timely response."

[0126] For example, consider a typical intelligent customer service call: the initial application scenario is debt collection, the system's interaction personality is configured as a conservative "better slow than aggressive" setting, and the output tag system is a unified decision vocabulary containing six actions (continue broadcasting, stop broadcasting, continue listening, take over, quietly accept, actively interrupt). During the call, if the system detects a need to switch the dialogue system's application scenario (e.g., the user switches from debt collection to service consultation, switching to a general customer service scenario), the system updates the interaction personality to a "balanced" setting in response to the switch. Subsequently, based on the updated interaction personality (balanced profile) and output labels, the multimodal model is invoked to execute subsequent turn-based state switching: upon receiving the audio segment of the current turn, a fixed prefix of the target dialogue is obtained (its key-value cache has been pre-written), the model input is constructed based on the fixed prefix, and the multimodal model is invoked to perform incremental forward computation on the model input based on the key-value cache of the fixed prefix. At the decision read point (i.e., the position of the last audio word of the beat), the predicted score is read, and a bias and mask are applied to the predicted score based on the bias parameters corresponding to the updated interaction personality and the compliant action set. The target action is determined from the preset action set, and subsequent turn-based state switching is executed. In this way, the same multimodal model can dynamically adapt to scene switching without retraining, outputting turn-based state switching decisions that conform to the interaction style of the new scene.

[0127] In the embodiments described in this specification, by responding to changes in the application scenario of the dialogue system, the interaction personality and / or output labels are updated, and the multimodal model is invoked based on the updated interaction personality and / or output labels to perform subsequent turn state switching. This allows the dialogue system to dynamically adapt to the interaction needs of various application scenarios without retraining the model or changing the model weights. In this way, the same set of model parameters can support online and combinable switching of multiple interaction styles and output label systems, improving the model's versatility and deployment efficiency, while maintaining the reusability of the key-value cache and reducing the computational cost of high-frequency calls.

[0128] Optionally, the method further includes: In response to a change in the state of the current talk turn, the fixed prefix of the target dialogue is reorganized based on the previously played text and historical audio corresponding to the new talk turn state, and the key-value cache is updated.

[0129] For example, taking a smart customer service call as an example: when the target action is to stop the broadcast, the system switches the current broadcast wheel from broadcasting state to listening state. In response to the switch in the current broadcast wheel's state, the system reorganizes the fixed prefix of the target dialogue based on the already broadcast text (i.e., the bot's "Are you available to communicate today?") and historical audio (i.e., the audio segment corresponding to the user's "Hello, may I interrupt?" inserted during the bot's broadcast) corresponding to the switched voice wheel state (listening state). Specifically, the current broadcast progress and the portion already spoken by the user at the wheel boundary are reduced to automatically recognized speech text and entered into the fixed prefix, while the real-time audio buffer is cleared and reopened. Subsequently, the system encodes the reorganized fixed prefix and updates the calculation result to the key-value cache for reuse in subsequent broadcast wheels. In the embodiments of this specification, in response to the switching of the current turn state, the fixed prefix of the target dialogue is reorganized based on the already played text and historical audio corresponding to the switched turn state, and the key-value cache is updated. This allows the fixed prefix to be dynamically updated as the dialogue progresses, ensuring that subsequent incremental forward calculations are based on accurate context. Thus, while maintaining the reusability of the key-value cache and reducing the computational cost of high-frequency calls, the coherence of turn state switching and decision accuracy in multi-turn dialogues are improved.

[0130] Optionally, step 108 includes: Based on the predicted score, the target action is determined from the preset action set of the current speaking turn state.

[0131] The preset action set for the current turn is a set of actions that the model is allowed to choose in the current turn, determined based on the current turn's state. In the embodiments of this specification, the preset action set corresponds to the current turn's state and is used to constrain the decision space, ensuring that the predicted score read at the decision read point falls only within the range of compliant actions, thus avoiding model output of actions incompatible with the current turn state. For example, when the turn state is in the broadcasting state, the preset action set includes the continue broadcasting action and the stop broadcasting action; when the turn state is in the listening state, the preset action set includes the continue listening action, the takeover action, the quiet acceptance action, and the active interruption action. Based on the predicted score, the target action is determined from the preset action set of the current turn's state. One possible approach is to filter actions belonging to the preset action set of the current turn's state from the predicted score and select the action with the highest score as the target action. Another possible approach is to apply a mask to the scores of actions whose directions do not belong to the preset action set in the predicted score, and determine the target action based on the predicted score after applying the mask. Yet another possible approach is to determine the target action from the preset action set based on the comparison between the scores of each action in the preset action set in the predicted score and a preset threshold. No specific method is specified here.

[0132] For example, consider a typical intelligent customer service call: the system's interaction mode is configured as a conservative "better slow than interrupt" mode, and the current turn is in the broadcasting state (the bot is broadcasting "Are you available to communicate today?"). The user inserts "Hello, may I interrupt?" into the background. The multimodal model is invoked to perform incremental forward computation on the model input based on a key-value cache with a fixed prefix, and the predicted score is read at the decision read point (i.e., the position of the last audio word in the current beat). Based on the current turn's state (broadcasting state), the corresponding preset action set is determined to be {continue broadcasting, stop broadcasting}. Then, based on the predicted score, the target action is determined from the preset action set of the current turn's state: actions belonging to this preset action set are filtered from the predicted scores, and the action with the highest score is selected as the target action. Since the conservative mode applies a positive bias to the stop broadcasting direction, and the user's voice clearly indicates an intention to interrupt, the score corresponding to "stop broadcasting" is the highest in the predicted scores. Based on this, the system determines the target action to stop broadcasting and switches the current turn from the broadcasting state to the listening state.

[0133] In the embodiments of this specification, the target action is determined from the preset action set of the current talk turn state based on the predicted score. This ensures that the determination of the target action is constrained to the range of compliant actions in the current talk turn state, avoiding the model output of actions that are incompatible with the current state. This improves the compliance and decision accuracy of talk turn state switching. At the same time, combined with the reuse of key-value cache with fixed prefixes, the low computing cost of high-frequency calls is maintained.

[0134] Optionally, the current talk wheel state includes a listening state and a talk state, and the preset action set for the current talk wheel state includes a preset listening state action set and a preset talk state action set; Based on the predicted score, the target action is determined from the preset action set of the current speaking turn state, including: If the current turn is in the listening state, the target action is determined from the preset listening state action set based on the predicted score. The preset listening state action set includes the continue listening action, take over action, quietly accept action, and actively interrupt action. If the current voice turn is in the voice turn state, then based on the predicted score, the target action is determined from the preset voice turn state action set, which includes the continue voice turn action and the stop voice turn action.

[0135] The listening state is the state in which the dialogue system is currently receiving user-input audio but has not actively generated a voice response. In the embodiments of this specification, the listening state corresponds to a preset set of listening state actions. In this state, the model continuously listens to the user's voice and can determine the target action based on the prediction score from actions such as continuing to listen, taking over, quietly accepting, and actively interrupting. For example, when the bot is listening to the user's statement, the system is in the listening state.

[0136] The broadcast state refers to the state in which the dialogue system is currently broadcasting a voice response to the user and has not actively received audio input from the user. In the embodiments of this specification, the broadcast state corresponds to a preset set of broadcast state actions. In this state, the model continuously broadcasts voice and can determine the target action from actions such as continuing broadcasting or stopping broadcasting based on the prediction score. For example, when the bot is broadcasting "Are you available to communicate today?", the system is in the broadcast state.

[0137] The preset listening state action set is a pre-defined set of actions that the multimodal model can choose in the listening state. In the embodiments of this specification, the preset listening state action set corresponds to the listening state and is used to constrain the decision space in the listening state, so that the prediction score read at the decision read point only falls within the range of compliant actions. For example, when the system is in the listening state, the preset listening state action set includes the continue listening action, takeover action, quietly accept action, and actively interrupt action. The preset broadcasting state action set is a pre-defined set of actions that the multimodal model can choose in the broadcasting state. In the embodiments of this specification, the preset broadcasting state action set corresponds to the broadcasting state and is used to constrain the decision space in the broadcasting state, so that the prediction score read at the decision read point only falls within the range of compliant actions. For example, when the system is in the broadcasting state, the preset broadcasting state action set includes the continue broadcasting action and the stop broadcasting action. The continue listening action is an action that indicates keeping the current broadcast wheel in the listening state and continuing to receive user input audio segments. In the embodiments described in this specification, the "continue listening" action is used to maintain the listening state while the user continues to speak and no system intervention is required, thus keeping the dialogue system in a listening state. For example, if the user normally states their needs during the bot's listening phase, the system determines the target action as "continue listening," maintains the listening state, and receives subsequent audio.

[0138] A takeover action is an action that instructs the current broadcast wheel to switch from listening mode to broadcast mode, and generates a corresponding voice response based on the target action for broadcasting. In the embodiments of this specification, the takeover action is used to proactively acquire the right to speak after the user finishes speaking or when a system response is required, stop receiving audio, and generate a response voice. For example, if the user pauses after finishing speaking, the system determines that the target action is takeover, switches to broadcast mode, and generates a voice response saying "How else can I help you?" for broadcasting.

[0139] A silent acknowledgment action is an action that instructs the bot to maintain the current playback cycle in a listening state and generate a corresponding silent acknowledgment voice for playback. In the embodiments of this specification, the silent acknowledgment action is used to provide a brief response without interrupting the user's speaking time, maintaining the continuity of the dialogue, and is a type of overlapping speech. For example, if the user pauses during a statement, and the bot emits an "um" as a silent acknowledgment, the system determines the target action as a silent acknowledgment, maintains the listening state, and generates the silent acknowledgment voice for playback.

[0140] An active interruption action is an action that instructs the current broadcast wheel to switch from listening mode to broadcast mode, stops receiving audio segments, and generates a corresponding interruption response voice for broadcast. In the embodiments of this specification, the active interruption action is used to proactively acquire the right to speak when urgent intervention is needed during a user's long statement, stops receiving user audio, and generates an interruption response. For example, if the user's statement deviates from the topic, the bot actively interrupts and guides the user back to the topic. The system determines the target action as an active interruption, switches to broadcast mode, and generates an interruption response voice for broadcast.

[0141] The "continue playing" action is an action that instructs the current playing wheel to remain in the playing state and generates corresponding continue playing voice based on the target action. In the embodiments of this specification, the continue playing action is used to maintain the current playing state when there is no need to switch the talking wheel; the system continues to maintain the playing state and generate subsequent voice. For example, if the bot is playing the opening remarks and the user has not inserted any voice, the system determines that the target action is to continue playing, maintains the playing state, and continues playing.

[0142] The "stop broadcasting" action is an action that instructs the system to switch the current broadcast cycle from broadcasting mode to listening mode, stop the current voice broadcasting, and wait to receive the audio segment of the next broadcast cycle. In the embodiments of this specification, the "stop broadcasting" action is used to relinquish the speaking right when a user inserts an interruption, and the system stops the current broadcasting and switches to listening mode. For example, if a user inserts "Hey, I'm interrupting" during bot broadcasting, the system determines the target action as "stop broadcasting," switches to listening mode, and stops the current broadcasting.

[0143] Based on the predicted scores, the target action is determined from a preset listening state action set. One possible approach is to filter actions belonging to the preset listening state action set from the predicted scores and select the action with the highest score as the target action. Another possible approach is to apply a mask to the scores of actions whose directions do not belong to the preset listening state action set based on the predicted scores, and determine the target action based on the predicted scores after applying the mask. Yet another possible approach is to determine the target action from the preset listening state action set based on the comparison results between the scores of each action in the preset listening state action set in the predicted scores and a preset threshold. No specific method is specified here.

[0144] Based on the predicted score, the target action is determined from the preset broadcast state action set. One possible method is to filter actions belonging to the preset broadcast state action set from the predicted scores and select the action with the highest score as the target action. Another possible method is to apply a mask to the scores of actions whose directions do not belong to the preset broadcast state action set based on the predicted scores, and determine the target action based on the predicted scores after applying the mask. Yet another possible method is to determine the target action from the preset broadcast state action set based on the comparison results of the scores of each action in the preset broadcast state action set with a preset threshold. No limitation is imposed here.

[0145] For example, consider a typical intelligent customer service call: the system's interaction style is configured as a conservative "better slow than aggressive" mode, and the current turn is in a listening state (e.g., the bot is listening to the user's statement). A multimodal model is invoked to perform incremental forward computation on the model input based on a key-value cache with a fixed prefix, and the predicted score is read at the decision read point (i.e., the position of the last audio word in the current beat). Based on the current turn's listening state, the target action is determined from the preset listening action set {continue listening, take over, quietly accept, actively interrupt}: if the user continues speaking, and the predicted score for "continue listening" is the highest, then the target action is determined to be "continue listening"; if the user finishes speaking and pauses, and the predicted score for "take over" is the highest, then the target action is determined to be "take over". When the current broadcast wheel is in broadcast mode (e.g., the bot is broadcasting "Is it convenient for you to communicate today?"), if the user inserts "Hello, I'd like to interrupt for a moment" in the background, based on the predicted score, the target action is determined from the preset broadcast action set {continue broadcasting, stop broadcasting}: Since the conservative mode applies a positive bias to the direction of stopping broadcasting, and the user's voice clearly shows an intention to interrupt, the score corresponding to "stop broadcasting" is the highest in the predicted score, the target action is determined to be stopping broadcasting, and the current broadcast wheel is switched from broadcasting mode to listening mode.

[0146] In the embodiments of this specification, if the current talk turn state is listening, the target action is determined from the preset listening action set; if it is talking, the target action is determined from the preset talking action set. This ensures that the determination of the target action is precisely constrained within the compliant action range of the current talk turn state, avoiding false triggering across states, improving the compliance and decision accuracy of talk turn state switching, and maintaining low computational cost for high-frequency calls by combining the key-value cache reuse with a fixed prefix.

[0147] Optionally, the preset action set is included in the unified decision vocabulary, which includes the actions of continuing to listen, taking over, quietly accepting, actively interrupting, continuing to broadcast, and stopping broadcasting. The multimodal model outputs the prediction scores for each action in a unified decision vocabulary at any decision readout point; The preset action set is a subset of actions in the unified decision word list that correspond to the current voice turn state.

[0148] A unified decision vocabulary is a pre-constructed set containing all available actions and their corresponding lexical units, used by multimodal models to output predicted scores at decision read points. In the embodiments of this specification, the unified decision vocabulary includes actions such as continue listening, take over, quietly accept, actively interrupt, continue broadcasting, and stop broadcasting. Each action is a word already existing in the unified decision vocabulary, optionally occupying a single lexical unit. For example, if the unified decision vocabulary contains six actions, the model outputs unnormalized scores on this vocabulary, and the probability distribution is obtained through softmax.

[0149] The preset action set is a subset of actions in the unified decision lexicon that corresponds to the current turn state. One possible approach is to query the preset state-action set mapping relationship based on the current turn state and select the corresponding action subset from the unified decision lexicon as the preset action set. Another possible approach is to generate a mask vector based on the current turn state to filter out the corresponding action subset from the unified decision lexicon as the preset action set. Yet another possible approach is to filter actions belonging to the current state from the unified decision lexicon based on the current turn state and use the filtered results as the preset action set. No specific limitations are imposed here.

[0150] For example, taking a smart customer service call as an example: the unified decision vocabulary configured by the system contains six actions (continue listening, take over, accept quietly, actively interrupt, continue broadcasting, stop broadcasting). When the current broadcast turn is in the listening state (e.g., the bot is listening to the user's statement), based on the current broadcast turn state, the corresponding action subset {continue listening, take over, accept quietly, actively interrupt} is selected from the unified decision vocabulary as the preset action set; when the current broadcast turn is in the broadcasting state (e.g., the bot is broadcasting "Are you available to communicate today?"), the corresponding action subset {continue broadcasting, stop broadcasting} is selected from the unified decision vocabulary as the preset action set. Subsequently, the multimodal model is invoked to perform incremental forward calculation on the model input based on the key-value cache with a fixed prefix, the predicted score is read at the decision read point, and based on the predicted score, the target action is determined from the preset action set of the current broadcast turn state. In the embodiments of this specification, by making the preset action set a subset of actions in the unified decision vocabulary that correspond to the current turn state, the preset action set can be dynamically determined according to the turn state. This ensures the constraint of the decision space and maintains the universality of the unified decision vocabulary, which facilitates the consistency of model training and online inference. At the same time, combined with the reuse of key-value cache with fixed prefixes, the computational cost of high-frequency calls is reduced.

[0151] Optionally, based on the target action, the current voice wheel state is switched, including: If the target action is a takeover action, the current broadcast wheel will be switched from listening mode to broadcast mode, and a corresponding voice response will be generated and broadcast based on the target action. If the target action is an active interruption action, the current broadcast wheel will be switched from listening mode to broadcast mode, the reception of audio segments will be stopped, and the corresponding interruption response voice will be generated and broadcast. If the target action is to stop broadcasting, the current broadcast wheel will be switched from broadcasting mode to listening mode, the current voice broadcasting will be stopped, and the system will wait to receive the audio segment of the next broadcast wheel.

[0152] For example, taking a smart customer service call as an example: if the target action is to continue listening, the system keeps the current playback wheel in a listening state and continues to receive audio segments input by the user. For instance, if the bot is listening to the user's stated needs and determines that the target action is to continue listening, the system continues to maintain a listening state and receives subsequent voice messages from the user.

[0153] For example, taking a smart customer service call as an example: when the target action is a takeover action, the system switches the current broadcast wheel from listening mode to broadcast mode, and generates a corresponding voice response based on the target action. For instance, if the user pauses after stating their statement, indicating that the target action is a takeover action, the system switches to broadcast mode and generates a voice response saying, "How else can I help you?"

[0154] For example, taking a smart customer service call as an example: when the target action is a silent acceptance action, the system keeps the current broadcast wheel in listening mode and generates the corresponding silent acceptance voice for broadcast. For instance, if the user pauses during the statement, confirming that the target action is a silent acceptance, the system remains in listening mode and simultaneously generates a silent acceptance voice of "um" for broadcast.

[0155] For example, taking a smart customer service call as an example: when the target action is an active interruption, the system switches the current playback wheel from listening mode to playback mode, stops receiving audio segments, and generates a corresponding interruption response voice for playback. For instance, if the user's statement deviates from the topic, determining that the target action is an active interruption, the system switches to playback mode, stops receiving audio, and generates an interruption response voice saying, "Excuse me, what is your question about?" for playback.

[0156] For example, taking a smart customer service call as an example: when the target action is to continue broadcasting, the system keeps the current broadcast wheel in broadcasting state and generates the corresponding continuation broadcast voice based on the target action. For example, if the bot is broadcasting "Are you available to communicate today?" and the user does not insert a voice message, the system determines that the target action is to continue broadcasting, keeps the broadcasting state, and continues to generate subsequent broadcast voice messages.

[0157] For example, taking a smart customer service call as an example: when the target action is to stop the broadcast, the system switches the current broadcast cycle from broadcast mode to listen mode, stops the current voice broadcast, and waits to receive the audio segment of the next broadcast cycle. For example, if the user inserts "Hello, I'd like to interrupt for a moment" during the bot's broadcast, and the target action is determined to be to stop the broadcast, the system switches to listen mode, stops the current broadcast, and waits to receive the user's subsequent audio.

[0158] In the embodiments of this specification, by executing corresponding turn state switching based on different target actions, the dialogue system can flexibly respond to various interactive intentions, realize precise turn control in full-duplex voice interaction, and while ensuring the continuity of interaction and the timeliness of response, effectively reduce the computing cost of high-frequency direct tuning of multimodal models by combining the key-value cache reuse with fixed prefixes.

[0159] Optionally, before step 104, the method further includes: Based on the audio segment, determine whether the model call gate is satisfied. The conditions for satisfying the model call gate include that the audio segment is valid speech. Step 104 includes: If so, then construct the model input for the current broadcast round based on the fixed prefix and audio segment.

[0160] Model invocation gating is a mechanism that determines whether to invoke a multimodal model to perform a turn-based state switch based on whether a preset condition is met for an audio segment. In the embodiments of this specification, model invocation gating is used to filter the currently received audio segments before constructing the model input. When an audio segment is determined to be valid speech (e.g., determined to be audible by speech activity detection, or determined by speaker gating to be from the caller and speaking to the bot), the subsequent steps of constructing model input based on a fixed prefix and audio segment and invoking the multimodal model are triggered. If an audio segment is determined to be silent, noise, or non-dialogue-related sound, the model invocation is skipped, and no model input is constructed, thereby avoiding unnecessary incremental forward computation on invalid audio. For example, based on speech activity detection, it is determined whether the acoustic energy of the audio segment exceeds a preset silence threshold; if it does, the model invocation gating is deemed to be satisfied. Based on the audio segment, to determine whether the model call gating is satisfied, one possible approach is to use speech activity detection to determine whether the audio segment contains valid speech; if so, the model call gating is satisfied. Another possible approach is to use speaker gating to determine whether the audio segment comes from the caller and is speaking to the bot, based on the audio segment and reference audio; if so, the model call gating is satisfied. Yet another possible approach is to use a comparison between the acoustic energy of the audio segment and a preset silence threshold to determine whether the model call gating is satisfied. No specific method is specified here.

[0161] For example, consider a typical intelligent customer service call: the bot is broadcasting the opening remarks (Phase A, the broadcasting phase), and the system receives the audio segments of the current broadcast round beat by beat according to the determined rhythm. Before constructing the model input, it first determines whether the audio segments meet the model call gating. Specifically, endpoint detection is performed on the audio segments through voice activity detection to determine whether the acoustic energy of the audio segments exceeds a preset silence threshold. If it exceeds the threshold, it is determined to contain valid speech and meets the model call gating. Subsequently, the model input for the current broadcast round is constructed based on a fixed prefix and the audio segment. If it does not exceed the preset silence threshold, it is determined to be a silent segment and does not meet the model call gating, so the model call is skipped. In this way, the model call gating filters silent or noisy segments out of the model inference process, avoiding invalid incremental forward computation and further reducing the computational cost of high-frequency calls to multimodal models.

[0162] In the embodiments of this specification, the model call gating is determined based on the audio segment. The model input is constructed and the multimodal model is called only when the gating condition is met. This enables the system to filter invalid audio segments and avoid unnecessary model inference for silence or noise. As a result, on the basis of reducing computing power cost by reusing fixed prefix key value cache, invalid calls are further reduced, which effectively reduces the computing power overhead of high-frequency direct modulation, while maintaining the real-time performance and accuracy of call state switching.

[0163] Optionally, after determining whether the model call gate is satisfied based on the audio segment, the following steps are also included: If not, continue caching the audio segment and return to the step of determining whether the model call gating is satisfied based on the audio segment.

[0164] Cached audio segments are audio data temporarily stored in memory or a buffer when an audio segment does not meet the model call gating. These cached audio segments are then merged with subsequently arriving audio segments for processing. In the embodiments described in this specification, cached audio segments do not trigger model calls during accumulation. When a newly arriving audio segment causes the accumulated cached audio segments to meet the model call gating, the cached audio segment is merged with the newly arrived audio segment to form the audio segment for the current broadcast round, used to construct the model input. For example, if a user is briefly silent for 200ms during a bot broadcast, this silent audio segment is accumulated and cached. When the user continues speaking, the cached audio is merged with the newly arrived audio into the model input sequence.

[0165] For example, consider a typical intelligent customer service call: the bot is broadcasting "Are you available to communicate today?" The user pauses briefly for about 200ms (without speaking). The system determines whether this audio segment meets the model call gating. Since the acoustic energy of this audio segment is below the preset silence threshold, it is determined that the model call gating is not met. The system accumulates this silent audio segment as a cached audio segment and returns to execute the step of determining whether the model call gating is met based on the audio segment. Subsequently, the user continues to say "Hello, I'd like to interrupt." The system again retrieves the current audio segment, combines it with the cached audio segment, and determines whether the accumulated audio data meets the model call gating (e.g., valid speech appears in the accumulated audio). If it does, the cached audio segment is merged with the current audio segment, and the model input for the current broadcast round is constructed based on a fixed prefix and the merged audio segment. In this way, the audio segment caching mechanism avoids the loss of user speech context due to brief silences, while maintaining the computational savings brought by gating filtering.

[0166] In the embodiments of this specification, by accumulating and caching audio segments when the model call gating is not satisfied and returning to re-evaluate, brief silences or non-speech segments will not cause context breakage. When subsequent valid speech arrives, the cached audio and the newly added audio can be merged for processing. This ensures the integrity of the model input and maintains the filtering effect of the model call gating on invalid calls. It reduces computing power costs while ensuring the decision continuity of the turn state switching.

[0167] Optionally, based on the audio segment, determine whether the model call gate is satisfied, including: Obtain the acoustic state corresponding to the audio segment, where the acoustic state includes a sound state and a silent state; Determine the current turn state of the target dialogue, where the current turn state includes the broadcast state and the listen state; Obtain a preset policy switch, wherein the policy switch is used to indicate whether to allow users to actively interrupt the user; Based on the acoustic state, the current turn state, and the policy switch, it is determined whether the model call gating is satisfied. The model call gating is satisfied when the acoustic state is in the sound state and the current turn state is in the broadcast state, the acoustic state is in the silent state and the current turn state is in the listening state, or the acoustic state is in the sound state and the current turn state is in the listening state and the policy switch indicates that active interruption of the user is allowed.

[0168] The acoustic state corresponding to an audio segment is the audio activity category determined based on the acoustic features of the audio segment (such as energy, zero-crossing rate, etc.). In the embodiments of this specification, the acoustic state is used to distinguish whether the current audio segment contains valid speech, serving as one of the criteria for model call gating, and includes two values: a spoken state and a silent state. For example, by performing speech activity detection on the audio segment of the current broadcast round, the acoustic state of the audio segment is determined to be either a spoken state (such as the user inserting "Hey, I'll interrupt for a moment") or a silent state (such as a brief silence of 200ms).

[0169] An audio state is defined as an audio segment whose acoustic energy exceeds a preset silence threshold and is therefore determined to contain valid speech. In the embodiments of this specification, an audio state indicates that the user may be speaking or attempting to interrupt the bot's broadcast, and is one of the conditions for satisfying the model's call gating. For example, if a user inserts "Hello, may I interrupt?" during the bot's broadcast of "Are you available to communicate today?", the acoustic energy of that audio segment exceeds the silence threshold and is therefore determined to be an audio state.

[0170] A silent state is defined as an audio segment whose acoustic energy is below a preset silence threshold and is therefore judged as silent or background noise. In the embodiments of this specification, a silent state indicates that there is currently no valid voice input, which is one of the conditions for satisfying the model call gating (when the current conversation round is in a listening state). For example, if a user is briefly silent for 200ms during a bot's broadcast, the acoustic energy of that audio segment is below the silence threshold and is judged as a silent state.

[0171] The preset policy switch is a pre-configured control parameter used to indicate whether the dialogue system allows users to actively interrupt the user while in a listening state. In the embodiments of this specification, the policy switch is used to control whether the active interruption function is enabled while in a listening state. When the policy switch indicates that active interruption of the user is allowed, the system will satisfy the model call gating if it detects an audio state while in a listening state; when the policy switch indicates that it is not allowed, it will only satisfy the gating if an audio state is detected while in a broadcast state. For example, in a debt collection scenario, the policy switch is configured to "not allow active interruption of the user" to avoid the bot accidentally interrupting the user's speech; in a navigation scenario, it is configured to "allow active interruption of the user" to enable timely intervention.

[0172] The model invocation gating condition is a combination of conditions determined by the acoustic state, the current turn-taking state, and the policy switch, used to decide whether to invoke the multimodal model. In the embodiments of this specification, the model invocation gating is satisfied in three ways: the acoustic state is active and the current turn-taking state is broadcasting (the user attempts to interrupt the bot); the acoustic state is silent and the current turn-taking state is listening (the user stops speaking, and the bot can take over); or the acoustic state is active and the current turn-taking state is listening, and the policy switch indicates that active interruption of the user is allowed (the bot actively interrupts the user). For example, if the bot is broadcasting an opening statement and the user inserts "Hey, I'd like to interrupt," the acoustic state is active and the current turn-taking state is broadcasting, satisfying the model invocation gating, and the multimodal model is invoked.

[0173] Based on the acoustic state, current turn state, and policy switch, it is determined whether the model call gating is satisfied. One possible approach is: if the acoustic state is active and the current turn state is broadcasting, the model call gating is satisfied; if the acoustic state is silent and the current turn state is listening, the model call gating is satisfied; if the acoustic state is active, the current turn state is listening, and the policy switch indicates that active interruption of the user is allowed, the model call gating is satisfied. Another possible approach is: query a preset call condition mapping table based on the acoustic state, current turn state, and policy switch to determine whether the model call gating is satisfied. Yet another possible approach is: generate the model call gating determination result based on the logical combination of the acoustic state, current turn state, and policy switch; no limitation is imposed here.

[0174] For example, consider a typical intelligent customer service call: the bot is broadcasting the opening line, "Are you available to communicate today?" (Phase A, broadcast phase). The system receives the audio segment of the current broadcast round step by step according to the determined rhythm. First, it obtains the acoustic state corresponding to the audio segment: it determines whether the acoustic energy of the audio segment exceeds a preset silence threshold through voice activity detection. If it does, the acoustic state is determined to be a voiced state (e.g., the user inserts "Hello, may I interrupt?"), otherwise it is a silent state. Next, it determines the current conversation state of the target dialogue: at this time, the bot is broadcasting, and the current conversation state is broadcast state. Then, it obtains the preset policy switch: for example, the system is configured to "allow active interruption of users". Finally, based on the acoustic state, the current turn state, and the policy switch, it is determined whether the model call gating is satisfied: Since the acoustic state is a voiced state and the current turn state is a broadcast state, it meets the condition of "acoustic state is a voiced state and the current turn state is a broadcast state," therefore, the model call gating is satisfied, and step 104 is then executed to construct the model input for the current broadcast turn based on the fixed prefix and the audio segment. If the acoustic state is a silent state and the current turn state is a listening state, the model call gating is also satisfied; if none of the three conditions are met, the model call is not satisfied, and the current model call is skipped. Through the above method, the model call gating is precisely triggered based on multi-dimensional conditions, avoiding invalid calls.

[0175] In the embodiments of this specification, by obtaining the acoustic state corresponding to the audio segment, determining the current turn state of the target dialogue, and obtaining the preset policy switch, and determining whether the model call gating is satisfied based on the three, the model call gating can comprehensively consider audio activity, system state, and interaction strategy, and accurately control whether to call the multimodal model in different scenarios. This further reduces unnecessary incremental forward computation, effectively reduces the computing power cost of high-frequency calls, and ensures the timeliness and accuracy of turn state switching.

[0176] Optionally, based on the predicted score, the target action is determined from a preset action set, including: Based on the judgment criteria, the speaker gating of the audio segment is first determined, and then the semantic content is judged to determine the target action; Among them, the speaker gating is used to determine whether the audio segment comes from the caller, is in the near field, or is spoken to the bot. If it is determined to be from a bystander, television, far-field echo, or environmental noise, the current broadcast state will be maintained. The semantic content is used to determine the user's intent. If it is determined to be a substantive request, a denial or question, a complete statement or question, a command to stop, or a substantive response to the bot's question, a yielding action will be triggered. If it is determined to be a mere tone of agreement, an isolated greeting, or something that is not clearly heard, the broadcast state will be maintained.

[0177] For example, taking a smart customer service call as an example: during the bot's broadcast phase, when the user inserts their voice into the background, the system determines the target action based on the prediction score. First, it uses speaker gating to determine whether the audio segment comes from the caller (if not, it directly maintains the broadcast state without triggering a stop broadcast). Then, it determines whether the semantic content contains a clear intention to interrupt (if so, it determines the target action to stop broadcast). This two-step determination method effectively avoids accidental triggering by bystanders / noise, improving the accuracy of turn-taking state switching.

[0178] Figure 2 A flowchart illustrating a dialogue control method according to an embodiment of this specification is shown, such as... Figure 2 As shown: This diagram uses an "intelligent customer service phone" scenario as an example to illustrate the overall process of the online inference phase, including steps 102 to 110. Specifically: In response to receiving the audio segment of the current broadcast round of the target dialogue (the beat is determined by marking points at 1 / 7 second per frame grid, approximately 142.9ms), a fixed prefix of the target dialogue is obtained (where the key-value cache of the fixed prefix has been pre-written); the model input for the current broadcast round is constructed based on the fixed prefix and the audio segment, and the model input includes text words and audio words; the multimodal model is invoked, and incremental forward computation is performed on the model input based on the key-value cache of the fixed prefix, and the prediction score corresponding to any decision read point (the position of the last audio word of the beat) is read; based on the prediction score, the target action is determined from a preset action set (such as HOLD, STOP, WAIT, TAKE, BC); and based on the target action, the talk state switch of the current broadcast round is executed. This process reflects a low-latency invocation method of "reading the last logits and not generating tokens".

[0179] Figure 3 This specification illustrates a schematic diagram of a dialogue timeline in a dialogue control method according to an embodiment, as shown below. Figure 3 As shown: This diagram, using the wall-clock as the horizontal axis and taking an "intelligent customer service phone" as an example, illustrates the complete dialogue timeline from the bot starting its broadcast (Phase A) to the user inserting voice (e.g., "Hello, I'm customer service, may I ask you something?") triggering the turnaround point (STOP, barge-in established, bot stops broadcasting), and then to the bot taking over after the user finishes speaking (entering Phase A'). The diagram uses a scale to represent the decision-making beat (frame grid 1 / 7 second ≈ 142.9ms) and marks key moments for turn-taking state switching: for example, the bot continuously outputs HOLD until it detects a clear interruption intention, then flips to STOP, stops broadcasting, and enters the listening state; during the user's statement phase, it continuously outputs WAIT, flipping to TAKE when semantic completeness is recognized and the pause meets the criteria, allowing the bot to take over speaking. This timeline intuitively reflects the temporal distribution of frame-level decisions in real-world interactions.

[0180] Figure 4 This diagram illustrates the interaction control delay in a dialogue control method according to an embodiment of this specification, as shown below. Figure 4 As shown: This diagram compares the interaction control latency of a traditional cascaded voice system with that of this solution. The cascaded Baseline adopts a serial architecture of VAD→ASR→rules→independent LLM, serially accumulating four latency segments (VAD endpoint stop judgment, ASR first-word silence threshold wait, rule judgment, and LLM generation decision-based token decoding), among which the VAD silence threshold wait is the largest segment of turn takeover latency. This solution uses a single full-modal large model (activation parameters >0.6B and ≤3B) to simultaneously complete interruption detection and stop judgment, eliminating ASR serial delay and error propagation, effectively reducing the computational cost and latency of high-frequency calls; after the audio arrives, only one incremental forward pass and reading of the calculated last logits are required, with the additional decoding latency being approximately zero.

[0181] Figure 5 This diagram illustrates a multimodal model updating interactive personalities and / or output labels in a dialogue control method according to an embodiment of this specification. Figure 5 As shown: This diagram illustrates a mechanism for updating interactive personalities and / or output labels based on the same underlying unified annotation (canonical fine-grained state, s_canon) and preset switching instructions. By applying a bias to the action direction score in the predicted score through the interactive personality bias parameter β (persona) (e.g., increasing the STOP / TAKE bias to make the model stop earlier and take over more proactively), online, combinable switching of interactive personalities (e.g., a conservative "better slow than aggressive" or an aggressive "intercepting" mode) and output label system (e.g., three sets of views) can be achieved without changing model weights or adding a new predictor. For example, when switching from a debt collection scenario to a general customer service scenario, the interactive personality is adjusted from a conservative mode (β<1) to one that allows moderate initiative (β>1), and the output label can be switched from a decision-making bot-action to a perceptual user-state. This switch is triggered by an instruction and requires no retraining.

[0182] Corresponding to the embodiments of the dialogue control method described above, Figure 6 A flowchart of a model training method provided in one embodiment of this specification is shown, including: Step 602: Obtain the sample fixed prefix, sample audio segments, and frame-level decision labels corresponding to the sample audio segments of the target dialogue.

[0183] The fixed prefix of a sample is the input data used to construct the starting part of the training sample during model training, based on system prompts, historical dialogue text, and preset turn control tags. In the embodiments of this specification, the fixed prefix of the sample corresponds to the fixed prefix during online inference, used to simulate the prefix area that remains unchanged throughout the entire dialogue. Its structure is consistent with online inference, including, for example, the selection of the output label system, the interaction profile, the judgment criteria, and the automatic speech recognition text of the previous round of dialogue. For example, in the training data, the fixed prefix of a sample is the system prompt (including a unified decision vocabulary of six actions and the "better slow than rush" conservative profile) combined with the historical dialogue text (the user said "wait a moment" in the previous round and the opening text already played by the bot).

[0184] The sample audio segments are user-side audio data used as training samples during model training. They are typically obtained by sampling a preset frame raster and segmenting it according to a preset window. In the embodiments of this specification, the sample audio segments correspond to the audio segments of the current broadcast round during online inference, used to simulate the real-time arriving audio stream. For example, an audio segment containing the user-inserted "Hey, let me interrupt" is generated through data synthesis, and its duration is aligned with the tag frame raster (1 / 7 second ≈ 142.9 milliseconds).

[0185] The frame-level decision label corresponding to the sample audio segment is a supervisory signal indicating the target action, marked on each labeled frame grid in the sample audio segment. In the embodiments of this specification, the frame-level decision label is generated by calculating a deterministic function. This function is determined based on multiple orthogonal objective axes (such as speaker, semantic content, silence duration, etc.) and the current effective determination rule, and is used to replace manual annotation. For example, for an audio segment corresponding to a user inserting "Hey, I'll interrupt for a moment," its frame-level decision label is marked as "stop broadcasting" when a clear interruption intention is detected in the frame.

[0186] Optionally, each training sample corresponds to a decision window, containing a fixed prefix of shared samples and a segment of real-time audio stream. The window length is approximately on the order of several seconds (typically 3–10 seconds). Due to the causal equivalence theorem, the training side can adopt a sample organization method of "one forward pass carrying all decisions in one window," and its sample size satisfies the following formula: N_sample = N_window (≈ N_turn), independent of decision density ρ Where N_sample is the number of training samples, and N_window is the number of windows, approximately equal to the number of turn N_turn. Assuming an audio frequency of approximately 7 tok / s and a lower limit of 1 / 7 second (approximately 142.9 milliseconds) for the decision beat interval, a window of approximately 5 seconds contains about 35 audio words and about 25 decision points, constituting only one sequence and one sample. However, it carries the loss of all 25 decisions in a single forward pass, thus avoiding the sample inflation problem of "one sample per decision point" and reducing training costs.

[0187] To obtain the sample fixed prefix of the target dialogue, one possible method is to construct the sample fixed prefix based on system prompts, historical dialogue text, and preset turn control flags; another possible method is to read the constructed sample fixed prefix from a preset training dataset; yet another possible method is to extract the corresponding system prompts and historical dialogue text from a preset dialogue script generated by a preset behavior recipe and construct the sample fixed prefix; no specific method is specified here.

[0188] To obtain sample audio segments and their corresponding frame-level decision labels, one possible approach is to generate a preset dialogue script based on a preset behavior recipe, perform audio rendering and synthesis training seeds, calculate frame-level labels using a deterministic function, and segment the audio segments and frame-level decision labels according to a preset window. Another possible approach is to acquire real dialogue recordings, generate frame-level decision labels through forced alignment, and segment the audio segments according to a preset window. Yet another possible approach is to generate sample audio segments in batches based on a data synthesis method, and calculate the decision labels corresponding to each frame based on a deterministic function determined by multiple orthogonal objective axes and the currently effective judgment rule. No specific limitations are imposed here.

[0189] For example, taking the training data of a smart customer service call as an example: First, a preset dialogue script is generated based on a preset behavior recipe. This script describes the scenario where the user inserts "Hello, let me interrupt" when the bot announces the opening line "Are you available to communicate today?". Then, audio rendering and synthesis are performed to generate training seeds, resulting in sample audio segments. Next, a frame-level decision label corresponding to the sample audio segment is calculated using a deterministic function (based on orthogonal objective axes such as speaker gating and semantic intent). The frame where the user inserts the interruption is marked as "stop broadcasting". Simultaneously, a fixed prefix for the sample is constructed: the system prompt (containing a unified decision vocabulary of six actions and a "better slow than rush" conservative option) is concatenated with the historical dialogue text (the text of the user saying "wait a moment" in the previous round and the opening line already broadcast by the bot). Finally, the fixed prefix for the sample, the sample audio segment, and the frame-level decision label are combined into a training sample.

[0190] Step 604: Construct the sample model input based on the sample fixed prefix and sample audio segments.

[0191] The sample model input is a single causal word sequence composed of a fixed sample prefix and a sample audio segment, used for forward computation of the multimodal model. In the embodiments of this specification, the sample model input corresponds to the model input of the current broadcast round during online inference, and may also include the already broadcast text and reference audio, used to simulate the multimodal input during online inference, such as the sequence obtained by concatenating the fixed sample prefix with the sample audio segment (the user inserts "Hello, may I interrupt?") and the already broadcast text ("Are you available to communicate today?").

[0192] Based on the sample fixed prefix and sample audio segments, the sample model input is constructed. One possible method is to concatenate the sample fixed prefix and sample audio segments to obtain the sample model input. Another possible method is to obtain the already played text and reference audio, and then concatenate the sample fixed prefix, sample audio segments, the already played text, and reference audio to obtain the sample model input. Yet another possible method is to insert the already played text and reference audio into the audio word sequence corresponding to the sample audio segments in chronological order, and then concatenate the sample fixed prefix and the sample audio segments after inserting the already played text and reference audio in a fixed arrangement order to obtain the sample model input. No particular limitation is made here.

[0193] For example, taking a training sample of an intelligent customer service call as an example: the sample model input is constructed based on a fixed prefix (including system prompts and historical dialogue text) and a sample audio segment (the user inserts "Hello, may I interrupt for a moment"). Specifically, the already played text ("Are you available to communicate today?") and reference audio (a 1-second audio segment from the previous round where the user said "Wait a moment") are obtained. The already played text and reference audio are inserted into the audio lexical sequence corresponding to the sample audio segment in chronological order, and then concatenated with the fixed prefix of the sample according to a fixed arrangement order (bot text first, then audio) to obtain the sample model input. This sample model input is a single causal lexical sequence, consistent with the model input structure during online inference.

[0194] Step 606: Invoke the multimodal model, perform forward computation on the sample model input based on the key-value cache of the sample fixed prefix, and read the prediction score corresponding to the decision read point at any decision read point.

[0195] The key-value cache for fixed sample prefixes is the result of encoding the fixed sample prefix region and caching the pre-computed key-value pairs. This cache is used to avoid redundant computation of the fixed prefix during training. In the embodiments of this specification, the key-value cache for fixed sample prefixes simulates the pre-filling process during online inference. Before the forward computation of each training sample begins, the fixed sample prefix region is encoded once and written to the key-value cache. Subsequently, computation is only performed on newly added sample audio segments, such as pre-filling the fixed sample prefix region containing system prompts and historical dialogue text, to obtain the key-value cache.

[0196] The process involves calling a multimodal model, performing forward computation on the sample model input based on a key-value cache with fixed sample prefixes, and reading the predicted score at any decision readout point. One possible approach is to call the multimodal model, perform forward computation on the sample model input based on a key-value cache with fixed sample prefixes, and read the predicted score at any decision readout point. Another possible approach is to perform forward propagation on the sample model input during the training phase and obtain the unnormalized score output by the multimodal model as the predicted score at the decision readout point. Yet another possible approach is to perform incremental forward computation only on newly added lexical units corresponding to the sample audio segments based on the key-value cache with fixed sample prefixes and read the predicted score at the decision readout point. No specific limitations are specified here.

[0197] For example, taking a training sample of an intelligent customer service call as an example: A multimodal model (a large full-modal model with activation parameters greater than 0.6B and no greater than 3B) is invoked. First, based on the key-value cache of the sample's fixed prefix (pre-filled with system prompts and historical dialogue text), forward computation is performed on the sample model input. Specifically, the word units of the sample audio segment (the user inserts "Hello, let me interrupt") are progressively input into the model. At any decision read point (i.e., the position of the last audio word unit of that beat, for example, the position corresponding to 1 / 7 second ≈ 142.9 milliseconds in the tag frame raster), the predicted score corresponding to that decision read point is read. This predicted score is the unnormalized score (logits) output by the model on a unified decision vocabulary (including continue broadcasting, stop broadcasting, continue listening, take over, quietly accept, actively interrupt).

[0198] Step 608: Based on the predicted scores, determine the training actions from the preset action set.

[0199] The training action is an action selected from a preset action set based on the predicted score during model training. It is used to compare the training action with the frame-level decision label to determine the decision loss. In the embodiments of this specification, the training action corresponds to the target action during online inference, and is obtained by selecting the action with the highest score from the preset action set. For example, based on the fact that "stop broadcasting" has the highest predicted score, the training action is determined to be "stop broadcasting".

[0200] Based on the predicted score, training actions are determined from a preset action set. One possible method is to determine the training actions from the preset action set based on the predicted score. Another possible method is to select the action with the highest score from all actions corresponding to the predicted score as the training action. Yet another possible method is to determine the training actions from the preset action set based on the comparison result between the predicted score and a preset threshold. No specific method is specified here.

[0201] For example, taking a training sample of an intelligent customer service call as an example: after reading the predicted score at the decision read point, the training action is determined from a preset action set (including continue broadcasting, stop broadcasting, continue listening, take over, quietly accept, and actively interrupt) based on the predicted score. Specifically, the action with the highest score is selected as the training action from each action corresponding to the predicted score. For example, if the predicted score for "stop broadcasting" is the highest, then the training action is determined to be "stop broadcasting".

[0202] Step 610: Determine the decision loss based on the training actions and frame-level decision labels.

[0203] The decision loss is a loss value used to measure the difference between the training action and the frame-level decision label, and is used to guide the parameter updates of the multimodal model. In the embodiments of this specification, the decision loss is calculated based on type weighting according to the frequency of occurrence of each action type in a unified decision vocabulary, and hard example modulation (focus loss) based on the prediction difference to reduce the weight of easily classified samples, in order to solve the class imbalance problem, such as assigning higher weights to rare but critical actions (such as active interruption).

[0204] The decision loss is determined based on the training actions and frame-level decision labels. One possible approach is to calculate the cross-entropy loss based on the training actions and frame-level decision labels, and then use type weighting based on the frequency of occurrence of each action type in the unified decision vocabulary, as well as hard example modulation to reduce the weight of easily classified samples based on prediction differences, to determine the decision loss. Another possible approach is to calculate the decision loss based on the difference between the prediction score corresponding to the training action and the frame-level decision label. Yet another possible approach is to compare the training actions and frame-level decision labels based on the decision loss function to determine the decision loss. No specific method is specified here.

[0205] For example, taking a training sample of an intelligent customer service call as an example: based on the training action (stop broadcasting) and frame-level decision labels (marked as "stop broadcasting" in the corresponding frame), the decision loss is determined. Specifically, cross-entropy loss is calculated, and type weighting (assigning weights based on the frequency of occurrence of each action in a unified decision vocabulary, for example, assigning a weight inversely proportional to the frequency to "stop broadcasting") and hard-example modulation (focus loss, reducing the weight of easily classified samples based on prediction differences) are introduced to obtain the final decision loss. Through this decision loss, the model can learn to output the "stop broadcasting" action when the user inserts an interruption speech.

[0206] Step 612: Based on the decision loss, train the multimodal model to obtain the trained multimodal model. The trained multimodal model is used to execute the above dialogue control method.

[0207] Based on decision loss, a multimodal model is trained to obtain a trained multimodal model. One possible approach is to update the parameters of the multimodal model using the backpropagation algorithm based on the decision loss. Another possible approach is to train the multimodal model using at least one of the following training paradigms: supervised fine-tuning, reinforcement learning, group relative policy optimization, or direct preference optimization, based on the decision loss. Yet another possible approach is to iteratively update the parameters of the multimodal model based on the decision loss until a preset convergence condition is met. No further limitations are specified here.

[0208] For example, taking a training sample from an intelligent customer service call: based on the decision loss, the parameters of the multimodal model (a large full-modal model with activation parameters greater than 0.6B and no greater than 3B) are updated using the backpropagation algorithm. During training, a supervised fine-tuning (SFT) paradigm can be adopted, iteratively training the model based on a large number of synthetic training samples (including various interaction scenarios such as debt collection and navigation) until the model converges, obtaining a trained multimodal model. This trained multimodal model can be deployed in a dialogue system for online turn-based state switching.

[0209] In this embodiment, the target dialogue sample fixed prefix, sample audio segments, and corresponding frame-level decision labels are obtained. A sample model input is constructed based on the sample fixed prefix and sample audio segments. The multimodal model performs forward computation on the sample model input based on the key-value cache of the sample fixed prefix and reads the prediction score at the decision read point. Training actions are determined from a preset action set based on the prediction score. A decision loss is determined based on the training actions and frame-level decision labels. The multimodal model is then trained based on the decision loss to obtain a trained multimodal model. This trained multimodal model is used to execute the aforementioned dialogue control method. Thus, by generating frame-level decision labels through data synthesis and deterministic functions, the problems of scarce real data and coarse labeling granularity are solved. The key-value caching mechanism avoids repeated computation of fixed prefixes during training, reducing training costs. Type weighting and hard example modulation alleviate the class imbalance problem, enabling the trained multimodal model to accurately execute end-to-end talk turn state switching, effectively reducing the computational cost of high-frequency online calls while ensuring interactive continuity. Optionally, obtain a sample fixed prefix of the target dialogue, including: Based on system prompts and historical dialogue text, a fixed prefix for the target dialogue is constructed.

[0210] For example, taking the training data of a smart customer service call as an example: based on the preset dialogue script generated by the preset behavior formula, system prompts (including a unified decision vocabulary of six actions and the "better to be slow than to rush" conservative mode) and historical dialogue text (the user said "wait a moment" in the previous round and the opening text already played by the bot) are extracted. Based on the system prompts and historical dialogue text, a sample fixed prefix of the dialogue system is constructed. Specifically, the system prompts and historical dialogue text are concatenated in sequence to form the sample fixed prefix. This sample fixed prefix corresponds to the fixed prefix during online inference and is used to simulate the prefix region that remains unchanged throughout the entire dialogue.

[0211] In the embodiments of this specification, a fixed prefix for the sample of the dialogue system is constructed based on system prompts and historical dialogue text, so that the training samples can cover behavioral instructions and historical context. This enables the multimodal model to learn the ability to perform incremental forward computation based on the fixed prefix during the training phase, providing a training basis for reusing key-value cache and reducing the cost of high-frequency call computing power during online inference.

[0212] Optionally, obtain the sample audio segment and the corresponding frame-level decision label, including: Based on preset behavior recipes, preset dialogue scripts are generated. The preset behavior recipes specify the range of values ​​for behavior families and each objective axis, and the preset dialogue scripts are used to schedule the time-level landing points of each event. Audio rendering is performed on the preset dialogue script to synthesize a training seed, which includes user-side recordings and time-level metadata. The deterministic function calculates frame-level labels for the training seeds and segments the user-side recordings according to a preset window to obtain the training sequence and the corresponding decision label track. The deterministic function is determined by multiple orthogonal objective axes and the currently effective judgment rule, and the features corresponding to the noise axis do not participate in the decision classification. The training sequence is validated by decision backfeeding, and the decision label track is validated for label correctness. If the validation passes, the training sequence is determined as the sample audio segment, and the decision label track is determined as the frame-level decision label.

[0213] A preset behavior formula is a pre-defined set of parameters used to define dialogue interaction behaviors, specifying the behavior family and the value range of each objective axis. In the embodiments of this specification, the preset behavior formula is used to generate preset dialogue scripts to synthesize training data in batches. For example, in a debt collection scenario, the preset behavior formula specifies the behavior family as "stop broadcasting", and specifies the speaker axis value range as "caller", the semantic content axis value range as "interruption intent", and the silence duration axis value range as "0-500ms".

[0214] A preset dialogue script is a dialogue script generated based on a preset behavior formula, which schedules the timing of each event. In the embodiments of this specification, the preset dialogue script is used to describe the time sequence of the bot's broadcast and the user's inserted voice, as the basis for audio rendering. For example, the script schedules the bot to start broadcasting "Are you available to communicate today?" at second 0, and the user to insert "Hello, may I interrupt for a moment?" at second 3.

[0215] A behavior family is a predefined set of action categories in dialogue interaction, corresponding to action groups in a unified decision vocabulary. In the embodiments of this specification, behavior families are used to specify the training target of the current sample in the preset behavior formula, such as behavior families including continue broadcasting, stop broadcasting, continue listening, take over, quietly accept, and actively interrupt.

[0216] The objective axis is an independent dimension used to describe the dialogue state, including the speaker, semantic content, silence duration, etc. In the embodiments of this specification, the objective axis serves as an input feature of a deterministic function to calculate frame-level decision labels. For example, the speaker axis indicates the audio source, the semantic content axis indicates the user intent, and the silence duration indicates the length of the silent segment.

[0217] An event is a specific interactive behavior that occurs during a dialogue, such as a user starting to speak, a bot starting to broadcast, or a user interrupting. In the embodiments of this specification, an event is a basic unit in a preset dialogue script; for example, a user inserting "Hey, I'd like to interrupt" is an event.

[0218] The time-level endpoint of an event refers to the specific moment the event occurs on the dialogue timeline. In the embodiments of this specification, a preset dialogue script schedules the time-level endpoints of each event for audio rendering and time-level metadata annotation. For example, the time-level endpoint of a user-inserted event is the 3rd second.

[0219] The training seed is the data obtained after audio rendering and synthesis of a preset dialogue script, including user-side recordings and time-level metadata. In the embodiments of this specification, the training seed is the basis for calculating frame-level labels and segmenting training sequences, such as synthesized audio containing the user saying "Hey, I'll interrupt for a moment" and metadata labeled with timestamps.

[0220] User-side recordings are audio data simulating user speech in the training seed. In the embodiments of this specification, user-side recordings correspond to the source of sample audio segments, such as audio recordings of users inserting phrases like "Hey, let me interrupt."

[0221] Time-level metadata consists of time-related information labeled in the training seed, including event type, timestamp, etc. In the embodiments of this specification, time-level metadata is used for deterministic function calculation of frame-level labels, for example, labeling a user insertion event as occurring at the 3rd second and lasting approximately 1 second.

[0222] The deterministic function is a function used to calculate frame-level decision labels, determined by multiple orthogonal objective axes and the current effective determination rule. In the embodiments of this specification, the deterministic function replaces manual annotation and outputs decision labels based on objective axis features, such as calculating the "stop broadcasting" label based on axes such as speaker and semantic content.

[0223] Multiple orthogonal objective axes are independent and non-overlapping dimensions describing the dialogue state, including speaker, semantic content, silence duration, etc. In the embodiments of this specification, multiple orthogonal objective axes serve as inputs to a deterministic function, such as the speaker axis, semantic content axis, and silence duration axis.

[0224] The currently effective determination rule is a set of rules that take effect during model training and are used to determine the target action. In the embodiments of this specification, the currently effective determination rule is used for deterministic function calculation. For example, the rule stipulates that "stop broadcasting" will be output when the user speaks and the semantics are not interrupted.

[0225] The noise axis is an objective axis that describes the noise characteristics in audio. In the embodiments of this specification, the features corresponding to the noise axis do not participate in decision classification, but are only used for verification, such as background noise type, energy, etc.

[0226] The features corresponding to the noise axis are the specific attributes described on the noise axis, such as noise type and signal-to-noise ratio. In the embodiments of this specification, the features corresponding to the noise axis do not participate in the decision classification, such as the acoustic features of background human voices.

[0227] Frame-level labels are the decision results for each tagged frame grid annotation. In the embodiments of this specification, frame-level labels are generated by calculating deterministic functions, for example, the frame corresponding to the user inserting interrupted speech is labeled "stop broadcasting".

[0228] The preset window is the time length for segmenting the user-side recordings to obtain the training sequence. In the embodiments of this specification, the preset window is aligned with the tag frame grid, for example, the window length is 1 second, containing approximately 7 frame grids.

[0229] The training sequence is an audio segment obtained by segmenting the user's recording according to a preset window. In the embodiments of this specification, the training sequence corresponds to a sample audio segment, such as an audio sequence containing the user's inserted "Hey, let me interrupt."

[0230] The decision label track corresponding to the training sequence is a frame-level label sequence arranged in time, corresponding to the training sequence. In the embodiments of this specification, the decision label track is used for training, for example, the "stop broadcasting" label sequence corresponding to each frame of the training sequence.

[0231] Decision backfeed verification is the process of verifying whether the noise characteristics of the training sequence match the noise audio waveform of the training seed. In the embodiments of this specification, decision backfeed verification is used to ensure the noise consistency of the training data, for example, to verify whether the synthesized noise is consistent with the noise in the user-side recording.

[0232] Tag correctness verification involves checking the timestamp offset and verifying whether the decision tag track is consistent with the currently effective judgment rule. In the embodiments of this specification, tag correctness verification is used to ensure tag accuracy, such as checking whether the tag timestamp is offset and whether the tag conforms to the judgment rule.

[0233] For example, taking the training data of a smart customer service call as an example: First, a preset dialogue script is generated based on a preset behavior recipe. This recipe specifies the behavior family as "stop broadcasting" and sets the speaker axis as "the caller" and the semantic content axis as "interruption intent". The script schedules the bot to broadcast "Are you available to communicate today?" at second 0, and the user to insert "Hello, I'd like to interrupt" at second 3. Next, the script is audio rendered to synthesize a training seed, which includes the user's recording (the audio of "Hello, I'd like to interrupt") and time-level metadata (the timestamp of the insertion event is marked as second 3). Then, a deterministic function (based on multiple orthogonal objective axes such as speaker and semantic content and the current effective judgment rule, with noise axis features not participating in the decision classification) is used to calculate frame-level labels for the training seed, and the user's recording is segmented according to a preset window (e.g., 1 second) to obtain the training sequence (including the audio segment containing the inserted speech) and the decision label track (the corresponding frame is marked as "stop broadcasting"). Finally, the training sequence is subjected to decision backfeed verification (verifying the matching of noise features with noise audio waveforms), and the decision label track is subjected to label correctness verification (verifying the consistency of timestamp offset and label with rules). After the verification is passed, the training sequence is determined as the sample audio segment, and the decision label track is determined as the frame-level decision label.

[0234] In the embodiments of this specification, a preset dialogue script is generated based on a preset behavior recipe, an audio rendering synthesis training seed is performed, and frame-level labels are calculated using a deterministic function. This achieves automated large-scale synthesis of training data, solving the problems of scarcity of real data and coarse labeling granularity. Through decision backfeed verification and label correctness verification, the quality of training data and the accuracy of labels are guaranteed, enabling the trained multimodal model to accurately perform end-to-end turn state switching.

[0235] Optionally, the training sequence is subjected to decision backfeed verification, and the decision label trajectory is subjected to label correctness verification, including: Verify whether the noise features of the training sequence match the noise audio waveform of the training seed to obtain the decision backfeeding verification result of the training sequence. The timestamp offset of the decision label track is verified, and the consistency between the decision label track and the current effective judgment rule is verified to obtain the label correctness verification result of the decision label track.

[0236] For example, taking the training data of a smart customer service call as an example: After obtaining the training sequence (including the audio segment of the user inserting "Hello, I'd like to interrupt") and the decision label track (the corresponding frame is marked as "Stop Broadcast"), the training sequence is subjected to decision backfeed verification, that is, verifying whether the noise features (such as background noise type and energy) in the training sequence match the noise audio waveform in the training seed to ensure the consistency of noise synthesis; the decision label track is subjected to label correctness verification, that is, verifying whether the label timestamp offset is within the allowable range (such as the offset not exceeding one frame grid), and verifying whether the decision label track (such as the "Stop Broadcast" label) is consistent with the currently effective judgment rule (such as outputting stop broadcast when the user inserts an interruption). If both verifications pass, it is determined that the training sequence and decision label track can be used for model training.

[0237] In the embodiments of this specification, by performing decision backfeed verification on the training sequence and label correctness verification on the decision label track, it is ensured that the noise characteristics of the training data are consistent with the original synthetic waveform, and that the label timestamps are accurate and conform to the judgment rules, thereby improving the reliability of the training data, avoiding interference from incorrect labels on model training, and further improving the accuracy of turn state switching of the trained multimodal model.

[0238] Optionally, before step 604, the method further includes: Get the text that has been played and the reference audio. The text that has been played is the text that has been played in the current round, and the reference audio is the audio segment of the historical round used for speaker gating. Step 604 includes: By concatenating the fixed prefix of the sample, the sample audio segment, the already played text, and the reference audio, the input of the sample model is obtained.

[0239] For example, taking a training sample of an intelligent customer service call as an example: the preset dialogue script generated based on the preset behavior recipe describes the scenario where the user inserts "Hello, let me interrupt" when the bot announces the opening line "Are you available to communicate today?". Before step 604, the already played text (i.e., the "Are you available to communicate today?" that the bot is currently playing) and the reference audio (i.e., the approximately 1-second audio clip where the user said "Please wait a moment" in the previous round) are obtained. When constructing the sample model input, the sample fixed prefix (containing system prompts and historical dialogue text), the sample audio clip (the audio item corresponding to the user inserting "Hello, let me interrupt"), the already played text, and the reference audio are concatenated to obtain the sample model input. Specifically, the already played text and the reference audio are inserted into the audio lexical sequence corresponding to the sample audio clip in chronological order, and then concatenated with the sample fixed prefix in a fixed arrangement order (e.g., bot text first, then audio) to obtain a single causal lexical sequence for subsequent forward computation.

[0240] In the embodiments of this specification, the sample model input is obtained by acquiring the already played text and reference audio, and concatenating the sample fixed prefix, sample audio segment, already played text, and reference audio. This ensures that the sample model input includes not only the sample audio segment to be judged, but also the already played text and reference audio. As a result, the model can simultaneously perceive the already played content and the user's reference audio during training, which helps improve the decision accuracy of turn-taking state switching. In particular, during the playback stage, it can make more reasonable decisions based on the already played content. At the same time, the reference audio helps the speaker gating judgment and avoids accidental triggering by bystanders / noise. This allows the trained multimodal model to adapt to complex full-duplex interaction scenarios, ensuring interaction continuity and response timeliness while taking into account computational efficiency and decision accuracy.

[0241] Optionally, the unified decision vocabulary includes decision words corresponding to each action in a preset action set, whereby the decision words represent actions in the preset action set; step 610 includes: Type-weighted processing is performed based on training actions and frame-level decision labels to determine the type-weighted loss. Type-weighting is to assign weights to each action type according to the frequency of occurrence of each action type in the unified decision vocabulary. The flip point weighting process is performed based on training actions and frame-level decision labels to determine the flip point weighting loss. The flip point weighting is to apply weights to the decision read points where the frame-level decision labels are flipped, and inject asymmetric weights according to the interaction profile. The weight coefficient of the earlier flip is greater than the weight coefficient of the later flip. Based on training actions and frame-level decision labels, tolerance soft label processing is performed to determine the tolerance soft label loss. The tolerance soft label is the conversion of hard labels into soft labels within the neighborhood at the flip time. Hard example modulation processing is performed based on training actions and frame-level decision labels to determine the hard example modulation loss. Hard example modulation is based on the prediction difference between training actions and frame-level decision labels, reducing the loss weight of easy-to-classify samples and increasing the loss weight of hard-to-classify samples. The decision loss is determined based on at least one of type-weighted loss, flip-point weighted loss, tolerance soft label loss, and hard-example modulation loss.

[0242] Type-weighted processing is a process of assigning weights to each action type according to the frequency of occurrence of each action type in a unified decision vocabulary, and weighting the differences between training actions and frame-level decision labels. In the embodiments of this specification, type-weighted processing is used to solve the class imbalance problem where the number of samples in each category is significantly different, so that rare but key actions occupy a reasonable proportion in the loss. For example, relatively high weight coefficients are assigned to actions with low occurrence frequency, such as "stop broadcasting", and relatively low weight coefficients are assigned to common actions, such as "continue listening".

[0243] Type-weighted loss is the loss value obtained after type-weighting processing. In the embodiments of this specification, type-weighted loss is used to measure the degree of difference between the training action and the frame-level decision label after considering the difference in action type frequency. For example, based on the frequency-assigned weights of each action in the unified decision vocabulary, the weighted cross-entropy loss between the training action (stop broadcasting) and the frame-level decision label (stop broadcasting) is calculated.

[0244] The flip-point weighting process involves assigning weights to decision read points where frame-level decision tags flip, and injecting asymmetric weights according to the interaction profile. In the embodiments of this specification, the flip-point weighting process is used to emphasize the critical moments of turn-by-turn state switching, making the model pay more attention to the boundary positions where tags flip from one action to another. For example, when a frame-level decision tag flips from "continue broadcasting" to "stop broadcasting," the flip-point decision read point is given a higher weight than the non-flip-point decision read point, and the weight coefficient for earlier flips is greater than the weight coefficient for later flips, in order to adapt to the conservative interaction profile of "better to be slow than to rush."

[0245] The flip-point weighted loss is the loss value obtained after flip-point weighting. In the embodiments of this specification, the flip-point weighted loss is used to measure the prediction accuracy of the model at the flip point of the turn state. For example, when the user inserts "Hello, I'm interrupting" during the bot's opening remarks, causing the label to flip to "Stop broadcasting", a weight higher than that of the silent segment continuing to broadcast frames is applied to calculate the loss.

[0246] Tolerance soft label processing is the process of converting hard labels into soft labels within the neighborhood at the flip point. In the embodiments of this specification, tolerance soft label processing is used to provide prediction tolerance near the flip point to avoid a sharp increase in loss due to small offsets in the label boundary. For example, it converts hard labels in the flip frame where the label is flipped from "continue broadcasting" to "stop broadcasting" and the grid (neighborhood range) one frame before and after it into soft labels, that is, to give the main label a smooth distribution with a higher probability and adjacent labels a lower probability.

[0247] The tolerance soft label loss is the loss value obtained after tolerance soft label processing. In the embodiments of this specification, the tolerance soft label loss is used to measure the degree of difference between the model's prediction and the soft label in the neighborhood of the flip point, such as calculating the cross-entropy or KL divergence loss between the model's predicted probability distribution and the soft label.

[0248] Hard example modulation processing is a process that reduces the loss weight of easily classified samples and increases the loss weight of hard-classified samples based on the prediction difference between training actions and frame-level decision labels. In the embodiments of this specification, hard example modulation processing corresponds to a focus loss mechanism, which is used to make training focus on hard-classified samples. For example, the loss weight of easily classified samples with high model prediction probability and consistent with frame-level decision labels (such as judging to continue listening in a silent segment) is reduced, while the loss weight of hard-classified samples with low prediction probability or incorrect prediction (such as judging to stop broadcasting in background noise) is increased.

[0249] The hard-example modulation loss is the loss value obtained after hard-example modulation processing. In the embodiments of this specification, the hard-example modulation loss is used to measure the model's prediction performance on hard-to-classify samples. For example, the modulation factor is calculated based on the prediction difference, and the modulation factor is multiplied by the cross-entropy loss to obtain the hard-example modulation loss.

[0250] Type-weighted loss is determined by performing type-weighted processing based on training actions and frame-level decision labels. One possible approach is to statistically analyze the historical frequency of each action type in the unified decision vocabulary, assign weight coefficients to each action type based on the inverse or inverse proportional function of the frequency, and use these weight coefficients to weight the difference between the training action and the frame-level decision label to determine the type-weighted loss. Another possible approach is to dynamically determine the weight coefficients based on the frequency of action types in preset behavior recipes, and calculate the type-weighted loss based on these weight coefficients. Yet another possible approach is to assign weights according to the frequency of each action type in the unified decision vocabulary, calculate the weighted cross-entropy loss between the predicted score corresponding to the training action and the frame-level decision label, and use this as the type-weighted loss. No specific method is specified here.

[0251] The flip point weighting process is performed based on training actions and frame-level decision labels to determine the flip point weighted loss. One possible approach is to identify the target decision read point in the frame-level decision labels where the state flips, apply a first weight to the target decision read point, and apply a second weight to the non-flip decision read points, with the first weight being greater than the second weight. The flip point weighted loss is then determined based on the first and second weights. Another possible approach is to determine asymmetric weights based on the interaction profile, making the weight coefficient of samples that flip earlier greater than that of samples that flip later, and then calculate the flip point weighted loss based on the sample weight coefficients. Yet another possible approach is to determine asymmetric weights based on the offset between the flip point position and the actual flip time, and then use the asymmetric weights to weight the cross-entropy loss to obtain the flip point weighted loss. No further limitations are specified here.

[0252] Tolerance soft label processing is performed based on training actions and frame-level decision labels to determine the tolerance soft label loss. One possible approach is to convert the hard labels in the frame-level decision labels into soft labels in the form of a probability distribution according to a preset conversion rule within the neighborhood of the flip time, and calculate the tolerance soft label loss based on the prediction scores corresponding to the soft labels and training actions. Another possible approach is to dynamically adjust the smoothness of the soft labels according to the neighborhood distance, with the smoothness decreasing as the distance to the flip time increases, and determine the tolerance soft label loss based on the dynamically smoothed soft labels. Yet another possible approach is to convert the hard labels in the neighborhood of the flip time into soft labels, calculate the KL divergence between the soft labels and the model's predicted probability distribution, and use this as the tolerance soft label loss. No further limitations are specified here.

[0253] Hard example modulation processing is performed based on training actions and frame-level decision labels to determine the hard example modulation loss. One possible approach is to calculate a modulation factor based on the difference between the predicted probability corresponding to the training action and the frame-level decision label, and use the modulation factor to reduce the loss contribution of easily classified samples with high predicted probabilities to obtain the hard example modulation loss. Another possible approach is to dynamically adjust the sample weights based on the prediction difference, so that the loss weight of hard-classified samples increases as the prediction difference increases, and calculate the hard example modulation loss based on the adjusted sample weights. Yet another possible approach is to calculate the cross-entropy loss between the training action and the frame-level decision label, and multiply the cross-entropy loss by the modulation factor determined based on the prediction probability to obtain the hard example modulation loss. No specific method is specified here.

[0254] A specific formula for calculating decision loss is as follows: L = Σ_{k=1}^{m} λ_{τ(y_k)} · w_k · (1-p_{k,y_k})^{γ} · CE(logits[a_k], _k) Wherein, λ_{τ(y_k)} is the type-weighted coefficient according to action type, used to solve the class imbalance problem (rare actions are assigned higher weights); w_k is the flip point weighted coefficient, which applies a greater weight to decision reads where the frame-level decision label flips (such as flipping from continue broadcasting to stop broadcasting) than to non-flip reads, and injects asymmetric weights according to the interaction profile (the weight coefficient for early flips is greater than the weight coefficient for late flips); (1-p_{k,y_k})^{γ} is the focus loss modulation term (γ is the modulation factor), used to reduce the loss weight of easily classified samples and increase the loss weight of difficult-to-classify samples. This focus loss modulation is an optional loss component and must be enabled after ablation confirmation; CE(logits[a_k], _k) represents the cross-entropy loss. _k represents the tolerance soft label, which means that hard labels are converted to soft labels within the neighborhood at the flip time (a smooth distribution with a higher probability for the main label and a lower probability for adjacent labels) to avoid a sharp increase in loss due to a small shift in the label boundary. The preset weight coefficients of each loss component are used to balance the contribution of different loss components to the total loss.

[0255] For example, taking a training sample of an intelligent customer service call: the bot is broadcasting the opening line "Are you available to communicate today?", and the user inserts "Hello, may I interrupt?" at the 3-second mark. Based on the prediction score, the training action is determined to be "stop broadcasting". The frame-level decision label marks the frame where the user inserts their voice as "stop broadcasting", and the previous frame is "continue broadcasting", that is, a flip has occurred from "continue broadcasting" to "stop broadcasting". First, type weighting is performed: "stop broadcasting" has a lower frequency in the unified decision vocabulary and is assigned a higher weight coefficient (e.g., 2.0), while "continue broadcasting" is assigned a lower weight coefficient (e.g., 0.5), and the type-weighted loss is calculated. Second, flip point weighting is performed: the decision read point where the flip occurs is identified, and a first weight (e.g., 3.0) is applied to the flipped read point, while a second weight (e.g., 1.0) is applied to non-flipped read points. Since the interaction characteristic is "better slow than fast" conservative, the weight coefficient of early flip (predicting stop broadcasting before the flip time) is greater than the weight coefficient of late flip, and the flip point weighted loss is calculated. Next, tolerance soft label processing is performed: within the neighborhood of the flipped frame (the flipped frame and one frame before and after it), hard labels are converted to soft labels. For example, the soft label of the flipped frame is "stop broadcasting" with a probability of 0.8 and "continue broadcasting" with a probability of 0.2. The tolerance soft label loss is then calculated. Then, hard example modulation processing is performed: based on the difference between the predicted probability and the label, a modulation factor is calculated. The loss weight is reduced for easily classified silent samples and increased for hard-to-classify samples. The hard example modulation loss is then calculated. Finally, based on at least one of the type-weighted loss, flip-point-weighted loss, tolerance soft label loss, and hard example modulation loss, a weighted sum is calculated according to preset weight coefficients to determine the decision loss, which guides the parameter updates of the multimodal model.

[0256] In the embodiments of this specification, the decision loss is determined from four dimensions: class balance, flip point emphasis, boundary tolerance, and hard example focusing, based on training actions and frame-level decision labels, through type weighting, flip point weighting, tolerance soft labeling, and hard example modulation. This enables the trained multimodal model to more accurately capture the key moments of turn-taking, adapt to different interaction characteristics, and maintain robustness in class imbalance and label noise, thereby effectively improving the accuracy and timeliness of turn-taking in online full-duplex voice interaction.

[0257] Optionally, the sample model input does not contain decision words, which belong to a unified decision vocabulary and are used to represent actions in a preset action set; Step 606 includes: The multimodal model is invoked, and forward computation is performed on the sample model input based on the key-value cache of the sample fixed prefix. At any decision read point, the prediction score corresponding to the decision read point is read. The decision word does not occupy the word position in the sample model input. The word position in the sample model input is obtained by arranging the sample fixed prefix and the sample audio segment in sequence. The decision read point is the position of the last audio word of the beat.

[0258] Decision words are dedicated lexical units in a unified decision lexicon used to represent actions within a preset action set. In the embodiments of this specification, a decision word corresponds to a specific interactive control action. The model outputs the prediction score corresponding to the decision word at the decision readout point, but the sample model input does not contain decision words; that is, no decision words representing actions are inserted into the input sequence to avoid decision words participating in attention calculation. For example, the lexical unit corresponding to "stop broadcasting" in the unified decision lexicon is a decision word.

[0259] Since the sample model input does not contain decision words, one possible approach is to remove all decision words corresponding to the preset action set from the word sequence of the sample model input when constructing the sample model input, so that the sample model input only contains audio words and text words; no restrictions are imposed here.

[0260] Under the parallel label track paradigm (where decision words do not enter the input stream), the training reads and online reads coincide unambiguously, and their decision loss satisfies the off-by-one alignment formula: L = Σ_{k=1}^{m} w_k · CE(logits[a_k], y_k), a_k = the index of the last audio token of this beat, y_k ∈ {continue broadcasting, stop broadcasting, continue listening, take over, quietly take over, actively interrupt} Where a_k is the decision read position (the position of the last audio word in that beat), logits[a_k] is the unnormalized score at that position, y_k is the frame-level decision label, and w_k is the optional sample weight. This ensures that the training read position a_k is completely consistent with the online read position a_k, eliminating off-by-one ambiguity and guaranteeing position-by-position alignment between training and online data.

[0261] For example, taking a training sample of an intelligent customer service call as an example: the fixed prefix of the sample includes system prompts (containing a unified decision vocabulary of six actions) and historical dialogue text, and the sample audio segment corresponds to the user-inserted "Hello, I'd like to interrupt for a moment". When constructing the sample model input, the sample model input does not contain decision words, that is, no action words such as "stop broadcasting" or "continue broadcasting" are inserted. The multimodal model is called, and forward computation is performed on the sample model input without decision words based on the key-value cache of the sample fixed prefix. At any decision read point (i.e., the position of the last audio word of the beat), the predicted score at that decision read point is read.

[0262] In the embodiments of this specification, by not including decision words in the sample model input, the decision words do not participate in the attention calculation of the model input stream, thus avoiding the shortcut of the model learning to copy the label of the previous step, thereby effectively preventing the problem of lag at the flip point, while ensuring the integrity of the key-value cache and preventing the cache from becoming invalid due to the insertion of decision words, thereby improving the accuracy and real-time performance of the talk turn state switching decision.

[0263] Optionally, the sample model input includes historical decision words, which are the decision words corresponding to the target action determined by the previous decision reading point. The historical decision words are inserted into the sample model input and participate in the attention calculation of subsequent positions as part of the model input stream. Step 606 includes: The multimodal model is invoked, and forward computation is performed on the sample model input containing historical decision words based on the key-value cache with fixed sample prefixes. At any decision read point, the prediction score corresponding to that decision read point is read.

[0264] The historical decision word is the decision word corresponding to the target action determined at the previous decision read point. This historical decision word is inserted into the sample model input and participates in the attention calculation of subsequent positions as part of the model input stream. In the embodiments of this specification, the historical decision word is used to feed back the decision result of the previous step to the model. However, because it participates in the attention calculation, it is easy for the model to form a shortcut of copying the label of the previous step, resulting in a lag in the flip point. For example, if the target action determined at the previous decision read point is "continue broadcasting", its corresponding decision word is inserted into the current sample model input.

[0265] Based on the fact that the sample model input contains historical decision words, one possible approach is to convert the decision word corresponding to the target action determined by the previous decision reading point into a word and insert it after the fixed prefix in the sample model input or before the audio segment, so that the historical decision word participates in the attention calculation of subsequent positions as part of the model input stream; no limitation is made here.

[0266] For example, taking a training sample of an intelligent customer service call as an example: Assume the target action determined by the model at the previous decision read point is "continue broadcasting," and its corresponding decision word is inserted into the current sample model input. The current sample model input includes the sample fixed prefix, the historical decision word (i.e., the lexical corresponding to "continue broadcasting"), and the sample audio segment (the user inserts "Hello, I'd like to interrupt"). The multimodal model is invoked, and forward computation is performed on the sample model input containing the historical decision word based on the key-value cache of the sample fixed prefix. At any decision read point, the predicted score at that decision read point is read.

[0267] In the embodiments described in this specification, while including historical decision words in the sample model input and participating in attention calculation can provide the model with the decision context of the previous step, it can easily lead the model to learn a shortcut of copying the label of the previous step, resulting in a lag in the flip point when the talk state needs to be switched. Therefore, as a preferred solution, the sample model input does not contain decision words to avoid this defect and improve the sensitivity and accuracy of decision-making.

[0268] Optionally, before constructing the sample model input based on the sample fixed prefix and sample audio segments, the following steps are also included: Obtain training instructions, which include output label system instructions and interactive personality instructions. The training instructions serve as system prompts in the fixed prefix of the samples and are used to constrain the output of the multimodal model. Obtain the underlying unified annotation, which includes fine-grained state annotations for each frame corresponding to the sample audio segment, and project the underlying unified annotations into the corresponding view labels based on the training instructions. The decision loss is determined based on the training actions and frame-level decision labels, including: determining the decision loss based on view labels and frame-level decision labels; After obtaining the trained multimodal model, the process also includes: obtaining evaluation instructions (the wording of which is not included in the training instructions), and based on the evaluation instructions, determining the accuracy of the output label system selection and the compliance of the interaction characteristics of the trained multimodal model.

[0269] Training instructions are instructions injected in the form of natural language text during model training to specify model output constraints and interaction styles. In the embodiments of this specification, training instructions include output label system instructions and interaction personality instructions (corresponding to the acquisition training instructions, which include output label system instructions and interaction personality instructions). This enables instructions to switch between output label systems and dialogue personalities, allowing the same multimodal model to learn multiple output label systems and interaction personalities through different training instructions, without needing to train a separate model for each set of outputs. For example, when training a sample of an intelligent customer service call, the training instructions are "Output label system: six actions; Interaction personality: slow and steady wins the race".

[0270] The output label system instruction is a sub-instruction in the training instructions used to specify the set of actions and a unified decision vocabulary output by the model at the decision readout point. In the embodiments of this specification, the output label system instruction is used together with the interaction personality instruction to constitute the training instructions, realizing the configurability of the output label system. For example, the instruction can specify one of three sets of views: view Figure 1 For user-side state prediction, see Figure 2 For AI-side status, see Figure 3 For AI state change decisions (outputting one of six action words, namely continue listening, take over, quietly accept, actively interrupt, continue broadcasting, and stop broadcasting), each view is obtained by a deterministic projection of the underlying unified annotation, and cross-view behavior is naturally self-consistent.

[0271] Interaction personality instructions are sub-instructions in the training instructions used to specify the model's interaction style preferences. In the embodiments of this specification, interaction personality instructions are used together with output label system instructions to constitute training instructions (corresponding to the projection of the underlying unified annotations onto the corresponding view labels based on the training instructions), enabling the model to learn decision biases under different interaction personalities during training, such as the conservative "better slow than fast" mode and the "timely response" mode, so that the model can adjust the bias of action direction in the predicted score according to the interaction personality instructions during inference.

[0272] The underlying unified annotation is a frame-level annotation generated from the training data based on a deterministic function, containing all action categories. In the embodiments of this specification, the underlying unified annotation corresponds to the frame-level decision label (which determines the decision loss based on the view label and the frame-level decision label). It is a fine-grained, unified raw supervision signal. For example, when a user inserts an audio clip saying "Hey, I'll interrupt for a moment," the underlying unified annotation marks the frame where a clear interruption intention is detected as "stop broadcasting."

[0273] View labels are frame-level labels obtained by projecting the underlying unified annotations based on training instructions, conforming to the requirements of the current instructions. In the embodiments of this specification, view labels are obtained by mapping the underlying unified annotations to a specified subset of actions according to the output label system instructions (corresponding to separating training instructions from evaluation instructions, and evaluating the accuracy of output label system selection and the degree of interaction character compliance on the wording of instructions not covered during training), and are used to calculate decision loss. For example, when the output label system instructions specify six actions, the view label is consistent with the underlying unified annotations, that is, "stop broadcasting".

[0274] Based on the training instructions, the underlying unified annotation is projected into the corresponding view labels. One possible approach is to parse the output label system instructions in the training instructions, determine the target action subset, and map the actions of each frame in the underlying unified annotation to the corresponding actions in the target action subset to obtain the view labels; however, no specific limitations are imposed here.

[0275] The underlying unified annotation corresponds to a set of fine-grained canonical states that are independent of specific output labeling systems. All external output labeling system (schema) views are deterministic projections of this state, satisfying the following formula: s_view = Π_view(s_canon), view∈{user-side state, AI-side state, AI state change decision} Here, `s_canon` represents the underlying unified canonical state, and `Π_view` is a deterministic projection function that maps the canonical state to a specified view label. For example, when the output label system instruction specifies "AI state change decision (six action words)," the view label is consistent with the underlying unified annotation; when other views are specified, the corresponding action subset is obtained through projection. Since all views are derived from the same `s_canon`, cross-view behavior is naturally consistent at any given time. The function of the instruction is to select which view to project to, thus enabling configurable switching of the output label system.

[0276] For example, taking a training sample of an intelligent customer service call as an example: Training instructions are obtained, including output label system instructions (such as "output a unified decision vocabulary of six actions") and interaction personality instructions (such as "better to be slow than to interrupt"). The underlying unified annotation (i.e., frame-level decision labels) is obtained by calculating a deterministic function on the training seed; for example, the frame where the user inserts "Hello, I'll interrupt for a moment" is labeled "stop broadcasting". Based on the training instructions, the underlying unified annotation is projected onto the corresponding view label: since the output label system instructions specify six actions, the view label is consistent with the underlying unified annotation, i.e., "stop broadcasting". Based on the view label and the frame-level decision label (which are the same at this point), the decision loss is determined (e.g., calculating type-weighted loss, flip-point-weighted loss, etc.). During the evaluation phase, the training instructions and evaluation instructions are separated. For example, the interaction personality instruction used during training is "better to be slow than to interrupt," while the evaluation uses an uncovered instruction phrase such as "preferring to interrupt users less." The evaluation model's output label system selection accuracy and interaction personality compliance under this instruction are evaluated.

[0277] In the embodiments of this specification, by acquiring training instructions including output label system instructions and interaction personality instructions, the underlying unified annotation is projected onto the corresponding view labels based on the training instructions, and the decision loss is determined based on the view labels and frame-level decision labels. This allows the same multimodal model to learn multiple output label systems and interaction personalities through different training instructions, realizing a switchable output label system and dialogue personality, and solving the problem in existing solutions where the output label system is fixed during training and cannot be switched online by natural language instructions. By separating training instructions from evaluation instructions, the accuracy of output label system selection and interaction personality compliance are evaluated on instruction wording not covered during training, verifying the model's generalization ability and zero-sample transfer ability to instruction wording. It can adapt to diverse online deployment needs without retraining or invasively modifying the model structure.

[0278] Alternatively, the above-mentioned dialogue control method and model training method can also adopt the following equivalent implementation methods: (1) Inference reading alternative: The category is directly output by reading the hidden state of the last layer using an external classification head, without decoding the decision word, so as to reduce the latency of a single inference; (2) Reading point density replacement: Densely mark each audio word in the real-time area to eliminate the fragility of frame grid alignment; (3) Cross-turn context management alternative: The prefix cache is reused throughout the process by only appending and not reverting the position, and is reorganized periodically, or only reorganized at the turn boundary (reducing the real-time audio to ASR text and reconstructing the prefix) to adapt to different deployment needs.

[0279] Figure 7 This diagram illustrates the synthesis and multi-system mapping of frame-level decision tags in a dialogue control method according to an embodiment of this specification. Figure 7 As shown: This diagram illustrates how, based on a small number of real-world seed dialogues and rare behaviors (such as targeted injection interruption and echoing), refined underlying raw state labels are obtained through procedural synthesis and automatic post-processing (the synthesis itself is the label, eliminating the need for manual frame-by-frame annotation). It also demonstrates the mechanism for selecting views via commands: the underlying unified annotation (s_canon) is mapped to multiple sets of views (views) through the deterministic projection function Π_view. Figure 1 User-side status; view Figure 2 AI-side status; view Figure 3 AI state change decision-making (one of the six action words) enables one-time labeling and multiple reuses, with naturally consistent behavior across views. The function of the instruction is to select which view to project onto, enabling configurable switching of the output labeling system.

[0280] Figure 8This specification illustrates a schematic diagram of a parallel tag track paradigm in a dialogue control method according to an embodiment of this specification, as shown below. Figure 8 As shown: This diagram illustrates that the input stream is always pure audio / text (without any decision words). Decisions are only made on parallel label tracks of the same length as the input stream, pinned to each decision read point (the position of the last audio word in that frame), while other positions are ignored (IGN). This paradigm has the advantages of zero positional drift (no token insertion, naturally monotonic positions 0..L-1), zero kernel modification to GDN (decision only reads the calculated hidden state), naturally established structural isolation (decision is not visible to each other, eliminating copy shortcuts), and positional alignment between train and serve (training reads logits[a_k], and online also reads logits[a_k], guaranteed by the causal equivalence theorem). The loss is only calculated at read point 'a', non-read points are ignored by IGN, and the last logits are read online; decisions are not retained in the preceding text, thus avoiding the lag at the flip point and state stickiness caused by decision backfeeding.

[0281] Figure 9 This diagram illustrates a decision feedback paradigm in a dialogue control method according to an embodiment of this specification, as shown below. Figure 9 As shown: This diagram illustrates the sequence structure of the decision-making feedback paradigm as a contrasting scheme to the parallel label-track paradigm (both are trained in pairs, their effects are compared, and both are included in the protection). In this paradigm, the decision word for each frame (such as continuing broadcast HOLD, stopping broadcast STOP, etc.) is expanded into micro-rounds (e.g., ... <im>assistant<im_end> The input stream is inserted and subsequently attended. Taking "intelligent customer service phone" as the main example, during the bot's broadcast phase (Phase A), the model incrementally moves forward frame by frame based on a key-value cache with a fixed prefix, continuously outputting HOLD. When a user is detected to insert a clear interruption intention such as "Hello, I'd like to interrupt," the decision is flipped to STOP. This STOP decision word is written into the input stream, and subsequent audio words (such as audio of the user's continued narration) can attend to this STOP decision word. This paradigm fully reuses existing conversational fine-tuning infrastructure and uses standard generative reasoning online, with the decision remaining in the preceding context. However, since subsequent audio can attend to the previous decision word, it is easy to learn a "copy shortcut" of "copying the previous label" (such as lag in the flip point and state stickiness), which is the consequence of violating the only additional condition of the causal equivalence theorem. Compared with the parallel label track paradigm, this invention advocates that both can be supported, but recommends the parallel label track paradigm to avoid cache invalidation and a decrease in decision quality.

[0282] Corresponding to the above method embodiments, Figure 10 This specification shows a schematic diagram of the structure of a dialogue system according to an embodiment. The dialogue system includes an online reasoning module, which includes: The prefix acquisition unit 1002 is used to acquire the fixed prefix of the target dialogue in response to receiving the audio segment of the current playback round of the target dialogue, wherein the key-value cache of the fixed prefix has been pre-written; Input building unit 1004 is used to construct the model input for the current broadcast round based on a fixed prefix and audio segment; Model calling unit 1006 is used to call the multimodal model, perform incremental forward calculation on the model input based on the key-value cache with a fixed prefix, and read the prediction score corresponding to any decision read point, where the decision read point is the position of the last audio word of the beat; Action determination unit 1008 is used to determine the target action from a preset action set based on the prediction score; The state switching unit 1010 is used to perform the state switching of the current broadcast wheel based on the target action.

[0283] In the embodiments of this specification, by pre-writing a fixed prefix of the target dialogue and reusing its key-value cache, when the audio segment of the current turn is received, only incremental forward computation needs to be performed on the model input constructed from the fixed prefix and the audio segment based on the cache. This avoids the repetitive computational cost caused by calculating all prefixes from scratch every time a beat is called, thereby effectively reducing the computational cost of high-frequency direct tuning of the multimodal model. By selecting the position of the last audio word of the current turn as the decision reading point to read the prediction score, the model outputs a decision only after fully perceiving the audio information of the beat, cutting off the transmission path of speech recognition serial delay and recognition errors in high-noise backgrounds to the decision layer in the cascaded architecture. Based on the prediction score, the target action is determined from the preset action set and the turn state is switched, realizing end-to-end turn control. While ensuring interaction continuity and response timeliness, it also takes into account computational efficiency and decision accuracy.

[0284] The above is an illustrative scheme of a dialogue system according to this embodiment. It should be noted that the technical solution of this dialogue system belongs to the same concept as the technical solutions of the dialogue control method and model training method described above. For details not described in detail in the technical solution of the dialogue system, please refer to the description of the technical solutions of the dialogue control method or model training method described above.

[0285] Figure 11 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 via a bus 1130, and a database 1150 is used to store data.

[0286] The computing device 1100 also includes an access device 1140, which enables the computing device 1100 to communicate via one or more networks 1160. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 1140 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0287] In one embodiment of this specification, the aforementioned components of the computing device 1100 and Figure 11 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 11 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0288] The computing device 1100 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1100 can also be a mobile or stationary server.

[0289] The processor 1120 is used to execute the following computer program / instruction, which, when executed by the processor, implements the steps of the above-mentioned dialogue control method or model training method.

[0290] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the dialogue control method and the model training method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the dialogue control method or the model training method described above.

[0291] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described dialogue control method or model training method.

[0292] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the dialogue control method and the model training method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the dialogue control method or the model training method described above.

[0293] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described dialogue control method or model training method.

[0294] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the dialogue control method and model training method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the dialogue control method or model training method described above.

[0295] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0296] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0297] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0298] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0299] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the embodiments described in this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.< / im>

Claims

1. A dialogue control method, comprising: In response to receiving an audio segment of the current playback round of the target dialogue, a fixed prefix of the target dialogue is obtained, wherein the key-value cache of the fixed prefix has been pre-written; Based on the fixed prefix and the audio segment, the model input for the current broadcast round is constructed, and the model input includes text words and audio words; The multimodal model is invoked, and incremental forward computation is performed on the model input based on the key-value cache of the fixed prefix. At any decision read point, the prediction score corresponding to the decision read point is read, wherein the decision read point is the position of the last audio word of the beat. Based on the predicted score, the target action is determined from the preset action set; Based on the target action, the current broadcast wheel state is switched.

2. The method according to claim 1, further comprising, before obtaining the fixed prefix of the target dialogue: Based on system prompts, historical dialogue text, and preset turn control tags, a fixed prefix is ​​constructed for the target dialogue; Write the fixed prefix into the key-value cache.

3. The method according to claim 2, wherein constructing a fixed prefix for the target dialogue based on system prompts, historical dialogue text, and preset turn control markers includes: The system prompts, historical dialogue texts, and preset turn control markers are used as a fixed prefix area; The fixed prefix region is encoded to obtain the fixed prefix of the target dialogue.

4. The method according to claim 1, further comprising, before constructing the model input for the current broadcast round based on the fixed prefix and the audio segment: Obtain the previously played text and reference audio, wherein the previously played text is the text already played in the current playing round, and the reference audio is a historical round audio segment used for speaker gating; The step of constructing the model input for the current broadcast round based on the fixed prefix and the audio segment includes: The model input for the current playback round is obtained by concatenating the fixed prefix, the audio segment, the already played text, and the reference audio.

5. The method according to claim 4, wherein concatenating the fixed prefix, the audio segment, the already played text, and the reference audio to obtain the model input for the current playback round includes: In chronological order, the broadcast text and the reference audio are inserted into the audio word sequence corresponding to the audio segment; The model input for the current broadcast round is obtained by splicing the fixed prefix, inserting the already broadcast text and the reference audio segment according to a fixed arrangement order.

6. The method according to claim 1, wherein calling the multimodal model, performing incremental forward computation on the model input based on the key-value cache of the fixed prefix, and reading the prediction score corresponding to any decision read point at any decision read point includes: The multimodal model is invoked, and incremental forward computation is performed on the model input based on the key-value cache of the fixed prefix to obtain the prediction scores output by the multimodal model at each word position of the model input; under the constraints of the interactive personality and / or the talk state of the current turn, the prediction score corresponding to the decision read point is read at any decision read point.

7. The method according to claim 6, wherein reading the predicted score corresponding to any decision point at any decision point, under the constraints of the interaction personality and the current turn state, comprises: By using preset switching instructions, the bias parameters corresponding to the interactive personality and the set of compliant actions corresponding to the current voice wheel state are determined; At any decision read point, obtain the prediction score output by the multimodal model; Based on the bias parameter, a bias is applied to the score of the action direction in the predicted score, wherein the action direction includes the direction of interrupting, the direction of giving way, and the direction of accepting. Based on the set of compliant actions, a mask is applied to the predicted scores of action directions outside the set of compliant actions, and the predicted scores after applying the bias and mask are determined as the predicted scores corresponding to the decision read point.

8. The method according to claim 6, further comprising: In response to switching the application scenario of the target dialogue, the interaction personality and / or output label are updated, wherein the updated interaction personality and / or output label are used to constrain the prediction score output by the multimodal model at the decision read point; Based on the updated interactive personality and / or output tags, the multimodal model is invoked to perform the switching of the current and subsequent broadcast rounds.

9. The method according to claim 1, further comprising: In response to a change in the state of the current talk turn, the fixed prefix of the target dialogue is reorganized based on the already played text and historical audio corresponding to the changed talk turn state, and the key-value cache is updated.

10. The method according to any one of claims 1 to 9, wherein determining the target action from a preset action set based on the predicted score comprises: Based on the predicted score, a target action is determined from a preset action set of the current broadcast turn's state.

11. The method according to claim 10, wherein the current turn state includes a listening state and a broadcasting state, and the preset action set of the current turn state includes a preset listening state action set and a preset broadcasting state action set; The step of determining the target action from a preset action set based on the predicted score includes: If the current turn of speech is in the listening state, then based on the predicted score, a target action is determined from the preset listening state action set, wherein the preset listening state action set includes the continue listening action, take over action, quietly accept action, and actively interrupt action; If the current voice turn is in the voice turn state, then based on the predicted score, a target action is determined from the preset voice turn state action set, wherein the preset voice turn state action set includes the continue voice turn action and the stop voice turn action.

12. The method according to claim 10, wherein the preset action set is included in a unified decision vocabulary, and the unified decision vocabulary includes the actions of continuing to listen, taking over, quietly accepting, actively interrupting, continuing to broadcast, and stopping broadcasting; The multimodal model outputs the predicted scores for each action in the unified decision vocabulary at any decision readout point; The preset action set is a subset of actions in the unified decision word list that correspond to the current voice turn state.

13. The method according to claim 11, wherein the step of switching the current broadcast wheel state based on the target action includes: If the target action is a takeover action, then the current broadcast wheel is switched from listening mode to broadcast mode, and a corresponding voice response is generated and broadcast based on the target action; If the target action is an active interruption action, the current broadcast wheel is switched from listening mode to broadcast mode, the reception of audio segments is stopped, and a corresponding interruption response voice is generated and broadcast. If the target action is to stop broadcasting, then the current broadcast wheel is switched from broadcasting mode to listening mode, the current voice broadcasting is stopped, and the system waits to receive the audio segment of the next broadcast wheel.

14. The method of claim 1, further comprising, before constructing the model input for the current broadcast round based on the fixed prefix and the audio segment: Based on the audio segment, it is determined whether the model invocation gate is satisfied, wherein the condition for satisfying the model invocation gate includes that the audio segment is valid speech; The step of constructing the model input for the current broadcast round based on the fixed prefix and the audio segment includes: If so, the model input for the current broadcast round is constructed based on the fixed prefix and the audio segment.

15. The method according to claim 14, after determining whether the model invocation gate is satisfied based on the audio segment, further comprising: If not, continue caching the audio segment and return to the step of determining whether the model call gating is satisfied based on the audio segment.

16. The method according to claim 14, wherein determining whether the model call gate is satisfied based on the audio segment includes: Obtain the acoustic state corresponding to the audio segment, wherein the acoustic state includes a sound state and a silent state; Determine the current turn state of the target dialogue, wherein the current turn state includes a broadcast state and a listen state; Obtain a preset policy switch, wherein the policy switch is used to indicate whether to allow users to actively interrupt; Based on the acoustic state, the current turn state, and the policy switch, it is determined whether the model call gating is satisfied. The model call gating is satisfied when the acoustic state is in a sound state and the current turn state is in a broadcast state, the acoustic state is in a silent state and the current turn state is in a listening state, or the acoustic state is in a sound state and the current turn state is in a listening state and the policy switch indicates that active interruption of the user is allowed.

17. A model training method, comprising: Obtain the fixed prefix of the target dialogue sample, the sample audio segment, and the frame-level decision label corresponding to the sample audio segment; Based on the fixed prefix of the sample and the audio segment of the sample, construct the sample model input; The multimodal model is invoked, and forward computation is performed on the sample model input based on the key-value cache of the fixed prefix of the sample. At any decision read point, the prediction score corresponding to that decision read point is read. Based on the predicted scores, training actions are determined from a preset action set; Based on the training action and the frame-level decision label, determine the decision loss; Based on the decision loss, the multimodal model is trained to obtain the trained multimodal model, which is used to execute the dialogue control method according to any one of claims 1 to 16.

18. The method according to claim 17, wherein obtaining a sample fixed prefix of the target dialogue includes: Based on system prompts and historical dialogue text, a fixed prefix for the target dialogue is constructed.

19. The method according to claim 17, obtaining a sample audio segment and a frame-level decision label corresponding to the sample audio segment, comprising: Based on a preset behavior formula, a preset dialogue script is generated, wherein the preset behavior formula specifies the value range of the behavior family and each objective axis, and the preset dialogue script is used to schedule the time-level landing point of each event. Audio rendering is performed on the preset dialogue script to synthesize a training seed, wherein the training seed includes user-side recordings and time-level metadata; Frame-level labels are calculated for the training seeds using a deterministic function, and the user-side recordings are segmented according to a preset window to obtain training sequences and decision label tracks corresponding to the training sequences. The deterministic function is determined by multiple orthogonal objective axes and the currently effective judgment rule, and the features corresponding to the noise axis do not participate in the decision classification. The training sequence is subjected to decision backfeed verification, and the decision label track is subjected to label correctness verification. If the verification passes, the training sequence is determined as the sample audio segment, and the decision label track is determined as the frame-level decision label.

20. The method according to claim 19, wherein performing decision backfeeding verification on the training sequence and label correctness verification on the decision label track includes: Verify whether the noise features of the training sequence match the noise audio waveform of the training seed to obtain the decision backfeeding verification result of the training sequence; The timestamp offset of the decision label track is verified, and it is also verified whether the decision label track is consistent with the currently effective judgment rule, so as to obtain the label correctness verification result of the decision label track.

21. The method of claim 17, further comprising, before constructing the sample model input based on the sample fixed prefix and the sample audio segment: Obtain the previously played text and reference audio, wherein the previously played text is the text already played in the current playing round, and the reference audio is a historical round audio segment used for speaker gating; The step of constructing the sample model input based on the sample fixed prefix and the sample audio segment includes: The sample model input is obtained by concatenating the sample fixed prefix, the sample audio segment, the broadcast text, and the reference audio.

22. The method according to claim 17, wherein the unified decision vocabulary includes decision words corresponding to each action in the preset action set, and the decision words are used to represent actions in the preset action set; The determination of decision loss based on the training action and the frame-level decision label includes: Based on the training actions and the frame-level decision labels, type-weighted processing is performed to determine the type-weighted loss, wherein the type weighting is to assign weights to each action type according to the frequency of occurrence of each action type in the unified decision vocabulary; Based on the training action and the frame-level decision label, a flip point weighting process is performed to determine the flip point weighting loss. The flip point weighting is to apply weights to the decision read points where the frame-level decision label flips, and inject asymmetric weights according to the interaction profile. The weight coefficient of early flips is greater than the weight coefficient of late flips. Based on the training action and the frame-level decision label, tolerance soft label processing is performed to determine the tolerance soft label loss, wherein the tolerance soft label is the conversion of hard labels into soft labels within the neighborhood at the flip time. Hard example modulation processing is performed based on the training action and the frame-level decision label to determine the hard example modulation loss. The hard example modulation is based on the prediction difference between the training action and the frame-level decision label, reducing the loss weight of easy-to-classify samples and increasing the loss weight of hard-to-classify samples. The decision loss is determined based on at least one of the type-weighted loss, the flip-point weighted loss, the tolerance soft tag loss, and the hard-case modulation loss.

23. The method according to claim 17, wherein the sample model input does not contain decision words, and the decision words belong to a unified decision vocabulary and are used to represent actions in a preset action set; The process of calling the multimodal model, performing forward computation on the sample model input based on the key-value cache of the sample fixed prefix, and reading the prediction score corresponding to any decision read point includes: The multimodal model is invoked, and forward computation is performed on the sample model input based on the key-value cache of the sample fixed prefix. At any decision read point, the prediction score corresponding to the decision read point is read. The decision word does not occupy the word position in the sample model input. The word position in the sample model input is obtained by arranging the sample fixed prefix and the sample audio segment in sequence. The decision read point is the position of the last audio word of the beat.

24. The method according to claim 17, wherein the sample model input includes historical decision words, the historical decision words being the decision words corresponding to the target action determined at the previous decision read point, and the historical decision words are inserted into the sample model input as part of the model input stream to participate in the attention calculation of subsequent positions; The process of calling the multimodal model, performing forward computation on the sample model input based on the key-value cache of the sample fixed prefix, and reading the prediction score corresponding to any decision read point includes: The multimodal model is invoked, and forward computation is performed on the sample model input containing the historical decision words based on the key-value cache of the fixed prefix of the sample. At any decision read point, the prediction score corresponding to that decision read point is read.

25. The method of claim 17, further comprising, before constructing the sample model input based on the sample fixed prefix and the sample audio segment: Obtain training instructions, which include output label system instructions and interactive personality instructions. The training instructions serve as system prompts in the fixed prefix of the samples and are used to constrain the output of the multimodal model. Obtain the underlying unified annotation, which includes fine-grained state annotations for each frame corresponding to the sample audio segment, and project the underlying unified annotations into corresponding view labels based on the training instructions; Determining the decision loss based on the training action and the frame-level decision label includes: determining the decision loss based on the view label and the frame-level decision label; After obtaining the trained multimodal model, the method further includes: obtaining evaluation instructions, the wording of which is not included in the training instructions, and determining the output label system selection accuracy and interaction personality compliance of the trained multimodal model based on the evaluation instructions.

26. A dialogue system, comprising an online reasoning module, the online reasoning module comprising: A prefix acquisition unit is used to acquire a fixed prefix of the target dialogue in response to receiving an audio segment of the current playback round of the target dialogue, wherein the key-value cache of the fixed prefix has been pre-written; An input construction unit is used to construct the model input for the current broadcast round based on the fixed prefix and the audio segment; The model invocation unit is used to invoke the multimodal model, perform incremental forward computation on the model input based on the key-value cache of the fixed prefix, and read the prediction score corresponding to the decision read point at any decision read point, wherein the decision read point is the position of the last audio word of the beat; An action determination unit is used to determine a target action from a preset action set based on the predicted score; The state switching unit is used to perform the state switching of the current broadcast wheel based on the target action.

27. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 25.

28. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 25.

29. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 25.

Images