A TailTurn speech-semantic fusion turn-based state prediction method and related equipment

The TailTurn model, by combining an audio encoder and a large language model, utilizes manually labeled data and thought chain reasoning text to solve the problem of accuracy in predicting turn states in complex scenarios in existing technologies, thereby improving the reliability and naturalness of real-time voice interaction.

CN123072397APending Publication Date: 2026-11-10CHINA TELECOM ARTIFICIAL INTELLIGENCE TECHNOLOGY (BEIJING) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611382131.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-07
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing turn-taking prediction methods lack accuracy in handling complex scenarios, especially when the text in streaming automatic speech recognition is incomplete or there are acoustic semantic conflicts, making it difficult to provide reliable decision-making basis in real-time voice interaction.

Method used

The TailTurn model is adopted, which extracts acoustic features through an audio encoder, maps the adapter module to a large language model, and combines manually labeled turn state samples for training. A thought chain reasoning text enhancement model is introduced to achieve joint modeling and prediction of acoustics and semantics.

Benefits of technology

It improves prediction accuracy in scenarios with incomplete streaming text or acoustic semantic conflicts, enhances the robustness and naturalness of real-time voice interaction, and provides explicit turn-taking state output.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123072397A_ABST
    Figure CN123072397A_ABST
Patent Text Reader

Abstract

This invention discloses a TailTurn speech-semantic fusion turn-taking state prediction method and related equipment. The invention utilizes a large language model to jointly represent acoustic embedding sequences and textual information, enhancing multimodal fusion capabilities. Based on automatic speech recognition corpora, the first text transcription is randomly truncated, and acoustically driven text completion training is performed using first sample audio. This forces the model to use audio to complete semantics when text is missing, enhancing the joint modeling capability of acoustics and semantics. Manually labeled turn-taking state tags are introduced for turn-taking state prediction training, enabling the model to output clear turn-taking states. Enhanced training is performed using pre-set thought chain reasoning text, improving interpretability in complex scenarios. When processing speech data to be predicted, the target model maintains high prediction accuracy in scenarios with incomplete streaming text or acoustic-semantic conflicts, improving the robustness and naturalness of real-time voice interaction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a TailTurn speech-semantic fusion turn-based state prediction method and related equipment. Background Technology

[0002] As voice interaction technology evolves towards full-duplex mode, systems need to determine the user's turn status in real time during speech to decide when to respond, remain silent, or provide brief feedback. Existing turn status prediction methods mainly fall into three categories. The first category is based on speech activity detection, which determines whether the user has finished speaking by detecting speech energy and silence intervals. A fixed silence duration threshold is typically set; exceeding the threshold indicates the end of the turn. This type of method struggles to distinguish between pauses of thought and the end of a speech, and is prone to false triggering of non-verbal acoustic events such as coughing and laughter. The second category is based on pure text semantics, relying solely on the text output of streaming automatic speech recognition to determine semantic completeness. This type of method utilizes semantic information but completely ignores acoustic cues such as prosody, ending sounds, and energy decay in the audio, easily leading to misjudgments when the semantics are complete but the acoustics have not yet ended. The third category is based on audio-text fusion, which fuses acoustic features extracted by the audio encoder with text features in the latent space before classification. This type of method theoretically takes into account both modalities, but during training, it tends to over-rely on text semantics and ignore the prosodic cues in the audio. When streaming automatic speech recognition loses tail words due to delay, or when acoustic information conflicts with semantic information, the accuracy drops significantly. Summary of the Invention

[0003] The main objective of this invention is to propose a TailTurn speech-semantic fusion turn state prediction method, device, electronic device, storage medium, and program product, aiming to solve at least one problem of the prior art.

[0004] To achieve the above objectives, one aspect of the present invention proposes a TailTurn speech-semantic fusion turn-based state prediction method, the method comprising: Obtain a training sample set; wherein, the training sample set includes automatic speech recognition corpus samples and manually annotated turn state samples, the automatic speech recognition corpus samples include a first sample audio and a first text transcription; the manually annotated turn state samples include a second sample audio, a second text transcription, and turn state labels; Configure the TailTurn model; the TailTurn model includes an audio encoder, an adapter module, and a large language model; the audio encoder is used to extract acoustic feature sequences from the input audio; the adapter module is used to map the acoustic feature sequences to the word embedding space of the large language model to obtain acoustic embedding sequences; the large language model is used to generate target sequences based on the acoustic embedding sequences; Based on the automatic speech recognition corpus samples, the first text transcription is randomly truncated, and then combined with the first sample audio to perform acoustic-driven text completion training on the TailTurn model to obtain the first training model. Based on manually labeled turn-around state samples, turn-around state labels are introduced to train the first training model for turn-around state prediction, thus obtaining the second training model. Based on manually labeled turn-taking state samples combined with pre-set thought chain reasoning text, the second training model is subjected to thought chain enhancement training to obtain the target model; Target model is used to process the speech state prediction of the speech data to be predicted.

[0005] In some optional ways, configuring the TailTurn model includes the following steps: Obtain a pre-trained automatic speech recognition model and extract the encoder part of the automatic speech recognition model as an audio encoder. The adapter module is obtained based on the configuration of a multi-layer Transformer network. Use pre-trained large language models with decoder-only architectures as large language models; The audio encoder, adapter module and large language model are connected in sequence to obtain the TailTurn model; The parameter training strategy of the TailTurn model includes: the audio encoder is always frozen; the adapter module always participates in training; the large language model performs full parameter training in text completion training, and freezes the main parameters and only updates the low-rank adaptation parameters in turn state prediction training and thought chain enhancement training.

[0006] In some alternative approaches, based on automatic speech recognition corpus samples, the first text transcription is randomly truncated, and then combined with the first sample audio to perform acoustic-driven text completion training on the TailTurn model to obtain the first training model, including the following steps: Based on the first sampling probability, automatic speech recognition corpus samples are collected from the training sample set as first candidate samples, and all first candidate samples are aggregated into the first complete text pattern sample. Based on the second sampling probability, automatic speech recognition corpus samples are collected from the training sample set as second candidate samples. The last word of the first text transcription in the second candidate samples is randomly truncated according to a preset ratio to update the first text transcription of the second candidate samples, and then the first part of the text pattern samples are summarized. Based on the third sampling probability, automatic speech recognition corpus samples are collected from the training sample set as third candidate samples. The first text transcription in the third candidate samples is removed, and then the first textless mode sample is obtained by summarizing. The first complete text pattern sample, the first partial text pattern sample, and the first no-text pattern sample are combined into a text completion training sample. The first acoustic feature sequence of the first sample audio of each text completion training sample is extracted using the audio encoder of the TailTurn model; Based on the first acoustic feature sequence, the first acoustic embedding sequence is obtained by processing it using the adapter module of the TailTurn model; For each text completion training sample, the first text transcription is embedded to obtain the first text word embedding; if the text completion training sample does not have a first text transcription, the corresponding first text word embedding is configured as empty. The first acoustic embedding sequence, the first text lexical embedding, and the preset first prompt text of the same text completion training sample are concatenated and input into the large language model of the TailTurn model so that the large language model can obtain the first completed text through autoregression. Based on the first completed text and its corresponding first text transcription in the automatic speech recognition corpus samples, the adapter module and large language model of the TailTurn model are optimized by cross-entropy loss to obtain the first training model.

[0007] In some alternative approaches, based on manually labeled turn-taking state samples, turn-taking state labels are introduced to train the first training model for turn-taking state prediction, resulting in a second training model, which includes the following steps: Based on the fourth sampling probability, manually labeled turn-taking state samples are collected from the training sample set as fourth candidate samples, and all fourth candidate samples are aggregated into second complete text pattern samples. Based on the fifth sampling probability, manually labeled turn-taking state samples are collected from the training sample set as fifth candidate samples. The last word of the second text transcription in the fifth candidate samples is randomly truncated according to a preset ratio to update the second text transcription of the fifth candidate samples. Then, the second part of the text pattern samples are summarized. Based on the sixth sampling probability, manually labeled turn-taking state samples are collected from the training sample set as the sixth candidate samples. The second text transcription in the sixth candidate samples is removed, and then the second textless mode samples are obtained by summarizing. The second complete text pattern sample, the second partial text pattern sample, and the second no-text pattern sample are combined into a state prediction training sample. The second acoustic feature sequence of the second sample audio of each state prediction training sample is extracted using the audio encoder of the first training model; Based on the second acoustic feature sequence, the second acoustic embedding sequence is obtained by processing it using the first training model adapter module. The second text transcription of each state prediction training sample is embedded to obtain the second text word embedding; if the state prediction training sample does not have a second text transcription, the corresponding second text word embedding is configured to be empty. The second acoustic embedding sequence, the second text lexical embedding, and the preset second prompt text of the same state prediction training sample are concatenated and input into the large language model of the first training model so that the large language model can process and obtain the second complete text and the predicted turn state; wherein, the predicted turn state includes the probability value of each turn state type; The first comparison data is constructed based on the second completed text and its corresponding second text transcription in the manually annotated turn-by-turn state samples. The second comparison data is constructed based on the predicted turn-by-turn state and the turn-by-turn state label. Based on the first comparison data and the second comparison data, the first loss is constructed using cross-entropy loss. A second loss is constructed by summing the probability values ​​of all turn-by-turn state types in the predicted turn-by-turn state and using the penalty loss. The total loss is obtained by weighted summation of the first and second losses. The second training model is obtained by optimizing the low-rank adaptation parameters of the adapter module of the first training model and the self-attention layer of the large language model based on the total loss.

[0008] In some alternative approaches, the second training model is subjected to thought chain enhancement training based on manually labeled turn-taking state samples combined with pre-set thought chain reasoning text to obtain the target model, including the following steps: A portion of manually labeled turn-taking state samples were collected from the training sample set as augmented training samples; The third acoustic feature sequence of the second sample audio of each enhanced training sample is extracted using the audio encoder of the second training model; Based on the third acoustic feature sequence, the third acoustic embedding sequence is obtained by processing it using the second training model adapter module. The second text transcription of each augmented training sample is embedded to obtain the third text word embedding; The third acoustic embedding sequence, the third text lexical embedding, the preset second prompt text, and the thought chain reasoning text of the same enhanced training sample are concatenated and input into the large language model of the second training model. Then, the teacher-forced training method is adopted to make the large language model perform explicit reasoning before outputting the results, so as to obtain the target model after training.

[0009] In some alternative approaches, the turn-based state prediction of the speech data to be predicted is processed using a target model, including the following steps: Acquire the speech data to be predicted; wherein, the speech data to be predicted includes user audio and streaming automatic speech recognition text prefix; The user's audio is input into the target model's audio encoder, and the target acoustic feature sequence is extracted. The target acoustic feature sequence is input into the adapter module of the target model to obtain the target acoustic embedding sequence; Embedding of text prefixes in streaming automatic speech recognition is performed to obtain target text word embeddings; Based on the target acoustic embedding sequence, target text lexical embedding, and preset second prompt text, the target output sequence is generated through an autoregressive method according to the large language model of the target model. The target completion text and target turn status label are parsed from the output sequence.

[0010] In some alternative approaches, based on the target acoustic embedding sequence, the target text lexical embedding, and a pre-defined second cue text, the target output sequence is generated using an autoregressive approach according to the large language model of the target model, including the following steps: If the thought chain output is enabled, the target acoustic embedding sequence, target text word embedding, preset second prompt text and thought chain reasoning text are concatenated and input into the target model's large language model, so that the large language model performs explicit reasoning before outputting the result, obtains reasoning text, and then drives the target model's large language model to generate the target output sequence through reasoning text. If the thought chain output is not enabled, the target acoustic embedding sequence, the target text lexical embedding, and the second prompt text are concatenated and input into the target model's large language model so that the large language model can generate the target output sequence through autoregression.

[0011] To achieve the above objectives, another aspect of the present invention proposes a TailTurn speech-semantic fusion turn-based state prediction device, the device comprising: The sample acquisition module is used to acquire the training sample set; wherein, the training sample set includes automatic speech recognition corpus samples and manually annotated turn state samples, the automatic speech recognition corpus samples include the first sample audio and the first text transcription; the manually annotated turn state samples include the second sample audio, the second text transcription and the turn state labels; The model configuration module is used to configure the TailTurn model. The TailTurn model includes an audio encoder, an adapter module, and a large language model. The audio encoder is used to extract acoustic feature sequences from the input audio. The adapter module is used to map the acoustic feature sequences to the word embedding space of the large language model to obtain acoustic embedding sequences. The large language model is used to generate target sequences based on the acoustic embedding sequences. The text completion training module is used to randomly truncate the first text transcription based on the automatic speech recognition corpus samples, and then combine the first sample audio to perform acoustically driven text completion training on the TailTurn model to obtain the first training model. The prediction training module is used to predict the turn-by-turn state of the first training model based on manually labeled turn-by-turn state samples and introduce turn-by-turn state labels to obtain the second training model. The enhanced training module is used to perform enhanced training on the second training model based on manually labeled turn-taking state samples combined with preset thought chain reasoning text, so as to obtain the target model. The model application module is used to process the speech state prediction of the speech data to be predicted using the target model.

[0012] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned method.

[0013] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0014] To achieve the above objectives, another aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0015] This invention provides at least the following beneficial effects: It offers a TailTurn speech-semantic fusion turn-taking state prediction method, apparatus, electronic device, storage medium, and program product. This solution involves acquiring a training sample set, which includes automatically recognized speech corpus samples and manually annotated turn-taking state samples. The automatically recognized speech corpus samples include a first sample audio and a first text transcription; the manually annotated turn-taking state samples include a second sample audio, a second text transcription, and turn-taking state labels. A TailTurn model is configured, comprising an audio encoder, an adapter module, and a large language model. The audio encoder extracts acoustic feature sequences from the input audio; the adapter module converts the input audio into a single audio string. Acoustic feature sequences are mapped to the word embedding space of a large language model to obtain acoustic embedding sequences. The large language model is used to generate target sequences based on acoustic embedding sequences. Based on automatic speech recognition corpus samples, the first text transcription is randomly truncated, and then combined with the first sample audio to perform acoustically driven text completion training on the TailTurn model to obtain the first training model. Based on manually labeled turn-taking state samples, turn-taking state labels are introduced to train the first training model for turn-taking state prediction to obtain the second training model. Based on manually labeled turn-taking state samples combined with preset thought chain reasoning text, the second training model is subjected to thought chain enhancement training to obtain the target model. The target model is used to process the turn-taking state prediction of the speech data to be predicted. This invention leverages a large language model to jointly represent acoustic embedding sequences and textual information, enhancing multimodal fusion capabilities. It randomly truncates the first text transcription based on automatic speech recognition corpus and combines it with first sample audio for acoustically driven text completion training. This forces the model to use audio to complete semantics when text is missing, avoiding reliance solely on text for judgment and enhancing the joint modeling capabilities of acoustics and semantics. It introduces manually labeled turn-taking states for turn-taking state prediction training, enabling the model to output explicit turn-taking states. Furthermore, it incorporates pre-defined thought chain reasoning text for enhanced training, improving interpretability in complex scenarios. Therefore, when processing speech data to be predicted, the target model maintains high prediction accuracy in scenarios with incomplete streaming text or acoustic-semantic conflicts, improving the robustness and naturalness of real-time voice interaction. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of an implementation environment for the TailTurn speech-semantic fusion turn state prediction method provided in this embodiment of the invention; Figure 2 This is a flowchart illustrating the TailTurn speech-semantic fusion turn-based state prediction method provided in an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the unfolded process of step S600 provided in an embodiment of the present invention; Figure 4This is a schematic diagram of the overall architecture principle of the TailTurn speech-semantic fusion turn state prediction method provided in this embodiment of the invention; Figure 5 This is a schematic diagram of the structure of the TailTurn speech-semantic fusion turn state prediction device provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0018] It is understood that the terms "first," "second," etc., used in this invention may be used to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of embodiments of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to determination," or "in the event of a determination."

[0019] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0020] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this invention is for descriptive purposes only and is not intended to limit the invention.

[0021] In related technologies, existing methods generally lack interpretable reasoning mechanisms, making it difficult to provide decision-making basis in complex conflict scenarios. Furthermore, they lack specific training strategies designed for real-world scenarios where streaming automatic speech recognition text is incomplete, resulting in insufficient robustness.

[0022] In view of this, this invention provides a TailTurn speech-semantic fusion turn-taking state prediction method and related equipment. This method involves acquiring a training sample set, which includes automatically recognized speech corpus samples and manually annotated turn-taking state samples. The automatically recognized speech corpus samples include a first sample audio and a first text transcription; the manually annotated turn-taking state samples include a second sample audio, a second text transcription, and turn-taking state labels. A TailTurn model is configured, comprising an audio encoder, an adapter module, and a large language model. The audio encoder extracts acoustic feature sequences from the input audio; the adapter module maps the acoustic feature sequences to... The word embedding space of the large language model yields an acoustic embedding sequence. The large language model is then used to generate a target sequence based on this acoustic embedding sequence. Based on samples from an automatic speech recognition corpus, the first text transcription is randomly truncated, and then combined with the first sample audio to perform acoustically driven text completion training on the TailTurn model, resulting in a first training model. Based on manually labeled turn-taking state samples, turn-taking state labels are introduced to train the first training model for turn-taking state prediction, resulting in a second training model. Based on manually labeled turn-taking state samples combined with pre-defined thought chain reasoning text, the second training model undergoes thought chain enhancement training, resulting in a target model. The target model is then used to process the turn-taking state prediction of the speech data to be predicted. This invention leverages a large language model to jointly represent acoustic embedding sequences and textual information, enhancing multimodal fusion capabilities. It randomly truncates the first text transcription based on automatic speech recognition corpus and combines it with first sample audio for acoustically driven text completion training. This forces the model to use audio to complete semantics when text is missing, avoiding reliance solely on text for judgment and enhancing the joint modeling capabilities of acoustics and semantics. It introduces manually labeled turn-taking states for turn-taking state prediction training, enabling the model to output explicit turn-taking states. Furthermore, it incorporates pre-defined thought chain reasoning text for enhanced training, improving interpretability in complex scenarios. Therefore, when processing speech data to be predicted, the target model maintains high prediction accuracy in scenarios with incomplete streaming text or acoustic-semantic conflicts, improving the robustness and naturalness of real-time voice interaction.

[0023] It is understood that the TailTurn speech-semantic fusion turn-based state prediction method provided by this invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various types of terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet, laptop, or desktop computer, but it is not limited to these.

[0024] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0025] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0026] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0027] Terminal 102 can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0028] For example, based on Figure 1The implementation environment shown in this embodiment of the invention provides a TailTurn voice-semantic fusion turn-taking state prediction method. The following description uses the application of this TailTurn voice-semantic fusion turn-taking state prediction method in server 101 as an example. It can be understood that this TailTurn voice-semantic fusion turn-taking state prediction method can also be applied in terminal 102.

[0029] Reference Figure 2 , Figure 2 This is an optional flowchart of the TailTurn speech-semantic fusion turn-taking state prediction method provided in the embodiments of the present invention. The executing entity of the TailTurn speech-semantic fusion turn-taking state prediction method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 The method may include, but is not limited to, steps S100 to S600.

[0030] Step S100: Obtain the training sample set; The training sample set includes automatic speech recognition corpus samples and manually annotated turn-taking state samples. The automatic speech recognition corpus samples include the first sample audio and the first text transcription; the manually annotated turn-taking state samples include the second sample audio, the second text transcription, and turn-taking state labels. For example, in some implementations, turn status labels are used to indicate states such as semantic completeness, semantic incompleteness, request to pause or terminate the conversation, brief agreement, and invalid dialogue. Specifically, turn-taking is one of the most fundamental interaction mechanisms in natural human dialogue. In interpersonal conversations, listeners need to anticipate whether the speaker is about to finish speaking by using various cues, including acoustic information such as phonetic prosody, the direction of the final sound, and semantic information, before the speaker has completely finished speaking, so as to respond or provide feedback at the appropriate time. This process is crucial to the fluency and naturalness of the interaction.

[0031] Step S200: Configure the TailTurn model; The TailTurn model includes an audio encoder, an adapter module, and a large language model. The audio encoder is used to extract acoustic feature sequences from the input audio. The adapter module is used to map the acoustic feature sequences to the word embedding space of the large language model to obtain acoustic embedding sequences. The large language model is used to generate target sequences based on the acoustic embedding sequences. It should be noted that, in some embodiments, configuring the TailTurn model may include the following steps: obtaining a pre-trained automatic speech recognition model, extracting the encoder part of the automatic speech recognition model as the audio encoder; configuring an adapter module based on a multi-layer Transformer network; using a pre-trained large language model with a decoder-only architecture as the large language model; and sequentially connecting the audio encoder, adapter module, and large language model to configure the TailTurn model. The parameter training strategy of the TailTurn model includes: the audio encoder is always frozen; the adapter module always participates in training; the large language model undergoes full parameter training during text completion training, and freezes the main parameters and only updates the low-rank adaptation parameters during turn state prediction training and thought chain enhancement training.

[0032] For example, in some specific embodiments, the TailTurn model architecture (i.e., the TailTurn model) proposed in this invention includes the following three core components: 1. Audio encoder. The encoder part of the pre-trained Zipformer automatic speech recognition model is used to extract high-level acoustic feature sequences from the input audio. During training, acoustic features are dynamically extracted in windows such as 160ms, 320ms, and 640ms, and the parameters are frozen and do not participate in gradient updates.

[0033] 2. Adapter Module. This module consists of a 4-layer Transformer network (the specific number of layers can be adjusted according to actual needs, such as 3, 5, or other layers; this embodiment uses 4 layers as an example), connecting the audio encoder and the large language model. The adapter maps the acoustic feature sequence output by the audio encoder to the word embedding space of the large language model, obtaining an acoustic embedding sequence. The adapter parameters are trainable parameters, transforming continuous acoustic representations into vectors understandable by the large language model.

[0034] 3. Large Language Model (LLM). This employs a pre-trained decode-only architecture (a decoder-only architecture, characterized by containing only the decoder part of the Transformer, generating text word-by-word through an autoregressive mechanism). It receives concatenated Prompts, acoustic embeddings, and text lexical embeddings, generating the target sequence in an autoregressive manner. The LLM injects low-rank adaptive parameters (LoRA) into some training phases for efficient fine-tuning; the main parameters are either frozen or trained with all parameters, depending on the training phase.

[0035] The parameter strategy of this architecture is as follows: the audio encoder is always frozen; the large language model is trained with all parameters in stage one, and the main parameters are frozen in stages two and three, with only LoRA weights being updated; the adapter participates in training in all stages.

[0036] Step S300: Based on the automatic speech recognition corpus samples, the first text transcription is randomly truncated, and then the TailTurn model is trained with acoustic-driven text completion by combining the first sample audio to obtain the first training model. It should be noted that in some embodiments, step S300 may include the following steps: collecting automatic speech recognition corpus samples from the training sample set based on a first sampling probability as first candidate samples, and summarizing all first candidate samples into a first complete text pattern sample; collecting automatic speech recognition corpus samples from the training sample set based on a second sampling probability as second candidate samples, randomly truncating the last words of the first text transcription in the second candidate samples according to a preset ratio to update the first text transcription of the second candidate samples, and then summarizing to obtain a first part of the text pattern sample; collecting automatic speech recognition corpus samples from the training sample set based on a third sampling probability as third candidate samples, removing the first text transcription in the third candidate samples, and then summarizing to obtain a first no-text pattern sample; summarizing the first complete text pattern sample, the first part of the text pattern sample, and the first no-text pattern sample into a text completion training sample; and using the TailTurn model. The audio encoder extracts the first acoustic feature sequence of the first sample audio of each text completion training sample; based on the first acoustic feature sequence, the adapter module of the TailTurn model is used to process it to obtain the first acoustic embedding sequence; the first text transcription of each text completion training sample is embedded to obtain the first text word embedding; if the text completion training sample does not have a first text transcription, the corresponding first text word embedding is configured as empty; the first acoustic embedding sequence, the first text word embedding, and the preset first prompt text of the same text completion training sample are concatenated and input into the large language model of the TailTurn model so that the large language model can obtain the first completed text through autoregression; based on the first completed text and its corresponding first text transcription in the automatic speech recognition corpus samples, the adapter module of the TailTurn model and the large language model are optimized through cross-entropy loss to obtain the first training model.

[0037] For example, in some specific implementations, the first phase of the entire training process (acoustic-driven text completion training) can be achieved as follows: The goal of Phase 1 is to teach the model to complete the deliberately obscured end text based on the audio content, thereby forcing the model to utilize the acoustic information provided by the audio encoder, enabling the large language model to establish the ability to "complete text by listening to audio." During this phase, the Zipformer audio encoder is frozen, and the adapter and large language model undergo full-parameter training.

[0038] The training data uses an automated speech recognition corpus, with each sample containing both audio and its complete transcribed text. The text is randomly truncated at the end according to a preset ratio to construct three text granularities: Full Text Mode: Provides all text terms; the target completion segment is empty. Partial text mode: Randomly truncate the last 30% to 70% of the words, keeping only the incomplete text as input, and the target completion segment is the truncated last words; No text mode: No text terms are provided; the target completion segment is a complete text transcription.

[0039] The sampling probabilities for the three modes are 10%, 40%, and 50%, respectively.

[0040] During training, the acoustic embedding of the audio is entered sequentially after the Prompt text.<asr_text> The model embeds the tags, followed immediately by the actual provided text input. It then predicts the masked tail terms in an autoregressive manner and outputs them at the end.< / asr_text> Labeling. The training loss uses the standard cross-entropy loss.

[0041] Prompt template (i.e., the first prompt text): Complete based on the audio.<asr_text> Subsequent transcription content and< / asr_text> Finish.

[0042] Step S400: Based on manually labeled turn-around state samples, turn-around state labels are introduced to train the first training model for turn-around state prediction, thereby obtaining the second training model; It should be noted that in some embodiments, step S400 may include the following steps: collecting manually labeled turn-taking state samples from the training sample set based on a fourth sampling probability as fourth candidate samples, and summarizing all fourth candidate samples into a second complete text pattern sample; collecting manually labeled turn-taking state samples from the training sample set based on a fifth sampling probability as fifth candidate samples, randomly truncating the last words of the second text transcription in the fifth candidate samples according to a preset ratio to update the second text transcription of the fifth candidate samples, and then summarizing to obtain a second part text pattern sample; collecting manually labeled turn-taking state samples from the training sample set based on a sixth sampling probability as sixth candidate samples, removing the second text transcription in the sixth candidate samples, and then summarizing to obtain a second no-text pattern sample; summarizing the second complete text pattern sample, the second part text pattern sample, and the second no-text pattern sample into a state prediction training sample; extracting the second acoustic feature sequence of the second sample audio of each state prediction training sample using the audio encoder of the first training model; processing the second acoustic feature sequence using the first training model adapter module to obtain a second acoustic embedding sequence; and for each The second text transcription of the state prediction training sample is embedded to obtain the second text word embedding. If the state prediction training sample does not have a second text transcription, the corresponding second text word embedding is configured as empty. The second acoustic embedding sequence, the second text word embedding, and the preset second prompt text of the same state prediction training sample are concatenated and input into the large language model of the first training model so that the large language model can process and obtain the second completed text and the predicted turn state. The predicted turn state includes the probability value of each turn state type. The first comparison data is constructed based on the second completed text and its corresponding second text transcription in the manually labeled turn state samples. The second comparison data is constructed based on the predicted turn state and the turn state label. Based on the first comparison data and the second comparison data, the first loss is constructed using cross-entropy loss. The second loss is constructed using penalty loss based on the sum of the probability values ​​of all turn state types in the predicted turn state. The first loss and the second loss are weighted and summed to obtain the total loss. The low-rank adaptation parameters of the adapter module of the first training model and the self-attention layer of the large language model are optimized based on the total loss to obtain the second training model.

[0043] For example, in some specific implementations, phase two (turn-based state prediction training) of the entire training process can be achieved as follows: Based on the model trained in Phase 1, a turn-based state prediction task is introduced. In this phase, the parameters of the Zipformer encoder and the main parameters of the large language model are frozen, and only the adapter module and the LoRA parameters injected into the self-attention layer of the large language model are trained.

[0044] The training data uses a manually labeled turn-taking state dataset. Each sample contains audio, the corresponding text transcription, and a turn-taking state label. Turn status tag set Defined as five special tokens: <complete>Indicates semantic completeness. <incomplete>This indicates that the meaning is incomplete. <wait>This indicates a request to suspend or terminate the communication. <backchannel>Indicates a brief agreement, <invalid>Indicates invalid dialogue (such as nonverbal acoustic events like coughing, laughter, or ambient noise).

[0045] During training, the text masking strategy from Phase 1 was used, while the Prompt template (i.e., the second cue text) was expanded to: Complete the audio based on the audio.<asr_text> Subsequent transcription content and< / asr_text> End, then output the talk round state, state definition: <complete>Indicates semantic completeness. <incomplete>This indicates that the meaning is incomplete. <wait>This indicates a request to suspend or terminate the communication. <backchannel>Indicates a brief agreement, <invalid>This indicates an invalid conversation. The training loss consists of two parts: (1) Cross-entropy loss (i.e., the first loss): calculate the loss of the model in generating the entire target sequence (including the comparison loss between the completed text and the original text, and the comparison loss between the predicted turn state and the turn state label). (2) Penalty loss To ensure that the model distributes the probability mass set to the five valid turn state labels at the end of the sequence, the penalty loss is defined as:

[0046] in, To prevent the logarithm from being an infinitely small constant, i.e., the total probability of the turn-based status tags... When it approaches 1, the penalty loss Approaching 0; conversely, increasing the penalty loss. This guides the model to assign the predicted probability of the final turn state label to the turn state label, completely eliminating the model's evasive behavior of generating irrelevant tokens or terminating prematurely at the decision point.

[0047] Therefore, the total loss : , It is the penalty coefficient (its value can be in the range of (0.3, 0.5)).

[0048] Step S500: Based on manually labeled turn-taking state samples and pre-set thought chain reasoning text, perform thought chain enhancement training on the second training model to obtain the target model. It should be noted that in some embodiments, step S500 may include the following steps: collecting some manually labeled turn-taking state samples from the training sample set as enhanced training samples; extracting the third acoustic feature sequence of the second sample audio of each enhanced training sample using the audio encoder of the second training model; processing the third acoustic feature sequence using the adapter module of the second training model to obtain the third acoustic embedding sequence; embedding the second text transcription of each enhanced training sample to obtain the third text word embedding; concatenating the third acoustic embedding sequence, the third text word embedding, the preset second prompt text, and the thought chain reasoning text of the same enhanced training sample and inputting them into the large language model of the second training model, and then using a teacher-forced training method to make the large language model perform explicit reasoning before outputting the results, so as to obtain the trained target model.

[0049] For example, in some specific implementations, phase three (mental chain reinforcement training) of the entire training process can be achieved as follows: To further enhance the model's interpretable reasoning capabilities in complex scenarios, we performed thought chain enhancement training. During this phase, we continued to freeze the Zipformer encoder and the main body of the large language model, training only the adapter and LoRA parameters.

[0050] For approximately 30% of the training samples, a second prompt text (or similar text) is inserted between the Prompt and the target sequence. <think> and< / think> The wrapped thought chain reasoning text. The reasoning text, generated manually or automatically, describes the analysis process for determining turn-taking states based on audio ending prosody and text semantic completeness. The target sequence format becomes: <think> chain of thought< / think> Complete the content< / asr_text><turn_label> ; Training employs a teacher-mandated approach, where the model performs explicit reasoning before outputting complete content and turn-based state labels. For samples without thought chains, the target sequence format is consistent with Phase Two. Loss calculation covers the autoregressive generation loss across the entire extended sequence, and the penalty loss from Phase Two is also applied. This phase grants the model optional reasoning capabilities; during inference, the generation of parts can be controlled via a system switch, enabling on-demand activation of interpretability.

[0051] Step S600: Use the target model to process the turn-taking state prediction of the speech data to be predicted; It should be noted that in some embodiments, such as Figure 3 As shown, step S600 may include the following steps: S610, acquiring speech data to be predicted; wherein, the speech data to be predicted includes user audio and streaming automatic speech recognition text prefixes; S620, inputting the user audio into the audio encoder of the target model to extract the target acoustic feature sequence; S630, inputting the target acoustic feature sequence into the adapter module of the target model to map the target acoustic embedding sequence; S640, performing embedding processing on the streaming automatic speech recognition text prefixes to obtain target text lexical embeddings; S650, based on the target acoustic embedding sequence, target text lexical embeddings, and preset second prompt text, generating the target output sequence through an autoregressive method according to the large language model of the target model; S660, parsing the target completion text and target turn status labels from the output sequence.

[0052] For example, in some implementations, during inference, the model receives user audio and optional streaming automatic speech recognition text prefixes. Using the Prompt template from training phase two, the large language model generates an output sequence in an autoregressive manner. Completed text (if present) and a final turn-based state tag token are parsed from the generated sequence, triggering corresponding system actions.

[0053] It should be noted that in some embodiments, step S650 may include the following steps: S651, if the thought chain output is enabled, the target acoustic embedding sequence, target text lexical embedding, preset second prompt text and thought chain reasoning text are concatenated and input into the target model's large language model, so that the large language model performs explicit reasoning before outputting the result, obtains reasoning text, and then drives the target model's large language model to generate the target output sequence through the reasoning text; S652, if the thought chain output is not enabled, the target acoustic embedding sequence, target text lexical embedding and second prompt text are concatenated and input into the target model's large language model, so that the large language model generates the target output sequence through autoregression.

[0054] For example, in some specific implementations, the model can choose whether to enable thought chain output during inference. If enabled, the model first generates the inference text, then performs completion and turn-taking prediction to provide interpretable evidence for decision-making; if disabled, the model directly outputs the completed text and turn-taking labels, resulting in lower latency.

[0055] It should also be noted that in some embodiments, the method may further include the following step: triggering a response behavior in response to the type of the target turn-around state tag. Specifically, the post-decision response behavior of the turn-around state tag can be implemented as follows: <complete>Immediately generate a system response and initiate TTS broadcast; <incomplete>: Remain silent and continue caching subsequent audio; <wait>After outputting a confirmation message such as "Close," the TTS will stop broadcasting. <backchannel>Output brief responses such as "um" or "okay" without interrupting the user; <invalid>Ignore this audio segment; no response will be triggered.

[0056] In some preferred embodiments, the present invention also provides a multi-level wake-up mechanism and fallback strategy in streaming inference: In streaming interactive scenarios, the inference cost of a large language model is far higher than that of a streaming automatic speech recognition decoder. Calling the large language model for every audio block (e.g., 320 ms) would result in significant computational waste and response latency. Therefore, this invention employs an encoder-CTC joint decoding architecture as the real-time speech recognition front-end: the encoder extracts acoustic features block by block, while the CTC decoder synchronously outputs the streaming recognition text prefix; simultaneously, consecutive blank tags generated during CTC decoding are accumulated and converted into physical silence durations, serving as the basis for scheduling the inference timing of the large language model.

[0057] The system runs the CTC decoding thread and the large language model inference scheduling thread in parallel. The specific control logic is as follows: (1) Short silence phase (cumulative duration of continuous Blank < first preset threshold, preferably 320 ms): This duration corresponds to the normal breathing or pronunciation intervals within a word or phrase, indicating that the user is in a continuous pronunciation state. At this time, the large language model is not activated; instead, the acoustic features corresponding to the current audio block are appended to the cache queue, and the streaming text prefix is ​​updated normally as the decoding process progresses.

[0058] (2) Effective silence phase (first preset threshold ≤ cumulative duration of continuous Blank < second preset threshold, preferably 320ms to 3000ms): When the cumulative duration of consecutive blanks reaches a first preset threshold, the system determines that a potential turn-taking boundary has been reached and immediately triggers inference using the large language model. During inference, the system maps the acoustic features in the cache queue into acoustic embedding sequences via an adapter, and feeds them, along with the current CTC decoding output streaming recognition text prefix and preset prompt text, into the large language model. The model then performs a multimodal joint decision based on acoustic prosody (end note direction, energy decay, etc.) and semantic integrity (dependency syntax, contextual logic), objectively outputting the turn-taking status label (e.g., ...). <complete> 、 <incomplete> 、 <wait>(etc.). During this reasoning process, the system does not inject any prior bias related to the duration of silence into the model, ensuring the independence and objectivity of the model's judgment.

[0059] (3) Ultra-long silent fallback phase (cumulative duration of continuous blanks ≥ second preset threshold, preferably 3000 ms): When the duration of continuous silence reaches a second preset threshold, the system activates a hard fallback correction strategy. This strategy does not participate in the reasoning process of the large language model, but only applies to the final decision output layer: if the predicted label output by the model in the current round is semantically incomplete ( <incomplete>If the tag is not fully semantically complete, then the tag will be forcibly corrected. <complete>), and immediately trigger a downstream system response; if the model has already output other explicit labels (such as <complete> 、 <backchannel> 、 <wait>If the forecast remains unchanged, the fallback strategy will not be implemented.

[0060] In the aforementioned three-level scheduling mechanism, the real-time recognition link consisting of the encoder and CTC continuously provides streaming text prefixes and Blank accumulation time, while the talk-turn state prediction link consisting of the encoder, adapter, and large language model is only awakened to perform inference during the effective silence phase. The long silence fallback strategy only takes effect when there is a conflict between model prediction and physical timeout, thus ensuring that the system never "wait indefinitely" while maximizing the integrity of the independent decision-making of the large language model based on multimodal information in normal scenarios, taking into account both the intelligence and robustness of the interaction.

[0061] The Zipformer encoder used in the front end of the large language model is an ASR model trained on CTC. Its CTC decoder has a frame shift of 40 ms / frame, and the absolute silence duration corresponding to the number of consecutive Blank tags N is T = N × 40 ms. The first preset threshold of 320 ms corresponds to 8 consecutive Blank tags, and the second preset threshold of 3000 ms corresponds to 75 consecutive Blank tags. Blank tags correspond to frames with "no effective output" in speech, naturally modeling silent states such as pronunciation gaps, pauses in thought, and sentence endings.

[0062] The streaming ASR employs a CTC greedy decoding strategy, predicting tags frame-by-frame from the acoustic feature sequence output for each audio block. Non-Blank tags are concatenated chronologically, and consecutive repeating tags are merged to form the streaming text output. This text prefix is ​​incrementally updated with each arriving audio block and cached in system memory for use during large language model inference.

[0063] To explain in detail the principle of the technical solution of the present invention, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0064] In view of the shortcomings of existing technologies, this invention proposes a TailTurn speech-semantic fusion turn-based state prediction method based on a large model. For example... Figure 4 The diagram illustrates the overall architecture principle of this invention. In some real-time scenarios, the core innovation of this invention lies in: randomly masking the last words of the corresponding audio content during training and requiring the model to complete them. This forces the large language model to actively perceive the acoustic information output by the audio encoder during text completion, implicitly establishing the alignment of acoustics and semantics in the language space. This mechanism enables the model to combine the last acoustic cues with existing semantic cues, accurately predicting turn states within a unified generation framework. The method of this invention includes three levels: model architecture design, training method, and inference method. Specifically, it can be implemented as follows: (a) Model Architecture: The TailTurn model architecture proposed in this invention comprises the following three core components: 1. Audio encoder. The encoder part of the pre-trained Zipformer automatic speech recognition model is used to extract high-level acoustic feature sequences from the input audio. During training, acoustic features are dynamically extracted in windows such as 160ms, 320ms, and 640ms, and the parameters are frozen and do not participate in gradient updates.

[0065] 2. Adapter Module. This module consists of a 4-layer Transformer network, connecting the audio encoder and the large language model. The adapter maps the acoustic feature sequence output by the audio encoder to the word embedding space of the large language model, obtaining an acoustic embedding sequence. The adapter parameters are trainable, transforming continuous acoustic representations into vectors that the large language model can understand.

[0066] 3. Large Language Model (LLM). This employs a pre-trained decode-only architecture (a decoder-only architecture, characterized by containing only the decoder part of the Transformer, generating text word-by-word through an autoregressive mechanism). It receives concatenated Prompts, acoustic embeddings, and text lexical embeddings, generating the target sequence in an autoregressive manner. The LLM injects low-rank adaptive parameters (LoRA) into some training phases for efficient fine-tuning; the main parameters are either frozen or trained with all parameters, depending on the training phase.

[0067] The parameter strategy of this architecture is as follows: the audio encoder is always frozen; the large language model is trained with all parameters in stage one, and the main parameters are frozen in stages two and three, with only LoRA weights being updated; the adapter participates in training in all stages.

[0068] (II) Training methods: The training process is carried out in three progressive stages, gradually endowing the model with text completion capabilities, turn-based state prediction capabilities, and interpretable reasoning capabilities.

[0069] Phase 1: Acoustic-driven text completion training; The goal of Phase 1 is to teach the model to complete the deliberately obscured end text based on the audio content, thereby forcing the model to utilize the acoustic information provided by the audio encoder, enabling the large language model to establish the ability to "complete text by listening to audio." During this phase, the Zipformer audio encoder is frozen, and the adapter and large language model undergo full-parameter training.

[0070] The training data uses an automated speech recognition corpus, with each sample containing both audio and its complete transcribed text. The text is randomly truncated at the end according to a preset ratio to construct three text granularities: Full Text Mode: Provides all text terms; the target completion segment is empty. Partial text mode: Randomly truncate the last 30% to 70% of the words, keeping only the incomplete text as input, and the target completion segment is the truncated last words; No text mode: No text terms are provided; the target completion segment is a complete text transcription.

[0071] The sampling probabilities for the three modes are 10%, 40%, and 50%, respectively.

[0072] During training, the acoustic embeddings of the audio are input sequentially after the Prompt text.<asr_text> The model embeds the tags, followed immediately by the actual provided text input. It then predicts the masked tail terms in an autoregressive manner and outputs them at the end.< / asr_text> Labeling. The training loss uses the standard cross-entropy loss.

[0073] Prompt template: Complete based on audio.<asr_text> Subsequent transcription content and< / asr_text> Finish.

[0074] Phase Two: Training on Turn-Taking Prediction; Based on the model trained in Phase 1, a turn-based state prediction task is introduced. In this phase, the parameters of the Zipformer encoder and the main parameters of the large language model are frozen, and only the adapter module and the LoRA (Low-Rank Adaptation) parameters injected into the self-attention layer of the large language model are trained.

[0075] The training data uses a manually annotated turn-taking state dataset. Each sample contains audio, the corresponding text transcription, and a turn-taking state label s∈S. The turn-taking state label set S is defined as five special tokens: <complete>Indicates semantic completeness. <incomplete>This indicates that the meaning is incomplete. <wait>This indicates a request to suspend or terminate the communication. <backchannel>Indicates a brief agreement, <invalid>Indicates invalid dialogue (such as nonverbal acoustic events like coughing, laughter, or ambient noise).

[0076] During training, the text masking strategy from Phase 1 was used, while the Prompt template was expanded to: Complete the audio based on the audio.<asr_text> Subsequent transcription content and< / asr_text> End, then output the talk round state, state definition: <complete>Indicates semantic completeness. <incomplete>This indicates that the meaning is incomplete. <wait>This indicates a request to suspend or terminate the communication. <backchannel>Indicates a brief agreement, <invalid>This indicates an invalid conversation. The training loss consists of two parts: (1) Cross-entropy loss : Calculate the loss of the model in generating the entire target sequence; (2) Penalty loss To ensure that the model distributes the probability mass set to the five valid turn state labels at the end of the sequence, the penalty loss is defined as:

[0077] in To prevent the logarithm from being an infinitely small constant, i.e., the total probability of the turn-based status tags... When it approaches 1, the penalty loss Approaching 0; conversely, increasing the penalty loss. This guides the model to assign the predicted probability of the final turn state label to the turn state label, completely eliminating the model's evasive behavior of generating irrelevant tokens or terminating prematurely at the decision point.

[0078] Therefore, the total loss , It is the penalty coefficient: .

[0079] Phase Three: Mind Chain Enhancement Training; To further enhance the model's interpretable reasoning capabilities in complex scenarios, we performed thought chain enhancement training. During this phase, we continued to freeze the Zipformer encoder and the main body of the large language model, training only the adapter and LoRA parameters.

[0080] For approximately 30% of the training samples, an insertion is made between the Prompt and the target sequence. <think> and< / think> The wrapped thought chain reasoning text. The reasoning text, generated manually or automatically, describes the analysis process for determining turn-taking states based on audio ending prosody and text semantic completeness. The target sequence format becomes: <think> chain of thought< / think> Complete the content< / asr_text><turn_label> ; Specifically, the thought chain text can be generated using the following automated rules: (1) Acoustic dimension: detect the fundamental frequency slope (F0 slope) at the end of the audio. If F0 shows a downward trend and the energy decay exceeds the preset threshold, the description "the tail tone drops significantly" is generated; if F0 shows an upward trend or the throat sound is obvious, the description "the tail tone rises, as if there is still not enough" is generated.

[0081] (2) Semantic dimension: The depth of the dependency syntax tree and the completeness of the argument structure of the core predicate are analyzed using the semantic big model. If the core argument is missing, a description of "incomplete semantic components" is generated.

[0082] (3) Comprehensive judgment: The above acoustic and semantic descriptions are combined into natural language reasoning text, such as "the audio tail tone shows a downward trend and is accompanied by a long pause, the subject-verb-object structure of the current sentence is complete, and it is judged to be semantically complete", thus forming an interpretable reasoning basis.

[0083] Training employs a teacher-mandated approach, where the model performs explicit reasoning before outputting complete content and turn-based state labels. For samples without thought chains, the target sequence format is consistent with Phase Two. Loss calculation covers the autoregressive generation loss across the entire extended sequence, and the penalty loss from Phase Two is also applied. This phase grants the model optional reasoning capabilities; during inference, the generation of parts can be controlled via a system switch, enabling on-demand activation of interpretability.

[0084] (III) Reasoning Methods: During inference, the model receives user audio and optional streaming automatic speech recognition text prefixes. Using the Prompt template from training phase two, the large language model generates an output sequence in an autoregressive manner. Completed text (if present) and a final turn-based state tag token are parsed from the generated sequence, triggering corresponding system actions.

[0085] This invention supports two inference modes: Mode 1 (Pure Audio Inference): Only audio is input, without any text. The large language model directly generates complete content text and turn state labels based solely on acoustic prefix embeddings.

[0086] Mode 2 (Audio + Streaming ASR Text Inference): Simultaneously input audio and potentially incomplete text output in real time by streaming automatic speech recognition. The large language model uses this text as a prefix and combines acoustic embedding to generate complete text and turn status labels.

[0087] The core value of Mode 2 lies in the fact that when streaming automatic speech recognition loses trailing words due to delay, the model uses acoustic information from the audio trailing words to complete the missing content, and jointly judges the turn state based on the complete semantics and the original acoustic cues. This "completeness-prediction" mechanism provides implicit calibration when there is a conflict between acoustics and semantics: if the audio has a falling tone or a long pause, even if the automatic speech recognition text is incomplete, the model will complete the trailing words and output the result. <complete>If the audio has a rising pitch and a strained tone, even if the text appears complete in the automatic speech recognition, the model tends to output the wrong message. <incomplete>The completion behavior itself becomes an intermediate task for the model to integrate acoustic and semantic cues, allowing conflict resolution to be completed naturally in the language space. Furthermore, this model significantly reduces the delay in predicting turn states by reusing ASR inference text.

[0088] During reasoning, the model can choose whether to enable thought chain output. If enabled, the model first generates the reasoning text, then performs completion and turn prediction to provide interpretable evidence for decision-making; if disabled, the model directly outputs the completed text and turn labels, resulting in lower latency.

[0089] In summary, this invention aims to improve the real-time voice interaction experience. By training a large model with tail completion to integrate speech and semantic information, the model must actively perceive the tail acoustic prosody information output by the audio encoder when predicting turn states, and make joint decisions with existing semantic cues. This allows for accurate prediction of turn states even in scenarios where the text of streaming automatic speech recognition is incomplete or there are acoustic-semantic conflicts, while ensuring that the model outputs clear turn decisions.

[0090] Compared with the prior art, the present invention has at least the following beneficial effects: 1. Forced Perceptual Acoustics: By randomly masking the end of the text and having the model complete it, the shortcut of relying solely on text semantics for judgment is cut off. This forces large language models to pay attention to the prosodic information of the end provided by the audio encoder, fundamentally solving the "text shortcut" problem in speech-semantic turn prediction.

[0091] 2. Multi-stage progressive training: Stage 1 establishes acoustic-text mapping; Stage 2 introduces turn-based state decision-making and applies penalty loss to ensure stable output, guaranteeing that each prediction is an effective decision and greatly improving the reliability of actual deployment; Stage 3 injects thought chains to enhance interpretability, with clear tasks and stable convergence in each stage.

[0092] 3. Robust in streaming scenarios: The model uses dynamic windows to extract acoustic embeddings during training, and a multi-granularity text feeding strategy (complete / partial / no text) directly simulates various states of streaming automatic speech recognition. The model can adapt to changes in text completeness and maintain high accuracy even when the last word of the automatic speech recognition is lost.

[0093] 4. Explainable reasoning ability: Through the enhanced training of the thought chain in stage three, the model can output the reasoning process in complex acoustic-semantic conflict scenarios, providing transparent basis for system debugging and user experience, without increasing the overhead of normal reasoning.

[0094] like Figure 5 As shown, this embodiment of the invention also provides a TailTurn speech-semantic fusion turn-based state prediction device 900, which can implement the above-described method. This device may include: The sample acquisition module 901 is used to acquire a training sample set; wherein, the training sample set includes automatic speech recognition corpus samples and manually annotated turn-taking state samples, the automatic speech recognition corpus samples include a first sample audio and a first text transcription; the manually annotated turn-taking state samples include a second sample audio, a second text transcription, and turn-taking state labels; Model configuration module 902 is used to configure the TailTurn model; wherein, the TailTurn model includes an audio encoder, an adapter module and a large language model; the audio encoder is used to extract acoustic feature sequences from the input audio; the adapter module is used to map the acoustic feature sequences to the word embedding space of the large language model to obtain acoustic embedding sequences; the large language model is used to generate target sequences based on acoustic embedding sequences; The completion training module 903 is used to randomly truncate the first text transcription based on the automatic speech recognition corpus samples, and then combine the first sample audio to perform acoustically driven text completion training on the TailTurn model to obtain the first training model. The prediction training module 904 is used to predict the turn-by-turn state of the first training model based on manually labeled turn-by-turn state samples and introduce turn-by-turn state labels to obtain the second training model. The enhanced training module 905 is used to perform enhanced training on the second training model based on manually annotated turn-taking state samples combined with preset thought chain reasoning text, so as to obtain the target model. Model application module 906 is used to process the turn state prediction of the speech data to be predicted using the target model.

[0095] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0096] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0097] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0098] like Figure 6 As shown, Figure 6 The hardware structure of an electronic device 1000 according to another embodiment is illustrated. The electronic device 1000 includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (aSIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 1002 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RaM). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0099] The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0101] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0102] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0103] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0104] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0105] The TailTurn speech-semantic fusion turn-taking state prediction method, apparatus, electronic device, storage medium, and program product provided in this invention obtain a training sample set. The training sample set includes automatically recognized speech corpus samples and manually annotated turn-taking state samples. The automatically recognized speech corpus samples include a first sample audio and a first text transcription. The manually annotated turn-taking state samples include a second sample audio, a second text transcription, and turn-taking state labels. A TailTurn model is configured. The TailTurn model includes an audio encoder, an adapter module, and a large language model. The audio encoder extracts acoustic feature sequences from the input audio. The adapter module maps the acoustic feature sequences to... The acoustic embedding sequence is obtained by mapping the word embedding space of the large language model. The large language model is used to generate the target sequence based on the acoustic embedding sequence. Based on the automatic speech recognition corpus samples, the first text transcription is randomly truncated, and then combined with the first sample audio to perform acoustically driven text completion training on the TailTurn model to obtain the first training model. Based on manually labeled turn-taking state samples, turn-taking state labels are introduced to train the first training model for turn-taking state prediction to obtain the second training model. Based on manually labeled turn-taking state samples and pre-set thought chain reasoning text, the second training model is subjected to thought chain enhancement training to obtain the target model. The target model is used to process the turn-taking state prediction of the speech data to be predicted. This invention leverages a large language model to jointly represent acoustic embedding sequences and textual information, enhancing multimodal fusion capabilities. It randomly truncates the first text transcription based on automatic speech recognition corpus and combines it with first sample audio for acoustically driven text completion training. This forces the model to use audio to complete semantics when text is missing, avoiding reliance solely on text for judgment and enhancing the joint modeling capabilities of acoustics and semantics. It introduces manually labeled turn-taking states for turn-taking state prediction training, enabling the model to output explicit turn-taking states. Furthermore, it incorporates pre-defined thought chain reasoning text for enhanced training, improving interpretability in complex scenarios. Therefore, when processing speech data to be predicted, the target model maintains high prediction accuracy in scenarios with incomplete streaming text or acoustic-semantic conflicts, improving the robustness and naturalness of real-time voice interaction.

[0106] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0107] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0108] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0109] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0110] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.< / incomplete> < / complete> < / invalid> < / backchannel> < / wait> < / incomplete> < / complete> < / invalid> < / backchannel> < / wait> < / incomplete> < / complete> < / wait> < / backchannel> < / complete> < / complete> < / incomplete> < / wait> < / incomplete> < / complete> < / invalid> < / backchannel> < / wait> < / incomplete> < / complete> < / invalid> < / backchannel> < / wait> < / incomplete> < / complete> < / invalid> < / backchannel> < / wait> < / incomplete> < / complete>

Claims

1. A TailTurn speech-semantic fusion turn-based state prediction method, characterized in that, The method includes the following steps: Obtain a training sample set; wherein the training sample set includes automatic speech recognition corpus samples and manually annotated turn state samples, the automatic speech recognition corpus samples include a first sample audio and a first text transcription; the manually annotated turn state samples include a second sample audio, a second text transcription, and turn state labels; Configure a TailTurn model; wherein the TailTurn model includes an audio encoder, an adapter module, and a large language model; the audio encoder is used to extract acoustic feature sequences from input audio; the adapter module is used to map the acoustic feature sequences to the word embedding space of the large language model to obtain acoustic embedding sequences; the large language model is used to generate target sequences based on the acoustic embedding sequences; Based on the automatic speech recognition corpus samples, the first text transcription is randomly truncated, and then the TailTurn model is acoustically driven to complete the text by combining the first sample audio, thus obtaining the first training model. Based on the manually labeled turn-around state samples, the turn-around state labels are introduced to train the first training model for turn-around state prediction, thereby obtaining the second training model. Based on the manually labeled turn-taking state samples and the preset thought chain reasoning text, the second training model is subjected to thought chain enhancement training to obtain the target model; The target model is used to process the turn-taking state prediction of the speech data to be predicted.

2. The method according to claim 1, characterized in that, Configuring the TailTurn model includes the following steps: Obtain a pre-trained automatic speech recognition model, and extract the encoder part of the automatic speech recognition model as the audio encoder; The adapter module is obtained based on a multi-layer Transformer network configuration; The large language model is a pre-trained decoder-only architecture. The audio encoder, the adapter module, and the large language model are connected sequentially to obtain the TailTurn model; The parameter training strategy of the TailTurn model includes: the audio encoder is always frozen; the adapter module always participates in training; the large language model performs full parameter training in the text completion training, and freezes the main parameters and only updates the low-rank adaptation parameters in the turn state prediction training and the thought chain enhancement training.

3. The method according to claim 1, characterized in that, The process of randomly truncating the first text transcription based on the automatic speech recognition corpus samples, and then combining the first sample audio to perform acoustic-driven text completion training on the TailTurn model to obtain the first training model includes the following steps: Based on the first sampling probability, the automatic speech recognition corpus samples are collected from the training sample set as first candidate samples, and all the first candidate samples are aggregated into a first complete text pattern sample. Based on the second sampling probability, the automatic speech recognition corpus samples are collected from the training sample set as second candidate samples. The last word of the first text transcription in the second candidate samples is randomly truncated according to a preset ratio to update the first text transcription of the second candidate samples, and then the first part of the text pattern samples are summarized. Based on the third sampling probability, the automatic speech recognition corpus samples are collected from the training sample set as the third candidate samples. The first text transcription in the third candidate samples is removed, and then the first textless mode samples are obtained by summarizing. The first complete text pattern sample, the first partial text pattern sample, and the first no-text pattern sample are summarized into text completion training samples. The first acoustic feature sequence of the first sample audio of each text completion training sample is extracted using the audio encoder of the TailTurn model; Based on the first acoustic feature sequence, the first acoustic embedding sequence is obtained by processing it using the adapter module of the TailTurn model; The first text transcription of each text completion training sample is embedded to obtain the first text word embedding; wherein, if the text completion training sample does not have the first text transcription, the corresponding first text word embedding is configured to be empty; The first acoustic embedding sequence, the first text lexical embedding, and the preset first prompt text of the same text completion training sample are concatenated and input into the large language model of the TailTurn model, so that the large language model can obtain the first completed text through autoregression. Based on the first completed text and its corresponding first text transcription in the automatic speech recognition corpus samples, the adapter module of the TailTurn model and the large language model are optimized by cross-entropy loss to obtain the first training model.

4. The method according to claim 1, characterized in that, The process of training the first training model to predict the turn-around state based on the manually labeled turn-around state samples, and then introducing the turn-around state labels to obtain the second training model, includes the following steps: Based on the fourth sampling probability, the manually annotated turn-taking state samples are collected from the training sample set as fourth candidate samples, and all the fourth candidate samples are aggregated into a second complete text pattern sample. Based on the fifth sampling probability, the manually annotated turn-taking state samples are collected from the training sample set as the fifth candidate samples. The last word of the second text transcription in the fifth candidate samples is randomly truncated according to a preset ratio to update the second text transcription of the fifth candidate samples, and then the second part of the text pattern samples are summarized. Based on the sixth sampling probability, the manually annotated turn-taking state samples are collected from the training sample set as the sixth candidate samples. The second text transcription in the sixth candidate samples is removed, and then the second textless mode samples are obtained by summarizing. The second complete text pattern sample, the second partial text pattern sample, and the second no-text pattern sample are summarized into state prediction training samples; The second acoustic feature sequence of the second sample audio of each state prediction training sample is extracted using the audio encoder of the first training model; Based on the second acoustic feature sequence, the second acoustic embedding sequence is obtained by processing the adapter module of the first training model; The second text transcription of each state prediction training sample is embedded to obtain a second text word embedding; wherein, if the state prediction training sample does not have a second text transcription, the corresponding second text word embedding is configured to be empty; The second acoustic embedding sequence, the second text lexical embedding, and the preset second prompt text of the same state prediction training sample are concatenated and input into the large language model of the first training model, so that the large language model can process and obtain the second complete text and the predicted turn state; wherein, the predicted turn state includes the probability value of each turn state type; First comparison data is constructed based on the second completed text and its corresponding second text transcription in the manually labeled turn-around state sample; second comparison data is constructed based on the predicted turn-around state and the turn-around state label; and a first loss is constructed based on the first comparison data and the second comparison data using cross-entropy loss. Based on the sum of the probability values ​​of all the turn-by-turn state types in the predicted turn-by-turn state, a second loss is constructed using the penalty loss; The first loss and the second loss are weighted and summed to obtain the total loss. The second training model is obtained by optimizing the low-rank adaptation parameters of the adapter module of the first training model and the self-attention layer of the large language model based on the total loss.

5. The method according to claim 1, characterized in that, The step of performing thought chain enhancement training on the second training model based on the manually annotated turn-taking state samples and the preset thought chain reasoning text to obtain the target model includes the following steps: A portion of the manually labeled turn-taking state samples are collected from the training sample set as augmented training samples; The third acoustic feature sequence of the second sample audio of each of the enhanced training samples is extracted using the audio encoder of the second training model; Based on the third acoustic feature sequence, the adapter module of the second training model is used to process and obtain the third acoustic embedding sequence; The second text transcription of each of the enhanced training samples is embedded to obtain the third text lexical embedding; The third acoustic embedding sequence, the third text lexical embedding, the preset second prompt text, and the thought chain reasoning text of the same enhanced training sample are concatenated and input into the large language model of the second training model. Then, the large language model is trained in a teacher-forced manner so that it performs explicit reasoning before outputting the result, thereby obtaining the trained target model.

6. The method according to claim 1, characterized in that, The process of predicting the turn state of the speech data to be predicted using the target model includes the following steps: Acquire the speech data to be predicted; wherein the speech data to be predicted includes user audio and streaming automatic speech recognition text prefix; The user audio is input into the audio encoder of the target model to extract the target acoustic feature sequence; The target acoustic feature sequence is input into the adapter module of the target model to obtain the target acoustic embedding sequence; The target text word embedding is obtained by embedding the text prefix of the streaming automatic speech recognition. Based on the target acoustic embedding sequence, the target text lexical embedding, and the preset second prompt text, the target output sequence is generated through an autoregressive method according to the large language model of the target model; The target completion text and target turn status label are parsed from the output sequence.

7. The method according to claim 6, characterized in that, The step of generating a target output sequence based on the target acoustic embedding sequence, the target text lexical embedding, and a preset second prompt text, using an autoregressive approach according to the large language model of the target model, includes the following steps: If the thought chain output is enabled, the target acoustic embedding sequence, the target text lexical embedding, the preset second prompt text and the thought chain reasoning text are concatenated and input into the large language model of the target model, so that the large language model performs explicit reasoning before outputting the result, obtains reasoning text, and then drives the large language model of the target model to generate the target output sequence through the reasoning text; If the thought chain output is not enabled, the target acoustic embedding sequence, the target text lexical embedding, and the second prompt text are concatenated and input into the large language model of the target model, so that the large language model generates the target output sequence through an autoregressive method.

8. A TailTurn speech-semantic fusion turn-based state prediction device, characterized in that, The device includes: The sample acquisition module is used to acquire a training sample set; wherein, the training sample set includes automatic speech recognition corpus samples and manually annotated turn state samples, the automatic speech recognition corpus samples include a first sample audio and a first text transcription; the manually annotated turn state samples include a second sample audio, a second text transcription, and turn state labels; A model configuration module is used to configure the TailTurn model; wherein, the TailTurn model includes an audio encoder, an adapter module, and a large language model; the audio encoder is used to extract acoustic feature sequences from input audio; the adapter module is used to map the acoustic feature sequences to the word embedding space of the large language model to obtain acoustic embedding sequences; the large language model is used to generate target sequences based on the acoustic embedding sequences; The completion training module is used to randomly truncate the first text transcription based on the automatic speech recognition corpus samples, and then combine the first sample audio to perform acoustically driven text completion training on the TailTurn model to obtain the first training model. The prediction training module is used to perform turn-by-turn state prediction training on the first training model based on the manually labeled turn-by-turn state samples and introduce the turn-by-turn state labels to obtain the second training model. The enhanced training module is used to perform enhanced training on the second training model based on the manually annotated turn-taking state samples and the preset thought chain reasoning text, so as to obtain the target model. The model application module is used to process the turn-taking state prediction of the speech data to be predicted using the target model.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.