Voice interaction method, voice interaction model training method, device, and storage medium

Abstract

Embodiments of the present application provide a voice interaction method, a training method of a voice interaction model, a device and a storage medium. The voice interaction method is applied to a terminal device deployed with an end-to-end voice interaction model, and includes: obtaining a voice signal; encoding the voice signal into an embedding sequence aligned with a text representation space of a decoding part through a voice coding part; performing semantic understanding on the embedding sequence in the same inference process through the decoding part, and jointly generating a reply text and structured label information carrying multiple information labels based on the semantic understanding result; generating and playing a reply voice according to the reply text, and triggering a corresponding interaction response operation according to the information label. This makes semantic understanding and reply generation complete at a time in a single model, avoids error accumulation of multi-model cascading, cooperates with end-side local execution, reduces interaction delay and resource occupation, and reduces data security pressure.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a voice interaction method, a training method for a voice interaction model, a device, and a storage medium. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, AI-based voice interaction is becoming increasingly natural and fluent. Voice interaction functions are being applied to various smart devices such as smart speakers, wearable devices, and in-vehicle terminals, and higher requirements are being placed on the real-time performance, accuracy, privacy, and personalization of the interaction.

[0003] Currently, voice interaction is mainly implemented in two ways. One is a multi-model cascade scheme, which consists of multiple independent models connected in series to form a processing chain. For example, speech recognition, semantic understanding (such as emotion recognition and intent recognition), response generation, and speech synthesis are handled sequentially by multiple independent models. The other is a cloud-based Application Programming Interface (API) call scheme, where the device on the edge collects the user's speech and transcribes it into text, then calls cloud models over the network to complete semantic understanding and response generation, and finally returns the results to the device on the edge for playback.

[0004] However, multi-model cascading solutions suffer from several drawbacks: processing errors at any stage propagate and accumulate, and the simultaneous loading of multiple models leads to high memory and computing power consumption. Cloud API call solutions, on the other hand, suffer from high response latency and data security issues. Therefore, an effective solution is urgently needed to address these problems. Summary of the Invention

[0005] This application provides a voice interaction method, a voice interaction model training method, a device, and a storage medium to complete semantic understanding and response generation of voice signals in the same inference process within a single model, reducing error accumulation caused by multi-stage processing, and lowering interaction latency and device resource consumption.

[0006] In a first aspect, embodiments of this application provide a voice interaction method applied to a terminal device, wherein an end-to-end voice interaction model is deployed on the terminal device. The method includes: acquiring a voice signal; encoding the voice signal into an embedding sequence through the voice encoding part of the voice interaction model, wherein the embedding sequence is aligned with the text representation space processed by the decoding part of the voice interaction model; performing semantic understanding on the embedding sequence in the same inference process through the decoding part, and jointly generating a response text and structured tag information based on the semantic understanding result, wherein the structured tag information carries multiple information tags in a machine-readable structure; generating and playing a response voice based on the response text, and triggering a corresponding interactive response operation based on the information tags in the structured tag information.

[0007] Secondly, embodiments of this application provide a training method for a voice interaction model. The initial model to be trained includes a voice encoding part and a decoding part. The method includes: generating multiple sample query information carrying information tags according to prompt information corresponding to various information tags; generating sample response text according to the sample query information; constructing sample structured label information carrying the information tags in a machine-readable structure according to the information tags carried in the sample query information; using the sample response text and the sample structured label information together as the annotation data corresponding to the sample query information; inputting the sample query information into the decoding part of the initial model, and having the decoding part jointly output predicted response text and predicted structured label information as predicted data; updating the weights of the decoding part according to the loss between the predicted data and the annotation data to obtain an end-to-end voice interaction model, wherein the voice interaction model is used to jointly output response text and structured label information in the same inference process.

[0008] Thirdly, embodiments of this application also provide a voice interaction device applied to a terminal device, wherein an end-to-end voice interaction model is deployed on the terminal device, and the device includes: an acquisition module for acquiring a voice signal; an encoding module for encoding the voice signal into an embedding sequence through the voice encoding part of the voice interaction model, wherein the embedding sequence is aligned with the text representation space processed by the decoding part of the voice interaction model; a generation module for performing semantic understanding on the embedding sequence in the same inference process through the decoding part, and jointly generating a response text and structured tag information based on the semantic understanding result, wherein the structured tag information carries multiple information tags in a machine-readable structure; and an interaction module for generating and playing a response voice according to the response text, and triggering a corresponding interactive response operation according to the information tags in the structured tag information.

[0009] Fourthly, this application also provides a training device for a voice interaction model. The initial model to be trained includes a voice encoding part and a decoding part. The device includes: a first generation module, used to generate multiple sample query information carrying information tags according to prompt information corresponding to multiple information tags; a second generation module, used to generate sample response text according to the sample query information; a construction module, used to construct sample structured label information carrying the information tags in a machine-readable structure according to the information tags carried by the sample query information; an annotation module, used to use the sample response text and the sample structured label information together as annotation data corresponding to the sample query information; a training module, used to input the sample query information into the decoding part of the initial model, and the decoding part jointly outputs predicted response text and predicted structured label information as prediction data; and an update module, used to update the weights of the decoding part according to the loss between the predicted data and the annotation data to obtain an end-to-end voice interaction model, wherein the voice interaction model is used to jointly output response text and structured label information in the same inference process.

[0010] Fifthly, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the voice interaction method provided in embodiments of this application, or implements the training method for the voice interaction model provided in embodiments of this application.

[0011] Sixthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the voice interaction method provided in embodiments of this application, or implements the training method for the voice interaction model provided in embodiments of this application.

[0012] Seventhly, embodiments of this application also provide a computer program product, including a computer program, which, when executed by a processor, implements the voice interaction method provided in embodiments of this application, or implements the training method for the voice interaction model provided in embodiments of this application.

[0013] The technical solution provided in this application adopts an end-to-end voice interaction model. The voice encoding part encodes the voice signal into an embedded sequence aligned with the text representation space. The decoding part of the end-to-end voice interaction model performs semantic understanding on the embedded sequence in the same inference process, and jointly generates response text and structured tag information carrying multiple information tags based on the semantic understanding results. This allows the semantic understanding and response generation of the voice signal to be completed in a single inference process within a single model, avoiding the problems of error accumulation and context fragmentation that occur when multiple independent models are processed in series. Information tags in the voice are preserved and directly applied to response generation, and the information tags and response content are naturally consistent. Simultaneously, the terminal device does not need to load multiple models simultaneously, reducing memory and computing power consumption. Combined with local execution on the terminal side, it shortens the interaction response latency. Voice data does not need to be uploaded, reducing the data security pressure caused by data transmission.

[0014] Alternatively, by introducing a switching mechanism between the first and second decoding constraints during the autoregressive decoding process, the decoding part first generates the response text in natural language form under the first decoding constraint in the same inference process. After detecting the preset constraint switching trigger information, it switches to the second decoding constraint and generates structured label information within the limited candidate word range. This ensures both the natural and fluent expression of the response text and the legality and parsability of the structured label information format, avoiding the pollution of structured content by natural language content.

[0015] Alternatively, during model training, the weights of the decoding part are updated using labeled data composed of sample response text and sample structured label information. This enables the model to learn to jointly output response text and structured label information carrying information labels while learning dialogue capabilities, providing a capability foundation for the model's joint generation during the reasoning process. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A schematic diagram illustrating an application scenario provided in an embodiment of this application; Figure 2 A flowchart illustrating a voice interaction method provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the process of jointly generating response text and structured tag information provided in this application embodiment; Figure 4 A schematic diagram of an output stage state machine and a restricted decoding process provided for an embodiment of this application; Figure 5 This is a schematic diagram of an end-side deployment structure provided in an embodiment of this application; Figure 6 A flowchart illustrating a training method for a voice interaction model provided in an embodiment of this application; Figure 7 A block diagram illustrating a training data construction process provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a voice interaction device provided in an embodiment of this application; Figure 9 A schematic diagram of the structure of a training device for a voice interaction model provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] In this embodiment, "at least one" refers to one or more; "and / or" describes the association relationship between related objects, indicating three possible relationships: A and / or B, which means: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0019] It should be noted that, in the cases involving user information in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with relevant laws, regulations, and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards.

[0020] To facilitate understanding of the technical solutions in the embodiments of this application, the technical concepts involved in the embodiments of this application will be explained first.

[0021] Speech signal: refers to audio data acquired through sound waves, carrying the language content of the speaker, as well as information such as tone, speed, and pitch. For example, a speech signal can be the waveform data of a user's speech acquired by a terminal device through a microphone.

[0022] An end-to-end voice interaction model refers to a neural network model that completes the process from inputting a voice signal to outputting response text and structured label information within a single model. It includes a speech encoding part and a decoding part. The speech encoding part encodes the voice signal into an embedding sequence; the decoding part uses the embedding sequence as input for semantic understanding and jointly generates the response text and structured label information.

[0023] An embedding sequence is a sequence of multiple embedding vectors arranged in a temporal order. In this embodiment, the embedding sequence output by the speech coding part is aligned with the text representation space processed by the decoding part. This means that each embedding vector in the embedding sequence and the vector representation of the text word are located in the same representation space, enabling the decoding part to process the embedding vectors corresponding to the speech signal in the same way as processing text words, without first transcribing the speech signal into text.

[0024] A token is the smallest unit of output generated by the decoding part. A token can correspond to a character, a word, a subword, or a symbol. The decoding part produces one token for each decoding step.

[0025] A single inference process refers to a continuous semantic understanding and content generation computation performed by the decoding part on a single input embedding sequence, without switching model weights, re-encoding speech, or calling external independent classification modules. In this process, the decoding part, based on a unified current semantic state (such as hidden layer state, context vectors, etc.), jointly produces the response text and structured label information through a single generation mechanism (including but not limited to autoregressive decoding, non-autoregressive parallel decoding, or hybrid decoding strategies). Both share the same semantic understanding foundation, constituting the complete output of this computation process, rather than being generated separately by multiple independent inference instances.

[0026] In one optional implementation, autoregressive decoding is employed. A single inference process refers to a continuous autoregressive decoding process performed by the decoding part on a single input embedding sequence. Jointly generating the response text and structured label information within the same inference process means that both are produced jointly by that autoregressive decoding process, rather than being produced separately by different models or different inference processes.

[0027] Autoregressive decoding refers to a decoding method in which the decoding part generates output content word by word. In this decoding method, each word is generated and then the generated word is concatenated into the decoding context. The next word is then predicted based on the updated decoding context. This process is repeated until the generation is complete.

[0028] The current hidden state refers to the internal representation vector calculated by the decoding part during the decoding process (e.g., autoregressive decoding). It reflects the semantic understanding of the embedding sequence and the currently generated words by the decoding part. The current hidden state is dynamically updated with each decoding step.

[0029] Response text: refers to the natural language text generated by the decoding part to respond to the voice signal, which is then converted into response speech and played back to the user.

[0030] Structured tag information refers to information organized in a machine-readable structure, consisting of at least one tag field and the values ​​of each tag field organized according to a uniform preset format, used to carry various information tags.

[0031] Information tags: These refer to the tag content carried by structured tag information and identified based on speech signals. Multiple information tags include at least two of the following: emotion tags, intent tags, and instruction tags. Emotion tags characterize the emotional state of the person emitting the speech signal, such as happy, sad, or calm; intent tags characterize the interactive intent corresponding to the speech signal, such as movie / TV show information, weather forecasts, or casual conversation; and instruction tags indicate the device control operation to be performed, such as adjusting volume, playing music, or performing a physical action.

[0032] Interactive response operations refer to the interaction-related operations triggered by the terminal device based on the information tags in the structured tag information, including but not limited to: adjusting the voice content and / or expression style of the reply voice based on the emotion tag, determining and executing subsequent interactive tasks based on the intent tag, and executing corresponding device control operations based on the instruction tag.

[0033] Decoding constraints refer to the constraints imposed on the generation range or generation method of the next lexical unit during the decoding process (e.g., autoregressive decoding). In the embodiments of this application, the first decoding constraint is the decoding constraint used when generating the response text, and the second decoding constraint is the decoding constraint used when generating structured tag information.

[0034] Preset constraint switching trigger information: refers to the trigger information used to indicate the switch from the first decoding constraint to the second decoding constraint, that is, the information used to determine that the response text has been generated and the generation of structured tag information can begin.

[0035] Preset stage switching lexical units: These are predefined special lexical units used to indicate the end of the generation of the response text and / or the start of the generation of structured tag information, such as pre-added separator lexical units in the vocabulary.

[0036] Preset end marker: This refers to a marker used to indicate the end of the generation of the reply text, such as a sentence-level terminator.

[0037] Semantic completeness score: refers to the score used to measure whether the currently generated response text is semantically complete and whether it fully responds to the content expressed by the speech signal.

[0038] Attention coverage: refers to the degree to which the semantic representation of the currently generated response text covers the embedded sequence. It is calculated based on the attention weight of the decoding part on the embedded sequence. The higher the attention coverage, the more fully the generated content responds to the information carried by the speech signal.

[0039] Generative confidence level: This indicates the likelihood that the structured tag information can be generated based on the currently generated content; the higher the generative confidence level, the more "ready" the decoding part is to start producing structured tag information.

[0040] Tag generation switching probability: This refers to the probability, determined based on the current hidden state, that indicates a switch from the response text generation stage to the structured tag information generation stage. It is generated jointly by the first switching probability and the second switching probability. The first switching probability is determined based on the dispersion of the next byte candidate probability distribution, while the second switching probability is obtained by converting the generation confidence level.

[0041] Next word candidate probability distribution: This refers to the probability distribution of all candidate words calculated by the decoding part based on the current hidden state, representing the probability that each candidate word will be selected as the next word. The degree of dispersion refers to the concentration or dispersion of this probability distribution, which can be measured, for example, by information entropy.

[0042] Output stage state machine: refers to the state machine maintained by the decoding part to identify the current output stage, including at least the response generation state and the tag generation state; the response generation state corresponds to the first decoding constraint, and the tag generation state corresponds to the second decoding constraint.

[0043] Candidate lexical set: refers to the set of lexical units used to generate structured tag information, determined according to the second decoding constraint, which is a subset of the full set of candidate lexical units.

[0044] Restricted candidate probability distribution: refers to the probability distribution obtained by adjusting the candidate probability distribution of the next word word based on the set of candidate word words.

[0045] A single output sequence refers to a continuous sequence of terms produced by the decoding part in the same inference process. The response text and structured tag information together constitute this sequence.

[0046] A unified preset format refers to the unified organizational format followed by structured tag information, such as key-value pair format or markup language format.

[0047] Backbone network: This refers to the pre-trained network structure in the speech interaction model that undertakes the main computation, including the speech encoding part and the raw decoding part. Raw decoding part: This refers to the pre-trained decoding network in the decoding part before the insertion of low-rank adapters.

[0048] A low-rank adapter is a low-rank parameter module inserted into the backbone network. Its parameter size is much smaller than that of the backbone network, and it participates in the computation of network layers through a bypass structure of low-rank decomposition. In the embodiments of this application, in the implementation where the voice interaction model includes a low-rank adapter, the decoding part includes the original decoding part and the low-rank adapter.

[0049] Preset character information: refers to pre-defined character attribute information used to constrain at least one of the following in response text: expression style, tone, and character background, so that the response text conforms to the expression habits of the predetermined character.

[0050] First quantization format: refers to the quantization format used for storing the weights of the original decoding part on the terminal device. First floating-point format: refers to the floating-point numerical format used for storing the weights of the speech coding part and for forward computation. Second floating-point format: refers to the floating-point numerical format used for independently storing the weights of the low-rank adapter.

[0051] Initial model: refers to the model to be trained before training begins, including the speech encoding and decoding parts.

[0052] Sample query information: refers to query samples constructed during the training phase that simulate user input. Sample response text: refers to the annotated response text generated based on the sample query information. Sample structured label information: refers to structured label information constructed based on the information labels carried in the sample query information, using a machine-readable structure to carry those information labels.

[0053] Annotated data: refers to the data that serves as training supervision signals, consisting of sample response text and sample structured label information. Predicted data: refers to the predicted response text and predicted structured label information jointly output by the decoding part during training.

[0054] The first generative model refers to the generative model used during the training phase to generate sample query information based on the prompt information. The second generative model refers to the generative model used during the training phase to generate sample response text based on the sample query information. The first generative model and the second generative model can be the same model or different models; this application embodiment does not limit this.

[0055] In this embodiment, the first and second generative models can be deep learning models with relatively large parameter sizes. It should be noted that "large model" is merely an illustrative description; this embodiment does not limit the specific number of model parameters supported by the first and second generative models. Their parameter sizes can be flexibly set according to the resource constraints and performance requirements of the actual application scenario. For example, the first and second generative models can be large language models (LLMs) or multimodal large models (MLLMs) based on artificial intelligence.

[0056] The application scenarios of the embodiments of this application will be described below with reference to the accompanying drawings.

[0057] Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. For example... Figure 1 As shown, this application scenario includes a terminal device 101. The terminal device 101 deploys an end-to-end voice interaction model 102, a voice synthesis model 103, and a device control component 104. Specifically, the terminal device 101 collects voice signals emitted by the user through a microphone and plays a response voice to the user through a speaker, forming a two-way voice interaction with the user. The voice interaction model 102 encodes the voice signals and jointly generates response text and structured tag information during the same inference process, providing the response text to the voice synthesis model 103 and the structured tag information to the device control component 104. The voice synthesis model 103 converts the response text into response voice and plays it through the speaker of the terminal device 101. The device control component 104 parses the information tags in the structured tag information and triggers the terminal device 101 to execute the corresponding interactive response operation.

[0058] The specific form of the terminal device 101 includes, but is not limited to, head-mounted display devices, smart speakers, smartphones, tablets, robots, vehicle terminals, etc. The embodiments of this application do not limit the specific form of the terminal device 101.

[0059] The voice interaction method provided in the embodiments of this application will be described in detail below. Figure 2This is a flowchart illustrating a voice interaction method provided in an embodiment of this application. The method is applied to a terminal device, which has an end-to-end voice interaction model deployed on it. This voice interaction model includes a voice encoding part and a decoding part. Figure 2 As shown, the method includes steps S201 to S204.

[0060] Step S201: Acquire the voice signal.

[0061] In this embodiment, the voice signal can be acquired in real time by the microphone of the terminal device. Specifically, the terminal device continuously or on-demand monitors ambient sound, and when it detects a user speaking, it acquires the audio data corresponding to that speech as the voice signal. In one possible implementation, the terminal device begins acquiring the voice signal after detecting a wake-up event (e.g., wake word, key press, or gaze trigger); in another possible implementation, the terminal device is in a continuous dialogue state, and determines the start and end positions of the user's speech based on voice activity detection, using the audio data between the start and end positions as the voice signal. The former method has a clear acquisition range and fewer false triggers, making it suitable for on-demand interaction scenarios; the latter method does not require repeated wake-ups and provides a more coherent interaction, making it suitable for continuous companionship dialogue scenarios. Optionally, after acquiring the voice signal, it can also be resampled, gain normalized, or noise-reduced to adapt to the input requirements of the voice interaction model. Optionally, resampling converts the sampling rate of the acquired speech signal to a fixed sampling rate required by the speech interaction model, such as 16kHz or 44.1kHz, ensuring that the frame rate of the speech signal input to the model is consistent with the data distribution during training, and avoiding feature extraction deviations caused by sampling rate mismatch. Optionally, gain normalization adjusts the amplitude of the acquired speech signal to a preset standard dynamic range, such as a mean of 0, a variance of 1, or peak normalization to [-1, 1], eliminating signal strength fluctuations caused by differences in sound distance or volume, ensuring that the energy distribution of the speech signal input to the model is consistent with the training data, and avoiding feature extraction distortion caused by amplitude abnormalities. Optionally, noise reduction suppresses environmental background noise or non-human voice interference components in the speech signal, improves the signal-to-noise ratio of the human voice subject, ensures that the purity of the speech signal input to the model matches the acoustic environment assumptions during training, and avoids interference from background noise on the extraction of semantic and paralinguistic features (such as emotion and tone). The above acquisition methods are merely examples and do not constitute a limitation on the scope of protection of this application.

[0062] Step S202: The speech signal is encoded into an embedding sequence by the speech encoding part of the speech interaction model, and the embedding sequence is aligned with the text representation space processed by the decoding part of the speech interaction model.

[0063] The speech encoding part of the speech interaction model takes the speech signal as input and extracts and vectorizes the speech signal to obtain an embedding sequence. Each embedding vector in the embedding sequence resides in the same representation space as the vector representation of the text words processed by the decoding part; that is, the embedding sequence is aligned with the text representation space. This alignment can be achieved through mapping mechanisms such as linear projection, adapter modules, or cross-modal contrastive loss in joint pre-training. This mapping transforms the representation of the speech signal in the high-dimensional acoustic feature space to the same semantic vector space as the text words, achieving a unified cross-modal semantic space. Under this alignment, the speech embedding vectors are homogeneous with the text word vectors in terms of mathematical distribution and semantic dimension. Therefore, the decoding part can process the speech embedding vectors and text word vectors in the same sequence without first transcribing the speech signal into text. Information such as tone, rate, and pitch in the speech is preserved along with the embedding vectors and directly participates in the generation of the response text and structured label information during subsequent decoding. This cross-modal alignment not only ensures that speech and text with the same semantics are close in vector space, allowing the decoding part to directly understand semantic intent based on speech embedding without intermediate text transcription, but more importantly, it preserves paralinguistic information (such as intonation, speech rate, and emotional tone) that is easily lost in the transcription process of traditional Automatic Speech Recognition (ASR). This allows it to directly participate in subsequent response generation and structured label prediction, which is the core foundation for achieving end-to-end direct speech understanding.

[0064] In this embodiment, the specific network structure of the speech coding part is not strictly limited. At the structural level, the speech coding part can employ an audio coding network based on convolutional downsampling and stacked self-attention layers. This involves first converting the speech signal into frame-level acoustic features (e.g., Mel-spectral features), then using convolutional downsampling layers to extract local features and perform temporal dimensionality reduction on the converted acoustic features, and finally using self-attention layers to perform global context modeling and semantic aggregation on the extracted features layer by layer, resulting in a high-dimensional embedding vector sequence. Alternatively, the speech coding part can employ a self-supervised pre-trained speech representation network. This network directly learns frame-level representations from the original speech signal waveform or simple acoustic features, and then maps and aligns the output feature vectors to the text representation space via a mapping layer. At the source level, the speech coding part can be jointly pre-trained with the decoding part (e.g., reusing the speech coding and decoding components in an open-source model that has been jointly pre-trained with multiple modalities, as they naturally share an aligned representation space), or it can be pre-trained independently and then trained through an adaptation layer to complete the alignment of the representation space.

[0065] In one alternative implementation, considering that end-to-end speech understanding requires the encoded output and decoded input to be as consistent as possible in semantic granularity, the speech encoding and decoding parts use corresponding components from the same multimodal jointly pre-trained model. The speech encoding part encodes the speech waveform into a multimodal embedding sequence aligned with the text lexical space, and the decoding part uses the autoregressive decoding component in the model. This implementation does not require additional alignment layer training, has high alignment quality, and is a low-cost option for edge deployment.

[0066] Optionally, the encoded embedded sequence can be concatenated with the word sequence corresponding to the historical dialogue content and input into the decoding part, so that the decoding process refers to the dialogue context. This application embodiment does not limit whether to introduce historical dialogue content. Among them, introducing the word sequence corresponding to the historical dialogue content enables the decoding part to make a comprehensive judgment not only based on the current voice input, but also combined with the previous interaction context when generating the current response. This context-aware mechanism helps to solve the problem of referential resolution (for example, when a user says "turn it off", the model needs to identify the object referred to by "it" in combination with the preceding text), maintain the logical coherence and topic consistency of multi-turn dialogue, and avoid discontinuities or ambiguities in the response; at the same time, information such as user preferences and emotional change trajectories contained in the historical dialogue can also be captured by the model, thereby generating more personalized, emotionally relevant and consistent responses that conform to the long-term interaction persona, improving the naturalness of the interaction and the user's immersion.

[0067] Step S203: Through the decoding part, semantic understanding is performed on the embedded sequence in the same inference process, and response text and structured tag information are jointly generated based on the semantic understanding results. The structured tag information carries multiple information tags in a machine-readable structure.

[0068] The decoding part performs a continuous semantic understanding and content generation decoding process using the embedded sequence as the decoding context. During this decoding process, based on a unified current semantic state, the decoding part jointly constructs a single output sequence containing both the response text and structured labels through a single generation mechanism (e.g., autoregressive decoding, non-autoregressive parallel decoding, or a hybrid decoding strategy). The response text responds to the speech signal in natural language, while the structured labels carry various information tags identified from the speech signal in a machine-readable structure. Both are produced based on the same semantic understanding result, and the information tags and response content are naturally consistent at the semantic level. This fundamentally avoids the problems of upstream recognition errors contaminating downstream generation and the disconnect between recognition results and response content when multiple independent models are connected in series.

[0069] For a detailed explanation of the implementation process of this step, please refer to the following text. Figure 3 and Figure 4 The embodiments described in detail are omitted here.

[0070] Step S204: Generate and play the reply voice based on the reply text, and trigger the corresponding interactive response operation based on the information tags in the structured tag information.

[0071] After obtaining the response text, it can be provided to the speech synthesis model, which converts it into response speech and plays it through the terminal device's speaker. In one possible implementation, the entire response text is input into the speech synthesis model, and the synthesized complete response speech is played. In another possible implementation, the response text is input into the speech synthesis model in a clause-by-clause manner. The speech synthesis model synthesizes the response speech clause by clause and plays it. That is, each complete clause produced by the decoding part (e.g., segmented by punctuation marks such as periods, commas, and question marks) is submitted for synthesis and played sequentially. The whole-segment synthesis method is simple to implement and has good prosodic integrity, making it suitable for scenarios with short responses. The clause-by-clause streaming synthesis method allows synthesis and decoding to proceed in parallel, which helps to reduce the waiting time from the start of decoding to the playback of the first speech segment, making it suitable for scenarios with high requirements for interactive immediacy.

[0072] Optionally, the speech synthesis model can be fine-tuned using speech data with a preset timbre to output a response speech with the preset timbre. Specifically, speech data with the target timbre is collected or authorized for acquisition, and the pre-trained speech synthesis model is fine-tuned to ensure that the output response speech has a stable preset timbre that matches the character image set by the preset character information.

[0073] It should be noted that the speech synthesis model is independent of the speech interaction model, and the two are connected through a text interface called reply text. Therefore, the speech synthesis model can be replaced or iterated independently (e.g., changing the timbre or upgrading the synthesis quality) without affecting the speech interaction model; the speech interaction model focuses on semantic understanding and joint generation, and does not require retraining due to adjustments in synthesis capabilities.

[0074] After obtaining the structured tag information, the device control component of the terminal device parses the information tags in the structured tag information and triggers the corresponding interactive response operation based on the information tags. For a detailed implementation of triggering interactive response operations according to information tag type, please refer to the description of the following embodiments, which will not be elaborated here.

[0075] In this embodiment, an end-to-end voice interaction model is adopted. The voice encoding part encodes the voice signal into an embedded sequence aligned with the text representation space. The decoding part of the end-to-end voice interaction model performs semantic understanding on the embedded sequence in the same inference process, and jointly generates the response text and structured tag information carrying multiple information tags based on the semantic understanding results. This allows the semantic understanding and response generation of the voice signal to be completed in a single inference process within a single model, avoiding the problems of error accumulation and context fragmentation when multiple independent models are processed in series. The information tags in the voice are preserved and directly applied to the response generation, and the information tags and response content are naturally consistent. At the same time, the terminal device does not need to load multiple models at the same time, reducing memory and computing power consumption. Combined with local execution on the terminal side, the interaction response latency is shortened. Voice data does not need to be uploaded, reducing the data security pressure caused by data transmission.

[0076] In this application embodiment, the specific implementation of the multiple information tags is not limited and may vary depending on the application scenario. In one possible implementation, the multiple information tags include at least two of emotion tags, intent tags, and instruction tags; the emotion tag represents the emotional state of the object emitting the voice signal, the intent tag represents the interaction intent corresponding to the voice signal, and the instruction tag indicates the device control operation to be performed.

[0077] Taking emotion tags as an example, their values ​​can include emotional states such as happy, sad, calm, angry, and tired. The determination of emotion tags includes information such as tone, speed, and pitch carried in the speech signal, as well as the emotional coloring of the language content itself. Since the embedded sequence retains the above information, the decoding part can combine the two to determine the value of the emotion tag. Taking intention tags as an example, their values ​​can include interactive intentions such as movie information, weather information, music request, and casual conversation. The determination is based on the interactive purpose pointed to by the language content. Taking command tags as an example, their values ​​can include device control operations such as adjusting volume, playing music, dimming lights, and performing body movements. The determination is based on the operation instructions contained in the language content. Among them, performing body movements refers to driving the embodied object associated with the terminal device to perform specific body movements or facial expressions. The embodied object includes both digital objects displayed on the terminal screen (such as virtual digital humans and 3D virtual images) and robot-shaped devices with physical entities (such as companion robots and service robots). For control commands targeting embodied objects, the determination is based on the indications of the object's behavior, posture, or expression contained in the language content (such as waving, nodding, smiling, etc.). After parsing the command, the device control component executes the corresponding driving operation according to the type of embodied object: if it is a digital object, it drives the virtual image displayed on the screen to make corresponding visual feedback by calling a preset motion library or animation engine; if it is a robot-shaped device, it sends commands to the robot's motion control module to drive its physical joints, servos, or actuators to move, so that the robot body can make corresponding limb movements or adjust the facial expressions on the display screen in physical space.

[0078] In one possible implementation, the multiple information tags include two types: emotion tags and intent tags, suitable for scenarios primarily involving companionship and dialogue without device control. In another possible implementation, the multiple information tags include three types: emotion tags, intent tags, and command tags, suitable for scenarios that combine companionship and device control capabilities. It should be noted that the types of information tags can be flexibly configured and expanded according to device capabilities and scenario needs; for example, topic tags and urgency tags can also be added, but this application embodiment does not limit this.

[0079] It should be noted that if the voice signal does not contain the information corresponding to a certain information tag (e.g., the user is only chatting casually, and there are no device control instructions), then the tag field corresponding to that information tag will be empty or a preset placeholder value. Therefore, the destination of all information tags is complete: information tags with non-empty values ​​will trigger subsequent interactive response operations, while information tags with empty values ​​will not trigger any operation.

[0080] Taking at least two of the above-mentioned information tags, including emotion tags, intent tags, and instruction tags, as examples, the implementation method of triggering interactive response operations based on information tags will be described.

[0081] In one possible implementation, triggering the corresponding interactive response operation based on the information tags in the structured tag information includes the following branch processing: If the multiple information tags include emotion tags, then the speech content and / or expression style of the response speech are adjusted according to the emotion tags. Specifically, adjusting the speech content means adapting the text content carried by the response speech to the emotion tags. For example, when "sad" is identified, the response content presents a soothing expression. Since the response text and the emotion tag are generated in the same reasoning process, the wording of the response text naturally matches the emotion. At this time, the adjustment of the speech content is naturally achieved by the joint generation mechanism. Adjusting the expression style means adjusting the prosodic parameters of the speech synthesis according to the emotion tags. For example, when "sad" is identified, the speech rate is reduced and the tone is slowed down, and when "happy" is identified, the tone is made more lively. The above two adjustments can be performed individually or in combination.

[0082] If the multiple information tags include intent tags, then the subsequent interaction task in the current interaction scenario is determined based on the intent tag, and the subsequent interaction task is executed. Specifically, a correspondence between intent tag values ​​and subsequent interaction tasks is established in advance; after identifying the intent tag, the corresponding subsequent interaction task is determined and executed in conjunction with the current interaction scenario. For example, if the intent tag is identified as "weather information" and the current interaction scenario is travel preparation, then the subsequent interaction task is determined to be querying weather information and proactively providing clothing suggestions in the reply; if the intent tag is identified as "movie / TV show information," then the subsequent interaction task is determined to be retrieving a list of recommended movies / TV shows for subsequent rounds of display.

[0083] If the various information tags include instruction tags, then the corresponding device control operation is executed according to the instruction tag. Specifically, the device control component parses the value of the instruction tag and triggers the device control operation indicated by the value, such as adjusting the volume, playing music, controlling the displayed content, or driving an embodied object (such as a virtual digital human, 3D virtual image, or robot) to perform limb movements.

[0084] The three branches mentioned above are independent of each other and are triggered separately based on whether the value of their respective information tags is non-empty. In the same interaction, only one branch can be triggered, or multiple branches can be triggered simultaneously (for example, when a user requests to play music while feeling down, the emotion branch and the instruction branch are triggered at the same time). Thus, the triggering logic of the interactive response operation is organized in the form of "condition + execution action", and the processing path of various information tags is clear and complete.

[0085] Optionally, to avoid the terminal device responding in a rigid manner when the user is in a negative emotion, a correspondence between emotion tag values ​​and environmental feedback operations can be pre-established. When the identified emotion tag matches the correspondence, the corresponding environmental feedback operation is triggered while playing the reply voice. For example, when the emotion tag is identified as "sad," environmental feedback operations such as dimming the lights and playing soothing background music are triggered, ensuring the reply voice and environmental feedback are consistent in timing and semantics. The above correspondence can be flexibly configured and expanded according to the terminal device's capabilities and the needs of the scenario. For example, facial animation feedback can be configured on devices with a display screen, and body movement feedback can be configured on robot-shaped devices. This application embodiment does not limit this.

[0086] In the following embodiments of this application, the detailed implementation process of jointly generating the response text and structured tag information in the decoding part will be described in detail. In the following embodiments of this application, the autoregressive decoding method used in the decoding part will be used as an example for explanation.

[0087] Specifically, the decoding part adopts an autoregressive decoding method. Each time a word is generated, the already generated word is concatenated into the decoding context to continue predicting the next word. When the voice interaction model continuously generates two different forms of content in the same inference process—the first part being the response text in natural language form and the second part being structured tag information with a machine-readable structure—it faces the following specific challenges: First, how does the decoding part know whether it is currently generating response text or structured tag information? Second, how does it determine when the response text generation has ended? Third, how does it trigger the start of structured tag information generation? Fourth, how does it prevent natural language content from mixing into structured tag information and polluting its machine-readable structure? Fifth, how does it ensure that the generated structured tag information conforms to a unified preset format and can be stably parsed downstream? Sixth, how does it complete the generation of the above two forms of content in a single continuous inference process without introducing additional delays caused by a second inference? In this embodiment, a multi-layered decoding control system is constructed to solve the above-mentioned technical challenges faced in jointly generating natural language responses and structured tag information in the same inference process. These challenges will be explained in detail below: First, a two-stage decoding framework based on a first decoding constraint and a second decoding constraint is established. During the autoregressive decoding process, the decoding part initially uses the first decoding constraint to generate the response text. Upon detecting a preset constraint switching trigger, it switches to the second decoding constraint to generate structured tag information. This phased constraint mechanism ensures the independence and standardization of content generation in different formats. Figure 3 This is a schematic diagram illustrating a process for jointly generating response text and structured tag information, provided as an embodiment of this application. For example... Figure 3As shown, in one possible implementation, the step of performing semantic understanding on the embedded sequence through the decoding part in the same inference process, and jointly generating response text and structured tag information based on the semantic understanding results, includes steps S301 to S305.

[0088] Step S301: Use the embedded sequence as the decoding context input to the decoding part for autoregressive decoding processing.

[0089] Specifically, the embedded sequence serves as the initial content of the decoding context and is input into the decoding part; as decoding progresses, the generated tokens are concatenated into the decoding context one by one.

[0090] In one possible implementation, in addition to the embedded sequence, the decoding context can also include the word vectors corresponding to the system prompts (used to define the output format and role settings) and the word vectors corresponding to the historical dialogue, concatenated before the embedded sequence. These three elements together constitute the decoding context, with positional encoding added to preserve temporal relationships. Thus, the decoding part simultaneously "sees" the semantic content and output format conventions of the speech signal within a unified context, establishing an input foundation for subsequent joint generation.

[0091] Step S302: During the autoregressive decoding process, the current hidden state of the decoding part is updated based on the embedded sequence and the currently generated lexical units; the current hidden state reflects the semantic understanding result of the decoding part on the embedded sequence and the currently generated lexical units.

[0092] Specifically, at each decoding step, the internal network layers perform attention and feedforward calculations based on the decoding context (including the embedded sequence and currently generated lexical units) to obtain the current hidden state for that step. The current hidden state is a comprehensive semantic representation of "what the speech signal said" and "what has been responded to so far" by the decoding part: on the one hand, through attention calculations on the embedded sequence, the current hidden state absorbs information such as the language content and tone of the speech signal; on the other hand, through attention calculations on the currently generated lexical units, the current hidden state tracks the progress of response generation. Thus, the current hidden state is both the basis for generating the next lexical unit and the source of information for judging "whether the response is complete and whether to switch to label generation" later. In other words, the utilization of semantic understanding results runs through the entire process of joint generation.

[0093] Step S303: When decoding using the first decoding constraint, generate the next term in the response text based on the current hidden state and the first decoding constraint. This step S303... Figure 3 In simple terms: Under the first decoding constraint, generate the next term of the response text based on the current hidden state.

[0094] The first decoding constraint is the decoding constraint used when generating the response text, and its goal is to ensure that the response text is natural and fluent. In one possible implementation, the first decoding constraint includes natural language sampling strategy constraints, such as temperature coefficient, kernel sampling (top-p) parameters, and repetition penalty coefficient. After calculating the probability distribution of the next candidate word based on the current hidden state, the decoding part selects the next word from it according to the above sampling strategy. Specifically, the natural language sampling strategy constraint controls the diversity and fluency of the generated text by adjusting the probability distribution or selection rules of the candidate words. The temperature coefficient is used to adjust the smoothness of the probability distribution. When the temperature coefficient is high, the probability distribution tends to be uniform, and the model tends to select words with lower probabilities, thereby increasing the randomness and creativity of the generated content; when the temperature coefficient is low, the probability distribution tends to be sharp, and the model tends to select words with the highest probabilities, thereby making the generated content more deterministic and conservative. Kernel sampling parameters are used to dynamically truncate low-probability words. The model sorts candidate words by probability from high to low and accumulates their probability values, retaining only the smallest set of candidate words whose cumulative probability reaches a preset threshold. Other words are excluded from the sampling range. This method ensures generation quality while avoiding the selection of highly unlikely words and can automatically adjust the size of the candidate set based on the certainty of the current context. A repetition penalty coefficient is used to suppress repeated generation. For words that have already appeared in the historical generation sequence, their probability value in the current candidate probability distribution is divided by this coefficient, thereby reducing the likelihood of them being selected again and effectively alleviating the problem of the model getting stuck in infinite loops or repeating phrases. In another possible implementation, the first decoding constraint can also include a response length upper limit constraint, that is, constraining the number of words in the response text to not exceed a preset upper limit, to adapt to the short expression habits of spoken dialogue. The former method has high generation freedom and natural expression, suitable for open chat scenarios; the method of adding a length upper limit has strong controllability and is suitable for scenarios with requirements on the interaction rhythm.

[0095] Step S304: During the process of generating the response text, the preset constraint switching trigger information is detected. If detected, the first decoding constraint is switched to the second decoding constraint.

[0096] This step addresses how to determine when the response text has ended and how to trigger the start of structured tag information. The decoding section performs a preset constraint switching trigger detection at each step of response text generation (or every preset number of steps); if no preset constraint switching trigger information is detected, it continues to generate subsequent words in the response text under the first decoding constraint; if preset constraint switching trigger information is detected, constraint switching is performed, subsequent decoding switches to the second decoding constraint, and the process proceeds to step S305.

[0097] Step S305: When decoding using the second decoding constraint, generate the next lexical unit of the structured tag information based on the current hidden state and the second decoding constraint. This step S305... Figure 3 In simple terms: Under the second decoding constraint, the next lexical unit of the structured label information is generated based on the current hidden state.

[0098] The second decoding constraint is used when generating structured tag information. Its goal is to ensure that the structured tag information has a valid format and is parsable. Unlike the first decoding constraint, which allows free sampling from all candidate lexical units, the second decoding constraint restricts the generation of the next lexical unit to a set of candidate lexical units that conform to the structured format, thus preventing natural language content from being mixed into the structured tag information. The full set of candidate lexical units refers to the complete set of all predefined lexical units contained in the vocabulary of the speech interaction model, covering the words, punctuation marks, and various markers required to form a natural language response and structured data. Under the first decoding constraint, the decoding part can freely select from this full set of candidate lexical units based on probability distribution to ensure the natural fluency of the response. Under the second decoding constraint, a valid subset of the full set of candidate lexical units can be dynamically selected as a restricted candidate lexical unit set according to the current structured grammar rules, allowing the selection of the next lexical unit only from this set, thereby ensuring that the generated tags strictly conform to the preset structured format requirements.

[0099] Through steps S301 to S305, the two types of content are generated sequentially in a single autoregressive decoding process: the first decoding constraint ensures the natural flow of the response text, while the second decoding constraint ensures the legality of the structured tag information format. The constraint switching mechanism establishes a clear stage boundary between the two. This avoids both the doubled latency and contextual fragmentation caused by splitting response generation and tag generation into two separate inference steps, and the problem of mutual contamination between the two types of content under a single constraint. The mechanism's effectiveness in ensuring format legality is particularly evident when the structured tag information needs to be stably parsed by downstream programs.

[0100] The detection methods for preset constraint switching trigger information are described below. To ensure the accuracy and robustness of the switching from the first decoding constraint to the second decoding constraint, embodiments of this application provide various detection methods for preset constraint switching trigger information.

[0101] In one possible implementation, during the generation of the response text, the preset constraint switching trigger information is detected, including at least one of the following detection methods: Detection Method 1: If the currently generated lexical unit contains a preset stage switching lexical unit, then the preset constraint switching trigger information is determined to have been detected. The preset stage switching lexical unit is used to indicate the end of the generation of the response text and / or the start of the generation of the structured label information. Specifically, a special stage switching lexical unit (e.g., denoted as a separator lexical unit) is predefined in the vocabulary of the decoding part, and this lexical unit is placed between the sample response text and the sample structured label information in the labeled data during the training phase, enabling the model to learn to actively generate this lexical unit after the response expression is completed. During the inference phase, the decoding part checks whether each generated lexical unit is a preset stage switching lexical unit; a match triggers the switching. This method allows the model to autonomously determine the switching timing, and the switching point has a high degree of fit with the semantic boundary, making it suitable for scenarios with sufficient training data and where the model has reliably learned the output format.

[0102] Detection Method 2: If the currently generated lexical unit contains a preset end marker, then the preset constraint switching trigger information is detected; wherein, the preset end marker is used to indicate the end of the generation of the response text. Specifically, the preset end marker can be a sentence-level terminator (such as a period, question mark, exclamation mark, or other sentence-ending punctuation, or a native text-ending lexical unit in the model's vocabulary); when the decoding part detects the presence of a preset end marker in the currently generated lexical unit, and the generated content constitutes a complete sentence, it determines that the generation of the response text has ended and triggers the switch. This method does not require expanding the vocabulary or special training for switching lexical units, resulting in lower implementation costs.

[0103] Detection Method 3: If the length of the response text reaches a preset length threshold, the preset constraint switching trigger information is determined to have been detected. Specifically, the number of tokens (or characters) in the generated response text is counted, and a forced switch is triggered when the preset length threshold is reached. This method is a fallback detection measure to prevent the model from continuously generating response text without producing structured label information under abnormal conditions, ensuring that both parts of content can be stably produced in the same inference process; it is usually used in conjunction with other detection methods as a last resort. The specific value of the preset length threshold is not limited and can be flexibly configured according to the role setting and device type.

[0104] Detection Method 4: If the semantic integrity score determined based on the current hidden state reaches a preset integrity threshold, then the preset constraint switching trigger information is detected. The semantic integrity score measures whether the currently generated response text is semantically complete and whether it has fully responded to the voice signal; when the score reaches the preset integrity threshold, it indicates that the marginal value of continuing to generate more response content is low, and the process can switch to the tag generation stage. The method for determining the semantic integrity score is described in the following embodiments. This method switches based on semantic-level judgment, and the switching timing is closely related to the response quality, which helps ensure response integrity.

[0105] Detection Method 5: If the tag generation switching probability determined based on the current hidden state is greater than a preset probability threshold, then the preset constraint switching trigger information is detected. The tag generation switching probability measures the switching timing based on the generation tendency of the decoding part itself, and its determination method is described in the following embodiments. This method directly utilizes the probability information already available during the decoding process, has low additional computational overhead, and is suitable for scenarios with limited computing power on the edge.

[0106] The five detection methods described above can be used individually or in any combination. In one possible implementation, detection method one is used as the primary method, with detection method three as a fallback: the model actively generates preset stage switching lexical units, and switches immediately; if the length reaches a threshold and switching has not occurred, a forced switch is initiated. In another possible implementation, detection methods four and five are used in combination: switching is triggered when either the semantic integrity score or the tag generation switching probability meets a certain threshold, taking into account both semantic integrity and generation tendency. Regardless of the detection method used, if preset constraint switching trigger information is detected, a switch from the first decoding constraint to the second decoding constraint is executed, and the decoding process continues.

[0107] In one alternative implementation, multiple detection methods can be used in combination. By using multiple detection methods together, the determination of the switching timing does not depend on a single signal. When any signal is abnormal (e.g., the model fails to generate a switching term), the other signals can still be triggered as a fallback, which can effectively improve the reliability of constraint switching.

[0108] The following provides an example of how to determine semantic integrity scores.

[0109] In one possible implementation, determining the semantic integrity score based on the current hidden state includes the following process: First, based on the current hidden state of the decoding part, the semantic representation of the currently generated response text is determined, and based on the attention weight of the embedded sequence, the attention coverage of the semantic representation on the embedded sequence is calculated.

[0110] Here, the current hidden state of the decoding part refers to the set of hidden state vectors output by each network layer after the decoding part has processed the currently generated response text tokens. Based on the current hidden state of the decoding part, the semantic representation of the currently generated response text can be determined using at least one of the following methods: Method 1 directly reuses the hidden state vectors already present in the decoding process. Specifically, it extracts the hidden state vectors corresponding to each word in the generated response text from the current hidden state in the decoding part. Then, it performs pooling processing on the hidden state vectors corresponding to each word in the generated response text, such as average pooling or taking the vector from the last word. The pooling result is used as the semantic representation. This method directly reuses the hidden states already present in the decoding process without changing the model structure.

[0111] Method 2 involves a dedicated semantic representation layer to aggregate and transform the current hidden state. Specifically, the model structure is modified by adding a semantic representation layer to the decoding part. Hidden state vectors corresponding to each word in the generated response text are extracted from the current hidden state of the decoding part. These vectors are then input into the semantic representation layer, which aggregates and transforms them to output a dedicated semantic representation vector. This semantic representation layer can be implemented as an additional attention aggregation layer or a lightweight feedforward transformation layer. Its parameters are trained together with the decoding part during the training phase. Optionally, in the implementation using a low-rank adapter, it can be updated along with the low-rank adapter. This method provides a specially learned semantic representation that is more targeted in representing the completeness of semantics.

[0112] The calculation of attention coverage is based on the attention weights applied to the embedded sequence by the decoding part. These attention weights refer to the numerical values ​​calculated by the decoding part when generating each word in the response text, using either self-attention or cross-attention mechanisms, to indicate the degree of attention the currently generated word receives from each position in the input embedded sequence. Specifically, during decoding, the decoding part calculates a relevance score between the currently generated word and each frame of acoustic feature vectors in the input embedded sequence. After normalization (e.g., softmax), a set of weight values ​​in the form of a probability distribution is obtained. These weight values ​​reflect the extent to which the currently generated semantic content depends on a certain part of the input speech information. When calculating attention coverage, the attention weights of each word in the generated response text to each position in the embedded sequence are accumulated or the maximum value is taken, resulting in a cumulative attention weight for each position in the input sequence. This cumulative attention weight represents the attention level at the corresponding position. Then, the percentage of positions in the embedded sequence where the cumulative attention weight reaches a preset level of attention is calculated, and this percentage is used as the attention coverage. If the cumulative attention weights at most positions in the input sequence reach the preset attention level, it indicates high attention coverage. This means the model has fully read and responded to the key information segments in the speech signal; in other words, higher attention coverage indicates a more comprehensive response to the information carried by the speech signal in the generated reply. Conversely, if the cumulative attention weights at certain key positions (such as time periods corresponding to specific intentions or entities) do not reach the preset attention level, it indicates low attention coverage. This means that the information in that part has not been effectively responded to. In this case, the low attention coverage prompts the decoding part to continue generating reply text to cover the remaining information. For example, if a user's speech expresses two layers of content: "work is not going well" and "want to listen to some music," and the generated reply only responds to the first layer, the cumulative attention weight at the position corresponding to "want to listen to some music" in the embedded sequence is low, and the attention coverage is still low. The decoding part then continues to generate reply text to respond to the second layer of content.

[0113] Secondly, based on the current hidden state of the decoding part, the confidence level for generating the structured tag information based on the currently generated response text is predicted. The confidence level for generation represents the probability that the structured tag information can be generated based on the currently generated content.

[0114] A prediction that can generate confidence can be made in at least one of the following ways: Method 1 involves attaching a lightweight Linear Projection Head to the output of the decoding section. Taking the current hidden state as input, it is mapped to a scalar value through a fully connected layer, and then processed by an activation function to compress the output value to the range of 0 to 1. The output value between 0 and 1 is used as the generative confidence score. This confidence score reflects the model's probability estimate of a binary judgment—whether the conditions for switching to the label generation stage are met—based on the current context. This method has a small number of parameters and negligible inference overhead, making it particularly suitable for scenarios with limited edge computing power.

[0115] Method 2 modifies the model structure by adding a Label Field Prediction Layer to the decoding section. This layer contains multiple binary classifiers or regression heads corresponding to the label fields (referred to as label fields) in the structured label information. Through this layer, the system predicts whether each label field has a currently available value (e.g., whether the values ​​of the sentiment and intent fields are determinable) based on the current hidden state. The proportion of label fields with available values ​​to the total number of fields is mapped to a generation confidence score. For example, if the structured label information contains 5 fields, and 3 of them have available values, the generation confidence score is 0.6. This method not only provides the overall confidence score but also the readiness status of each label field, providing a field-level reference for subsequent label generation. It is suitable for scenarios with many label fields where fine-grained control over the switching timing is desired.

[0116] The parameters of the aforementioned prediction head or prediction layer are trained during the training phase using the "switch point position" as the supervision signal: the last position of the sample response text in the labeled data is labeled as a positive sample, and the remaining positions are labeled as negative samples, and the prediction head or prediction layer is trained with binary classification loss.

[0117] Finally, based on the aforementioned attention coverage and generative confidence, the semantic integrity score of the response text is determined. Optionally, the attention coverage and generative confidence can be fused to obtain the semantic integrity score of the response text. The fusion method can employ at least one of the following: Method 1: Weighted summation of attention coverage and generative confidence, with the weighted sum used as the semantic integrity score of the response text; the weights can be flexibly determined according to the application scenario, for example, each taking 0.5. Method 2: Taking the smaller value of the two as the semantic integrity score of the response text, requiring both dimensions to meet the standard for semantic integrity. This method is relatively conservative and can avoid premature switching due to an inflated single dimension. Thus, the semantic integrity score simultaneously characterizes two dimensions: "whether the response to the input is sufficient" (attention coverage) and "whether the label information can be produced" (generative confidence), providing a semantic-level quantitative criterion for switching timing.

[0118] In the above implementation, by establishing the switching determination on the quantitative fusion of attention coverage and the confidence that can be generated, the switching timing is directly linked to the semantic completeness of the response content. This can effectively avoid the problem of being truncated and switched before the response is finished, or the response being redundant and long, thus delaying the generation of tags. Especially when the user's voice contains multiple layers of expression and the response length fluctuates greatly, the advantage of this mechanism over fixed length threshold switching is more obvious.

[0119] It should be noted that the determination of semantic integrity score is not limited to the methods described in the above embodiments. For example, in another possible implementation, the determination of semantic integrity score may not rely on the fusion calculation of attention coverage and generate confidence, but may use a single-dimensional quantitative indicator or rule-based combinational logic. For example, in a first variation, the semantic integrity score may be directly equivalent to the attention coverage calculated above. Or, in a second variation, the semantic integrity score may be directly equivalent to the generate confidence predicted above. Or, in a third variation, the semantic integrity score is determined using threshold-based logical decision rather than numerical fusion. Specifically, a first threshold for attention coverage and a second threshold for generate confidence are set respectively. When the attention coverage is greater than or equal to the first threshold and the generate confidence is greater than or equal to the second threshold, the semantic integrity score is determined to be qualified (or set to 1); otherwise, it is determined to be unqualified (or set to 0). Under this Boolean scoring system, the decoding part can perform a switch from generating response text to generating structured tag information when the score is qualified.

[0120] The following is an example of how to determine the label generation switching probability.

[0121] In one possible implementation, determining the tag generation switching probability based on the current hiding state includes the following process: First, the probability distribution of the next word candidate is calculated based on the current hidden state, and the first switching probability is determined according to the dispersion of the probability distribution of the next word candidate.

[0122] Specifically, as the response text generation nears completion, the decoding part's certainty about "what to generate next" decreases, and the shape of the next word candidate probability distribution changes accordingly. Therefore, the degree of dispersion of the distribution can serve as a signal for the switching timing. The degree of dispersion can be measured in at least one of the following ways: Method 1: Calculate the information entropy of the next word candidate probability distribution. A higher entropy value indicates a more dispersed distribution and a lower certainty from the model regarding continuing to generate natural language. Map the normalized entropy value to the first switching probability. Method 2: Calculate the probability difference between the highest-probability candidate word and the second-highest-probability candidate word. A smaller difference indicates a flatter distribution. Map the inverse normalization result of the difference to the first switching probability. Method 1 is sensitive to the overall shape of the distribution and provides a more comprehensive signal; Method 2 only requires comparing the head probabilities and involves less computation.

[0123] Secondly, based on the current hidden state, the generateability confidence level for the structured tag information can be predicted based on the currently generated response text, and this generateability confidence level is converted into a second switching probability. The prediction method for the generateability confidence level is described in the preceding embodiments and will not be repeated here. The conversion from generateability confidence level to the second switching probability can be achieved using at least one of the following methods: Method 1, directly using the generateability confidence level as the second switching probability; Method 2, using a biased sigmoid function mapping for conversion, where the bias term controls the sensitivity range of the conversion, making the second switching probability change more sensitively when the confidence level approaches the decision boundary.

[0124] Finally, the tag generation switching probability is generated based on the first switching probability and the second switching probability. The generation method can employ at least one of the following: Method 1, a weighted sum of the first and second switching probabilities, with the weights calibrated based on the reliability of the two signals in the verification data; Method 2, taking the larger of the two, i.e., a tendency to switch when either signal is strong. This method is more proactive in switching and can reduce the risk of prolonged response. Wherein, when the tag generation switching probability is greater than a preset probability threshold, it is determined that a preset constraint switching trigger information has been detected. The specific value of the preset probability threshold is not limited and can be flexibly configured according to the preference for switching proactivity, for example, 0.8.

[0125] In the above implementation, the first switching probability comes from the decoding distribution pattern and the second switching probability comes from the prediction of the confidence level. The two signal sources are independent of each other. The switching probability generated by the fused tag has the effect of suppressing the noise of the single signal and can effectively improve the robustness of the switching decision. Especially when there are colloquial pauses, interjections and other content in the reply text that are easy to interfere with the single signal, the robustness advantage of dual-channel fusion is more obvious.

[0126] Furthermore, in this embodiment, an output stage state machine is introduced to explicitly manage the decoding process. The state machine includes at least a response generation state and a tag generation state, corresponding to the first decoding constraint and the second decoding constraint, respectively. When a preset constraint switching trigger information is detected, the state machine is triggered to switch from the response generation state to the tag generation state, thereby clarifying the current generation stage at the system level and avoiding confusion between the two decoding logics. The implementation method of the output stage state machine is described below.

[0127] Figure 4 This is a schematic diagram of an output stage state machine and a restricted decoding process provided in an embodiment of this application. Figure 4 As shown, in one possible implementation, the decoding part maintains an output stage state machine, which includes at least a response generation state and a tag generation state, used to explicitly identify the current output stage of the decoding process. This is precisely the solution to the problem of how the decoding part knows what content is being generated; that is, each step of decoding reads the current state of the output stage state machine and performs generation according to the decoding constraints corresponding to the current state. The stage attribution is uniformly maintained by the state machine without step-by-step deduction.

[0128] In this embodiment, the above-mentioned process of switching from the first decoding constraint to the second decoding constraint includes: in response to detecting preset constraint switching trigger information, triggering the output stage state machine to switch from the response generation state to the tag generation state; when switching from the response generation state to the tag generation state, stopping the use of the first decoding constraint corresponding to the response generation state and starting to use the second decoding constraint corresponding to the tag generation state.

[0129] Specifically, the output stage state machine can be implemented in at least one of the following ways: Method 1: Maintain explicit state variables in the control logic of the decoding loop. The state variables are initially in the response generation state, and are set to the label generation state when a preset constraint switching trigger information is detected. Each step of the decoding loop selects the currently effective decoding constraint based on the state variables. This method is intuitive, provides observable states, facilitates debugging and log tracking, and is suitable for scenarios with self-developed decoding engines. Method 2: Drive state transitions using preset stage switching terms or start delimiters of the structured portion in the generated response text. That is, the decoding part monitors whether the aforementioned special terms or symbols used to identify stage boundaries appear in the output sequence, and implicitly propels the state machine from the response generation state to the label generation state accordingly. This method is naturally integrated with the term processing of the decoding part and requires no additional state storage.

[0130] Optionally, the output stage state machine can be extended with more states. For example, a generation end state can be added after the tag generation state, entering the generation end state and terminating decoding when the structured tag information is generated (a structure terminator is detected); another example is adding a preamble state before the return generation state to handle the pre-filling stage of system prompts and embedded sequences. The specific granularity of state division can be flexibly configured according to the implementation needs of the decoding part, and this application embodiment does not limit it.

[0131] In the above implementation, the output stage state machine transforms the implicit content judgment of "which output stage is currently in" into a controlled state management. The constraint switching has a clear time point and is irreversible (it will not revert after switching from the response generation state to the tag generation state). It can effectively avoid the content mixing caused by the interleaving of the two decoding constraints. Especially when the boundaries of the two forms of content in a single output sequence are adjacent, the boundary control effect of the state machine is more obvious.

[0132] Regardless of the method used, after switching from the first decoding constraint to the second decoding constraint, decoding processing can continue based on the current hidden state of the decoding part and the second decoding constraint, thereby generating the next term of the structured tag information. In an optional embodiment, during the tag generation state, a restricted decoding mechanism based on the second decoding constraint can be used to ensure the format validity of the structured tag information. The following section combines... Figure 4 The restricted decoding process based on the second decoding constraint is explained.

[0133] In one possible implementation, such as Figure 4 As shown, based on the current hidden state and the second decoding constraint, the next lexical unit of the structured tag information is generated, including the following process: The first step is to calculate the probability distribution of the next candidate word across all candidate words based on the current hidden state. Specifically, the probability distribution of all candidate words in the vocabulary is obtained through the output projection layer and normalization process of the decoding part.

[0134] The second step is to determine a set of candidate lexical units for generating structured tag information based on the second decoding constraint. The set of candidate lexical units is a subset of the full set of candidate lexical units.

[0135] The second decoding constraint includes at least one of the following: a structured tag template, a set of field names, an enumeration of field values, a punctuation constraint, or a structured syntax state.

[0136] The structured tag template defines the overall syntactic skeleton of the structured tag information, specifying the fixed order of field names, delimiters, and value positions. During decoding, the template determines the syntactic node at the current decoding position: if the current position corresponds to a fixed part in the template (such as field names, colons, commas, or parentheses), the next term is uniquely determined or restricted to a specific punctuation mark; if the current position corresponds to a variable part in the template (i.e., a field value slot), then terms can be selected from the set of valid enumerated values ​​corresponding to that field. In this way, the template decomposes the generation task into a series of constrained local selection problems, thereby precisely defining the range of terms allowed in each decoding step.

[0137] The field name set refers to the set of tokens corresponding to all legal field names (such as "emotion", "intention", "instruction", etc.) in the structured tag information. During the decoding process, when it is determined that a field name should be generated at the current position based on the structured syntax state, tokens in the field name set can be retained as legal candidates, thereby ensuring that the generated field names conform to the preset structure definition and avoiding illegal or undefined field identifiers.

[0138] Field value enumeration refers to the set of lexical units corresponding to the allowed legal values ​​of each tag field in the structured tag information (e.g., the values ​​of the emotion field are "happy", "sad", "calm"). During the decoding process, when it is determined that a specific field value should be generated at the current position based on the structured syntax state, lexical units in the field value enumeration corresponding to that field can be retained as legal candidates. This ensures that the generated value content is strictly limited to the predefined semantic range, avoiding out-of-bounds or invalid tag values.

[0139] Punctuation constraints refer to the set of tokens corresponding to delimiters and separators (such as colons between key-value pairs, commas between fields, and opening and closing brackets of an overall structure) specified in the structured syntax. During decoding, when it is determined from the structured syntax state that a structured punctuation mark should be generated at the current position, tokens can be selected from this set of punctuation constraints. This ensures that the generated structured tags conform to the standard syntax requirements and maintain the integrity of the machine-readable structure.

[0140] The structured grammar state refers to the parsing progress status maintained in real time by the decoding part during the generation of structured tag information, based on preset structured format grammar rules (such as JSON format, key-value pair format, etc.). This state records the currently generated structural components (such as whether field names have been generated, whether delimiters have been generated, and which field is currently being evaluated). Based on the current structured grammar state, the decoding part dynamically deduces the grammatically valid lexical type (field name, value, or punctuation) for the next decoding position, thereby determining the specific candidate lexical set corresponding to that position, achieving controlled generation that proceeds step by step according to grammatical logic.

[0141] The above-mentioned constraints can be used individually or in combination; for example, using structured syntax states as the main framework and overlaying field value enumerations at the field value positions can simultaneously ensure both the structure and the value.

[0142] The third step is to adjust the probability distribution of the next candidate lexicon based on the candidate lexicon set to obtain a restricted candidate probability distribution.

[0143] The adjustments include masking and / or reordering. Specifically: masking involves setting the probability of words outside the candidate word set to zero (or to a negative infinite logarithmic probability), retaining only the probabilities of words within the candidate word set, and then normalizing the retained probabilities. Masking largely prevents illegal words from being selected and is a direct means of preventing natural language content from contaminating structured tag information. Reordering involves rearranging or weighting the probabilities of words within the candidate word set according to their compliance with the second decoding constraint (e.g., applying a boosting coefficient to the word that best matches the current grammatical state), thus prioritizing words that better conform to the second decoding constraint. Masking and reordering can be used individually or in combination: first, masking eliminates illegal words, then reordering optimizes the selection among legal words.

[0144] The fourth step is to select the next lexicon of the structured tag information from the candidate lexicon set based on the restricted candidate probability distribution.

[0145] Regarding the selection strategy for choosing the next lexicon of structured tag information from the candidate lexicon set, this application embodiment does not limit this. In an optional embodiment, the grammatical attributes of the current decoding position determined in the second decoding constraint can be flexibly configured: for fixed grammatical node positions (such as field names, key-value separators, field separators, etc.) determined by the structured tag template or structured grammatical state, a greedy selection strategy can be adopted to directly select the lexicon with the highest probability in the restricted candidate probability distribution to maximize the stability and accuracy of the structural skeleton generation; for variable value positions (i.e., the specific content slots of each tag field) that are open by field value enumeration, a restricted sampling strategy can be adopted to randomly sample according to the restricted candidate probability distribution, retaining a certain degree of generation diversity while ensuring that the value conforms to the predefined enumeration range, so as to adapt to multiple legal expressions that may exist under the same intent.

[0146] The aforementioned restricted decoding process is repeated word by word until the structured tag information is generated. Therefore, the problem of ensuring the second part is in a legal structured format is solved during the generation process: each word is generated within a legal range, and the structured tag information naturally conforms to a unified preset format upon completion, without needing post-generation repair.

[0147] In the above implementation, by restricting the candidate word set and adjusting the probability distribution, the format legality of the structured tag information is guaranteed by the generation mechanism at the word granularity. This can effectively eliminate format defects such as natural language word mixing, field value out-of-bounds, and missing delimiters, and reduce the downstream parsing failure rate. Especially when the end-to-end voice interaction model is small in scale and the stability of the freely generated format is limited, the effect of restricted decoding on improving the format legality rate is more obvious.

[0148] Optionally, as a supplementary safeguard to restricted decoding, after the decoded output, format verification can be performed on the structured tag information. If the verification passes, the information is then provided to the device control component; if the verification fails, a preset fallback strategy is applied, such as only playing the response voice and not triggering any interactive response. Therefore, even in implementations where restricted decoding is not enabled or the constraint configuration is incomplete, format deviations will be intercepted before parsing, ensuring the stable operation of the interactive response chain.

[0149] In summary, in the embodiments described above, a complete joint generation control mechanism is constructed through the organic integration of multiple technical means, including two-stage decoding constraints, constraint switching trigger detection, output stage state machine, and restricted decoding. These technical means do not exist in isolation but work synergistically: two-stage decoding constraints establish the basic framework for generating response text and structured tag information; constraint switching trigger detection provides precise timing judgment for the transition from natural language generation to structured generation; the output stage state machine realizes explicit management and logical isolation of the generation stage; and restricted decoding ensures the legality of the final output format at the lexical granularity. This multi-layered integration mechanism ensures both the natural fluency and semantic integrity of the response text and the format standardization and parsability of the structured tag information, significantly reducing the risk of mutual interference or boundary confusion between two different forms of content during a single inference process. This achieves high-quality, low-latency, and highly reliable end-to-end voice interaction response within a single model.

[0150] In this embodiment, the response text and structured tag information are organized and managed through a single output sequence. In this approach, the response text and structured tag information are no longer independent parallel output streams, but are integrated into a continuous lexical sequence for autoregressive generation. In one possible implementation, the structured tag information is organized according to a uniform preset format, and the response text and structured tag information together constitute a single output sequence of the decoding part in the same inference process.

[0151] Specifically, a single output sequence consists of a natural language part and a structured part connected sequentially: the natural language part is the response text, and the structured part is structured tag information organized according to a unified preset format. The boundary between the two can be distinguished by switching lexical units or structure start delimiters in a preset stage. The unified preset format includes, but is not limited to, key-value pair format or markup language format. Taking the key-value pair format as an example, the structured tag information can be organized into a set of fields in the following form: the key of the emotion field is "emotion" and the value is the identified emotion tag; the key of the intent field is "intent" and the value is the identified intent tag; the key of the command field is "action" and the value is the identified command tag. Taking an interaction as an example, the form of a single output sequence can be: the response text "Okay, I'll play you a cheerful song," the transition stage switches lexical units, and then the structured part "{emotion: happy, intent: music request, action: play music}" is connected. The above format and field naming are only examples and do not constitute a limitation on the scope of protection of this application.

[0152] By employing a single output sequence organization, the response text and structured label information are produced sequentially within the same lexical stream. The achievement of "completing two output formats in a single inference" is directly achievable through this output organization: the decoding section requires no modification to the output header or parallel maintenance of two output channels; downstream, the two parts can be accessed separately by segmenting the data using boundary delimiters. Correspondingly, during the training phase, the labeled data is also organized using the same single output sequence format, consisting of concatenated sample response text and sample structured label information. This consistency in data organization between training and inference allows the model to directly learn the conversion rules from natural language to structured formats during training, achieving a closed loop between output format learning and practical application. This improves the accuracy and stability of the model in generating structured label information conforming to the preset format during the inference phase.

[0153] It should be noted that the arrangement of reply text first and structured tag information later also brings benefits to the interactive experience: the reply text is generated first and can immediately enter the sentence-by-sentence streaming speech synthesis (see step S204), the structured tag information is generated subsequently and triggers the interactive response operation, the speech playback and operation execution are naturally connected in time sequence, and the user's perceived first response delay is short.

[0154] In this embodiment, the detailed implementation structure of the voice interaction model is not limited. In an optional embodiment, the voice interaction model may include a low-rank adapter (LoRA). The following description, in conjunction with the accompanying drawings, illustrates the implementation method and edge deployment of this low-rank adapter-included model.

[0155] Figure 5 This is a schematic diagram of the deployment structure of an end-to-end voice interaction model on the terminal side, as provided in an embodiment of this application. Figure 5 As shown, the voice interaction model includes a backbone network and a low-rank adapter inserted into the backbone network. The backbone network includes a speech encoding part and a raw decoding part. The decoding part includes the raw decoding part and the low-rank adapter. The low-rank adapter is used to carry the output format of preset role information and structured tag information. The preset role information is used to constrain at least one of the expression style, timbre and character background of the response text.

[0156] Specifically, the backbone network is a pre-trained network structure that undertakes the main computations for speech encoding and language generation. The low-rank adapter is inserted into the original decoding layer of the backbone network as a bypass structure with low-rank decomposition, and its parameter scale is much smaller than that of the backbone network. The character expression ability and structured output ability obtained by the model after fine-tuning or training are mainly concentrated in the parameters or weights of the low-rank adapter: at least one of the following: the preset character information constrains the expression style of the response text (e.g., a lively or calm language style), timbre (e.g., timbre identifiers that match the character image, for the speech synthesis model to select the corresponding timbre), and character background (e.g., the character's identity setting and knowledge background, so that the response content is consistent with the character setting); the output format of the structured label information (field composition and organization format) is also carried by the low-rank adapter, so that the model can stably produce a unified preset format.

[0157] In the implementation using a low-rank adapter, the embedded sequence is semantically understood during the same inference process through the decoding part, and the response text and structured tag information are jointly generated based on the semantic understanding results. This includes: using the embedded sequence as the decoding context input to the original decoding part for autoregressive decoding processing; during the autoregressive decoding process, for the network layer into which the low-rank adapter is inserted in the original decoding part, the output of the network layer is added to the output of the low-rank adapter and used as the input of the next network layer in the original decoding part, and the response text and the structured tag information are generated in the same inference process based on the output format of the preset role information and structured tag information carried by the low-rank adapter.

[0158] To ensure a complete understanding of the inference process involving low-rank adapters, the forward computation process described above is expanded below at the network layer granularity. Let the network layer in the original decoding part without a low-rank adapter be denoted as the first network layer, and the network layer with a low-rank adapter be denoted as the second network layer. In each step of the autoregressive decoding, the forward computation of each network layer is performed as follows: For the first network layer in the original decoding part without a low-rank adapter, if the first network layer is the first network layer in the original decoding part, its input is the embedded sequence output by the speech coding part; if the first network layer is not the first network layer in the original decoding part, the output of the previous network layer is used as the input of the first network layer, and after processing by the first network layer, the final output of the first network layer is obtained. The final output of the first network layer is then directly used as the input of the next network layer in the original decoding part; for the second network layer, the second network layer first calculates its original output based on its input. Wherein, if the second network layer is the original... The first network layer of the initial decoding part takes the embedded sequence output by the speech coding part as its input. If the second network layer is not the first network layer of the original decoding part, its input is the final output of the previous network layer. If the second network layer is the first network layer of the original decoding part, the embedded sequence output by the speech coding part is input to the low-rank adapter. If the second network layer is not the first network layer of the original decoding part, the final output of the previous network layer is input to the low-rank adapter. The low-rank adapter calculates the input according to its own parameters (i.e., it first performs low-rank dimensionality reduction projection and then low-rank dimensionality increase projection through a bypass calculation) to obtain the output of the low-rank adapter. The output of the low-rank adapter is added to the original output of the second network layer. The sum is used as the final output of the second network layer, and this final output is used as the input of the next network layer in the original decoding part. The above calculation is performed layer by layer until the output layer, obtaining the current hidden state and the probability distribution of the next word candidate at this step.

[0159] Thus, the role-expressive and structured output capabilities carried by the low-rank adapter are integrated into each step of the decoding computation through a layer-by-layer bypass addition: the original decoding part provides general semantic understanding and language generation capabilities, and the low-rank adapter superimposes incremental adjustments for role-based and formatted elements on top of it. Together, these two determine the wording style of the response text and the output format of the structured tag information. This forward computation method does not require any modification to the weights of the original decoding part; the role capabilities take effect in a bypass form, providing a structural foundation for the multi-role switching discussed later.

[0160] Optionally, the weights of the original decoding part are stored in the terminal device in a first quantization format, and the weights of the speech coding part are stored in the terminal device in a first floating-point format; the weights of the low-rank adapter are stored independently in a second floating-point format and are not merged with the weights of the original decoding part stored in the first quantization format.

[0161] Specifically, the parameter size of the original decoding part is much larger than that of the speech coding part. Performing low-bit quantization only on the weights of the original decoding part, such as 4-bit integer quantization, can significantly compress the model size. The speech coding part is most sensitive to speech understanding accuracy, and its weights are stored in a first floating-point format (e.g., 16-bit floating-point format) to preserve the accuracy of the most sensitive component. If the low-rank adapter is incorporated into the original decoding part before performing low-bit quantization, the low-bit quantization will compromise the consistency of the role carried by the low-rank adapter and the accuracy of the structured output format. Storing the weights of the low-rank adapter independently in a second floating-point format (e.g., 16-bit floating-point format) avoids this compromise, and the parameter size of the low-rank adapter is only in the tens of megabytes range, resulting in minimal additional space overhead from independent storage.

[0162] In the above implementation, by adopting a differentiated storage strategy, namely, quantizing and compressing the original decoding part to save storage space, retaining the floating-point format for the speech coding part to ensure the accuracy of acoustic feature extraction, and storing the weights of the low-rank adapter independently to avoid performance loss caused by quantization, the model size is greatly compressed at a lower accuracy cost, enabling the end-to-end voice interaction model to run resident within the storage and memory constraints of the terminal device. This advantage is more obvious in wearable devices with limited resources such as smartwatches and headphones, greatly improving the feasibility of model deployment in various edge scenarios.

[0163] Due to the differences in storage format and computational precision between the speech coding part and the original decoding part, in order to maintain the numerical stability of the forward computation while ensuring the model compression rate, the method may optionally further include: during the autoregressive decoding process, dequantizing the weights of the original decoding part stored in the first quantization format to the first floating-point format for forward computation. Specifically, in each forward computation step of the autoregressive decoding, the decoding part is calculated sequentially according to the network layer order. For each network layer in the original decoding part, the weights of that layer stored in the first quantization format are dequantized to the first floating-point format, and the calculation of that layer is performed with the weights in the first floating-point format; in particular, for the second network layer with the inserted low-rank adapter, the dequantized backbone computation output (i.e., the output of the current second network layer) is directly added to the bypass output of the low-rank adapter stored in the second floating-point format (both are in floating-point format, and no precision conversion is required for addition), and the addition result is passed to the next network layer. Thus, quantization only affects the storage format, and the computation path maintains floating-point precision throughout, maximizing the preservation of the model's performance in precision-sensitive tasks such as semantic understanding and structured output while compressing the volume.

[0164] It should be noted that the specific formats of the first quantization format, the first floating-point format, and the second floating-point format are not limited and can be flexibly configured according to the hardware support of the deployment environment; for example, the first quantization format can be a 4-bit integer quantization format or an 8-bit integer quantization format, and the first floating-point format and the second floating-point format can be a 16-bit floating-point format or a 32-bit floating-point format; the first floating-point format and the second floating-point format can be the same floating-point format or different floating-point formats, and this application embodiment does not limit this.

[0165] Optionally, the terminal device stores multiple low-rank adapters corresponding to various preset role information; the method further includes: in response to a role switching instruction, loading the low-rank adapter corresponding to the target role, and the loaded low-rank adapter participating in inference together with the backbone network. Specifically, the backbone network resides permanently in the terminal device's memory, and the low-rank adapters corresponding to different roles are stored in independent files; when the user issues a role switching instruction through voice, interface operation, or application configuration, the current low-rank adapter can be unloaded and the low-rank adapter corresponding to the target role can be loaded without reloading the backbone network. After switching, the expression style, tone, and background of the response text change with the target role, while the model's semantic understanding ability and the output format of the structured label information remain stable.

[0166] In the above implementation, the overhead of role switching is reduced from reloading the entire model to reloading adapter parameters (i.e. weights) on the order of tens of megabytes, and the switching time is greatly shortened, making it feasible to switch multiple role capabilities on the device side.

[0167] It should be noted that the voice interaction method in this application embodiment is applied to a terminal device. The encoding of the voice signal, the generation of the response text, and the generation of structured tag information can all be executed locally on the terminal device. Optionally, the voice synthesis model and the device control component are also deployed locally on the terminal device, forming a complete local interaction closed loop. As a result, the interaction link does not rely on network transmission, has low response latency, can still run in an offline environment, and the voice data does not need to leave the terminal device, reducing the data security pressure caused by data transmission.

[0168] The voice interaction model in this application embodiment can be trained on a pre-trained model using specific training samples and a specific training method. The training method for the voice interaction model provided in this application embodiment is described below. In this application embodiment, the pre-trained model can be referred to as the initial model.

[0169] Figure 6 A flowchart illustrating a training method for a voice interaction model provided in an embodiment of this application. Figure 7This is a block diagram illustrating a training data construction process provided in an embodiment of this application. The model training process is based on an initial model to be trained, which includes a speech encoding part and a decoding part. The following describes the process in conjunction with... Figure 6 and Figure 7 The training process of this model will be explained. For example... Figure 6 As shown, the training method includes steps S601 to S606.

[0170] Step S601: Generate multiple sample query information carrying information tags based on the prompt information corresponding to the various information tags.

[0171] The various information tags are used to generate sample query information, each with its own corresponding prompts. The initial model lacks the ability to jointly output response text and structured tag information in the same inference process. The primary training step is to construct sample query data that covers the target scenario and includes information tags for each sample. Specifically, for each information tag, corresponding prompts are constructed to describe the characteristics of the desired query. For example, for the instruction tag, the prompt describes "generating a colloquial expression in which the user requests the device to perform an action," resulting in sample queries such as "Give me a round of applause" and "Let's dance," each with its corresponding instruction tag. For the intent tag, the prompt describes "generating a colloquial expression in which the user seeks information," resulting in sample queries such as "What are some good movies lately?" and "What's the weather like tomorrow?" each with intent tags for movie / television / weather information. For the emotion tag, the prompt describes "generating a user expression with a strong emotional tone," resulting in sample queries such as "I'm so happy today" and "I'm a little sad," each with its corresponding emotion tag.

[0172] In one possible implementation, such as Figure 7 As shown, the step of generating multiple sample query information carrying information tags based on the prompt information corresponding to the various information tags includes: inputting the prompt information corresponding to the various information tags into a first generation model, and having the first generation model generate multiple sample query information carrying the corresponding information tags. The first generation model can be a general text generation model (e.g., a large language model) that generates sample query information in batches according to the prompt information. In another possible implementation, the sample query information can also be generated by combining a preset query template with tag values, for example, using the template "help me + [action phrase]" to generate instruction-type queries in batches. The queries generated by the first generation model are diverse in expression and highly colloquial, resulting in more diverse samples; the template filling method produces controllable results and has a lower cost.

[0173] Since the information tags are defined in the prompts when the sample query information is generated, the tag annotations are obtained synchronously with the generation, eliminating the need for manual annotation of each item afterward. Therefore, it is possible to quickly obtain a large-scale, wide-coverage sample query information with built-in annotations at a lower cost. Optionally, the prompts can also specify diverse requirements such as expression style, sentence length, and scenario to enhance the diversity of the sample query information. This application embodiment does not limit the specific content of the prompts.

[0174] Step S602: Generate sample response text based on the sample query information.

[0175] Specifically, for each sample query, a sample response text is generated to address the query, serving as annotation content on the response text side. As shown in the figure, in one possible implementation, preset role information and the sample query information are input together into the second generation model, which then generates sample response text that conforms to the constraints of the preset role information. In another possible implementation, role constraints are not introduced; the sample query information is directly input into the second generation model to generate sample response text in a general style. This method is suitable for training voice interaction models without role settings.

[0176] Optionally, the length of the sample response text can be controlled within a preset length threshold: on the one hand, this makes the response style more in line with the expression habits of spoken dialogue; on the other hand, it shortens the generation length of the response text and the speech synthesis time during the inference stage, reducing the overall latency of the interaction on the device side. The specific value of the preset length threshold is not limited and can be flexibly configured according to the role settings and device type.

[0177] Step S603: Based on the information tags carried by the sample query information, construct sample structured tag information in a machine-readable structure to carry the information tags.

[0178] Specifically, such as Figure 7 As shown, for each sample query, the information tags it carries (at least two of the values ​​of emotion tags, intent tags, and instruction tags) are read and organized into sample structured tag information according to a structured format. For information tags not mentioned in the sample query, the corresponding tag field values ​​are set to empty or preset placeholder values ​​to ensure that the field composition of the sample structured tag information is consistent across all samples.

[0179] Step S604: Combine the sample response text with the sample structured label information as the annotation data corresponding to the sample query information.

[0180] In one possible implementation, the sample structured label information is organized according to a unified preset format. The sample response text and the sample structured label information together constitute a single output sequence, which serves as the labeled data. That is, the labeled data is not a separate response text or a separate classification label, but rather a holistic sequence composed of "sample response text + preset stage switching lexical units + sample structured label information." The structured label information is modeled as a component of the generation task, and the model learns the recognition and formatted output of information labels while learning dialogue generation. This organization method is consistent with the organization method of the single output sequence in the inference stage, ensuring that the output formats of training and inference are seamlessly connected.

[0181] Optionally, after constructing the labeled data, a format validity check can be performed on the structured portion of a single output sequence to remove samples with invalid formats, thereby ensuring the consistency of the format of the structured label information of the samples in the labeled data.

[0182] Step S605: Input the sample query information into the decoding part of the initial model. The decoding part jointly outputs the predicted response text and the predicted structured label information as the prediction data, i.e., the joint output prediction data.

[0183] It should be noted that the training method in this embodiment only trains the decoding part of the initial model; the speech encoding part is not involved in the training. Therefore, the training input does not require speech morphology. The sample query information in text form is converted into a word vector sequence through the embedding layer and then directly input into the decoding part. Specifically, the decoding part uses the word vector sequence corresponding to the sample query information as the decoding context, performs autoregressive decoding, and jointly outputs the predicted response text and predicted structured label information, which together constitute the prediction data. This method eliminates the need for constructing sample speech and performing forward computation of speech encoding, resulting in lower training data construction costs and lower single-step training overhead. The reason why the decoding part can be trained separately is that the embedding sequence output by the speech encoding part is aligned with the text representation space. This allows the joint generation capability obtained by the decoding part on text input to be directly transferred to the inference stage where the embedding sequence is used as input.

[0184] It should be noted that training the decoding part separately is only one model training method. In another optional implementation, the entire initial model can also be trained: sample speech is generated in reverse based on sample query information. For example, the text-based sample query information can be converted into audio-based sample speech using a speech synthesis model, and multiple speaker timbres and noise perturbations can be superimposed to improve robustness. Then, the sample speech is used as the input of the initial model, encoded into an embedding sequence by the speech coding part, and then jointly output as prediction data by the decoding part. The prediction data and labeled data are combined to train the entire initial model. This training method enables the speech coding and decoding parts to be jointly optimized during training, which is more suitable for the distribution of speech input and is conducive to further improving the alignment quality of the representation space of the speech coding and decoding parts.

[0185] Step S606: Based on the loss between the predicted data and the labeled data, update the weights of the decoding part (referred to as updating the weights) to obtain an end-to-end voice interaction model. The voice interaction model is used to jointly output the response text and structured label information in the same inference process.

[0186] In this embodiment, the specific loss calculation method between predicted data and labeled data is not limited, and can be flexibly selected according to actual training needs. Several possible loss calculation strategies are listed below: In one implementation, the sample response text and sample structured label information can be treated as a single, complete output sequence, and the overall sequence generation loss can be calculated. For example, word-by-word cross-entropy loss can be used. Specifically, first, the single output sequence in the labeled data is used as a supervision signal; this sequence consists of a series of real labeled words. Second, for each word position in the sequence, the probability distribution of the model's predicted output is obtained, and the predicted probability value corresponding to the real labeled word at that position is determined. Then, the negative logarithm of this predicted probability value is taken to obtain the local loss value for that word position. Finally, the local loss values ​​of all valid word positions in the sequence are summed or averaged to obtain the overall sequence generation loss for the sample. Since the single output sequence simultaneously contains both the sample response text and the sample structured label information, this loss naturally supervises both response generation and label output capabilities. Optionally, higher loss weights can be applied to the word positions in the sample structured label information portion of the sequence to enhance the model's learning of structured formats. Backpropagation and weight updates are performed on the decoding part based on the loss, and training is iterated until the loss converges (e.g., the validation set loss no longer decreases within a preset number of consecutive training rounds) or the preset number of training rounds is reached, thus obtaining an end-to-end voice interaction model.

[0187] In another implementation, the losses for the response text portion and the structured label information portion can be calculated separately, and the two can be weighted and summed to obtain the overall loss. Specifically, firstly, based on the separator marker or preset position in a single output sequence, the predicted data and labeled data are divided into a response subsequence portion and a label subsequence portion. Secondly, for the response subsequence portion, a word-by-word cross-entropy loss calculation method is used, that is, for each word position in the response subsequence portion, the probability distribution of the predicted data output at that position is obtained, and the predicted probability value of the corresponding real word in the labeled data is extracted, and its negative log-likelihood is calculated as a local loss. Finally, the results are summed to obtain the response text generation loss. At the same time, for the label subsequence portion, the same word-by-word cross-entropy loss calculation method is used, and the above probability comparison and negative log calculation are performed for each word position in the label subsequence portion, and the results are summed to obtain the structured label generation loss. Finally, the response text generation loss and the structured label generation loss are weighted and combined according to preset weight coefficients to obtain the final training loss. Based on the final training loss, backpropagation and weight updates are performed on the decoding part, and training is iterated until the loss converges (e.g., the validation set loss no longer decreases within a preset number of consecutive training rounds) or the preset number of training rounds is reached, resulting in an end-to-end voice interaction model. This approach allows for independent adjustment and optimization priorities for natural language generation tasks and structured format generation tasks.

[0188] In another implementation, the losses for the response text portion and the structured label information portion can be calculated separately, and the training state can be determined based on their respective loss errors. Specifically, firstly, the response text generation loss and the structured label generation loss are calculated separately as described above; then, it is monitored whether the response text generation loss meets a preset text convergence threshold, and whether the structured label generation loss meets a preset structured convergence threshold; when the response text generation loss and the structured label generation loss meet their respective convergence thresholds, it is determined that the current model has reached the training objective, and an end-to-end voice interaction model is obtained.

[0189] Thus, the training of the voice interaction model was completed through sample construction, the establishment of network roles and optimization objectives, the execution of the forward inference process, and the calculation and convergence judgment of the loss function.

[0190] It should be noted that the training process in this embodiment does not limit the introduction of low-rank adapters and preset role information; that is, weight updates can be applied to all parameters of the decoding part, i.e., full-parameter fine-tuning; the full-parameter fine-tuning method is highly flexible. In an optional embodiment, low-rank adapters and preset role information can be introduced during the above model training process. The model training method that introduces low-rank adapters has low overhead and does not destroy pre-training capabilities.

[0191] The following describes the training implementation method that incorporates a low-rank adapter and preset role information.

[0192] In one possible implementation, the initial model includes a backbone network and a low-rank adapter inserted into the backbone network. The backbone network includes the speech coding part and the original decoding part. The decoding part includes the original decoding part and the low-rank adapter. The low-rank adapter is used to carry the output format of preset role information and structured label information. The preset role information is used to constrain at least one of the following: the expression style, timbre, and background of the responding text. In this implementation, the decoding part is weighted according to the loss between the predicted data and the labeled data. This includes updating the weights of the low-rank adapter according to the loss between the predicted data and the labeled data, while keeping the weights of the original decoding part unchanged, to obtain an end-to-end speech interaction model.

[0193] In this implementation, the generation of sample response text synchronously incorporates role constraints: preset role information and the sample query information are input into a second generation model, which then generates sample response text that conforms to the preset role information constraints. The preset role information is used to constrain at least one of the following: expression style, tone, and character background of the sample response text. The injection of preset role information can be achieved in at least one of the following ways: Method 1: The preset role information is organized as a system prompt and input into the second generation model along with the sample query information, constraining the output to conform to the character's expression style and character background; Method 2: A small number of manually written role-based responses are used as examples and input into the second generation model along with the preset role information in an example-guided manner, aligning the generated result stylistically with the examples. Method 1 has low implementation costs and is suitable for scenarios with clearly defined roles; Method 2 has higher style fit and is suitable for scenarios with stringent requirements for role consistency. Thus, the labeled data as a whole carries the role style, and the character's expressive ability after fine-tuning is condensed in the parameters of the low-rank adapter.

[0194] In the training implementation with the low-rank adapter, the forward propagation and parameter update process during training is as follows: Sample query information is input into the decoding part, and forward computation is performed layer by layer as described in the previous inference embodiment. The output of the second network layer with the low-rank adapter is added to the bypass output of the low-rank adapter and then passed to the next network layer. The loss propagates backward along the network, and the gradient passes through each network layer of the original decoding part, but only the gradient is accumulated and updated for the parameters of the low-rank adapter, while the weights of the original decoding part remain unchanged. Thus, the character expression ability and structured output ability are reflected in the low-rank adapter, and the semantic understanding and language generation ability obtained by the pre-training of the original decoding part are not destroyed.

[0195] Alternatively, the same backbone network can be used with labeled data for different roles to train multiple low-rank adapters, providing a training basis for multi-role switching on the edge.

[0196] In one possible implementation, the weights of the speech coding portion are stored in a first floating-point format, the weights of the original decoding portion are stored in a first quantization format, and the weights of the low-rank adapter are stored independently in a second floating-point format, without being merged with the weights of the original decoding portion stored in the first quantization format. In this implementation, the training process includes: during forward propagation, dequantizing the weights of the original decoding portion stored in the first quantization format to the first floating-point format for forward computation to obtain the predicted data; during backpropagation, calculating the gradient based on the loss between the predicted data and the labeled data; updating the weights of the low-rank adapter according to the gradient, while keeping the weights of the original decoding portion unchanged, to obtain an end-to-end speech interaction model.

[0197] Specifically, the weights of the original decoding part are loaded in the first quantization format (e.g., 4-bit quantization format), which significantly reduces memory usage. During forward propagation, the weights are dequantized layer by layer to the first floating-point format for computation, and the computational accuracy is not affected by quantization storage. During backpropagation, the gradient flows only to and updates the low-rank adapter parameters stored in the second floating-point format. As a result, fine-tuning can be completed even without large-scale training resources, and the training threshold is low. Moreover, the weight storage format during the training phase is consistent with that during the edge deployment phase, and the accuracy strategies of the training, storage, and inference stages are seamlessly integrated, ensuring that the model's capabilities are fully preserved on the edge.

[0198] In this embodiment, the specific network structure of the original decoding part is not limited. In one possible implementation, the network structure of the original decoding part may include multiple stacked attention layers and feedforward network layers. In this network structure, a low-rank adapter can be inserted into the attention layers and feedforward network layers of the decoding part. Specifically, the low-rank adapter can be inserted into the query projection, key projection, value projection, and output projection positions of each attention layer in the decoding part, as well as the projection position of the feedforward network layer. The low-rank adapter of the attention layer mainly affects how the model pays attention to the context, while the adapter of the feedforward network layer mainly affects the expression of content and the organization of format. The two can be used together to simultaneously carry both role expression and structured output capabilities. Optionally, the rank of the low-rank adapter can be flexibly configured according to the complexity of the role setting and the constraints of the end-side resources: the larger the rank, the richer the role and format knowledge that the adapter can carry, but the number of parameters and inference overhead will increase accordingly; in one possible implementation, the rank can be 32, and after fine-tuning, only tens of megabytes of adapter weights need to be stored.

[0199] Optionally, after fine-tuning, the voice interaction model can be evaluated in multiple dimensions. Evaluation dimensions include the format validity rate of structured tag information, the recognition accuracy of each information tag, and a score given by an independent evaluation model based on the role fit of the response text according to preset role information. The evaluation results can be used to select training data allocation and rank-equal training configurations. This application embodiment does not limit whether to perform the evaluation or the specific method of evaluation.

[0200] To facilitate a full understanding of the technical solutions described in the embodiments of this application, several scenario embodiments are given below.

[0201] Scenario Example 1: Companion dialogue scenario based on head-mounted display device. The user wears the head-mounted display device and says, "My work didn't go well today, and I'm a little sad." The device's microphone collects the speech signal (corresponding to step S201); the speech encoding part encodes the speech signal into an embedding sequence aligned with the text representation space, and the low-pitched tone information in the speech is preserved along with the embedding vector (corresponding to step S202); the decoding part performs autoregressive decoding, first generating the response text "Thank you for your hard work, let me hug you first, would you like to tell me what happened?" under the first decoding constraint, during the generation process, the model output preset stage switching word unit is detected, and the output stage state machine switches from the response generation state to the label generation state, and then generates structured label information within the limited candidate word unit range under the second decoding constraint, where the emotion field is "sad", the intent field is "chatting", and the instruction field is empty (corresponding to the detailed implementation process of step S203); the response text is input into the speech synthesis model in a clause-by-clause manner, and is synthesized and played clause by clause with a preset timbre. The expression style of the response speech slows down the tone according to the emotion label, and at the same time, the device control component triggers the environmental feedback operation of dimming the lights according to the emotion label, the instruction field is empty, and no device control operation is triggered (corresponding to the detailed implementation process of step S204). The entire process is completed locally on the head-mounted display device in a single inference step, with the response tone and reassuring feedback perfectly synchronized.

[0202] Scenario Example 2: Abnormal Fallback Branch. In the same interaction as Scenario Example 1, it is assumed that the client-side model fails to actively produce the preset stage switching terminology due to quantization, and the response text continues to be generated; when the length of the generated response text reaches the preset length threshold, the preset constraint switching trigger information is forcibly triggered, the output stage state machine switches to the tag generation state, the structured tag information is still produced normally under restricted decoding, the interactive response operation is unaffected, and the user is unaware of this fallback process.

[0203] Scenario Example 3: Configuring Overlay Branches and Role Switching. When a terminal device (e.g., a head-mounted display device) is in an environment without network connectivity, the voice interaction model, voice synthesis model, and device control components are all deployed locally on the device. The voice interaction function is unaffected by network conditions and can still operate normally. When the device is configured in "Children's Mode," even if the instruction tag generated by the decoding part in restricted decoding is a device control operation that the device can execute, the device control component still filters high-risk operations according to the Children's Mode configuration strategy (e.g., prohibiting volume from being increased to the maximum). The configuration strategy takes precedence over the default execution of tag-driven operations. After the user issues a role switching instruction, the terminal device remains active on the backbone network, replacing the current low-rank adapter with the low-rank adapter corresponding to the target role. After the switch, the expression style, timbre, and character background of the reply text change accordingly, while the model's semantic understanding ability and the output format of the structured tag information remain stable. The above scenario examples are merely illustrative and do not constitute a limitation on the scope of protection of this application.

[0204] Figure 8 This is a schematic diagram of a voice interaction device provided in an embodiment of this application. The device is applied to a terminal device, which deploys an end-to-end voice interaction model. For example... Figure 8 As shown, the voice interaction device 800 includes: an acquisition module 801, an encoding module 802, a generation module 803, and an interaction module 804.

[0205] The system includes: an acquisition module 801 for acquiring a speech signal; an encoding module 802 for encoding the speech signal into an embedding sequence using the speech encoding part of the speech interaction model, wherein the embedding sequence is aligned with the text representation space processed by the decoding part of the speech interaction model; a generation module 803 for performing semantic understanding on the embedding sequence during the same inference process using the decoding part, and jointly generating a response text and structured tag information based on the semantic understanding results, wherein the structured tag information carries multiple information tags in a machine-readable structure; and an interaction module 804 for generating and playing a response speech based on the response text, and triggering corresponding interactive response operations based on the information tags in the structured tag information.

[0206] It should be noted that the optional implementation methods in the above method embodiments are also applicable to the corresponding modules of this device embodiment, and will not be repeated here.

[0207] Figure 9 This is a schematic diagram of a training device for a voice interaction model provided in an embodiment of this application. The initial model to be trained includes a voice encoding part and a decoding part. Figure 9As shown, the training device 900 includes: a first generation module 901, a second generation module 902, a construction module 903, a labeling module 904, a training module 905, and an update module 906.

[0208] The system comprises the following modules: a first generation module 901, used to generate multiple sample query information carrying information tags based on the prompt information corresponding to various information tags; a second generation module 902, used to generate sample response text based on the sample query information; a construction module 903, used to construct sample structured label information carrying the information tags in a machine-readable structure based on the information tags carried in the sample query information; an annotation module 904, used to use the sample response text and the sample structured label information together as the annotation data corresponding to the sample query information; a training module 905, used to input the sample query information into the decoding part of the initial model, and the decoding part jointly outputs the predicted response text and the predicted structured label information as the prediction data; and an update module 906, used to update the weights of the decoding part based on the loss between the predicted data and the annotation data to obtain an end-to-end voice interaction model, wherein the voice interaction model is used to jointly output the response text and the structured label information in the same inference process.

[0209] It should be noted that the optional implementation methods in the above training method embodiments are also applicable to the corresponding modules of this training device embodiment, and will not be repeated here.

[0210] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 10 As shown, the electronic device 1000 includes: a processor 1001, a memory 1002, and a communication interface 1003.

[0211] Memory 1002 is used to store computer programs and can be configured to store various other data to support operation on electronic devices. Memory 1002 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0212] The processor 1001, coupled to the memory 1002, is used to execute the computer program stored in the memory 1002 to implement the voice interaction method provided in the embodiments of this application, or to implement the training method of the voice interaction model provided in the embodiments of this application.

[0213] Communication interface 1003 is used for wired or wireless communication between electronic devices and other devices. Communication interface 1003 can support mobile communication standards such as 2G, 3G, 4G, and 5G, or communication methods such as wireless local area networks. In an exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0214] Further optionally, the electronic device 1000 also includes optional components such as an audio component, a power supply component, and a display. The audio component includes a microphone and a speaker; the microphone is used to capture voice signals, and the speaker is used to play back voice responses. The power supply component provides power to various components of the electronic device and may include a power management system, one or more power supplies, and other components associated with the generation, management, and distribution of power. The display may include a liquid crystal display (LCD) or a touch panel for displaying interactive content or receiving user touch operations. Figure 10 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 10 The components shown.

[0215] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium.

[0216] Accordingly, this application also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above method embodiments.

[0217] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0218] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A voice interaction method, characterized in that, Applied to a terminal device, wherein an end-to-end voice interaction model is deployed on the terminal device, the method includes: Acquire audio signals; The speech signal is encoded into an embedding sequence by the speech encoding part of the speech interaction model, and the embedding sequence is aligned with the text representation space processed by the decoding part of the speech interaction model. Through the decoding part, semantic understanding is performed on the embedded sequence in the same inference process, and response text and structured tag information are jointly generated based on the semantic understanding results. The structured tag information carries multiple information tags in a machine-readable structure. The system generates and plays a response voice based on the response text, and triggers the corresponding interactive response operation based on the information tags in the structured tag information.

2. The method according to claim 1, characterized in that, The multiple information tags include at least two of emotion tags, intent tags, and instruction tags; the emotion tag represents the emotional state of the object emitting the voice signal, the intent tag represents the interactive intent corresponding to the voice signal, and the instruction tag indicates the device control operation to be performed.

3. The method according to claim 2, characterized in that, The step of triggering the corresponding interactive response operation based on the information tags in the structured tag information includes: If the multiple information tags include emotion tags, then the voice content and / or expression style of the response voice are adjusted according to the emotion tags; If the multiple information tags include intent tags, then the subsequent interaction task in the current interaction scenario is determined based on the intent tags, and the subsequent interaction task is executed. If the multiple information tags include an instruction tag, then the corresponding device control operation is executed according to the instruction tag.

4. The method according to claim 1, characterized in that, The step of performing semantic understanding on the embedded sequence through the decoding part during the same inference process, and jointly generating response text and structured tag information based on the semantic understanding results, includes: The embedded sequence is used as the decoding context input to the decoding part for autoregressive decoding processing; During the autoregressive decoding process, the current hidden state of the decoding part is updated based on the embedded sequence and the currently generated lexical units; the current hidden state reflects the semantic understanding result of the decoding part of the embedded sequence and the currently generated lexical units. When decoding using the first decoding constraint, the next term of the response text is generated based on the current hidden state and the first decoding constraint; During the generation of the response text, a preset constraint switching trigger information is detected. If detected, the first decoding constraint is switched to the second decoding constraint. When decoding using the second decoding constraint, the next term of the structured tag information is generated based on the current hidden state and the second decoding constraint.

5. The method according to claim 4, characterized in that, During the generation of the response text, preset constraint switching trigger information is detected, including at least one of the following: If the currently generated lexical unit contains a preset stage switching lexical unit, then it is determined that the preset constraint switching trigger information has been detected; wherein, the preset stage switching lexical unit is used to indicate the end of the generation of the response text and / or the start of the generation of the structured tag information; If the currently generated lexical unit contains a preset end marker, then it is determined that the preset constraint switching trigger information has been detected; wherein, the preset end marker is used to indicate the end of the generation of the response text; If the length of the reply text reaches a preset length threshold, it is determined that the preset constraint switching trigger information has been detected; If the semantic integrity score determined based on the current hidden state reaches a preset integrity threshold, then it is determined that the preset constraint switching trigger information has been detected. If the tag generation switching probability determined based on the current hidden state is greater than a preset probability threshold, then it is determined that the preset constraint switching trigger information has been detected.

6. The method according to claim 5, characterized in that, Determining a semantic integrity score based on the current hidden state includes: Based on the current hidden state, determine the semantic representation of the currently generated response text, and calculate the attention coverage of the semantic representation on the embedded sequence based on the attention weight of the embedded sequence; Based on the current hidden state, predict the confidence level of the ability to generate the structured tag information based on the currently generated response text; Based on the attention coverage and the generative confidence, the semantic integrity score of the response text is determined.

7. The method according to claim 5, characterized in that, Determining the tag generation switching probability based on the current hidden state includes: The probability distribution of the next word candidate is calculated based on the current hidden state, and the first switching probability is determined according to the dispersion of the probability distribution of the next word candidate. Based on the current hidden state, predict the generative confidence level of the structured tag information that can be generated based on the currently generated response text, and convert the generative confidence level into a second switching probability; The tag switching probability is generated based on the first switching probability and the second switching probability.

8. The method according to claim 4, characterized in that, The decoding section maintains an output stage state machine, and the switching from the first decoding constraint to the second decoding constraint includes: In response to the detection of the preset constraint switching trigger information, the output stage state machine is triggered to switch from the response generation state to the tag generation state; When switching from the response generation state to the tag generation state, the first decoding constraint corresponding to the response generation state is stopped, and the second decoding constraint corresponding to the tag generation state is started.

9. The method according to claim 4, characterized in that, The step of generating the next term of the structured tag information based on the current hidden state and the second decoding constraint includes: Based on the current hidden state, calculate the probability distribution of the next word candidate on all candidate words; Based on the second decoding constraint, a set of candidate lexical units for generating the structured tag information is determined, wherein the set of candidate lexical units is a subset of the full set of candidate lexical units; Based on the set of candidate lexical units, the probability distribution of the next lexical unit candidate is adjusted to obtain a restricted candidate probability distribution; Based on the restricted candidate probability distribution, the next lexicon of the structured tag information is selected from the candidate lexicon set.

10. The method according to any one of claims 1 to 9, characterized in that, The structured tag information is organized according to a unified preset format, and the response text and the structured tag information together constitute a single output sequence of the decoding part in the same reasoning process.

11. The method according to any one of claims 1 to 9, characterized in that, The voice interaction model includes a backbone network and a low-rank adapter inserted into the backbone network. The backbone network includes the voice encoding part and the original decoding part. The decoding part includes the original decoding part and the low-rank adapter. The low-rank adapter is used to carry the output format of preset role information and structured tag information. The preset role information is used to constrain at least one of the following: expression style, tone, and background of the person in the reply text.

12. The method according to claim 11, characterized in that, Through the decoding part, semantic understanding is performed on the embedded sequence during the same inference process, and response text and structured tag information are jointly generated based on the semantic understanding results, including: The embedded sequence is used as the decoding context input to the original decoding part for autoregressive decoding processing; During the autoregressive decoding process, for the network layer into which the low-rank adapter is inserted in the original decoding part, the output of the network layer is added to the output of the low-rank adapter and used as the input of the next network layer in the original decoding part. Based on the output format of the preset role information and structured tag information carried by the low-rank adapter, the response text and the structured tag information are generated in the same inference process.

13. The method according to claim 12, characterized in that, The weights of the original decoding part are stored in the terminal device in a first quantization format, and the weights of the speech encoding part are stored in the terminal device in a first floating-point format. The weights of the low-rank adapter are stored independently in a second floating-point format and are not merged with the weights of the original decoded portion stored in the first quantization format.

14. The method according to claim 13, characterized in that, Also includes: During the autoregressive decoding process, the weights of the original decoded portion stored in the first quantization format are dequantized to the first floating-point format to participate in the forward computation.

15. The method according to claim 11, characterized in that, The terminal device stores multiple low-rank adapters corresponding to various preset role information; the method further includes: In response to a role switching instruction, a low-rank adapter corresponding to the target role is loaded, and the loaded low-rank adapter participates in inference together with the backbone network.

16. A training method for a voice interaction model, characterized in that, The initial model to be trained includes a speech encoding part and a decoding part, and the method includes: Based on the prompts corresponding to various information tags, generate multiple sample query messages carrying information tags; Based on the sample query information, generate sample response text; Based on the information tags carried by the sample query information, construct sample structured tag information in a machine-readable structure to carry the information tags; The sample response text and the sample structured tag information are used together as the annotation data corresponding to the sample query information; The sample query information is input into the decoding part of the initial model, and the decoding part jointly outputs the predicted response text and the predicted structured label information as the prediction data. Based on the loss between the predicted data and the labeled data, the weights of the decoding part are updated to obtain an end-to-end voice interaction model, which is used to jointly output the response text and structured label information in the same inference process.

17. The method according to claim 16, characterized in that, The step of generating multiple sample query information carrying information tags based on the prompt information corresponding to various information tags includes: The prompt information corresponding to the various information tags is input into the first generation model, and the first generation model generates multiple sample query information carrying the corresponding information tags.

18. The method according to claim 16, characterized in that, The step of generating sample response text based on the sample query information includes: The preset role information and the sample query information are input into the second generation model, and the second generation model generates sample response text that conforms to the constraints of the preset role information; the preset role information is used to constrain at least one of the expression style, timbre and background of the person in the sample response text.

19. The method according to claim 16, characterized in that, The sample structured label information is organized according to a unified preset format. The sample response text and the sample structured label information together constitute a single output sequence, which serves as the annotation data.

20. The method according to claim 16, characterized in that, The initial model includes a backbone network and a low-rank adapter inserted into the backbone network. The backbone network includes the speech coding part and the original decoding part. The decoding part includes the original decoding part and the low-rank adapter. The low-rank adapter is used to carry the output format of preset role information and structured tag information. The preset role information is used to constrain at least one of the following: the expression style, timbre, and background of the character in the reply text. Based on the loss between the predicted data and the labeled data, the decoding part is weighted and updated to obtain an end-to-end voice interaction model, including: Based on the loss between the predicted data and the labeled data, the weights of the low-rank adapter are updated, while the weights of the original decoding part remain unchanged, to obtain an end-to-end voice interaction model.

21. The method according to claim 20, characterized in that, The weights of the speech coding part are stored in a first floating-point format, the weights of the original decoding part are stored in a first quantization format, and the weights of the low-rank adapter are stored independently in a second floating-point format and are not merged with the weights of the original decoding part stored in the first quantization format. Based on the loss between the predicted data and the labeled data, the weights of the low-rank adapter are updated, while the weights of the original decoding part remain unchanged, to obtain an end-to-end speech interaction model, including: During the forward propagation process, the weights of the original decoded portion stored in the first quantization format are dequantized to the first floating-point format to participate in the forward calculation, thereby obtaining the predicted data. During backpropagation, the gradient is calculated based on the loss between the predicted data and the labeled data; Based on the gradient, the weights of the low-rank adapter are updated, while the weights of the original decoding part remain unchanged, to obtain an end-to-end voice interaction model.

22. The method according to claim 20 or 21, characterized in that, The low-rank adapter is inserted into the attention layer and feedforward network layer of the decoding section.

23. A voice interaction device, characterized in that, Applied to a terminal device, wherein an end-to-end voice interaction model is deployed on the terminal device, the device includes: The acquisition module is used to acquire voice signals; The encoding module is used to encode the speech signal into an embedding sequence through the speech encoding part of the speech interaction model, wherein the embedding sequence is aligned with the text representation space processed by the decoding part of the speech interaction model. The generation module is used to perform semantic understanding on the embedded sequence in the same inference process through the decoding part, and jointly generate response text and structured tag information based on the semantic understanding results. The structured tag information carries multiple information tags in a machine-readable structure. The interaction module is used to generate and play a reply voice based on the reply text, and to trigger corresponding interactive response operations based on the information tags in the structured tag information.

24. A training device for a voice interaction model, characterized in that, The initial model to be trained includes a speech encoding part and a decoding part, and the device includes: The first generation module is used to generate multiple sample query information carrying information tags based on the prompt information corresponding to various information tags. The second generation module is used to generate sample response text based on the sample query information; the construction module is used to construct sample structured tag information in a machine-readable structure based on the information tags carried in the sample query information. The annotation module is used to combine the sample response text and the sample structured tag information as annotation data corresponding to the sample query information. The training module is used to input the sample query information into the decoding part of the initial model, and the decoding part jointly outputs the predicted response text and the predicted structured label information as prediction data. An update module is used to update the weights of the decoding part based on the loss between the predicted data and the labeled data, so as to obtain an end-to-end voice interaction model. The voice interaction model is used to jointly output the response text and structured label information in the same inference process.

25. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program, and when the processor executes the computer program, it implements the voice interaction method as described in any one of claims 1 to 15, or the training method as described in any one of claims 16 to 22.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the voice interaction method as described in any one of claims 1 to 15, or the training method as described in any one of claims 16 to 22.

27. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements the voice interaction method as described in any one of claims 1 to 15, or implements the training method as described in any one of claims 16 to 22.

Images