End-to-end voice interaction control method and device based on multi-head output

Abstract

Embodiments of the present application provide an end-to-end voice interaction control method, device and computer readable storage medium based on multi-head output. The method comprises obtaining target acoustic feature data corresponding to an instruction of a target voice; generating a target agent matching result according to the target acoustic feature data; inputting the target agent matching result and the target acoustic feature data into an end-to-end model corresponding to the target acoustic feature data to obtain multiple output results; and generating a target voice feedback result by fusing the multiple output results of the end-to-end model. In this way, in complex scenarios such as smart home control that require accurate and structured instruction output, control commands (for example, JSON objects containing devices, actions and parameters) can be directly and efficiently generated, greatly improving user experience.

Description

Technical Field

[0001] Embodiments of this application relate to the field of data processing, and more particularly to end-to-end voice interaction control methods, apparatus, devices, and computer-readable storage media based on multi-head output. Background Technology

[0002] Currently, most mainstream smart home voice interaction systems, such as those used in common smart speakers and smart assistants, are based on a pipeline architecture. While they can achieve voice interaction control, they have revealed many shortcomings in practical applications, specifically:

[0003] a. High latency, impacting user experience: Because multiple modules must complete their respective processing tasks sequentially, the cumulative latency of the entire interaction process is very significant;

[0004] b. Error accumulation reduces system robustness: In a cascaded architecture, the output of each module becomes the input of the next module. That is, any recognition or comprehension error generated by any module will be propagated to subsequent modules and may be amplified, thus affecting the accuracy of the final result;

[0005] c. High resource consumption and high deployment cost: Cascaded architectures typically require the deployment and maintenance of independent models for core modules such as ASR, NLU, and TTS.

[0006] d. The interactive experience is not natural and smooth enough: Since each module works independently and has different optimization goals, there may be a lack of close coordination between the voice response generated by the system and the control operation executed, which will make the interaction seem stiff and not user-friendly. Summary of the Invention

[0007] According to embodiments of this application, an end-to-end voice interaction control scheme based on multi-head output is provided. This scheme abandons the traditional multi-module serial cascade architecture and employs a single, unified deep learning model to directly map the user's original voice input to the system output. That is, a unified deep learning model enables direct mapping from user voice input to voice responses and device control commands, significantly improving response speed, recognition accuracy, and user experience.

[0008] In a first aspect of this application, an end-to-end voice interaction control method based on multi-head output is provided. The method includes:

[0009] Obtain the target acoustic feature data corresponding to the target speech command;

[0010] Based on the target acoustic feature data, a target agent matching result is generated;

[0011] The target intelligent agent matching result is used to input the target acoustic feature data into the end-to-end model corresponding to the target acoustic feature data, resulting in multiple output results;

[0012] The target speech feedback result is generated by fusing multiple outputs from the end-to-end model.

[0013] Furthermore, generating the target agent matching result based on the target acoustic feature data includes:

[0014] The target acoustic feature data is input into a pre-constructed audio buffer;

[0015] If the data in the audio buffer meets the preset requirements, then the features of the target acoustic feature data already stored in the audio buffer are extracted;

[0016] The features are input into the routing classification module to determine the Agent type of the target acoustic feature data;

[0017] Based on the Agent type, the target agent matching result is obtained.

[0018] Furthermore, the matching result of the target agent, the input of the target acoustic feature data to the end-to-end model corresponding to the target acoustic feature data, and the resulting outputs include:

[0019] The target agent matching result is input into the end-to-end model corresponding to the target acoustic feature data;

[0020] The target agent matching results are feature-encoded to obtain encoded features;

[0021] The encoded features are input in parallel to a multi-head output encoder to generate audio output, text sequences, and structured device control commands.

[0022] Furthermore, before inputting the encoded features in parallel to a multi-head output encoder to generate audio output, text sequences, and structured device control commands, the method further includes:

[0023] Based on the encoding characteristics, determine the user's intent type;

[0024] Generate a gating value based on the intent type;

[0025] The output strength is determined based on the stated gate value;

[0026] If the output intensity is greater than or equal to the control threshold, then a structured device control command is generated through the multi-head output encoder.

[0027] Furthermore, the generation of structured device control commands through the multi-head output encoder includes:

[0028] The encoded features are input into the control command head of the multi-head output encoder to obtain the target device type, control action, and control parameters.

[0029] The target device type, control action, and control parameters are combined to obtain structured device control commands.

[0030] Furthermore, the end-to-end model adopts a multi-Agent collaborative architecture.

[0031] Furthermore, the process of fusing and generating the target speech feedback result based on multiple outputs of the end-to-end model includes:

[0032] Based on the voice data in the output, generate the first feedback result;

[0033] Based on the control commands in the output, a second feedback result is generated;

[0034] Based on the first feedback result and the second feedback result, the target speech feedback result is generated by fusing them together.

[0035] In a second aspect of this application, an end-to-end voice interaction control device based on multi-head output is provided. The device includes:

[0036] The acquisition module is used to acquire the target acoustic feature data corresponding to the target speech command;

[0037] The generation module is used to generate target intelligent agent matching results based on the target acoustic feature data;

[0038] The output module is used to input the target intelligent agent matching result and the target acoustic feature data into the end-to-end model corresponding to the target acoustic feature data to obtain multiple output results;

[0039] The feedback module is used to fuse multiple outputs from the end-to-end model to generate the target speech feedback result.

[0040] In a third aspect of this application, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0041] In a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method according to the first aspect of this application.

[0042] The end-to-end voice interaction control method based on multi-head output provided in this application embodiment acquires target acoustic feature data corresponding to the target voice command; generates target agent matching results based on the target acoustic feature data; inputs the target agent matching results into the end-to-end model corresponding to the target acoustic feature data to obtain multiple output results; and fuses the multiple output results of the end-to-end model to generate target voice feedback results, which significantly improves response speed, recognition accuracy, and user experience.

[0043] It should be understood that the description in the Summary Section is not intended to limit the key or essential features of the embodiments of this application, nor is it intended to restrict the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0044] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0045] Figure 1 A system architecture diagram relating to the methods provided in the embodiments of this application.

[0046] Figure 2 A flowchart of an end-to-end voice interaction control method based on multi-head output according to an embodiment of this application;

[0047] Figure 3 This is a flowchart of the streaming inference process according to an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of a condition control generation mechanism according to an embodiment of this application;

[0049] Figure 5 This is a structural diagram of a multi-head output model according to an embodiment of this application;

[0050] Figure 6 This is a flowchart illustrating the model training process according to an embodiment of this application;

[0051] Figure 7 This is a comparative architectural diagram of embodiments according to this application;

[0052] Figure 8 This is a block diagram of an end-to-end voice interaction control device based on multi-head output according to an embodiment of this application;

[0053] Figure 9 This is a schematic diagram of the structure of a terminal device or server suitable for implementing the embodiments of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0055] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0056] Figure 1 A system architecture diagram is shown in which embodiments of the present disclosure can be implemented, including:

[0057] User voice input: Used to acquire voice commands input by users through devices such as microphones.

[0058] Feature extraction: This is used for preprocessing and feature extraction of the received raw speech signal. Preprocessing includes noise reduction, voice activity detection (VAD), etc.; feature extraction typically involves converting the raw waveform into acoustic features more suitable for deep learning model processing, such as Mel-frequency cepstral coefficients (MFCC), Fbank features, or directly using the spectrogram of the raw waveform.

[0059] Classification module: Used to initially classify the user's voice input into different predefined Agent types;

[0060] Multiple dedicated end-to-end models: Based on the classification results of the classification module, the user's voice features are input into the corresponding Agent's dedicated end-to-end (E2E) model for processing;

[0061] Multi-head output: Used to output different results based on different Agent types;

[0062] The Info E2E model outputs voice, text, and query results (query results can be viewed as a special form of text or structured data).

[0063] The above outputs are generated in parallel. Voice output is used to provide real-time feedback to the user, text output can be used for display or recording, and control commands are used to operate the corresponding devices or services.

[0064] Player: Used to aggregate all the voices generated by the Agent and play the synthesized voices to the user;

[0065] Device Controller: Used to send control commands generated by TV Agent, Music Agent, and IoT Agent to a unified device controller; the device controller is responsible for parsing these structured commands and interacting with the target device through the corresponding communication protocols (such as Wi-Fi, Zigbee, Bluetooth, etc.) to execute user instructions.

[0066] The overall architectural advantage of this disclosure lies in its initial task distribution through a classification module, enabling the backend to use more targeted E2E models, thus balancing model specialization with system scalability. Furthermore, all E2E models share a similar multi-head output concept, simplifying system design and implementation.

[0067] Figure 2 A flowchart of an end-to-end voice interaction control method based on multi-head output according to an embodiment of the present disclosure is shown. The method includes:

[0068] S210, acquire the target acoustic feature data corresponding to the target speech command.

[0069] In some embodiments, the target voice input by the user can be obtained through devices such as microphones and smart speakers.

[0070] Preprocessing of the target speech, such as noise reduction and voice activity detection (VAD), can improve the data quality of the target speech and provide a foundation for subsequent steps.

[0071] After preprocessing, feature extraction is performed to convert the original waveform into target acoustic feature data that is more suitable for deep learning models to process; for example, Mel frequency cepstral coefficients (MFCC), Fbank features, or directly using the spectrogram of the original waveform.

[0072] S220, Based on the target acoustic feature data, generate the target intelligent agent matching result.

[0073] In some embodiments, the extracted target acoustic feature data is input into a trained routing classification module; the user's voice input is initially classified into different predefined Agent types; for example, based on the extracted target acoustic feature data, it can be classified into:

[0074] Chat Agent (General Chat): Handles general conversations such as small talk and question-and-answer sessions;

[0075] TV Agent: Handles television-related commands, such as changing channels and adjusting volume;

[0076] Music Agent: Handles music playback-related commands, such as playing songs and switching songs;

[0077] IoT Agent (Smart Home Controller): Handles control commands to other smart home devices;

[0078] Info Agent: Handles information queries in specific fields, such as weather and stocks.

[0079] The routing classification module can be a lightweight classification model, the purpose of which is to distribute tasks to more specialized backend models, thereby improving overall processing efficiency and accuracy.

[0080] It should be noted that in practical applications, if the application is only for a specific field (e.g., only smart home control), then the routing classification module may not be necessary.

[0081] In some embodiments, streaming inference can be used for data processing to address the stuttering and latency issues caused by the traditional "wait for the whole sentence to finish before processing" (non-streaming) model.

[0082] Specifically, such as Figure 3 As shown, it continuously receives audio data streams from the microphone. Audio data streams are typically transmitted and processed in very small time chunks.

[0083] An audio buffer is constructed. Whenever a new audio segment arrives, it is determined whether the data in the buffer is sufficient for a valid feature extraction and model inference (e.g., accumulating audio of a specific duration or detecting the end of a speech segment). If insufficient, reception continues; otherwise, feature extraction is performed by retrieving sufficient audio data from the buffer for acoustic feature extraction. In streaming processing, feature extraction is typically incremental. The features are input to a routing classification module to determine the Agent type of the target acoustic feature data; based on the Agent type, a target agent matching result is obtained.

[0084] Data processing is performed using streaming inference. Once the audio decoder generates a portion of the speech data (e.g., a few milliseconds or tens of milliseconds of speech), it can be immediately sent to the audio player for playback. This achieves extremely low time-to-first-byte (TTFB) latency, allowing users to hear the system's initial feedback (e.g., a simple "hmm" or a start playback response) almost simultaneously with their speech.

[0085] S230, the target agent matching result is used to input the target acoustic feature data into the end-to-end model corresponding to the target acoustic feature data, and multiple output results are obtained.

[0086] In some embodiments, such as Figure 1 As shown, based on the classification results of the classification module, the target acoustic feature data is input into the corresponding Agent's dedicated end-to-end (E2E) model for processing. For example:

[0087] The architecture includes Chat E2E models, E2E models, Music E2E models, IoT E2E models, and / or Info E2E models; each dedicated E2E model is a multi-head output model. The multi-agent architecture allows for the use of models of different scales and optimization objectives for different types of tasks. For example, a Chat Agent might require stronger language generation capabilities, while an IoT Agent might need more precise control command generation capabilities.

[0088] like Figure 4 As shown, feature extraction is performed on the target acoustic feature data, converting the original audio signal (Audio Input) into an acoustic feature sequence (e.g., Fbank features). This acoustic feature sequence is then input into a shared encoder. The shared encoder is the core component of the model, responsible for learning deep, unified acoustic and semantic representations (EncodedFeatures, D) of the input speech. Currently mainstream sequence modeling networks can be used, such as Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), or Transformer encoders. It should be noted that the features extracted by this encoder can be used by all subsequent output heads, facilitating knowledge transfer and complementarity between different tasks, enabling the model to learn more robust and generalized representations.

[0089] The encoded features obtained by the shared encoder will be fed in parallel into multiple different output heads (decoders), each head responsible for generating a specific type of output:

[0090] Audio Head: Used to directly generate audio output from coded features. It can be a vocoder or a sequence-to-sequence acoustic parameter generation model, ultimately outputting a token sequence of an audio codec or directly a playable speech waveform; the goal of this head is to generate natural, fluent audio that matches the user's expectations. Figure 1 A respectful voice response;

[0091] Text Head: Used to generate a corresponding text sequence from encoded features. It can be an attention-based sequence-to-sequence decoder, and the generated text can be used for various purposes, such as:

[0092] Displayed to the user on the screen; stored as a conversation log; used to assist in debugging and model analysis;

[0093] In some scenarios where voice output is not required, it can be used directly as a text reply;

[0094] Control Header: Used to generate structured device control commands from coded features.

[0095] The model described above includes a shared encoder and all output heads, and can be trained end-to-end using a unified loss function. This avoids the suboptimal problems that may result from independent optimization of each module in a cascaded system, and thus better optimizes overall performance.

[0096] The shared encoder design allows most model parameters to be shared across different tasks (speech generation, text generation, control command generation), significantly reducing the total number of model parameters, lowering deployment costs and resource consumption, while improving training efficiency.

[0097] Since all outputs are generated in parallel from the same shared features, it is possible to achieve synchronous generation of voice responses and control commands (which is difficult to achieve in traditional cascaded systems).

[0098] Decoders for different tasks can indirectly influence each other during training by sharing an encoder, promoting knowledge transfer. For example, learning for text generation tasks may help improve the encoding of semantic information in speech, thereby indirectly improving the accuracy of control command generation.

[0099] In some embodiments, traditional voice interaction systems generate JSON-formatted strings directly when generating control commands. While this method is direct, it is easily susceptible to minor errors in the model that render the JSON format invalid, thus causing the entire control command to fail.

[0100] To address the aforementioned issues, this disclosure proposes a structured control command generation mechanism. Its core idea is to decompose the complex command generation task into several simpler and more controllable sub-tasks.

[0101] Specifically, such as Figure 5 As shown, the control command header includes a device classifier, an action classifier, and a parameter generator.

[0102] Among them, the device classifier:

[0103] Function: This classifier is responsible for identifying the type of target device to which the user intent is directed from the encoded features provided by the shared encoder;

[0104] Implementation: A Softmax activation function can be added to a simple linear layer to output the probability distribution of each predefined device type; for example, given the input "turn on the living room light", the device ID corresponding to "living room light" can be output.

[0105] Action Classifier:

[0106] Function: Similar to a device classifier, this classifier is responsible for identifying control actions in the user's intent;

[0107] Implementation: Similarly, a Softmax layer can be added to a linear layer to output the probability distribution of each predefined action type; for example, given the input "turn on the living room light", the action ID corresponding to "turn on" can be output.

[0108] Param Generator:

[0109] Function: Some control commands require additional parameters besides the device and action. For example, in the command "set the air conditioner to 26 degrees", the parameter generator is responsible for extracting or generating the parameter value from the encoded features.

[0110] Implementation: The types of parameters can be diverse (numerical, enumeration, text, etc.), making the implementation of parameter generators more complex. For simple enumeration parameters, a device / action classifier-like approach can be used; for numerical parameters, a regression model can be used; for text parameters (such as song titles, notification content), a lightweight sequence-to-sequence decoder can be used.

[0111] Furthermore, the outputs of each classifier (generator) are combined (device, action, and control parameters) to form a structured control command that may include an object or dictionary containing device, action, and parameter fields. This has the following advantages compared to directly generating a JSON string:

[0112] 1. Higher robustness: Classification tasks are generally easier to learn and control than unconstrained text generation tasks. Even if there are slight deviations in the parameter generation part, as long as the device and action are classified correctly, the system can still understand most of the user's intent or provide clarification;

[0113] 2. Easier error localization: When a command fails to execute, it is easier to determine whether the problem lies in the incorrect identification of the device, the incorrect identification of the action, or the problem with parameter extraction.

[0114] 3. Easy to expand and maintain: Adding new device types or action types only involves modifying the output dimension of the classifier and the training data, with minimal impact on other parts of the model;

[0115] 4. Ensure correct format: Since the command is generated by combining predefined fields and classification results, its basic format is always correct, avoiding problems such as JSON syntax errors.

[0116] In some embodiments, users' intentions in everyday human-computer voice interaction are often diverse. For example, a user may explicitly want to control a device (e.g., "turn on the light"); want to engage in casual conversation (e.g., "I'm happy today, how about you?"); or have mixed intentions (e.g., "I'm happy today, can you play some upbeat music?"). If the system indiscriminately attempts to generate control commands for all inputs, it may lead to accidental device triggering in purely conversational scenarios or the generation of meaningless control instructions, affecting user experience and system security.

[0117] To address the aforementioned issues, this disclosure establishes a conditional control generation mechanism. The core purpose of this mechanism is to intelligently determine whether the current user's voice input contains a genuine control intent, and based on the determination result, decide whether to activate the control command generation process.

[0118] Specifically, such as Figure 5 As shown, a conditional control generation mechanism can be added before the control command head of the multi-head output model, or as an early processing stage within the control command head. Its workflow is as follows:

[0119] The feature encoding is input into the intent classifier to determine the user's intent graph; for example, intents can be categorized as follows:

[0120] Chat / Informational Intent: The user's primary purpose is to obtain information, engage in casual conversation, or perform non-controlling tasks;

[0121] Control Intent: A user's explicit desire to control a device or perform an action.

[0122] Mixed Intent: A user intent that includes both dialogic and control elements;

[0123] The intent classifier can be a lightweight model trained independently, or it can be jointly trained with the main E2E model.

[0124] Furthermore, based on the classification results, a gate value is generated; this gate value can be a continuous scalar (e.g., between 0 and 1) used to adjust the strength or probability of generating control commands.

[0125] If the intent is determined to be purely conversational, the gating value can be set to a lower value (e.g., 0 or close to 0).

[0126] If the intention to control is clearly defined, the gate value can be set to a higher value (e.g., 1 or close to 1).

[0127] If the intention is determined to be mixed, the gate value can be an intermediate value (e.g., 0.5).

[0128] Gating values ​​can be applied to the output of the control command generator. This can be achieved in several ways, for example:

[0129] The final output (or its confidence level) of the control command is directly multiplied by the gating value;

[0130] The gating value can be fed as an additional input condition into various components of the control command generator (device classifier, motion classifier, parameter generator).

[0131] Gating values ​​can be used to adjust the threshold of activation functions on the control command generation path.

[0132] Furthermore, through gating adjustment, the decision on whether to actually generate and execute the control command can be made based on the final control output strength or confidence level.

[0133] If the output strength is lower than the control threshold, it is considered that it is not suitable to generate a control command at this time, and the system will not generate a control command.

[0134] If the output strength is greater than or equal to the control threshold, a control command is generated.

[0135] Through the above mechanism, the probability of erroneous activation of device control in pure dialogue or unrelated scenarios is significantly reduced, thereby improving system security and user experience.

[0136] Improved naturalness of interaction: enabling the system to more accurately understand the boundaries of user intent, remain silent when no control is needed, and respond decisively when control is required;

[0137] Resource optimization: When the intent is determined to be non-control intent, the calculation of the control command header can be stopped in advance to save computing resources;

[0138] Learnable and adaptive: The gating mechanism itself can also learn; the model can automatically learn to determine when to generate control commands through data-driven methods, thereby better adapting to various complex scenarios.

[0139] In some embodiments, end-to-end model training can be performed in the following manner, such as... Figure 6As shown, data accumulated by existing cascaded voice interaction systems (as described in the background art) (including user voice, ASR-transcribed text, NLU analysis results (intent, slot), system response text, TTS-generated speech, and actual executed control commands, etc.) can be used as sample data.

[0140] To improve the robustness and generalization ability of the model, the collected sample data can be augmented in the following ways:

[0141] Speech enhancement: Add background noise, reverberation, change speech rate, pitch, etc.;

[0142] Text enhancement: synonym replacement, sentence rewriting, back translation, etc.;

[0143] The enhanced data is fed into the model in mini-batches for training to balance computational efficiency and gradient estimation accuracy. Each batch of data is input into the E2E model to perform forward propagation. That is, input speech features are passed through a shared encoder and then in parallel through an audio generation head, a text generation head, and a control command head (including a device classifier, action classifier, parameter generator, etc.) to obtain various predicted outputs. The model's forward propagation produces multiple outputs, each of which needs to be compared with the ground truth in the training data to calculate the corresponding loss function. Due to the multi-head output architecture, the total loss is typically a weighted sum of the losses from each head:

[0144] Audio loss: Compares the difference between the audio generated by the model (or its acoustic parameters) and the target audio. The loss function can be mean squared error (MSE), spectral distance, or cross-entropy loss based on the codec token predicted by the vocoder, etc.

[0145] Text loss: Compares the difference between the text generated by the model and the target text transcription. A commonly used loss function is cross-entropy loss, such as the typical approach based on the Transformer decoder;

[0146] Losses related to control commands: For structured control command headers, there will also be corresponding losses for each of their internal components:

[0147] Device classification loss: Compare the predicted device ID with the actual device ID, typically using cross-entropy loss.

[0148] Action classification loss: Compare the predicted action ID with the real action ID, typically using cross-entropy loss;

[0149] Furthermore, all individual losses are weighted and summed using preset weights (α, β, γ, δ, etc.) to obtain the total loss value:

[0150] L = α * L_audio + β * L_text + γ * L_device + δ * L_action + ε* L_param

[0151] Furthermore, the gradient of the loss function with respect to all trainable parameters of the model (including the parameters of the shared encoder and each output head) is calculated using the backpropagation algorithm. To prevent gradient explosion, especially when training deep networks such as RNNs or Transformers, the calculated gradients are clipped to a reasonable range (defined according to the specific application scenario). An optimizer (e.g., Adam, SGD) is then used to update the model parameters based on the calculated and clipped gradients, iteratively optimizing the model in the direction of loss reduction.

[0152] An epoch (L) is completed when the model has traversed all the training data once.

[0153] After each epoch (or after a certain number of training steps), evaluate the model's performance on a separate validation dataset. The validation set is not used in training; it is used to monitor the model's generalization ability and help determine if overfitting has occurred.

[0154] Evaluation metrics include: perceptual quality of speech generation (e.g., MOS score), accuracy of text generation (e.g., BLEU, WER), accuracy of various control commands (device, action, parameters), and end-to-end task completion rate.

[0155] Furthermore, based on the performance on the validation set, it is determined whether the model has achieved the expected target (N);

[0156] If the target is not achieved, the following adjustments can be made:

[0157] Adjusting the learning rate: The learning rate is a key hyperparameter that controls the step size of parameter updates; a learning rate decay strategy or dynamic adjustment of the learning rate can be used.

[0158] Adjust loss weights, model structure, data augmentation strategies, etc.

[0159] Continue training for the next batch or epoch;

[0160] If the objective is achieved (e.g., performance on the validation set no longer improves, or a preset performance threshold is reached), the model can be saved and training can end.

[0161] The above training process ensures that all parts of the E2E model are effectively optimized and can work together to complete complex voice interaction tasks.

[0162] S240, based on multiple outputs of the end-to-end model, fuses and generates the target speech feedback result.

[0163] like Figure 1 As shown, based on the voice data in the output results, the first feedback result is generated. That is, all the voice outputs generated by all agents are aggregated into the player, and the synthesized voice is played to the user;

[0164] Based on the control commands in the output, a second feedback result is generated. That is, the control commands generated by the TV Agent, Music Agent, and IoT Agent are sent to a unified device controller. The device controller parses the structured commands and interacts with the target device through the corresponding communication protocol (such as Wi-Fi, Zigbee, Bluetooth, etc.) to execute the user's instructions.

[0165] That is, the target speech feedback result is generated by fusing the first feedback result and the second feedback result.

[0166] According to the embodiments of this disclosure, the following technical effects are achieved.

[0167] like Figure 7 As shown, the end-to-end architecture of this disclosure significantly improves data processing time compared to the cascaded architecture in existing technologies. In complex scenarios requiring precise and structured command output, such as smart home control, it can directly and efficiently generate control commands (e.g., JSON objects containing devices, actions, and parameters), greatly enhancing the user experience.

[0168] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0169] The above is an introduction to the method embodiments. The following describes the solution described in this application through device embodiments.

[0170] Figure 8 A block diagram 800 of an end-to-end voice interaction control device based on multi-head output according to an embodiment of this application is shown, as follows: Figure 8 The following are included:

[0171] The acquisition module 810 is used to acquire the target acoustic feature data corresponding to the target speech command;

[0172] The generation module 820 is used to generate a target intelligent agent matching result based on the target acoustic feature data;

[0173] The output module 830 is used to input the target acoustic feature data into the end-to-end model corresponding to the target acoustic feature data, and obtain multiple output results by inputting the target intelligent agent matching result into the target acoustic feature data.

[0174] Feedback module 840 is used to fuse multiple outputs from the end-to-end model to generate target speech feedback results.

[0175] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0176] Figure 9 A schematic diagram of a terminal device or server suitable for implementing embodiments of this application is shown.

[0177] like Figure 9 As shown, the terminal device or server includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage section 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the terminal device or server. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0178] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0179] Specifically, according to embodiments of this application, the above method flow steps can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a machine-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs the functions defined above in the system of this application.

[0180] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0182] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be located in a processor. The names of these units or modules do not, in certain circumstances, constitute a limitation on the unit or module itself.

[0183] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium stores one or more programs that, when used by one or more processors, execute the methods described in this application.

[0184] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing application concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions claimed in this application.

Claims

1. An end-to-end voice interaction control method based on multi-head output, characterized in that, include: Obtain the target acoustic feature data corresponding to the target speech command; Based on the target acoustic feature data, a target agent matching result is generated; Based on the target agent matching result, the target acoustic feature data is input into the end-to-end model corresponding to the target acoustic feature data to obtain multiple output results; Based on the multiple outputs of the end-to-end model, the target speech feedback result is generated by fusing them. The step of generating the target agent matching result based on the target acoustic feature data includes: The target acoustic feature data is input into a pre-constructed audio buffer; If the data in the audio buffer meets the preset requirements, then the features of the target acoustic feature data already stored in the audio buffer are extracted; The features are input into the routing classification module to determine the Agent type of the target acoustic feature data; Based on the Agent type, the target agent matching result is obtained; Based on the target agent matching result, the target acoustic feature data is input into the end-to-end model corresponding to the target acoustic feature data to obtain multiple output results, including: Based on the target agent matching result, the target acoustic feature data is input into the end-to-end model corresponding to the target acoustic feature data; The target agent matching results are feature-encoded to obtain encoded features; The encoded features are input in parallel to a multi-head output encoder to generate audio output, text sequences, and structured device control commands; Before inputting the encoded features in parallel into a multi-head output encoder to generate audio output, text sequences, and structured device control commands, the method further includes: Based on the encoding characteristics, determine the user's intent type; Generate a gating value based on the intent type; The output strength is determined based on the stated gate value; If the output intensity is greater than or equal to the control threshold, then a structured device control command is generated through the multi-head output encoder.

2. The method according to claim 1, characterized in that, The generation of structured device control commands through the multi-head output encoder includes: The encoded features are input into the control command head of the multi-head output encoder to obtain the target device type, control action, and control parameters. The target device type, control action, and control parameters are combined to obtain structured device control commands.

3. The method according to claim 2, characterized in that, The end-to-end model adopts a multi-agent collaborative architecture.

4. The method according to claim 3, characterized in that, The process of fusing and generating the target speech feedback result based on multiple outputs of the end-to-end model includes: Based on the voice data in the output, generate the first feedback result; Based on the control commands in the output, a second feedback result is generated; Based on the first feedback result and the second feedback result, the target speech feedback result is generated by fusing them together.

5. An end-to-end voice interaction control device based on multi-head output, characterized in that, include: The acquisition module is used to acquire the target acoustic feature data corresponding to the target speech command; The generation module is used to generate target intelligent agent matching results based on the target acoustic feature data; The output module is used to input the target intelligent agent matching result and the target acoustic feature data into the end-to-end model corresponding to the target acoustic feature data to obtain multiple output results; The feedback module is used to fuse and generate the target speech feedback result based on multiple outputs of the end-to-end model. The step of generating the target agent matching result based on the target acoustic feature data includes: The target acoustic feature data is input into a pre-constructed audio buffer; If the data in the audio buffer meets the preset requirements, then the features of the target acoustic feature data already stored in the audio buffer are extracted; The features are input into the routing classification module to determine the Agent type of the target acoustic feature data; Based on the Agent type, the target agent matching result is obtained; The matching result of the target agent is used to input the target acoustic feature data into the end-to-end model corresponding to the target acoustic feature data, resulting in multiple output results, including: Based on the target agent matching result, the target acoustic feature data is input into the end-to-end model corresponding to the target acoustic feature data; The target agent matching results are feature-encoded to obtain encoded features; The encoded features are input in parallel to a multi-head output encoder to generate audio output, text sequences, and structured device control commands; Before inputting the encoded features in parallel into a multi-head output encoder to generate audio output, text sequences, and structured device control commands, the method further includes: Based on the encoding characteristics, determine the user's intent type; Generate a gating value based on the intent type; The output strength is determined based on the stated gate value; If the output intensity is greater than or equal to the control threshold, then a structured device control command is generated through the multi-head output encoder.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1 to 4.

Images