Voice interaction method and apparatus, device, and storage medium

Abstract

A voice interaction method and apparatus, a device, a storage medium, and a program product. The method comprises: in a voice interaction, acquiring an audio signal collected from an environment while outputting a voice response to a user, wherein the voice response is generated on the basis of a machine learning model (610); on the basis of context information related to the voice interaction and a first audio clip collected from the environment at a first time, using the machine learning model to determine whether a next output unit in the voice response outputted to the user is a voice unit or an interrupt symbol, wherein the interrupt symbol indicates that turn-taking has occurred in the voice interaction (620); and in response to determining that the next output unit is the interrupt symbol, stopping outputting the voice response to the user (630).

Description

Methods, apparatus, devices and storage media for voice interaction

[0001] This application claims priority to Chinese Patent Application No. 202411053858.4, filed on August 1, 2024, entitled "Method, Apparatus, Device and Storage Medium for Voice Interaction", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and particularly to methods, apparatus, devices and computer-readable storage media for voice interaction. Background Technology

[0003] Dialogue is a form of human-computer interaction (HCI). With the rapid development of language models (LMs) and transformer architectures, text-based conversational artificial intelligence has become an important part of daily life. However, these models are limited to text input and output and cannot directly interact with humans in any scenario. Voice-based HCI supports both voice input and output, and can match more practical interaction needs. Summary of the Invention

[0004] In a first aspect of this disclosure, a method for voice interaction is provided. The method includes: during voice interaction, acquiring audio signals collected from the environment while outputting a voice response to a user, the voice response being generated based on a machine learning model; using the machine learning model, based on contextual information related to the voice interaction and a first audio segment collected from the environment at a first moment, determining whether the next output unit in the voice response output to the user is a voice unit or an interrupt symbol, the interrupt symbol indicating a turn-taking transition occurring during the voice interaction; and in response to determining that the next output unit is an interrupt symbol, stopping the output of the voice response to the user.

[0005] In a second aspect of this disclosure, a voice interaction apparatus is provided. The apparatus includes: an audio signal acquisition module configured to acquire audio signals collected from the environment while outputting a voice response to a user during voice interaction, the voice response being generated based on a machine learning model; an output unit determination module configured to use the machine learning model to determine, based on contextual information related to the voice interaction and a first audio segment collected from the environment at a first moment, whether the next output unit in the voice response output to the user is a voice unit or an interrupt symbol, the interrupt symbol indicating a turn-taking transition occurring during the voice interaction; and a stop output module configured to stop outputting the voice response to the user in response to determining that the next output unit is an interrupt symbol.

[0006] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the electronic device to perform the method of the first aspect.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The medium stores a computer program that, when executed by a processor, implements the method of the first aspect.

[0008] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes a computer program that, when executed by a processor, implements the method of the first aspect.

[0009] It should be understood that the description in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] Figures 2A to 2D illustrate the voice interaction modes from a communication perspective;

[0013] Figure 3 illustrates a schematic diagram of the architecture of a machine learning model according to some embodiments of the present disclosure;

[0014] Figure 4A illustrates a schematic diagram of performing a fusion operation prior to at least one feature processing layer according to some embodiments of the present disclosure;

[0015] Figure 4B illustrates a schematic diagram of performing a fusion operation in each feature processing layer according to some embodiments of the present disclosure;

[0016] Figure 4C illustrates a schematic diagram of performing a fusion operation after at least one feature processing layer according to some embodiments of the present disclosure;

[0017] Figure 5 illustrates the probability distribution of interruption symbols over time according to some embodiments of the present disclosure;

[0018] Figure 6 shows a flowchart of a voice interaction method according to some embodiments of the present disclosure;

[0019] Figure 7 illustrates an exemplary structural block diagram of a voice interaction device according to some embodiments of the present disclosure; and

[0020] Figure 8 shows a block diagram of an electronic device that can implement one or more embodiments of the present disclosure. Detailed Implementation

[0021] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0022] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0023] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0024] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0025] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0026] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0028] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0029] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.

[0030] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as an input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. The testing phase can sometimes be integrated into the training phase. In the application or inference phase, the trained model can be used to process actual model inputs based on the trained parameter values ​​to determine the corresponding model output.

[0031] Figure 1 illustrates a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. In this example environment 100, an application 120 is installed on an electronic device 110. A user 130 can interact with the application 120 via the electronic device 110 and / or an attached device of the electronic device 110. For example, the application 120 can capture the voice 135 of the user 130 via a voice capture device (e.g., a microphone) of the electronic device 110.

[0032] In embodiments of this disclosure, application 120 can be any suitable application with human-computer dialogue capabilities. For example, application 120 can provide a digital assistant for human-computer dialogue. This digital assistant supports text-based dialogue services, voice interaction services, and content dialogue in other modalities with user 130. In some embodiments, application 120 or its digital assistant can utilize machine learning model 140 (which may include one or more machine learning models, such as machine learning model 140-1, machine learning model 140-2, ..., machine learning model 140-N, etc., where N is a positive integer. For ease of description, one or more machine learning models are collectively referred to herein as machine learning model 140) to support interaction with user 130. For example, application 120 or its digital assistant can utilize one or more machine learning models 140 to provide question-and-answer services to user 130.

[0033] In environment 100, if application 120 is active, electronic device 110 can present the user interface 150 of application 120. User interface 150 may include various pages that application 120 can provide, such as a user-digital assistant dialogue page (where current and historical dialogues, including text dialogue content, may be presented), and so on. In some embodiments, electronic device 110 may play voice 152 in user interface 150. Voice 152 may, for example, include voice 135 from user 130 or voice response to voice 145.

[0034] Machine learning model 140 can be of different types. In some embodiments, one or more machine learning models 140 may be built based on a language model. The machine learning model used is a content-generating model, capable of generating corresponding outputs based on model inputs. In some embodiments, the language model-based machine learning model can handle text-modal model inputs (e.g., natural language and / or machine language) and / or non-text-modal model inputs (e.g., images, speech, video, etc.), and can generate the desired output based on the model inputs and prompts. Here, prompts are used to guide the machine learning model to generate outputs that address the user needs indicated by the model inputs. In application scenarios supporting user dialogue, user 130's input can be provided to machine learning model 140 as at least a part of the model inputs (other parts may include prompts). This user input is considered a question. Based on the model outputs, corresponding responses can be generated and provided to user 130.

[0035] In Figure 1, electronic device 110 can be any type of computing device, including terminal devices or server devices. Terminal devices can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. Server devices can include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0036] It should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0037] As mentioned earlier, conversational AI cannot directly interact with humans. Integrating speaking and listening interfaces into conversational AI can improve the ease of human-computer interaction. Using text language modeling techniques, speech language models (SLMs) process speech in a text-like manner. This paradigm involves encoding speech signals into discrete tokens or continuous embeddings, modeling them with a language model, and decoding the tokens or embeddings back into the speech signal. Some studies have used this paradigm for speech continuation, generating expressive speech and natural multi-turn dialogues. Other studies have applied this paradigm to task-specific applications, such as decoder-only high-fidelity speech synthesis (text-to-speech, TTS) and decoder-only streaming automatic speech recognition (ASR). Furthermore, some related techniques use language models to initialize speech language models, expanding the speech tokens into the language model's dictionary and continuing speech training. This enables speech language models to understand semantic information and gives them conversational capabilities. However, these models are limited to turn-based dialogues and cannot handle real-time audio or interruptions, which limits their application in real life.

[0038] Interaction and turn-taking are fundamental abilities in natural human communication. Full-duplex modeling (FDM) in interactive speech language models (iSLMs) is a crucial aspect affecting user experience. Some related techniques propose using a separate understanding module to handle real-time audio input. Others suggest modifying the organization order of text lexical units in the language model to address the duplex modeling problem. These models are all based on text-centric language models and require external ASR and TTS modules for speech interaction. Therefore, latency remains perceptible, and paralinguistic ability is still lacking.

[0039] To address the aforementioned issues and enable full-duplex modeling for interactive speech language models, this disclosure proposes a speech interaction scheme. Specifically, in speech interaction, while outputting a speech response to the user, audio signals collected from the environment are acquired. The speech response is generated based on a machine learning model. Using the machine learning model, based on contextual information related to the speech interaction and a first audio segment collected from the environment at a first moment, the next output unit in the speech response output to the user is determined to be either a speech unit or an interrupt symbol, where the interrupt symbol indicates a turn-taking transition in the speech interaction. In response to determining that the next output unit is the interrupt symbol, the output of the speech response to the user is stopped.

[0040] According to the scheme disclosed herein, the machine learning model can acquire the collected audio signal while outputting the speech response, achieving a full-duplex working mode. By determining that the next output unit in the speech response is an interruption symbol, the machine learning model can quickly stop outputting. In this way, the machine learning model can simultaneously utilize information from multiple channels, improving the accuracy and fluency of real-time interaction capabilities, and enabling the determination of turn-taking in speech-based human-computer interaction processes.

[0041] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0042] Figures 2A-2D illustrate different voice interaction modes from a communication perspective.

[0043] Simplex speech language models are limited to a single channel, i.e., listening or speaking. Figure 2A shows the speech interaction mode of a simplex speech language model with listening capabilities, and Figure 2B shows the speech interaction mode of a simplex speech language model with speaking capabilities. With the help of the language model, simplex speech language models exhibit robust comprehension and generation capabilities. However, simplex speech language models are limited to one-way communication (human to machine or machine to human). Figure 2C shows a half-duplex speech interaction mode, the corresponding model of which is a half-duplex speech language model with listening and speaking capabilities. Although this model can listen and speak, it only performs one action at a time, thus failing to solve the turn-taking problem.

[0044] Figure 2D illustrates a full-duplex voice interaction mode, corresponding to a full-duplex speech language model capable of simultaneous listening and speaking. This model allows for turn-taking when a human interrupts the machine. As shown in Figure 2D, at time 205, noise and human voice are detected, thus triggering the first turn-taking and halting the machine's (e.g., application 120) speech output. At time 210, the end of the human voice is detected, triggering the second turn-taking, and the machine resumes speech output.

[0045] Before describing the full-duplex machine learning model 140 of this application, we will first introduce the simplex or half-duplex models.

[0046] Simplex or half-duplex voice interaction systems can be modeled by finding a parameter θ that maximizes the log-likelihood function. This process can be represented as follows:

[0047] Where (C,R) represents a context-response pair in dataset D, P θ (R|C) represents the probability of response R given context C and parameters θ. In one example, if the voice interaction system is modeled using an autoregressive language model, where the response R is generated word-by-word, the training loss for each sample is... It can be represented as follows:

[0048] Where R 1:t-1 =[r1,r2,…,r t-1 [], where T represents the sequence length. During the inference phase, the model can only predict the next term autoregressively based on previous outputs within the current channel and not on information from other channels.

[0049] When modeling a full-duplex voice interaction system in an autoregressive language model, the model needs to be based not only on the context C, but also on the response history R generated in the current channel. 1:t-1 =[r1,r2,…,r t-1 ], and simultaneously utilize information S from another channel. 1:t-1 =[s1,s2,…,s t-1 Predict the next term r in the response R. t This allows us to extend the modeling methods used for simplex or half-duplex dialogue systems to meet the requirements of full-duplex modeling, and to adjust the training loss. It can be represented as follows:

[0050] The key point of full-duplex modeling is that the sequence S is generated by facts and is unpredictable. Taking a full-duplex speech language model as an example, in inference step t-1, the current speaking channel generates the output r. t-1 And the input s obtained from the listening channel t-1 Simultaneously, it is fed into the model, affecting the next output r of the speaking channel. t The prediction.

[0051] Embodiments of this disclosure propose automatic prediction of turn-taking in voice-based human-computer interaction, thereby supporting full-duplex voice interaction. Figure 3 illustrates a schematic diagram of the architecture of a machine learning model 140 according to some embodiments of this disclosure. The machine learning model 140 can be configured to process voice input data. As shown in Figure 3, the machine learning model 140 may include a voice encoder 305 and a language model-based voice decoder 310.

[0052] In some embodiments, the machine learning model 140 includes a speaking channel and a listening channel, the speaking channel being configured to output a voice response for a user, and the listening channel being configured to acquire audio signals in the environment (e.g., noise, human voices, etc.).

[0053] In voice interaction, the voice decoder 310 outputs a voice response 315 to the user 130, which is generated based on the machine learning model 140. Simultaneously with outputting the voice response 315, audio signals 320 collected from the environment can also be acquired. In some embodiments, the audio signals 320 can be acquired using a voice acquisition device (e.g., a microphone) of the electronic device 110.

[0054] To simulate the speaking ability of machine learning model 140, a speech decoder 310 can be implemented using an autoregressive, lexical-based TTS model. The speech decoder 310 employs a single-layer discrete audio lexical structure, a design that better meets the requirements of real-time interaction and eliminates the need to wait for autoregressive lexical synthesis before performing non-autoregressive operations. Given target speech X... R The embedded representation R can be obtained using the speech encoder 305, and this process can be represented as follows: R = Enc(X R (4)

[0055] To train an autoregressive TTS model based on discrete lexical units, the speech embedding representation R can be quantized. This process can be represented as follows: R q =Qnt(R), (5)

[0056] Where Qnt represents the discretization operation, R q Let C represent discrete lexical units. Given contextual information C, in this case, the model autoregressively synthesizes the corresponding discrete lexical units for the text content to be synthesized. The negative log-likelihood of the target sequence can be minimized, conditioned on the previous lexical units and context, to train the TTS model. The loss function for this model is defined as follows:

[0057] Where θS t represents the parameters used to model speaking ability. EOS Indicates the time step until the end of the sequence term. This represents the target discrete word at time step t. This represents all words up to time step t-1. During inference, the model moves from the generated words... Conditional distribution and sampling in C above This process can be represented as follows:

[0058] Using machine learning model 140, based on contextual information 325 related to the voice interaction and a first audio segment captured from the environment at a first moment, the next output unit of the voice response 315 output to user 130 is determined to be either a speech unit or an interruption symbol (IRQ). An interruption symbol indicates a turn-taking event occurring during the voice interaction. In some embodiments, the first moment may be the initial time at which the voice response 315 is output to user 130.

[0059] In some embodiments, taking time 345 as an example, a fusion operation can be performed on the first embedded representation 330 of the first audio segment acquired at the first time and the second embedded representation 335 of the current output unit to obtain a fused embedded representation, and based on the context information 325 and the fused embedded representation.

[0060] In some embodiments, the speech decoder 310 includes at least one feature processing layer (sometimes also called a transformer layer) and an output layer (sometimes also called a softmax layer), the output layer being after the at least one feature processing layer. The following description, with reference to FIG4A-4C, illustrates the fusion of the first audio segment and the current output unit at different locations within the speech decoder 310.

[0061] Figure 4A illustrates a schematic diagram 400A of performing a fusion operation before at least one feature processing layer according to some embodiments of the present disclosure. As shown in Figure 4A, taking a first embedded representation 405 and a second embedded representation 410 as examples, a fusion operation is performed on the first embedded representation 405 and the second embedded representation 410 before at least one feature processing layer 415, that is, the listening channel and the speaking channel are fused at the input embedded representation before autoregressive prediction. Therefore, the fused embedded representation can be processed by at least one feature processing layer 415.

[0062] FIG. 4B shows a schematic diagram 400B of performing a fusion operation in each feature processing layer according to some embodiments of the present disclosure. As shown in FIG. 4B, taking the first embedding representation 405 and the second embedding representation 410 as examples, a fusion operation is performed on the first embedding representation 405 and the second embedding representation 410 in each feature processing layer of at least one feature processing layer 415. Specifically, in addition to the hidden state of the speech channel and the positional embedding representation, the listening channel can be additionally added to the input of each feature processing layer.

[0063] FIG. 4C shows a schematic diagram 400C of performing a fusion operation after at least one feature processing layer according to some embodiments of the present disclosure. As shown in FIG. 4C, continuing to take the first embedding representation 405 and the second embedding representation 410 as examples, after the embedding representation 402 of the current output unit is processed by at least one feature processing layer 415, a fusion operation is performed on the first embedding representation 405 and the second embedding representation 410.

[0064] In this way, the embedding representation related to the collected audio segment and the embedding representation related to the input speech response can be fused in various ways, improving the efficiency of fusion, so that the machine learning model proposed in this application can effectively utilize multi-channel information and improve the accuracy and fluency of real-time interaction capabilities.

[0065] In some embodiments, based on the context information 325 and the first audio segment, it can be detected whether the first audio segment includes a predetermined instruction. If it is detected that the first audio segment includes a predetermined instruction, it is determined that the next output unit is the interruption symbol. In one example, the predetermined instruction can be a predetermined English word or Chinese, such as, "honey" or "宝贝" etc.

[0066] In some embodiments, based on the context information 325 and the first audio segment, it can be detected whether the first audio segment contains human voices. If it is detected that the first audio segment includes human voices, it is determined that the next output unit is the interruption symbol. In one example, the human voices can include a passage or a few words spoken by different users.

[0067] In this way, the output of the machine learning model 140 can be flexibly interrupted, thereby reducing redundant information and saving computing resources.

[0068] In some embodiments, the interval between the time of the interruption symbol and the occurrence time of the predetermined instruction and / or human voices is determined within a predetermined time interval. The machine learning model 140 may need a certain amount of time to determine whether the acquired audio signal includes a predetermined instruction and / or human voices, and then output the interruption symbol. The predetermined interval can be set to 0.5 seconds, or less, so as not to affect the user experience.

[0069] In some embodiments, training samples for training the machine learning model 140 can be constructed based on time intervals. For example, if the time interval is set to 0.5 seconds, the ground truth data for outputting interrupt symbols in the training samples can be set within 0.5 seconds of the occurrence time of a predetermined instruction and / or human voice. Therefore, after training, the machine learning model 140 has a greater probability of outputting interrupt symbols within the predetermined time interval. The effect of constructing training samples based on time intervals will be explained below with reference to FIG5.

[0070] Figure 5 shows the probability distribution of the interruption symbol over time. As shown in the figure, when the machine learning model 140 is not interrupted, the probability of outputting the interruption symbol remains at 1×10. -3 Below. When the listening channel begins receiving real-time voice-over transitions, the machine learning model 140 can detect whether it is an interruption or noise. Shortly after an interruption occurs, the probability of outputting an interruption symbol begins to increase rapidly. Therefore, the machine learning model 140 has a greater probability of outputting an interruption symbol within a predetermined time period during which a predetermined instruction and / or voice occurs.

[0071] In some embodiments, the context information 325 includes at least the voice signals acquired in the current dialogue turn of the voice interaction. For example, the context information 325 is a question asked by user 130.

[0072] After determining that the next output unit is an interruption symbol (IRQ), the output of voice response to user 130 can be stopped. At this point, the voice response output by the device is stopped because a turn-taking mechanism has been detected by the model, and the turn has been switched to the user, for example, the user expects to further supplement or modify the input voice query. After stopping the output of voice response to user 130, ambient audio signals can continue to be acquired to obtain the user's voice information. If another turn-taking mechanism is detected in the voice interaction, such as the user stopping speaking (e.g., the user's voice pause time is detected to exceed a threshold), then the turn can be switched back to the device so that the machine learning model 140 can restart the generation of voice response. The contextual information relied upon at this time may include the audio signals received between the two turn-taking mechanisms, as well as some or all of the previously used contextual information.

[0073] In some embodiments, in response to the machine learning model 140 determining that the next output unit is a speech unit (i.e., no turn-taking occurs during the output of the speech response), the speech response to be output can be determined based on the determined speech unit and the previous speech units in the speech response, and the speech response can continue to be output to the user.

[0074] According to embodiments of this disclosure, ambient sounds can be monitored in real time during voice interaction to determine whether a turn-taking switch occurs, thereby enabling full-duplex voice dialogue.

[0075] In some embodiments, a vocoder Dec can be used to extract discrete tokens. (e.g., the identified speech unit and previous speech units) to recover the speech signal (The voice response output to the user), this process can be represented as follows:

[0076] Here, A represents the acoustic cues that provide the synthesized speech timbre. This approach decouples timbre from content, allowing the speech decoder 310 to focus more on semantic information rather than auxiliary linguistic information.

[0077] For the language ability of machine learning model 140, given the audio input X from the listening channel. S The same speech encoder 305 used in formula (4) can be used to obtain a continuous embedding representation S, which can be expressed as follows: S = Enc(X S (9)

[0078] Where X S It represents various audio signals, including ambient noise and human voices. Unlike training speaking ability, it involves a discretization module. The embedded representation S of the listening channel is fed into the neural network via the projection module Proj. This process can be represented as follows: S p =Proj(S) (10)

[0079] The received audio signal is projected onto the space that the speech decoder 310 can process through the projection module, so that the embedded representation of the audio signal is aligned with the dimension of the embedded representation of each output unit in the speech decoder 310.

[0080] In some embodiments, after the machine learning model 140 determines that the next output unit is a speech unit (i.e., no turn-taking occurs during the output of the speech response) and continues to output the speech response based on the determined speech unit, the machine learning model 140 continues to work to predict the next output unit in the speech response. The process of determining the next output unit is similar to that discussed above. During each prediction of the output unit, the machine learning model 140 determines whether the next output unit is a speech unit or a breakpoint symbol.

[0081] For example, in the example of Figure 3, based on the first embedding representation 330 of the first audio segment at time 345 and the second embedding representation 335 of the current output unit, the next output unit is determined to be speech unit 345. Next, the machine learning model 140 continues to utilize speech unit 345 and the audio segment acquired at the second time (e.g., the audio segment at the second time in audio signal 320), and also determines the next output unit in the speech response based on the context information 325 at this time. The prediction process for the output unit is performed iteratively until the next output unit is predicted to be an interruption symbol (IRQ). At this point, the speech response for the current dialogue turn is stopped.

[0082] In some embodiments, the machine learning model 140 can be trained. Training of the machine learning model 140 can be performed locally on the electronic device 110 or by an external device. During the training of the machine learning model 140, in response to a sample speech response including an interruption symbol, from the initial time of the sample speech response to the time preceding the occurrence of the interruption symbol, a first loss function of the machine learning model is determined based at least on the sample speech units output at each time, sample audio segments collected from the environment at each time, sample context information, the predicted output unit at the time of the interruption symbol occurrence, and the parameters of the machine learning model.

[0083] Alternatively or additionally, in response to the absence of the interruption symbol in the sample speech response, from the initial time of the sample speech response to the time preceding the end time of the sample speech response, a first loss function of the machine learning model is determined based at least on the sample speech units output at each time, the sample audio segments collected from the environment at each time, the sample context information, the predicted output unit at the end time, and the parameters of the machine learning model. The machine learning model 140 is then trained by reducing or minimizing the value of either the first or second loss function.

[0084] Machine learning model 140 has a speaking channel and a listening channel. At time step t, all previous information of the speaking channel is... and processing information of the listening channel This is considered simultaneously by the model. Therefore, the formula (6) for the loss function can be modified as follows:

[0085] Where θ LS These represent the parameters used by machine learning model 140 to model full-duplex capability. This represents the cumulative time from the initial time of the sample speech response to the time when the interrupt symbol occurs. This represents the sum from the initial time of the sample speech response to the end time of the sample speech response. In addition to the terminator representing the end of the sequence (represented by EOS), terminators representing interruption symbols (represented by IRP) can be added to the segmenter dictionary to allow the machine learning model 140 to stop output as early as possible when a turn-taking occurs. For example, if a human interrupts, the machine learning model 140 can stop outputting the speech response within a predetermined time interval. During inference, the machine learning model 140 learns from the already generated terminators... Context C and real-time captured audio lexical units Sampling in conditional distribution Formula (8) can be rewritten as follows:

[0086] One of the basic requirements for the speech encoder 305 is that the speech encoder 305 is streaming, so that the machine learning model 140 can acquire real-time audio features during the inference phase.

[0087] In some embodiments, the audio signal acquired from the environment includes human voice. In response to determining that the human voice meets predetermined conditions, the machine learning model, based on the context information, the human voice, and a second speech segment acquired from the environment at a second time, determines the voice response output to the user in the next turn of the voice interaction. In one example, the human voice initiates the first turn of the voice interaction. During the first turn, the machine learning model 140 stops the voice response, and the user 130 begins speaking. If the human voice ends for a predetermined duration or a predetermined button is clicked, the voice interaction enters the second turn. During the second turn, the machine learning model 140, based on the context information 325, the human voice, and a second speech segment acquired from the environment at a second time, determines the voice response output to the user in the next turn of the voice interaction.

[0088] Figure 6 shows a flowchart of a voice interaction method 600 according to some embodiments of the present disclosure. Method 600 can be implemented at the electronic device 110 of Figure 1. Method 600 will be described with reference to the environment 100 of Figure 1.

[0089] In frame 610, electronic device 110, during voice interaction, acquires audio signals collected from the environment while outputting voice responses to the user; the voice responses are generated based on a machine learning model.

[0090] In box 620, electronic device 110 uses a machine learning model to determine, based on contextual information related to the voice interaction and a first audio segment captured from the environment at the first moment, whether the next output unit in the voice response to the user is a speech unit or an interrupt symbol, with the interrupt symbol indicating a turn-taking transition in the voice interaction.

[0091] In box 630, electronic device 110 stops outputting voice responses to the user in response to determining that the next output unit is an interrupt symbol.

[0092] In some embodiments, determining the next output unit based on context information and a first audio segment includes: performing a fusion operation on a first embedded representation of the first audio segment and a second embedded representation of the current output unit in the speech response to obtain a fused embedded representation; and determining the next output unit based on context information and the fused embedded representation.

[0093] In some embodiments, the machine learning model includes a language model-based speech decoder, the speech decoder including at least one feature processing layer and an output layer, the output layer being after the at least one output layer, wherein performing a fusion operation on a first audio segment and a current output unit includes at least one of the following: performing a fusion operation before the at least one feature processing layer to process the fused embedding representation by the at least one feature processing layer, performing a fusion operation in each of the at least one feature processing layer, or performing a fusion operation on the first embedding representation and the processed second embedding representation after the embedding representation of the current output unit has been processed by the at least one feature processing layer.

[0094] In some embodiments, determining whether the next output unit in a voice response to user output is a voice unit or an interrupt symbol includes: detecting whether the first audio segment includes a predetermined instruction based on context information and the first audio segment; and if the first audio segment is detected to include a predetermined instruction, determining that the next output unit is an interrupt symbol.

[0095] In some embodiments, determining whether the next output unit in a voice response to user output is a voice unit or an interrupt symbol includes: detecting whether the first audio segment contains human voice based on context information and the first audio segment; and if the first audio segment contains human voice, determining that the next output unit is an interrupt symbol.

[0096] In some embodiments, the interval between the time of the interruption symbol and the time of occurrence of the predetermined instruction and / or human voice is determined to be within a predetermined time interval, and the training samples for training the machine learning model are constructed based on the predetermined time interval.

[0097] In some embodiments, contextual information includes at least the voice signals acquired during the current turn of the voice interaction.

[0098] In some embodiments, method 600 further includes: in response to determining that the next output unit is a voice unit, outputting a voice response to the user based on the determined voice unit and the previous voice unit in the voice response.

[0099] In some embodiments, the machine learning model is trained by: in response to the inclusion of interrupt symbols in the sample speech response, determining a first loss function of the machine learning model from the initial time of the sample speech response to the time preceding the occurrence of the interrupt symbols, based at least on the sample speech units output at each time, sample audio segments collected from the environment at each time, sample context information, the predicted output unit at the time of the interrupt symbols, and the parameters of the machine learning model; or in response to the exclusion of interrupt symbols in the sample speech response, determining a first loss function of the machine learning model from the initial time of the sample speech response to the time preceding the end time of the sample speech response, based at least on the sample speech units output at each time, sample audio segments collected from the environment at each time, sample context information, the predicted output unit at the end time, and the parameters of the machine learning model; and reducing or minimizing the value of the first loss function or the second loss function.

[0100] In some embodiments, the audio signal acquired from the environment includes human voice, and the method 600 further includes: in response to determining that the human voice meets predetermined conditions, using a machine learning model, based on contextual information, the human voice, and a second speech segment acquired from the environment at a second time, determining a voice response to the user's output in the next round of the voice interaction.

[0101] Embodiments of this disclosure also provide corresponding apparatus for implementing the methods or processes described above. FIG7 shows an exemplary structural block diagram of a voice interaction apparatus 700 according to some embodiments of this disclosure. The apparatus 700 may be implemented as or included in an electronic device 110. The various modules / components in the apparatus 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0102] As shown in Figure 7, the device 700 includes an audio signal acquisition module 710, configured to acquire audio signals collected from the environment while outputting a voice response to the user during voice interaction. The voice response is generated based on a machine learning model. The device 700 also includes an output unit determination module 720, configured to use a machine learning model to determine, based on contextual information related to the voice interaction and a first audio segment collected from the environment at a first moment, whether the next output unit in the voice response output to the user is a voice unit or an interrupt symbol. The interrupt symbol indicates a turn-taking transition during the voice interaction. The device 700 also includes a stop output module 730, configured to stop outputting the voice response to the user in response to determining that the next output unit is an interrupt symbol.

[0103] In some embodiments, the output unit determination module 720 is further configured to: perform a fusion operation on the first embedded representation of the first audio segment and the second embedded representation of the current output unit in the speech response to obtain a fused embedded representation; and determine the next output unit based on context information and the fused embedded representation.

[0104] In some embodiments, the machine learning model includes a language model-based speech decoder, the speech decoder including at least one feature processing layer and an output layer, the output layer being after the at least one output layer, wherein the output unit determination module 720 is further configured to: perform a fusion operation before the at least one feature processing layer to process the fused embedding representation by the at least one feature processing layer, perform a fusion operation in each of the at least one feature processing layer, or perform a fusion operation on a first embedding representation and a processed second embedding representation after the embedding representation of the current output unit has been processed by the at least one feature processing layer.

[0105] In some embodiments, the output unit determination module 720 is further configured to: detect whether the first audio segment includes a predetermined instruction based on context information and the first audio segment; and if the first audio segment is detected to include a predetermined instruction, determine that the next output unit is an interrupt symbol.

[0106] In some embodiments, the output unit determination module 720 is further configured to: detect whether the first audio segment contains human voice based on context information and the first audio segment; and if the first audio segment contains human voice, determine that the next output unit is an interrupt symbol.

[0107] In some embodiments, the interval between the time of the interruption symbol and the time of occurrence of the predetermined instruction and / or human voice is determined to be within a predetermined time interval, and the training samples for training the machine learning model are constructed based on the predetermined time interval.

[0108] In some embodiments, contextual information includes at least the voice signals acquired during the current turn of the voice interaction.

[0109] In some embodiments, the apparatus 700 further includes a voice response output module configured to output a voice response to a user based on the determined voice unit and the previous voice unit in the voice response, in response to determining that the next output unit is a voice unit.

[0110] In some embodiments, the machine learning model is trained by: in response to a sample speech response including an interruption symbol, determining a first loss function of the machine learning model from the initial time of the sample speech response to the time preceding the time the interruption symbol occurs, based at least on sample speech units output at each time, sample audio segments collected from the environment at each time, sample context information, predicted output units at the time the interruption symbol occurs, and the parameters of the machine learning model. In some embodiments, the machine learning model is trained by: in response to a sample speech response not including an interruption symbol, determining a first loss function of the machine learning model from the initial time of the sample speech response to the time preceding the time preceding the time the sample speech response ends, based at least on sample speech units output at each time, sample audio segments collected from the environment at each time, sample context information, predicted output units at the time the end occurs, and the parameters of the machine learning model; and reducing or minimizing the value of the first loss function or a second loss function.

[0111] In some embodiments, the audio signal acquired from the environment includes human voice, and the device 700 further includes: a voice response determination module configured to, in response to determining that the human voice meets predetermined conditions, use a machine learning model to determine a voice response to the user's output in the next dialogue round of the voice interaction based on contextual information, the human voice, and a second voice segment acquired from the environment at a second time.

[0112] The units and / or modules included in device 700 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 700 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0113] It should be understood that one or more steps in the above methods can be performed by a suitable electronic device or combination of electronic devices. Such an electronic device or combination of electronic devices may, for example, include the electronic device 110 in FIG1.

[0114] Figure 8 shows a block diagram of an electronic device 800 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 800 shown in Figure 8 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 800 shown in Figure 8 can be used to implement the electronic device 110 of Figure 1 or the device 700 of Figure 7.

[0115] As shown in Figure 8, the electronic device 800 is in the form of a general-purpose electronic device. Components of the electronic device 800 may include, but are not limited to, one or more processors or processing units 810, memory 820, storage devices 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. The processor 810 may be a physical or virtual processor and is capable of performing various processes according to the programs stored in the memory 820. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 800.

[0116] Electronic device 800 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 820 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 830 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 800.

[0117] Electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG8, disk drives for reading or writing from removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading or writing from removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 820 may include computer program product 825 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0118] The communication unit 840 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 800 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 800 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0119] Input device 850 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 860 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 800 can also communicate with one or more external devices (not shown) via communication unit 840 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 800, or with any device that enables electronic device 800 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0120] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0121] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0122] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0123] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some newer implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0125] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method of voice interaction, comprising: acquiring, in a voice interaction, an audio signal captured from an environment while outputting a voice response to a user, the voice response being generated based on a machine learning model; determining, with the machine learning model, that a next output unit in the voice response output to the user is a speech unit or a turn-taking symbol based on context information related to the voice interaction and a first audio segment captured from the environment at a first time, the turn-taking symbol indicating that a turn transition occurs in the voice interaction; and in response to determining that the next output unit is the turn-taking symbol, stopping outputting the voice response to the user.

2. The method of claim 1, wherein determining the next output unit based on the context information and the first audio segment comprises: performing a fusion operation on a first embedding representation of the first audio segment and a second embedding representation of a current output unit in the voice response to obtain a fused embedding representation; and determining the next output unit based on the context information and the fused embedding representation.

3. The method of claim 2, wherein the machine learning model comprises a speech decoder based on a language model, the speech decoder comprising at least one feature processing layer and an output layer, the output layer being subsequent to the at least one output layer, wherein performing the fusion operation on the first audio segment and the current output unit comprises at least one of: performing the fusion operation prior to the at least one feature processing layer to process the fused embedding representation by the at least one feature processing layer, performing the fusion operation in each of the at least one feature processing layer, or performing the fusion operation on the first embedding representation and a processed second embedding representation after processing the embedding representation of the current output unit by the at least one feature processing layer.

4. The method of claim 1, wherein determining that the next output unit in the voice response output to the user is a speech unit or a turn-taking symbol comprises: detecting, based on the context information and the first audio segment, whether the first audio segment includes a predetermined instruction; and if detecting that the first audio segment includes the predetermined instruction, determining that the next output unit is the turn-taking symbol.

5. The method of claim 1, wherein determining that the next output unit in the voice response output to the user is a speech unit or a turn-taking symbol comprises: detecting, based on the context information and the first audio segment, whether the first audio segment includes a human voice; and if detecting that the first audio segment includes the human voice, determining that the next output unit is the turn-taking symbol.

6. The method of claim 4 or 5, wherein a time of determining the turn-taking symbol is within a predetermined time interval from a time of occurrence of the predetermined instruction and / or the human voice, and wherein training samples used to train the machine learning model are constructed based on the predetermined time interval. ​ ​ ​ ​ 7. The method of claim 1, wherein the context information comprises at least a speech signal captured in a current turn of conversation of the voice interaction.

8. The method of claim 1, further comprising: responsive to determining that the next output unit is a speech unit, outputting a speech response to the user based on the determined speech unit and a previous speech unit in the speech response.

9. The method of claim 1, wherein the machine learning model is trained by: responsive to the interrupt symbol being included in a sample speech response, determining a first loss function of the machine learning model based on at least a sample speech unit output at each time, a sample audio segment captured from an environment at each time, sample context information, a predicted output unit at a time at which the interrupt symbol occurs, and parameters of the machine learning model, from an initial time of the sample speech response to a time preceding a time at which the interrupt symbol occurs; or responsive to the interrupt symbol not being included in a sample speech response, determining a second loss function of the machine learning model based on at least the sample speech unit output at each time, the sample audio segment captured from the environment at each time, the sample context information, a predicted output unit at an end time of the sample speech response, and the parameters of the machine learning model, from the initial time of the sample speech response to the end time; and causing a value of the first loss function or the second loss function to decrease or be minimized.

10. The method of claim 1, the audio signal captured from the environment including a human voice, the method further comprising: responsive to determining that the human voice meets a predetermined condition, determining, with the machine learning model, a speech response output to the user in a next turn of conversation of the voice interaction based on the context information, the human voice, and a second speech segment captured from the environment at a second time.

11. An apparatus of a voice interaction, comprising: an audio signal obtaining module configured to obtain, in a voice interaction, an audio signal captured from an environment while outputting a speech response to a user, the speech response generated based on a machine learning model; an output unit determining module configured to determine, with the machine learning model, that a next output unit in the speech response output to the user is a speech unit or an interrupt symbol indicating a turn change occurring in the voice interaction based on context information related to the voice interaction and a first audio segment captured from the environment at a first time; and a stopping output module configured to stop outputting the speech response to the user responsive to determining that the next output unit is the interrupt symbol.

12. An electronic device, comprising: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions when executed by the at least one processor cause the device to perform the method of any of claims 1-10. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 13. A computer-readable storage medium having stored thereon computer- executable instructions that, when executed by a processor, implement the method of any of claims 1-10.

14. A computer program product comprising computer-executable instructions that, when executed by a processor, implement the method of any of claims 1-10.

Images