Streaming speech-to-speech model with automatic speaker rotation detection

Abstract

A method (400) for rotation detection in a speech-to-speech (S2S) model (200) includes receiving a sequence of acoustic frames (102) corresponding to an utterance (108). The method further comprises, at each of a plurality of output steps, generating, by the audio encoder (210), a higher order feature representation (213) for a corresponding acoustic frame of the sequence of acoustic frames, and determining, by an alternating detector (215) of the S2S model, whether the utterance is at a breakpoint at the corresponding output step based on a higher order feature representation generated by the audio encoder at the corresponding output step. When the rotation detector determines that the utterance is at the breakpoint, the method includes synthesizing a sequence of output audio frames (222) output by a speech decoder (220) of the S2S model into a time domain audio waveform representing a synthesized speech of the utterance spoken by the user.

Description

Technical Field

[0001] The present disclosure relates to a speech-to-speech model including a turn detector. Background Art

[0002] Speech-to-speech (S2S) models can be used to convert the speech of a source speaker into synthesized speech without changing the linguistic information of the original speech. For example, an S2S model can produce canonical, fluent synthesized speech for a user with dysarthria or atypical speech. Alternatively, an S2S model can translate the user's speech into synthesized speech in another language. Typically, an S2S model is manually activated by user input indicating when the input speech begins and ends. After the user finishes speaking, the S2S model then processes the input speech to produce synthesized speech. Summary of the invention

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations, the operations comprising: receiving a sequence of acoustic frames corresponding to an utterance spoken by a user in streaming audio captured by a client device associated with a user as input to a speech-to-speech (S2S) model. At each of a plurality of output steps, the operations further comprise: generating a higher-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames by an audio encoder of the S2S model; determining by a turn detector of the S2S model whether the utterance is at a breakpoint at the corresponding output step based on the higher-order feature representation generated by the audio encoder at the corresponding output step; and when the turn detector determines that the utterance is at a breakpoint, synthesizing a sequence of output audio frames output by a speech decoder of the S2S model into a time-domain audio waveform representing a synthesized speech of the utterance spoken by the user. Here, each output audio frame in the sequence of output audio frames is based on a corresponding one of the higher-order feature representations generated by the audio encoder until the corresponding output step when the turn detector determines that the utterance is at a breakpoint.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operation further includes providing an output from the client device when the utterance is at a breakpoint: a speaker turn indication notifying the user to stop speaking; and a time-domain audio waveform of a synthesized speech representing the utterance spoken by the user. The utterance spoken by the user in the streaming audio captured by the client device may be associated with an atypical voice, and the time-domain audio waveform representing the synthesized speech of the utterance may include a time-domain audio waveform of a synthesized canonical fluent speech of the same utterance spoken by the user. Additionally, the utterance spoken by the user in the streaming audio captured by the client device may be in a first language, and the time-domain audio waveform representing the synthesized speech of the utterance may include a time-domain audio waveform of a synthesized translated speech of the same utterance in a second language different from the first language.

[0005] In some examples, determining whether the utterance is at a breakpoint at a corresponding output step is further based on one or more of the higher order feature representations generated by the audio encoder at an output step preceding the corresponding output step. Additionally or alternatively, the operation may further include, at each of the plurality of output steps, generating, by a speech decoder of the S2S model, an output audio frame of the corresponding higher order feature representation generated by the audio encoder at the corresponding output step.

[0006] In some implementations, in response to determining that the utterance is at a breakpoint at a corresponding output step, the operations further include: receiving a sequence of higher-order feature representations generated by the audio encoder as input to a speech decoder of the S2S model until the corresponding output step; and generating, by the speech decoder of the S2S model, a sequence of output audio frames. The turn detector may include a deep neural network that may be arranged between the audio encoder and the speech decoder.

[0007] In some additional implementations, determining whether the utterance is at a breakpoint at the corresponding output step includes generating a turn output from a turn detector as an output indicating whether the utterance is at a breakpoint. Here, the turn output includes a bit or a probability distribution.

[0008] In some examples, the operation further includes training the S2S model by receiving a set of training utterances, wherein each training utterance in the set of training utterances includes a corresponding sequence of training acoustic frames and is paired with a corresponding real-value synthetic speech representation of the training utterance. Each training acoustic frame in the sequence of training acoustic frames is annotated with a label indicating whether the corresponding training acoustic frame corresponds to a breakpoint frame or a non-breakpoint frame. In these examples, the S2S model is further trained by obtaining a first label indicating a target output spectrogram of the training input audio data, obtaining a second label indicating a target rotation output of the training input audio data, and generating a training output using the speech conversion model and the training input audio data, the training output including: a training output spectrogram corresponding to a synthetic speech representation of the training input audio data; and a training rotation output indicating a breakpoint in the training input audio data. Finally, the S2S model is trained by: comparing the training output spectrogram with the first label to determine a first loss; determining a second loss by comparing the training rotation output with the second label; and optimizing the speech conversion model based on the first loss and the second loss associated with the training input audio data.

[0009] In some implementations, the operations further include determining a speech type of an utterance spoken by a user captured in streaming audio by the client device; and selecting a speech decoder from a plurality of available speech decoders for generating a sequence of output audio frames. In some examples, each output audio frame in the sequence of output audio frames output by the speech decoder comprises a spectrogram frame.

[0010] Another aspect of the present disclosure provides a system, the system including data processing hardware and memory hardware in communication with the data processing hardware. The memory stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including: receiving a sequence of acoustic frames corresponding to an utterance spoken by a user in streaming audio captured by a client device associated with the user as input to a speech-to-speech (S2S) model. At each of a plurality of output steps, the operations also include: generating a higher-order feature representation for a corresponding acoustic frame in the sequence of acoustic frames by an audio encoder of the S2S model; determining by a turn detector of the S2S model whether the utterance is at a breakpoint at the corresponding output step based on the higher-order feature representation generated by the audio encoder at the corresponding output step; and when the turn detector determines that the utterance is at a breakpoint, synthesizing the sequence of output audio frames output by the speech decoder of the S2S model into a time-domain audio waveform representing a synthesized speech of the utterance spoken by the user. Here, each output audio frame in the sequence of output audio frames is based on a corresponding one of the higher order feature representations generated by the audio encoder until a corresponding output step when the turn detector determines that the utterance is at a breakpoint.

[0011] This aspect may include one or more of the following optional features. In some implementations, the operation further includes, when the utterance is at a breakpoint, providing an output from the client device: a speaker turn indication notifying the user to stop speaking; and a time-domain audio waveform of synthesized speech representing the utterance spoken by the user. The utterance spoken by the user in the streaming audio captured by the client device may be associated with an atypical voice, and the time-domain audio waveform representing the synthesized speech of the utterance may include a time-domain audio waveform of synthesized canonical fluent speech of the same utterance spoken by the user. Additionally, the utterance spoken by the user in the streaming audio captured by the client device may be in a first language, and the time-domain audio waveform representing the synthesized speech of the utterance may include a time-domain audio waveform of synthesized translated speech of the same utterance in a second language different from the first language.

[0012] In some examples, determining whether the utterance is at a breakpoint at a corresponding output step is further based on one or more of the higher order feature representations generated by the audio encoder at an output step preceding the corresponding output step. Additionally or alternatively, the operation may further include, at each of the plurality of output steps, generating, by a speech decoder of the S2S model, an output audio frame of the corresponding higher order feature representation generated by the audio encoder at the corresponding output step.

[0013] In some implementations, in response to determining that the utterance is at a breakpoint at a corresponding output step, the operations further include: receiving a sequence of higher-order feature representations generated by the audio encoder as input to a speech decoder of the S2S model until the corresponding output step; and generating, by the speech decoder of the S2S model, a sequence of output audio frames. The turn detector may include a deep neural network that may be arranged between the audio encoder and the speech decoder.

[0014] In some additional implementations, determining whether the utterance is at a breakpoint at the corresponding output step includes generating a turn output from a turn detector as an output indicating whether the utterance is at a breakpoint. Here, the turn output includes a bit or a probability distribution.

[0015] In some examples, the operation further includes training the S2S model by receiving a set of training utterances, wherein each training utterance in the set of training utterances includes a corresponding sequence of training acoustic frames and is paired with a corresponding real-value synthetic speech representation of the training utterance. Each training acoustic frame in the sequence of training acoustic frames is annotated with a label indicating whether the corresponding training acoustic frame corresponds to a breakpoint frame or a non-breakpoint frame. In these examples, the S2S model is further trained by obtaining a first label indicating a target output spectrogram of the training input audio data, obtaining a second label indicating a target rotation output of the training input audio data, and generating a training output using the speech conversion model and the training input audio data, the training output including: a training output spectrogram corresponding to a synthetic speech representation of the training input audio data; and a training rotation output indicating a breakpoint in the training input audio data. Finally, the S2S model is trained by: comparing the training output spectrogram with the first label to determine a first loss; determining a second loss by comparing the training rotation output with the second label; and optimizing the speech conversion model based on the first loss and the second loss associated with the training input audio data.

[0016] In some implementations, the operations further include determining a speech type of an utterance spoken by a user captured in streaming audio by the client device; and selecting a speech decoder from a plurality of available speech decoders for generating a sequence of output audio frames. In some examples, each output audio frame in the sequence of output audio frames output by the speech decoder comprises a spectrogram frame.

[0017] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of an example speech-to-speech system with a turn detector.

[0019] Figure 2 is a diagram of an example speech-to-speech model with a turn detector.

[0020] Figure 3 is a diagram of an example training process for a speech-to-speech model with a turn detector.

[0021] Figure 4 is a flow diagram of an example operational arrangement of a method for performing speech-to-speech conversion under a turn detector.

[0022] Figure 5 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0023] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0024] There is a growing interest in developing more inclusive speech technologies, especially those that can help people with speech disorders. With the introduction of models based on end-to-end (E2E) deep learning, speech-to-speech (S2S) conversion has made great progress in recognizing speech from speakers with dysarthria or atypical speech patterns for conversion into synthesized speech. For example, atypical speech patterns can include, but are not limited to, impaired speech, heavily accented speech, and deaf speech due to physical or neurological conditions (e.g., speakers with amyotrophic lateral sclerosis (ALS) disease). Another use of the S2S model is translation, in which a user speaks in a first language and the S2S model generates synthesized speech in a second language.

[0025] Conventional S2S models require that the entire input speech (i.e., utterance) be received before generating synthesized speech. Typically, the user provides input indicating the start and end of an utterance for the S2S model. For example, a user may hold down a button on a user interface corresponding to the S2S model when they begin speaking (i.e., begin providing input speech to the S2S model). The user then releases the button when they finish speaking. The S2S model then converts the input speech received during the time the user holds down the button into synthesized speech. Such manual methods of activating the S2S model may be inconvenient for the user due to the requirement for a large amount of user input. Further, requiring user input allows for human error (e.g., the user may release the button before they finish speaking).

[0026] The present disclosure introduces a turn detector that automatically determines appropriate breakpoints in a user's speech at which to output synthesized speech converted from the user's speech by an S2S model in a streaming manner. Thus, the current disclosure does not require user input to indicate when an utterance begins / ends, but rather can receive streaming speech from a user and automatically determine appropriate moments in the speech (so-called "breakpoints") for the user to pause and receive / listen to the synthesized speech produced by the S2S model. By automatically determining breakpoints, implementations of the present disclosure provide a more natural user experience because the turn-by-turn nature of the system mimics a normal conversation (i.e., the user speaks a turn, and then the system speaks a turn, and so on).

[0027] As used herein, and unless otherwise specified, the terms "speech-to-speech system" and "speech-to-speech model" may refer to any system / model that converts input speech directly into synthesized speech without performing intermediate speech recognition on the input speech. In other words, the speech-to-speech system / model is configured to convert an input audio waveform, sequence of acoustic frames, or spectrogram corresponding to the input speech directly into an output audio waveform or spectrogram corresponding to the synthesized speech, without converting the input audio waveform into an intermediate representation (e.g., text or phonemes). It is apparent that speech-to-speech models and techniques for training speech-to-speech models will enable users with atypical speech to speak to and be understood by other humans and speech interfaces (e.g., digital assistants) by enabling recognition and reproduction of the user's intended speech.

[0028] Although the examples herein depict a speech-to-speech model that receives input utterances corresponding to atypical speech for conversion into synthesized speech corresponding to canonical fluent speech, the speech-to-speech model may be similarly adapted to perform other types of speech conversion tasks without departing from the scope of this disclosure. For example, an S2S model may convert an input utterance in a first language into synthesized speech corresponding to a translation of the input utterance in a different second language. The S2S model may similarly receive spoken input by a user and output synthesized speech containing the same linguistic content of the spoken input but with different voice characteristics of a target speaker.

[0029] Figure 1 A speech-to-speech system 100 is shown that includes a speech-to-speech (S2S) model 200 that is configured to directly convert input audio data 102 (e.g., a sequence of acoustic frames or an input audio waveform) corresponding to an utterance 108 spoken by a target speaker 104 into output audio data 106 (e.g., a sequence of output audio frames or an output audio waveform) corresponding to a synthesized speech representation of the same utterance 114 spoken by the target speaker 104. Notably, the S2S conversion model 200 is configured to directly convert the input audio data 102 into the output audio data 106 without performing speech recognition or otherwise requiring generation of any intermediate discrete representation (e.g., text or phonemes) from the input audio data 102.

[0030] The S2S conversion model 200 includes: an audio encoder 212, which is configured to encode the input audio data 102 into a hidden feature representation (e.g., a series of vectors); a turn detector 215, which is configured to determine a breakpoint in the speech 108 based on the hidden feature representation output by the encoder; and a speech decoder 220, which is configured to decode the hidden representation into output audio data 106 corresponding to the synthetic canonical fluent speech representation. For example, when the audio encoder 200 receives the input audio data 102 of the speech 108, the audio encoder 200 can process five frames in the audio and convert the five frames in the audio into ten vectors. These vectors are not transcriptions of the frames of the audio data 102, but mathematical representations of the frames in the audio data 102. Then, the turn detector 215 can determine whether there is a breakpoint within the ten vectors. If there is a breakpoint, the speech decoder 220 can generate output audio data 106 corresponding to the synthetic canonical fluent speech representation for the vector before the breakpoint based on the vector received from the audio encoder 200. For example, the turn detector 215 may determine that the tenth vector is a breakpoint, so the speech decoder 220 may receive ten vectors representing five frames in the audio from the audio encoder 200. Here, the speech decoder 220 may generate five frames in the output audio data 106 corresponding to a synthetic canonical fluent speech representation of the utterance 114, the utterance including the expected words or portions of words as the five frames in the input audio data 102, but without the disfluencies of the atypical speech.

[0031] The S2S conversion system 100 may further include a synthesizer 275 for synthesizing the output audio data 106 into a time domain waveform as an audible output of the synthesized speech 114 of the speech 108. The time domain audio waveform includes an audio waveform that defines the amplitude of the audio signal over time. The synthesizer 275 may include a unit selection module or a WaveNet module for synthesizing the output audio data 106 into a time domain waveform for synthesizing standard fluent speech. In some implementations, the synthesizer 275 includes a vocoder network, that is, a neural vocoder that is trained separately and adjusted on a Mel-spectrogram for conversion to a time domain audio waveform. In an additional implementation, the synthesizer 275 includes a streaming vocoder that is configured to convert / invert the logarithmic magnitude spectrogram output from the speech decoder 220 as the output audio data 106 into a time domain audio waveform in real time.

[0032] In some implementations, the target speaker 104 is associated with an atypical speech, such that the target speaker 104 speaks in an atypical speech pattern that may be difficult to understand. Atypical speech patterns may include, but are not limited to, impaired speech due to a physical or neurological condition (e.g., a speaker with amyotrophic lateral sclerosis (ALS) disease), heavily accented speech, and deaf speech. In other implementations, the target speaker 104 speaks in a first language, and S2S translates the speech into a second language.

[0033] Thus, the speech-to-speech conversion system 100 is trained to directly convert input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 associated with an atypical speech into output audio data 106 corresponding to a synthesized canonical fluent speech representation of the same utterance 108. Thus, the synthesized canonical fluent speech representation provided by the output audio data 106 improves the intelligibility of the atypical speech (e.g., heavily accented speech or ALS speech) spoken by the target speaker 104.

[0034] Without departing from the scope of the present disclosure, the speech-to-speech conversion system 100 may be trained to convert input audio data 102 corresponding to an utterance 108 associated with an atypical voice in a first language directly into output audio data 106 corresponding to a synthetic canonical fluent speech representation of the same utterance 108 in the same voice but in a different second language.

[0035] Further, the turn detector 215 can be trained to determine the breakpoints in the input audio data 102 in real time when the utterance 108 spoken by the target speaker 104 is captured in the streaming audio by the user device 110 associated with the target speaker 104. In some implementations, the turn detector 215 includes a deep neural network. The turn detector 215 can be arranged between the audio encoder 212 and the speech decoder 220, so that the turn detector 215 receives the output 213 from the audio encoder 212 (e.g., a higher-order feature representation of each acoustic frame in the sequence of acoustic frames) and transmits the turn output 216 to the decoder 220. The turn output indicates whether the utterance is at a breakpoint. In some implementations, the turn output is a series of bits (e.g., 1 and 0), where 1 indicates that the speech is at a breakpoint and 0 indicates a non-breakpoint. Here, each bit in the series of bits can represent a corresponding acoustic frame of the input audio data 102. In other implementations, the turn output is a probability score (i.e., a number between 0 and 1) indicating the likelihood of a breakpoint at the corresponding acoustic frame. For example, when the probability score of the corresponding acoustic frame satisfies a breakpoint threshold, then the acoustic frame indicates a breakpoint. When the turn output 216 indicates that the speech is at a breakpoint, the S2S model 200 can provide an instruction 117 to the user device 110 that causes the user device 110 to output a turn indication 115 indicating that the user should stop speaking.

[0036] The speech decoder 220 may then receive the output 213 from the encoder 212 and the turn output 216 from the turn detector 215 during each of the plurality of time steps. In some implementations, the turn detector 215 modifies the output from the encoder such that the output from the encoder indicates a breakpoint.

[0037] A user device (interchangeably referred to as a 'computing device') 110 associated with a target speaker 104 may capture an utterance 108 spoken by the target speaker 104 in streaming audio and transmit corresponding input audio data 102 to a speech-to-speech conversion system 100 for conversion into output audio data 106. Thereafter, the speech-to-speech conversion system 100 may transmit output audio data 106 corresponding to a synthesized speech representation of the same utterance 114 spoken by the target speaker 104 to another computing device 116 associated with a user 118, whereby the other computing device 116 audibly outputs a synthesized speech representation of the utterance 108 spoken by the target speaker 104. In this example, the target speaker 104 and the user 118 speak to each other through their respective computing devices 110, 116, such as through a telephone call or other type of voice communication protocol (e.g., voice over internet protocol). Although the target speaker 104 and other users 118 can speak the same language, the other users 118 may have difficulty understanding the target speaker 104 because the target speaker 104 has an atypical voice due to the ALS disease. Therefore, although the target speaker 104 speaks in an atypical voice (e.g., ALS voice) that may be difficult to understand, other users 118 who hear the synthesized canonical fluent speech representation will more easily understand the utterance 108 intended by the target speaker 104. In other words, the synthesized canonical fluent speech representation provides a more consistent rhythm that another user can more easily understand than the original utterance 108 spoken by the target speaker in an atypical voice. It is noteworthy that the synthesized canonical fluent speech representation is in the voice of the target speaker 104.

[0038] In some other examples, the S2S conversion system 100 instead delivers output audio data 106 corresponding to a synthesized speech representation of an utterance spoken by the target speaker 104 to an output audio device so that the synthesized speech representation is audibly output to a listener in the voice of the target speaker 104. For example, the target speaker 104 may be a psychology professor providing a lecture to a class of students, wherein the utterance spoken by the target speaker 104 includes medical terms belonging to a particular field (e.g., psychology). It is apparent that the S2S conversion system 100 is trained to learn the linguistic diversity from the linguistic content present in the training utterances and the acoustic diversity associated with the particular type of atypical speech associated with the speaker speaking the target utterance.

[0039] Alternatively, another computing device 116 may be associated with a downstream automated speech recognition (ASR) system, with the speech-to-speech conversion system 100 acting as a front end to provide output audio data 106 corresponding to the synthesized speech representation as input to the ASR system for conversion into recognized text. The recognized text may be presented to other users 118 and / or may be provided to a natural language understanding (NLU) system for further processing.

[0040] In any of the above examples, when the target speaker 104 is speaking, if the turn detector 215 determines that the speech has reached a breakpoint, the S2S model 200 can provide instructions 117 to cause the user device 110 associated with the target speaker 104 to output a turn indication 115. For example, Figure 1 10 shows that the user device 110 displays a graphical turn indication 115a as a series of exclamation marks that inform the target speaker 104 to pause and allow the S2S model 200 to generate the output audio data 106 corresponding to the synthetic canonical fluent speech representation of the input audio data 102. In an additional example, the user device 110 outputs an audible turn indication 115b (e.g., emitting a tone or a series of tones) that informs the target speaker 104 to pause. The user device 110 can output other types of turn indications 115, such as by vibration and / or flashing lights, to inform the target speaker 104 to pause speaking. In some implementations, the S2S model 200 provides instructions 117 to cause another device (not shown) associated with the target speaker 104 to output the turn indication 115t. In these implementations, a device other than the user device 110 that captures the utterance 108 can receive the instructions 117 and output the turn indication 115. For example, a smart watch worn by the user can output a turn indication 115 by vibration / beeping to notify the target speaker 104 to pause. Once the instruction 117 is provided so that the user device 110 (or another device) outputs the turn indication 115, the synthesizer 275 can provide a time domain audio waveform of the synthesized speech 114 for output from the device 110, 116 or any other device.

[0041] The functionality of the speech-to-speech conversion system 100 may reside on a remote server 112, on either or both of the computing devices 110, 116, or on any combination of the remote server and the computing devices 110, 116. The computing devices 110 and 116 may include, but are not limited to, smartphones, tablets, desktop / laptop computers, smart speakers, smart displays, smart appliances, assistant-enabled wearable devices (e.g., smart watches, smart headphones, smart glasses, etc.), or vehicle infotainment systems.

[0042] Figure 2The system includes an encoder 212, a rotation detector 215 and a decoder 220. Figure 1 The S2S model 200 processes speech with the target speaker 104 ( Figure 1 ) to generate a sequence of output audio frames 222 corresponding to the synthesized speech. The encoder 212 may receive the input audio data 102 of the speech 108 and generate a higher-order feature representation 213 (also referred to herein as a hidden feature representation) for each frame 102 in the sequence of audio frames 102 of the audio data 102. Each higher-order feature representation 213 is generated by the encoder 212 at an output step corresponding to the frame 102 in the sequence of audio frames. The rotation detector 215 may receive the higher-order feature representation 213 generated by the encoder 212 at each corresponding output step to generate a rotation output 216 as a single bit. That is, the rotation detector 215 outputs a rotation output 216 for each corresponding output step, the rotation output indicating whether the corresponding audio frame 102 is at a breakpoint. Here, the rotation detector 215 may correspond to a deep neural model including attention with a logic function to predict a single bit (0 or 1) based on the encoder output 213 at each output step. In some implementations, the turn output 216 includes a probability distribution indicating the likelihood that the encoder output 212 at each output step is at a breakpoint. The turn detector 215 can analyze each higher-order feature representation 213 separately to determine the turn output 216 corresponding to the higher-order feature representation 213. In some implementations, the encoder 212 processes the audio data 102 under the attention mechanism to obtain one or more higher-order feature representations 213 (which can be transmitted as a single vector), and the turn detector 215 can use the higher-order feature representations 213 to generate the turn output 216. Therefore, the turn detector 215 can determine whether the utterance 108 is at a breakpoint based on the current higher-order feature representation 213 (i.e., the current output step) and the one or more higher-order feature representations 213 generated by the encoder at the previous output step (i.e., the turn detector 215 receives a history of encoder states in order to predict whether the utterance 108 is currently at a breakpoint).

[0043] The rotation detector 215 may be arranged between the encoder 212 and the decoder 220. Alternatively, the rotation detector 215 may be part of the encoder 212. In either case, the rotation detector 215 may provide the decoder 220 with a rotation output 216 and the higher-order feature representation 213 output by the encoder 212 at each corresponding output step, so that the decoder 220 may generate a sequence of output audio frames 222. The decoder 220 may process the sequence of higher-order feature representations 213 received at each output step before and including the output step corresponding to the breakpoint to generate a sequence of output audio frames 222, wherein each output audio frame in the sequence of output audio frames 222 is based on a corresponding one of the higher-order feature representations 213 generated by the audio encoder 212.

[0044] In some implementations, the decoder 220 does not begin generating a sequence of output audio frames 222 until a carousel output 216 corresponding to a breakpoint is received. In other implementations, the decoder 220 operates in a streaming mode by generating a corresponding output audio frame 222 for each higher-order feature representation 213 generated by the encoder 212. Here, the synthesizer 275 may not begin synthesizing a sequence of output audio frames 212 until the turn detector 215 determines that the utterance 108 is at a breakpoint (i.e., the synthesizer begins synthesizing output audio data 222 at an output step corresponding to the carousel output 216 indicating the breakpoint).

[0045] In some implementations, the S2S model 200 includes a plurality of decoders 220, and selects an appropriate decoder 220 based on the speech type of the utterance 108. In these implementations, the S2S model first determines the speech type, and then selects an appropriate decoder 220. The S2S model 200 can determine the speech type in a variety of ways. In one implementation, the S2S model 200 receives the utterance 108 and determines the speech type based on the characteristics of the utterance 108 (e.g., the speech may have an accent, delay, stutter, articulation disorder, etc.). In other implementations, the S2S model 200 receives the speech type from a user profile associated with the target speaker 104 (e.g., the user profile indicates that the user has articulation disorder speech). In yet other implementations, the S2S model 200 may receive an indication of the speech type via user input (e.g., a user input for translating speech into a specific language).

[0046] Based on the speech type, the S2S model 200 can then select an appropriate decoder 220 to use when generating a sequence of output audio frames 222. For example, if the S2S model 200 receives speech corresponding to a speech type of an accented speech, the S2S model 200 can select a streaming decoder 220. Here, the streaming decoder 220 can be suitable for synthesizing the accented speech into a non-accented speech because each word in the accented speech is directly synthesized into a non-accented speech. However, if the speech type is a translation, the streaming decoder 220 may not be appropriate because there may be syntactic differences between languages, and a correct translation may require an entire sentence or phrase. For example, "Benimle gitti" in Turkish literally translates to "me with he went" in English, although the correct translation of this phrase in English should be "he went with me". Therefore, for speech types associated with translation, the S2S model 200 can select a standard decoder 220 that processes the entire sentence. The standard decoder 220 may synthesize speech at each received breakpoint.

[0047] Figure 3 A training process 300 of a speech-to-speech model 200 including a turn detector 215 is shown. In some implementations, the S2S model 200 is initialized / pre-trained before being further fine-tuned by training. For example, pre-training may include initiating the S2S model 200 with pre-training data 301 including a plurality of spoken utterances of a typical speaker associated with canonical fluent speech. The pre-training data 301 may further include spoken utterances paired with corresponding true value synthetic canonical fluent speech representations of the spoken utterances. In some implementations, the pre-training data 301 includes one or more breakpoints corresponding to each utterance in the plurality of utterances. The pre-trained S2S model may then be trained on data 302 from a user with an atypical voice to further fine-tune the S2S model 200 for a user with an atypical voice. In another example, the S2S model 200 is pre-configured with general parameters and then fine-tuned throughout the training process 300.

[0048] The training process 300 may include training any one of the audio encoder 212, the turn detector 215, or the speech decoder 220 separately or jointly in any suitable combination. The process 300 includes feeding the training data 302 to the S2S model 200. In some implementations, the training data 302 includes training input audio data 320 associated with a plurality of training atypical speech samples spoken by one or more speakers associated with atypical speech. That is, for each training atypical speech sample, the training input audio data 320 may include a corresponding training sequence of acoustic frames. In these implementations, the training data 302 also includes training output audio data 321 associated with a plurality of true value canonical speech samples, each true value canonical speech sample being paired with a corresponding one of the training atypical speech samples. Here, each true value canonical speech sample of the training output audio data 321 may include a sequence of acoustic frames corresponding to a canonical fluent speech representation of the same speech as the training input audio data 320. Further, the training data 302 may also include a sequence of breakpoint labels 322 for each of a plurality of training atypical speech samples of the input audio data 320, the sequence of breakpoint labels including a series of bits labeled as 1 and 0 to indicate whether a corresponding acoustic frame of the input audio data 320 is at a breakpoint. For example, the training data 302 may include training input audio data 320 corresponding to training atypical speech samples of an utterance spoken by a user with an atypical voice, training output audio data 321 corresponding to a canonical fluent speech representation of the same utterance as the training input audio data 320, and labels 321 that may indicate which acoustic frame in the training input audio data 320 includes a breakpoint of the utterance (e.g., each period in the transcription of the utterance). It is noteworthy that the labels 321 may additionally correspond to a true value transcription of the training input audio data 320 indicating where the breakpoint occurs.

[0049] Further, the training data 302 may be based on the user's interaction with the output of the trained S2S model 200. In this way, the S2S model 200 may be fine-tuned cyclically based on real-world feedback. For example, the S2S model 200 may receive streaming speech from a user. When the S2S model 200 determines that the speech has reached a breakpoint, the S2S model 200 provides an instruction 117 so that the user device outputs a rotation indication 115 notifying the user to stop speaking. However, if the user overrides the rotation indication 115 by continuing to speak or otherwise, the determined breakpoint may be considered incorrect. Here, the user's speech may be used as training data, where the point at which the user stops speaking is used as a label 322 corresponding to the target breakpoint.

[0050] Upon receiving the training data 302, the S2S model 200 may generate an output 350 (e.g., a sequence of output audio frames 222, a time-domain audio waveform of synthesized speech representing an utterance spoken by a user, and / or a turn-taking output 216). The S2S model 200 may generate an output 350 (e.g., a sequence of output audio frames 222, a time-domain audio waveform of synthesized speech representing an utterance spoken by a user, and / or a turn-taking output 216). Figure 2 The training data 302 may be processed in the manner described above or in any other suitable manner for speech-to-speech conversion.

[0051] In some implementations, one or more loss functions 305, 305a-b use the output 350 to generate one or more losses 310, 310a-c. That is, the loss function 305a compares the output 350 to the corresponding true value canonical speech sample of the training output audio data 321 to generate a loss 310a, where the loss 310a indicates the difference between the predicted output 350 and the corresponding true value canonical speech sample of the training output audio data 321. For example, the loss function 305a can compare the output 350 of the S2S model 200 (i.e., the sequence of predicted output audio frames 222) to the training output audio data 321 including a sequence of true value output audio frames representing a canonical fluent speech representation of the training input audio data 120 to determine the loss 310a.

[0052] Further, the loss function 305b compares the output 350 with the breakpoint label 322 to generate a loss 310b, wherein the loss 310b indicates the difference between the breakpoint label 322 and the position of the predicted breakpoint in the output 350. For example, the loss function 305b may compare the output of the S2S model 200 (i.e., the rotation output 216) with the breakpoint label 322 corresponding to the target breakpoint of the training data 302 to determine the loss 310b. In some implementations, the rotation output 216 may be a bit (i.e., 0 or 1) generated by the rotation detector 215 of the S2S model 200 based on the higher-order feature representation corresponding to each frame of the sequence of acoustic frames of the utterance, wherein '0' corresponds to a non-breakpoint and '1' corresponds to a breakpoint. Alternatively, the rotation output 216 may be a probability distribution. In some implementations, the loss 310 may be combined to form the loss 310c. Alternatively, the loss 310 may be sent to the S2S model 200 separately.

[0053] The loss function 305 may implement any suitable technique to determine the loss 310, such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross entropy, hinge loss, multi-class loss, etc. The loss 310 may then be fed directly to the S2S model 200. The S2S model 200 may then process the loss and adjust one or more parameters to account for the loss.

[0054] Figure 44 is a flow chart of an exemplary operational arrangement of the method 400 of the speech-to-speech model 200 with the turn detector 215. The data processing hardware 510 ( Figure 5 ) may be performed by executing a program stored in memory hardware 520 (which is in communication with data processing hardware 510) Figure 5 ) to perform the operations of method 400. Data processing hardware 510 and memory hardware 520 may reside on computing device 500 ( Figure 5 ) Figure 1 remote server 112 and / or user computing device 110.

[0055] At operation 402, the method 400 includes receiving a sequence of acoustic frames 102 corresponding to an utterance 108 spoken by the user 104 in streaming audio captured by a client device 110 associated with the user 104 as input to a speech-to-speech (S2S) model 200. At each of a plurality of output steps, operation 404 of the method 400 includes generating, by an audio encoder 212 of the S2S model 200, a higher-order feature representation 213 for a corresponding acoustic frame 102 in the sequence of acoustic frames 102. At each of the plurality of output steps, operation 406 of the method 400 includes determining, by a turn detector 215 of the S2S model 200, whether the utterance 108 is at a breakpoint at the corresponding output step based on the higher-order feature representation 213 generated by the audio encoder 212 at the corresponding output step. At operation 408, the method includes synthesizing a sequence of output audio frames 222 output by the speech decoder 220 of the S2S model 200 into a time-domain audio waveform representing a synthesized speech of the speech 108 spoken by the user 104 when the turn detector 215 determines that the speech is at a breakpoint. Here, each output audio frame 222 in the sequence of output audio frames 222 is based on a corresponding one of the higher-order feature representations 213 generated by the audio encoder 212 until the corresponding output step when the turn detector 215 determines that the speech is at a breakpoint. The sequence of output audio frames 222 corresponds to the output audio data 106.

[0056] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0057] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0058] Figure 5 5 is a schematic diagram of an example computing device 500 that can be used to implement the systems and methods described in this document. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only, and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0059] The computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 connected to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 connected to a low-speed bus 570 and the storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 510 can process instructions for execution within the computing device 500, including instructions stored in the memory 520 or on the storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 580 coupled to the high-speed interface 540. In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. In addition, multiple computing devices 500 can be connected, each of which provides a portion of the necessary operations (for example, as a server group, a blade server group, or a multi-processor system).

[0060] Memory 520 stores information non-temporarily within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 520 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0061] Storage device 530 is capable of providing mass storage for computing device 500. In some implementations, storage device 530 is a computer-readable medium. In various different implementations, storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0062] The high-speed controller 540 manages bandwidth-intensive operations of the computing device 500, while the low-speed controller 560 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner; or, for example, coupled to a networking device, such as a switch or a router, through a network adapter.

[0063] As shown, computing device 500 can be implemented in a variety of different forms. For example, computing device can be implemented as a standard server 500a or multiple implementations in a group of such servers 500a, implemented as a laptop computer 500b, or implemented as part of a rack server system 500c.

[0064] Various implementations of the systems and techniques described herein may be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that may be executed and / or interpreted on a programmable system that includes at least one programmable processor that may be special purpose or general purpose and that may be coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0065] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0066] The processes and logic flows described in this specification may be performed by one or more programmable processors (also referred to as data processing hardware) that execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows may also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or may be operably coupled to receive data from one or more mass storage devices or to transfer data to one or more mass storage devices or both. However, a computer need not have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0067] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor) or a touch screen for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0068] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (400) which, when executed on data processing hardware (510), causes the data processing hardware (510) to perform operations comprising: receiving as input to a speech-to-speech (S2S) model (200) a sequence of acoustic frames (102) corresponding to utterances (108) spoken by a user in streaming audio captured by a client device (110) associated with the user; and At each of the multiple output steps: Generating, by an audio encoder (212) of the S2S model (200), a higher-order feature representation (213) for a corresponding acoustic frame (102) in the sequence of acoustic frames (102); determining, by a turn detector (215) of the S2S model (200), whether the utterance (108) is at a breakpoint at the corresponding output step based on the higher-order feature representation (213) generated by the audio encoder (212) at the corresponding output step; and When the turn detector (215) determines that the utterance (108) is at the breakpoint, a sequence of output audio frames (222) output by the speech decoder (220) of the S2S model (200) is synthesized into a time-domain audio waveform of synthesized speech (114) representing the utterance (108) spoken by the user, each output audio frame (222) in the sequence of output audio frames (222) being based on a corresponding one of the higher-order feature representations (213) generated by the audio encoder (212) until the corresponding output step when the turn detector (215) determines that the utterance (108) is at the breakpoint.

2. The computer-implemented method (400) of claim 1, wherein the operations further comprise, when the utterance (108) is at the breakpoint, providing output from the client device (110): a speaker turn indication (115) notifying the user to stop speaking; and The time-domain audio waveform represents synthesized speech (114) of the utterance (108) spoken by the user.

3. The computer-implemented method (400) of claim 1 or 2, wherein: The utterance (108) spoken by the user in the streaming audio captured by the client device (110) is associated with an atypical voice; and The time-domain audio waveform representing synthesized speech (114) of the utterance (108) includes a time-domain audio waveform of synthesized canonical fluent speech of the same utterance (108) spoken by the user.

4. The computer-implemented method (400) of any one of claims 1 to 3, wherein: The utterance (108) spoken by the user in the streaming audio captured by the client device (110) is in a first language; and The time-domain audio waveform representing synthesized speech (114) of the utterance (108) includes a time-domain audio waveform of synthesized translated speech of the same utterance (108) in a second language different from the first language.

5. A computer-implemented method (400) as described in any one of claims 1 to 4, wherein determining whether the utterance (108) is at the breakpoint at the corresponding output step is further based on one or more of the higher-order feature representations (213) generated by the audio encoder (212) at an output step preceding the corresponding output step.

6. A computer-implemented method (400) as described in any one of claims 1 to 5, wherein the operation further includes, at each of the plurality of output steps, generating, by the speech decoder (220) of the S2S model (200), an output audio frame (222) for a corresponding higher-order feature representation (213) generated by the audio encoder (212) at the corresponding output step.

7. The computer-implemented method (400) of any one of claims 1 to 6, wherein in response to determining that the utterance (108) is at the breakpoint at the corresponding output step, the operations further comprise: receiving a sequence of said higher order feature representations (213) generated by said audio encoder (212) as input to said speech decoder (220) of said S2S model (200) until said corresponding output step; as well as The sequence of output audio frames (222) is generated by the speech decoder (220) of the S2S model (200).

8. The computer-implemented method (400) of any one of claims 1 to 7, wherein the turn detector (215) comprises a deep neural network.

9. The computer-implemented method (400) of any one of claims 1 to 8, wherein the turn detector (215) is arranged between the audio encoder (212) and the speech decoder (220).

10. A computer-implemented method (400) as described in any one of claims 1 to 9, wherein determining whether the utterance (108) is at the breakpoint at the corresponding output step includes generating as an output from the turn detector (215) a turn output (216) indicating whether the utterance (108) is at the breakpoint.

11. The computer implemented method (400) of claim 10, wherein the toggle output (216) comprises bits.

12. The computer-implemented method (400) of claim 10, wherein the rotation output (216) comprises a probability distribution.

13. The computer-implemented method (400) of any one of claims 1 to 12, wherein the operations further comprise training the S2S model (200) by: A set of training utterances (302) is received, each training utterance (302) in the set of training utterances (302): A corresponding sequence of training acoustic frames (320) is included, each training acoustic frame (320) in the sequence of training acoustic frames (320) being annotated with a label (321, 322) indicating whether the corresponding training acoustic frame (320) corresponds to a breakpoint frame or a non-breakpoint frame; and paired with a corresponding real-valued synthesized speech representation of said training utterance (302); Obtaining a first label (321) indicating a target output spectrogram of the training utterance (302); Obtaining a second label (322) indicating a target rotation output (216) of the training utterance (302); Generating a training output (350) using the S2S model (200) and the training utterance (302), the training output comprising: a training output spectrogram corresponding to a synthesized speech representation of the training acoustic frame (320); and a training rotation output indicating breakpoints in the training acoustic frame (320); determining a first loss (310) by comparing the training output spectrogram with the first label (321); determining a second loss (310) by comparing the training rotation output (216) to the second label (322); and The S2S model (200) is optimized based on the first loss (310) and the second loss (310) associated with the training input audio data (320).

14. The computer-implemented method (400) of any one of claims 1 to 13, wherein the operations further comprise: determining a voice type of the utterance (108) spoken by the user captured in the streaming audio by the client device (110); as well as The speech decoder (220) for generating the sequence of output audio frames (222) is selected from among a plurality of available speech decoders.

15. The computer-implemented method (400) of any one of claims 1 to 14, wherein each of the output audio frames (222) in the sequence of output audio frames (222) output by the speech decoder (220) comprises a spectrogram frame.

16. A system (100), comprising: Data processing hardware (510); as well as Memory hardware (520) storing instructions that, when executed on the data processing hardware (510), cause the data processing hardware (510) to perform operations comprising: Receiving a sequence of acoustic frames (102) corresponding to utterances (108) spoken by a user in streaming audio captured by a client device (110) associated with the user as input to a speech-to-speech (S2S) model; and At each of the multiple output steps: Generating, by an audio encoder (212) of the S2S model (200), a higher-order feature representation (213) for a corresponding acoustic frame (102) in the sequence of acoustic frames (102); determining, by a turn detector (215) of the S2S model (200), whether the utterance (108) is at a breakpoint at the corresponding output step based on the higher-order feature representation (213) generated by the audio encoder (212) at the corresponding output step; and When the turn detector (215) determines that the utterance (108) is at the breakpoint, a sequence of output audio frames (222) output by the speech decoder (220) of the S2S model (200) is synthesized into a time-domain audio waveform of synthesized speech (114) representing the utterance (108) spoken by the user, each output audio frame (222) in the sequence of output audio frames (222) being based on a corresponding one of the higher-order feature representations (213) generated by the audio encoder (212) until the corresponding output step when the turn detector (215) determines that the utterance (108) is at the breakpoint.

17. The system (100) of claim 16, wherein the operations further comprise, when the utterance (108) is at the breakpoint, providing output from the client device (110): a speaker turn indication (115) notifying the user to stop speaking; and The time-domain audio waveform represents synthesized speech (114) of the utterance (108) spoken by the user.

18. The system (100) of claim 16 or 17, wherein: The utterance (108) spoken by the user in the streaming audio captured by the client device (110) is associated with an atypical voice; and The time-domain audio waveform representing synthesized speech (114) of the utterance (108) includes a time-domain audio waveform of synthesized canonical fluent speech of the same utterance (108) spoken by the user.

19. The system (100) of any one of claims 16 to 18, wherein: The utterance (108) spoken by the user in the streaming audio captured by the client device (110) is in a first language; and The time-domain audio waveform representing synthesized speech (114) of the utterance (108) includes a time-domain audio waveform of synthesized translated speech of the same utterance (108) in a second language different from the first language.

20. The system (100) of any one of claims 16 to 19, wherein determining whether the utterance (108) is at the breakpoint at the corresponding output step is further based on one or more of the higher-order feature representations (213) generated by the audio encoder (212) at an output step prior to the corresponding output step.

21. The system (100) of any one of claims 16 to 20, wherein the operation further comprises, at each of the plurality of output steps, generating, by the speech decoder (220) of the S2S model (200), an output audio frame (222) for a corresponding higher-order feature representation (213) generated by the audio encoder (212) at the corresponding output step.

22. The system (100) of any one of claims 16 to 21, wherein in response to determining that the utterance (108) is at the breakpoint at the corresponding output step, the operations further comprise: receiving a sequence of said higher order feature representations (213) generated by said audio encoder (212) as input to said speech decoder (220) of said S2S model (200) until said corresponding output step; as well as The sequence of output audio frames (222) is generated by the speech decoder (220) of the S2S model (200).

23. The system (100) of any one of claims 16 to 22, wherein the turn detector (215) comprises a deep neural network.

24. The system (100) of any one of claims 16 to 23, wherein the turn detector (215) is arranged between the audio encoder (212) and the speech decoder (220).

25. The system (100) of any one of claims 16 to 24, wherein determining whether the utterance (108) is at the breakpoint at the corresponding output step comprises generating as an output from the turn detector (215) a turn output (216) indicating whether the utterance (108) is at the breakpoint.

26. The system of claim 25, wherein the toggle output (216) comprises bits.

27. The system of claim 25, wherein the rotation output (216) comprises a probability distribution.

28. The system (100) of any one of claims 16 to 27, wherein the operations further comprise training the S2S model (200) by: A set of training utterances (302) is received, each training utterance (302) in the set of training utterances (302): A corresponding sequence of training acoustic frames (320) is included, each training acoustic frame (320) in the sequence of training acoustic frames (320) being annotated with a label (321, 322) indicating whether the corresponding training acoustic frame (320) corresponds to a breakpoint frame or a non-breakpoint frame; and paired with a corresponding real-valued synthesized speech representation of said training utterance (302); Obtaining a first label (321) indicating a target output spectrogram of the training utterance (302); Obtaining a second label (322) indicating a target rotation output (216) of the training utterance (302); Generating a training output (350) using the S2S model (200) and the training utterance (302), the training output comprising: a training output spectrogram corresponding to a synthesized speech representation of the training acoustic frame (320); and a training rotation output indicating breakpoints in the training acoustic frame (320); determining a first loss (310) by comparing the training output spectrogram with the first label (321); determining a second loss (310) by comparing the training rotation output (216) to the second label (322); and The S2S model (200) is optimized based on the first loss (310) and the second loss (310) associated with the training input audio data (320).

29. The system (100) of any one of claims 16 to 28, wherein the operations further comprise: determining a voice type of the utterance (108) spoken by the user captured in the streaming audio by the client device (110); as well as The speech decoder (220) for generating the sequence of output audio frames (222) is selected from among a plurality of available speech decoders.

30. The system (100) of any one of claims 16 to 29, wherein each output audio frame (222) in the sequence of output audio frames (222) output by the speech decoder (220) comprises a spectrogram frame.

Images