End-to-end voice conversion

Abstract

Methods, systems, apparatus, including computer programs encoded on a computer storage medium, for end-to-end speech conversion are disclosed. In one aspect, a method includes actions of receiving first audio data of a first utterance of one or more first terms spoken by a user. The actions further include providing the first audio data as an input to a model configured to receive first given audio data spoken in the first sound and output second given audio data spoken in the synthetic sound without performing speech recognition on the first given audio data. The actions also include receiving second audio data of a second utterance of the one or more first terms spoken in the synthetic sound. The actions also include providing, as an output, second audio data of a second utterance of the one or more first terms spoken in the synthetic sound.

Description

This application is a divisional application of the following application: International application number of the original application: PCT / US2019 / 063334 International filing date of the original application: November 26, 2019 The national application number of the original application is: 201980094770.8, The title of the invention of the original application: End-to-end speech conversion. CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. application 62 / 808,627, filed on February 21, 2019, which is incorporated herein by reference. Technical Field

[0002] This specification generally relates to speech processing. Background Art

[0003] Speech processing is the study of speech signals and signal processing methods. Signals are usually processed in digital representation, so speech processing can be seen as a special case of digital signal processing applied to speech signals. Aspects of speech processing include the acquisition, processing, storage, transmission and output of speech signals. Summary of the invention

[0004] Speech synthesizers typically require a transcription as input. Speech synthesizers receive the transcription and output audio data of the transcribed synthesized utterance. In order to convert the user's speech into a synthesized utterance, an automatic speech recognizer must perform automatic speech recognition on the audio data of the user's speech to generate a transcription of the user's speech. The speech synthesizer will then generate a synthesized utterance of the transcription of the user's speech.

[0005] This technology for performing automatic speech recognition and speech synthesis may place a burden on the computing system. It would be beneficial to be able to convert speech audio received from a user into speech audio of a sound different from the user's voice (Voice) without performing automatic speech recognition on the user's voice. The following discussion describes the process of converting speech audio in a speaker's voice into speech audio in a different voice using a model trained using machine learning without performing speech recognition. The model receives speech audio spoken by a speaker and converts the speech audio into a mathematical representation. The model converts the mathematical representation into speech audio of a different voice without performing speech recognition on the speech audio spoken by the speaker.

[0006] In some embodiments, the speech synthesis system is capable of converting first audio data including speech in a first voice into second audio data including the same speech in a second voice. The conversion can be done by acting directly on samples or features of the first audio data without converting the audio into an intermediate representation (e.g., text, phone, etc.). The system can use sequence to sequence to normalize any speech that potentially includes background noise and generate the same content in the voice of a single predefined target speaker. The source speech can come from any speaker or accent and may contain complex prosodic patterns, imperfections, and background noise, all of which are removed by the normalization process because the first audio data is converted into clean second audio data with a fixed accent and coherent articulation and prosody. In other words, the system can be used to throw away all non-linguistic information including speaker characteristics and retain only what is said, not who, how, or where it is said.

[0007] This type of normalization has a variety of potential applications. Fully normalizing any speech to a single speaker with clean audio can significantly simplify speech recognition models, which can be reduced to support a single speaker. Removing the identity of the speaker can be useful when recording sensitive and private speech data, allowing the user to transmit only the transformed speech to a server with the "acoustic" identity removed. Reducing all accents to a single voice with a predefined accent, unlike acoustically disguised audio, can also mitigate bias and discrimination while maintaining a natural human voice, for example, for telephone interviews or recordings of candidate conversations provided to hiring committees. Other applications will help understand the speech content of accents that are unfamiliar to the listener, i.e., improve the intelligibility of heavily accented speech.

[0008] According to the innovative aspects of the subject matter described in the present application, an end-to-end speech conversion method includes the following actions: a computing device receives first audio data of a first utterance of one or more first terms spoken by a user; the computing device provides the first audio data as input to a model, the model being configured to receive first given audio data of a first given utterance of one or more first given terms spoken in a first voice, and output second given audio data of a second given utterance of the one or more first given terms spoken in a synthesized voice without performing speech recognition on the first given audio data; in response to providing the first audio data as input to the model, the computing device receives second audio data of a second utterance of the one or more first terms spoken in the synthesized voice; and the computing device provides, as output, the second audio data of the second utterance of the one or more first terms spoken in the synthesized voice.

[0009] Each of these and other embodiments may include one or more of the following features. The action also includes: the computing device receives data indicating that the robot configured to conduct a conversation with a given human is not configured to generate a response to a third utterance received from the human; and based on receiving the data indicating that the robot is not configured to generate the response from the third utterance received from the human, the computing device sends a request for a response to the third utterance received from the human to a human operator. The action of receiving the first audio data of the first utterance of the one or more first terms spoken by the user includes receiving the first audio data of the first utterance of the one or more first terms spoken by the human operator in response to the third utterance. The action of receiving the first audio data of the first utterance of the one or more first terms spoken by the user includes receiving the first audio data of the first utterance of the one or more first terms spoken by the human operator when answering a phone call.

[0010] The actions also include receiving audio data of an utterance combination; obtaining a transcription of each utterance in the utterance set; providing the transcription of each utterance as input to a text-to-speech model; for each transcription of each utterance, receiving audio data of an additional utterance set in a synthesized voice; and training the model using the audio data of the utterance set and the audio data of the additional utterance set in a synthesized voice. The actions also include: the computing device receiving third audio data of a third utterance of one or more third terms spoken by an additional user; the computing device providing the third audio data as input to the model; in response to providing the third audio data as input to the model, the computing device receiving fourth audio data of a fourth utterance of one or more third terms spoken in the synthesized voice; and the computing device providing the fourth audio data of the fourth utterance of the one or more third terms spoken in the synthesized voice as output. The actions also include the computing device bypassing obtaining a transcription of the first utterance. The model is configured to adjust the time period between each of the one or more first given terms. The model is configured to adjust the speaking time between each of the one or more first given terms.

[0011] Other embodiments of this aspect include corresponding systems, apparatus, and computer programs recorded on computer storage devices, each of which is configured to perform the operations of the above-described methods.

[0012] Particular implementations of the subject matter described in this specification can be implemented so as to achieve one or more of the following advantages: A computing system can receive audio data of an utterance spoken in a user voice and output audio data of a synthesized utterance spoken in a synthesized voice without the additional overhead of performing automated speech recognition on the received audio data.

[0013] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following detailed description. Other features, aspects, and advantages of the subject matter will become apparent from the detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 An example system is shown that converts speech audio received from a user into synthesized speech audio without performing speech recognition.

[0015] Figure 2 An example system is shown that converts voice audio received from a human operator interrupting a conversation between an automated robot and a user into voice audio that mimics the automated robot.

[0016] Figure 3An example system is shown that converts voice audio received from a callee screening a call into voice audio that blocks the caller from determining that the callee is answering the call.

[0017] Figure 4 Flowchart of an example process for converting speech audio received from a user into synthesized speech audio without performing speech recognition.

[0018] Figure 5 An example network architecture of a system is shown that converts speech audio received from a user into synthesized speech audio having a higher pitch and faster speaking rate than the user without performing speech recognition.

[0019] Figure 6 Examples of computing devices and mobile computing devices are shown.

[0020] Like reference numbers and numerals in the various drawings represent like elements. DETAILED DESCRIPTION

[0021] Figure 1 An example system 100 is shown that converts speech audio 102 received from a user 104 into synthesized speech audio 106 without performing speech recognition. Briefly, and as described in more detail below, a user 104 speaking with a British accent speaks an utterance 108 near a computing device 110. The computing device 110 transmits the audio data 102 of the utterance 108 to a speech-to-speech conversion server 112. The speech-to-speech conversion server 112 converts the audio data 102 of the utterance 108 into audio data 106 of a synthesized utterance 114. The speech-to-speech conversion server 112 transmits the audio data 106 of the synthesized utterance 114 to a computing device 116, and the computing device 116 outputs the synthesized utterance 114. In some implementations, the functionality of the end-to-end speech conversion server 112 is built into the computing device 110 or the computing device 116, or both.

[0022] In more detail, user 104 and user 118 are talking to each other via computing device 110 and computing device 116. User 104 and user 118 may be talking via a telephone, or another type of voice communication protocol, such as an Internet voice protocol. Although user 104 and user 118 speak the same language, user 118 may have difficulty understanding user 104 because user 104 has a strong accent. In this example, user 104 may be British, and user 118 may understand American accent 120 more easily than British accent 122 of user 104.

[0023] To solve this problem, computing device 110 can provide speech 108 and audio data 102 of other speech uttered by user 104 to speech-to-speech conversion server 112. Speech-to-speech conversion server 112 can be configured to convert the audio data of speech uttered by the user who may have an accent into the audio data of synthesized speech that may have different accents. In order to complete this conversion, the server will usually be configured to perform speech recognition on the audio data of the speech uttered by the user with an accent. The speech recognizer can be configured to recognize the speech uttered with the user's accent or can be configured to recognize the speech uttered with any accent. The server then provides transcription to a speech synthesizer, which generates audio data of synthesized speech with different accents. Speech-to-speech conversion server 112 operates in different ways.

[0024] The speech-to-speech conversion server 112 receives the audio data 102 of the speech 108 from the computing device 110 and provides the audio data 102 of the speech 108 to the model 124. The speech-to-speech conversion server 112 trains the model 124 to convert the audio data 102 of the speech 108 spoken in the British accent 122 into the audio data 106 of the speech 114 spoken in the American accent 120. The speech-to-speech conversion server 112 does not use the speech recognizer 126 to perform the conversion. The speech recognizer 126 can remain inactive during the conversion process. Instead, the model 124 provides the audio data 102 of the speech 108 to the encoder 128. The encoder 128 can be configured to convert the audio data 102 of the speech 108 into an internal representation, such as a series of vectors. For example, when the encoder 128 receives the audio data 102 of the speech 108, the encoder 128 can process five frames of audio and convert the five frames of audio into ten vectors. These vectors are not transcriptions of the frames of audio data 102, but rather mathematical representations of the frames of audio data 102. Model 124 provides the series of vectors to a spectrogram decoder 130. Spectrogram decoder 130 may be configured to generate audio data for a synthesized utterance based on the vectors received from encoder 128. For example, spectrogram decoder 130 may receive ten vectors representing five frames of audio from encoder 128. Spectrogram decoder 130 generates five frames of audio data 106 for a synthesized utterance 114 that include the same words or parts of words as the five frames of audio data, but with a different voice from user 104.

[0025] The speech-to-speech conversion server 112 provides the audio data 106 of the synthesized utterance 114 to the computing device 116. Figure 1In the example shown, the speech-to-speech server 112 provides audio data 106 of a synthesized utterance of “Can I make an appointment tomorrow?” The synthesized utterance 114 may have an American accent 120. In some implementations, the synthesized utterance 114 may have the same accent as the user 104 and a different voice than the user 104. The sound of the synthesized utterance 114 may make it impossible for the user 118 or another user to recognize the user 104 as the speaker of the utterance 108. In some implementations, the candence of the synthesized utterance 114 may be different from the candence of the utterance 108. The speech-to-speech server 112 may adjust the candence of the synthesized utterance 114 to increase the likelihood that the user 118 can understand the synthesized utterance 114.

[0026] The computing device 116 receives the audio data 106 of the synthesized utterance 114 and outputs the audio data 106 through a speaker or other audio output device. In some implementations, as the user 104 speaks portions of the utterance 108, the speech-to-speech conversion server 112 continuously generates corresponding portions of the synthesized utterance 114. For example, the speech-to-speech conversion server 112 may generate one second of the synthesized utterance 114 after receiving one second of the utterance 108. By continuously generating portions of the synthesized utterance 114, the conversation between the users 104 and 118 may be more naturally synchronized. In some implementations, the speech-to-speech conversion server 112 may determine when the user 104 has stopped speaking. After determining that the user 104 has stopped speaking, the speech-to-speech conversion server 112 converts the audio data 102 of the utterance 108 into the audio data 106 of the synthesized utterance 114.

[0027] The speech-to-speech server 112 includes various components for generating training data and training models 124. The speech-to-speech server 112 includes utterance transcriptions 132 and utterance audio data 134. The utterances may be utterances spoken by different users with different types of accents. In some embodiments, the utterance transcriptions 132 are generated by an automatic speech recognizer. Before the transcriptions are stored in the utterance transcriptions 132 and the audio data is stored in the utterance audio data 134, the speaker of each utterance can verify the accuracy of the transcriptions. In some embodiments, the utterance transcriptions 132 are generated by one or more people.

[0028] The speech-to-speech conversion server 112 provides the speech transcription 132 to a speech synthesizer 136. The speech synthesizer is configured to generate synthesized speech audio data 138 of the transcription 132. The speech synthesizer is configured to generate the synthesized speech audio data 138 with a single voice. The voice may have a specific accent, such as an American accent or a British accent. The synthesized speech audio data 138 may be free of any background noise or other audio artifacts.

[0029] The speech-to-speech conversion server 112 provides the synthesized speech audio data 138 and the speech audio data 134 to the model trainer 140. The model trainer 140 trains the model 124 using machine learning techniques. The model trainer 140 trains the model 124 to receive audio data similar to the speech audio data 134 and output audio data similar to the synthesized speech audio data 138 without performing speech recognition on the received audio data. Even when the model 124 receives different inputs of audio data including different speech of different voices, the model trainer 140 trains the model 124 to output speech of the same sound as the synthesized speech from the speech synthesizer 136.

[0030] In some implementations, the speech-to-speech conversion server 112 may use utterance audio data 134 that includes different audio features. This may result in the model 124 being configured to process input of audio data having those different features. In some implementations, the speech-to-speech conversion server 112 may add audio features to the utterance audio data 134 so that the model trainer 140 trains the model 124 to process audio features similar to the added audio features.

[0031] For example, speech to speech conversion server 112 can add different levels of noise to speech audio data 134. Different levels of noise can include different types of noise, such as stationary noise and / or non-stationary noise. Stationary noise can include different levels of road noise, different levels of background voice noise similar to a cocktail party or restaurant, different levels of fan noise, and / or any other similar type of noise. Non-stationary noise can include different levels of television noise, different levels of gust noise, different levels of background music noise, and / or any other similar type of noise. Speech to speech conversion server 112 can add different levels and different types of noise to the audio data of the same speech. This may result in generating multiple audio data samples that match the same transcription, each audio data sample includes the same underlying speech audio data, and the underlying speech audio data has different levels and different types of noise added. By adding noise, model 124 can be better configured to process the received audio data, and in addition to the speech audio data, the received audio data also includes background noise.

[0032] As another example, the speech-to-speech conversion server 112 can process speech audio data 134 from a user with a speech disorder that may cause the user to speak unfluently, such as those with amyotrophic lateral sclerosis. The model trainer 140 can train the model 124 using speech audio data and transcriptions of speech from the user with the speech disorder, so that the model 124 can receive speech audio data spoken by the user with the speech disorder and output speech audio data with a more consistent rhythm that can be more easily understood by another user.

[0033] As another example, the speech-to-speech conversion server 112 may be configured to translate an utterance into a different language without transcribing the utterance. In this case, the utterance audio data 134 may include an utterance spoken in a first language, such as English. The utterance transcription 132 may include a transcription of a translation of the utterance in a second language, such as a transcription of a Spanish translation. The speech synthesizer 136 may be configured to generate synthesized speech in a second language, such as a synthesized Spanish utterance. The model trainer 140 uses machine learning to train the model 124 using the utterance audio data 134 in the first language and the synthesized utterance audio data 138 in the second language. The resulting model 124 is configured to receive utterance audio data in a first language (e.g., English) without transcribing the received utterance, and output synthesized utterance audio data in a second language (e.g., Spanish).

[0034] Figure 2 An example system 200 is shown that converts speech audio 202 received from an operator 204 into speech audio 210 that mimics an automated agent 206, the operator 204 interrupting a conversation between the automated agent 206 and a user 208. In short, as described in more detail below, the user 208 is having a conversation with the automated agent 206. During the conversation, the user 208 speaks an utterance 212 to which the automated agent 206 cannot respond. The operator 204 receives an indication that the automated agent 206 cannot respond to the utterance 212. The operator 204 provides an utterance 214 in response to the utterance 212. The speech-to-speech conversion server 216 converts the speech audio 202 of the utterance 214 into the speech of the automated agent 206, thereby making the user 208 believe that the user 208 is still talking to the same party. In some embodiments, the functionality of the automated agent 206 is built into the computing device 220 or the computing device 234 or both. In some embodiments, the functionality of the end-to-end speech conversion server 216 is built into the computing device 220 or the computing device 234 or both.

[0035] In more detail, in stage A, user 208 initiates a telephone conversation with automated agent 206. Computing device 220 is connected to automated agent 206. The user speaks utterance 218 and asks the automated agent, "Can I reserve a table for two?" Automated agent 206 can mimic a human being, such that user 208 cannot distinguish automated agent 206 from a real person. In some embodiments, automated agent 206 can initiate a telephone conversation with user 208. In some embodiments, the conversation between user 208 and automated agent 206 can be a communication channel other than a telephone call, such as a VOIP call or other type of voice communication.

[0036] In phase B, computing device 220 detects speech 218 through a microphone or another input device, and uses an audio subsystem to process audio data of speech 218. The audio subsystem may include a microphone, an analog-to-digital converter, a buffer, and various other audio filters. The microphone may be configured to detect sounds in the surrounding area, such as the voice of speech 218. The analog-to-digital converter may be configured to sample the audio data detected by the microphone. The buffer may store the sampled audio data for processing by computing device 220 or for transmission by computing device 220. In some embodiments, the audio subsystem may be continuously active, or may be active during a period when computing device 220 is waiting to receive audio, such as during a phone call. In this case, in response to the initiation of a phone call with automation agent 206, the microphone may detect audio. The analog-to-digital converter may continuously sample the detected audio data during the phone call. The buffer may store the latest sampled audio data, such as the last ten seconds of sound. Computing device 220 may provide sampled and filtered audio data 222 of speech 218 to automation agent 206.

[0037] Automated agent 206 receives audio data 222 of utterance 218 and determines an appropriate response. Automated agent 206 may apply a series of rules, a decision tree, a neural network, and / or another decision process to determine an appropriate response. Automated agent 206 may generate a transcription of the appropriate response and provide the transcription to a speech synthesizer. At stage C, the speech synthesizer may generate audio data 224 representing utterance 226, "tonight?", as a response to "Can I reserve a table for two?". Although the speech synthesizer may generate utterance 226, user 208 cannot determine that user 208 is speaking to a computer.

[0038] At stage D, computing device 220 receives audio data 224 of utterance 226. Computing device 226 outputs audio data 224 through a speaker or other type of audio output device. User 208 hears utterance 226 in synthesized speech 228.

[0039] At stage E, user 208 responds to utterance 226 by uttering utterance 212 "I have to watch the game tonight. Did you know Smith is playing?" Computing device 220 detects utterance 212 and processes utterance 212 using the audio subsystem. At stage F, computing device 220 provides audio data 230 of utterance 212 to automated agent 206.

[0040] The automated agent 206 receives the audio data 230 of the utterance 212. The automated agent 206 processes the audio data 230 of the utterance 212 in a manner similar to the processing of the audio data 222 of the utterance 218. The automated agent 206 may apply a series of rules, a decision tree, a neural network, and / or another decision-making process to determine an appropriate response to the utterance 212. In this case, the automated agent 206 is unable to determine an appropriate response. In the case where the user utterance deviates from the topic of the conversation, the automated agent 206 may be unable to determine an appropriate response to the user utterance.

[0041] In order to continue the conversation, when the automated agent 206 or other automated agents are unable to generate an appropriate response to the user utterance 212, the automated agent 206 can notify the operator 204 on standby to provide assistance. In stage G, the automated agent 206 generates a summary 232 of the conversation between the automated agent 206 and the user 208. The summary 232 may include a record of the conversation up to the point where the automated agent 206 is unable to generate an appropriate response to the user utterance 212. Alternatively or additionally, the summary 232 may include a description of any tasks completed as a result of the conversation or agreement between the automated agent 206 and the user 208. Figure 2 In the example of , summary 232 includes a record of a conversation between automated agent 206 and user 212. Automated agent 206 may also include a state 236 of the conversation. State 236 may describe a task that automated agent 206 was unable to perform. State 236 may indicate that automated agent 206 was unable to perform speech recognition on audio data 230 of utterance 212. In that case, summary 232 would include audio data 230 of utterance 212. State 236 may indicate that automated agent 206 was unable to generate a response to utterance 212.

[0042] At stage H, computing device 234 of operator 204 receives summary 232 and status 236. Operator 204 reviews summary 232 and status 236. Operator 204 speaks utterance 214 as a response to utterance 212 of user 208 and as an attempt to direct the conversation back to the original topic, or a topic that automated agent 206 can understand. Computing device 234 detects utterance 214 and processes utterance 214 using the audio subsystem in a manner similar to that computing device 220 processes utterance 212 and utterance 218. Figure 2In the example shown, user 204 says, "I don't know. This should be exciting. What day would you like to book?"

[0043] In stage I, the computing device 234 transmits the audio data 202 of the utterance 214 to the speech-to-speech conversion server 216. The speech-to-speech conversion server 216 can be similar to the speech-to-speech conversion server 112, wherein the speech-to-speech conversion server 216 is configured to receive the utterance audio data spoken in a first voice and output the utterance audio data including the same words and terms spoken in a second and different voice without performing speech recognition on the utterance audio data spoken in the first voice. The speech-to-speech conversion server 216 can be configured to generate the utterance audio data of the same synthesized speech 228 as the automated agent 206.

[0044] At stage J, the speech-to-speech server 216 converts the audio data of the utterance 214 into audio data 210 of the utterance 238. The speech-to-speech server 216 transmits the audio data 210 of the utterance 238 to the computing device 220. In some implementations, the speech-to-speech server 216 transmits the audio data 210 of the utterance 238 to the computing device 234. The computing device 234 then transmits the audio data 210 of the utterance 238 to the automated agent 206. The automated agent transmits the audio data 210 of the utterance 238 to the computing device 220. In some implementations, the speech-to-speech server 216 transmits the audio data 210 of the utterance 238 to the automated agent 206. The automated agent transmits the audio data 210 of the utterance 238 to the computing device 220. In some implementations, the computing device 234 provides the automated agent 206 with a transcription of the utterance 214 so that the automated agent still knows the content of the conversation with the user 208. The automated agent 206 may use the transcription of the utterance 214 to update models and / or rules used by the automated agent 206 to generate responses.

[0045] At stage K, computing device 220 outputs utterance 238 through a speaker or other type of audio output device. User 208 hears utterance 238, and because utterance 238 is spoken in the same synthesized voice 228 as utterance 226, user 208 is unaware that another party is participating in the conversation. User 208 may respond to utterance 238 by speaking a new utterance. Operator 204 may continue to monitor the conversation to ensure that automated agent 206 is able to seamlessly take over the conversation. If necessary, operator 204 may continue to talk to user 208 through speech-to-speech conversion server 216 for the remainder of the conversation or a portion of the remainder of the conversation. During the conversation, user 208 may believe that user 208 is talking to the same real person.

[0046] Figure 3 An example system 300 is shown that converts voice audio 302 received from a callee 304 screening a call into voice audio 306 that prevents a caller 308 from determining that it was the callee 304 who answered the call. In short, as described in more detail below, a caller 308 places a telephone call to a callee 304. The callee 304 may be unsure about answering the call, and instead of having the call go to voicemail, the callee 304 answers and screens the call. The callee 304 may use the call screening functionality of a computing device 210, which accesses a speech-to-speech conversion server 312. The speech-to-speech conversion server 312 converts audio data 302 of utterances 314 spoken in the voice of the callee into audio data 306 of synthesized utterances 316 spoken in a generic voice. The caller 308 answers the screening questions without being aware that the callee 304, and possibly a real person, answered the call. In some implementations, the functionality of the end-to-end speech conversion server 312 is built into the computing device 310 or the computing device 318 or both.

[0047] In more detail, in stage A, caller 308 Alice initiates a phone call with callee 304 Bob. Computing device 310 indicates that computing device 310 is receiving an incoming call by outputting notification 311. Unlike a phone call, caller 308 may initiate a voice communication through an alternative type of communication channel, such as VOIP or a similar type of voice communication. Caller 308 initiates a phone call from computing device 318. The computing device 310 of callee 304 indicates that callee 304 is receiving a phone call. Computing device 310 may give callee 304 the option of answering the phone call directly, ignoring the phone call, sending the phone call to voicemail, or initiating call screening.

[0048] At stage B, callee 304 initiates the option of call screening. Upon selecting the option of call screening, computing device 310 initiates communication with speech-to-speech conversion server 312. Computing device 310 indicates that computing device 310 will send audio data to speech-to-speech conversion server 312 for conversion into another voice.

[0049] In stage C, callee 304 speaks speech 314. Computing device 310 detects speech 314 through microphone or another type of audio input device, and uses audio subsystem to process audio data. Audio subsystem may include microphone, analog-to-digital converter, buffer and various other audio filters. Microphone may be configured to detect sounds in surrounding area, such as voice of speech 314. Analog-to-digital converter may be configured to sample audio data detected by microphone. Buffer may store sampled audio data for processing by computing device 310 or for transmission by computing device 310. In some embodiments, audio subsystem may be continuously active or may be active during the period when computing device 310 is waiting to receive audio, such as active during telephone call. In this case, microphone may detect audio in response to initiation of call screening option. Analog-to-digital converter may continuously sample detected audio data during telephone call. Buffer may store latest sampled audio data, such as the sound of the last ten seconds. In stage D, computing device 310 may provide sampled and filtered audio data 302 of speech 314 to voice-to-voice conversion server 312.

[0050] The speech-to-speech conversion server 312 receives audio data 302 of an utterance 314 spoken by the callee 304 from the computing device 310. In some implementations, the computing device 310 provides instructions to the speech-to-speech conversion server 312 to convert the audio data 302 of the utterance 314 spoken by the callee 304 into an utterance spoken in a different voice. In some implementations, the computing device 310 provides instructions as to where the speech-to-speech conversion server 312 should send the audio data 306 of the synthesized utterance 316 spoken in a different voice. For example, the computing device 310 may provide a phone number or a device identifier of the computing device 318, and instructions to transmit the audio data 306 of the synthesized utterance 316 spoken in a different voice. In some implementations, the computing device 310 may provide instructions to the speech-to-speech conversion server 312 to transmit the audio data 306 of the synthesized utterance 316 spoken in a different voice back to the computing device 310, so that the computing device can transmit the data 306 of the synthesized utterance 316 spoken in a different voice to the computing device 318.

[0051] At stage E, the speech-to-speech conversion server 312 generates audio data 306 of a synthesized utterance 316 spoken in a voice different from the voice of the callee 304. The speech-to-speech conversion server 312 can be similar to the speech-to-speech conversion server 112, wherein the speech-to-speech conversion server 312 is configured to receive audio data of an utterance spoken in a first voice and output audio data including an utterance of the same words and terms spoken in a second and different voice without performing speech recognition on the audio data of the utterance spoken in the first voice. In this example, the speech-to-speech conversion server 312 receives audio data 302 of an utterance 314 spoken in the voice of the callee 304. The speech-to-speech conversion server 312 provides the audio data 302 of the utterance 314 spoken in the voice of the callee 304 to the model, which generates audio data 306 of an utterance 316 spoken in a generic voice without performing speech recognition on the audio data 302. The speech-to-speech server 312 provides the audio data 306 of the utterance 316 to the computing device 318. In some implementations, the speech-to-speech server 312 provides the audio data 306 of the utterance 316 to the computing device 310, and the computing device 310 provides the audio data 306 of the utterance 316 to the computing device 318.

[0052] At stage F, computing device 318 outputs audio data 306 of utterance 316 through a speaker or other audio output device of computing device 318. Utterance 316 is not in the voice of callee 304, but in a different generic voice that sounds like a real person and not like callee 304. Figure 3 In the example of , caller 308 hears “Please state your name and purpose of call” in a voice that does not sound like callee 304 . Caller 308 may think that caller 308 is talking to callee 304 's secretary or assistant.

[0053] At stage G, caller 308 responds to utterance 316 by speaking utterance 320. Utterance 320 is detected by a microphone or other audio input device of computing device 318. The audio subsystem of computing device 318 processes utterance 320. Figure 3 In the example of , caller 308 says: "This is Alice, and I'm calling to schedule a meeting with Bob."

[0054] At stage H, computing device 318 transmits audio data 322 of speech 320 to computing device 310. The call screening function of this example can work in one direction. In other words, the voice of callee 304 who activated the call screening function is disguised by the call screening function. The voice of caller 308 remains unchanged.

[0055] In stage I, computing device 310 outputs audio data 322 of utterance 320 through a speaker or other audio output device of computing device 310. Utterance 320 is the voice of caller 308. Callee 304 hears caller 308 saying, "This is Alice. I'm calling to schedule a meeting with Bob."

[0056] At stage J and while call screening is still active, callee 304 speaks utterance 326. Utterance 326 is detected by a microphone or other audio input device of computing device 310. The audio subsystem of computing device 310 processes utterance 326. Figure 3 In the example, the callee says, "Wait a moment."

[0057] At stage K and while call screening is still active, computing device 310 transmits audio data 328 of utterance 326 to speech-to-speech conversion server 312. Speech-to-speech conversion server 312 provides audio data 328 of utterance 326 to the same model as in stage D. The model generates audio data 330 of utterance 332 in a voice different from the voice of callee 304. In some implementations, the voice of utterance 332 is the same as the voice of utterance 316. Speech-to-speech conversion server 312 generates audio data 330 of utterance 332 without performing speech recognition on audio data 328.

[0058] At stage L, speech-to-speech server 312 provides audio data 330 of utterance 332 to computing device 318. In some implementations, speech-to-speech server 312 provides audio data 330 of utterance 332 to computing device 310, and computing device 310 provides audio data 330 of utterance 332 to computing device 318.

[0059] At stage 24, computing device 318 outputs audio data 330 of speech 332 through a speaker or other audio output device of computing device 318. Speech 332 is not the voice of callee 304, but is a generic voice that is the same as speech 316 or another voice that sounds like a real person. Figure 3 In the example of , caller 308 hears “wait a minute” that does not sound like the voice of callee 304 . Caller 308 may continue to believe that caller 308 is talking to callee 304 's secretary or assistant.

[0060] At stage N, callee 304 speaks utterance 334. Speech 334 is detected by a microphone or other audio input device of computing device 310. The audio subsystem of computing device 310 processes utterance 334. Figure 3In the example of , callee 304 says, "Hi! Alice, this is Bob." Prior to uttering utterance 334, callee 304 may deactivate call screening mode of computing device 310. Callee 304 may deactivate call screening mode at any time during execution of stages K, L, or M and prior to stage N. By deactivating call screening mode, computing device 310 returns to transmitting audio data of the utterance spoken by caller 304 to computing device 318, rather than transmitting audio data of the utterance spoken by caller 304 to speech-to-speech conversion server 312. In some implementations, computing device 310 provides an indication to speech-to-speech conversion server 312 that computing device 310 will not transmit audio data of subsequently received utterances to speech-to-speech conversion server 312 for conversion to a different voice.

[0061] In stage 0 and when call screening is deactivated, computing device 310 transmits audio data 336 to computing device 318. This transmission of audio data may be similar to the audio data transmission that occurs during a typical voice conversation between two users using computing devices similar to computing device 310 and computing device 318.

[0062] At stage P, computing device 318 outputs speech 338 via a speaker or other audio output device of computing device 318. Figure 3 In the example of , computing device 318 outputs speech 338 "Hi Alice, this is Bob". The voice of speech 338 is the voice of callee 304. Caller 308 is likely to think that the person screening the call transferred the call to callee 304 and that caller 308 had not spoken to callee 304 during the entire call.

[0063] Figure 4 Flow chart of an example process for converting speech audio received from a user into synthesized speech audio without performing speech recognition. Generally, process 400 receives audio data of an utterance spoken by a user. Process 400 converts the utterance audio data into audio data of another utterance in a different voice by applying the utterance audio data to a model. The different voice is a synthesized voice that sounds like a real person. A person who hears the other utterance in the different voice may not realize that the original user said the utterance before converting it to the different voice. Process 400 generates audio data of the other utterance in the different voice without performing speech recognition on the received audio data. Process 400 outputs audio data of the other utterance in the different voice. Process 400 will be described as being performed by, for example, Figure 1 System 100 in Figure 2 System 200, or Figure 3 The system 300 is executed by a computer system of one or more computers.

[0064] The system receives first audio data of a first utterance of one or more first terms spoken by a user (410). The user can speak in a typical voice of the user. In some embodiments, the user speaks the first utterance while answering a call. In some embodiments, the user can activate a call screening function of the system before answering the call.

[0065] The system provides first audio data as input to a model, which is configured to receive first given audio data of a first given utterance of one or more first given terms spoken in a first voice, and output second given audio data of a second given utterance of one or more first given terms spoken in a synthesized voice, without performing speech recognition on the first given audio data (420). The model can use an encoder to encode the first audio data into a series of vectors representing the audio data. These vectors can be different from the transcription of the first audio data. The model can use a decoder to generate the output audio data. The decoder can be configured to convert each vector into a synthesized speech of a voice different from the user's voice. In some embodiments, the model bypasses transcribing the first audio data of the first utterance.

[0066] In response to providing the first audio data as input to the model, the system receives second audio data of a second utterance of the one or more first terms spoken in a synthesized voice (430). In some implementations, a speaking time of each of the one or more first terms in the first utterance can be different from a speaking time of each of the one or more first terms in the second utterance. In some implementations, a time period between each of the one or more first terms in the first utterance can be different from a time period between each of the one or more first terms in the second utterance.

[0067] The system provides as output (440) second audio data of a second utterance of one or more first terms spoken in a synthesized voice. The system can output the second audio data to a speaker or other audio output device. Another user can hear the second utterance and may not know that the original user said the first utterance. Even if the system generates the audio data of the second utterance using a model, the second utterance may sound like a real person's voice. In some embodiments, the synthesized voice can have gender-neutral characteristics so that a listener cannot determine whether the speaker is male or female. The pitch of the gender-neutral synthesized voice can be an average of the pitch of a female synthesized voice and the pitch of a male synthesized voice.

[0068] In some embodiments, the system may receive utterances from different users. The system may apply audio data of utterances from different users to the model. The model may output audio data of synthesized utterances in the same synthesized voice. In other words, the model may be configured to convert audio data of utterances spoken by different people into utterances in the same synthesized voice.

[0069] In some embodiments, the system can train a model using a collection of utterances received by the system and other systems. The system obtains a transcription of each utterance in the collection of utterances. The system can generate the transcription using automatic speech recognition or by manual transcription. The system provides each transcription to a speech synthesizer or text-to-speech model that generates a synthesized utterance with a synthesized voice. The system trains the model using machine learning, a collection of utterances, and corresponding synthesized utterances. The trained model is configured to generate a synthesized utterance with the same synthesized voice based on the utterances received from the user. The trained model does not use speech recognition to generate the synthesized utterances.

[0070] In some embodiments, the system may be part of an automated agent or robot configured to conduct a voice conversation with a user. The user may have the impression that the user is not speaking to a computer, but rather to a real person. The automated agent may not be able to generate a suitable response to every utterance that the automated agent receives from the user. In such cases, an operator may be on standby to have the automated agent generate responses to the user's utterances so that the conversation can continue. The system may help disguise the operator's voice so that the user thinks the user is still speaking to the same person. The system may convert the operator's voice into the automated agent's voice so that the user hears the same voice even when the operator, rather than the automated agent, generates the response.

[0071] In more detail, this paper describes an end-to-end speech-to-speech model that maps an input spectrogram directly to another spectrogram without any intermediate discrete representation. The network consists of an encoder, a spectrogram decoder, a phoneme decoder, and a vocoder that synthesizes a time-domain waveform. The model can be trained to normalize speech from any speaker, including accents, emotions, complex prosodic patterns, defects, and background noise, to a clean single predefined target speaker's voice with a fixed accent and consistent pronunciation and prosody. This paper describes the impact of this method on speech recognition performance. In addition, this paper illustrates that the same architecture can be trained on a speech separation task. In some embodiments, an end-to-end speech-to-speech model can translate Spanish speech into synthesized English speech.

[0072] Encoder-decoder models with attention can be used to model a variety of complex sequence-to-sequence problems. These models can be used for speech and natural language processing, such as machine translation, speech recognition, and combined speech translation. Using a single neural network that directly generates a target sequence given a virtual raw input, these models can also be used for end-to-end text-to-speech (TTS) synthesis and automatic speech recognition (ASR).

[0073] This paper describes combining state-of-the-art speech recognition and synthesis models to build a direct end-to-end speech-to-speech sequence converter that generates speech spectrograms as a function of different input spectrograms without relying on intermediate discrete representations. The model can be first applied to speech normalization and speech separation tasks. The model can be used to translate directly from one language to another, for example, from Spanish speech to English speech.

[0074] In some embodiments, a unified sequence-to-sequence model can normalize arbitrary speech, potentially including background noise, and generate the same content in the voice of a single predefined target speaker. The source speech can be from any speaker or accent, can include complex prosodic patterns, imperfections, and background noise, all of which are converted into a clean signal with a fixed accent and consistent pronunciation and prosody. The task is to throw away all non-linguistic information including speaker characteristics and retain only what is said, not who, how, or where it is said.

[0075] This normalization system has a variety of potential applications. Completely normalizing any voice to a single speaker with clean audio can significantly simplify ASR models, which can be reduced to support a single speaker. Removing the identity of the speaker can be useful when recording sensitive and private speech data, allowing the user to transmit only the transformed speech to a server with the "acoustic" identity removed. Reducing all accents to a single voice with a predefined accent, as opposed to acoustically disguised audio, can also mitigate bias and discrimination while maintaining a natural human voice, for example, for telephone interviews or recordings of candidate conversations provided to hiring committees. Other applications will help understand the speech content of accents that are unfamiliar to the listener, such as improving the intelligibility of heavily accented speech.

[0076] In some embodiments, voice conversion may include the use of mapping code books, neural networks, dynamic frequency warping, and Gaussian mixture models. These techniques may modify only the voice of the input speaker. In some embodiments, voice conversion may include accent conversion. The model described herein may normalize all speakers to a single voice and accent, as well as normalize prosody and use an end-to-end neural architecture that directly generates a target signal. In some embodiments, voice conversion may be a filtering and / or conversion based approach.

[0077] The end-to-end sequence-to-sequence model architecture takes as input a source speech and generates / synthesizes a target speech as output. In some embodiments, the only training requirement for this model is a parallel corpus of paired input-output speech.

[0078] like Figure 5 As shown, the network consists of an encoder and decoder with attention, and a vocoder that synthesizes time domain waveforms. The encoder converts a series of acoustic frames into hidden feature representations, which the decoder uses to predict spectrograms. In some embodiments, the core architecture of the model includes an end-to-end ASR model and / or an end-to-end TTS model based on attention.

[0079] The basic encoder configuration can be similar to other encoders discussed below with some changes. From an example input speech signal sampled at 16kHz, the encoder can extract 80-dimensional log-mel spectrogram acoustic feature frames in the range of 125-7600Hz, using a Hann window, 50ms frame length, 12.5ms frame shift, and 1024-point Short-Time-Fourier (STFT) calculation.

[0080] In this example, the input features are passed to a stack of two convolutional layers with ReLU activation, each consisting of 32 kernels, shape 3x3 in time x frequency, stride 2x2, and temporally downsampled by a total factor of 4, reducing the amount of computation for subsequent layers. Batch normalization is applied after each layer.

[0081] The resulting downsampled sequence is passed to a bidirectional convolutional LSTM (CLSTM) layer using a 1x3 filter, i.e., convolution is performed only on the frequency axis at each time step. Finally, it is passed to a stack of three bidirectional LSTM layers of size 256 in each direction, interleaved with 512-dimensional linear projections, followed by batch normalization and ReLU activation to compute the final 512-dimensional encoder representation.

[0082] In some embodiments, the decoder targets a 1025-dimensional STFT magnitude, computed using the same number of frames as the input feature 2048-point STFT.

[0083] The system uses a decoder network consisting of an autoregressive RNN to predict the output spectrogram from the encoded input sequence one frame at a time. The prediction from the previous decoder time step first passes through a small pre-network (pre-net), which includes two fully connected layers with 256 ReLU units, which may help learn attention. The pre-network output and the attention context vector can be concatenated and passed through a stack of two unidirectional LSTM layers with 1024 units. The concatenation of the LSTM output and the attention context vector is then projected through a linear transformation to produce a prediction for the target spectrogram frame. Finally, these predictions pass through a 5-layer convolutional pre-network, which predicts the residual to be added to the initial prediction. Each post-net layer has 512 filters of shape 5x 1, followed by batch normalization and tanh activation.

[0084] To synthesize an audio signal from a predicted magnitude spectrogram, the system uses the Griffin-Lim algorithm to estimate a phase consistent with the predicted magnitude, followed by an inverse STFT. In some embodiments, a neural vocoder such as WaveNet can provide improved synthesis quality. In some embodiments, WaveNet can replace Griffin-Lim.

[0085] In some embodiments, the system can be configured to generate speech rather than arbitrary audio. Jointly training the encoder network to simultaneously learn a high-level representation of the underlying language helps bias the predictions of the spectrogram decoder towards representations of the same underlying speech content. An auxiliary ASR decoder can be added to predict a (grapheme or phoneme) transcription of the output speech conditioned on the encoder latent representation. This multi-task trained encoder can be thought of as learning a latent representation of the input that maintains information about the underlying transcription, e.g., a latent representation closer to that learned in a TTS sequence-to-sequence network.

[0086] In some embodiments, the decoder input is created by concatenating the 64-dimensional embedding of the grapheme emitted at the previous time step, and the 512-dimensional attention context vector. This is passed to a 256-unit LSTM layer. Finally, the concatenation of the attention context and the LSTM output is passed to a softmax layer, which predicts the probability of each grapheme in the output vocabulary.

[0087] Speech-to-speech models can be used to convert speech from an arbitrary speaker into the voice of a predefined canonical speaker. As described above, the system can require a parallel corpus of utterances spanning a variety of speakers and recording conditions, with each utterance mapped to speech from a canonical speaker. Since it may be impractical to have a single speaker record hours of speech in a clean acoustic environment, a TTS system can be used to generate training targets from a large corpus of human or machine-transcribed speech. Essentially, this reduces the task of reproducing any input speech in the voice of a single-speaker TTS system. Using a TTS system to generate this parallel corpus has several advantages: (1) the audio is spoken by a single predefined speaker and in a standard language accent; (2) there is no background noise; (3) high-quality pronunciation is used without disfluencies; and (4) large amounts of data can be synthesized as needed to scale to large corpora.

[0088] This article describes an end-to-end speech-to-speech model that maps an input spectrogram directly to another spectrogram without any intermediate discrete representation. The model is trained to normalize any utterance from any speaker to the voice of a single predefined speaker, retaining the language content and discarding the non-language content. In some embodiments, the same model can be trained to successfully identify, separate, and reconstruct the loudest speaker in a mixture of overlapping voices, which improves ASR performance. In some cases, the model can directly translate speech in one language into speech in another language.

[0089] For tasks where the goal is to preserve speaker identity, TTS synthesis can be used to introduce variations in the target speaker identity, such as matching the original speaker and directly controlling speaking style or prosody. This technique can be used to synthesize training targets for a normalized model that preserves speaker identity but enforces a neutral prosody, or vice versa, a model that normalizes speaker identity but maintains the prosody of the input speech.

[0090] Figure 6Examples of computing device 600 and mobile computing device 650 that can be used to implement the techniques described herein are shown. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Mobile computing device 650 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components, their connections and relationships, and their functions shown here are only by way of example and are not meant to be limiting.

[0091] The computing device 600 includes a processor 602, a memory 604, a storage device 606, a high-speed interface 608 connected to the memory 604 and a plurality of high-speed expansion ports 610, and a low-speed interface 612 connected to the low-speed expansion port 610. Each of the processor 602, the memory 604, the storage device 606, the high-speed interface 608, the high-speed expansion port 610, and the low-speed interface 612 is interconnected using various buses and can be installed on a common motherboard or other suitable installation methods. The processor 602 can process instructions for execution within the computing device 600, including instructions stored in the memory 604 or on the storage device 606, to display graphical information of a GUI on an external input / output device such as a display 616 connected to the high-speed interface 608. In other embodiments, multiple processors and / or multiple buses and multiple memories and multiple memory types can be used as appropriate. In addition, multiple computing devices can be connected, each device providing a portion of the necessary operations, for example, as a server group, a blade server group, or a multi-processor system).

[0092] The memory 604 stores information within the computing device 600. In some implementations, the memory 604 is a volatile memory unit or units. In some implementations, the memory 604 is a non-volatile memory unit or units. The memory 604 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0093] Storage device 606 can provide mass storage for computing device 600. In some embodiments, storage device 606 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state storage device, or a group of devices, including devices in a storage area network or other configurations. Instructions can be stored in an information carrier. When executed by one or more processing devices (e.g., processor 602), the instructions perform one or more methods such as those described above. Instructions can also be stored in one or more storage devices, such as a computer or machine-readable medium (e.g., memory 604, storage device 606, or memory on processor 602).

[0094] The high-speed interface 608 manages bandwidth-intensive operations of the computing device 600, while the low-speed interface 612 manages less bandwidth-intensive operations. This allocation of functions is merely an example. In some embodiments, the high-speed interface 608 is connected to the memory 604, the display 616 (e.g., through a graphics processor or accelerator), and the high-speed expansion interface 610, which can accept a variety of expansion cards (not shown). In this embodiment, the low-speed interface 612 is connected to the storage device 606 and the low-speed expansion port 614. The low-speed expansion port 614, which includes various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be connected to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device, such as a switch or a router, for example, through a network adapter.

[0095] As shown, computing device 600 can be implemented in a variety of different forms. For example, it can be implemented in a standard server 620, or multiple times in a group of such servers. In addition, it can be implemented in a personal computer such as a laptop 622. It can also be implemented as part of a rack server system 624. Optionally, components from computing device 600 can be combined with other components in a mobile device (not shown), such as mobile computing device 650. Each such device can contain one or more of computing device 600 and mobile computing device 650, and the entire system can be composed of multiple computing devices that communicate with each other.

[0096] The mobile computing device 650 includes a processor 652, a memory 664, an input / output device such as a display 654, a communication interface 666, and a transceiver 668, among other components. The mobile computing device 650 may also be equipped with a storage device, such as a microdrive or other device, to provide additional storage. Each of the processor 652, the memory 664, the display 654, the communication interface 666, and the transceiver 668 are interconnected using a variety of buses, and several components may be mounted on a common motherboard or in other suitable ways.

[0097] The processor 652 can execute instructions within the mobile computing device 650, including instructions stored in the memory 664. The processor 652 can be implemented in a chipset including separate or multiple analog and digital processors. The processor 652 can provide, for example, coordination for other components of the mobile computing device 650, such as controlling a user interface, applications running on the mobile computing device 650, and wireless communications of the mobile computing device 650.

[0098] The processor 652 can communicate with the user through the control interface 658 and the display interface 656 connected to the display 654. The display 654 can be, for example, a TFT display (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 656 may include suitable circuits for driving the display 654 to present graphics and other information to the user. The control interface 658 can receive commands from the user and convert them for submission to the processor 652. In addition, the external interface 662 can provide communication with the processor 652 so as to enable the mobile computing device 650 to communicate with other devices in a near area. The external interface 662 can provide, for example, wired communication in some embodiments, or wireless communication in other embodiments, and multiple interfaces can also be used.

[0099] The memory 664 stores information within the computing device 650. The memory 664 may be implemented in one or more of a computer-readable medium or media, a volatile memory unit, or a non-volatile memory unit. An expansion memory 674 may also be provided and connected to the mobile computing device 650 via an expansion interface 672, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory 674 may provide additional storage space for the mobile computing device 650, or may also store applications or other information for the mobile computing device 650. Specifically, the expansion memory 674 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Thus, for example, the expansion memory 674 may be provided as a security module for the mobile computing device 650, and may be programmed with instructions that allow secure use of the mobile computing device 650. In addition, secure applications and additional information may be provided via a SIMM card, such as placing identification information on a SIMM card in an unbreakable manner.

[0100] As described below, the memory may include, for example, flash memory and / or NVRAM memory (non-volatile random access memory). In some embodiments, instructions are stored in an information carrier, which when executed by one or more processing devices (e.g., processor 652) performs one or more methods as described above. The instructions may also be stored in one or more storage devices, such as one or more computer or machine readable media (e.g., memory 664, storage device 674, or memory on processor 652). In some embodiments, the instructions may be received in a propagated signal, for example, via transceiver 668 or external interface 662.

[0101] The mobile computing device 650 can communicate wirelessly via a communication interface 666, which may include digital signal processing circuitry if necessary. The communication interface 666 may provide communications in various modes or protocols, such as GSM (Global System for Mobile Communications) voice calls, SMS (Short Message Service), EMS (Enhanced Message Service) or MMS messages (Multimedia Messages), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000 or GPRS (General Packet Radio Service), etc. For example, such communications may occur using radio frequencies through a transceiver 668. In addition, short-range communications may occur, such as using Bluetooth, WiFi or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 670 may provide additional navigation and location-related wireless data to the mobile computing device 650, which may be used appropriately by applications running on the mobile computing device 650.

[0102] The mobile computing device 650 may also communicate audibly using an audio codec 660, which may receive voice information from a user and convert it into usable digital information. The audio codec 660 may also generate audible sounds for the user, such as through a speaker such as in a handset of the mobile computing device 650. The sounds may include sounds from a voice phone call, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on the mobile computing device 650.

[0103] As shown, computing device 650 can be implemented in many different forms. For example, it can be implemented as a cellular phone 680. It can also be implemented as part of a smart phone 682, a personal digital assistant, or other similar mobile devices.

[0104] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which can be special purpose or general purpose, that receives instructions from a storage system, at least one input device, and at least one output device.

[0105] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural, and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term machine-readable signal refers to any signal for providing machine instructions and / or data to a programmable processor.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, a keyboard, and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form including acoustic, voice, or tactile input.

[0107] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of these back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet. In some embodiments, the systems and techniques described herein can be implemented on an embedded system, where speech recognition and other processing are performed directly on the device.

[0108] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises from computer programs running on respective computers and having a client-server relationship with each other.

[0109] Although some embodiments have been described in detail above, other modifications are possible. For example, although the client application is described as accessing a delegate, in other embodiments, the delegate can be used by other applications implemented by one or more processors, such as applications executed on one or more servers. In addition, the logic flow depicted in the figure does not require the specific order shown, or the sequential order, to achieve the desired results. In addition, other actions can be provided from the described process, or actions can be eliminated, and other components can be added to the described system, or deleted from the described system. Therefore, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method executed on data processing hardware, the computer-implemented method causing the data processing hardware to perform an operation, characterized in that The operations include: receiving a sequence of source audio frames representing utterances spoken in a first accent; processing the sequence of source audio frames using an encoder of an acoustic conversion model to generate a sequence of source internal representations representing the utterance spoken in the first accent; processing the sequence of source internal representations using a decoder of the acoustic conversion model to generate a sequence of target audio frames representing a synthesized speech representation of the utterance in a second accent different from the first accent; and The synthesized speech representation of the utterance in the second accent is provided for output by a computing device.

2. The computer-implemented method of claim 1, wherein: Processing the sequence of source internal representations to generate a synthesized speech representation of the utterance in a second accent includes processing the sequence of source internal representations to generate the synthesized speech representation without performing any speech recognition on the sequence of source audio frames.

3. The computer-implemented method of claim 1 , wherein: The sequence of source audio frames comprises a sequence of input spectrograms.

4. The computer-implemented method of claim 1, wherein: The target audio frame sequence includes an output spectrogram sequence.

5. The computer-implemented method of claim 1, wherein: The rhythm of the utterance spoken in the first accent is different from the rhythm of the synthesized speech representation of the utterance spoken in the second accent.

6. The computer-implemented method of claim 1, wherein: The encoder includes a bidirectional long short-term memory (LSTM) layer.

7. The computer-implemented method of claim 1, wherein: The decoder comprises a spectrogram decoder with attention.

8. The computer-implemented method of claim 1, wherein: The operations also include: receiving audio data of a collection of utterances; obtaining a transcription of each utterance in the set of utterances; providing a transcription of each utterance as input to a text-to-speech model; For each transcription of each utterance, receiving audio data for a set of additional utterances in a synthesized voice; and The model is trained using the audio data for the set of utterances and the audio data for the additional set of utterances in synthesized voice.

9. The computer-implemented method of claim 1, wherein: The operations also include bypassing obtaining a transcription of the utterance.

10. The computer-implemented method of claim 1, wherein: The speech conversion model is configured to adjust a time period between each term in the utterance spoken in the first accent.

11. A system, characterized in that: include: Data processing hardware; and Memory hardware that communicates with the data processing hardware and stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, including: receiving a sequence of source audio frames representing utterances spoken in a first accent; processing the sequence of source audio frames using an encoder of an acoustic conversion model to generate a sequence of source internal representations representing the utterance spoken in the first accent; processing the sequence of source internal representations using a decoder of the acoustic conversion model to generate a sequence of target audio frames representing a synthesized speech representation of the utterance in a second accent different from the first accent; and The synthesized speech representation of the utterance in the second accent is provided for output by a computing device.

12. The system according to claim 11, characterized in that Processing the sequence of source internal representations to generate a synthesized speech representation of the utterance in a second accent includes processing the sequence of source internal representations to generate the synthesized speech representation without performing any speech recognition on the sequence of source audio frames.

13. The system according to claim 11, characterized in that The sequence of source audio frames comprises a sequence of input spectrograms.

14. The system according to claim 11, characterized in that The target audio frame sequence includes an output spectrogram sequence.

15. The system according to claim 11, characterized in that The rhythm of the utterance spoken in the first accent is different from the rhythm of the synthesized speech representation of the utterance spoken in the second accent.

16. The system according to claim 11, characterized in that The encoder includes a bidirectional long short-term memory (LSTM) layer.

17. The system according to claim 11, characterized in that The decoder comprises a spectrogram decoder with attention.

18. The system according to claim 11, characterized in that The operations also include: receiving audio data of a collection of utterances; obtaining a transcription of each utterance in the set of utterances; providing a transcription of each utterance as input to a text-to-speech model; For each transcription of each utterance, receiving audio data for a set of additional utterances in a synthesized voice; and The model is trained using the audio data for the set of utterances and the audio data for the additional set of utterances in synthesized voice.

19. The system according to claim 11, characterized in that The operations also include bypassing obtaining a transcription of the utterance.

20. The system according to claim 11, characterized in that The speech conversion model is configured to adjust a time period between each term in the utterance spoken in the first accent.

Images