Synthetic Speech Processing

Abstract

A speech processing system receives both text data and natural understanding data (e.g., domain, intent, and / or entity) related to a command represented in the text data. The system uses the natural understanding data to modify acoustic characteristics when determining spectrogram data corresponding to the text data based on the natural understanding data.

Description

[0001] Cross-references to related application data

[0002] This application claims priority to U.S. patent application No. 16 / 818,542, filed on March 13, 2020, in the name of Antonio Bonafonte et al. and entitled “SYNTHETIC SPEECH PROCESSING.” Background Art

[0003] The speech processing system may include a dialog manager that processes input data corresponding to a command from a user and determines output data corresponding to a response to the command. The command may initially be represented by audio data; an automatic speech recognition component may process the audio data to determine corresponding text data, and a natural language understanding component may process the text data to determine one or more of a domain of the command, an intent corresponding to the command, and / or an entity corresponding to the command. The speech processing system may also include a text-to-speech component for generating audio representing the response to the command. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] For a more complete understanding of the present disclosure, reference is now made to the following description taken in conjunction with the accompanying drawings.

[0005] Figure 1 A speech processing method according to an embodiment of the present disclosure is shown.

[0006] Figure 2A Components of a user device and a remote system for performing speech processing according to an embodiment of the present disclosure are shown.

[0007] Figure 2B Components of a user device for performing speech processing according to an embodiment of the present disclosure are shown.

[0008] Figure 3A and Figure 3B Components for generating output data given input data according to embodiments of the present disclosure are shown.

[0009] Figure 4 Components for rewriting input text and / or changing the acoustic characteristics of output audio are shown according to embodiments of the present disclosure.

[0010] Figure 5 Examples of NLU and conversation data according to embodiments of the present disclosure are shown.

[0011] Figure 6 Components for rewriting text data according to an embodiment of the present disclosure are shown.

[0012] Figure 7 Components for changing the sound characteristics of output audio data according to an embodiment of the present disclosure are shown.

[0013] Figure 8 Components for training a TTS system according to an embodiment of the present disclosure are shown.

[0014] Figure 9A and Figure 9B A speech encoder and a voice decoder according to an embodiment of the present disclosure are shown.

[0015] Figure 10 A natural network for speech processing according to an embodiment of the present disclosure is shown.

[0016] Figure 11 Components of a user device for performing speech processing according to an embodiment of the present disclosure are shown.

[0017] Figure 12 Components of a remote system for performing speech processing according to an embodiment of the present disclosure are shown.

[0018] Figure 13 A networked computing environment according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0019] The speech processing system may include multiple components, each of which may be a trained neural network model for performing various functions associated with speech processing. Together, these components may enable the speech processing system to receive, interpret, and respond to commands from a user. An automatic speech recognition (ASR) component (which may be or include one or more first-trained models) may process audio data (and / or other input data) corresponding to a command and determine corresponding output data, which may be text data, audio data, and / or video data. A natural language understanding (NLU) component (which may be or include one or more second-trained models) may process the output data to determine the domain, intent, and / or slot corresponding to the command. A dialog manager component (which may be or include one or more third-trained models) may process the output data and / or the output of the NLU component to determine output data corresponding to a response to the command. Finally, a text-to-speech (TTS) component (which may be or include one or more fourth-trained models) may process the output data to determine the corresponding audio data. In some embodiments, the ASR, NLU, dialog manager, and / or TTS components may be combined into a single model.

[0020] The TTS component can modify aspects of the audio data based on the output of the NLU component and / or the dialog manager component. These modified aspects may include changes in the wording of the words represented in the audio data and / or changes in vocal characteristics, such as intonation, speech rate, emphasis, and / or accent. These vocal characteristics may also be referred to as prosody. Variations in word phrasing and / or vocal characteristics can improve the user experience by more closely resembling human speech. For example, if a person is asked the same question more than once, the response to the question may vary. For example, if the question is "What time is it now?" one response might be "It's 12 noon," while another might be "It's noon." Humans may similarly alter the vocal characteristics of their speech or emphasize specific words in different contexts. For example, a response might be neutral when reporting the weather, but excited when reporting an unexpected event. Similarly, certain words in a response, such as the name of a famous band, may be pronounced more strongly than others, such as in the response "TOTO is playing next week!"

[0021] Therefore, the present disclosure relates to systems and methods for generating synthesized speech given input text data and natural language understanding (NU) data, such that the synthesized speech includes variations based at least in part on the NU data. Consequently, these variations in the synthesized speech can more closely resemble naturally occurring variations in human speech. In various embodiments, a speech encoder processes the input text data to determine first encoded data representing the input, such as an utterance. A second encoder can be trained to similarly process the input text data and determine second encoded data representing natural language understanding information related to the utterance. This natural language understanding information can also be referred to as the context of the utterance. This second encoded data can be combined with NLU embedding data, which represents encoded information determined by an NLU component that processes the input text data. The second encoded data can be combined with the NLU embedding data and then used to condition an attention network. The attention network can then process the first encoded data determined by the speech encoder into determined weighted encoded data, which can then be processed by a speech decoder to determine audio data, such as spectrogram data. For example, if the NLU embedding data indicates that an entity is represented in the input text data, the attention network can weight the output of the speech encoder so that the audio data includes a representation of the emphasized entity.

[0022] refer to Figure 1, user 10 utters speech represented by input audio 12. User device 110 receives input audio 12 and converts it (using, for example, a microphone) into corresponding audio data. As explained in further detail below, user device 110 may perform additional speech processing and / or may send the audio data to a remote system 120 via network 199 for further audio processing. Regardless of whether the audio data is processed by user device 110 and / or remote system 120, an ASR component may process the audio data to determine corresponding text data, and an NLU component may process the text data to determine NLU data, such as domains, intents, and / or entities associated with the text data.

[0023] The user device 110 and / or the remote system 120 may then use a Figure 7 The language encoder 702 shown processes (130) input data, such as input text data, using a language encoder. As explained in more detail with reference to the accompanying drawings, the language encoder 702 can be one or more neural networks, such as a feed-forward neural network (FFNN) and / or a recurrent neural network. The language encoder 702 can therefore include one or more nodes connected to one or more other nodes; the nodes can be arranged in layers such that the output of one layer is processed by a second layer. The nodes can each be associated with a weight that multiplies the input by a first value and / or an offset that adds a second value to the input. The output of the language encoder 702 can be first encoded data consisting of a first vector of multiple encoded values; the first vector can uniquely identify the words of the input text data as a single set of values.

[0024] The user device 110 and / or the remote system 120 may then use a second encoder such as Figure 7 The second encoder 720 processes (132) the input text data. Like the speech encoder 702, the second encoder 720 can determine second encoded data consisting of a second vector of values. However, unlike the output of the speech encoder 720, the second vector can define variations in the synthesized speech corresponding to the natural understanding data, such as which words should be emphasized.

[0025] The user device 110 and / or the remote system 120 may then process (134) the first encoded data using the second encoded data and the natural understanding data (determined by the NLU component). For example, an attention network, such as Figure 7The attention network 704 can determine one or more weights of one or more patterns based at least in part on the second encoded data and / or the natural understanding data to determine third encoded data by processing the first encoded data using the one or more weights. As explained in more detail below, the attention network 704 can be a neural network of nodes arranged in one or more layers; each node of the attention network 704 can be connected not only to the output layer of the language encoder 702, but also to one or more nodes in one or more other layers of the language encoder 702. Each connection between the attention network 704 and the language encoder 702 can be associated with a weight. By changing at least some of the weights of the attention network 704 according to the first encoded data, the second encoded data, and the natural understanding data, the attention network 704 causes changes in the synthesized speech represented in the output audio to change according to the natural understanding data.

[0026] The user device 110 and / or the remote system 120 may then process 136 the third encoded data to determine audio data representing synthesized speech corresponding to a response to the command. The decoder may be, for example, Figure 7 The decoder 706 of FIG. 702 and, like the encoder 702 , can be a neural network, such as an FFNN or an RNN. The audio data can be a digital representation of an audio waveform, a spectrogram such as a mel-spectrogram, or any other such form of audio data. If the audio data is a spectrogram, it can be further processed by another component (such as a vocoder) to convert the spectrogram into a corresponding audio waveform. The data corresponding to the audio waveform can be determined by the user device 110 or can be sent to the user device to be output as output audio 14.

[0027] refer to Figure 2A , the speech processing system can be distributed across one or more user devices 110 and one or more remote systems 120 (such as a server or local hub). The user device 110 can capture audio including speech and then either process the audio itself and / or transmit audio data representing the audio to the remote system for further processing. The remote system 120 may have access to greater computing resources, such as more and / or faster computer processors than the user device, and therefore may be able to process the audio data and determine an appropriate response faster than the user device. The user device may have, for example, a wake-up word determination component that detects the presence of a wake-up word in the audio and transmits the corresponding audio data to the remote system only when the wake-up word is detected. As used herein, a “wake-up word” is one or more specific words that a user of a user device can say to cause the user device to start processing audio data, such as “Alexa”, which may also include command representations, such as “turn on the lights”.

[0028] Also refer to Figure 2B, the speech processing system can be entirely located on the user device 110. In other embodiments, some additional components, such as the ASR component, are located on the user device 110, while other components, such as the speech generation component 280, are located on the remote system 120. Therefore, any distribution of components of the speech processing system of the present disclosure is within its scope. Therefore, the following discussion relates to Figure 2A and Figure 2B The distribution of components and similar distributions are also involved.

[0029] The user device 110 and / or the remote system 120 may therefore include an automatic speech recognition (ASR) component that processes the audio data to determine corresponding text data and a natural language understanding (NLU) component that processes the text data to determine the user's intent expressed in the text data and, thereby, to determine an appropriate response to the intent. The remote system 120 may determine data representing a response and send it to the user device 110 (or other device), which may then output the response. In other embodiments, the user of the user device 110 may wish to transmit audio data for reasons other than ASR / NLU processing, such as one-way or two-way audio communication with one or more other parties or remote systems.

[0030] Before processing the audio data, the device 110 can use various techniques to first determine whether the audio data includes a representation of the speech of the user 10. For example, the user device 110 can use a voice-activity detection (VAD) component 202 to determine whether the audio data represents speech based on various quantitative aspects of the audio data, such as the spectral slope between one or more frames of the audio data, the energy level of the audio data in one or more spectral bands, the signal-to-noise ratio of the audio data in one or more spectral bands, and / or other quantitative aspects. In other examples, the VAD component 202 can be a trained classifier that is configured to distinguish speech from background noise. The classifier can be a linear classifier, a support vector machine, and / or a decision tree. In other examples, a hidden Markov model (HMM) and / or a Gaussian mixture model (GMM) technique can be applied to compare the audio data with one or more acoustic models in a speech storage device, which can include models corresponding to speech, noise (e.g., ambient noise or background noise), and / or silence.

[0031] The user device 110 may alternatively or additionally determine that the audio data represents an utterance by using a wake word detection component 204. If the VAD component 202 is being used and it determines that the audio data includes speech, the wake word detection component 204 may then only be activated to process the audio data to determine whether a wake word may be represented therein. In other embodiments, the wake word detection component 204 may continuously process the audio data (e.g., in a system that does not include the VAD component 202). The user device 110 may also include an ASR component for determining text data corresponding to the speech represented in the input audio 12, and may send the text data to the remote system 120.

[0032] The training model of the VAD component 202 and / or the wake-up word detection component 204 can be a CNN, an RNN, an acoustic model, a hidden Markov model (HMM) and / or a classifier. These trained models can apply a general large vocabulary continuous speech recognition (LVCSR) system to decode the audio signal, wherein the wake-up word search is performed in the generated lattice and / or confusion network. Another wake-up word detection method constructs an HMM for each key wake-up word term and non-wake-up word speech signal respectively. Non-wake-up word speech includes other spoken words, background noise, etc. One or more HMMs can be constructed to model the non-wake-up word speech characteristics, which can be called a filling model. Viterbi decoding can be used to search for the best path in the decoding graph, and the decoding output is further processed to make a decision about the presence of the wake-up word. This method can be extended to include identification information by incorporating a hybrid DNN-HMM decoding framework. In another example, the wake-up word detection component 204 can directly use a convolutional neural network (CNN) / recurrent neural network (RNN) structure without using an HMM. The wake-up word detection component 204 can estimate the wake-up word posterior with contextual information by stacking frames within the context window of a DNN or using an RNN. Subsequent posterior threshold adjustment and / or smoothing can be applied to the decision. Other techniques for wake-up word detection can also be used.

[0033] After the user device 110 detects the wake word and / or speech, the remote system 120 may be used for additional audio processing, possibly beginning to process the audio data using ASR and / or NLU, and / or sending corresponding audio data 212. In some cases, the remote system 120 may receive the audio data 212 from the user device 110 (and / or other devices or systems) and perform speech processing thereon. Figure 2A or Figure 2B Each component shown in can therefore be located on the user device 110 or the remote system 120. The remote system 120 can be located at a different location than the user device 110 (e.g., a cloud server) and / or can be located at the same location as the user device 110 (e.g., a local hub server).

[0034] The audio data 212 can be sent to, for example, a coordinator component 230 of a remote system 120. The coordinator component 230 may include memory and logic that enable the coordinator component 230 to transmit various data segments and various forms of data to various components of the system 120. For example, the ASR component 250 may first transcribe the audio data into text data that represents one or more hypotheses corresponding to the speech represented in the audio data 212. The ASR component 250 may transcribe the speech based on the similarity between the speech in the audio data and a pre-established language model. For example, the ASR component 250 may compare the audio data with a model of sound (which may include, for example, subword units, such as phonemes) and the sound sequence represented in the audio data to identify words that match the sound sequence spoken in the speech. These models may include, for example, one or more finite state transducers (FSTs). The FST may include multiple nodes connected by paths. The ASR component 250 may select the first node of the FST based on the similarity between the first node and the first subword unit of the audio data. The ASR component 250 may thereafter transition to a second node and subsequent nodes of the FST based on similarities between subsequent sub-word units and based on the likelihood that the second sub-word unit follows the first sub-word unit.

[0035] After determining the text data, the ASR component 250 may send (either directly and / or via the coordinator component 230) the text data to the corresponding NLU component 260. The text data output by the ASR component 250 may include the highest-scoring hypothesis and / or may include an N-best list (e.g., a list of ranked possible interpretations of the text data representing the audio data) containing multiple hypotheses. The N-best list may also include a score associated with each hypothesis represented therein. Each score may indicate the confidence level of the ASR processing performed to generate the hypothesis associated with the score.

[0036] NLU component 260 can process text data to determine semantic interpretations of words represented in the text data. That is, NLU component 260 determines one or more meanings associated with words represented in the text data based on the individual words represented in the text data. Meanings can include domains, intents, and one or more entities. As these terms are used herein, a domain represents a general category associated with a command, such as "music" or "weather." An intent represents a type of command, such as "play a song" or "tell me tomorrow's forecast." An entity represents a specific person, place, or thing associated with a command, such as "Toto" or "Boston." However, the present disclosure is not limited to these categories associated with meanings (generally referred to herein as "natural understanding data," which can include data determined by NLU component 260 and / or dialog manager component).

[0037] NLU component 260 can determine the intent represented by the text data (e.g., an action that the user desires user device 110 and / or remote system 120 to perform) and the relevant information segments in the text data that allow a device (e.g., device 110, system 120, etc.) to perform the intent. For example, if the text data corresponds to "play Africa by Toto," NLU component 260 can determine that the user wants the system to output the song "Africa" ​​by the band Toto, which NLU component 260 determines is represented by the "play music" intent. NLU component 260 can also process speaker identifier 214 to determine the intent and / or output. For example, if the text data corresponds to "play my favorite Toto song," and if the identifier corresponds to "speaker A," NLU component can determine that speaker A's favorite Toto song is "Africa."

[0038] The coordinator component 230 can send the NLU data to the dialog component 270; in some embodiments, the dialog component 270 is a voice applet component 290. The dialog component 270 uses the NLU data to determine output text data. The system may include more than one dialog component 270; each dialog component 270 can be associated with one or more different domains, intents, and / or entities. For example, if the NLU data includes intent data corresponding to the "play music" intent and entity data corresponding to "artist: Toto," the coordinator 230 can send the NLU data to the music-related dialog component 270, which can determine a response including Toto music audio data for output by the user device 110.

[0039] Dialogue component 270 can be a trained model that processes NLU data (and, in some embodiments, input text data) and determines output text data representing a response to a command represented in the input text data. In some embodiments, the trained model of dialogue component 270 can be a sequence-to-sequence ("seq2seq") model that includes an encoder and a decoder (in addition to other possible components such as an attention network). The encoder can process the input text data and / or NLU data word by word to determine a vector representing the entire input text data and / or NLU data. The encoder can include one or more nodes with recurrent connections (as described herein); for example, these nodes can be long short-term memory (LSTM) nodes. The encoder can also process text data and / or NLU data in a first direction (e.g., from beginning to end) and a second direction (e.g., from beginning to end); such an encoder can be referred to as a bidirectional encoder. The decoder can then process the vector determined by the encoder (and / or one or more outputs of the attention network, in some embodiments) to determine a response to the input text data. The response can be a sequence of words represented by the output text data. Dialog component 270 can be trained using training data; for example, the training data can be text data representing a set of commands and corresponding responses.

[0040] The voice applet 290 may be software such as an application. That is, the voice applet 290 may enable the device 110 and / or system 120 to perform a specific function to provide data and / or generate some other output requested by the user 10. The device 110 and / or system 120 may be configured with more than one voice applet 290. For example, the weather voice applet 290 may enable the device 110 and / or system 120 to provide weather information, the ridesharing voice applet may enable the device 110 and / or system 120 to book a ride with a taxi and / or ridesharing service, and the food order voice applet may enable the device 110 and / or system 120 to order a pizza with a restaurant's online ordering system.

[0041] In some cases, the speech applet 290 may provide output text data in response to the received NLU result data. The device 110 and / or system 120 may include a synthesized speech generation component 280 that generates output audio data from input text data and / or input audio data. The synthesized speech generation component 280 may use one of a variety of speech synthesis techniques. In one method of synthesis, known as unit selection, the synthesized speech generation component 280 analyzes the text data against a database of recorded speech. The synthesized speech generation component 280 may then select recorded speech units that match the text data and concatenate these units to form the output audio data. In another method of synthesis, known as parametric synthesis, the synthesized speech generation component 280 varies parameters such as frequency, volume, and noise to create output audio data comprising an artificial speech waveform. Parametric synthesis utilizes a computerized speech generator (sometimes also referred to as a vocoder). In another speech synthesis method, a trained model, which may be a sequence-to-sequence model, generates output audio data directly based on the input text data.

[0042] Device 110 and / or system 120 may include a speaker identification component 295. Speaker identification component 295 may determine a score indicating whether audio data 212 originates from a particular user or speaker. For example, a first score may indicate the likelihood that audio data 212 is associated with a first synthesized speech, and a second score may indicate the likelihood that the speech is associated with a second synthesized speech. Speaker identification component 295 may also determine an overall confidence level regarding the accuracy of the speaker identification operation. Speaker identification component 295 may perform speaker identification by comparing audio data 212 with stored user audio characteristics of other synthesized speech. The output of speaker identification component 295 may be used to inform NLU processing and processing performed by speech applet 290.

[0043] System 120 may include a profile storage 275. Profile storage 275 may include various information related to individual users and / or groups of users interacting with device 110. Profile storage 275 may similarly include information related to individual speakers and / or groups of speakers that are not necessarily associated with a user account. Profile storage 275 of user device 110 may include user information, while profile storage 275 of remote system 120 may include speaker information.

[0044] Profile storage 275 may include one or more profiles. Each profile may be associated with a different user and / or speaker. Profiles may be specific to a user or speaker and / or a group of users or speakers. For example, a profile may be a "family" profile that includes profiles associated with multiple users or speakers in a single family. A profile may include preferences shared by all of its included profiles. Each profile included in a single profile may include preferences specific to the user or speaker associated with it. In other words, each profile may include unique preferences from one or more user profiles included in the same user profile. A profile may be a standalone profile and / or may be included within another user profile. As shown, profile storage 275 is implemented as part of remote system 120. However, profile storage 275 may be located on user device 110 and / or in a different system and / or system 120 that communicates with user device 110, for example, via network 199. Profile data may be used to inform NLU processing, dialog manager processing, and / or speech processing.

[0045] Each profile may include information indicating various devices, the output capabilities of each of the various devices, and / or the location of each of the various devices 110. This device profile data represents a device-specific profile. For example, the device profile data may represent various profiles associated with a device 110, voice processing performed on audio data received from the device 110, instances when the device 110 detected a wake word, etc. In contrast, user or speaker profile data represents a user- or speaker-specific profile.

[0046] Figure 3A and Figure 3B Components of a system for generating output audio given input data according to an embodiment of the present disclosure are shown. Figure 3A As described above, the system can introduce changes in the output audio data 214, which changes can include changes in the wording of words and / or changes in sound characteristics to more closely resemble human speech. As described above, the ASR component 250 processes the input audio data 212 to determine the input text data 302. Also, as described above, the input text data 302 can be data representing words corresponding to the input audio data 212 and / or can be an N-best list of possible words corresponding to the input audio data 212. The input text data 302 can alternatively or additionally include sub-word units corresponding to the input audio data 212, such as phonemes.

[0047] Although Figure 3A214 (e.g., one "turn" of a conversation), but during an interaction with a user 10 involving multiple turns of conversation (a "multi-turn" conversation), more input data items may be received and more output data items may be generated. Thus, when generating output audio data 214, the synthesized speech recognition generation component 280 may use data from previous conversation turns in addition to the NLU data 304 and / or conversation data 308. In some embodiments, the conversation data 308 includes information about previous turns of conversation. For example, if the input audio data 212 indicates that the user 10 did not hear or understand a word represented in a previous output audio data 214, the synthesized speech generation component 280 may generate additional output audio data 214 that includes an indication that the word was spoken louder or slower.

[0048] refer to Figure 3B In other embodiments of the present disclosure, other input data 312 in addition to input audio data 212 may be processed. Input data 312 may be, for example, text data; a user 10 of user device 110 may use an input device of user device 110 (such as a touch screen and / or a keyboard) to input text data. Input data 312 may alternatively or additionally be video data; video data may include, for example, a representation of the user's lips while speaking, a representation of the user's gestures (such as sign language), or other such data. The present disclosure is not limited to any type of input data 312.

[0049] If the input data 312 includes text data, the NLU component 260 can process the input data 312 without further processing. If the input data 312 includes other types of data (such as video data), a video processing component (such as an image recognition component) can process the input data 312 to determine corresponding text data, which can then be processed by the NLU component 260.

[0050] Similar to how input data 312 is not limited to only audio data, output data 314 is also not limited to only audio data or only one type of data. The output data can be, for example, text data; as described herein, this text data can be a rewritten version of output text data 306. Output generation component 380 can also process output text data 306 to determine video data as output data 314. Output data 314 can also be or include other data, such as video data; this video data can include, for example, a face that can express different facial expressions. Output data 314 can be a combination of data types, such as a video of a face moving its lips and corresponding audio data representing speech.

[0051] Return to Figure 3A, the NLU component 260 determines NLU data 304, which, as described above, can include domains, intents, and / or entities corresponding to the input text data 302. Also as described above, the dialog manager component 270 processes the input text data 302 and / or the NLU data 304 to determine output text data 306, which represents a response to the command corresponding to the input text data 312. The dialog manager component 270 can also determine dialog data 308, which, like the NLU data 304, can include domains, intents, and / or entities that differ from the entities in the NLU data 304. For example, the output text data 306 can reference new entities that do not appear in the NLU data 304 and / or can change the domain determined by the NLU component 260.

[0052] As explained in greater detail below, the synthesized speech generation component 280 processes the output text data 306, the NLU data 304, and / or the dialog data 308 to determine the output audio data 214. As described herein, the output audio data 214 can include different words or phrases than the output text data 306 and / or can represent changes in vocal characteristics based at least in part on the NLU data 304 and / or the dialog data 308.

[0053] Figure 4 Components of a system for rewriting input text and / or changing the acoustic characteristics of output audio according to an embodiment of the present disclosure are shown. The synthesized speech generation component 280 may include a rewriting component 402 (see Figure 6 ) and / or TTS component 406 (see Figure 7 In some embodiments, the synthesized speech generation component 280 includes only the TTS component 406 and does not perform rewriting. In other embodiments, the rewriting component 402 generates the rewritten text data 404, but the TTS component 406 does not introduce changes in the sound characteristics in the output audio data 214.

[0054] In some embodiments, rewrite component 402 and TTS component 406 can communicate to determine whether output audio data 214 should include one or more changes due to rewriting, one or more changes due to the change of sound characteristics, or both. For example, synthetic speech generation component 280 can for example determine the score of each type of change corresponding to the degree of change. For example, the score of rewriting can be determined by output text data 306 compared with the different word quantity of the text data 404 of rewriting. The score of sound characteristics change can be determined by determining the maximum loudness of output audio data 310, the maximum or minimum speech rate of output audio data 310 or by measuring other such attributes of output audio data 310. If any score is higher than corresponding threshold value, or if the sum of scores is higher than another threshold value, then synthetic speech generation component 280 can determine to only include the change from rewriting or only include the change of sound characteristics in output audio data 310 in output audio data 310. Synthetic speech generation component 280 can alternatively or additionally cause any one of rewriting component 402 and / or TTS component 406 to reduce the change they determine separately.

[0055] Figure 5An example of NLU and conversation data according to an embodiment of the present disclosure is shown. NU data 502 may include NLU data 302 and / or conversation data 308. As described herein, NLU data 302 may include: domain data 502a, which may include representations of one or more domains; intent data 504a, which may include representations of one or more intents; and / or entity data 506a, which may include representations of one or more entities. Conversation data 308 may include: domain data 502b, which may include representations of one or more domains; intent data 504b, which may include representations of one or more intents; and / or entity data 506b, which may include representations of one or more entities. For example, if input audio data 212 includes a representation of the phrase "Alexa, play Africa by Toto," NLU component 260 may determine that domain data 502a corresponds to "music," intent data 504b corresponds to "play music," and entity data 506a corresponds to "band: Toto" and "song: Africa." If conversation manager component 270 determines that output text data 306 includes a representation of the phrase "got it." Do you want to listen to 'Rosanna' again? The dialog manager component 270 may also determine that the entity data 506b corresponds to "Band: Toto" and "Song: Rosanna." The dialog data 308 may also include some or all of the NLU data 304; in these embodiments, the rewriting component 402 and / or the TTS component 406 may process the text data 306 / 404 and the dialog data 308, but not the NLU data 304. The NU data 502 may also include additional data, such as data representing the user's emotional state.

[0056] Figure 6 Components of a rewriting component 402 for rewriting text data according to an embodiment of the present disclosure are shown. The rewriting component 402 can include a seq2seq model including an encoder 602 and a decoder 606. One particular implementation of the rewriting component 402 is shown; however, the present disclosure is not limited to this implementation.

[0057] The encoder 602 processes the output text data 306 (output by the dialog manager component 270) to determine the encoded data. As with other encoders discussed herein, the encoder 602 can be a trained neural network model consisting of one or more nodes arranged in one or more layers.

[0058] The encoded data can then be processed by a max pooling component 604. The max pooling component 604 can be a mathematical function that reduces the size of one or more dimensions of the encoded data output by the encoder 602. The max pooling component 604 can divide each dimension into multiple regions and, for each region, find the maximum value of all values ​​in that region. The max pooling component 604 can then output these maximum values ​​and their corresponding regions.

[0059] The decoder 606 receives as input the output of the max pooling component 604. Like the encoder 602, the decoder 606 can be a trained neural network model consisting of one or more nodes arranged in one or more layers, as described with respect to the other decoders described herein.

[0060] The decoder 606 can be autoregressive, meaning that each output it generates is fed back as its input until the generation of a special "end of sentence" output marks the end of the sentence represented by the rewritten text data 404. In order to condition the rewriting component 402 on the desired properties that the decoded sentence should have, the decoder 606 can also receive as input the NU data 502, which can be, for example, an encoded version of the NU data 502 determined by the encoder. The summing / averaging component 612 can first process the NU data by summing and / or averaging some or all values ​​(or groups of values) of the NU data 502. In some embodiments, the values ​​corresponding to one or more intents 504a, 504b are summed or averaged with the values ​​corresponding to one or more entities 506a, 506b. The decoder 606 can process the output of the summing / averaging component 612 as a first step before processing the outputs of the encoder 602 and the max pooling component 604.

[0061] The output of the decoder 606 can be processed by an attention network 608. The attention network 608 can be an RNN, a DNN, and / or other network discussed herein, and can include nodes with weights and / or cost functions arranged into one or more layers. The attention probability can be calculated after projecting the input into (for example) a 128-dimensional hidden representation. In some embodiments, the attention network 608 weights certain values ​​of the output of the decoder 606 before sending them to the projection component 610. For example, the attention network 608 can weight certain parts of the context vector by increasing the values ​​of certain parts of the context vector and can weight other parts by decreasing the values ​​of other parts of the context vector. The increased values ​​can correspond to values ​​that the projection component 610 should pay more attention to, and the decreased values ​​can correspond to value features that the projection component 610 should pay less attention to. The attention network 608 can thus allow the projection component 610 to "pay attention" to different parts of the output of the decoder 606 at each step of output generation. The attention network 608 can thus allow the decoder 606 and / or the linear projection component 610 to learn what to pay attention to.

[0062] The linear projection component 610 can perform a projection on the output of the attention network 608. The projection can be a linear transformation from the vector space to itself. P , making P 2 = P。 The output of the linear projection component 610 can be combined with the output of one or more fully connected layers 614; the combination can be processed using a softmax component 616 to determine the rewritten text data 404. The softmax component 616 can perform a normalization function that adjusts its input so that while each value in the output maintains the same ratio as the corresponding value of its input, all values ​​of the output are between two values ​​(such as 0 and 1), and the sum of all output values ​​is the larger of the two ranges, such as 1. The fully connected layer 614 can determine its output by processing the output of the summation / average component 612.

[0063] Figure 7 Components of a system for changing the acoustic characteristics of output audio data according to an embodiment of the present disclosure are shown. A speech encoder 702 processes the output text data 306, an attention network 704 determines weighted encoding data by processing the output of the encoder 702, and a speech decoder 706 determines audio data (such as spectrogram data 708) by processing the output of the speech encoder 702. These components will be referred to as Figure 9A and Figure 9B Describe in more detail.

[0064] The second encoder 720 also processes the output text data 306. The second encoder 720 can be similar in architecture to the language encoder 702, but can have a different number and / or type of nodes and layers. The second encoder 720 is trained to determine information related to natural understanding of the output text data 306, such as vectors representing domains, intents, and / or values ​​corresponding to entities in the output text data 306.

[0065] One or more recurrent layers 712 process the output of the second encoder 720. Although depicted separately, the recurrent layer 712 can be entirely or partially within the second encoder 720. In these embodiments, the recurrent layer 712 can process the intermediate output of the second encoder 720. The recurrent layer 712 can include recurrent elements such as a gated recurrent unit (GRU) and / or a long short-term memory (LSTM). Because the recurrent layer 712 has a recurrent connection property, the recurrent layer can determine the output corresponding to the entire output text data 306.

[0066] Then, one or more feed-forward layers 714 can receive the output of the recurrent layer 712 and the NU data 502. The feed-forward layer 714 can be or include, for example, a convolutional neural network (CNN). Thus, when the speech encoder 702 processes the output text data 306, the output of the feed-forward layer 714 can be used to adjust the attention network 704. That is, when the speech encoder 702 processes data corresponding to one or more particular words of the output text data, the feed-forward layer 714 can instruct the attention network 704 to adjust one or more of its weights, thereby causing corresponding changes in one or more sound characteristics represented in the spectrogram data 708.

[0067] The attention network 704 may further receive style data 710 corresponding to one or more general speech styles. Example styles include "newscaster," "English accent," or "silly voice." The style data 710 may be a one-hot vector; each value of the vector may correspond to a particular speech style, and only one value in the vector holds a certain value (e.g., 1), while all other values ​​hold some other value (e.g., 0). Thus, the attention network 704 may use the style data 710 to focus on portions of the speech encoder 702 that correspond to the selected style, while reducing the weights associated with other styles. The style data 710 may be determined by associating one or more of a domain, intent, or entity with a particular style and then indicating that style when determining the particular domain, intent, or entity.

[0068] Figure 8Components for training a TTS system according to an embodiment of the present disclosure are shown. Output text data 306, style data 710, language encoder 702, attention network 704, speech decoder 706, and spectrogram data 708 correspond to reference Figure 7 However, in Figure 8 In [ 306 ], a reference encoder 810 processes reference spectrogram data 808, which may be a set of training data. The training data may also include annotations describing the acoustic characteristics of the reference spectrogram data 808. The reference encoder 810 may be a variational autoencoder (VAE), meaning it is trained to produce one or more data distributions rather than exact results. By producing these distributions, the reference encoder 810 can produce a more complete dataset than would otherwise be possible, especially when training data is limited. In other words, the reference encoder 810 intelligently "fills in" the gaps between training data items during training, thereby estimating additional training data. The output of the reference encoder 810 is therefore parameters defining one or more distributions. In some embodiments, these parameters may include μ, which represents the mean of the distribution, and σ, which represents the variance of the distribution. The sampling component 812 can then use the parameters to determine a posterior z, which can then be used to train the language encoder 702, the attention network 704, and / or the speech decoder 706 as they process the output text data 306.

[0069] Figure 9A One embodiment of a speech encoder 702 is shown; however, the present disclosure is not limited to any particular embodiment of the encoder 702. Other encoders described herein, such as Figure 6 The encoder 602 and / or Figure 7 The encoder 720 may have a similar architecture. The language encoder 702 may receive input data, such as text data 306, and the character embedding component 902 may create character embeddings based on the input data. The character embedding can represent the text data 306 as a defined list of characters, which may include, for example, English characters (e.g., az and AZ), numbers, punctuation marks, special characters, and / or unknown characters. The character embedding can use, for example, one-hot encoding to convert the character list into one or more corresponding vectors. The vectors can be multi-dimensional; in some embodiments, the vectors represent learned 512-dimensional character embeddings.

[0070] The character embeddings may be processed by one or more convolutional layers 904, which may apply one or more convolution operations to the vectors corresponding to the character embeddings. In some embodiments, the convolutional layers 904 correspond to three convolutional layers, each of which includes 512 layers with 5 ×1-shaped filters, i.e., each filter spans five characters. The convolutional layer 904 can model long-term context (e.g., N-grams) in character embeddings. The final output of the convolutional layer 904 (i.e., the output of the only or final convolutional layer) can be passed to the bidirectional LSTM layer 906 to generate output data. In some embodiments, the bidirectional LSTM layer 906 includes 512 units: 256 in the first direction and 256 in the second direction.

[0071] Figure 9B One embodiment of one or more speech decoders 706 is shown; however, the present disclosure is not limited to any particular embodiment of decoder 706. Other encoders described herein, such as Figure 6 The encoder 606 can have a similar architecture. The speech decoder 706 can be a network, such as a neural network; in some embodiments, the decoder is an autoregressive recurrent neural network (RNN). The speech decoder 706 can generate the encoded features 908 from the participating encoded features 920 one frame at a time. The participating encoded features 920 can represent predictions corresponding to the frequencies of the input spectrogram data 306. For example, if the participating encoded features 920 correspond to speech expressing the emotion of fear, the input spectrogram data 306 may include predictions for higher frequencies; if the participating encoded features 920 correspond to speech expressing a whisper, the input spectrogram data 306 may include predictions for lower frequencies. In some embodiments, the input spectrogram data 306 includes frequencies adjusted according to a Mel scale, wherein the input spectrogram data 306 corresponds to a perceptual scale of pitches that are judged by listeners to be equally distant from each other. In these embodiments, the input spectrogram data 306 may include or be referred to as a Mel-spectrogram and / or a Mel-frequency cepstrum (MFC).

[0072] The speech decoder 706 may include one or more front-end layers 916. The front-end layers 916 may include two fully connected layers of 256 hidden units, such as rectified linear units (ReLUs). The front-end layers 916 receive the input spectrogram data 306 from the previous time step and may act as an information bottleneck, thereby helping the attention network 704 focus its attention on a specific output of the attention network 704. In some embodiments, the use of the front-end layers 916 allows the decoder 706 to place more emphasis on the output of the attention network 704 and less emphasis on the input spectrogram data 306 from the previous time step.

[0073] The output of the front network layer 916 can be connected to the output of the attention network 704. One or more LSTM layers 910 can receive the output of this connection. The LSTM layer 910 can include two unidirectional LSTM layers, each LSTM layer having (for example) 1124 units. The output of the LSTM layer 910 can be transformed with a linear transformation 912 (such as a linear projection). In other embodiments, a different transformation, such as an affine transformation, can be used. One or more back network layers 914, which can be convolutional layers, can receive the output of the linear transformation 912; in some embodiments, the back network layer 914 includes five layers, and each layer includes (for example) 512 units with 5 × 1-shaped filters with batch normalization. Tanh activation can be performed on all outputs except the last layer. The connection element can connect the output of the post-network layer 914 with the output of the linear transformation 912 to generate the spectrogram data 306.

[0074] Figure 10 An exemplary neural network is shown in FIG, which may be an encoder 602, a language encoder 702, an encoder 720, a decoder 606, and / or a speech decoder 706. The neural network may include nodes organized into an input layer 1002, one or more hidden layers 1004, and an output layer 1006. The input layer 1002 may include m nodes, the hidden layer 1004 may include n nodes, and the output layer 1006 may include o nodes, of which m 、 n and o can be any number and can represent the same or different number of nodes for each layer. The nodes of the input layer 1002 can receive input (e.g., audio data 302), and the nodes of the output layer 1006 can produce output (e.g., input spectrogram data 306). Each node of the hidden layer 1004 can be connected to one or more nodes in the input layer 1002 and one or more nodes in the output layer 1004. Although Figure 10 The neural network shown includes a single hidden layer 1004, but other neural networks may include multiple hidden layers 1004; in these cases, each node in a hidden layer can be connected to some or all nodes in an adjacent hidden (or input / output) layer. Each connection from one node to another node in an adjacent layer can be associated with a weight and / or score. The neural network can output one or more outputs, a set of weighted possible outputs, or any combination thereof.

[0075] Neural networks can also be constructed using recurrent connections, so that one or more outputs of the network's hidden layer 1004 are fed back into the hidden layer 1004 as the next set of inputs. Every node in the input layer is connected to every node in the hidden layer; and every node in the hidden layer is connected to every node in the output layer. As shown in the figure, one or more outputs of the hidden layer are fed back into the hidden layer to process the next set of inputs. A neural network that includes recurrent connections can be called a recurrent neural network (RNN).

[0076] The processing performed by a neural network is determined by the learned weights of each node's input and the network's structure. Given a specific input, the neural network determines one output layer at a time until the output layer for the entire network is calculated. Connection weights can be initially learned by the neural network during training, where a given input is associated with a known output. In a set of training data, various training examples are fed into the network. For each example, the correct connection from the input to the output is typically set to 1, and all connections are assigned a weight of 0. As examples in the training data are processed by the neural network, inputs can be sent to the network and compared to the associated outputs to determine how the network's performance compares to a target performance. Using training techniques such as backpropagation, the neural network's weights can be updated to reduce the errors made by the neural network when processing the training data. In some cases, a neural network can be trained on a lattice to improve speech recognition when processing the entire lattice.

[0077] Figure 11 1 is a block diagram conceptually illustrating the user device 110 . Figure 12 is a block diagram conceptually illustrating exemplary components of a remote system 120, which may be one or more servers and which may facilitate TTS processing, NLU processing, and the like. The term "system" as used herein may refer to a traditional system as understood in a system / client computing architecture, but may also refer to many different computing components that may facilitate the operations discussed herein. For example, a server may include one or more physical computing components (such as a rack system) that are physically and / or network-connected to other devices / components and capable of performing computing operations. A server may also include one or more virtual machines that simulate a computer system and run on one device or across multiple devices. A server may also include other combinations of hardware, software, firmware, and the like to perform the operations discussed herein. A server may be configured to operate using one or more of a client system model, a computer bureau model, grid computing technology, fog computing technology, mainframe technology, utility computing technology, a peer-to-peer model, sandbox technology, or other computing technologies.

[0078] The system 120 may include multiple servers, such as one or more servers for performing speech processing. In operation, each of these servers (or groups of devices) may include computer-readable and computer-executable instructions residing on the respective servers, as will be discussed further below. Each of these devices / systems (110 / 120) may include one or more controllers / processors (1104 / 1204), each of which may include a central processing unit (CPU) for processing data and computer-readable instructions and a memory (1106 / 1206) for storing data and instructions for the respective device. The memory (1106 / 1206) may individually include volatile random access memory (RAM), non-volatile read-only memory (ROM), non-volatile magnetoresistive memory (MRAM), and / or other types of memory. Each device (110 / 120) may also include a data storage component (1108 / 1208) for storing data and controller / processor executable instructions. Each data storage component (1108 / 1208) can individually include one or more non-volatile storage device types, such as magnetic storage devices, optical storage devices, solid-state storage devices, etc. Each device (110 / 120) can also be connected to removable or external non-volatile memory and / or storage devices (such as removable memory cards, memory key drives, network storage devices, etc.) through corresponding input / output device interfaces (1102 / 1202). Device 110 can also include a speaker 1112, a microphone 1120, a display 1116, and / or a camera 1118. Remote system 120 can similarly include an antenna 1214, a speaker 1212, a microphone 1220, a display 1216, and / or a camera 1218.

[0079] Computer instructions for operating each device / system (110 / 120) and its various components can be executed by the controller / processor (1104 / 1204) of the corresponding device at runtime using the memory (1106 / 1206) as a temporary "working" storage device. The computer instructions of the device can be stored in a non-volatile memory (1106 / 1206), storage device (1108 / 1208) or external device in a non-transitory manner. Alternatively, in addition to or in place of software, some or all of the executable instructions can be embedded in hardware or firmware on the corresponding device.

[0080] Each device / system (110 / 120) includes an input / output device interface (1102 / 1202). Various components can be connected via the input / output device interface (1102 / 1202), as will be discussed further below. In addition, each device (110 / 120) can include an address / data bus (1124 / 1224) for transferring data between the components of the respective device. In addition to (or instead of) connecting to other components across the bus (1124 / 1224), each component within the device (110 / 120) can also be directly connected to other components.

[0081] refer to Figure 13 , the device 110 may include an input / output device interface 1102, which is connected to various components, such as an audio output component (e.g., a microphone 1304 and / or a speaker 1306), a wired headset or a wireless headset (not shown), or other components capable of outputting audio. The device 110 may also include an audio capture component. The audio capture component may be, for example, a microphone 1120 or a microphone array, a wired headset or a wireless headset, etc. If a microphone array is included, the approximate distance to the origin of the sound can be determined by acoustic positioning based on the time and amplitude differences between the sounds captured by different microphones in the array. The device 110 may additionally include a display for displaying content. The device 110 may also include a camera.

[0082] Via antenna 1114, the input / output device interface 1102 can connect to one or more networks 199 via a wireless local area network (WLAN) (such as WiFi) radio, Bluetooth, and / or a wireless network radio, such as a radio capable of communicating with a wireless communication network such as a Long Term Evolution (LTE) network, a WiMAX network, a 3G network, a 4G network, a 5G network, etc. Wired connections such as Ethernet may also be supported. Through the network 199, the system can be distributed in a network environment. The I / O device interface (1102 / 1202) may also include communication components that allow data to be exchanged between devices (such as different physical systems or other components in a collection of systems).

[0083] Components of device 110 and / or system 120 may include their own dedicated processors, memory, and / or storage devices. Alternatively, one or more components of device 110 and / or system 120 may utilize the I / O interfaces (1102 / 1202), processors (1104 / 1204), memories (1106 / 1116), and / or storage devices (1108 / 1208) of device 110 and / or system 120.

[0084] As described above, multiple devices can be employed in a single system. In such a multi-device system, each of the devices can include different components for performing different aspects of the system processing. Multiple devices can include overlapping components. As described herein, the components of device 110 and / or system 120 are illustrative and can be positioned as independent devices or can be included in whole or in part as components of a larger device or system.

[0085] The network 199 can also connect voice-controlled devices 110a, tablet computers 110d, smartphones 110b, refrigerators 110c, desktop computers 110e, and / or laptop computers 110f via wireless service providers, via WiFi or cellular network connections, etc. Other devices can be included as network-connected support devices, such as system 120. Support devices can be connected to the network 199 via wired or wireless connections. Networked devices 110 can capture audio using one or more built-in or connected microphones and / or audio capture devices, with processing performed by the same device or another device connected via the network 199. The concepts disclosed herein can be applied within many different devices and computer systems, including, for example, general-purpose computing systems, speech processing systems, and distributed computing environments.

[0086] The foregoing content can also be understood in accordance with the following terms.

[0087] 1. A computer-implemented method for generating speech, the method comprising:

[0088] receiving first audio data representing a command from a user device;

[0089] processing the first audio data using an automatic speech recognition (ASR) component to determine first text data representing the speech;

[0090] processing the first text data using a natural language understanding (NLU) component to determine natural language understanding data including representations of entities in the first text data;

[0091] processing the natural understanding data using a dialog manager component to determine second text data representing a response to the first audio data, wherein the response includes a reference to the entity;

[0092] processing the second text data with a speech encoder of a text-to-speech (TTS) component to determine first encoded data representing words in the command;

[0093] processing the second text data with a second encoder of the TTS component to determine second encoded data corresponding to the natural understanding data;

[0094] processing the first encoded data, the second encoded data, and the natural understanding data with an attention network of the TTS component to determine weighted encoded data, the weighted encoded data corresponding to a variation of synthesized speech that emphasizes the name of the entity; and

[0095] The weighted encoded data is processed with a speech decoder of the TTS component to determine second audio data corresponding to the variation of the synthesized speech.

[0096] 2. The computer-implemented method of clause 1, further comprising:

[0097] processing, using the dialog manager component, the second NLU data to determine third textual data representing a second response to the second command;

[0098] processing the third text data and the second natural understanding data with a rewriting component to determine fourth text data, the fourth text data including a representation of the entity and at least a first term not represented in the third text data; and

[0099] The fourth text data is processed with the TTS component to determine third audio data.

[0100] 3. The computer-implemented method of clause 1 or 2, further comprising:

[0101] processing third text data using the NLU component to determine second natural understanding data including an intent to repeat a word represented in the first audio data;

[0102] processing the second natural understanding data using the dialog manager component to determine fourth text data representing a response to the third text data;

[0103] processing the second natural understanding data with the attention network to determine second weighted encoding data, the second weighted encoding data corresponding to the emphasis on the word; and

[0104] The second weighted encoded data is processed with the speech decoder to determine third audio data corresponding to a second sound characteristic associated with the word.

[0105] 4. The computer-implemented method of clause 1, 2, or 3, further comprising:

[0106] determining a domain associated with the natural understanding data;

[0107] determining that first data stored in a computer memory indicates that a speech style is associated with the domain;

[0108] determining second data representing the voice style,

[0109] Wherein said weighted coded data is also based at least in part on said second data.

[0110] 5. A computer-implemented method comprising:

[0111] receiving first input data corresponding to a response to the command;

[0112] receiving second input data comprising a machine representation of the command;

[0113] processing the first input data using a first model to determine first encoded data representing a word;

[0114] processing the first input data with a second model to determine second encoded data corresponding to the second input data;

[0115] processing the first coded data using the second coded data and the second input data to determine third coded data; and

[0116] The third encoded data is processed with a third model to determine audio data corresponding to a variation of the synthesized speech associated with the second input data.

[0117] 6. The computer-implemented method of clause 5, further comprising:

[0118] processing the audio data using a vocoder to determine output audio data; and

[0119] The output audio data is caused to be output.

[0120] 7. The computer-implemented method of clause 5 or 6, further comprising:

[0121] receiving third input data corresponding to a second response to the second command;

[0122] processing the third input data with a fourth model to determine fourth input data different from the third input data, the fourth input data corresponding to the second input data; and

[0123] The fourth input data is processed with the first model, the second model, and the third model to determine second audio data.

[0124] 8. The computer-implemented method of clause 7, further comprising:

[0125] Before processing the third input data, it is determined that the response corresponds to the second response and the command corresponds to the second command.

[0126] 9. The computer-implemented method of clause 5, 6, 7, or 8, further comprising:

[0127] determining a voice style associated with a domain associated with the response;

[0128] Wherein the third encoded data is also based at least in part on the speech style.

[0129] 10. The computer-implemented method of clause 5, 6, 7, 8, or 9, further comprising:

[0130] determining a score representing the extent of the change; and

[0131] It is determined that the score is less than a threshold.

[0132] 11. The computer-implemented method of clause 5, 6, 7, 8, 9, or 10, wherein processing the first input data with the second model further comprises:

[0133] The intermediate output of the second model is processed using at least one recurrent layer.

[0134] 12. The computer-implemented method of clause 5, 6, 7, 8, 9, 10, or 11, further comprising:

[0135] The second input data and the fourth encoded data are processed with the third model to determine second audio data corresponding to a second variation of the synthesized speech associated with the fourth input data.

[0136] 13. A system comprising:

[0137] at least one processor; and

[0138] at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:

[0139] receiving first input data corresponding to a response to the command;

[0140] receiving second input data comprising a machine representation of the command;

[0141] processing the first input data using a first model to determine first encoded data representing a word;

[0142] processing the first input data with a second model to determine second encoded data corresponding to the second input data;

[0143] processing the first encoded data using the second encoded data and the second input data with an attention network to determine third encoded data; and

[0144] The third encoded data is processed with a third model to determine audio data corresponding to a variation of the synthesized speech associated with the second input data.

[0145] 14. The system of clause 13, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

[0146] processing the audio data using a vocoder to determine output audio data; and

[0147] The output audio data is caused to be output.

[0148] 15. The system of clause 13 or 14, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

[0149] receiving third input data corresponding to a second response to the second command;

[0150] processing the third input data with a fourth model to determine fourth input data different from the third input data; and

[0151] The fourth input data is processed with the first model, the second model, and the third model to determine second audio data.

[0152] 16. The system of clause 15, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

[0153] Before processing the third input data, it is determined that the response corresponds to the second response and the command corresponds to the second command.

[0154] 17. The system of clause 13, 14, 15, or 16, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

[0155] determining a voice style associated with a domain associated with the response;

[0156] Wherein the third encoded data is also based at least in part on the speech style.

[0157] 18. The system of clause 13, 14, 15, 16, or 17, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

[0158] determining a score representing the extent of the change; and

[0159] It is determined that the score is less than a threshold.

[0160] 19. The system of clause 13, 14, 15, 16, 17, or 18, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

[0161] The intermediate output of the second model is processed using at least one recurrent layer.

[0162] 20. The system of clause 13, 14, 15, 16, 17, 18, or 19, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

[0163] The second input data and the fourth encoded data are processed with the third model to determine second audio data corresponding to a second variation of the synthesized speech associated with the fourth input data.

[0164] 21. A computer-implemented method comprising:

[0165] receiving first input data corresponding to a response to the command;

[0166] receiving second input data comprising a machine representation of the command;

[0167] processing the first input data with a first model to determine first encoded data representing words of the command, the first model including a speech encoder of a text-to-speech (TTS) component;

[0168] processing the first input data with a second model to determine second encoded data corresponding to the second input data, the second model including a second encoder of the TTS component;

[0169] processing the first encoded data using the second encoded data and the second input data with an attention network of the TTS component to determine third encoded data; and

[0170] The third encoded data is processed with a third model to determine audio data corresponding to a variation of the synthesized speech associated with the second input data.

[0171] 22. The computer-implemented method of clause 21, further comprising:

[0172] processing the audio data using a vocoder to determine output audio data; and

[0173] The output audio data is caused to be output.

[0174] 23. The computer-implemented method of clause 21 or 22, further comprising:

[0175] receiving third input data corresponding to a second response to the second command;

[0176] processing the third input data with a fourth model to determine fourth input data different from the third input data, the fourth input data corresponding to the second input data; and

[0177] The fourth input data is processed with the first model, the second model, and the third model to determine second audio data.

[0178] 24. The computer-implemented method of clause 23, further comprising:

[0179] Before processing the third input data, it is determined that the response corresponds to the second response and the command corresponds to the second command.

[0180] 25. The computer-implemented method of clause 21, 22, 23, or 24, further comprising:

[0181] determining a voice style associated with a domain associated with the response;

[0182] Wherein the third encoded data is also based at least in part on the speech style.

[0183] 26. A computer-implemented method as described in clause 21, 22, 23, 24 or 25, wherein the third encoded data includes weighted encoded data corresponding to the variation of the synthesized speech, the variation emphasizing the name of the entity represented in the audio data.

[0184] 27. The computer-implemented method of clause 21, 22, 23, 24, 25, or 26, wherein processing the first input data with the second model further comprises:

[0185] The intermediate output of the second model is processed using at least one recurrent layer.

[0186] 28. The computer-implemented method of clause 21, 22, 23, 24, 25, 26, or 27, further comprising:

[0187] The second input data and the fourth encoded data are processed with the third model to determine second audio data corresponding to a second variation of the synthesized speech associated with the fourth input data.

[0188] 29. A system comprising:

[0189] at least one processor; and

[0190] at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:

[0191] receiving first input data corresponding to a response to the command;

[0192] receiving second input data comprising a machine representation of the command;

[0193] processing the first input data with a first model to determine first encoded data representing words of the command, the first model including a speech encoder of a text-to-speech (TTS) component;

[0194] processing the first input data with a second model to determine second encoded data corresponding to the second input data, the second model including a second encoder of the TTS component;

[0195] processing the first encoded data using the second encoded data and the second input data with an attention network of the TTS component to determine third encoded data; and

[0196] The third encoded data is processed with a third model to determine audio data corresponding to a variation of the synthesized speech associated with the second input data.

[0197] 30. The system of clause 29, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

[0198] processing the audio data using a vocoder to determine output audio data; and

[0199] The output audio data is caused to be output.

[0200] 31. The system of clause 30, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

[0201] receiving third input data corresponding to a second response to the second command;

[0202] processing the third input data with a fourth model to determine fourth input data different from the third input data; and

[0203] The fourth input data is processed with the first model, the second model, and the third model to determine second audio data.

[0204] 32. The system of clause 31, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

[0205] Before processing the third input data, it is determined that the response corresponds to the second response and the command corresponds to the second command.

[0206] 33. The system of clause 29, 30, 31 or 32, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

[0207] determining a voice style associated with a domain associated with the response;

[0208] Wherein the third encoded data is also based at least in part on the speech style.

[0209] 34. A system as described in clause 29, 30, 31, 32 or 33, wherein the third encoded data includes weighted encoded data corresponding to the variation of the synthesized speech, the variation emphasizing the name of the entity represented in the audio data.

[0210] 35. The system of clause 29, 30, 31, 32, 33, or 34, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

[0211] The intermediate output of the second model is processed using at least one recurrent layer.

[0212] The above aspects of the present disclosure are intended to be illustrative. The aspects are selected to illustrate the principles and applications of the present disclosure and are not intended to be exhaustive or to limit the present disclosure. Many modifications and variations of the disclosed aspects may be apparent to those skilled in the art. Those of ordinary skill in the art of computer and speech processing will recognize that the components and processing steps described herein may be interchangeable with other components or steps or combinations of components or steps and still achieve the benefits and advantages of the present disclosure. In addition, those skilled in the art will appreciate that the present disclosure may be practiced without some or all of the specific details and steps disclosed herein.

[0213] Aspects of the disclosed systems may be implemented as computer methods or articles of manufacture, such as a memory device or a non-transitory computer-readable storage medium. The computer-readable storage medium may be readable by a computer and may include instructions for causing a computer or other device to perform the processes described herein. The computer-readable storage medium may be implemented by volatile computer memory, non-volatile computer memory, a hard drive, a solid-state memory, a flash drive, a removable disk, and / or other media. Additionally, one or more of the components and engines may be implemented as firmware or hardware, such as an acoustic front end (AFE), which may include, among other things, analog and / or digital filters (e.g., filters configured as firmware for a digital signal processor (DSP)).

[0214] As used in this disclosure, unless specifically stated otherwise, the term "a" or "an" may include one or more items. In addition, unless specifically stated otherwise, the phrase "based on" is intended to mean "based at least in part on."

Claims

1. A computer-implemented speech processing method comprising: receiving first input data corresponding to a response to the command; receiving second input data comprising a machine representation of the command; processing the first input data with a first model to determine first encoded data representing words of the command, the first model including a speech encoder of a text-to-speech component; processing the first input data with a second model to determine second encoded data corresponding to the second input data, the second model including a second encoder of the text-to-speech component; processing the first encoded data using the second encoded data and the second input data with an attention network of the text-to-speech component to determine third encoded data; as well as The third encoded data is processed with a third model to determine audio data corresponding to a variation of the synthesized speech associated with the second input data.

2. The computer-implemented speech processing method of claim 1 , further comprising: processing the audio data using a vocoder to determine output audio data; as well as The output audio data is caused to be output.

3. The computer-implemented speech processing method of claim 1 , further comprising: receiving third input data corresponding to a second response to the second command; processing the third input data using a fourth model to determine fourth input data different from the third input data, the fourth input data corresponding to the second input data; as well as The fourth input data is processed with the first model, the second model, and the third model to determine second audio data.

4. The computer-implemented speech processing method of claim 3 , further comprising: Before processing the third input data, it is determined that the response corresponds to the second response and the command corresponds to the second command.

5. The computer-implemented speech processing method of any one of claims 1 to 4, further comprising: determining a voice style associated with a domain associated with the response; Wherein the third encoded data is also based at least in part on the speech style.

6. A computer-implemented speech processing method as claimed in any one of claims 1 to 4, wherein the third encoded data comprises weighted encoded data corresponding to the variation of the synthesized speech, the variation emphasizing the name of the entity represented in the audio data.

7. The computer-implemented speech processing method of any one of claims 1 to 4, wherein processing the first input data with the second model further comprises: The intermediate output of the second model is processed using at least one recurrent layer.

8. The computer-implemented speech processing method of any one of claims 1 to 4, further comprising: The second input data and fourth encoded data are processed with the third model to determine second audio data corresponding to a second variation of the synthesized speech associated with the fourth input data.

9. A speech processing system comprising: at least one processor; as well as at least one memory comprising instructions that, when executed by the at least one processor, cause the system to: receiving first input data corresponding to a response to the command; receiving second input data comprising a machine representation of the command; processing the first input data with a first model to determine first encoded data representing words of the command, the first model including a speech encoder of a text-to-speech component; processing the first input data with a second model to determine second encoded data corresponding to the second input data, the second model including a second encoder of the text-to-speech component; processing the first encoded data using the second encoded data and the second input data with an attention network of the text-to-speech component to determine third encoded data; and The third encoded data is processed with a third model to determine audio data corresponding to a variation of the synthesized speech associated with the second input data.

10. The speech processing system of claim 9, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: processing the audio data using a vocoder to determine output audio data; and The output audio data is caused to be output.

11. The speech processing system of claim 10, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: receiving third input data corresponding to a second response to the second command; processing the third input data with a fourth model to determine fourth input data different from the third input data; and The fourth input data is processed with the first model, the second model, and the third model to determine second audio data.

12. The speech processing system of claim 11 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: Before processing the third input data, it is determined that the response corresponds to the second response and the command corresponds to the second command.

13. The speech processing system of any one of claims 9 to 12, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: determining a voice style associated with a domain associated with the response; Wherein the third encoded data is also based at least in part on the speech style.

14. The speech processing system according to any one of claims 9 to 12, wherein the third encoded data includes weighted encoded data corresponding to the variation of the synthesized speech, the variation emphasizing the name of the entity represented in the audio data.

15. The speech processing system of any one of claims 9 to 12, wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to: The intermediate output of the second model is processed using at least one recurrent layer.

Images