Synthesized speech generation

Abstract

A device for speech generation includes one or more processors configured to receive one or more control parameters indicative of target speech characteristics, the one or more processors also configured to process, using a multi-encoder, an input representation of speech based on the one or more control parameters to generate encoded data corresponding to an audio signal representing a version of the speech based on the target speech characteristics.

Description

Technical Field

[0001] Priority Claim This application claims the benefit of priority of U.S. Non-Provisional Patent Application No. 17 / 154,372, filed on January 21, 2021, and owned by the same applicant, the entire content of which is hereby incorporated by reference in its entirety.

[0002] The present disclosure generally relates to synthetic speech generation.

Background Art

[0003] Technological advancements have led to smaller and more powerful computing devices. For example, there are now various portable personal computing devices, including wireless telephones such as mobile phones and smartphones, tablets, and laptop computers, which are small, lightweight, and easily carried by users. These devices can communicate voice and data packets via wireless networks. Additionally, many such devices incorporate additional functions such as digital still cameras, digital video cameras, digital recorders, and audio file players. Also, such devices can process executable instructions, including software applications such as web browser applications that can be used to access the Internet. Thus, these devices can include significant computing power.

[0004] Such computing devices often incorporate the ability to receive audio signals from one or more microphones. For example, the audio signals may represent user utterances captured by the microphones, external sounds captured by the microphones, or a combination thereof. Such devices may include applications that use speech models. For example, voice-activated applications recognize user utterance commands. Speech-based applications are trained using audio speech samples. Using larger and more diverse datasets of audio speech samples for training results in more robust speech models. However, obtaining a large number of diverse audio speech samples from people can be time-consuming and inconvenient. [Overview of the Initiative] [Means for solving the problem]

[0005] According to one implementation of the present disclosure, a device for speech generation includes one or more processors configured to receive one or more control parameters indicating a target utterance characteristic. The one or more processors are also configured to process an input representation of an utterance based on one or more control parameters, using a multiencoder, to generate encoded data corresponding to an audio signal representing a version of the utterance based on the target utterance characteristic.

[0006] According to another implementation of the present disclosure, a speech generation method includes the step of receiving one or more control parameters in a device that indicate a target utterance characteristic. The method also includes the step of processing an input representation of an utterance based on one or more control parameters, using a multiencoder to generate encoded data corresponding to an audio signal representing a version of the utterance based on the target utterance characteristic.

[0007] According to another implementation of the present disclosure, a non-temporary computer-readable storage medium stores instructions that, when executed by one or more processors, cause one or more processors to receive one or more control parameters representing a target utterance characteristic. The instructions also, when executed by one or more processors, cause one or more processors to use a multiencoder to process an input representation of an utterance based on one or more control parameters and generate encoded data corresponding to an audio signal representing a version of the utterance based on the target utterance characteristic.

[0008] According to another implementation of the present disclosure, the device includes means for receiving one or more control parameters indicating a target utterance characteristic. The device also includes means for processing an input representation of an utterance based on one or more control parameters, using a multiencoder to generate encoded data corresponding to an audio signal representing a version of the utterance based on the target utterance characteristic.

[0009] After reviewing the entire application, including the following sections, namely the brief description of the drawings, the modes for carrying out the invention, and the claims, other aspects, advantages, and features of the present disclosure will become apparent. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram of a particular exemplary embodiment of a system capable of operating to generate synthesized speech, as illustrated by some examples of the present disclosure. [Figure 2] This figure shows a specific implementation of the speech generator of the system in Figure 1, with some examples from the present disclosure. [Figure 3] This figure shows another specific implementation of the speech generator shown in Figure 1, with some examples from the present disclosure. [Figure 4A] This figure shows another specific implementation of the speech generator shown in Figure 1, with some examples from the present disclosure. [Figure 4B]This figure shows another specific implementation of the speech generator shown in Figure 1, with some examples from the present disclosure. [Figure 5] This figure shows an exemplary embodiment of the encoder of the speech generator shown in Figure 1, according to some examples of the present disclosure. [Figure 6] This figure shows illustrative embodiments of the operation of the components of the system in Figure 1, based on several examples of the present disclosure. [Figure 7] This figure shows illustrative embodiments of the operation of the components of the system in Figure 1, based on several examples of the present disclosure. [Figure 8] This figure shows an example of an integrated circuit capable of operating to generate synthesized speech, with some examples from the present disclosure. [Figure 9] This figure shows a mobile device capable of operating to generate synthesized speech, as illustrated by some examples of the present disclosure. [Figure 10] This figure shows a headset capable of operating to generate synthesized speech, as illustrated by some examples of the present disclosure. [Figure 11] This figure shows some examples of wearable electronic devices capable of generating synthesized speech. [Figure 12] This figure shows a voice-controlled speaker system capable of operating to generate synthesized speech, according to some examples of the present disclosure. [Figure 13] This figure shows a camera capable of operating to generate synthesized speech, as illustrated by some examples of the present disclosure. [Figure 14] The diagram shows a headset, such as a virtual reality or augmented reality headset, which is capable of operating to generate synthesized speech, as illustrated by some examples of the present disclosure. [Figure 15] This figure shows a first example of a mobile device capable of operating to generate synthesized speech, according to some examples of the present disclosure. [Figure 16] This figure shows a second example of a mobile device capable of operating to generate synthesized speech, according to some examples of the present disclosure. [Figure 17]Figure 1 shows a specific implementation of a method for generating synthesized speech, which may be performed by the device shown in Figure 1, with some examples from the present disclosure. [Figure 18] Figure 1 shows another specific implementation of a method for generating synthesized speech, which may be performed by the device shown in some examples of the present disclosure. [Figure 19] This is a block diagram of a specific exemplary example of a device capable of operating to generate synthesized speech, as illustrated by several examples of the present disclosure. [Modes for carrying out the invention]

[0011] Using larger and more diverse datasets of audio speech samples for training results in more robust speech models. Preparing a large number of audio speech samples for speech training can be inconvenient and time-consuming for users. A system and method for synthetic speech generation are disclosed. For example, a speech generator receives (or accesses) an input representation of an utterance (e.g., an audio signal, video signal, text, etc.). The speech generator receives control parameters that indicate target utterance characteristics, such as target emotion (e.g., calm, happy, sad, etc.), target utterance rate (e.g., slow, medium, or fast), target person on whom the utterance characteristics are used (e.g., a user with a particular intonation, a celebrity, a child, etc.), or a combination thereof. The speech generator includes a multiencoder, such as a multiencoder converter, which processes the input representation based on the control parameters to produce encoded data based on the target utterance characteristics, and a decoder that generates an audio signal (e.g., a synthetic speech signal) by decoding the encoded data. The speech generator can generate a diverse set of synthetic audio samples by applying different sets of control parameters to the same input representation.

[0012] Certain aspects of the present disclosure are described below with reference to the drawings. In the description, common features are designated by common reference numerals. As used herein, the various terms are used only for the purpose of describing a particular implementation and are not intended to limit the implementation. For example, the singular forms "a", "an", and "the" are intended to include the plural form as well unless the context clearly indicates otherwise. Further, some of the features described herein are singular in some implementations and plural in other implementations. By way of example, FIG. 1 shows a device 102 that includes one or more processors (the "processor" 190 in FIG. 1), which indicates that in some implementations, the device 102 includes a single processor 190 and in other implementations, the device 102 includes multiple processors 190. For ease of reference herein, such features are generally introduced as "one or more" features and are then referred to in the singular form unless otherwise described in connection with aspects related to multiple of such features.

[0013] As used herein, the terms “comprise,” “comprises,” and “comprising” may be used interchangeably with “include,” “includes,” or “including.” Furthermore, the term “wherein” may be used interchangeably with “where.” As used herein, “exemplary” indicates an example, implementation, and / or aspect, and should not be construed as limiting or indicating a preference or suitable implementation. As used herein, ordering terms used to modify elements such as structure, components, and behavior (e.g., “first,” “second,” “third,” etc.) do not in themselves indicate priority or order of an element relative to another element, but rather merely distinguish an element from another element having the same name (apart from the use of ordering terms). As used herein, the term “set” refers to one or more of a particular element, and the term “plurality” refers to multiple (e.g., two or more) of a particular element.

[0014] As used herein, "coupled" may include "communicatively coupled", "electrically coupled", or "physically coupled", or (alternatively) any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) via one or more other devices, components, wires, buses, networks (e.g., a wired network, a wireless network, or a combination thereof), etc. By way of illustrative and non-limiting example, two electrically coupled devices (or components) may be included in the same device or in different devices, respectively, and may be connected via an electronic device, one or more connectors, or an inductive coupling. In some implementations, two devices (or components) communicatively coupled in, e.g., electrical communication, may directly or indirectly transmit and receive signals (e.g., digital signals or analog signals) via one or more wires, buses, networks, etc. As used herein, "directly coupled" may include two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without intervening components.

[0015] In this disclosure, terms such as “determine,” “calculate,” “estimate,” “shift,” and “adjust” may be used to describe how one or more actions are performed. Such terms should not be construed as restrictive, and it should be noted that other techniques may be used to perform similar actions. In addition, “generate,” “calculate,” “estimate,” “use,” “select,” “access,” and “determine” as used herein may be used interchangeably. For example, “generate,” “calculate,” “estimate,” or “determine” a parameter (or signal) may refer to actively generating, estimating, calculating, or determining the parameter (or signal), or it may refer to using, selecting, or accessing a parameter (or signal) that has already been generated by another component or device.

[0016] Referring to Figure 1, a specific exemplary embodiment of a system configured to generate synthesized speech is disclosed and is designated collectively as 100. System 100 includes a device 102 configured to generate synthesized speech using a speech generator 120. Device 102 includes one or more processors 190 coupled to memory 132. One or more processors 190 include the speech generator 120. In some implementations, device 102 is coupled to a microphone 182, a speech repository 184, or both. In certain implementations, the microphone 182, the speech repository 184, or both are located outside of device 102. In alternative implementations, the microphone 182, the speech repository 184, or both are incorporated into device 102.

[0017] The audio generator 120 includes an audio analyzer 122, a style data generator 126, a multi-encoder transformer 128, an audio synthesizer 130, or a combination thereof. In certain embodiments, the audio analyzer 122 is coupled to the style data generator 126, the multi-encoder transformer 128, or both. The style data generator 126 is coupled to the audio synthesizer 130 via the multi-encoder transformer 128. In certain embodiments, the audio synthesizer 130 is coupled to an audio repository 184.

[0018] The audio analyzer 122 is configured to process an input signal 121 to generate an input representation 123 of an utterance. In a particular example, the input signal 121 includes, for example, an audio signal received via a microphone 182 or retrieved from a storage device. In another example, the input signal 121 includes a video signal representing audio and image information. The input representation 123 includes, based on the input signal 121, text, a Mel-scale spectrogram, fundamental frequency (F0) features, one or more other representations of the speech, or a combination thereof. In a particular example, the audio analyzer 122 is configured to perform feature extraction to generate a Mel-scale spectrogram, fundamental frequency (F0) features, or a combination thereof as the input representation 123. In a particular example, the audio analyzer 122 is configured to perform speech-to-text conversion on the input signal 121 to generate text as the input representation 123.

[0019] The style data generator 126 is configured to generate merged style data based on an input representation 123 and one or more control parameters 125, as will be further described with reference to Figure 2. The one or more control parameters 125 indicate one or more target utterance characteristics 141. For example, the target utterance characteristics 141 indicate a target person 143 (e.g., person 180), a target emotion 145 (e.g., happy, angry, excited, sad, or calm), a target utterance rate 147 (e.g., intermediate utterance rate), or a combination thereof. In a particular embodiment, the merged style data 127 corresponds to an embedding (e.g., an embedding vector) representing the input representation 123 and one or more control parameters 125.

[0020] The multi-encoder transformer 128 is configured to process an input representation 123 and merged style data 127 to generate output spectral data, as will be further described with reference to Figures 4A to 5. The multi-encoder transformer 128 includes a multi-encoder 192 and a decoder 194. The multi-encoder 192 is configured to process the input representation 123 based on one or more control parameters 125 to generate encoded data 193 corresponding to a synthesized speech signal 131. For example, the synthesized speech signal 131 is based on the encoded data 193. In certain embodiments, as will be further described with reference to Figure 4A, the multi-encoder 192 includes a first encoder configured to process the input representation 123 independently of one or more control parameters 125 (for example, independently of a target utterance characteristic 141) to generate first encoded data. The multi-encoder 192 includes one or more second encoders configured to process merged style data 127 (based on one or more control parameters 125) to generate second encoded data, as will be further described with reference to Figures 4A and 4B. Thus, the second encoded data is based on one or more control parameters 125 (e.g., target utterance characteristics 141). The decoder 194 is configured to generate output spectral data 129 (e.g., first encoded data and second encoded data) based on the encoded data 193, as will be further described with reference to Figures 4A and 4B. The speech synthesizer 130 is configured to perform speech synthesis on the output spectral data 129 to generate a synthesized speech signal 131 representing a version of the utterance (e.g., indicated by the input representation 123) based on the target utterance characteristics 141.

[0021] Memory 132 is configured to store data used or generated by the speech generator 120. For example, memory 132 is configured to store at least a portion of the input signal 121, the input representation 123, one or more control parameters 125, merged style data 127, encoded data 193, output spectral data 129, at least a portion of the synthesized speech signal 131, or a combination thereof. In certain embodiments, memory 132 stores mapping data that shows a mapping between one or more control parameters 125 and a target speech characteristic 141. For example, the mapping data shows a first mapping between the value of a first control parameter from one or more control parameters 125 and a set of people 151, a second mapping between the value of a second control parameter from one or more control parameters 125 and a set of emotions 161, a third mapping between the value of a third control parameter from one or more control parameters 125 and a set of speech rates 171, or a combination thereof. In response to the mapping data indicating that the value of a first control parameter, indicated by one or more control parameters 125, maps to a target person 143 from the set of people 151, the style data generator 126 determines that one or more control parameters 125 indicate a target person 143. Similarly, in response to the mapping data indicating that the value of a second control parameter, indicated by one or more control parameters 125, maps to a target emotion 145 from the set of emotions 161, the style data generator 126 determines that one or more control parameters 125 indicate a target emotion 145. In response to the mapping data indicating that the value of a third control parameter, indicated by one or more control parameters 125, maps to a target speech rate 147 from the set of speech rates 171, the style data generator 126 determines that one or more control parameters 125 indicate a target speech rate 147.

[0022] In some implementations, device 102 corresponds to or includes one or various types of devices. In the illustrated example, one or more processors 190 are incorporated into a headset device including a microphone 182, as further described with reference to Figure 10. In other examples, one or more processors 190 are incorporated into at least one of the following: a mobile phone or tablet computer device, as described with reference to Figure 9; a wearable electronic device, as described with reference to Figure 11; a voice-controlled speaker system, as described with reference to Figure 12; a camera device, as described with reference to Figure 13; or a virtual reality, augmented reality, or mixed reality headset, as described with reference to Figure 14. In yet another illustrated example, one or more processors 190 are incorporated into a mobile device, also including a microphone 182, as further described with reference to Figures 15 and 16.

[0023] During operation, the audio analyzer 122 receives an input signal 121 via the microphone 182. For example, the microphone 182 captures an utterance from a person 180 (e.g., "Please unlock the door") and provides the audio analyzer 122 with an input signal 121 representing the utterance. The utterance represented by the input signal 121 (e.g., "Please unlock the door") has a specific speech rate (e.g., fast speech rate) and a specific emotion (e.g., anxiety).

[0024] The audio analyzer 122 generates an input representation 123 of the input signal 121. For example, the audio analyzer 122 performs speech-to-text conversion on the input signal 121 to generate text (e.g., "Please unlock the door"). Alternatively, the audio analyzer 122 performs audio feature extraction on the input signal 121 to generate a Mel-scale spectrogram, fundamental frequency (F0) features, or a combination thereof. The input representation 123 includes text (e.g., "Please unlock the door"), a Mel-scale spectrogram, fundamental frequency (F0) features, or a combination thereof.

[0025] Device 102 receives or accesses one or more control parameters 125. In certain embodiments, one or more control parameters 125 are based on default data, configuration settings, user input, or a combination thereof. One or more control parameters 125 indicate the target speech characteristics 141 of the synthesized speech produced by the speech generator 120.

[0026] In certain embodiments, the target utterance characteristic 141 indicates a target person 143 (e.g., person 180) from a set of people 151. The set of people 151 includes one or more additional people, such as a first person (person 1) 153, a second person (person 2) 155, an Nth person (person N) 157, or a combination thereof. In exemplary and non-limiting examples, the set of people 151 includes a specific celebrity, a specific user, a person with a specific tone of voice, a person of a specific age, a person of a specific gender, or a combination thereof. The utterance characteristic of target person 143 is used by the speech generator 120 to produce synthesized speech. In a particular example, target person 143 indicates "President Obama." In another example, target person 143 indicates "a middle-aged man with a Texan accent." In a particular example, target person 143 indicates person 180.

[0027] In certain embodiments, the target utterance characteristic 141 indicates a target emotion 145 (e.g., calmness) selected from a set of emotions 161. The set of emotions 161 includes one or more additional emotions, such as a first emotion (emotion 1) 163, a second emotion (emotion 2) 165, an Nth emotion (emotion N) 167, or a combination thereof. In an exemplary and non-restrictive example, the set of emotions 161 includes joy, fear, anxiety, excitement, anger, calmness, calmness, or a combination thereof.

[0028] In certain embodiments, the target utterance characteristic 141 indicates a target utterance rate 147 (e.g., an intermediate utterance rate) selected from a set of utterance rates 171. The set of utterance rates 171 includes one or more additional utterance rates, such as a first utterance rate (utterance rate 1) 173, a second utterance rate (utterance rate 2) 175, an Nth utterance rate (utterance rate N) 177, or a combination thereof. In illustrative and non-restrictive examples, the set of utterance rates 171 includes a slow utterance rate, an intermediate utterance rate, a fast utterance rate, or a combination thereof.

[0029] The style data generator 126 generates merged style data 127 based on the input representation 123 and one or more control parameters 125, as will be further explained with reference to Figure 2. For example, the merged style data 127 corresponds to an embedding (e.g., an embedding vector) representing the input representation 123 and one or more control parameters 125.

[0030] The multi-encoder transformer 128 processes the input representation 123 and merged style data 127 to generate output spectral data 129, as will be further described with reference to Figures 4A and 4B. For example, the first encoder of the multi-encoder 192 processes the input representation 123 independently of one or more control parameters 125 (for example, independently of the target utterance characteristic 141) to generate first encoded data, as will be further described with reference to Figure 4A. One or more second encoders of the multi-encoder 192 process the merged style data 127 to generate second encoded data, as will be further described with reference to Figures 4A and 4B.

[0031] In certain embodiments, the merged style data 127 is used by the multi-encoder 192 while processing the input representation 123. For example, one or more encoders process the merged style data 127 simultaneously with the first encoder processing the input representation 123. The multi-encoder 192 provides the encoded data 193 (e.g., first encoded data, second encoded data, or a combination thereof) to the decoder 194. In certain embodiments, the encoded data 193 corresponds to the output embedding of the multi-encoder 192 (e.g., an embedding vector).

[0032] The decoder 194 processes the encoded data 193 (for example, the first encoded data and the second encoded data) to generate output spectral data 129, as further explained with reference to Figure 4A. The speech synthesizer 130 performs speech synthesis on the output spectral data 129 to generate a synthesized speech signal 131. The synthesized speech signal 131 corresponds to the version of the utterance shown by the input representation 123 based on the target utterance characteristics 141.

[0033] The multi-encoder 192 and decoder 194 are described as being included in the multi-encoder transformer 128 as an exemplary example. In some examples, the multi-encoder 192 and decoder 194 have configurations other than transformers.

[0034] If the multi-encoder 192 for one or more control parameters 125 has one or more second encoders (separate from the first encoder which is independent of one or more control parameters 125), then the second encoders can be trained (e.g., optimized) to encode the input representation 123 based on each of the one or more control parameters 125, as will be further explained with reference to Figure 3.

[0035] In certain examples, the synthesized speech signal 131 has a target speech characteristic 141. In certain examples, the synthesized speech signal 131 comes closer to having the target speech characteristic 141 (compared to the input signal 121). In certain embodiments, the speech generator 120 includes an adjuster configured to tune the configuration parameters (e.g., neural network weights) of the multi-encoder transformer 128 based on one or more control parameters 125 and estimated control parameters corresponding to the synthesized speech signal 131, as further described with reference to Figure 3. Thus, the speech generator 120 can dynamically train the multi-encoder transformer 128 to improve the accuracy of synthesized speech generation.

[0036] Device 102 is described, as an exemplary example, as receiving an input signal 121 and generating an input representation 123 based on the input signal 121. For example, each of the style data generator 126 and the multi-encoder transformer 128 receives the input representation 123 from the audio analyzer 122. In some examples, device 102 receives the input representation 123 (for example, instead of the input signal 121). For example, device 102 (for example, each of the multi-encoder transformer 128 and the style data generator 126) receives the input representation 123 as input from person 180, another device, or both.

[0037] As an illustrative example, input expression 123 corresponds to the utterance of person 180 having a specific emotion (e.g., anxiety) and a specific speech rate (e.g., high speech rate). Synthesized speech signal 131 corresponds to the utterance of target person 143 (e.g., person 180) having a target emotion 145 (e.g., calm) and a target speech rate 147 (e.g., medium speech rate). In this example, performing speech recognition on the synthesized speech signal 131 (e.g., calm utterance with medium speech rate) results in improved recognition accuracy compared to performing speech recognition on input signal 121 (e.g., anxious utterance with high speech rate).

[0038] In certain embodiments, the speech generator 120 appends the synthesized speech signal 131 to the speech repository 184. For example, the speech generator 120 generates (or updates) a training dataset based on the input signal 121, as further described with reference to Figure 6. For illustrative purposes, the speech generator 120 generates a first synthesized speech signal 131 based on a first set of one or more control parameters 125, a second synthesized speech signal 131 based on a second set of one or more control parameters 125, one or more additional synthesized speech signals 131 based on one or more additional sets of one or more control parameters 125, or a combination thereof. The speech generator 120 appends the input signal 121, the first synthesized speech signal 131, the second synthesized speech signal 131, one or more additional synthesized speech signals 131, or a combination thereof, to the training dataset in the speech repository 184.

[0039] In a specific example, the speech generator 120 generates multiple synthesized speech signals corresponding to various emotions and speech rates based on the same input signal 121. In this example, the synthesized speech signals can be used to train a robust speech model (e.g., a neural network) that represents the utterances of person 180 using a few words uttered by person 180. For illustrative purposes, during registration, utterances of person 180 with specific speech characteristics (e.g., calm utterances with a medium speech rate) (e.g., registered utterances) are received as input signal 121 for training the neural network. By training the neural network on synthesized speech corresponding to various speech characteristics, the neural network can recognize utterances with speech characteristics different from those of the registered speech (e.g., calm utterances with a medium speech rate) (e.g., anxious utterances with a fast speech rate).

[0040] In a specific example, the speech generator 120 generates multiple synthesized speech signals based on the same input signal 121, corresponding to various emotions, different speech rates, and different speech characteristics of different individuals. In this example, a large and diverse dataset of synthesized speech can be generated using a few words spoken by person 180.

[0041] Microphone 182 is shown coupled to device 102, but in other implementations, microphone 182 may be integrated into device 102. While one microphone 182 is illustrated, other implementations may include one or more additional microphones configured to capture user utterances. In some implementations, microphone 182 is omitted, for example, when the speech generator 120 retrieves an input signal 121 or input representation 123 from storage (for example, as text data retrieved from memory 132 or received from another device). System 100 is shown containing all components of speech generator 120 within device 102, but in other implementations, a second device may be added, which may perform some of the operations described as being performed in device 102.

[0042] Referring to Figure 2, a specific implementation of the speech generator 120 is shown. The style data generator 126 includes multiple speech characteristic encoders, such as a person encoder 206, an emotion encoder 208, a speech rate encoder 210, one or more additional speech characteristic encoders, or a combination thereof.

[0043] In certain embodiments, the audio analyzer 122 is coupled to each of a plurality of speech characteristic encoders. Each of the plurality of speech characteristic encoders is coupled to the encoder planet 212. Each of the plurality of speech characteristic encoders is configured to generate an encoded version of the input representation 123 based on the corresponding speech characteristic indicated by one or more control parameters 125. For example, the person encoder 206 is configured to process the input representation 123 and the person control parameter (e.g., indicating a target person 143) from one or more control parameters 125 to generate a person encoded input representation 207. For illustrative purposes, the person encoder 206 includes a neural network (e.g., a convolutional neural network (CNN)) configured to receive the input representation 123 and the person control parameter (e.g., indicating a target person 143) as input and output a person encoded input representation 207 (e.g., an embedding vector) representing the input representation 123 and the person control parameter.

[0044] Similarly, the emotion encoder 208 is configured to process the input expression 123 and one or more emotion control parameters 125 (for example, indicating a target emotion 145) to generate an emotion-encoded input expression 209. In a particular example, the speech rate encoder 210 is configured to process the input expression 123 and one or more speech rate parameters 125 (for example, indicating a target speech rate 147) to generate an speech rate-encoded input expression 211.

[0045] The encoder planet 212 includes a neural network (e.g., a CNN), a linear projection layer, or both. In a particular example, a person-coded input representation 207, an emotion-coded input representation 209, an utterance-rate-coded input representation 211, or a combination thereof, corresponds to the input embeddings of the neural network (e.g., a CNN). The neural network of the encoder planet processes the input embeddings (e.g., coded input representations) to produce output embeddings. The neural network provides the output embeddings to the linear projection layer, which produces merged style data 127 by performing a linear projection on the output embeddings of the neural network. In a particular embodiment, the multi-encoder transformer 128 does not include convolutions, which are performed by processing the person-coded input representation 207, an emotion-coded input representation 209, an utterance-rate-coded input representation 211, or a combination thereof using the encoder planet 212. The encoder planet 212 provides the merged style data 127 to the multi-encoder 192.

[0046] The multi-encoder 192 receives an input representation 123 from the audio analyzer 122 and merged style data 127 from the style data generator 126. The multi-encoder 192 processes the input representation 123 and the merged style data 127 to generate encoded data 193. For example, as further described with reference to Figures 4A and 4B, the multi-encoder 192 processes the input representation 123 to generate first encoded data independent of the target utterance characteristic 141 in Figure 1, and processes the merged style data 127 to generate second encoded data corresponding to the target utterance characteristic 141 in Figure 1. In certain embodiments, the encoder planet 212 also provides the merged style data 127 to the decoder 194. In this embodiment, the decoder 194 processes the encoded data 193 based on the merged style data 127 to generate output spectral data 129.

[0047] Referring to Figure 3, a specific implementation of the speech generator 120 is shown, in which the speech synthesizer 130 is coupled to the adjuster 314 via a control parameter estimator 310 and a loss calculator 312.

[0048] The control parameter estimator 310 processes the synthesized speech signal 131 to generate one or more estimated control parameters 311. For example, the control parameter estimator 310 determines the estimated person, estimated emotion, estimated speech rate, or a combination thereof, corresponding to the synthesized speech signal 131.

[0049] In a specific example, the control parameter estimator 310 generates a model match score by comparing the synthesized speech signal 131 with a speech model associated with each of the set of people 151 in Figure 1. The control parameter estimator 310 selects from the model match scores the model match score corresponding to the specific speech model that best matches the synthesized speech signal 131 (e.g., the best match score). In response to determining that the specific speech model represents the utterance of a specific person, the control parameter estimator 310 generates one or more estimated control parameters 311 that indicate the specific person as the estimated person.

[0050] In certain embodiments, the control parameter estimator 310 uses emotion detection techniques to detect the estimated emotion indicated by the synthesized speech signal 131. For example, the control parameter estimator 310 includes a neural network configured to process an input embedding representing the synthesized speech signal 131 to produce an output (e.g., a classifier output or embedding) representing the estimated emotion.

[0051] In a particular embodiment, the control parameter estimator 310 uses audio analysis techniques to determine the average (e.g., average, intermediate, or mode) speech rate detected in the synthesized speech signal 131 and generates one or more estimated control parameters 311 that represent the average speech rate as the estimated speech rate. The control parameter estimator 310 provides one or more estimated control parameters 311 to the loss calculator 312.

[0052] The loss calculator 312 generates a loss metric 313 based on a comparison of one or more control parameters 125 with an estimated control parameter 311. For example, the loss metric 313 may include a person loss metric, an emotion loss metric, a speech rate loss metric, or a combination thereof.

[0053] In a particular embodiment, the target person 143 is represented by a first person embedding vector in a vector space, and the person to be estimated is represented by a second person embedding vector in a vector space. For example, the dimensions of the vector space represent various speech characteristics (e.g., pitch, tone, etc.). The control parameter estimator 310 determines the person loss metric based on the vector distance between the first person embedding vector and the second person embedding vector.

[0054] In a particular embodiment, the target emotion 145 is represented by a first emotion embedding vector in a vector space, and the estimated emotion is represented by a second emotion embedding vector in a vector space. For example, the vector space represents an emotion chart. The control parameter estimator 310 determines the emotion loss metric based on the vector distance between the first emotion embedding vector and the second emotion embedding vector.

[0055] In certain embodiments, the control parameter estimator 310 determines the speech rate loss metric based on the difference between the target speech rate 147 and the estimated speech rate (for example, speech rate loss metric = target speech rate 147 minus estimated speech rate).

[0056] The loss calculator 312 provides the adjuster 314 with a loss metric 313 (e.g., a person loss metric, an emotion loss metric, a speech rate loss metric, or a combination thereof). In response to determining that the loss metric 313 indicates a loss greater than a loss threshold, the adjuster 314 uses machine learning (e.g., neural network training) techniques to train one or more configuration parameters (e.g., neural network weights) of the multi-encoder 192, person encoder 206, emotion encoder 208, speech rate encoder 210, encoder planenet 212, decoder 194, or a combination thereof, to reduce the loss metric 313 for subsequent processing of the input representation.

[0057] In a particular embodiment, the adjuster 314 trains the configuration parameters (e.g., neural network weights) of the person encoder 206, encoder planet 212, person encoder 420 in Figure 4A, masked multi-head attention 466, multi-head attention 478, or a combination thereof, in response to determining that the person loss metric indicates a loss greater than the person loss threshold.

[0058] In a particular embodiment, the adjuster 314 trains the configuration parameters (e.g., neural network weights) of the emotion encoder 208, the encoder planenet 212, the emotion encoder 430 in Figure 4A, the masked multi-head attention 466, the multi-head attention 480, or a combination thereof, in response to determining that the emotion loss metric indicates a loss greater than the emotion loss threshold.

[0059] In a particular embodiment, the adjuster 314 trains the configuration parameters (e.g., neural network weights) of the speech velocity encoder 210, the encoder planet 212, the velocity encoder 440 in Figure 4A, the masked multi-head attention 466, the multi-head attention 482, or a combination thereof, in response to determining that the speech velocity loss metric indicates a loss greater than the speech velocity loss threshold.

[0060] Therefore, the adjuster 314 enables training of the speech generator 120, improving the accuracy of synthesized speech generation in matching the target speech characteristics 141. For example, the adjuster 314 can automatically adjust the configuration parameters of the speech generator 120 (e.g., neural network weights) independently of user input to improve the accuracy of synthesized speech generation.

[0061] Referring to Figure 4A, a specific implementation of the speech generator 120 is shown. The multi-encoder 192 includes an input expression encoder 450 and speech characteristic encoders for each speech characteristic. For example, the multi-encoder 192 includes a person encoder 420, an emotion encoder 430, a velocity encoder 440, or a combination thereof.

[0062] Each encoder in the multi-encoder 192 is configured to generate encoded data by processing input data, independently of the other encoders in the multi-encoder 192, as will be further explained with reference to Figure 5. For example, the input representation encoder 450 is configured to process the input representation 123 to generate the first encoded data 458. The person encoder 420 is configured to process the merged style data 127 to generate person encoded data 428. The emotion encoder 430 is configured to process the merged style data 127 to generate emotion encoded data 438. The velocity encoder 440 is configured to process the merged style data 127 to generate speech encoded data 448. Since each encoder in the multi-encoder 192 operates independently of the other encoders in the multi-encoder 192, each encoder can be specifically adjusted by the adjuster 314 based on the corresponding speech characteristic loss metric, as will be explained with reference to Figure 3, to improve the accuracy of synthesized speech generation.

[0063] Each encoder in the multi-encoder 192 includes an input stage, one or more coding layers, and a normalization layer (layer normalization), as will be further described with reference to Figure 5. For example, the input representation encoder 450 includes an input stage 452, one or more coding layers 454, and a layer normalization 456. The person encoder 420 includes an input stage 422, one or more coding layers 424, and a layer normalization 426. The emotion encoder 430 includes an input stage 432, one or more coding layers 434, and a layer normalization 436. The velocity encoder 440 includes an input stage 442, one or more coding layers 444, and a layer normalization 446.

[0064] In some implementations, encoders 420, 430, 440, and 450 are configured and operate as described with reference to encoder 500 shown in Figure 5. In Figure 5, encoder 500 includes an input stage 522, one or more coding layers 554, and a layer normalizer 526. Each of the one or more coding layers 554 includes an attention layer and a feedforward layer. The attention layer includes an attention network, such as a multi-head attention 564. The feedforward layer includes a feedforward neural network, such as a feedforward 570 (for example, a fully connected feedforward neural network). In a particular example, the attention layer includes a layer normalizer 562 coupled to a combiner 566 via a multi-head attention 564. The feedforward layer includes a layer normalizer 568 coupled to a combiner 572 via a feedforward 570. The attention layer is coupled to the feedforward layer. For example, combiner 566 is coupled to layer normalizer 568 and then to combiner 572.

[0065] One or more coding layers 554, each containing a single coding layer, are provided as exemplary examples. In other examples, one or more coding layers 554 include multiple coding layers, where the output of an input stage 522 is coupled to the attention layer of an initial coding layer (e.g., layer normalization 562), the feedforward layer of each preceding coding layer (e.g., combiner 572) is coupled to the attention layer of a subsequent coding layer (e.g., layer normalization 562), and the feedforward layer of the last coding layer (e.g., combiner 572) is coupled to layer normalization 526.

[0066] The input stage 522 receives the input data 521. In a particular example, the encoder 500 corresponds to the input representation encoder 450 in Figure 4. For example, the input data 521 corresponds to the input representation 123, the input stage 522 corresponds to the input stage 452, one or more coding layers 554 corresponds to one or more coding layers 454, and layer normalization 526 corresponds to layer normalization 456. In a particular implementation, the input stage 452 includes an encoder planet (e.g., a CNN, a linear projection layer, or both) that processes the input representation 123 to produce an output. For example, the encoder planet of the input representation encoder 450 is used to perform a convolution. The input stage 452 is configured to process the output of the encoder planet. For example, the input stage 452 is configured to produce an output by combining the output of the encoder planet with position-coded data.

[0067] In certain embodiments, the encoder 500 corresponds to a person encoder 420, an emotion encoder 430, or a velocity encoder 440, the input data 521 corresponds to merged style data 127, and the input stage 522 is configured to generate an output by processing the merged style data 127. For example, the input stage 522 is configured to generate an output by combining the merged style data 127 with position-encoded data.

[0068] In a specific example, encoder 500 corresponds to person encoder 420. For illustrative purposes, input stage 522 corresponds to input stage 422, one or more coding layers 554 correspond to one or more coding layers 424, and layer normalization 526 corresponds to layer normalization 426. In a specific example, encoder 500 corresponds to emotion encoder 430. For illustrative purposes, input stage 522 corresponds to input stage 432, one or more coding layers 554 correspond to one or more coding layers 434, and layer normalization 526 corresponds to layer normalization 436. In a specific example, encoder 500 corresponds to speed encoder 440. For illustrative purposes, input stage 522 corresponds to input stage 442, one or more coding layers 554 correspond to one or more coding layers 444, and layer normalization 526 corresponds to layer normalization 446.

[0069] The output of the input stage 522 is provided to the multi-head attention 564, following normalization by layer normalization 562. The multi-head attention 564 can construct context vectors from different aspects using different attention heads. For example, the multi-head attention 564 includes attention heads that process inputs to the multi-head attention 564 in parallel. For illustrative purposes, the input is multiplied by a first matrix, a second matrix, and a third matrix to produce a first query vector, a first key vector, and a first value vector, respectively. The first query vector, the first key vector, and the first value vector are processed by the first attention head. The input is multiplied by a fourth matrix, a fifth matrix, and a sixth matrix to produce a second query vector, a second key vector, and a second value vector, respectively. The second query vector, second key vector, and second value vector are processed by the second attention head in parallel with or simultaneously with the first attention head that processes the first query vector, first key vector, and first value vector.

[0070] In certain configurations, the output of the attention head corresponds to the following formula:

[0071]

number

[0072] In the above equation, Z corresponds to the output of the attention head, Q corresponds to the query vector, x corresponds to the multiplication operation, K corresponds to the key vector, V corresponds to the value vector, and d k corresponds to the dimension of the key vector, and softmax corresponds to the normalization operation.

[0073] The independent outputs of the attention heads are concatenated and converted linearly to produce the output of the multi-head attention 564. The combiner 566 produces its output by combining the input to the layer normalizer 562 with the output of the multi-head attention 564.

[0074] The output of combiner 566 is normalized by layer normalization 568 and then provided to feedforward 570 (for example, a fully connected feedforward neural network). In a particular example, feedforward 570 includes a first linear transformation layer coupled to a second linear transformation layer via a normalized linear function (ReLU) layer. Feedforward 570 generates its output by processing the output of combiner 566.

[0075] The combiner 572 generates an output by combining the output of the combiner 566 and the output of the feedforward 570. In certain embodiments, layer normalization 526 generates encoded data 528 by applying normalization to the output of the combiner 572 (e.g., the last encoding layer of one or more encoding layers 554). For example, applying normalization involves adjusting each value of the output of the combiner 572 (e.g., the output vector) to be within a specific range.

[0076] Referring to Figure 4A, the input expression encoder 450 is configured to encode the input expression 123 independently of one or more control parameters 125 to generate first encoded data 458. Each of the person encoder 420, emotion encoder 430, and velocity encoder 440 is configured to encode the input expression 123 based on one or more control parameters 125 to generate encoded data. For example, encoding the input expression 123 based on one or more control parameters 125 includes encoding merged style data 127. For illustrative purposes, the person encoder 420 generates person encoded data 428 by encoding the merged style data 127. The emotion encoder 430 generates emotion encoded data 438 by encoding the merged style data 127. The velocity encoder 440 generates speech velocity encoded data 448 by encoding the merged style data 127.

[0077] In certain embodiments, the person encoder 420, emotion encoder 430, and velocity encoder 440 use different weights or matrices from each other, as used by the corresponding multi-head attention 564. For example, the multi-head attention 564 for the person encoder 420 generates query vectors, key vectors, and value vectors using a first set of matrices representing a first set of weights. The multi-head attention 564 for the emotion encoder 430 generates query vectors, key vectors, and value vectors using a second set of matrices representing a second set of weights. Similarly, the multi-head attention 564 for the velocity encoder 440 generates query vectors, key vectors, and value vectors using a third set of matrices representing a third set of weights.

[0078] The multi-encoder 192 provides encoded data 193 to the decoder 194. For example, the encoded data 193 includes first encoded data 458 independent of one or more control parameters 125. In certain embodiments, the encoded data 193 also includes second encoded data based on one or more control parameters 125 (e.g., person encoded data 428, emotion encoded data 438, speech rate encoded data 448, or a combination thereof).

[0079] The decoder 194 includes a decoder planenet 460, a combiner 462, one or more decoding layers 498, a layer normalizer 492, a linear transformation layer (linear) 494, or a combination thereof. Each of the one or more decoding layers 498 includes a masked attention layer, an attention layer, and a feedforward layer. For example, the masked attention layer includes a layer normalizer 464 coupled to multiple normalization layers (layer normalizers) such as layer normalizer 468, layer normalizer 470, layer normalizer 472, layer normalizer 474, or a combination thereof via a masked multi-head attention 466 (e.g., a masked decoder attention network). The attention layer includes multiple decoder attention networks coupled to the layer normalizer 488 via a combiner 486. For example, multiple decoder attention networks (e.g., a decoder multi-head attention network) include a multi-head attention 478, a multi-head attention 480, a multi-head attention 482, a multi-head attention 484, or a combination thereof. The feedforward layer includes a feedforward 490 (for example, a fully connected feedforward neural network).

[0080] A masked attention layer is coupled to an attention layer. For example, each of the multiple layer normalizations of a masked attention layer is coupled to the corresponding multi-head attention network of the attention layer. For example, layer normalization 468, layer normalization 470, layer normalization 472, and layer normalization 474 are coupled to multi-head attention 478, multi-head attention 480, multi-head attention 482, and multi-head attention 484, respectively. A masked multi-head attention 466 is coupled to layer normalization 488 via combiner 486. Attention layers are coupled to feedforward layers. For example, layer normalization 488 is coupled to feedforward 490.

[0081] The combiner 462 is coupled to one or more decoding layers 498. For example, the combiner 462 is coupled to the layer normalization 464 of the decoding layer. One or more decoding layers 498 is coupled to the layer normalization 492. For example, the feedforward 490 of the decoding layer is coupled to the layer normalization 492.

[0082] In a particular embodiment, the decoder planet 460 receives feedback output spectral data 429 corresponding to the output spectral data 129 generated by the decoder 194 for the previous time step. The decoder planet 460 includes a neural network (e.g., a CNN) that processes the feedback output spectral data 429 to produce an output. In a particular embodiment, the decoder planet 460 performs a convolution. In a particular embodiment, the combiner 462 produces an output by combining the output of the decoder planet 460 with position-coded data 461.

[0083] The output of the combiner 462 is processed by a decoding layer among one or more decoding layers 498. For example, the output of the combiner 462 is normalized by the layer normalization 464 of the decoding layer and then provided to the masked multi-head attention 466. In certain embodiments, the masked multi-head attention 466 masks future positions in the input to the masked multi-head attention 466. The masked multi-head attention 466 generates a query vector, a key vector, and a value vector from the masked version of the input to the masked multi-head attention 466. Each attention head of the masked multi-head attention 466 processes the query vector, key vector, and value vector to generate an output. The independent outputs of the attention heads of the masked multi-head attention 466 are concatenated and linearly transformed to generate the output of the masked multi-head attention 466.

[0084] The output of the masked multi-head attention 466 is provided to the combiner 486, which, after normalization, provides it to multiple multi-head attentions in the decoding layer. For example, the output of the multi-head attention 466 is provided to multi-head attention 478, multi-head attention 480, multi-head attention 482, and multi-head attention 484, respectively, after normalization by layer normalization 468, layer normalization 470, layer normalization 472, and layer normalization 474.

[0085] Each of the multiple multi-head attentions in the decoding layer is configured to process a normalized version of the output of the masked multi-head attention 466 and encoded data received from the corresponding encoder of the multi-encoder 192. For example, each of the multiple multi-head attentions in the decoding layer processes (e.g., generates and processes) a query vector based on the normalized version of the output of the masked multi-head attention 466 and processes (e.g., generates and processes) key and value vectors based on encoded data received from the corresponding encoder of the multi-encoder 192. For illustrative purposes, multi-head attention 484 is configured to process a query vector based on the output of layer normalization 474 and to process key and value vectors based on the first encoded data 458. Multi-head attentions 478, 480, and 482 are configured to process person encoded data 428, emotion encoded data 438, and speech rate encoded data 448, respectively, and to process the outputs of layer normalization 468, layer normalization 470, and layer normalization 472, respectively. For illustrative purposes, the multi-head attention 478 processes a query vector based on the output of the layer normalization 468 and processes key and value vectors based on the person coding data 428.

[0086] The combiner 486 generates an output by combining the output of the masked multi-head attention 466 with the output of the multi-head attention 478, the output of the multi-head attention 480, the output of the multi-head attention 482, the output of the multi-head attention 484, or a combination thereof. The output of the combiner 486 is provided to the feedforward 490 of the decoding layer, following normalization by the layer normalization 488.

[0087] The output of a feedforward 490 of a particular decoding layer of one or more decoding layers 498 is provided to the speech synthesizer 130 as output spectral data 129, following normalization by layer normalization 492 and linear transformation by layer normalization 492. In a particular embodiment, the speech synthesizer 130 performs speech synthesis based on the output spectral data 129, encoded data 193, or a combination thereof, to generate a synthesized speech signal 131. The output spectral data 129 is provided to the decoder planet 460 as feedback output spectral data 429 for subsequent time steps.

[0088] One or more decoding layers 498, each containing a single decoding layer, are provided as exemplary examples. In other examples, one or more decoding layers 498 include multiple decoding layers, where the output of a combiner 462 is coupled to the layer normalization 464 of an initial decoding layer, the feedforward 490 of each preceding decoding layer is coupled to the layer normalization 464 of a subsequent coding layer, and the feedforward 490 of the last decoding layer is coupled to the layer normalization 492. The last coding layer of the multi-encoder 192 provides coded data 193 to each of the multiple multi-head attentions of one or more decoding layers 498.

[0089] In certain embodiments, the adjuster 314 in Figure 3 adjusts the configuration parameters (e.g., neural network weights) of the multi-encoder 192 and decoder 194 based on the loss metric 313. For example, in response to determining that the person loss metric of the loss metric 313 does not meet the person loss threshold, the adjuster 314 trains the neural network weights of one or more coding layers 424 of the person encoder 420, multi-head attention 478, masked multi-head attention 466, feedforward 490 of one or more decoding layers 498, decoder planet 460, or a combination thereof.

[0090] Referring to Figure 4A, a specific implementation of the speech generator 120 is shown. The multi-encoder 192 includes a characteristic encoder 451 for each speech characteristic instead of a separate speech characteristic encoder for each speech characteristic. For example, the characteristic encoder 451 includes an input stage 453, one or more coding layers 455, and a layer normalization 457. In a particular embodiment, the encoder 500 in Figure 5 is an example of an implementation of the characteristic encoder 451. For example, the input stage 522 corresponds to the input stage 453, one or more coding layers 554 correspond to one or more coding layers 455, and the layer normalization 526 corresponds to the layer normalization 457.

[0091] The characteristic encoder 451 receives merged style data 127 (e.g., input embedding vectors) from the style data generator 126. In a particular embodiment, the input stage 453 generates an output by combining the merged style data 127 with positional encoded data. The output of the input stage 453 is processed by one or more encoding layers 455, as described with reference to one or more encoding layers 554 in Figure 5. The output of one or more encoding layers 455 is normalized by layer normalization 457 and then provided to the decoder 194 as second encoded data 459. For example, the encoded data 193 includes first encoded data 458 and second encoded data 459.

[0092] One or more decoding layers 498 include a masked multi-head attention 466 coupled to two multi-head attentions via a normalization layer (layer normalization). For example, the masked multi-head attention 466 is coupled to a multi-head attention 485 via layer normalization 475 and to a multi-head attention 484 via layer normalization 474. The combiner 486 is coupled to the masked multi-head attention 466, the multi-head attention 485, and the multi-head attention 484.

[0093] The output of the masked multi-head attention 466 is provided to the multi-head attention 485 after being normalized by layer normalization 475. The characteristic encoder 451 provides the multi-head attention 485 with second encoded data 459. The multi-head attention 485 generates a query vector based on the output of the masked multi-head attention 466 and generates a key vector and a value vector based on the second encoded data 459. Each attention head of the multi-head attention 485 processes the query vector, key vector, and value vector to generate an output. The outputs of each attention head of the multi-head attention 485 are concatenated and linearly transformed to generate the output of the multi-head attention 485. The combiner 486 generates an output by combining the output of the masked multi-head attention 466 with the output of the multi-head attention 485, the output of the multi-head attention 484, or both.

[0094] In certain embodiments, the adjuster 314 in Figure 3 adjusts the configuration parameters (e.g., neural network weights) of the multi-encoder 192 and decoder 194 based on the loss metric 313. For example, in response to determining that any of the person loss metric, emotion loss metric, or speech rate loss metric of the loss metric 313 does not meet the corresponding loss threshold, the adjuster 314 trains the neural network weights of one or more coding layers 455 of the characteristic encoder 451, multi-head attention 485, masked multi-head attention 466, feedforward 490 of one or more decoding layers 498, decoder planet 460, or a combination thereof.

[0095] Referring to Figure 6, an exemplary mode of operation of the components of system 100 in Figure 1 is shown, which is collectively designated as 600. The input signal 121 includes a plurality of audio samples 602, for example, one or more additional sets of audio samples such as audio sample 602A, audio sample 602B, the Nth audio sample 602N (where N is an integer greater than 2), or a combination thereof. The audio samples 602 represent the speech of a particular person (for example, person 180), a particular emotion (for example, calm or bored), a particular speech rate (for example, medium speed), or a combination thereof.

[0096] The speech generator 120 receives (e.g., accesses) one or more control parameters 621. One or more control parameters 621 indicate a target person 623 (e.g., person 180), a target emotion 625 (e.g., excitement), a target speech rate 627 (e.g., fast speech rate), or a combination thereof. One or more control parameters 621 indicate the speech characteristics of the synthesized speech signal 661 produced by the speech generator 120 based on the input signal 121. In a particular example, at least one of the target person 623, target emotion 625, and target speech rate 627 differs from a specific person, a specific emotion, and a specific speech rate, respectively. The speech generator 120 processes the input signal 121 based on one or more control parameters 621 to produce the synthesized speech signal 661, as described with reference to Figure 1. The synthesized speech signal 661 includes a synthesized audio sample 629 based on the target person 623, target emotion 625, target speech rate 627, or a combination thereof. For example, the synthesized audio sample 629 represents an utterance having similar speech characteristics to the target person 623 (e.g., person 180), similar speech characteristics associated with the target emotion 625 (e.g., excitement) (e.g., pitch or tone), the target speech rate 627 (e.g., fast speech rate), or a combination thereof. In a particular embodiment, the speech generator 120 (or another component of device 102) adds the audio sample 602 and the synthesized audio sample 629 to the training dataset 665.

[0097] In certain embodiments, the speech generator 120 generates multiple sets of audio samples from the same input signal 121, each having different speech characteristics corresponding to various control parameters. For example, the speech generator 120 receives (e.g., accesses) one or more control parameters 631 indicating a target person 633 (e.g., person 180), a target emotion 635 (e.g., fatigue), a target speech rate 637 (e.g., slow speech rate), or a combination thereof. The speech generator 120 processes the input representation 121 based on one or more control parameters 631 to generate a synthesized speech signal 663, as described with respect to Figure 1. The synthesized speech signal 663 includes synthesized audio samples 639 based on a target person 633 (e.g., person 180), a target emotion 635 (e.g., fatigue), a target speech characteristic 637 (e.g., slow speech rate), or a combination thereof. In certain embodiments, the speech generator 120 (or another component of device 102) adds the synthesized audio samples 639 to a training dataset 665.

[0098] The network trainer 670 uses machine learning techniques to train the speech recognition network 672 based on the training dataset 665. Including synthesized audio samples 629 and 639 (in addition to audio sample 602) in the training dataset 665 improves the accuracy of the speech recognition network 672 in recognizing utterances. For example, the speech recognition network 672 (trained on synthesized audio samples 629 and 639) is more robust in recognizing utterances of person 180 that have different characteristics (e.g., fast or slow, and fatigued or excited) than the characteristics of the utterance provided by person 180 in the input signal 121 (e.g., calm).

[0099] Figure 7 is a diagram illustrating an exemplary mode of operation of the components of the system in Figure 1. The audio analyzer 122 is configured to receive a sequence of audio data samples 710, such as a sequence of continuously captured frames of the input signal 121, shown as one or more additional frames (where N is an integer greater than 2), including a first frame (F1) 712, a second frame (F2) 714, and an Nth frame (FN) 716. The audio analyzer 122 is configured to output a sequence of input representation data sets 720, such as a sequence of sets of input representations 123, including a first set 722, a second set 724, and an Nth set 726.

[0100] The style data generator 126 is configured to receive a sequence 720 of input representation data sets and generate a sequence 730 of merged style data sets based on sequence 720 and one or more control parameters 125. Sequence 730 includes one or more additional sets, including a first set (M1) 732, a second set (M2) 734, and an Nth set (MN) 736. The multi-encoder 192 is configured to receive sequences 720 and 730 and generate a sequence 740 of data encoding sets, including one or more additional encoding sets, such as a first encoding set (E1) 742, a second encoding set (E2) 744, and an Nth encoding set (EN) 746. The decoder 194 is configured to receive a sequence 740 of encoding sets and generate a sequence 750 of spectral data, including one or more additional sets of spectral data, including a first spectral data (S1) 752, a second spectral data (S2) 754, and an Nth spectral data (SN) 756.

[0101] During operation, the audio analyzer 122 processes the first frame (F1) to generate a first set 722 of the input representation 123, and the style data generator 126 processes the first set 722 to generate a first set (M1) 732 of the merged style data 127. The multi-encoder 192 processes the first set 722 of the input representation 123 and the first set (M1) 732 of the merged style data 127 to generate a first coded set (E1) of the coded data 193. The decoder 194 processes the first coded set (E1) of the coded data 193 to generate a first spectral data (S1) 752 of the output spectral data 129. Subsequent coded sets (e.g., a second coded set (E2)) are processed by the decoder 194 at least in part based on the first spectral data (S1) 752. Such processing continues with the audio analyzer 122 processing the nth frame 716 to generate the nth set 726 of the input representation 123; the style data generator 126 processing the nth set 726 of the input representation 123 to generate the nth set (MN) 736 of the merged style data 127; the multi-encoder 192 processing the nth set 726 of the input representation 123 and the nth set (MN) 736 of the merged style data 127 to generate the nth coded set (EN) 746 of the coded data 193; and the decoder 194 processing the nth coded set (EN) 746 of the coded data 193 to generate the nth spectral data (SN) 756. The nth spectral data (SN) 756 is based on the nth set 726 of the feature data and is at least partially based on one or more of the previous sets of feature data from sequence 720. By decoding based on one or more previous sets of spectral data, the accuracy of synthesized speech generation by decoder 194 may be improved for audio signals that may span multiple frames of audio data.

[0102] Figure 8 shows one implementation form 800 of device 102 as an integrated circuit 802 including one or more processors 190. The integrated circuit 802 also includes an audio input 804, such as one or more bus interfaces, to enable the reception of an input signal 121 for processing. The integrated circuit 802 also includes an audio output 806, such as a bus interface, to enable the transmission of an output signal, such as a synthesized speech signal 131. The integrated circuit 802 enables the implementation of synthesized speech generation as a component in a system including a microphone, such as a mobile phone or tablet as shown in Figure 9, a headset as shown in Figure 10, a wearable electronic device as shown in Figure 11, a voice-controlled speaker system as shown in Figure 12, a camera as shown in Figure 13, a virtual reality headset or augmented reality headset as shown in Figure 14, or a mobile device as shown in Figure 15 or Figure 16.

[0103] Figure 9 shows an implementation form 900 in which device 102 includes a mobile device 902, such as a telephone or tablet, as an exemplary and non-restrictive example. The mobile device 902 includes a microphone 182 and a display screen 904. Components of the processor 190, including a speech generator 120, are shown using dotted lines to indicate internal components that are incorporated into the mobile device 902 and are generally not visible to the user of the mobile device 902. In a particular example, the speech generator 120 operates to produce synthesized speech (or the speech recognition network 672 in Figure 6 operates to recognize utterances), and the synthesized speech is then processed to perform one or more actions on the mobile device 902, such as launching a graphical user interface or, optionally, displaying other information related to the user's utterances on the display screen 904 (for example, via an integrated "Smart Assistant" application).

[0104] Figure 10 shows an implementation configuration 1000 in which device 102 includes a headset device 1002. The headset device 1002 includes a microphone 182. Components of the processor 190, including a speech generator 120, are incorporated into the headset device 1002. In a particular example, the speech generator 120 operates to produce synthesized speech (or the speech recognition network 672 in Figure 6 operates to recognize speech), thereby causing the headset device 1002 to perform one or more operations on the headset device 1002, send audio data corresponding to the synthesized speech to a second device (not shown) for further processing, or a combination thereof.

[0105] Figure 11 shows one implementation form 1100, which includes a wearable electronic device 1102, indicated as a “smartwatch” in device 102. A voice generator 120 and a microphone 182 are incorporated into the wearable electronic device 1102. In a particular example, the voice generator 120 operates to produce synthesized speech (or the speech recognition network 672 in Figure 6 operates to recognize speech), and the synthesized speech is then processed to perform one or more actions on the wearable electronic device 1102, such as invoking a graphical user interface or, optionally, displaying other information related to the user’s speech on the display screen 1104 of the wearable electronic device 1102. For illustrative purposes, the wearable electronic device 1102 may include a display screen configured to display notifications based on user speech detected by the wearable electronic device 1102. In a particular example, the wearable electronic device 1102 includes a haptic device that provides haptic notifications (e.g., vibrates) in response to the detection of user voice activity or the generation of synthesized speech. For example, haptic notification can cause the user to look at the wearable electronic device 1102 and see a displayed notification indicating that a keyword spoken by the user has been detected. Thus, the wearable electronic device 1102 can alert a user with hearing impairment or a user wearing a headset that the user's voice activity has been detected.

[0106] Figure 12 shows an implementation configuration 1200 in which device 102 includes a wireless speaker and a voice-activated device 1202. The wireless speaker and voice-activated device 1202 may have wireless network connectivity and is configured to perform assistant actions. A processor 190 including a speech generator 120, a microphone 182, or a combination thereof is included in the wireless speaker and voice-activated device 1202. The wireless speaker and voice-activated device 1202 also includes a speaker 1204. During operation, the wireless speaker and voice-activated device 1202 may perform assistant actions, such as executing an integrated assistant application, in response to receiving verbal commands and generating synthesized speech via the operation of the speech generator 120 (or recognizing utterances via the operation of the speech recognition network 672 in Figure 6). Assistant actions may include adjusting the temperature, playing music, turning on lights, etc. For example, an assistant action may be performed in response to receiving a command following a keyword or key phrase (e.g., "Hello, Assistant").

[0107] Figure 13 shows one implementation configuration 1300 in which device 102 includes a portable electronic device corresponding to camera device 1302. A voice generator 120, a microphone 182, or a combination thereof are included in camera device 1302. During operation, camera device 1302 can receive verbal commands and, in response to generating synthesized speech via the operation of voice generator 120 (or recognizing speech via the operation of the speech recognition network 672 in Figure 6), perform actions in response to spoken user commands, such as adjusting image or video capture settings, image or video playback settings, or image or video capture commands, as exemplary examples.

[0108] Figure 14 shows one implementation form 1400, in which device 102 includes a portable electronic device corresponding to a virtual reality, augmented reality, or mixed reality headset 1402. A voice generator 120, a microphone 182, or a combination thereof is incorporated into the headset 1402. In certain embodiments, the headset 1402 includes a microphone 182 positioned primarily to capture user utterances. Synthetic speech generation can be performed based on the audio signal received from the microphone 182 of the headset 1402. A visual interface device is positioned in front of the user's eyes to enable the display of augmented reality or virtual reality images or scenes to the user while the headset 1402 is being worn. In certain examples, the visual interface device is configured to display notifications indicating user utterances detected in the audio signal.

[0109] Figure 15 shows an implementation configuration 1500 in which device 102 corresponds to or is incorporated within a mobile body 1502, which is shown as a manned or unmanned aerial device (e.g., a delivery drone). A voice generator 120, a microphone 182, or a combination thereof is incorporated into the mobile body 1502. Synthesized speech generation can be performed based on audio signals received from the microphone 182 of the mobile body 1502, such as delivery orders from an authorized user of the mobile body 1502.

[0110] Figure 16 shows another implementation form 1600 in which device 102 corresponds to or is incorporated into a mobile body 1602, shown as a car. The mobile body 1602 includes a processor 190, which includes a speech generator 120. The mobile body 1602 also includes a microphone 182. The microphone 182 is positioned to capture the utterances of the operator of the mobile body 1602. Synthetic speech generation can be performed based on audio signals received from the microphone 182 of the mobile body 1602. In some implementation forms, synthetic speech generation can be performed based on audio signals received from an internal microphone (e.g., microphone 182), such as for voice commands from an authorized occupant. For example, synthetic speech generation can be used to detect voice commands from the operator of the mobile body 1602 (e.g., setting the volume to 5 or setting a destination for an autonomous vehicle). In some implementations, synthesized speech generation can be performed based on an audio signal received from an external microphone (e.g., microphone 182), such as from an authorized user of the mobile device. In certain implementations, the voice activation system 162 receives a verbal command and, in response to generating synthesized speech via the operation of the speech generator 120, initiates one or more actions of the mobile device 1602 based on one or more keywords detected in the synthesized speech signal 131 (e.g., "unlock," "start engine," "play music," "show weather forecast," or another voice command), such as by providing feedback or information via a display or one or more speakers (e.g., speaker 1610).

[0111] Referring to Figure 17, a specific implementation of the synthesized speech generation method 1700 is shown. In a particular embodiment, one or more operations of the method 1700 are performed by at least one of the following: the style data generator 126, multiencoder 192, decoder 194, multiencoder transformer 128, speech generator 120, one or more processors 190, device 102, system 100, the person encoder 206, emotion encoder 208, speech rate encoder 210, encoder planet 212 in Figure 2, the person encoder 420, emotion encoder 430, rate encoder 440, input expression encoder 450 in Figure 4A, the characteristic encoder 451 in Figure 4B, the encoder 500 in Figure 5, or a combination thereof.

[0112] Method 1700 includes the step in 1702 of receiving one or more control parameters that represent a target utterance characteristic. For example, the style data generator 126 in Figure 1 receives one or more control parameters 125 that represent a target utterance characteristic 141, as will be explained with reference to Figure 1.

[0113] Method 1700 also includes the step in 1704 of processing an input representation of an utterance based on one or more control parameters to generate encoded data corresponding to an audio signal representing a version of the utterance based on a target utterance characteristic using a multiencoder. For example, the multiencoder 192 processes the input representation 123 based on one or more control parameters 125 to generate encoded data 193 corresponding to a synthesized speech signal 131 representing a version of the utterance based on a target utterance characteristic 141, as described with reference to Figure 1.

[0114] Method 1700 enables the generation of multiple synthesized speech signals corresponding to various emotions, different speech rates, and different speech characteristics of different individuals, based on the same input signal 121. A large and diverse set of synthesized speech datasets can be generated using a relatively small number of utterances.

[0115] The method 1700 in Figure 17 may be implemented by a processing unit such as a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a central processing unit (CPU), a DSP, a controller, another hardware device, a firmware device, or any combination thereof. For example, the method 1700 in Figure 17 may be implemented by an instruction-executing processor, as described with reference to Figure 19.

[0116] Referring to Figure 18, a specific implementation of the synthesized speech generation method 1800 is shown. In a particular embodiment, one or more operations of the method 1800 are performed by at least one of the following: the style data generator 126, multiencoder 192, decoder 194, multiencoder transformer 128, speech generator 120, speech synthesizer 130, one or more processors 190, device 102, system 100, the person encoder 206, emotion encoder 208, speech rate encoder 210, encoder planet 212 in Figure 2, the person encoder 420, emotion encoder 430, rate encoder 440, input expression encoder 450 in Figure 4A, the characteristic encoder 451 in Figure 4B, the encoder 500 in Figure 5, or a combination thereof.

[0117] Method 1800 includes the step in 1802 of receiving one or more control parameters that represent a target utterance characteristic. For example, the style data generator 126 in Figure 1 receives one or more control parameters 125 that represent a target utterance characteristic 141, as will be explained with reference to Figure 1.

[0118] Method 1800 also includes the step of generating merged style data based on one or more control parameters in 1804. For example, the style data generator 126 in Figure 1 generates merged style data based on one or more control parameters 125, as described with reference to Figure 1.

[0119] Method 1800 further includes the step in 1806 of encoding an input representation independently of one or more control parameters in the first encoder of the multiencoder to generate first encoded data. For example, as described with reference to Figure 4A, the input representation encoder 450 of the multiencoder 192 encodes an input representation 123 independently of one or more control parameters 125 to generate first encoded data 458.

[0120] Method 1800 also includes the step in 1808 of encoding an input representation based on one or more control parameters in one or more second encoders of a multiencoder to generate second encoded data. For example, the person encoder 420, emotion encoder 430, and velocity encoder 440 in Figure 4A encode an input representation 123 based on one or more control parameters 125 to generate second encoded data (e.g., person encoded data 428, emotion encoded data 438, and speech velocity encoded data 448), as described with reference to Figure 4A. As another example, the characteristic encoder 451 in Figure 4B encodes an input representation 123 based on one or more control parameters 125 to generate second encoded data 459, as described with reference to Figure 4B.

[0121] Method 1800 further includes the step in 1810 of generating output spectral data based on first encoded data and second encoded data in a decoder network of decoders. For example, decoder 194 (including multiple decoder networks such as multi-head attention 478, multi-head attention 480, multi-head attention 482, and multi-head attention 484 in Figure 4A, and multi-head attention 485 in Figure 4B) generates output spectral data 129 based on encoded data 193, as described with reference to Figure 1 and Figures 4A and 4B.

[0122] Method 1800 also includes the step in 1812 of generating an audio signal in a speech synthesizer based on output spectral data, representing a version of the utterance based on a target utterance characteristic. For example, the speech synthesizer 130 in Figure 1 generates a synthesized speech signal 131 based on output spectral data 129, representing a version of the utterance based on a target utterance characteristic 141, as described with reference to Figure 1.

[0123] Method 1800 enables the generation of multiple synthesized speech signals corresponding to various emotions, different speech rates, and different speech characteristics of different individuals, based on the same input signal 121. A large and diverse set of synthesized speech datasets can be generated using a relatively small number of utterances.

[0124] The method 1800 in Figure 18 may be implemented by a processing unit such as a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a central processing unit (CPU), a DSP, a controller, another hardware device, a firmware device, or any combination thereof. For example, the method 1800 in Figure 18 may be implemented by an instruction-executing processor, as described with reference to Figure 19.

[0125] Referring to Figure 19, a block diagram of a specific exemplary implementation of the device is shown, designated as 1900 overall. In various implementations, device 1900 may have more or fewer components than shown in Figure 19. In the exemplary implementation, device 1900 may correspond to device 102. In the exemplary implementation, device 1900 may perform one or more operations described with reference to Figures 1 to 18.

[0126] In certain implementations, device 1900 includes a processor 1906 (e.g., a central processing unit (CPU)). Device 1900 may also include one or more additional processors 1910 (e.g., one or more DSPs). In certain embodiments, one or more processors 190 in Figure 1 correspond to processor 1906, processor 1910, or a combination thereof. Processor 1910 may include a speech and music coder-decoder (codec) 1908, which may include a voice coder ("vocoder") encoder 1936, a vocoder decoder 1938, a speech generator 120, a network trainer 670, or a combination thereof.

[0127] Device 1900 may include memory 132 and codec 1934. Memory 132 may include instructions 1956 that can be executed by one or more additional processors 1910 (or processor 1906) to implement the functions described with reference to the voice generator 120, network trainer 670, or both. Device 1900 may include modem 1970 coupled to antenna 1952 via transceiver 1950.

[0128] Device 1900 may include a display 1928 coupled to a display controller 1926. A speaker 1992, a microphone 182, or both may be coupled to a codec 1934. The codec 1934 may include a digital-to-analog converter (DAC) 1902, an analog-to-digital converter (ADC) 1904, or both. In certain implementations, the codec 1934 may receive an analog signal from the microphone 182, convert the analog signal to a digital signal using the analog-to-digital converter 1904, and provide the digital signal to a speech and music codec 1908. The speech and music codec 1908 may process the digital signal, which may be further processed by a speech generator 120. In certain implementations, the speech and music codec 1908 may provide the digital signal to the codec 1934. Codec 1934 may use a digital-to-analog converter 1902 to convert the digital signal to an analog signal, and may provide the analog signal to speaker 1992.

[0129] In certain implementations, device 1900 may be included in a system-in-package or system-on-chip device 1922. In certain implementations, memory 132, processor 1906, processor 1910, display controller 1926, codec 1934, and modem 1970 are included in a system-in-package or system-on-chip device 1922. In certain implementations, input device 1930 and power supply 1944 are coupled to the system-on-chip device 1922. Furthermore, in certain implementations, as shown in Figure 19, the display 1928, input device 1930, speaker 1992, microphone 182, antenna 1952, and power supply 1944 are located outside the system-on-chip device 1922. In certain implementations, each of the display 1928, input device 1930, speaker 1992, microphone 182, antenna 1952, and power supply 1944 may be coupled to a component of the system-on-chip device 1922, such as an interface or controller.

[0130] Device 1900 may include virtual assistants, home appliances, smart devices, Internet of Things (IoT) devices, communication devices, headsets, mobile devices, computers, display devices, televisions, gaming consoles, music players, radios, video players, entertainment units, personal media players, digital video players, cameras, navigation devices, smart speakers, speaker covers, mobile communication devices, smartphones, cellular phones, laptop computers, tablets, personal digital assistants, digital video disc (DVD) players, tuners, augmented reality headsets, virtual reality headsets, aircraft, home automation systems, voice-activated devices, wireless speakers and voice-activated devices, portable electronic devices, automobiles, mobile devices, computing devices, virtual reality (VR) devices, base stations, mobile devices, or any combination thereof.

[0131] In relation to the described implementation, the apparatus includes means for receiving one or more control parameters that represent target speech characteristics. For example, the means for receiving include the style data generator 126 in Figure 1, the speech generator 120, one or more processors 190, device 102, the transceiver 1950 in Figure 19, the modem 1970, the input device 1930, the input interface of device 1900, one or more other circuits or components configured to receive one or more control parameters, or any combination thereof.

[0132] The apparatus also includes means for processing an input representation of an utterance based on one or more control parameters to generate encoded data corresponding to an audio signal representing a version of the utterance based on target utterance characteristics, using a multiencoder. For example, the means for processing include the multiencoder 192, multiencoder transformer 128, speech generator 120, one or more processors 190, device 102, the utterance and music codec 1908, processor 1906, device 1900, one or more other circuits or components configured to process an input representation of an utterance based on one or more control parameters, or any combination thereof.

[0133] In some implementations, a non-temporary computer-readable medium (e.g., a computer-readable storage device such as memory 132) includes an instruction (e.g., instruction 1956) that, when executed by one or more processors (e.g., one or more processors 1910 or processor 1906), causes one or more processors to receive one or more control parameters (e.g., one or more control parameters 125) that indicate a target speech characteristic (e.g., target speech characteristic 141). The instruction, when executed by one or more processors, causes one or more processors to process an input representation of an utterance (e.g., input representation 123) based on the control parameters (e.g., one or more control parameters 125) to generate encoded data (e.g., encoded data 193) corresponding to an audio signal (e.g., synthesized speech signal 131) representing a version of the utterance based on the target speech characteristic (e.g., target speech characteristic 141) using a multi-encoder (e.g., multi-encoder 192).

[0134] Specific aspects of this disclosure are described below in the first set of interrelated clauses.

[0135] According to Clause 1, a device for speech generation includes one or more processors configured to receive one or more control parameters indicating a target utterance characteristic and to process an input representation of an utterance based on one or more control parameters, using a multiencoder, to generate encoded data corresponding to an audio signal representing a version of the utterance based on the target utterance characteristic.

[0136] Clause 2 includes the device described in Clause 1, wherein the control parameters indicate the target person, target emotion, target speech rate, or combination thereof, on which the speech characteristics are used.

[0137] Clause 3 further comprises the device described in Clause 1 or Clause 2, wherein one or more processors are configured to generate merged style data based on one or more control parameters, and the merged style data is used by a multiencoder while processing an input representation.

[0138] Clause 4 includes a device described in any one of Clauses 1 to 3, wherein the multiencoder includes a first encoder configured to encode an input representation independently of one or more control parameters to generate first encoded data, and one or more second encoders configured to encode an input representation based on one or more control parameters to generate second encoded data, wherein the encoded data includes the first encoded data and the second encoded data.

[0139] Clause 5 further includes the device described in Clause 4, wherein one or more processors are configured to process an input expression in an utterance characteristic encoder based on at least one of one or more control parameters to generate an encoded input utterance expression, and in an encoder pre-network, generate style data merged at least in part on the encoded input utterance expression, provide the input expression to a first encoder to generate first encoded data, and provide the input expression and merged style data to one or more second encoders to generate target second encoded data.

[0140] Clause 6 further includes a multi-encoder transformer including a multi-encoder and a decoder, wherein the first encoder includes a first attention network, each of one or more second encoders includes a second attention network, and the decoder includes a decoder attention network, as described in Clause 4 or Clause 5.

[0141] Clause 7 includes the device described in Clause 6, wherein the first encoder comprises a first layer including a first attention network, the first attention network corresponding to a first multi-head attention network, and a second layer including a first neural network, and each of one or more second encoders comprises a first layer including a second attention network, the second attention network comprising a first layer corresponding to a second multi-head attention network, and a second layer including a second neural network.

[0142] Clause 8 includes the device described in any one of Clauses 4 to 7, further comprising a decoder coupled to a multiencoder, the decoder including a decoder network configured to generate output spectral data based on first encoded data and second encoded data of a target, and a speech synthesizer configured to generate an audio signal representing a version of speech based on target speech characteristics based on the output spectral data.

[0143] Clause 9 includes the device described in Clause 8, wherein the decoder network includes a decoder attention network comprising a first multi-head attention network configured to process first encoded data, one or more second multi-head attention networks configured to process target second encoded data, and a combiner configured to combine the outputs of the first multi-head attention network and one or more second multi-head attention networks.

[0144] Clause 10 includes the device described in Clause 9, wherein the decoder further comprises a masked multi-head attention network coupled to the input of a decoder attention network and a decoder neural network coupled to the output of the decoder attention network.

[0145] Clause 11 includes the device described in any one of Clauses 1 to 10, wherein one or more processors are further configured to generate one or more estimated control parameters from an audio signal and to train one or more neural network weights of a multi-encoder, one or more audio modification characteristic encoders, encoder pre-networks, decoder networks, or combinations thereof, based on a comparison of one or more control parameters with one or more estimated control parameters.

[0146] Clause 12 includes the device described in any one of Clauses 1 to 11, wherein one or more processors are further configured to receive an input audio signal and generate an input representation based on the input audio signal.

[0147] Clause 13 includes the device described in any one of Clauses 1 to 11, wherein one or more processors are further configured to receive an input representation.

[0148] Clause 14 includes devices described in any one of Clauses 1 to 13, wherein the input representation includes text, Mel-scale spectrograms, fundamental frequency (F0) features, or a combination thereof.

[0149] Specific aspects of this disclosure are described below in the second set of interrelated clauses.

[0150] According to Clause 15, a speech generation method includes, in a device, receiving one or more control parameters that indicate a target utterance characteristic, and processing an input representation of an utterance based on one or more control parameters using a multiencoder to generate encoded data corresponding to an audio signal representing a version of the utterance based on the target utterance characteristic.

[0151] Clause 16 includes the method described in Clause 15, wherein the control parameters indicate the target person, target emotion, target speech rate, or combination thereof, on which the speech characteristics are used.

[0152] Clause 17 further includes the step of generating merged style data in a device based on one or more control parameters, wherein the merged style data is used by a multiencoder while processing an input representation, and includes the method of Clause 15 or Clause 16.

[0153] Clause 18 further includes the steps of encoding an input representation in a first encoder of a multiencoder independently of one or more control parameters to generate first encoded data, and encoding an input representation in one or more second encoders of the multiencoder based on one or more control parameters to generate target second encoded data, wherein the audio signal contained in the encoded data includes the method of any one of Clauses 15 to 17, based on the first encoded data and the second encoded data.

[0154] Clause 19 further includes the method of Clause 18, wherein in an utterance characteristic encoder, the steps are: processing an input representation based on at least one of one or more control parameters to generate an encoded input utterance representation; in an encoder pre-network, the steps are: generating style data merged at least in part on the encoded input utterance representation; providing the input representation to a first encoder to generate first encoded data; and providing the input representation and merged style data to one or more second encoders to generate target second encoded data.

[0155] Clause 20 includes the method of Clause 18, wherein the multi-encoder and decoder are included in the multi-encoder transformer, the first encoder includes a first attention network, each of one or more second encoders includes a second attention network, and the decoder includes a decoder attention network.

[0156] Clause 21 includes the method according to Clause 20, wherein a first encoder comprises a first layer including a first attention network, the first attention network comprising a first layer corresponding to a first multi-head attention network and a second layer including a first neural network, and each of one or more second encoders comprises a first layer including a second attention network, the second attention network comprising a first layer corresponding to a second multi-head attention network and a second layer including a second neural network.

[0157] Clause 22 further includes the method of Clause 18, comprising the steps of generating output spectral data based on first encoded data and second encoded data of a target in a decoder's decoder network, and generating an audio signal representing a version of speech based on target speech characteristics in a speech synthesizer based on the output spectral data.

[0158] Clause 23 further includes the method of Clause 22, comprising the steps of processing first encoded data in a first multi-head attention network of a decoder attention network, wherein the decoder network includes a decoder attention network; processing target second encoded data in one or more second multi-head attention networks of the decoder attention network; and combining the outputs of the first multi-head attention network and one or more second multi-head attention networks in a combiner.

[0159] Clause 24 includes the method described in any one of Clauses 15 to 23, further comprising the steps of generating one or more estimated control parameters from an audio signal in a device, and training one or more neural network weights of a multi-encoder, one or more audio modification characteristic encoders, encoder pre-networks, decoder networks, or combinations thereof, based on a comparison of one or more control parameters with one or more estimated control parameters.

[0160] Clause 25 includes the method of any one of Clauses 15 to 24, further comprising the steps of: receiving an input audio signal in the device; and generating an input representation based on the input audio signal in the device.

[0161] Clause 26 includes the method described in any one of Clauses 15 to 24, further comprising the step of receiving an input representation in the device.

[0162] Specific aspects of this disclosure are described below in the third set of interrelated clauses.

[0163] According to Article 27, a non-temporary computer-readable storage medium for storing instructions, wherein when an instruction is executed by one or more processors, one or more processors cause one or more processors to receive one or more control parameters indicating a target utterance characteristic, and to process an input representation of an utterance based on one or more control parameters using a multiencoder, and generate encoded data corresponding to an audio signal representing a version of the utterance based on the target utterance characteristic.

[0164] Clause 28 includes non-temporary computer-readable storage media as described in Clause 27, in which the input representation includes text, Mel-scale spectrograms, fundamental frequency (F0) features, or a combination thereof.

[0165] Specific aspects of this disclosure are described below in the fourth set of interrelated clauses.

[0166] According to Article 29, an apparatus comprising means for receiving one or more control parameters indicating a target utterance characteristic, and means for processing an input representation of an utterance based on one or more control parameters using a multiencoder to generate encoded data corresponding to an audio signal representing a version of the utterance based on the target utterance characteristic.

[0167] Clause 30 includes the apparatus of Clause 29, in which the means for receiving and processing are incorporated into at least one of the following: a virtual assistant, a consumer electronics appliance, a smart device, an Internet of Things (IoT) device, a communication device, a headset, a mobile device, a computer, a display device, a television, a gaming console, a music player, a radio, a video player, an entertainment unit, a personal media player, a digital video player, a camera, or a navigation device.

[0168] Those skilled in the art will further understand that various exemplary logic blocks, configurations, modules, circuits, and algorithmic steps described herein with respect to the implementation forms disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or a combination of both. Various exemplary components, blocks, configurations, modules, circuits, and steps have been described above in relation to their functions. Whether such functions are implemented as hardware or as processor-executable instructions depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functions in various ways for each specific application, and such decisions should not be construed as causing a departure from the scope of this disclosure.

[0169] Steps of methods or algorithms described in relation to the implementations disclosed herein may be embodied directly in hardware, in software modules executed by a processor, or in a combination of both. The software modules may reside in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, compact disk read-only memory (CD-ROM), or any other form of non-temporary storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor. The processor and storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computing device or user terminal. Alternatively, the processor and storage medium may reside as separate components within a computing device or user terminal.

[0170] The above description of the disclosed embodiments is provided to enable those skilled in the art to make or use the disclosed embodiments. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the principles set forth herein may be applied to other embodiments without departing from the scope of this disclosure. Accordingly, this disclosure is not intended to be limited to the embodiments shown herein, but should be given the broadest possible scope, consistent with the principles and novel features set forth by the following claims. [Explanation of symbols]

[0171] 100 Systems 102 devices 120 Voice Generators 121 Input Signals 122 Audio Analyzer 123 Input Representation 125 Control Parameters 126 Style Data Generator 127 Merged style data 128 Multi-Encoder Transformer 129 Output Spectrum Data 130 Speech Synthesizers 131 Synthesized speech signal 132 memory 141 Target utterance characteristics 143 Target Person 145 Target Emotions 147 Target speech rate Set of 151 characters 153 The first person 155 The second person 157 The Nth Person 161 Set of Emotions 163 The first emotion 165 The second emotion 167 The Nth Emotion 171 Speech speed set 173 First speech rate 175 Second speech rate 177 The Nth speech rate 180 people 182 Microphone 184 Audio Repository 190 processors 192 Multi-Encoder 193 Encoded data 194 Decoder 206 People Encoder 207 Human Encoded Input Representation 208 Emotion Encoders 209 Emotion-Encoded Input Representation 210 speech rate encoder 211 Speech rate coded input representation 212 Encoder Plane 310 Control parameter estimator 311 Estimated control parameters 312 Loss Calculator 313 Loss Metrics 314 Adjuster 420 People Encoder 422 Input Stages 424 coding layer 426 layer normalization 428 Person encoded data 429 Feedback output spectral data 430 Emotion Encoders 432 Input Stages 438 Emotion-Coding Data 440 Speed ​​Encoder 442 Input Stages 444 Coding layer 446 layer normalization 448 speech rate coded data 450 Input Representation Encoders 451 Characteristic Encoder 452 Input Stages 453 Input Stage 454 Coding layer 455 Coding layer 456 layer normalization 457 layer normalization 458 First encoded data 459 Second encoded data 460 Decoder Plane 461 Location-coded data 462 Combiner 464 layer normalization 466 Masked Multi-Head Attention 468 layer normalization 470 layer normalization 472 layer normalization 474 layer normalization 475 layer normalization 478 Multi-Head Attention 480 Multi-Head Attention 482 Multi-Head Attention 484 Multi-Head Attention 485 Multi-Head Attention 486 Combiner 488 layer normalization 490 Feedforward 492 layer normalization 494 Linear transformation layer 498 Decoding Layer 500 encoders 521 Input data 522 Input Stages 526 layer normalization 528 Encoded Data 554 coding layer 562 layer normalization 564 Multi-Head Attention 566 Combiner 568 layer normalization 570 Feedforward 572 Combiner 602, 602A, 602B, 602N Audio Samples 621 Control Parameters 623 Target Person 625 Target Emotion 627 Target speech rate 629 Synthetic Audio Samples 631 Control Parameters 633 Target Person 635 Target Emotion 637 Target speech rate 639 Synthetic Audio Samples 661 Synthesized speech signal 663 Synthesized speech signal 665 training datasets 670 Network Trainer 672 Speech Recognition Network 710 Sequence 712 First Frame 714 Second Frame 716th Nth Frame 720 sequences 722 First set 724 Second set 726 Set N 730 Sequences 732 First set 734 Second set 736 Set N 740 sequences 742 First coding set 744 Second coding set 746th Nth coding set 750 sequences 752 First spectral data 754 Second spectral data 756. Spectral data of the Nth element 800 Implementation Forms 802 integrated circuit 804 Audio Input 806 Audio Output 900 Implementation Forms 902 Mobile Devices 904 Display screen 1000 Implementation Forms 1002 Headset Device 1100 Implementation Forms 1102 Wearable electronic devices 1104 Display screen 1200 Implementation Forms 1202 Voice-activated device 1204 Speaker 1300 Implementation Forms 1302 Camera Device 1400 Implementation Forms 1402 Headset 1500 Implementation Forms 1502 Mobile Unit 1600 Implementation Forms 1602 Mobile Unit 1610 speaker 1620 display 1700 methods 1900 devices 1902 Digital-to-Analog Converter (DAC) 1904 Analog-to-Digital Converter (ADC) 1906 Processor 1908 Speech and music coder-decoder (codec) 1910 Processor 1922 System-in-Package or System-on-Chip Device 1926 Display Controller 1928 Display 1930 Input Devices 1934 codec 1936 Voice Coder Encoder 1938 Vocoder Decoder 1944 power supply 1950 Transceiver 1952 Antenna 1956 instruction 1970 Modem 1992 Speaker

Claims

1. A device for generating speech, One or more processors, Receiving an input audio signal, Receiving one or more control parameters that represent the target speech characteristics, Performing audio feature extraction on the input audio signal in order to generate a Mel-scale spectrogram and / or fundamental frequency (F0) of the input audio signal, Processing the input representation of the utterance based on one or more control parameters to generate encoded data corresponding to an audio signal representing a version of the input speech signal based on the target utterance characteristics, using the multi-encoder of the device, The input representation of the utterance includes the Mel-scale spectrogram and / or fundamental frequency (F0) of the input speech signal, The aforementioned multi-encoder, A first encoder configured to encode the input representation independently of one or more control parameters in order to generate first encoded data, One or more second encoders configured to encode the input representation based on one or more control parameters in order to generate second encoded data, and to be equipped with One or more processors configured to perform A device equipped with the following features.

2. The device according to claim 1, wherein one or more control parameters indicate a target person, target emotion, target speech rate, or a combination thereof, in which speech characteristics are used.

3. The device according to claim 1, wherein the one or more processors are further configured to generate merged style data for the input representation based on the one or more control parameters, and the merged style data is used by the multiencoder while processing the input representation.

4. The device according to claim 1, wherein the encoded data includes the first encoded data and the second encoded data.

5. The one or more processors described above In a speech characteristic encoder, the input expression is processed based on at least one of the one or more control parameters to generate an encoded input speech expression. In the encoder pre-network, style data is generated that is merged at least partially based on the encoded input utterance representation. The input representation is provided to the first encoder in order to generate the first encoded data. To generate the second encoded data, the merged style data is provided to the one or more second encoders. The device according to claim 4, further configured as follows.

6. The device according to claim 4, further comprising a multi-encoder transformer including the multi-encoder and decoder, wherein the first encoder includes a first attention network, each of the one or more second encoders includes a second attention network, and the decoder includes a decoder attention network separate from the first and second attention networks of each of the one or more second encoders.

7. The first encoder, A first layer including the first attention network, wherein the first attention network corresponds to a first multi-head attention network, The second layer includes the first neural network and Equipped with, Each of the one or more second encoders is A first layer including the second attention network, wherein the second attention network corresponds to a second multi-head attention network, The second layer includes a second neural network and The device according to claim 6, comprising:

8. A decoder coupled to the multi-encoder, comprising a decoder network configured to generate output spectral data based on the first encoded data and the second encoded data, A speech synthesizer configured to generate the audio signal representing the version of the utterance based on the target utterance characteristics, based on the output spectral data. The device according to claim 4, further comprising:

9. The decoder network includes a decoder attention network, and the decoder attention network is A first multi-head attention network configured to process the first encoded data, One or more second multi-head attention networks configured to process the second encoded data, A combiner configured to combine the outputs of the first multi-head attention network and the one or more second multi-head attention networks. The device according to claim 8, comprising:

10. The decoder, A masked multi-head attention network coupled to the input of the decoder attention network, A decoder neural network coupled to the output of the decoder attention network and The device according to claim 9, further comprising:

11. The one or more processors described above A synthesized speech signal is generated based on the aforementioned encoded data. One or more estimated control parameters are generated from the synthesized speech signal. Based on a comparison of the one or more control parameters with the one or more estimated control parameters, one or more neural network weights of the multi-encoder, one or more speech characteristic encoders, encoder pre-networks, decoder networks, or combinations thereof are trained. The device according to claim 1, further configured as follows.

12. The device according to claim 1, further comprising a microphone, wherein one or more processors are configured to receive the input audio signal via the microphone.

13. The device according to claim 1, wherein one or more processors are further configured to receive the input audio signal from an audio repository.

14. The device according to claim 1, wherein one or more processors are configured to receive input signals including the input audio signal and video signal.

15. A method of generating speech, In the device, the steps include receiving an input audio signal, The device includes the step of receiving one or more control parameters that represent target speech characteristics, The device includes the steps of performing audio feature extraction on the input audio signal to generate a Mel-scale spectrogram and / or fundamental frequency (F0) of the input audio signal, A step of processing an input representation of an utterance based on one or more control parameters to generate encoded data corresponding to an audio signal representing a version of the input speech signal based on the target utterance characteristics, using a multi-encoder, The input representation of the utterance includes the Mel-scale spectrogram and / or fundamental frequency (F0) of the input speech signal, The aforementioned multi-encoder, A first encoder configured to encode the input representation independently of one or more control parameters in order to generate first encoded data, One or more second encoders configured to encode the input representation based on one or more control parameters in order to generate second encoded data, and Equipped with steps and Methods that include...

Images