Keyframe Network

Abstract

A method (400) for generating frame values ​​(242) using a keyframe network includes receiving a text utterance (130) having at least one phoneme (132), and for each respective phoneme of the at least one phoneme, predicting using a predictive model (232) a fixed amount of keyframes (232), each respective keyframe including a representation of a component of the respective phoneme. The method also includes generating frame values ​​using the fixed amount of keyframes, where each respective frame value of the plurality of frame values ​​is a representation of a fixed duration of audio.

Description

Technical Field

[0001] The present disclosure relates to a key-frame network for predicting frame values.

Background Art

[0002] Text-to-speech (TTS) systems, which read digital text aloud to users, are becoming increasingly popular on mobile devices. Certain TTS models aim to synthesize various aspects of speech, such as speaking styles, to generate speech with a natural-sounding voice. Neural TTS models generally convert the input representational text into an intermediate representation of the output audio, such as in the form of frames (e.g., 10 ms frames) that represent fixed-length sections of audio. These intermediate representations are typically more than one order of magnitude larger than the amount of text being input, so neural TTS models require significant computational resources and time, thereby reducing efficiency during training and inference.

Summary of the Invention

[0003] One aspect of the present disclosure provides a method for generating frame values using a key-frame network that causes data processing hardware to perform operations when executed on the data processing hardware. The operations include receiving a text utterance having at least one phoneme and predicting a fixed amount of key frames for each respective phoneme of the at least one phoneme using a prediction model. Each respective key frame of the fixed amount of key frames includes a representation of a component of the respective phoneme. The operations also include generating values for a plurality of frames using the fixed amount of key frames. Here, each respective frame value of the plurality of frame values is a representation of a fixed duration of audio.

[0004] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the duration of a first keyframe of a fixed amount of keyframes differs from the duration of a second keyframe of a fixed amount of keyframes. In some examples, the fixed amount is 3, or 10.

[0005] In some embodiments, the number of frame values ​​per phoneme is variable. Optionally, the audio represents synthesized speech. In some examples, generating frame values ​​further involves using a deterministic algorithm. In these examples, the deterministic algorithm includes one of the following: interpolation matrices, cubic spline interpolation, or discrete cosine transform.

[0006] In some embodiments, the operation further includes receiving a set of training samples, each training sample having at least one phoneme, and each phoneme in each training sample having a fixed amount of ground-truth Keyframes. In these examples, the operation also includes generating a loss for the predictive model based on the ground-truth Keyframes and training the predictive model to predict a fixed amount of keyframes for each phoneme using the set of training samples by minimizing the loss. Here, each of the ground-truth Keyframes in the fixed amount of ground-truth Keyframes may be annotated with the temporal position of the respective ground-truth Keyframe in the phoneme.

[0007] Another aspect of the present disclosure provides a system for generating frame values ​​using a keyframe network. The system includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions, which, when executed by the data processing hardware, cause the data processing hardware to perform an operation that includes receiving a text utterance having at least one phoneme, and for each of the at least one phoneme, predicting a fixed amount of keyframes using a prediction model. Each of the fixed amount of keyframes contains a representation of the components of the respective phoneme. The operation also includes generating a plurality of frame values ​​using the fixed amount of keyframes, where each of the frame values ​​is a representation of a fixed duration of audio.

[0008] This embodiment may include one or more of the following optional features. In some embodiments, the duration of the first keyframe of a fixed amount keyframe is different from the duration of the second keyframe of a fixed amount keyframe. In some examples, the fixed amount is 3, or 10.

[0009] In some embodiments, the number of frame values ​​per phoneme is variable. Optionally, the audio represents synthesized speech. In some examples, generating frame values ​​further involves using a deterministic algorithm. In these examples, the deterministic algorithm includes one of the following: interpolation matrices, cubic spline interpolation, or discrete cosine transform.

[0010] In some embodiments, the operation further includes receiving a set of training samples, each training sample having at least one phoneme, and each phoneme in each training sample having a fixed amount of ground-truth Keyframes. In these examples, the operation also includes training the predictive model to predict a fixed amount of keyframes for each phoneme using the set of training samples by generating a loss for the predictive model based on the ground-truth Keyframes and minimizing the loss, where each of the ground-truth Keyframes in the fixed amount of ground-truth Keyframes may be annotated with the temporal position of the respective ground-truth Keyframe in the phoneme.

[0011] Details of one or more embodiments of this disclosure are described in the accompanying drawings and the following description. Other embodiments, features, and advantages will become apparent from the description and drawings, as well as from the claims. [Brief explanation of the drawing]

[0012] [Figure 1] This is an exemplary system for training a deep neural network to provide a keyframe network for use in generating frame values ​​from keyframes. [Figure 2] Figure 1 is a schematic diagram of an exemplary component of a keyframe network. [Figure 3A] This is a schematic diagram of an exemplary system for generating frame values. [Figure 3B] This is a schematic diagram of a time plot that includes phonemes, keyframes, and frame values. [Figure 4] This is an illustrative flowchart of the operation of how to generate frame values ​​using a keyframe network. [Figure 5] This is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein. [Modes for carrying out the invention]

[0013] Similar reference symbols in various drawings refer to the same elements. Text-to-Speech (TTS) models, commonly used in speech synthesis systems, typically take text input in the form of phonemes and generate intermediate audio representations (e.g., frames) of the utterance to be synthesized into speech. Phonemes can have significantly varying durations, ranging from a few milliseconds to several hundred milliseconds. Each of these intermediate audio representations constitutes a short section of audio, typically ranging in length from 5 to 20 milliseconds. Therefore, the number of intermediate audio representations is usually one to two orders of magnitude larger than the number of phonemes in the text input. Predicting this large number of intermediate audio representations requires considerable time and computational resources from the TTS model.

[0014] Embodiments of this specification relate to keyframe networks. In a keyframe network, rather than directly predicting all intermediate audio representations, the TTS model is trained to predict a fixed number of keyframes representing the number of intermediate audio representations for each phoneme in the text input. That is, instead of predicting frames by a fixed transient duration, the model may predict a fixed number of keyframes (e.g., 3, 5, 10, etc.) for each phoneme. The TTS model can then extend the fixed number of keyframes to intermediate audio representations (e.g., frame values) through interpolation. Advantageously, the step of extending the fixed number of keyframes to intermediate audio representations is deterministic, does not require learned parameters, and requires less time and computational resources than directly predicting intermediate audio representations from input text using the TTS model.

[0015] During training, the TTS model is trained to predict keyframes at a phoneme-level granularity based on the input phonemes (i.e., predicting the same number of keyframes for each phoneme regardless of the phoneme's temporal length). In practice, this results in far fewer predictions per phoneme than traditional TTS models that predict intermediate representations (e.g., frame values ​​with a fixed duration) for each phoneme. Furthermore, predicting outputs corresponding to a fixed number of keyframes for each phoneme allows for the use of a different neural network architecture than those required to predict outputs of variable length. That is, the neural network architecture used to predict a fixed number of keyframes may be simpler and more computationally efficient (not autoregressive or requiring fewer prediction iterations). During inference, the TTS model is configured to receive text utterances, predict keyframe values, and then expand the keyframes to predict intermediate representations. This leads to a significant improvement in efficiency, both in terms of time and computational resources, for training and running the TTS model.

[0016] Referring now to Figure 1, in some embodiments, the exemplary system 100 includes a text-to-speech (TTS) model 200 for generating synthesized speech 152 from an input text utterance 130. The TTS model 200 runs on a computing system 120 and includes a keyframe network 230 that predicts keyframes 232 for use by a frame generator 240 to generate frame values ​​242 representing the text utterance 130. The TTS model 200 may include an autoregressive or non-autoregressive model. The computing system 120 includes data processing hardware 122 and memory hardware 124 that communicates with the data processing hardware 122 and stores instructions that cause the data processing hardware 122 to perform actions. The computing system 120 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) with scalable / elastic resources including computing resources 122 (e.g., data processing hardware) and / or storage resources 124 (e.g., memory hardware). During inference, the TTS model 200 can use the keyframe network 230 to predict a fixed number of keyframes 232, which can then be converted into frame values ​​242 by the frame generator 240.

[0017] In some embodiments, the TTS model 200 receives a text utterance 130 containing a sequence of phonemes 132. The text utterance 130 may contain one or more feature representations of the text input to the TTS model 200. The TTS model 200 receives the text utterance 130 and each phoneme 132 of at least one phoneme 132 as input. Using the text utterance 130, the TTS model 200 uses a keyframe network 230 to generate / predict a fixed amount of keyframes 232. In other words, the keyframe network 230 predicts a fixed amount of keyframes 232 for each phoneme 132 of the text utterance 130 (i.e., the same amount of keyframes 232 for each phoneme 132). The fixed amount of keyframes together may represent a variable number of frame values ​​corresponding to each phoneme 132, and may further include the end of the previous phoneme 132 in that sequence or the beginning of the following phoneme 132 in the sequence. Here, each of the fixed number of keyframes 232 contains a representation of the components of each phoneme 132. As will be described in more detail below, each keyframe 232 may represent any number of frame values ​​242 for each phoneme 132, and the fixed number of keyframes 232 are uniformly distributed across each phoneme 132. As used herein, the term “component” of each phoneme 132 may include a portion of each phoneme 132, a portion of the previous phoneme 132, and / or a portion of the next phoneme 132, as well as the IDCT coefficient. Each phoneme 132 may contain audio of different fixed durations (i.e., some phonemes take longer to speak than others), so the number of frame values ​​242 per phoneme 132 is variable, but the number of keyframes 232 per phoneme remains fixed.

[0018] When the keyframe network 230 predicts a fixed amount of keyframes 232 for some or all of the phonemes 132, the frame generator 240 receives the keyframes 232 and uses them to generate multiple frame values ​​242. Each of the multiple frame values ​​242 represents a fixed duration of audio. That is, each frame value 242 represents a short section of audio with the same duration (e.g., 5 to 20 milliseconds) and includes audio features such as a Mel frequency spectrogram, source filter vocoder parameters, or pitch and energy values. In some embodiments, a computing system 120 (e.g., data processing hardware 122) running the TTS model 200 transmits the frame values ​​242 predicted by the TTS model 200 from the input text utterance 130 to a synthesizer 150, which converts them into a time-domain audio waveform representing a synthesized speech 152 that can be output as a speech representation of the input text utterance 130. The time-domain audio waveform includes an audio waveform that defines the amplitude of the audio signal over time. The synthesizer 150 can be separately trained and conditioned with Mel frequency spectrograms or source filter vocoder parameters for conversion to time-domain audio waveforms. The type of audio features to which the synthesizer is conditioned and trained is not limited.

[0019] In some embodiments, the keyframe network 230 is trained on a large training dataset 210 of training samples 220, 220a–n. Each training sample 220 may contain at least one training phoneme 132T (e.g., a feature representation of text input) paired with a fixed amount of ground-truth keyframes 232T. During training, the keyframe network 230 may receive multiple training examples 220, 220a–n, each training example 220 containing training phonemes 132T of variable length in the time domain (i.e., those of varying durations). The keyframe network 230 is configured to encode the representation (e.g., a feature representation of text utterance) associated with each training phoneme 132T in the training example 220 into a corresponding prediction containing a fixed amount of keyframes 232 representing a quantity of frame values ​​242.

[0020] Referring to Figure 2, the TTS model 200 may include a preprocessing module 228 configured to receive phonemes 132 corresponding to a received text utterance 130 as input, convert the phonemes 132 into audio feature representations 132R, and input them into the keyframe network 230. The keyframe network 230 uses the text feature representations 132R to generate a fixed number of keyframes 232 for each phoneme 132. The fixed number of keyframes 232 may be any number; for example, the number may be three keyframes 232 representing three components / parts of each phoneme 132. As another example, the number may include ten keyframes 232 representing ten components / parts of each phoneme 132. However, these numbers are illustrative, and the keyframe network 230 can be trained to predict any fixed number of keyframes 232 depending on the specific use case of the TTS model 200 and the hardware capabilities of the computing system 120 running the TTS model 200.

[0021] In some embodiments, the frame generator 240 of the TTS model 200 executes a deterministic and computationally inexpensive algorithm 246 that receives, as input, a fixed amount of key frames 232 from the key frame network 230 and uses the deterministic algorithm 246 to generate / predict a plurality of frame values 242 that each represent a section of audio of a fixed duration. In some examples, the deterministic algorithm 246 includes an interpolation process based on an interpolation matrix. However, the deterministic algorithm 246 can include other deterministic approaches such as, but not limited to, cubic spline interpolation, discrete cosine transform, or linear interpolation. Advantageously, since the deterministic algorithm 246 is a deterministic process, the frame generator 240 does not include learned parameters and thus does not require training, enabling a computationally inexpensive implementation (e.g., no training time or resources are required). For example, both discrete cosine transform and linear interpolation can be implemented by constructing an appropriate interpolation matrix, and the interpolation can be performed as a multiplication using the corresponding matrix.

[0022] Referring now to Figure 3A, schematic Figure 300a shows a sequence of phonemes 132 for a received text utterance 130, converted to frame values ​​242 using the TTS model 200. For illustrative purposes only, the text utterance 130 contains a sequence of four phonemes 132a-d. As shown, each of the phonemes 132 requires a different duration when uttered. In some languages, the duration of phoneme 132 can vary considerably (e.g., from a few milliseconds to several hundred milliseconds). Here, a phoneme has a duration represented by the length of its contour, and therefore phonemes 132a and 132d have relatively longer durations than phonemes 132b and 132c. The keyframe network 230 receives phonemes 132a-d and generates a fixed amount of keyframes 232a1-d3 for each phoneme 132a-d. In the example shown, the fixed amount of frame values ​​242 predicted by the keyframe network 230 is 3. Specifically, the keyframe network 230 predicts three keyframes 232a1-3 as the representation of phoneme 132a, three keyframes 232b1-b3 as the representation of phoneme 132b, three keyframes 232c1-c3 as the representation of phoneme 132c, and three keyframes 232d1-d3 as the representation of phoneme 132d. Since phonemes 132a-d each require different durations (i.e., different amounts of time are needed when they are spoken aloud), the fixed number of keyframes 232 for each of phonemes 132a-d can be different. For example, the durations of 232c1-c3 may be the same, but the number of frames for keyframe 232c1 is less than the number of frames for keyframe 232d2. Conversely, the number of frames for keyframe 232c1 is greater than the number of frames for keyframe 232b1.

[0023] The frame generator 240 uses the twelve key frames 232a1 to d3 generated by the key frame network 230 to generate values 242 for a plurality of frames, and the value 242 for each respective frame represents a fixed duration of the audio. As shown, the key frames 232a to d are extended to different numbers of frame values 242 for each phoneme 132a to d. Thus, the number of values 242 for the plurality of frames is greater than both the number of phonemes 132a to d and the number of key frames 232a to d. For example, if the duration of each frame value is 10 ms and the duration of each of the total key frames 232a1 to d3 is 400 ms, the frame generator 240 generates 40 frame values 242 (i.e., 400 divided by 10).

[0024] Referring to FIG. 3B, the schematic diagram 300b shows a sequence of phonemes 132 of 3B set on the x-axis representing duration. Here, the text utterance 130 corresponds to the word "link" including the phoneme / l / 132a, the phoneme / i / 132b, the phoneme / n / 132c, and the phoneme / k / 132d. As described above, each phoneme 132a to d has the duration t shown in FIG. 3B 1~4These can be represented by different durations of audio, and each phoneme 132a~d is represented by the same number of keyframes 132 for each phoneme 132 (i.e., 3 in this example). For example, the phoneme / l / 132a is represented by three keyframes 232a1-a3, which contain three keyframes evenly distributed over time t1. The three keyframes 232a1~a3 are converted (e.g., interpolated) to seven frame values ​​242a~g, and can also be extended beyond the boundaries of phonemes, as will be discussed in more detail below. That is, here the three keyframes 232a1~a3 are converted to seven frame values ​​242a~g distributed over time t1. Similarly, the phoneme / i / 132b corresponds to three keyframes 232b1~b3, but the three keyframes 232b1~b3 are converted to two frame values ​​142b distributed over time t2. The phoneme / n / 132c corresponds to three keyframes 232c1~c3, and these three keyframes 232c1~c3 are converted to three frame values ​​242j~l distributed over time t3. Finally, the phoneme / k / 132d corresponds to three keyframes 232d1~d3, and these three keyframes 232c1~c3 are converted to seven frame values ​​242m~s distributed over time t4.

[0025] In some embodiments, the frame generator 240 generates frame values ​​242 individually from keyframes 232 for each phoneme 132. That is, the frame generator 240 generates a number of frame values ​​242 from a first set of keyframes 232 representing phoneme 132, and then generates a different number of frame values ​​242 from a second set of keyframes 232 representing another phoneme 132 (and so on). In other embodiments, the frame generator 240 generates frame values ​​242 using all of the keyframes 232 for multiple phonemes 132. In these embodiments, some frame values ​​242 may represent components of the first keyframe 232 and components of the second keyframe 232. That is, the interpolation that generates the frame values ​​242 may extend beyond the boundaries of keyframes 232 and / or phoneme 132.

[0026] Referring again to Figure 1, a training process 280 is shown for training a keyframe network 230 to generate a fixed number of keyframes 232 for each phoneme 132. The training process 280 is run on the computing system 120. As shown, the training process 280 retrieves one or more training datasets 210 stored in a datastore 201 and trains the keyframe network 230 (e.g., a deep neural network) on the training datasets 210. The datastore 201 may reside in the memory hardware 124 of the computing system 120. Each training dataset 210 contains multiple training examples 220, 220a-n, and each training example 220 may contain at least one training phoneme 132T, which is paired with a fixed amount of ground-truth keyframes 232T. In short, the training process 280 trains the keyframe network 230 to learn to predict a fixed number of ground-truth keyframes 232T (e.g., three ground-truth keyframes 232T) for each training phoneme 132T.

[0027] In some embodiments, each of the training examples 220 is associated with the temporal position of the corresponding training phoneme 132T in each ground-truth keyframe 232T, so the keyframe network 230 learns through the training process 280 to generate a fixed amount of keyframes 232 at the corresponding time in response to receiving phoneme 132 as input. In the example shown, the keyframe network 230 receives the training example 220 as input and predicts the output y r Generates the output prediction y. rThis includes a predicted fixed number of keyframes 232, which are tested for accuracy. During the training process 280, and in each time step or batch of time steps, the keyframe network 230 may be trained to minimize the loss using a loss function 260 based on the output prediction yr and a fixed number of ground truth keyframes 232T from the corresponding training example 220, which serve as ground truth.

[0028] Figure 4 is a flowchart illustrating an exemplary arrangement of the operation of Method 400 for generating frame values ​​242 using a keyframe network 230. Method 400 includes, in operation 402, receiving a text utterance 130 having at least one phoneme 132. For each of the at least one phoneme 132, Method 400 also includes, in operation 404, predicting a fixed amount of keyframes 232 using a prediction model 230 (i.e., the keyframe network 230). Each of the fixed amount of keyframes 232 contains a representation of the components of each phoneme 132. In operation 406, Method 400 further includes generating a multi-frame value 242 using the fixed amount of keyframes 232, where each of the multi-frame value 242 represents a fixed duration of audio.

[0029] Figure 5 is a schematic diagram of an exemplary computing device 500 that can be used to implement the systems and methods described in this document. The computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown herein, their connections and relationships, and their functions are for illustrative purposes only and are not intended to limit the embodiments of the invention described and / or claimed in this document.

[0030] The computing device 500 includes a processor 510, memory 520, a storage device 530, a high-speed interface / controller 540 connected to memory 520 and a high-speed expansion port 550, and a low-speed bus 570 and a low-speed interface / controller 560 connected to storage device 530. Each component 510, 520, 530, 540, 550, and 560 is interconnected using various buses and may be mounted on a common motherboard or otherwise present as needed. The processor 510 (e.g., data processing hardware 122 in Figure 1) processes instructions for execution within the computing device 500, including instructions stored in memory 520 or storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 580 connected to the high-speed interface 540. In other embodiments, multiple processors and / or multiple buses may be used as needed, along with multiple memories and memory types. Additionally, multiple computing devices 500 may be connected, with each device performing some of the necessary operations (for example, as a server bank, a group of blade servers, or a multiprocessor system).

[0031] Memory 520 (for example, the memory hardware in Figure 1) stores information non-temporarily within the computing device 500. Memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). Non-temporarily, memory 520 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and disk or tape.

[0032] The storage device 530 can provide high-capacity storage to the computing device 500. In some embodiments, the storage device 530 is a computer-readable medium. In various different embodiments, the storage device 530 may be a device array including a floppy disk® device, a hard disk device, an optical disk device, or a tape device, flash memory or other similar solid-state memory device, or a storage area network or other configuration device. In additional embodiments, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that perform one or more of the above-described methods at runtime. The information carrier is a computer-readable or machine-readable medium such as memory 520, the storage device 530, or memory on the processor 510.

[0033] The high-speed controller 540 manages the bandwidth-intensive operation of the computing device 500, and the low-speed controller 560 manages the low-bandwidth-intensive operation. Such role assignments are merely examples. In some embodiments, the high-speed controller 540 is coupled to memory 520, a display 580 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 550 that can accept various expansion cards (not shown). In some embodiments, the low-speed controller 560 is coupled to a storage device 530 and a low-speed expansion port 590. The low-speed expansion port 590 may include various communication ports (such as USB, Bluetooth®, Ethernet®, or Wireless Ethernet) that can connect to one or more input / output devices such as a keyboard, pointing device, scanner, or network devices such as switches or routers via a network adapter, etc.

[0034] The computing device 500 can be implemented in many different forms, as shown in the figure. For example, it may be implemented as a standard server 500a, or multiple times within a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0035] Various embodiments of the systems and technologies described herein can be realized in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor, at least one input device, and at least one output device, which may be specialized or general-purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system.

[0036] A software application (i.e., a software resource) can refer to computer software that causes a computing device to perform a task. In some examples, a software application may be called an “application,” “app,” or “program.” Exemplary applications include, but are not limited to, system diagnostic applications, system administration applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0037] These computer programs (also known as programs, software, software applications, or code) include machine instructions for programmable processors and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-temporary computer-readable medium, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0038] The processes and logical flows described herein can be performed by one or more programmable processors, called data processing hardware, executing one or more computer programs to perform functions by acting on input data and producing outputs. Processes and logical flows can also be performed by special-purpose logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Processors suitable for executing computer programs include, for example, one or more processors from both general-purpose and special-purpose processors, and from either type of digital computer. Generally, processors receive instructions and data from read-only memory, random-access memory, or both. The basic elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely connected to them to receive data from them, transmit data to them, or both. However, a computer is not required to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices including EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory may be complemented by or integrated into dedicated logic circuits.

[0039] To interact with a user, one or more aspects of this disclosure can be implemented in a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touchscreen), and optionally a keyboard and pointing device (e.g., a mouse or trackball) by which the user can input to the computer. Other types of devices may also be used to provide interaction with a user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic, spoken language, or haptic input. Furthermore, the computer can interact with the user by sending and receiving documents to and from the user's device, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0040] Several embodiments have been described. Nevertheless, it is understood that various modifications can be made without departing from the spirit and scope of this disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer implementation method (400) for causing data processing hardware (122) to perform an operation when executed on data processing hardware (122), wherein the operation is Receiving a text utterance (130) having at least one phoneme (132), For each of the at least one phoneme (132), predict a fixed amount of keyframes (232) using a prediction model (230), wherein each of the fixed amount of keyframes (232) includes a representation of the components of the respective phoneme (132). A computer implementation method (400) comprising generating a plurality of frame values ​​(242) using the fixed amount of keyframes (232), wherein each of the plurality of frame values ​​(242) represents audio of a fixed duration.

2. The computer implementation method (400) according to claim 1, wherein each keyframe (232) further includes at least one of the following: a representation of a portion of a phoneme (132) preceding the text utterance (130) or a representation of a portion of a subsequent phoneme (132) following the text utterance (130).

3. The computer implementation method (400) according to claim 1 or 2, wherein the fixed amount is 3.

4. The computer implementation method (400) according to claim 1 or 2, wherein the fixed amount is 10.

5. The computer implementation method (400) according to claim 1 or 2, wherein the number of values ​​(242) of the plurality of frames for each phoneme (132) is variable.

6. The computer implementation method (400) according to claim 1 or 2, wherein the audio represents synthesized speech (152).

7. The computer implementation method (400) according to claim 1 or 2, further comprising using a deterministic algorithm (246) to generate the values ​​(242) of the plurality of frames.

8. The aforementioned deterministic algorithm (246) Interpolation matrix, Cubic spline interpolation, or A computer implementation method (400) according to claim 7, comprising an interpolation process based on one of the discrete cosine transforms.

9. The aforementioned operation, Receiving a set consisting of multiple training samples (220), wherein each training sample has at least one phoneme (132), and each phoneme (132) of each training sample (220) has a fixed amount of ground-truth key frames (232T), Using the set of the multiple training samples (220), the prediction model (230) predicts the fixed amount of keyframes (232) for each phoneme (132). Based on the ground-truth key frame (232T), the loss of the prediction model (230) is generated, and The computer implementation method (400) according to claim 1 or 2, further comprising training the predictive model (230) by minimizing the aforementioned loss.

10. The computer implementation method (400) according to claim 9, wherein each of the fixed amount of ground-truth key frames (232T) is annotated at the temporal position of each of the ground-truth key frames (232) in the phoneme (132).

11. System (100), Data processing hardware (122), and A memory hardware (124) that communicates with the data processing hardware (122), wherein the memory hardware (124) stores instructions, and when an instruction is executed by the data processing hardware (122), it communicates to the data processing hardware (122): Receiving a text utterance (130) having at least one phoneme (132), For each of the at least one phoneme (132), predict a fixed amount of keyframes (232) using a prediction model (230), wherein each of the fixed amount of keyframes (232) includes a representation of the components of the respective phoneme (132). A system (100) including memory hardware (124) that causes the system to perform an operation which includes generating a plurality of frame values ​​(242) using a fixed amount of keyframes (232), wherein each of the plurality of frame values ​​(242) represents audio of a fixed duration.

12. The system (100) according to claim 11, wherein each keyframe (232) further comprises at least one of the following: a representation of a portion of a phoneme (132) preceding the text utterance (130) or a representation of a portion of a subsequent phoneme (132) following the text utterance (130).

13. The system (100) according to claim 11 or 12, wherein the fixed amount is 3.

14. The system (100) according to claim 11 or 12, wherein the fixed amount is 10.

15. The system (100) according to claim 11 or 12, wherein the number of the multiple frame values ​​(242) for each phoneme (132) is variable.

16. The system (100) according to claim 11 or 12, wherein the audio represents synthesized speech (152).

17. The system (100) according to claim 11 or 12, further comprising using a deterministic algorithm (246) to generate the values ​​(242) of the plurality of frames.

18. The aforementioned deterministic algorithm (246) Interpolation matrix, Cubic spline interpolation, or The system (100) according to claim 17, comprising an interpolation process based on one of the discrete cosine transforms.

19. The aforementioned operation is, Receiving a set consisting of multiple training samples (220), wherein each training sample (220) has at least one phoneme (132), and each phoneme (132) of each training sample has a fixed amount of ground-truth key frames (232T), Using the set of the multiple training samples (220), the prediction model (230) predicts the fixed amount of keyframes (232) for each phoneme (132). Based on the ground-truth key frame (232T), the loss of the prediction model (230) is generated, and The system (100) according to claim 11 or 12, further comprising training the predictive model (230) by minimizing the aforementioned loss.

20. The system (100) according to claim 19, wherein each of the fixed amount of ground-trus ski frames (232T) is annotated at the temporal position of each of the ground-trus ski frames (232) in the phoneme (132).

Images