Method, device, apparatus and medium for generating voice and video information based on input

Abstract

A method, apparatus, device, and medium for generating speech and video information from input include receiving text input comprising a sequence of text components, determining the temporal duration of each of the text components using a duration model, generating spectrogram frames based on the duration model, generating audio waveforms based on the spectrogram frames, and generating video information based on the audio waveforms. The audio waveforms are provided as output along with corresponding video.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 16 / 549,068, filed in the U.S. Patent and Trademark Office on August 23, 2019, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to speech synthesis technology, and in particular to a method, device, apparatus and medium for generating speech and video information based on input. Background Art

[0004] U.S. application Ser. No. 16 / 397,349, filed Apr. 29, 2019, is incorporated herein by reference in its entirety.

[0005] Recently, end-to-end speech synthesis systems such as Tacotron have demonstrated impressive text-to-speech (TTS) results in terms of naturalness and prosody of the synthesized speech. However, such systems suffer from significant shortcomings in terms of skipping or repeating some words in the input text when synthesizing speech. This problem arises from the end-to-end nature of speech generation, which uses an uncontrollable attention mechanism.

[0006] Previously, end-to-end attention-based models have shown improvements over traditional non-end-to-end TTS frameworks. However, end-to-end attention-based models also suffer from the problem of missing and repeating words in the original input text, which is a common flaw in end-to-end attention frameworks. Summary of the Invention

[0007] According to some possible implementations, a method for generating voice and video information based on input may include: a device receiving text input including a sequence of text components; the device determining the respective temporal durations of the text components using a duration model; the device generating a first spectrum group based on the sequence of text components; the device generating a second spectrum group based on the first spectrum group and the respective temporal durations of the sequence of text components; the device generating a spectrogram frame based on the second spectrum group; the device generating an audio waveform based on the spectrogram frame; the device generating video information corresponding to the audio waveform; and the device providing an audio waveform and a corresponding video as output of the device based on the video information.

[0008] According to some possible implementations, the method may include training a duration model.

[0009] According to some possible implementations, in the method, text input can be obtained by: receiving an input video including a corresponding input audio waveform as input; generating input video information corresponding to the input audio waveform by the device; generating an input spectrogram frame based on the input audio waveform by the device; generating a first input spectrum group based on the input spectrogram frame by the device; generating a second input spectrum group based on the first input spectrum group by the device; and determining the text input by the device and using a duration model.

[0010] According to some possible implementations, in the method, the text constituent elements may be phonemes or characters.

[0011] According to some possible implementations, the method may further include receiving, by the device, information corresponding to an emotional state, the emotional state being associated with the text input, and wherein the audio waveform and corresponding video provided as output of the device may be based on the information corresponding to the emotional state.

[0012] According to some possible implementations, in the method, an audio waveform and a corresponding video that may be based on video information may be provided as output simultaneously.

[0013] According to some possible implementations, in the method, training the duration model may include multi-task training.

[0014] According to some possible implementations, in the method, the output audio waveform and the output corresponding video may be applied to a virtual character.

[0015] According to some possible implementations, in the method, the second spectrum group may include Mel-frequency cepstrum spectra.

[0016] According to some possible implementations, in the method, training the duration model may include using prediction frame groups and training text components.

[0017] According to some possible implementations, a computer device may include: at least one memory configured to store program code; at least one processor configured to read the program code and operate as instructed by the program code to execute the method of generating voice and video information based on input as described in any of the aforementioned implementations.

[0018] According to some possible implementations, a device for generating voice and video information based on input may include: a receiving module for receiving text input including a sequence of text components; a determination module for determining the individual temporal durations of the text components using a duration model; a first generation module for generating a first spectrum group based on the text component sequence; a second generation module for generating a second spectrum group based on the first spectrum group and the individual temporal durations of the text component sequence; a third generation module for generating a spectrogram frame based on the second spectrum group; a fourth generation module for generating an audio waveform based on the spectrogram frame; a fifth generation module for generating video information corresponding to the audio waveform; and an output module for providing the audio waveform and corresponding video as output of the device based on the video information.

[0019] According to some possible implementations, a non-transitory computer-readable medium storing instructions may be provided, wherein the instructions include one or more of the following instructions: when the one or more instructions are executed by one or more processors of the device, the one or more processors are caused to execute the method of generating voice and video information based on input as described in any of the aforementioned implementations.

[0020] Embodiments described herein relate to methods and apparatus for modeling and generating both, and in some embodiments, simultaneously, speech and video information of a talking face. These embodiments are based on a novel model, Duration Informed Attention Network (DurIAN), described not only herein but also in U.S. application Ser. No. 16 / 397,349, which is incorporated herein in its entirety as described above.

[0021] As can be seen from the above, the present application provides a method and apparatus for generating voice and video information based on input. The method includes: receiving, by a device, a text input comprising a sequence of text components; determining, by the device, the temporal durations of the text components using a duration model; generating, by the device, a first spectrum group based on the sequence of text components; generating, by the device, a second spectrum group based on the first spectrum group and the temporal durations of the sequence of text components; generating, by the device, a spectrogram frame based on the second spectrum group; generating, by the device, an audio waveform based on the spectrogram frame; generating, by the device, video information corresponding to the audio waveform; and providing, by the device, an audio waveform and a corresponding video as output of the device based on the video information.

[0022] Prior art Tacotron systems suffer from instabilities such as skipping and / or repeating input text. This instability is primarily due to its uncontrollable attention mechanism, which cannot guarantee that every input text can be synthesized sequentially without skipping or repeating. The present invention replaces this unstable and uncontrollable attention mechanism with a duration-based attention mechanism that ensures that the input text is synthesized sequentially without skipping or repeating. Embodiments of the present disclosure introduce independent call duration modeling into an end-to-end attention framework, successfully addressing issues inherent in traditional end-to-end attention frameworks. Embodiments of the present disclosure use the newly proposed Duration-Informed Attention Network (DurIAN) framework to simultaneously model both the speech and video information of a talking face. Embodiments of the present disclosure demonstrate superior performance compared to conventional audio-visual modeling methods. Embodiments of the present disclosure also support the modeling and synthesis of speech and faces with diverse styles, such as happy, sad, annoying, and natural. Embodiments of the present disclosure also demonstrate superior duration and system controllability compared to conventional frameworks.

[0023] Embodiments of the present disclosure may also be applied to virtual characters, virtual faces, and the like.

[0024] Embodiments of the present disclosure utilize the DurIAN model to provide better and more synchronized audiovisual modeling and synthesis methods.

[0025] Implementations of the present disclosure support multi-style audiovisual modeling and synthesis.

[0026] Implementations of the present disclosure provide better controllability over audio-visual modeling and synthesis than conventional methods.

[0027] The embodiments of the present disclosure may also be applied to only audio features or only visual features, or modeling audio features or visual features as multi-task training. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a diagram that provides an overview of an example implementation described herein;

[0029] Figure 2 is a diagram of an example environment in which the systems and / or methods described herein may be implemented;

[0030] Figure 3 yes Figure 2 diagrams of example components of one or more devices;

[0031] Figure 4 is a flow chart of an example process for generating an audio waveform and corresponding video, according to an embodiment; and

[0032] Figure 5A and Figure 5B is a diagram including inputs and outputs of a duration model, according to an embodiment. DETAILED DESCRIPTION

[0033] Text-to-speech (TTS) systems have a variety of applications. However, most commercially available systems are based on parametric systems that are significantly inferior to natural human speech. Tacotron is a TTS synthesis system that differs significantly from conventional parameter-based TTS systems and is capable of producing highly natural-sounding sentences. The entire system can be trained end-to-end, using an encoder-convolution-bank-highway network-bidirectional-gated-recurrent unit (CBHG) module to replace the conventional complex language feature extraction component.

[0034] An end-to-end attention mechanism is used to replace the duration model that has been used in conventional parameter systems. In the end-to-end attention mechanism, the alignment between the input text (or phoneme sequence) and the speech signal is learned from the attention model instead of the alignment based on the Hidden Markov Model (HMM). Another major difference associated with the Tacotron system is that the Tacotron system directly predicts the mel / linear spectrum that can be directly used by advanced vocoders such as Wavenet and WaveRNN to synthesize high-quality speech.

[0035] The Tacotron-based system is able to generate more accurate and natural-sounding speech. However, the Tacotron system includes instabilities such as skipping and / or repeating input text, which is an inherent shortcoming when synthesizing speech waveforms.

[0036] Some implementations herein address the aforementioned issues of input text skipping and repetition associated with Tacotron-based systems while maintaining the excellent synthesis quality of Tacotron-based systems. In addition, some implementations herein address these instability issues and achieve significantly improved naturalness in synthesized speech.

[0037] The instability of Tacotron is mainly caused by its uncontrollable attention mechanism, and it cannot guarantee that every input text can be synthesized sequentially without skipping or repetition.

[0038] Some implementations in this paper replace this unstable and uncontrollable attention mechanism with a duration-based attention mechanism, in which the input text is guaranteed to be synthesized sequentially without skipping or repeating. The main reason for the need for attention in Tacotron-based systems is the lack of alignment information between the source text and the target spectrogram.

[0039] Typically, the length of the input text is much shorter than the length of the generated spectrogram. A single character / phoneme from the input text may generate multiple frames of spectrograms, and this information is required to model the input / output relationship using any neural network architecture.

[0040] Tacotron-based systems have primarily addressed this problem using an end-to-end mechanism, where the generation of the spectrogram depends on learning attention to the source input text. However, such an attention mechanism is fundamentally unstable because its attention is highly uncontrollable. Some implementations herein replace the end-to-end attention mechanism within the Tacotron system with a duration model that predicts how long a single input character and / or phoneme lasts. In other words, the alignment between the output spectrogram and the input text is achieved by copying each input character and / or phoneme for a predetermined duration. The ground truth duration of the input text learned from the system is achieved using forced alignment based on an HMM. Using the predicted duration, each target frame in the spectrogram can be matched to a character / phoneme in the input text. The entire model architecture is depicted in the following figure.

[0041] Figure 1 FIG is a diagram that provides an overview of the embodiments described herein. Figure 1 As shown and indicated by reference numeral 110, a platform (e.g., a server) may receive text input including a sequence of text components. As shown, the text input may include a phrase such as "this is a cat." The text input may include a sequence of text components displayed as the characters "DH," "IH," "S," "IH," "Z," "AX," "K," "AE," and "T."

[0042] like Figure 1 As further shown, and by reference numeral 120, the platform can determine the individual temporal durations of text components using a duration model. The duration model can include a model that receives input text components and determines the temporal durations of the text components. As an example, the phrase "this is a cat" can include an overall temporal duration of one second when audibly output. Individual text components of the phrase can include different temporal durations that together form an overall temporal duration.

[0043] As an example, the word "this" may include a temporal duration of 400 milliseconds, the word "is" may include a temporal duration of 200 milliseconds, the word "a" may include a temporal duration of 100 milliseconds, and the word "cat" may include a temporal duration of 300 milliseconds. The duration model may determine the temporal durations of the individual components of a textual element.

[0044] like Figure 1 As further shown, and indicated by reference numeral 130, the platform can generate a first set of spectra based on the sequence of text components. For example, the platform can input the text components into a model that generates output spectra based on the input text components. As shown, the first set of spectra can include a corresponding spectrum for each text component (e.g., displayed as "1," "2," "3," "4," "5," "6," "7," "8," and "9").

[0045] like Figure 1 As further shown and indicated by reference numeral 140, the platform can generate a second spectrum group based on the first spectrum group and the respective temporal durations of the text component sequence. The platform can generate the second spectrum group by replicating the spectrum based on the respective temporal durations of the spectrum. As an example, spectrum "1" can be replicated such that the second spectrum group includes three spectrum components corresponding to spectrum "1," and so on. The platform can use the output of the duration model to determine how to generate the second spectrum group.

[0046] like Figure 1 As further shown, and by reference numeral 140, the platform can generate a spectrogram frame based on the second spectrum group. The spectrogram frame can be formed by the individual component spectrum elements of the second spectrum group. Figure 1 As shown, the spectrogram frames can be aligned with the predicted frames. In other words, the spectrogram frames generated by the platform can be accurately aligned with the expected audio output of the text input.

[0047] like Figure 1 As shown in

[15] , the phone duration pattern can be introduced into the end-to-end attention framework to align the input language text with the output acoustic features. Figure 1 As shown in the figure, both audio features and visual features can be used as autoregressive outputs. In addition, style type and emotion type can be added to the encoded language features to control the audio-visual style.

[0048] The platform can use various techniques to generate audio waveforms based on spectrogram frames and provide the audio waveforms as output. Similarly, the platform can also generate and output corresponding videos.

[0049] In this manner, some implementations herein enable more accurate audio and video output generation associated with speech-to-text synthesis by utilizing a duration model that determines the respective temporal durations of input text constituents.

[0050] Figure 2 is a diagram of an example environment 200 in which the systems and / or methods described herein may be implemented. Figure 2 As shown, environment 200 may include user device 210, platform 220, and network 230. The devices of environment 200 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0051] User device 210 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information related to platform 220. For example, user device 210 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smartphone, a wireless phone, etc.), a wearable device (e.g., a pair of smart glasses or a smart watch), or the like. In some implementations, user device 210 may receive information from platform 220 and / or send information to platform 220.

[0052] The platform 220 includes one or more devices capable of using a duration-informed attention network to generate an audio waveform for text-to-speech synthesis, as described elsewhere herein. In some implementations, the platform 220 may include a cloud server or a cloud server group. In some implementations, the platform 220 may be designed to be modular so that software components can be swapped in or out according to specific needs. In this way, the platform 220 may be easily and / or quickly reconfigured for different purposes.

[0053] In some implementations, as shown, the platform 220 can be hosted in a cloud computing environment 222. It is worth noting that while the implementations described herein describe the platform 220 as being hosted in the cloud computing environment 222, in some implementations, the platform 220 is not cloud-based (i.e., can be implemented outside of a cloud computing environment) or can be partially cloud-based.

[0054] Cloud computing environment 222 includes an environment hosting platform 220. Cloud computing environment 222 can provide computing, software, data access, storage, and other services without requiring end users (e.g., user devices 210) to be aware of the physical location and configuration of the systems and / or devices hosting platform 220. As shown, cloud computing environment 222 can include a set of computing resources 224 (collectively referred to as "computing resources 224" and individually as "computing resource 224").

[0055] Computing resources 224 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, computing resources 224 can host platform 220. Cloud resources can include computing instances executed in computing resources 224, storage devices provided in computing resources 224, data transmission devices provided by computing resources 224, etc. In some implementations, computing resources 224 can communicate with other computing resources 224 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0056] Further Figure 2 As shown, the computing resources 224 include a set of cloud resources, such as one or more applications ("application, APP") 224-1, one or more virtual machines ("virtual machines, VM") 224-2, virtualized storage devices ("virtualized storage, VS") 224-3, one or more hypervisors ("hypervisor, HYP") 224-4, etc.

[0057] The applications 224-1 include one or more software applications that can be provided to or accessed by the user device 210 and / or the sensor device. The applications 224-1 do not require the software applications to be installed and executed on the user device 210. For example, the applications 224-1 may include software associated with the platform 220 and / or any other software that can be provided through the cloud computing environment 222. In some implementations, one application 224-1 can send information to or receive information from one or more other applications 224-1 through the virtual machine 224-2.

[0058] Virtual machine 224-2 comprises a software implementation of a machine (e.g., a computer) that executes programs, similar to a physical machine. Virtual machine 224-2 can be either a system virtual machine or a process virtual machine, depending on the use and degree of correspondence between virtual machine 224-2 and any real machine. A system virtual machine can provide a complete system platform that supports the execution of a complete operating system (OS). A process virtual machine can execute a single program and support a single process. In some implementations, virtual machine 224-2 can execute on behalf of a user (e.g., user device 210) and can manage the infrastructure of cloud computing environment 222, such as data management, synchronization, or long-term data transfer.

[0059] Virtualized storage 224-3 includes one or more storage systems and / or one or more devices that utilize virtualization technology within the storage systems or devices of computing resource 224. In some implementations, within the context of storage systems, types of virtualization may include block virtualization and file virtualization. Block virtualization refers to the abstraction (or separation) of logical storage from physical storage, enabling access to the storage system regardless of the physical storage or heterogeneous architecture. This separation allows storage system administrators to flexibly manage end-user storage. File virtualization eliminates dependencies between data accessed at the file level and the physical location of the files. This can optimize storage utilization, server consolidation, and / or performance for non-disruptive file migration. Hypervisor 224-4 provides hardware virtualization technology that allows multiple operating systems (e.g., guest operating systems) to execute simultaneously on a host computer, such as computing resource 224. Hypervisor 224-4 can provide a virtual operating platform for guest operating systems and manage their execution. Multiple instances of various operating systems can share virtualized hardware resources.

[0060] The network 230 includes one or more wired networks and / or wireless networks. For example, the network 230 may include a cellular network (e.g., a fifth generation (5G) network, a Long-Term Evolution (LTE) network, a third generation (3G) network, a Code Division Multiple Access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-optic-based network, etc., and / or combinations of these or other types of networks.

[0061] Figure 2 The number and arrangement of devices and networks shown are provided as examples. Figure 2 There may be more devices and / or networks, fewer devices and / or networks, different devices and / or networks, or devices and / or networks arranged differently than those shown. Figure 2 Two or more of the devices shown may be implemented in a single device, or Figure 2 The single device shown may be implemented as multiple distributed devices. Additionally or alternatively, one or more devices of one group of devices (eg, one or more devices) of environment 200 may perform one or more functions described as being performed by another group of devices of environment 200.

[0062] Figure 3 is a diagram of example components of a device 300. Device 300 may correspond to user device 210 and / or platform 220. Figure 3 As shown, device 300 may include a bus 310 , a processor 320 , a memory 330 , a storage component 340 , an input component 350 , an output component 360 , and a communication interface 370 .

[0063] Bus 310 includes components that allow communication between components of device 300. Processor 320 is implemented in hardware, firmware, or a combination of hardware and software. Processor 320 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some implementations, processor 320 includes one or more processors that can be programmed to perform functions. Memory 330 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by processor 320.

[0064] The storage component 340 stores information and / or software related to the operation and use of the device 300. For example, the storage component 340 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-transitory computer-readable medium and a corresponding drive.

[0065] Input components 350 include components that allow device 300 to receive information, for example, through user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 350 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 360 include components that provide output information from device 300 (e.g., a display, a speaker, and / or one or more light-emitting diodes (LEDs)).

[0066] Communication interface 370 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable device 300 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 370 may allow device 300 to receive information from another device and / or provide information to another device. For example, communication interface 370 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, and the like.

[0067] Device 300 can perform one or more of the processes described herein. Device 300 can perform these processes in response to processor 320 executing software instructions stored by non-transitory computer-readable media, such as memory 330 and / or storage component 340. Computer-readable media is defined herein as non-transitory memory devices. Memory devices include memory space within a single physical storage device or memory space distributed across multiple physical storage devices.

[0068] The software instructions may be read into memory 330 and / or storage component 340 from another computer-readable medium or from another device through communication interface 370. When executed, the software instructions stored in memory 330 and / or storage component 340 may cause processor 320 to perform one or more processes described herein.

[0069] Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0070] Figure 3 The number and arrangement of components shown are provided as examples. Figure 3 The device 300 may include more components, fewer components, different components, or components arranged differently than those shown. Additionally or alternatively, one or more components of the device 300 may perform one or more functions described as being performed by another group of components of the device 300.

[0071] Figure 4 is a flow chart of an example process 400 for generating an audio waveform and corresponding video using a duration-informed attention network for text-to-speech synthesis. In some implementations, Figure 4 One or more processing blocks of may be executed by platform 220. In some implementations, Figure 4One or more processing blocks of FIG. 2 may be performed by another device or group of devices, such as user device 210 , separate from or including platform 220 .

[0072] like Figure 4 As shown, processing may include receiving, by a device, text input comprising a sequence of text components (block 410 ).

[0073] For example, the platform 220 may receive text input to be converted into audio output. Text components may include characters, phonemes, n-grams, words, letters, etc. A sequence of text components may form a sentence, a phrase, etc.

[0074] like Figure 4 As further shown, processing may include determining, by the device, respective temporal durations of the text components using a duration model (block 420 ).

[0075] The duration model may include a model that receives input text components and determines the temporal duration of the input text components. Platform 220 may train the duration model. For example, platform 220 may use machine learning techniques to analyze data (e.g., training data such as historical data) and create the duration model. Machine learning techniques may include supervised and / or unsupervised techniques, such as artificial networks, Bayesian statistics, learning automata, hidden Markov modeling, linear classifiers, quadratic classifiers, decision trees, association rule learning, and the like.

[0076] The platform 220 can train the duration model by aligning spectrogram frames and text component sequences of known duration. For example, the platform 220 can use HMM-based forced alignment to determine the ground truth duration of the text components of the input text sequence. The platform 220 can train the duration model by utilizing predicted spectrogram frames or target spectrogram frames of known duration and a known input text sequence including text components.

[0077] Platform 220 can input text components into a duration model and, based on the output of the model, determine information identifying or associated with each temporal duration of the text components. As described below, the information identifying or associated with each temporal duration can be used to generate a second spectral group.

[0078] like Figure 4 As further shown, processing may include generating a first spectral group based on the text component sequence (block 430 ).

[0079] For example, platform 220 can generate an output spectrum of text components corresponding to an input sequence of text components. Platform 220 can utilize a CBHG module to generate the output spectrum. The CBHG module can include a one-dimensional convolutional filter bank, a highway network group, a bidirectional gated recurrent unit (GRU), a recurrent neural network (RNN), and / or other components.

[0080] In some implementations, the output spectrum may be a mel-frequency cepstrum (MFC) spectrum. The output spectrum may include any type of spectrum used to generate a spectrogram frame.

[0081] like Figure 4 As further shown, processing may include generating a second spectral group based on the first spectral group and respective temporal durations of the textual component sequences (block 440 ).

[0082] For example, the platform 220 may generate a second spectral group using the first spectral group and information identifying or associated with respective temporal durations of text components.

[0083] As an example, platform 220 can replicate various spectra in the first set of spectra based on respective temporal durations corresponding to the elementary textual components of the spectra. In some cases, platform 220 can replicate spectra based on a replication factor, a temporal factor, etc. In other words, the output of the duration model can be used to determine factors by which to replicate specific spectra, generate additional spectra, etc.

[0084] like Figure 4 As further shown, processing may include generating a spectrogram frame based on the second set of spectra (block 450 ).

[0085] For example, platform 220 can generate a spectrogram frame based on the second set of spectrum. Collectively, the second set of spectrum forms a spectrogram frame. As mentioned elsewhere herein, spectrogram frames generated using a duration model can more accurately resemble a target frame or a predicted frame. In this way, some implementations herein improve the accuracy of TTS synthesis, enhance the naturalness of generated speech, improve the prosody of generated speech, and the like.

[0086] like Figure 4 As further shown, processing may include generating an audio waveform based on the spectrogram frame (460).

[0087] For example, the platform 220 can generate an audio waveform based on the spectrogram frame and provide the audio waveform for output. As examples, the platform 220 can provide the audio waveform to an output component (e.g., a speaker), provide the audio waveform to another device (e.g., user device 210), transmit the audio waveform to a server or another terminal, etc.

[0088] like Figure 4 As further shown, processing may include generating, by the device, video information corresponding to the audio waveform.

[0089] Finally, if Figure 4 As shown, processing may include providing an audio waveform and corresponding video as output.

[0090] although Figure 4 Example blocks of process 400 are shown, but in some implementations, Figure 4 Process 400 may include more blocks, fewer blocks, different blocks, or blocks arranged differently than those depicted in . Additionally or alternatively, two or more of the blocks of process 400 may be executed in parallel.

[0091] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

[0092] As used herein, the term component is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0093] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0094] Even if particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features can be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each dependent claim listed below may be directly dependent on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

[0095] The elements, actions or instructions used in this article should not be interpreted as key or necessary elements, actions or instructions, unless clearly described to this. In addition, as used in this article, "a kind of" and "an" are intended to include one or more items and can be used interchangeably with "one or more". In addition, as used in this article, the term "group (set)" is intended to include one or more items (for example, related items, unrelated items, combinations of related items and unrelated items, etc.), and can be used interchangeably with "one or more". When it is intended to be only one item, the term "one" or similar language is used. In addition, as used in this article, the terms "has", "have", "having (having)" etc. are intended to be open terms. In addition, the phrase "based on" is intended to mean "based at least in part on", unless otherwise clearly stated.

Claims

1. A method for generating voice and video information based on input, characterized in that: The method comprises: Receiving, by a device, text input comprising a sequence of text components, wherein the text components are characters; Determining, by the device and using a duration model, the respective temporal durations of the text components; The device generates a first spectrum group based on the text component sequence; generating, by the device, a second spectrum group based on the first spectrum group and the respective temporal durations of the text component sequence, wherein the second spectrum group is generated by copying the respective spectra in the first spectrum group based on the respective temporal durations of the basic text components corresponding to the spectra; generating, by the device, a spectrogram frame based on the second spectrum group; generating, by the device, an audio waveform based on the spectrogram frame; generating, by the device, video information corresponding to the audio waveform; and providing, by the device, the audio waveform and the corresponding video as outputs of the device based on the video information; The text input is obtained by: receiving as input an input video including a corresponding input audio waveform; generating, by the device, input video information corresponding to the input audio waveform; generating, by the device, an input spectrogram frame based on the input audio waveform; generating, by the device, a first input spectrum group based on the input spectrogram frame; generating, by the device, a second input spectrum group based on the first input spectrum group; and The text input is determined by the device using the duration model.

2. The method according to claim 1, characterized in that The method further comprises: The duration model is trained.

3. The method according to claim 1, characterized in that The method further comprises: receiving, by the device, information corresponding to an emotional state associated with the text input, and wherein the audio waveform and corresponding video provided as output of the device are based on the information corresponding to the emotional state.

4. The method according to any one of claims 1 to 3, characterized in that The device concurrently provides the audio waveform and the corresponding video as outputs of the device based on the video information.

5. The method according to claim 2, wherein: The training duration model includes multi-task training.

6. The method according to any one of claims 1 to 3, characterized in that The output audio waveform and the output corresponding video are applied to the avatar.

7. The method according to any one of claims 1 to 3, characterized in that The second spectrum group includes Mel-frequency cepstral spectra.

8. The method according to claim 2, characterized in that Training the duration model includes using prediction frame groups and training text components.

9. A computer device, characterized in that: The computer device comprises: at least one memory configured to store program code; At least one processor is configured to read the program code and operate according to instructions of the program code to execute the method according to any one of claims 1 to 8.

10. A device for generating voice and video information based on input, characterized in that: The device comprises: A receiving module, configured to receive a text input including a sequence of text constituent elements, wherein the text constituent elements are characters; a determination module, configured to determine the temporal duration of each of the text constituent elements using a duration model; A first generating module, configured to generate a first spectrum group based on the text constituent element sequence; a second generating module, configured to generate a second spectrum group based on the first spectrum group and the respective temporal durations of the text constituent sequence, wherein the second spectrum group is generated by copying the respective spectrums in the first spectrum group based on the respective temporal durations of the basic text constituents corresponding to the spectrums; a third generating module, configured to generate a spectrum graph frame based on the second spectrum group; a fourth generating module, configured to generate an audio waveform based on the spectrogram frame; a fifth generating module, configured to generate video information corresponding to the audio waveform; and an output module for providing the audio waveform and corresponding video as output of the apparatus based on the video information; The text input is obtained by: receiving an input video including a corresponding input audio waveform; generating input video information corresponding to the input audio waveform; generating an input spectrogram frame based on the input audio waveform; generating a first input spectrum group based on the input spectrogram frame; generating a second input spectrum group based on the first input spectrum group; and The text input is determined using the duration model.

11. A non-transitory computer-readable medium storing instructions, characterized in that: The instructions include one or more instructions that, when executed by one or more processors of a device, cause the one or more processors to perform the method according to any one of claims 1 to 8.

Images