Text to synchronized joint video and audio generation

Abstract

Methods and apparatus for generating a synchronized video-audio pair. According to an example embodiment, a method of generating a synchronized video-audio pair includes: applying one or more text inputs to a generative model, the one or more text inputs including a speech text, the generative model including a neural network; with the generative model, converting the speech text into an audio segment for the synchronized video-audio pair; and with the generative model, generating a video segment for the synchronized video-audio pair, the video segment including a talking head having lip movements corresponding to the speech text and in synchronization with the audio segment.

Description

TEXT TO SYNCHRONIZED JOINT VIDEO AND AUDIO GENERATION 1. Cross-Reference to Related Applications

[0001] This application claims the benefit of priority from U.S. Provisional Application No.63 / 555,166, filed on February 19, 2024, and European Patent Application No.24171641.4, filed April 22, 2024, which is incorporated by reference herein in its entirety. 2. Field of the Disclosure

[0002] Various example embodiments relate to generating a talking face video having lip-syncing that matches an arbitrary speech segment. 3. Background

[0003] With the continuing rise in the consumption of audio-visual content, rapid video content creation has become an important need. For example, a lecture series, a movie, or a public address can be translated to a desired target language to become accessible to millions of new viewers. An important aspect of translating the corresponding talking-face videos to a different language or creating new talking-face videos includes producing a correct lip sync that matches the target speech segment in the target language. Consequently, various approaches to lip-syncing a talking-face video to an arbitrarily selected speech segment are currently being developed. BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS

[0004] Various embodiments disclosed herein achieve text-to-(synchronized joint video and audio) generation and provide users with multiple selectable options for generating synchronized video-audio pairs. In various examples, the generative model is configured to accept different types of text as input, thereby making it convenient for everyday user applications. The use of low-bitrate texts as the model’s inputs significantly reduces transmission costs associated with the generation of synchronized video-audio pairs compared to that incurred under some previous frameworks. Various enhancements of the base generative model focus on expanding video caption categories, improving the generation speed, fine-tuning a pre-trained model for unlabeled data guidance, and enabling talking-face / head video generation from a single static image.

[0005] According to an example embodiment, provided is a method of generating a synchronized video-audio pair, the method comprising: applying one or more text inputs to agenerative model, the one or more text inputs including a speech text, the generative model including a neural network; with the generative model, converting the speech text into an audio segment for the synchronized video-audio pair; and with the generative model, generating a video segment for the synchronized video-audio pair, the video segment including a talking head having lip movements corresponding to the speech text and in synchronization with the audio segment.

[0006] The generative model can include denoising processing based on a diffusion model, such as a denoising U-net, examples of which are provided further herein (see Figure 6A). The denoising U-net model is configurable to perform both unconditional and conditional generation of audio-video pairs. The speech text may be processed by a pretrained text-to-speech model of the generative model (for example, YourTTS, described further herein), which generates speech audio which can be provided as a condition to the denoising U-net for generating audio for the audio-video pairs. When no speech text is input, the denoising U-net is configured to process random distributions for video and audio and generate an audio-video pair based on this random input.

[0007] According to another example embodiment, provided is a non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising the above method.

[0008] According to yet another example embodiment, provided is an apparatus for generating a synchronized video-audio pair, the apparatus comprising: at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: apply one or more text inputs to a generative model, the one or more text inputs including a speech text, the generative model including a neural network; with the generative model, convert the speech text into an audio segment for the synchronized video-audio pair; and with the generative model, generate a video segment for the synchronized video-audio pair, the video segment including a talking head having lip movements corresponding to the speech text and in synchronization with the audio segment.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Other aspects, features, and benefits of various disclosed embodiments will become more fully apparent, by way of example, from the following detailed description and the accompanying drawings, in which:

[0010] FIG.1 is a block diagram illustrating a process for a video-audio delivery pipeline according to an embodiment.

[0011] FIGS.2A-2D are block diagrams illustrating various examples of a receiver that can be used in the process of FIG.1 according to some embodiments.

[0012] FIGS.3A-3B illustrate training and inference processes used with the generative model of the process of FIG.1 according to some examples.

[0013] FIG.4 is a block diagram illustrating a denoising diffusion probabilistic model (DDPM) that can be used in the generative model of the process of FIG.1 according to some examples.

[0014] FIG.5 is a block diagram illustrating a U-Net that can be used in the DDPM of FIG.4 according to some examples.

[0015] FIGS.6A-6C are block diagrams illustrating a double-branch U-Net block that can be used in the DDPM of FIG.4 according to some examples.

[0016] FIG.7 is a block diagram illustrating a double branch U-Net that can be used in the DDPM of FIG.4 according to some examples.

[0017] FIG.8 is a block diagram illustrating a pipeline for unconditional generation of audio-video pairs that can be used in the process of FIG.1 according to some examples.

[0018] FIG.9 is a block diagram illustrating a pipeline for conditional generation of audio-video pairs that can be used in the process of FIG.1 according to some examples.

[0019] FIG.10 is a block diagram illustration a pipeline of a joint video-audio classifier that can be used with pipeline of FIG.9 according to some examples.

[0020] FIG.11 is a block diagram illustrating a pipeline for conditional generation of audio-video pairs that can be used in the process of FIG.1 according to further examples.

[0021] FIG.12 is a block diagram illustrating a pipeline that can be used in the process of FIG.1 according to various embodiments.

[0022] FIG.13 is a block diagram illustrating a modification to the backward processing of the DDPM of FIG.4 that can be used in the generative model of the process of FIG.1 according to some examples.

[0023] FIG.14 is a block diagram illustrating video prediction that can be used in the process of FIG.1 according to some examples.

[0024] FIGS.15A-15B are block diagrams illustrating a training process and an inference process corresponding to the video prediction illustrated in FIG.14 according to some examples.

[0025] FIG.16 is a block diagram illustrating a pre-trained model that can be used in the process of FIG.1 according to some examples.

[0026] FIG.17 is a block diagram illustrating a modification to the backward processing that can be used in the in the process of FIG.1 according to some examples.

[0027] FIG.18 is a block diagram illustrating a further modification to the backward processing that can be used in the in the process of FIG.1 according to some examples.

[0028] FIG.19 is a block diagram illustrating a computing device used to implement the process of FIG.1 according to some examples. DETAILED DESCRIPTION

[0029] Example embodiments disclosed herein are directed at generating video-audio segments with lip synchronization based on text inputs. Some applications of such embodiments may help to solve an ultra-low bit rate video conferencing problem and / or other related problems in the generative taking-face audio / video scenarios. Some examples employ a Denoising Diffusion Probabilistic Model (DDPM) with a double-branch U-Net architecture to produce synchronized video-audio pairs from textual inputs. In some examples, only low bitrate texts need to be transmitted from a source to the video-audio generator, which beneficially reduces bandwidth requirements to the corresponding communication channel. In some examples, the user is provided with multiple options based on which to generate a desired video-audio content. Examples of such options include using video captions, speech text, and / or metadata. In some examples, the user has the flexibility of inputting one or more static images to customize thestyle of the output video. Various embodiments are generally directed at enhancing the efficiency and versatility of multimedia applications.

[0030] FIG.1 is a block diagram illustrating a process (100) for a video-audio delivery pipeline according to an embodiment. The process (100) includes a producer (110) and a receiver (120) connected, directly or indirectly, via a communication link (114). In some examples, the producer (110) intends to share video content, either through an online intermediary or directly, with the receiver (120) who seeks to receive a shared talking-face video. The process (100) enables the producer (110) to transmit via the communication link (114) only low-bitrate texts, thereby significantly reducing the transmission costs compared to the costs of sending a corresponding video through the communication link (114). Based on the received low-bitrate texts, which may include video description texts and / or talking texts, the receiver (120) operates to generate corresponding synchronized video-audio pairs (124, 126) using a generative model (122).

[0031] FIGS.2A-2D are block diagrams illustrating various examples of the receiver (120) that can be used in the process (100) according to some embodiments. The input to the receiver (120) can be one or more texts. In some cases, there are more than one type of text that is being received via the communication link (114). Once the text(s) is (are) received by the receiver (120), a user may consider various options for converting the text(s) into the joint video / audio content. Some examples of such options are individually illustrated in FIGS.2A- 2D.

[0032] FIG.2A illustrates a first example option according to which the received text(s) (202) are directly converted into the synchronized video and audio pair (124, 126) without any additional text modifications. In this case, the video (124) may be based on a default stock video. FIG.2B illustrates a second example option according to which the user alters a video description communicated via the received texts (202) to provide a modified input (204) to the generative model (122). In the example shown in FIG.2B, the video description is changed by the user from “man is talking” to “woman is talking.” Accordingly, the video portion of the output of the generative model (122) becomes a different video (206) featuring a woman, instead of the video (124) used for the first option. FIG.2C illustrates a third example option according to which the user alters the received text(s) (202) such that the pronounced text (speech) is modified to be a different text (208), e.g., by changing one or more words or sentences in the text(s) (202). Accordingly, the audio portion of the output of the generative model (122) becomes a different audio (210) corresponding to the text (208), as indicated in FIG.2C. FIG.2D illustrates a fourth example option according to which the user provides a static image (212) as an additional input to the generative model (122). Accordingly, the video portion of the output of the generative model (122) becomes a different video (214) generated based on the static image (212). In various examples, the static image (212) may cause the video (214) to have a different talking face, head, or style compared to the otherwise used default video, such as the video (124) used in the first option.

[0033] Some embodiments described herein below introduce a novel text to synchronized joint video and audio generation framework that uses text inputs to generate synchronized video-audio pairs. Beneficially, this framework typically leads to a substantial reduction in the transmission bitrates, thereby enhancing the overall efficiency compared to at least some conventional approaches. Additional embodiments described herein below provide users with several flexible options for generating video-audio pairs from texts, e.g., as illustrated in FIGS.2A-2D. This feature empowers users to generate videos that meet their preferences and / or requirements, allowing for a more personalized and satisfying video experience. In at least some examples, the disclosed text-to-speech conditional generation operates in full zero- shot mode, thereby enabling acceptance of substantially any input texts for to the generative model (122). This important capability makes the process (100) highly versatile and applicable to many real-life scenarios. Example Elements of the Process (100)

[0034] In some examples, the generative model (122) is based on a type of neural network that excels at learning the underlying distribution of a training dataset and is subsequently used for generating new instances.

[0035] FIGS.3A-3B pictorially illustrate, at a high abstraction level, training and inference processes (310, 320) used with the generative model (122) according to some examples. In the training phase (310), a neural network of the generative model (122) is exposed to a training dataset, allowing the model to grasp the overall distribution of the data. Subsequently, during the inference phase (320), the trained generative model (122) relies on a random noise distribution (304) to generate a model’s output (322), adhering to the learned characteristics of the training dataset (302). In some examples, the generative model (122) is optionally configured to also respond to additional conditions (306). The conditions (306) may include, e.g., text descriptions and / or images that influence the model’s output (322) and guide the generative model (122) towards producing desired outputs. This ability to conditionallyguide the generative model (122) may beneficially enhance the model’s flexibility and utility for different applications.

[0036] FIG.4 is a block diagram illustrating a denoising diffusion probabilistic model (DDPM) (400) that can be used in the generative model (122) according to some examples. The DDPM (400) uses forward processing (402) and backward processing (404). In some examples, the DDPM (400) surpasses other suitable alternatives in terms of performance for the intended purpose.

[0037] We denote an original sample (410) without noise and its pure Gaussian distribution as ^^^^0and ^^^^^^^^, respectively. In the forward processing (402), noise (412) is progressively added to the original sample ^^^^0over T time steps, gradually transforming the sample (410) into a pure Gaussian noise pattern (418), denoted as ^^^^^^^^. On the other hand, the backward processing (404) is configured to reconstruct the sample (410) from the Gaussian noise pattern (418). The combination of the forward and backward processes (402, 404) enables the DDPM (400) to excel in generating high-quality samples while maintaining the integrity of the original data distribution. Its efficacy makes it an excellent choice for various generative tasks. In general, a goal of the DDPM (400) is to reconstruct ^^^^0from ^^^^^^^^. Therefore, the forward processing (402) following the Markov Chain can be represented using the following expressions:where ^^^^ ∈ [0,^^^^] is the time steps that are pre-defined to gradually add noise from the true data^^^^^^^^to the pure Gaussian noise ^^^^^^^^. The diffusion forward process (402) is used to get the sample^^^^^^^^ from ^^^^^^^^−^^^^ by adding noise (412) using the normal distribution ^^^^(∙). The factor�1 − ^^^^^^^^ isthe mean value, and ^^^^ is a pre-defined variance. ^^^^ is identity matrix. Using the property of the forward pass, we can compute ^^^^^^^^from ^^^^0using the following expression:where ^^^^^^^^ = 1 − ^^^^^^^^; ^^�^^^^^^ = ∏^^^^^^^^=1 ^^^^^^^^ ; and ^^^^ is the noise from the normal distribution.

[0038] For the backward processing (404), a goal is to train a model ^^^^ that canapproximate ^^^^(^^^^^^^^−1|^^^^^^^^ , ^^^^0) using ^^^^^^^^(^^^^^^^^−1|^^^^^^^^). Therefore, we have the following formula forthe reverse process: ^^^^^^^^(^^^^^^^^−1|^^^^^^^^) = ^^^^(^^^^^^^^−1; ^^^^^^^^(^^^^^^^^ , ^^^^); ∑^^^^ (^^^^^^^^ , ^^^^) ) (4)where ^^^^^^^^is the mean value predicted from the trained model ^^^^. In some examples, we use a U- Net to implement the model ^^^^, and the training objective involves minimizing the loss function between the predicted noise and ground-truth noise, or between the predicted de-noised sample and the ground-truth sample (4100. In mathematical terms, the training objective can be formulated as follows:where ^^^^^^^^(^^^^^^^^ , ^^^^) is the predicted noise from the U-Net.

[0039] FIG.5 is a block diagram illustrating a U-Net (500) that can be used in the model (400) according to some examples. In the example shown, the U-Net (500) is a neural network having a U-shaped topology that includes a down-sampling branch (502) and an up-sampling branch (504) connected as indicated in FIG.5. Skip connections between the branches (502, 504) indicated by the horizontal arrows are added at different scales of the latent features. In some examples, self-attention layers are inserted between convolution layers. In some examples, implementation of the U-Net (500) may benefit from certain features described in Ronneberger, O., Fischer, P., and Brox, T., “U-net: Convolutional networks for biomedical image segmentation,” Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp.234-241, which is incorporated herein by reference in its entirety.

[0040] In some examples, to optimize training costs and enhance the quality of generated outcomes, we employed pre-trained models. One example pre-trained model used for this purpose is the YourTTS model disclosed in Casanova Edresson, Julian Weber, Christopher D. Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir A. Ponti, “Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,” International Conference on Machine Learning, pp.2709-2720, PMLR, 2022, which is incorporated herein by reference in its entirety. Another example pre-trained model used for this purpose is the Wav2Lip model disclosed in Prajwal, K. R., Rudrabha Mukhopadhyay, Vinay P. Namboodiri, and C. V. Jawahar, “A lip sync expert is all you need for speech to lip generation in the wild,” Proceedings of the 28th ACM international conference on multimedia, pp.484-492, 2020, which is also incorporated herein by reference in its entirety. In some examples, YourTTS is utilized to convert text into speech audio, while Wav2Lip is applied to enhance the synchronization between video and audio elements. By leveraging these two pre-trained models, the generative model (122) can be configured to efficiently generate high-quality results with improved video- audio coherence. In other examples, other suitable pre-trained models can also be used in the generative model (122).

[0041] In some examples, YourTTS provides a powerful zero-shot text-to-speech audio model, boasting pre-training with multiple speakers and languages. In various examples, YourTTS can take inputs of texts, language IDs, and speaker IDs, thereby supporting various options, some of which are described above in reference to FIGS.2A-2D. For English speech audio, the fixed language ID can be set to “English.” Regarding the speaker ID, in the absence of caption input, one can randomly select a speaker index from the entire pool of speakers. However, when a caption is provided, the choice of female or male speaker index is determined accordingly. The resulting audio output is converted into a tensor format, serving as a guiding element for the subsequent video generation process. A benefit of this approach lies in the ability to achieve text-to-speech video-audio generation without inflating the training parameters of the diffusion model. Indeed, the utilization of YourTTS streamlines at least some embodiments and empowers the process (100) to achieve video-audio generation with enhanced flexibility. By incorporating the capabilities of YourTTS, the process (100) can readily adapt and generate speech audio from diverse texts, pre-defined language IDs, and pre-defined speaker IDs. It should be noted though that the pre-defined language IDs and speaker IDs are limited by the training data used in the pre-training. The aforementioned features beneficially enhance the versatility of the process (100), thereby enabling the use thereof in a wide range of applications and scenarios. For example, the process (100) can be configured to produce video-audio content with various linguistic nuances and speaker characteristics, which can significantly enrich the overall user experience.

[0042] Example embodiments are directed to generating synchronized video-audio pairs (124, 126) for talking heads or faces. As such, particular attention is given to achieving sufficiently precise lip motion and audio synchronization. However, the subtlety of lip motions poses a challenge, making the synchronization task challenging. Additionally, discrepancies in lip-sync can be easily noticeable in videos compared with natural ambient sounds like ocean waves or wind, impacting the overall viewing experience. To address these concerns, some embodiments integrate a pre-trained Wav2Lip model into the process (100). This model is specifically designed to enhance lip pose and audio synchronization. Wav2Lip leverages a pre- trained Lip-synchronization expert during training, with the objective of minimizing synchronization loss. Some embodiments, strategically incorporate this model after the final de- noising step, thereby enabling the corresponding process (100) to significantly improve the synchronization of generated audio-video pairs (124, 126). By leveraging the capabilities of Wav2Lip, we aim to produce talking face or head videos with accurate lip motion and audio coherence, thereby delivering a seamless and immersive viewing experience.

[0043] FIGS.6A-6C are block diagrams illustrating a double-branch U-Net block (600) that can be used in the model (400) according to some examples. More specifically, FIG.6A is a block diagram illustrating the overall architecture of the U-Net block (600). FIG.6B is a block diagram illustrating a video convolution block (610) used in the double-branch U-Net block (600). FIG.6C is a block diagram illustrating an audio convolution block (620) used in the double-branch U-Net block (600). For illustration purposes and without any implied limitations, the double-branch U-Net block (600) is shown in FIG.6A as having two instances of the video convolution block (610) and two instances of the audio convolution block (620). In other embodiments, the number of such instances may be different from two.

[0044] As previously mentioned, the diffusion model (400) can be implemented using the U-Net (500) as its fundamental network architecture. However, the U-Net (500) is configured to accept only one type of input. In various examples, the double-branch U-Net block (600) is used to modify the U-Net (500) to enable the resulting modified U-Net to accept and process both video and audio inputs in parallel.

[0045] FIGS.6B and 6C illustrate an example structure of the video convolution block (610) and the audio convolution block (620). The video convolution block (610) includes a group normalization block (612) and a Swish activation function block (614). A video feature (615) outputted by the block (614) undergoes processing in a 3D convolutional layer (616). Similarly, the audio convolution block (620) includes a group normalization block (622) and a Swish activation function block (624). An audio feature (625) outputted by the block (624) undergoes processing in a 1D convolutional layer (626). In some examples, one or both of the Swish activation functions (614) and (624) can be replaced by other types of activation functions, such as ELU or GELU activation functions. As shown in FIG.6A, the double-branch U-Net block (600) includes both video and audio convolution blocks (610, 620), augmented by a cross attention block (630) to facilitate learning the alignments between the audio and video elements. In some examples, the cross-attention function ^^^^(∙) used in the block (630) is as follows:^^^^^^^^ = ^^^^^^^^^^^^^^^^^^^^^^^^�^^^^^^^^^^^^^^^^^^^^^^^^^^^^(^^^^^^^^)� (7)where ^^^^^^^^and ^^^^^^^^are the video and audio, respectively, in the ^^^^-th time step; ^^^^, ^^^^ and ^^^^ are query, key, and value, respectively; and ^^^^^^^^denotes the dimension of ^^^^^^^^. The cross-attention function ^^^^(^^^^^^^^,^^^^^^^^)can be computed similarly, in a symmetrical way.

[0046] FIG.7 is a block diagram illustrating a double branch U-Net (700) that can be used in the model (400) according to some examples. The U-Net (700) can be considered to be a modification of the U-Net (500) by incorporating therein double-branch U-Net blocks (600). For illustration purposes and without any implied limitations, both a down-sampling branch (702) and an up-sampling branch (704) of the U-Net (700) include three respective double-branch U- Net blocks (600), which are labeled (6001-6003) and (6004-6006), respectively. In other examples, a different from three number of the double-branch U-Net blocks (600) can be used in each of the down-sampling and up-sampling branches (702, 704). Skip connections between the branches (702, 704) indicated by the horizontal block arrows are added at different scales of the latent features corresponding to different respective sizes of the double-branch U-Net blocks (600i). In some examples, self-attention layers are inserted between convolution layers.

[0047] The U-Net architecture (700) enables synchronized processing of video and audio, thereby providing precise and coherent generation of the video-audio pairs (124, 126). In some examples, further modifications of the U-Net architecture (700) can be implemented by inserting one or more additional branches into the constituent U-Net block blocks (600i). Note that more branches typically result in more training parameters and possibly more cross-attention blocks to catch their alignments.

[0048] FIG.8 is a block diagram illustrating a pipeline (800) for unconditional generation of audio-video pairs that can be used in the process (100) according to some examples. The pipeline (800) includes the double branch U-Net (700) and a pre-trained lip- synchronization model (810) serially connected to one another. In some examples, the lip- synchronization model (810) can be or include the above-mentioned Wav2Lip model. Two distinct random distributions (802, 804), each of a different respective size, are employed as inputs to the U-Net (700), which is configured to perform de-noising operations. After N times of de-noising, the outputs of the U-Net (700) are used as inputs to the lip-synchronization model (810), which serves to the enhance the synchronization a final output video-audio pair (824, 826). The pipeline (800) underwent training with numerous synchronized audio-video sentence pairs, enabling the pipeline (800) to attain the unconditional generation capability.

[0049] FIG.9 is a block diagram illustrating a pipeline (900) for conditional generation of audio-video pairs that can be used in the process (100) according to some examples. In the example shown, a condition (908) applied to the U-Net (700) includes a video caption. The condition (908) serves to control the generation process by providing captions as input. For example, when we supply the caption “a man is talking” to an unconditional generative model,the model can produce video-audio content featuring a man’s head. To realize the captioned conditional generation, we offer two distinct approaches: classifier guidance and conditional generative diffusion model, which are described in more detail below. The pipeline (900) is generally similar to the pipeline (800), except for the application of the caption condition (908) to the denoising U-Net (700). Through the inclusion of video captions, the pipeline (900) achieves fine-grained control over the generated video-audio pairs, thereby providing an enhanced level of customization and specificity.

[0050] FIG.10 is a block diagram illustration a pipeline (1000) of a joint video-audio classifier (1010) that can be used with pipeline (900) according to some examples. The classifier (1010) is trained to identify and classify the video-audio pairs (1002, 1004) using captions. The training of the classifier (1010) includes supervised learning with labeled video-caption pairs. The unconditional model is guided using the score of the classifier given an input caption ^^^^. To this end, we employed U-Net blocks (600) used in the main generative model. However, two differences were introduced: 1) the output blocks of the U-Net block (600) were removed, and 2) a pooling layer and a fully connected layer were added to serve as a classification head. Initially, we focused on training a video classifier alone. Thereafter, the joint video and audio classifier (1010) was introduced. The classifier (1010) differs from a conventional classifier in that it constitutes a type of noisy classifier capable of accepting noisy samples and the time step ^^^^ (1006) as inputs. The added input (1006) for time step t is important, as the classifier (1010) plays a role in guiding each de-noising step of the backward diffusion processing (404).

[0051] Once the classifier (1010) is trained, we leverage its guidance for the unconditional generative model. The unconditional generative model can be expressed aswhere ^^^^ and ^^^^ refer to video and audio, respectively. On the otherhand, we define the joint video-audio classifier as ^^^^∅(^^^^| (^^^^^^^^−1, ^^^^^^^^−1)), with y denoting the inputcaption. Consequently, the classifier-guided generation can be expressed as:where ^^^^ is a normalizing constant, which is used to make any probability density function to have a total probability of 1. Conceptually, we can explain the classifier guidance as follows.One goal is to predict the score function ^^^^^^^^^^^^^^^^^^^^( (^^^^^^^^ , ^^^^^^^^) | ^^^^). We define the ^^^^ as ^^^^(^^^^^^^^,^^^^^^^^). Usingthe Bayes Rule:since ^^^^^^^^^^^^^^^^^^^^(y)=0, Eq. (9) can be rewritten as: ^^^^^^^^^^^^^^^^^^^^(^^^^^^^^ , ^^^^^^^^ | ^^^^) = ^^^^^^^^^^^^^^^^^^^^(^^^^^^^^ , ^^^^^^^^) + ^^^^^^^^^^^^^^^^^^^^(^^^^|^^^^^^^^ , ^^^^^^^^) (10)From Eq. (10), it is readily apparent that the score of caption conditional generation is equal to the unconditional score plus the classifier score given the input caption ^^^^.

[0052] The classifier-guided model, while effective, may incur significant time and computational costs when training on large datasets. As an alternative, one can adopt a different approach by directly incorporating caption embeddings into the diffusion training process, thereby eliminating the need for a separate classifier. In some examples, we use a conditionaldiffusion model. Specifically, we set ^^^^ = [^^^^^^^^ ,^^^^^^^^] as the time step and caption embedding pair.It is important to bear in mind that the diffusion model systematically adds noise to true data ^^^^0up to the full Gaussian noise ^^^^^^^^. Each time step represents one stage in the diffusion process and is pre-defined using a linear method from [0, T], where T denotes to the total number of time steps. In one example, for the caption embedding, we use 0 and 1 to represent the caption “a man is talking” and the “a woman is talking,” respectively. To seamlessly integrate the caption embeddings into each U-Net block (600), we employ the adaptive group normalization (AdaGN) layer. By doing so, we establish the captioned conditional layer as follows: ^^^^^^^^^^^^^^^^^^^^(ℎ,^^^^) = ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^(ℎ) + ^^^^^^^^ (11)where ℎ is the activations after the first convolutional layer of each block (600).

[0053] FIG.11 is a block diagram illustrating a pipeline (1100) for conditional generation of audio-video pairs that can be used in the process (100) according to further examples. For speech text conditional generation, the pipeline (1100) is configured to use a pre- trained text-to-speech model (1110) to convert any given input text (114, 202, 204, 208, 1108) into speech audio. Subsequently, this speech audio is used to guide the video generation process. In some examples, the pre-trained text-to-speech model (1110) is the commercially available YourTTS model, which is a versatile multi-speaker and multi-language text-to-speech model. With the speech text (1108) as an input, the pipeline (1100) enhances the flexibility and expressiveness of the available embodiments, thereby empowering the users to produce compelling video content guided by diverse textual inputs.

[0054] We denote the speech audio converted from the text (1108) by the pre-trained text-to-speech model (1110) as ^^�^^0. Then, one can obtain ^^�^^^^^^, e.g., as follows:Once we have ^�^^^^^^^, we replace ^^^^^^^^with ^�^^^^^^^in the original conditional backward processing. As aresult, we haveFinally, we update the video ^^�^^^^^^−1 using thefollowing:where C is a constant for the conditional scale that can be set by the user.

[0055] Incorporating static images for conditional generation can be illustrated with two example cases: one with the speech text input (1108) and the other without such speech text input.

[0056] In the case with the speech text input (1108), we employ the text-converted speech audio to drive the static image using the pre-trained lip-synchronization model (810). In some examples, the pre-trained lip-synchronization model (810) is the commercially available Wav2Lip model. The lip-synchronization model (810) accepts image and audio as inputs and, in typical use cases, the inputs are not synchronized. The lip-synchronization model (810) can generate synchronized video using the synchronization expert model. The model can accept both latent and audio / image native format as inputs. This feature allows us to synchronize the static image with the corresponding speech audio, producing a coherent and immersive video- audio combination at the output.

[0057] When the speech text input is absent, the process (100) is configured to generate the audio using a random distribution and subsequently uses it to drive the static image. By leveraging this approach, we can still achieve dynamic and engaging video content, even in scenarios where a specific speech text input (1108) is not available.

[0058] FIG.12 is a block diagram illustrating a pipeline (1200) that can be used in the process (100) according to various embodiments. The pipeline (1200) incorporates various features of the above-described pipelines (800, 900, 1100), which enables the corresponding model to exhibit significant flexibility in terms of accommodating various types of conditional inputs, including fully conditional inputs, partial conditional inputs, or no conditional inputs. Note that in scenarios in which no speech text input is available, a random distribution is inputted for audio generation, and the pre-trained text-to-speech model (1110) is removed from the pipeline or is bypassed. In the latter examples, despite the absence of a specific speech text input, the pipeline (1200) still produces a dynamic video content through generation of audio from a random distribution and aligning the audio and video components to create a synchronized output pair. Note that in the unconditional setting, the pre-trained lip- synchronization model (810) is not used since the audio is generated from the random distribution. In the text conditional setting, the pre-trained lip-synchronization model (810) is optional as well if the input speech text is in the distribution of the training dataset, such as the same word as the training set or the words containing similar pronunciation. In various examples, the pipeline (1200) is configured to accept any speech text from the user. Byaccommodating various conditional inputs, the pipeline (1200) offers the users significant freedom and versatility in creating video-audio content that suits their specific needs and preferences. This adaptability beneficially makes the pipeline (1200) into a powerful tool for generating rich and diverse multimedia content.

[0059] The remainder of this section provides a description of the training procedures that can be used to implement various embodiments of the process (100). The following aspects of the training are covered: (i) Diffusion Training; (ii) Classifier Training; (iii) Inference Setting; and (iv) Training Dataset.

[0060] In some examples, in the U-Net setting, we configured the number of ResNet blocks for each U-Net block (600) as two, and the number of head channels was set to 64. The video and audio fps rates were selected to be 16 and 16,000, respectively. A fixed learning rate of 0.0001 was employed, and the model saving was performed every 5,000 steps during the training process. To schedule the diffusion noise, we utilized a linear method and the total number N of diffusion steps during inference was set to 1,000. To ensure meaningful outcomes, the model underwent training for a minimum of 50,000 steps. A primary objective during training was to minimize the loss function between the predicted noise and the ground-truth noise. Furthermore, all video inputs were resized to 64 × 64. We define B as the batch size, F is the video frames, C is the channel number, and ^^^^^^^^as input audio data points. Therefore, the input video tensor size is [B, F, C, 64, 64] and audio tensor size is [B, 1, ^^^^^^^^].

[0061] In some examples, during the training of the classifier (1010), we set the initial learning rate to 0.001 and implemented a learning rate decay to optimize its performance. To prevent overfitting and enhance generalization, we set the weight decay to 0.05. Given the classifier’s role in handling noisy input, we utilized the uniform method as the schedule sampler. We conducted a total of 300,000 iterations to ensure robust convergence and acquisition of meaningful classification results.

[0062] In some examples, during the inference phase, we set the total number of de- noising steps to 1,000 and employed a linear noise schedule to effectively guide the de-noising process. To ensure the highest possible generation quality, we opted for the DDPM reverse sampling method.

[0063] In some examples, the training dataset comprises a total of 24 speakers, equally divided to include 12 females and 12 males. For each speaker, there are 40 videos available. To optimize the training efficiency and cost, we opted to extract a one-second segment from eachvideo, featuring one digit alongside one word. Consequently, the entire dataset spans a duration of 16 minutes. To enable caption conditional generation, we labeled the videos with two distinct video captions, namely “a man is talking” and “a woman is talking." The original resolution of each video is 360 × 288. Optional Enhancements to the Process (100)

[0064] FIG.13 is a block diagram illustrating a modification to the backward processing (404) that can be used in the generative model (122) according to some examples. To achieve high quality content generation, the backward processing (404) of the DDPM (400) uses many de-noising steps to reconstruct the sample from pure Gaussian noise. Such multiple de-noising steps may take a relatively long time that may present a problem for long video generation. In some examples, one can change the training objective to enforce the self-consistency property based on the consistency model illustrated in FIG.13. Instead of iterative de-noising used in the backward processing (404), the consistency model is configured to achieve one step generation from any point along the t axis to ^^^^0. A goal is to approximate the consistency function ^^^^^^^^, thereby enforcing the self-consistency property. The self-consistency property ensures that no matter which point we select along the t axis, the result will be ^^^^0. Therefore, the trained consistency model empowers one step generation. Some embodiments may benefit from the use of certain features of the consistency model disclosed in Song, Yang, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever, “Consistency models,” (2023), which is incorporated herein by reference in its entirety.

[0065] FIG.14 is a block diagram illustrating video prediction (1400) that can be used in the process (100) according to some examples. In some examples, when provided with several static images (212) representing continuous video frames as an input (referred to as the image batch condition), the generative model (122) can generate natural talking head videos with dynamic head motion and eye blinking. However, when using only one static image as input, the model’s output mainly focuses on lip movements, resulting in a video with limited head motion. To address this limitation of video generation from one static image, the generative model (122) can be configured to use the single input image (212) as the first frame and predict a plurality of following frames using the video prediction (1400).

[0066] In some examples, the training objective of the generative model (122) enhanced with video prediction (1400) is formulated as: ^^^^(^^^^) = ^^^^^^^^,^^^^0~^^^^^^^^^^^^^^^^,^^^^~(0,^^^^)�‖^^^^−^^^^^^^^(^^^^^^^^|^^^^,^^^^)‖22� (14)where ^^^^ is the static image (212) we input as the first frame. The basic outline of the video prediction (1400) is illustrated in FIG.14. We input only one image and iteratively predict (1402) next images (1412n). The audio part can undergo a similar process if we want to predict audio from audio samples. During training, the input of the model can be ground-truth frame combined with the pure Gaussian noise and the past known frame can be used as a condition added to the U-Net block (600).

[0067] FIGS.15A-15B are block diagrams illustrating a training process (1502) and an inference process (1504) corresponding to the video prediction (1400) according to some examples. The training process (1502) is configured in accordance with Eq. (14). The inference process (1504) produces a talking head video corresponding to the single input image (212) with dynamic head motion and eye blinking.

[0068] In some examples, due to limitations of the GPU memory, we resized the training video resolution to 64 × 64. To address a desire for a higher resolution video output, two approaches for super-resolution can be used. The first approach involves training a separate super-resolution model using the same training dataset as that used for the generative model (122). Alternatively, for a zero-shot video super-resolution, a pre-trained model can be used. However, it is important to note that employing pre-trained zero-shot models may result in abnormal coloration, as these models can only generate instances from their pre-existing dataset.

[0069] Super-resolution methods are well-established and offer various possibilities to increase the video resolution. Inclusion of the above-mentioned two super-resolution approaches as options for the process (100) is relatively straightforward because they share the same baseline code base, thereby simplifying their integration. Other suitable super resolution approaches can also be in further alternative embodiments.

[0070] One limitation of the process (100) is the use of labeled video data for the caption conditional model. This feature inherently relies on the availability of the corresponding video captions for each video. However, the process of labeling diverse video-caption pairs may be challenging and / or time-consuming. Additionally, the need for labeling typically restricts the input captions to match those present in the training set. To address this limitation, some embodiments are configured to use caption conditioning without the need for explicit labeling, thereby allowing the model to accept any input caption. Further embodiments extend this approach to images conditional generation as well.

[0071] FIG.16 is a block diagram illustrating a pre-trained vision-language model (1600) that can be used in the process (100) according to some examples. The model (1600) is constructed in accordance with CLIP (Contrastive Language-Image Pre-Training) disclosed in Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, et al. “Learning transferable visual models from natural language supervision,” International conference on machine learning, pp.8748-8763, PMLR, 2021, which is incorporated herein by reference in its entirety. CLIP is a robust and sophisticated vision and text model, which is trained on vast amounts of image-text data. Its core objective revolves around predicting image-text matches. In the example shown, the model (1600) includes a text encoder (1610) and an image encoder (1620).

[0072] In example embodiments, we extend the image CLIP into a video CLIP model using a temporal convolution. A resulting image encoder of video CLIP can process multiple input frames. The basic idea behind this approach involves utilizing the pre-trained text encoder and image encoder from CLIP. The CLIP image encoder is then fine-tuned to create a noised image encoder specific to the dataset used with the process (100). The initial process of fine- tuning the CLIP image encoder for this purpose can be summarized as follows: 1) We define the pre-trained image encoder as ^^^^^^^^(1620), and the noised image encoder we want to get as ^^�^^^^^^ (1720).2) The pure video label is generated from ^^^^^^^^ using ^^^^^^^^^^^^^^^^ = ^^^^^^^^(^^^^0). We generatenoised image samples using the same formulation as in DDPM, which is expressed as:3) We can obtain the noised image feature using ^^^^^^^^^^^^ = ^^�^^^^^^(^^^^^^^^ , ^^^^).4) The training objective of ^^�^^^^^^is to maximize the cosine similarity between ^^^^^^^^^^^^^^^^and ^^^^^^^^^^^^.

[0073] Once a noised image encoder ^^�^^^^^^ (1720) is trained, it can be used to guide theunconditional generation via the model (1600). We define the function to compute the similarity as ^^^^(∙). We revisit the classifier guided generation as:where ^^^^∅(∙) is the trained classifier, and ^^^^ is the input text. The pre-trained CLIP text encoder is defined as ^^^^^^^^(∙) (1710). By using a fine-tuned CLIP image encoder (1720), we substantially follow the equation for caption conditional generation:

[0074] FIG.17 is a block diagram illustrating a modification (1700) to the backward processing (404) that can be used in the generative model (122) according to some examples. The modification (1700) supports the process of caption conditional generation using a fine- tuned CLIP model (1600). For illustration purposes and without any implied limitations, the modification (1700) is shown to include only two de-noising steps (17021, 17022). In various examples, the modification (1700) may include more than two de-noising steps (1702i). At each denoising step (1702i), a text encoder (1710) and an image encoder (1720) provide inputs to a classifier (1730), the output of which guides the processing of the denoising step (1702i).

[0075] FIG.18 is a block diagram illustrating a further modification (1800) to the backward processing (404) that can be used in the generative model (122) according to some examples. Compared to the modification (1700), each denoising step (1802i) the modification (1800) is additionally guided by the output of an image classifier (1830) configured to use reference images. For the modification (1800), Eq. (17) is modified to become:where ^^^^1and ^^^^2are the weights of text and reference images, respectively. Example Hardware

[0076] FIG.19 is a block diagram illustrating a computing device (1900) used to implement the process (100) according to various embodiments. In some examples, two or more instances of the computing device (1900) can be used to implement the process (100). The computing device (1900) comprises input / output (I / O) devices (1910), a processing engine (1920), and a memory (1930). The I / O devices (1910) may be used to enable the device (1900) to receive various input signals (1902) and to output various output signals (1904). For example, the I / O devices (1910) may be operatively connected to send and / or receive signals via the communication link (114).

[0077] The memory (1930) may have buffers to receive data. Once the data are received, the memory (1930) may provide parts of the data to the processing engine (1920) for processing therein. The processing engine (1920) includes a processor (1922) and a memory (1924). The memory (1924) may store therein program code, which when executed by the processor (1922) enables the processing engine (1920) to perform various processing operations, including but not limited to at least some operations associated with the process (100) and / or the corresponding training procedures.

[0078] Various aspects of the present disclosure may be appreciated from the following Enumerated Example Embodiments (EEEs):

[0079] EEE 1. method of generating a synchronized video-audio pair, the method comprising: applying one or more text inputs to a generative model, the one or more text inputs including a speech text, the generative model including a neural network; with the generative model, converting the speech text into an audio segment for the synchronized video-audio pair; and with the generative model, generating a video segment for the synchronized video-audio pair, the video segment including a talking head having lip movements corresponding to the speech text and in synchronization with the audio segment.

[0080] EEE 2. The method of EEE 1, wherein the one or more text inputs further include one or more texts selected from the group consisting of: a video caption; a description of the talking head; a condition for the video segment or for the audio segment; and a description of the video segment.

[0081] EEE 3. The method of EEE 1 or 2, further comprising: receiving a first text input of the one or more text inputs via a communication link; modifying the first text input based on a user input; and applying the modified text input to the generative model, wherein the synchronized video-audio pair is generated based on the modified text input.

[0082] EEE 4. The method of any of EEEs 1 to 3, further comprising applying a static image to the generative model, wherein the video segment is generated based on the static image.

[0083] EEE 5. The method of any of EEEs 1 to 4, wherein the generative model includes denoising processing of a diffusion model.

[0084] EEE 6. The method of EEE 5, wherein the denoising processing is implemented with a denoising U-Net.

[0085] EEE 7. The method of EEE 6, wherein the denoising U-Net includes plurality of double-branch U-Net blocks corresponding to different scales of a latent feature, each of the double-branch U-Net blocks having a respective video branch and a respective audio branch.

[0086] EEE 8. The method of EEE 7, wherein the respective video branch and the respective audio branch are connected via a respective cross attention block configured to facilitate relative alignment between the video segment and the audio segment.

[0087] EEE 9. The method of any of EEEs 6 to 8, , wherein the one or more text inputs include a video caption; and wherein the denoising U-Net is configured to receive the video caption as a condition.

[0088] EEE 10. The method of any of EEEs 6 to 9, further comprising running an output video-audio pair generated by the denoising U-Net through a pretrained lip-synchronization model to generate the video and audio segments for the synchronized video-audio pair.

[0089] EEE 11. The method of EEE 10, further comprising applying a static image to the pretrained lip-synchronization model, wherein the video segment is generated based on the static image.

[0090] EEE 12. The method of any of EEEs 6 to 11, further comprising applying the speech text to a pretrained text-to-speech model to generate an input audio segment for the denoising U-Net, wherein the audio segment for the synchronized video-audio pair is generated based on the input audio segment.

[0091] EEE 13. The method of any of EEEs 6 to 12, further comprising predicting a sequence of video frames from a single input image, wherein the denoising U-Net is configured to use the sequence of video frames as an image batch condition for generating the video segment.

[0092] EEE 14. The method of any of EEEs 6 to 13, wherein the denoising U-net is configured to receive a random distribution as a video input or as an audio input.

[0093] EEE 15. The method of any of EEEs 5 to 14, wherein the denoising processing includes using a consistency model to reduce a number of denoising steps.

[0094] EEE 16. The method of any of EEEs 5 to 15, wherein the denoising processing is guided using at least one of a text classifier and an image classifier.

[0095] EEE 17. The method of any of EEEs 5 to 16, wherein the denoising processing is guided using a joint video-audio classifier.

[0096] EEE 18. The method of any of EEEs 5 to 17, further comprising using a vision- language model to provide conditional input to the denoising processing, the vision-language model configured to broaden a pool of acceptable video captions provided via the one or more text inputs.

[0097] EEE 19. The method of any of EEEs 5 to 19, further comprising using a super- resolution model to increase resolution of the video segment.

[0098] EEE 20. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising any of EEEs 1 to 19.

[0099] EEE 21. An apparatus for generating a synchronized video-audio pair, the apparatus comprising: at least one processor; and at least one memory including program code; wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: apply one or more text inputs to a generative model, the one or more text inputs including a speech text, the generative model including a neural network; with the generative model, convert the speech text into an audio segment for the synchronized video-audio pair; and with the generative model, generate a video segment for the synchronized video audio pair, the video segment including a talking head having lip movements corresponding to the speech text and in synchronization with the audio segment.

[0100] With regard to the processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes could be practiced with the described steps performed in an order other than the order described herein. It further should be understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating certain embodiments, and should in no way be construed so as to limit the claims.

[0101] Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent upon reading the above description. The scope should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the application is capable of modification and variation.

[0102] All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those knowledgeable in the technologies described herein unless an explicit indication to the contrary is made herein. In particular, use of the singular articles such as “a,” “the,” “said,” etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0103] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in fewer than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

[0104] While this disclosure includes references to illustrative embodiments, this specification is not intended to be construed in a limiting sense. Various modifications of the described embodiments, as well as other embodiments within the scope of the disclosure, which are apparent to persons skilled in the art to which the disclosure pertains are deemed to lie within the principle and scope of the disclosure, e.g., as expressed in the following claims.

[0105] Some embodiments may be implemented as circuit-based processes, including possible implementation on a single integrated circuit.

[0106] Some embodiments can be embodied in the form of methods and apparatuses for practicing those methods. Some embodiments can also be embodied in the form of program code recorded in tangible media, such as magnetic recording media, optical recording media, solid state memory, floppy diskettes, CD-ROMs, hard drives, or any other non-transitory machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the patented invention(s). Some embodiments can also be embodied in the form of program code, for example, stored in a non-transitory machine-readable storage medium including being loaded into and / or executed by a machine, wherein, when the program code is loaded into and executed by a machine, such as a computer or a processor, the machine becomes an apparatus for practicing the patented invention(s). When implemented on a general-purpose processor, theprogram code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits.

[0107] Unless explicitly stated otherwise, each numerical value and range should be interpreted as being approximate as if the word “about” or “approximately” preceded the value or range.

[0108] The use of figure numbers and / or figure reference labels in the claims is intended to identify one or more possible embodiments of the claimed subject matter in order to facilitate the interpretation of the claims. Such use is not to be construed as necessarily limiting the scope of those claims to the embodiments shown in the corresponding figures.

[0109] Although the elements in the following method claims, if any, are recited in a particular sequence with corresponding labeling, unless the claim recitations otherwise imply a particular sequence for implementing some or all of those elements, those elements are not necessarily intended to be limited to being implemented in that particular sequence.

[0110] Reference herein to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the disclosure. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments necessarily mutually exclusive of other embodiments. The same applies to the term “implementation.”

[0111] Unless otherwise specified herein, the use of the ordinal adjectives “first,” “second,” “third,” etc., to refer to an object of a plurality of like objects merely indicates that different instances of such like objects are being referred to, and is not intended to imply that the like objects so referred-to have to be in a corresponding order or sequence, either temporally, spatially, in ranking, or in any other manner.

[0112] Unless otherwise specified herein, in addition to its plain meaning, the conjunction “if” may also or alternatively be construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” which construal may depend on the corresponding specific context. For example, the phrase “if it is determined” or “if [a stated condition] is detected” may be construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event].”

[0113] Also, for purposes of this description, the terms “couple,” “coupling,” “coupled,” “connect,” “connecting,” or “connected” refer to any manner known in the art or later developed in which energy is allowed to be transferred between two or more elements, and the interposition of one or more additional elements is contemplated, although not required. Conversely, the terms “directly coupled,” “directly connected,” etc., imply the absence of such additional elements.

[0114] As used herein in reference to an element and a standard, the term compatible means that the element communicates with other elements in a manner wholly or partially specified by the standard and would be recognized by other elements as sufficiently capable of communicating with the other elements in the manner specified by the standard. The compatible element does not need to operate internally in a manner specified by the standard.

[0115] The functions of the various elements shown in the figures, including any functional blocks labeled as “processors” and / or “controllers,” may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing software, random access memory (RAM), and nonvolatile storage. Other hardware, conventional and / or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.

[0116] As used in this application, the terms “circuit,” “circuitry” may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry); (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions); and (c) hardwarecircuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.” This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0117] It should be appreciated by those of ordinary skill in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0118] “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” in this specification is intended to introduce some example embodiments, with additional embodiments being described in “DETAILED DESCRIPTION” and / or in reference to one or more drawings. “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

Claims

CLAIMS What is claimed is:

1. A method of generating a synchronized video-audio pair, the method comprising: applying one or more text inputs to a generative model, the one or more text inputs including a speech text, the generative model including a neural network; with the generative model, converting the speech text into an audio segment for the synchronized video-audio pair; and with the generative model, generating a video segment for the synchronized video-audio pair, the video segment including a talking head having lip movements corresponding to the speech text and in synchronization with the audio segment, wherein converting the speech text into an audio segment for the synchronized video-audio pair comprises generating speech audio based only on the one or more text inputs.

2. The method of claim 1, wherein the one or more text inputs further include one or more texts selected from the group consisting of: a video caption; a description of the talking head; a condition for the video segment or for the audio segment; and a description of the video segment.

3. The method of claim 1 or 2, further comprising: receiving a first text input of the one or more text inputs via a communication link; modifying the first text input based on a user input; and applying the modified text input to the generative model, wherein the synchronized video-audio pair is generated based on the modified text input.

4. The method of any of claims 1 to 3, further comprising applying a static image to the generative model, wherein the video segment is generated based on the static image.

5. The method of any of claims 1 to 4, wherein the generative model includes denoising processing based on a diffusion model.

6. The method of claim 5, wherein the denoising processing is implemented with a denoising U-Net.

7. The method of claim 6, wherein the denoising U-Net includes a plurality of double- branch U-Net blocks corresponding to different scales of a latent feature, each of the double- branch U-Net blocks having a respective video branch and a respective audio branch.

8. The method of claim 7, wherein the respective video branch and the respective audio branch are connected via a respective cross attention block configured to facilitate relative alignment between the video segment and the audio segment.

9. The method of claim 8, wherein the denoising U-Net includes one or more additional upsampling blocks and / or one or more additional downsampling located between the double- branch U-Net blocks; and wherein the denoising U-Net includes one or more skip connections between the double- branch U-Net blocks that share a same respective feature size.

10. The method of any of claims 6 to 9, wherein the one or more text inputs include a video caption; and wherein the denoising U-Net is configured to receive the video caption as a condition.

11. The method of any of claims 6 to 10, further comprising running an output video-audio pair generated by the denoising U-Net through a pretrained lip-synchronization model to generate the video and audio segments for the synchronized video-audio pair.

12. The method of claim 11, further comprising applying a static image to the pre-trained lip- synchronization model, wherein the video segment is generated based on the static image.

13. The method of any of claims 6 to 12, further comprising applying the speech text to a pretrained text-to-speech model to generate the speech audio, the speech audio being an input audio segment for the denoising U-Net, wherein the audio segment for the synchronized video- audio pair is generated based on the input audio segment.

14. The method of any of claims 6 to 13, further comprising predicting a sequence of video frames from a single input image, wherein the denoising U-Net is configured to use the sequence of video frames as an image batch condition for generating the video segment.

15. The method of any of claims 6 to 14, wherein the denoising U-Net is configured to receive a random distribution as a video input or as an audio input.

16. The method of any of claims 5 to 15, wherein the denoising processing includes using a consistency model to reduce a total number of denoising steps.

17. The method of any of claims 5 to 16, wherein the denoising processing is guided using at least one of a text classifier and an image classifier.

18. The method of any of claims 5 to 17, wherein the denoising processing is guided using a joint video-audio classifier.

19. The method of any of claims 5 to 18, further comprising using a vision-language model to provide conditional input to the denoising processing, the vision-language model being configured to broaden a pool of acceptable video captions provided via the one or more text inputs.

20. The method of any of claims 5 to 19, further comprising using a super-resolution model to increase the resolution of the video segment.

21. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising any of claims 1 to 20.

22. An apparatus for generating a synchronized video-audio pair, the apparatus comprising: at least one processor; and at least one memory including program code; wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: apply one or more text inputs to a generative model, the one or more text inputs including a speech text, the generative model including a neural network;with the generative model, convert the speech text into an audio segment for the synchronized video-audio pair, wherein converting the speech text into an audio segment for the synchronized video-audio pair comprises generating speech audio based only on the one or more text inputs; and with the generative model, generate a video segment for the synchronized video-audio pair, the video segment including a talking head having lip movements corresponding to the speech text and in synchronization with the audio segment.

Images