Video compositing within a messaging system

Abstract

Aspects of the disclosure relate to a system comprising a computer-readable storage medium storing a program for video synthesis and a method for video synthesis. The program and method provide for accessing a primary generative adversarial network (GAN) comprising a pre-trained image generator, a motion generator comprising a plurality of neural networks, and a video discriminator; generating an updated GAN based on the primary GAN by performing operations comprising identifying input data for the updated GAN, the input data comprising an initial latent code and a motion domain dataset, training the motion generator based on the input data, and adjusting weights of the plurality of neural networks of the primary GAN based on an output of the video discriminator; and generating a synthetic video based on the primary GAN and the input data.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 198,151, filed September 30, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to image and video processing, including video compositing within messaging systems. Background Technology

[0004] Image and video compositing is a related field aimed at generating content from noise. Focused areas include image compositing methods that produce image-based models capable of achieving wider variation and improved resolution in image content and rendering. Attached Figure Description

[0005] In accompanying drawings that are not necessarily drawn to scale, similar reference numerals may describe similar parts in different views. To facilitate identification of any particular element or action being discussed, one or more of the highest-order digits in the reference numerals indicate the drawing number in which the element was first introduced. Some non-limiting examples are shown in the accompanying drawings:

[0006] Figure 1 It is a diagrammatic representation of a networked environment in which the content of this disclosure can be deployed, based on some examples.

[0007] Figure 2 This is a diagram of a generative adversarial network architecture based on some examples.

[0008] Figure 3 The flowchart illustrates a video synthesis technique for generating videos using a pre-trained image generator and motion generator, based on some examples.

[0009] Figure 4 A flowchart of an image discrimination technique based on some examples is shown.

[0010] Figure 5 A flowchart of a feature extractor, including a contrastive image discriminator, is shown based on some examples.

[0011] Figure 6 Example output sequences generated from cross-domain videos are shown.

[0012] Figure 7A and Figure 7B Another set of example output sequences for video synthesis based on the examples described in this article is shown.

[0013] Figure 8It is a graphical representation of a machine in the form of a computer system, based on some examples, within which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein.

[0014] Figure 9 It is a block diagram showing a software architecture in which examples can be implemented. Detailed Implementation

[0015] The following description includes systems, methods, techniques, instruction sequences, and computer program products for implementing illustrative embodiments of the present disclosure. In the following description, numerous specific details are set forth for illustrative purposes to provide an understanding of various embodiments. However, it will be apparent to those skilled in the art that embodiments can be practiced without these specific details. Generally, well-known examples of instructions, protocols, structures, and techniques are not necessarily shown in detail.

[0016] Image and video synthesis is a related field aimed at generating content from noise. Advances have focused on improving image synthesis methods to produce image-based models capable of achieving significant changes in image content, high resolution, and high-quality rendering. Image synthesis models are able to render images that are typically indistinguishable (or nearly indistinguishable) from real images.

[0017] However, advancements in video synthesis have yielded relatively limited improvements. Larger model sizes and increased statistical complexity mean that current video synthesis methods produce relatively low-resolution videos while requiring longer training times and more computational resources. This is particularly relevant on low-resource computers, such as mobile devices with limited memory and processing power. For example, generating a video with a target resolution of 256×256 pixels using contemporary image generators could require a substantial computational budget, resulting in training costs in the thousands of dollars. Furthermore, there are specific hardware requirements for such tasks.

[0018] Synthetic video has two main, but not necessarily exclusive, desired properties: (i) high quality (e.g., resolution) of each individual frame, and (ii) temporal consistency of the entire frame sequence (e.g., depicting the same subject or content with reasonable motion). Previous efforts have attempted to achieve both of these goals using a single frame, making such methods computationally demanding when high resolution is desired.

[0019] Therefore, a novel video synthesis technique may be needed to render high-resolution, visually consistent videos with low training time and fewer computational resources.

[0020] As discussed, some video rendering techniques are computationally complex and may not be suitable for implementation on low-resource computers (e.g., mobile devices with limited memory and processing power). The video synthesis techniques disclosed in this paper involve a multi-part framework that leverages current image synthesis methods with novel motion generators to render high-resolution, time-consistent videos with high training efficiency and therefore with significantly reduced computational resources compared to current video synthesis techniques. The techniques disclosed herein can be used, for example, in messaging systems. Messaging systems typically allow users to exchange content items (e.g., messages, images, and / or videos) with each other in message threads.

[0021] The example video synthesis technique uses a generative adversarial network (“GAN”) that includes a video generator and a video discriminator. In one or more implementations, the video generator uses a combination of a pre-trained image generator and a motion generator to create “fake videos” (e.g., synthetic videos). The pre-trained image generator is trained before the GAN is trained on a training set of real videos. The motion generator is trained during the training mode of the GAN. The GAN is first trained by feeding initial latent codes selected from a distribution (e.g., a normal distribution) to the pre-trained image generator and the motion generator. The motion generator also takes motion trajectory vectors that influence (e.g., randomize) the motion codes generated by the motion generator. The output of the motion generator is processed and fed to the pre-trained image generator, which in turn outputs vectors of all frames in the sequence (e.g., the synthetic video). The synthetic video vectors are then fed to the video discriminator to evaluate the synthetic video vectors against the training dataset and to provide feedback to the generator to adjust the neural network variables. The video discriminator restricts the generated motion patterns to resemble the motion patterns of the training video. In the example, the process is run until the discriminator can no longer distinguish between real and synthetic videos in the dataset; that is, the discriminator determines that the synthetic video is real.

[0022] In one example, the video synthesis technique includes an image discriminator to force the content of frames to be temporally continuous. In this example, a motion generator is implemented via two recurrent neural networks that operate on initial content codes to obtain a motion representation. This method models motion as a residual between consecutive latent codes, which are then passed to the image generator for individual frame generation. Such a residual representation also facilitates the detangling of motion and content. In another example, the motion generator is trained using a selected image discriminator with a contrast loss to force the content to be temporally continuous.

[0023] The disclosed implementations are characterized by several key advantages over conventional video synthesis pipelines. First, these implementations allow for the generation of high-quality videos at high resolution (e.g., 1024×1024). Second, the disclosed implementations are computationally more efficient when searching for latent trajectories in the image generator, requiring training time that is orders of magnitude less than previous video-based work. Third, the image generator can be trained on separate high-quality image datasets as it is updated or otherwise improved. Due to the untangled representation of motion and content, the system also enables the learning of motion from video datasets and its application to image datasets, even when the two datasets belong to different domains. When a domain (e.g., dogs) contains many high-quality images but not corresponding high-quality videos, the system provides the ability to synthesize high-quality videos using the image generator. In this way, realistic videos of objects whose movement was not observed during training can be generated (e.g., generating realistic pet facial videos using motion extracted from images of people speaking). This novel video generation task is referred to herein as cross-domain video synthesis.

[0024] Networked computing environment

[0025] Figure 1 This is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of client devices 102, each hosting multiple applications including messaging client 104 and other applications 106. Each messaging client 104 is communicatively coupled via a network 112 (e.g., the Internet) to other instances of the messaging client 104 (e.g., hosted on corresponding other client devices 102), a messaging server system 108, and a third-party server 110. The messaging client 104 can also communicate with the locally hosted applications 106 using an application programming interface (API).

[0026] The messaging client 104 can communicate and exchange data with other messaging clients 104 and messaging server system 108 via network 112. The data exchanged between messaging clients 104 and between messaging clients 104 and messaging server system 108 includes functions (e.g., commands to activate functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0027] The messaging server system 108 provides server-side functionality to a specific messaging client 104 via network 112. While some functions of the messaging system 100 are described herein as being performed by either the messaging client 104 or the messaging server system 108, the location of certain functions within the messaging client 104 or the messaging server system 108 can be a design choice. For example, it might be technically preferred that certain technologies and functions are initially deployed within the messaging server system 108, but later migrated to the messaging client 104 of the client device 102 with sufficient processing power.

[0028] The messaging server system 108 supports various services and operations provided to the messaging client 104. Such operations include sending data to and receiving data from the messaging client 104, and processing data generated by the messaging client 104. As an example, this data may include message content, client device information, geolocation information, media enhancements and coverage, message content persistence conditions, social network information, and live event information. Data exchange within the messaging system 100 is activated and controlled through functions available via the user interface (UI) of the messaging client 104.

[0029] Specifically, turning to message transceiver server system 108, application programming interface (API) server 116 is coupled to application server 114 and provides a programming interface to application server 114. Application server 114 is communicatively coupled to database server 120, which facilitates access to database 126, which stores data associated with messages processed by application server 114. Similarly, web server 128 is coupled to application server 114 and provides a web-based interface to application server 114. To this end, web server 128 processes incoming network requests via Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0030] Application Programming Interface (API) server 116 receives and sends message data (e.g., commands and message payloads) between client device 102 and application server 114. Specifically, API server 116 provides a set of interfaces (e.g., routines and protocols) that message sending and receiving client 104 can invoke or query to activate the functionality of application server 114. Application Programming Interface (API) server 116 exposes various functions supported by application server 114, including: account registration; login functionality; sending messages from one messaging client 104 to another messaging client 104 via application server 114; sending media files (e.g., images or videos) from messaging client 104 to messaging server 118 and providing possible access for another messaging client 104; setting up media data collections (e.g., stories); retrieving the friend list of the user of client device 102; retrieving such collections; retrieving messages and content; adding and deleting entities (e.g., friends) in an entity graph (e.g., a social graph); locating friends in the social graph; and opening application events (e.g., related to messaging client 104).

[0031] Application server 114 hosts multiple server applications and subsystems, including, for example, messaging server 118, image processing server 122, social networking server 124, and video compositing system 130. Messaging server 118 implements multiple messaging technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of messaging client 104. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to messaging client 104. Given the hardware requirements for such processing, additional processor- and memory-intensive data processing can also be performed on the server side by messaging server 118.

[0032] Application server 114 also includes image processing server 122, which is dedicated to performing various image processing operations, typically relative to the images or videos within the payload of messages sent from message transceiver server 114 or received at message transceiver server 118.

[0033] Social networking server 124 supports various social networking functions and services and makes these functions and services available to messaging server 118. To this end, social networking server 124 maintains and accesses an entity graph within database 126. Examples of functions and services supported by social networking server 124 include identifying other users in messaging system 100 who have a relationship with a specific user or who are "following" that specific user, as well as identifying a specific user's interests and other entities.

[0034] Returning to messaging client 104, the features and functionality of external resources (e.g., application 106 or applet) are available to the user via the interface of messaging client 104. In this context, "external" refers to the fact that application 106 or applet is outside of messaging client 104. External resources are typically provided by third parties, but may also be provided by the creator or provider of messaging client 104. Messaging client 104 receives user selections regarding options to launch or access the features of such external resources. External resources may be application 106 installed on client device 102 (e.g., a "native app"), or a smaller version (e.g., an "app") of an application hosted on client device 102 or located away from client device 102 (e.g., on a third-party server 110). A smaller version of an application includes a subset of the features and functionality of the application (e.g., a full, native version of the application) and is implemented using markup language documentation. In one example, a smaller version of the application (e.g., an "app") is a web-based markup language version of the application and is embedded in messaging client 104. In addition to using markup language documents (e.g., .*ml files), applets can incorporate scripting languages ​​(e.g., .*js files or .json files) and stylesheets (e.g., .*ss files).

[0035] In response to a user selection of an option to launch or access an external resource, messaging client 104 determines whether the selected external resource is a web-based external resource or a locally installed application 106. In some cases, application 106, locally installed on client device 102, can be launched independently of messaging client 104 and separately from it, for example, by selecting the icon corresponding to application 106 on the home screen of client device 102. A smaller version of such an application can be launched or accessed via messaging client 104, and in some examples, no part of the smaller application can be accessed (or only a limited part can) outside of messaging client 104. A smaller application can be launched by messaging client 104, for example, by receiving and processing markup language documents associated with the smaller application from a third-party server 110.

[0036] In response to determining that the external resource is a locally installed application 106, the messaging client 104 instructs the client device 102 to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the messaging client 104 communicates with a third-party server 110 (e.g.) to obtain a markup language document corresponding to the selected external resource. The messaging client 104 then processes the obtained markup language document to render the web-based external resource within the user interface of the messaging client 104.

[0037] The messaging client 104 can notify users of client device 102 or other users (e.g., "friends") associated with such users of one or more external resources. For example, the messaging client 104 can provide participants in a conversation (e.g., a chat session) within the messaging client 104 with notifications related to the current or recent use of external resources by one or more members of a group of users. One or more users can be invited to join an active external resource or to activate an external resource that was recently used but is currently inactive (within the group of friends). External resources can provide participants in the conversation, each using the corresponding messaging client 104, with the ability to share items, states, conditions, or locations within the external resource to the chat session with one or more members of a group of users. Shared items can be interactive chat cards that chat members can use to interact, for example, to activate the corresponding external resource, view specific information within the external resource, or take chat members to a specific location or state within the external resource. Within a given external resource, response messages can be sent to users on the messaging client 104. External resources can selectively include different media items in the response based on the current context of the external resource.

[0038] The messaging client 104 can present a list of available external resources (e.g., application 106 or applet) to the user to launch or access a given external resource. This list can be presented in a context-sensitive menu. For example, the icons representing different applications 106 (or applets) can change based on how the user launches the menu (e.g., from a conversational interface or a non-conversational interface).

[0039] In one example, video synthesis system 130 uses a generative adversarial network to generate images or image sequences (e.g., videos). In another example, video synthesis system 130 generates images or image sequences (e.g., videos) using video synthesis techniques. In some examples, video synthesis system 130 receives conditional data as input to parameters controlling the generated images or image sequences. Parts of video synthesis system 130 may reside on messaging client 104, while other parts of video synthesis system 130 may reside on application server 114. In some examples, the image or video generator neural network resides on messaging client 104, while the image or video discriminator neural network resides on application server 114.

[0040] Generative Adversarial Network Architecture

[0041] Figure 2 This is an illustration of a representative generative adversarial network architecture 200 based on some examples. In some examples, the video synthesis system 130 includes a GAN. As mentioned above, a generative adversarial network is a machine learning framework in which two neural networks, a discriminator network and a generator network, compete against each other (e.g., in a zero-sum arrangement). Given a training dataset, a GAN trains a model to learn to generate new data with the same characteristics (e.g., style, resolution) as the training set. That is, a GAN is a type of generative model that generates realistic synthetic data with the same quality as the underlying training data. For example, given a set of images of real human faces, a GAN will generate entirely new artificial (e.g., fake) faces through appropriate training.

[0042] Typically, generator networks attempt to create fake data that resembles real data (e.g., fake videos that look like real videos). In this example, the generator network takes random vectors as input and outputs real images or sequences of images (e.g., videos). On the other hand, discriminator networks attempt to correctly determine whether a given image or video was generated by the generator (e.g., fake or artificial) or comes from a dataset (e.g., real).

[0043] The generative adversarial network architecture 200 includes a GAN generator 202 and a GAN discriminator 208. Both the GAN generator 202 and the GAN discriminator 208 are neural networks. Each of the GAN generator 202 and the GAN discriminator 208 can be any type of artificial neural network (e.g., a long short-term memory neural network, a convolutional neural network). The output of the GAN generator 202 is fake data 206 (e.g., artificial), which is directly linked to the input of the GAN discriminator 208. Using backpropagation, the GAN discriminator 208 classifies information that the GAN generator 202 uses to update its weights.

[0044] The GAN discriminator 208 is a classifier that attempts to distinguish between real data 214 (e.g., from real images, real videos) and artificial data or fake data 206 (e.g., created by the GAN generator 202). The GAN discriminator 208 is trained using two data sources: real data 214 and fake data 206. For example, real data 214 may include real faces, while fake data 206 may include synthetic or artificial faces (e.g., faces entirely generated by a computer). Fake data 206 is data generated by the GAN generator 202. During training, the GAN discriminator 208 classifies real data 214 and fake data 206. The discriminator loss 210 takes into account that the GAN discriminator 208 misclassifies real data 214 as fake and fake data 206 as real. The GAN discriminator 208 uses the discriminator loss 210 to update its weights (e.g., the weights of the neural network layers) through backpropagation.

[0045] GAN generator 202 is a neural network that generates fake data 206 based on feedback from GAN discriminator 208. GAN generator 202 learns to enable GAN discriminator 208 to classify fake data 206 as real. GAN generator 202 takes random noise 204 as input and transforms the random noise 204 into meaningful output data. Generator loss 212 considers the fake data 206 generated by GAN generator 202 and classified as fake by GAN discriminator 208. GAN generator 202 updates its weights using generator loss 212 through backpropagation via GAN discriminator 208 and GAN generator 202.

[0046] Video compositing

[0047] Figure 3 A flowchart of an example video synthesis process 300 for generating video using a generative adversarial network (“GAN”) is shown. The video synthesis process 300 includes a generator 302 and a discriminator 304. The generator 302 includes a pre-trained image generator 306 (“GAN”). I ") and motion generator 308 ("G M The discriminator 304 includes a video discriminator 310 (“D”). V ”).

[0048] Generator 302 uses the initial input latent code 312 (“z1”) and motion trajectory vector 318 to synthesize each individual frame. The resulting video sequence or the generated video 320 It is given by the following formula: For each composite frame Where t = 1, 2, ..., n. On the other hand, a real video sequence or real video 322 (“v”) is defined as v = {x1, x2, ..., xn}.n And the training distribution is p v In the example, real video 322 is the training dataset used to train generator 302 and discriminator 304. The generated video 320 and real video 322 are fed into video discriminator 310.

[0049] The pre-trained image generator 306 receives information from the motion generator 308 to generate high-quality image frames and utilizes manageable computational resources to improve the training efficiency of the method. In the example, each of the pre-trained image generator 306 and the motion generator 308 uses an initial input latent code 312. In the example, the pre-trained image generator 306 is pre-trained, for example, before training the rest of the generator 302. The pre-trained image generator 306 can be pre-trained with a primary dataset, while the generator 302 is trained based on a secondary dataset. In the example, the pre-trained image generator 306 is trained based on a dataset of real images. In yet another example, the pre-trained image generator 306 is trained based on a content dataset. In the example, while the pre-trained image generator 306 is pre-trained, the process of training the generator 302 is actually the process of training the motion generator 308 with a motion dataset. In some other examples, the pre-trained image generator 306 is not pre-trained and is trained with a training dataset within the video synthesis process 300.

[0050] In the example, the pre-trained image generator 306 may follow an image generator architecture based on one or more of the following: training a generative adversarial network using limited data; analyzing and improving the image quality of a StyleGAN; providing discriminative enhancements for data-efficient GAN training; and / or image augmentation for GAN training. In another example, with the pre-trained image generator 306 fixed during generator training, video motion is detangled from image content, enabling video synthesis even when image content and video motion come from different domains.

[0051] The motion generator 308 uses the initial input latent code 312 (“z1”) to predict successive latent codes. In the example, z1∈Z, and Z={z1, z2, …, z n}, where n is the number of frames in the generated video 320. As used in this paper, the z vector is a vector containing random values ​​from a Gaussian (normal) distribution or some other distribution. In the example, from the normal distribution p z The input code z1 is sampled from p. In another example, StyleGAN2 is used to sample the input code z1 from p. z Sample z1, where p zIt is the distribution following the multilayer perceptron (“MLP”) because the underlying code within this distribution can be semantically untangled better than when using a normal distribution.

[0052] As an example, the motion generator 308 is implemented using two Long Short-Term Memory (“LSTM”) neural networks: an LSTM encoder 314 (“LSTMenc”) and an LSTM decoder 316 (“LSTMdec”). As will be noted, the motion generator 308 and the pre-trained image generator 306 can be implemented using other types of neural network architectures. The latent code z1 is fed into the LSTM encoder 314, which encodes z1 to obtain the initial hidden state h1. The output of the LSTM encoder 314 is then passed to the LSTM decoder 316, which recursively estimates n-1 consecutive states. The LSTM encoder 314 and the LSTM decoder 316 can be modeled by the following equation (“Equation 1”):

[0053] h1, c1 = LSTM enc (z1)

[0054] h t c t =LSTM dec (ε t , (h t-1 c t-1 ), t=2,3,…,n

[0055] Where h and c represent the hidden state and the cell state, respectively, and the motion trajectory vector is 318(“ε”). t ”) is a noise vector sampled from a normal distribution to model the motion diversity at timestamp t.

[0056] Note that the conventional method will use h t The motion code serves as the basis for generating motion for a specific frame, while the content code remains constant across all frames. However, such a design requires a recurrent network to estimate motion while maintaining consistent content from the latent vectors, which is difficult to learn in practice. Traditional GANs struggle to extract and control discrete features within the same image due to feature entanglement. GANs may not be able to distinguish fine discrete details in an image as well as humans, causing features to become "entangled" with each other. Entanglement limits the ability of GANs to generate images without entangled features (e.g., a combination of hair color and eye color, or a combination of hairstyle and skin color).

[0057] Therefore, the technique described in this paper involves motion untangling and uses a sequence of motion residuals to estimate motion trajectories. Specifically, the motion residuals are modeled as a linear combination of a set of interpretable directions in the latent space. In the example, to create the motion trajectory vector, first, principal component analysis (“PCA”) is performed on the latent vector m randomly sampled from Z to obtain the basis V. Then, as described in Equation (“Equation 2”), by using h t And V, from the previous frame z t-1 up to the current frame z t To estimate the direction of motion:

[0058] z t =z t-1 +λ·h i ·V, t=2,3,…,n

[0059] Wherein, the hidden state h i ∈[-1, 1], and λ controls the step size given by the residual. Combining Equations 1 and 2, we obtain G. M (z1) = {z1, z2, ..., z} n}, and the generated video 320 Given as

[0060] Temporarily return to the motion trajectory vector 318 (e.g., the noise vector ε) t As will be noted, the motion trajectory vector 318 allows for control over the diversity of motion. In some examples, the LSTM decoder 316 can ignore ε without further processing. t This leads to the collapse of the motion pattern. In other words, the motion generator 308(G) M It may be impossible to capture diverse motion patterns from training videos and generate different videos with similar motion patterns for different noise vectors from an initial latent code. However, the technique described in this paper, by including mutual information loss (“L…”),… m () to make the hidden vector h t With noise vector ε t These problems are solved by maximizing mutual information between them. Where sim(u, v) = u T v / ||u||||v|| represents the cosine similarity between vectors u and v. Therefore, L is defined by Equation 3. m :

[0061]

[0062] Here, H is a 2-layer MLP used as the mapping function.

[0063] During training, motion generator 308 uses data from video discriminator 310 (“D…”).V The information received is used to modify the network parameters. In the example, the video discriminator 310 is a multi-scale video discriminator configured to determine whether a video sequence or the generated video 320 is real or synthetic. In the example, the video discriminator 310 is based on the PatchGAN architecture. However, in the example, the described technique uses 3D convolutional layers in the video discriminator 310 because 3D convolutional layers can model temporal dynamics better than 2D convolutional layers. The video discriminator 310 divides the input video sequence into small 3D patches and classifies each patch as real or fake. The local responses of the input sequence are averaged to produce the final output. In addition, each frame in the input video sequence is conditioned on the first frame when the first frame falls into the distribution of the pre-trained image generator 306 for more stable training. To optimize the adversarial loss to learn G... M and D V Apply the following formula ("Formula 4"):

[0064]

[0065] Figure 4 A flowchart of an image discrimination process 400 according to some examples is shown. In one example, the image discrimination process 400 is used as part of the training process of a motion generator 308 during a video synthesis process 300. In another example, the image discrimination process 400 is used as part of training the motion generator 308 and a pre-trained image generator 306. The image discrimination process 400 includes a contrastive image discriminator 402 that takes frames of the generated video 320 as input (e.g., and It generates new frames or a sequence of frames that are coherent with the input frames in at least one aspect of motion and quality. It should be noted that in the example where the pre-trained image generator 306 is pre-trained, the image generator can be trained based on a dataset (e.g., a video dataset) of a first domain (e.g., images of animal faces) while learning motion generator parameters (e.g., trained based on motion generator parameters) using a dataset of a second domain (e.g., images of human facial expressions).

[0066] In some examples, Equation 4 alone may not provide explicit constraints on the generated image. With the first image The ability to achieve similar quality and content, first image It is sampled from the image space of image generator 302, and therefore has high fidelity. Image discrimination process 400 utilizes contrast image discriminator 402 (“D”). I to match and Both quality and content.

[0067] The image discriminator 402 compares the D... I and G M Adversarial training is used to improve perceptual quality. The image discriminator 402 receives... And will Used as a real sample and Used as fake samples. The loss function is described by the following equation (“Equation 5”):

[0068]

[0069] To achieve content similarity between frames within a video, a contrastive image discriminator 402 is used as a feature extractor. In the example, the contrastive image discriminator 402 is trained with a contrastive loss called InfoNCE, which is based on representation learning utilizing contrastive predictive encoding. It should be noted that the goal of this method is to place image pairs with the same content close together in the embedding space, while images containing different content should be placed far apart.

[0070] Now go to Figure 5 The flowchart illustrates a feature extractor 500, including a contrast image discriminator (similar to contrast image discriminator 402), based on some examples. In the examples, given a small batch of N generated videos 320... Randomly sample 502 ("t") frames from each video: Then, using Enhanced 504, for each frame Create two enhanced versions (506a and 506b), thus generating the following 2N samples: where, It is positive for 510 because they share the same content, and All are negative pairs 512, where i≠j.

[0071] In the example, feature extractor 500 includes a similar encoder 508 (“F”). The similar encoder 508 shares the same weights and architecture as the contrastive image discriminator 402. In another example, the similar encoder 508 shares the same weights and architecture as the contrastive image discriminator 402, but excludes layers from the contrastive image discriminator 402. In yet another example, the similar encoder 508 includes a 2-layer MLP as a projection head that produces a representation of the input image. The contrastive loss function L, representing the cross-entropy of the 2N augmentation computation, is described by the following equation (“Equation 6”). contr :

[0072]

[0073] Where sim(·,·) is the cosine similarity function defined in Equation 3. If j≠i, It equals 1, and τ is a temperature parameter. In the example, τ is determined empirically and set to 0.07.

[0074] In the example, feature extractor 500 may include a momentum decoder mechanism to maintain a storage volume to remove the oldest negative pair 512 and update new negative pairs 512. In some examples, enhancement 504 includes translation, color dithering, and cropping on the synthesized image. For example, cropping is used to enhance the image on enhanced frames 506a and 506b. By utilizing positive and negative pairs dynamically generated during training, the discriminator can effectively focus on the content of the input samples.

[0075] In yet another example, the selection of the positive pair 510 in Equation 6 is specifically designed for cross-domain video synthesis, since video containing arbitrary content from the image domain may not be available. Positive and negative pairs are more easily obtained when the image and video originate from the same domain. Furthermore, two frames can be randomly selected from a real video and augmented to create a positive pair sharing the same content, while a negative pair contains augmented images from different real videos.

[0076] In another example, besides the contrastive loss function L contr In addition, the feature extractor 500 may also include a feature matching loss L f By changing L1 regularization to cosine similarity, the feature matching loss L... f It is applied to the first frame and other frames generated.

[0077] In view of the above, the total loss function used to train the motion generator 308, the video discriminator 310, and the contrast image discriminator 402 is defined by the following equation (“Equation 7”):

[0078]

[0079] Where, λ m , λ contr and λ f It is a hyperparameter for balancing loss.

[0080] Figure 6 An example output sequence of cross-domain video generation is shown, based on the examples described herein. Figure 6 In the video synthesis process 300, a sequence of frames of the smiling woman is synthesized using a pair of separate training datasets (e.g., the FFHQ dataset and the VoxCeleb dataset). As mentioned above... Figure 3The video synthesis process 300 is capable of untangling motion with image content and transferring motion patterns from one domain to another. In this example, the video synthesis process 300 learns human facial motion from VoxCeleb and then uses image content from FFHQ to synthesize video, thereby generating a high-quality, temporally continuous video (e.g., 1024×1024 pixels). Figure 6 The composite frame shown (t = {2, 4, 6, 8, 10, 12, 14, 16}) illustrates a continuous motion trajectory.

[0081] Figure 7A and Figure 7B Another set of example output sequences for video synthesis based on the examples described in this article is shown. Similar to... Figure 6 The output sequence in the video synthesis process 300 uses separate datasets (e.g., AFHQ-Dog for image content and VoxCeleb for motion content) to synthesize a sequence of dog frames. For Figure 7A and Figure 7B Each initial frame t=2 is the same. However, each frame in each subsequent sequence (e.g., t={4,6,8,10,12,14,16}) is different from the corresponding frame in another sequence.

[0082] Machine architecture

[0083] Figure 8This is a schematic representation of machine 800, within which instructions 810 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 800 to perform any or more of the methods and techniques discussed herein. For example, instructions 810 can cause machine 800 to perform any or more of the methods described herein. Instructions 810 transform a general, unprogrammed machine 800 into a specific machine 800 programmed to perform the described and illustrated functions in the described manner. Machine 800 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a network deployment, machine 800 can operate as a server machine or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 800 may include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web devices, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 810 specifying actions to be taken by machine 800. Furthermore, although only a single machine 800 is shown, the term "machine" should also be considered to include a collection of machines that individually or jointly execute instructions 810 to perform any one or more of the methods discussed herein. For example, machine 800 may include client device 102 or any of a plurality of server devices forming part of message transceiver server system 108. In some examples, machine 800 may also include both client and server systems, wherein certain operations of a particular method or algorithm are performed on the server side and certain operations of a particular method or algorithm are performed on the client side.

[0084] Machine 800 may include a processor 804, a memory 806, and an input / output (I / O) unit 802 that can be configured to communicate with each other via a bus 840. In the example, processor 804 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processors 808 and 812 that execute instruction 810. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although Figure 8 Multiple processors 804 are shown, but machine 800 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0085] Memory 806 includes main memory 814, static memory 816, and memory cells 818, both of which are accessible by processor 804 via bus 840. Main memory 806, static memory 816, and memory cells 818 store instructions 810 that implement any one or more of the methods or functions described herein. Instructions 810 may also reside wholly or partially in main memory 814, in static memory 816, in machine-readable medium 820 within memory cells 818, within at least one processor in processor 804 (e.g., within the processor's cache memory), or any suitable combination thereof during execution by machine 800.

[0086] I / O component 802 may include a wide variety of components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O component 802 included in a particular machine will depend on the type of machine. For example, portable machines such as mobile phones may include touch input devices or other such input mechanisms, while headless server machines are unlikely to include such touch input devices. It should be understood that I / O component 802 may include... Figure 8 Many other components are not shown. In various examples, I / O component 802 may include user output component 826 and user input component 828. User output component 826 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. User input component 828 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens that provide the position and force of touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), etc.

[0087] In other examples, I / O component 802 may include biometric component 830, motion component 832, environmental component 834, or position component 836, as well as various other components. For example, biometric component 830 includes components for detecting expressions (e.g., hand gestures, facial expressions, vocal expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 832 includes accelerometer components (e.g., accelerometers), gravity sensor components, and rotation sensor components (e.g., gyroscopes).

[0088] The environmental component 834 includes, for example: one or more camera devices (with still image / photograph and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting the concentration of hazardous gases or measuring pollutants in the atmosphere for safety purposes), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0089] Regarding the camera device, client device 102 may have a camera device system including, for example, a front-facing camera on the front surface of client device 102 and a rear-facing camera on the rear surface of client device 102. The front-facing camera may be used, for example, to capture still images and videos (e.g., "selfies") of the user of client device 102, which can then be enhanced using the aforementioned enhancement data (e.g., filters). For example, the rear-facing camera may be used to capture still images and videos in a more conventional camera device mode, which are similarly enhanced using the enhancement data. In addition to the front and rear-facing cameras, client device 102 may also include a 360° camera for capturing 360° photos and videos.

[0090] Furthermore, the camera system of the client device 102 may include dual rear cameras (e.g., a main camera and a depth-sensing camera), or even include triple, quadruple, or quintuple rear camera configurations on the front and rear sides of the client device 102. For example, these multi-camera systems may include wide-angle cameras, ultra-wide-angle cameras, telephoto cameras, macro cameras, and depth sensors.

[0091] The position component 836 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure to obtain altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0092] A wide variety of technologies can be used to implement communication. I / O component 802 also includes communication component 838, which is operable to couple machine 800 to network 822 or device 824 via appropriate coupling or connection. For example, communication component 838 may include a network interface component or another suitable device to interface with network 822. In further examples, communication component 838 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, etc. Components (e.g.) (low power consumption) Components and other communication components that provide communication via other modes. Device 824 can be another machine or any peripheral device among various peripheral devices (e.g., a peripheral device coupled via USB).

[0093] Furthermore, the communication component 838 can detect identifiers or include components operable to detect identifiers. For example, the communication component 838 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, UltraCode, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying audio signals from tags). Additionally, various information can be derived via the communication component 838, such as location derived via Internet Protocol (IP) geolocation, etc. Location can be obtained through signal triangulation or by detecting NFC beacon signals that indicate a specific location.

[0094] Various memories (e.g., main memory 814, static memory 816, and the memory of processor 804) and storage units 818 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instruction 810) cause various operations to implement the disclosed examples when executed by processor 804.

[0095] Instructions 810 can be sent or received over network 822 via a network interface device (e.g., a network interface component included in communication component 838), using a transmission medium and any of several known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instructions 810 can be sent or received via a transmission medium coupled to device 824 (e.g., peer-to-peer coupling).

[0096] Software Architecture

[0097] Figure 9 This is a block diagram 900 illustrating a software architecture 904 that can be installed on any or more devices described herein. The software architecture 904 is supported by hardware such as a machine 902 including a processor 920, memory 926, and I / O components 938. In this example, the software architecture 904 can be conceptualized as a stack of layers, where each layer provides specific functionality. The software architecture 904 includes layers such as an operating system 912, libraries 910, a framework 908, and an application 906. Operationally, the application 906 activates API calls 950 via the software stack and receives messages 952 in response to API calls 950.

[0098] Operating system 912 manages hardware resources and provides public services. Operating system 912 includes, for example, a kernel 914, services 916, and drivers 922. The kernel 914 acts as an abstraction layer between the hardware layer and other software layers. For example, the kernel 914 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 916 can provide other public services to other software layers. Drivers 922 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 922 may include display drivers, camera drivers, etc. or Low-power drives, flash drives, serial communication drives (e.g., USB drives), Drivers, audio drivers, power management drivers, etc.

[0099] Library 910 provides common low-level infrastructure used by application 906. Library 910 may include system libraries 918 (e.g., the C standard library), which provide functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 910 may include API libraries 924, such as media libraries (e.g., libraries supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codecs, Joint Picture Experts Group (JPEG or JPG) or Portable Web Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering graphic content on a display in two-dimensional (2D) and three-dimensional (3D) formats), database libraries (e.g., SQLite, providing various relational database functions), web libraries (e.g., WebKit, providing web browsing capabilities), etc. Library 910 may also include various other libraries 928 to provide application 906 with many other APIs.

[0100] Framework 908 provides common high-level infrastructure for use by Application 906. For example, Framework 908 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. Framework 908 can provide a wide range of other APIs that can be used by Application 906, some of which may be specific to a particular operating system or platform.

[0101] In the example, application 906 may include a home application 936, a contacts application 930, a browser application 932, a book reader application 934, a location application 942, a media application 944, a messaging application 946, a game application 948, and a wide variety of other applications such as third-party application 940. Application 906 is a program that performs the functions defined in the program. One or more applications 906 can be created using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 940 (e.g., an entity other than the vendor of a particular platform using Android) TM or iOS TM Applications developed using a Software Development Kit (SDK) can be used on platforms such as iOS. TM ANDROID TM , Mobile software running on the phone's mobile operating system or other mobile operating systems. In this example, a third-party application 940 can activate API calls 950 provided by the operating system 912 to facilitate the functions described herein.

[0102] Glossary

[0103] "Carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions to be executed by a machine, including digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0104] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, laptop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0105] "Communication network" refers to one or more parts of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a POTS (Plain Old-Style Telephone Service) network, a cellular telephone network, a wireless network, etc. A network, other types of networks, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, the coupling can implement any data transmission technology of various types, such as Single Carrier Radio Transmission (1xRTT), Evolved Data Optimization (EVDO), General Packet Radio Service (GPRS), Enhanced Data Rate Evolution of GSM (EDGE), the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long-distance protocols, or other data transmission technologies.

[0106] A “component” refers to a device, physical entity, or logic having boundaries defined by functional or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularity for a particular processing or control function. Components can be combined with other components via their interfaces to perform machine processing. A component can be a packaged functional hardware unit designed for use with other components, and part of a program that typically performs a related function. Components can constitute software components (e.g., code implemented on a machine-readable medium) or hardware components. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various examples, one or more hardware components (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to operate to perform certain operations described herein. Hardware components can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuitry systems that are temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a specific part of a machine) uniquely tailored to perform the configured function and is no longer a general-purpose processor. It will be understood that the implementation of hardware components can be determined, for cost and time considerations, whether mechanically implemented in a dedicated and permanently configured circuitry system or in a temporarily configured (e.g., software-configured) circuitry system. Therefore, the phrase “hardware component” (or “hardware-implemented component”) should be understood to encompass tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain way or perform certain operations described herein. Consider the example of hardware components being temporarily configured (e.g., programmed), without requiring each of the hardware components to be configured or instantiated at any given time. For example, in cases where the hardware components include a general-purpose processor that is configured as a dedicated processor via software, the general-purpose processor can be configured as a different dedicated processor (e.g., including different hardware components) at different times. The software accordingly configures one or more specific processors to constitute a specific hardware component at one time and different hardware components at different times. Hardware components can provide information to and receive information from other hardware components. Therefore, the described hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from that memory structure. For example, a hardware component can perform an operation and store the output of that operation in a communication-coupled memory device. Another hardware component can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., collections of information). The various operations of the example methods described herein can be performed, at least in part, by one or more processors configured, either temporarily (e.g., by software) or permanently, to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented at least in part by processors, where one or more specific processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors 1004 or processor-implemented components. Furthermore, the one or more processors can also be configured to support the execution of related operations in a “cloud computing” environment or to operate as “Software as a Service” (SaaS). For example, at least some operations can be performed by a group of computers (as an example of machines including processors), where these operations are accessible via a network (e.g., the Internet) and via one or more suitable interfaces (e.g., APIs). The execution of some operations can be distributed among processors, not just residing within a single machine, but can be deployed across multiple machines. In some examples, the processor or processor-implemented component can reside in a single geographic location (e.g., in a home environment, office environment, or server cluster). In other examples, the processor or processor-implemented component can be distributed across multiple geographic locations.

[0107] "Computer-readable storage medium" refers to both machine-readable storage media and transmission media. Therefore, these terms encompass both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" refer to the same thing and may be used interchangeably in this disclosure.

[0108] A "brief message" is a message that is accessible for a limited period of time. Brief messages can be text, images, videos, etc. The access time for a brief message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting method, the message is transient.

[0109] "Machine storage medium" refers to one or more storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Therefore, this term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" refer to the same thing and are used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal medium."

[0110] "Non-transitory computer-readable storage medium" refers to a tangible medium capable of storing, encoding, or carrying instructions that can be executed by a machine.

[0111] "Signal medium" means any intangible medium capable of storing, encoding, or carrying instructions executable by a machine, and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

Claims

1. A video synthesis method, comprising: Access the main generative adversarial network (GAN), which includes a pre-trained image generator, a motion generator comprising multiple neural networks, and a video discriminator; An updated GAN is generated based on the primary GAN by performing operations, including identifying the input data of the updated GAN, the input data including initial latent codes and a motion domain dataset, and The motion generator is trained based on the input data; as well as Synthetic videos are generated based on the main GAN and the input data. The pre-trained image generator is trained using a first dataset from a first domain and a second dataset from a second domain to learn motion generator parameters. The first and second domains correspond to different types of captured objects, and the first and second datasets correspond to different types of data. The motion generator learns motion patterns from the second domain to synthesize time-consistent video frames in the first domain, thereby enabling the generation of realistic video in the first domain using motion extracted from the second domain.

2. The video synthesis method according to claim 1, wherein, The GANs that generate updates based on the main GANs also include: The weights of the multiple neural networks of the main GAN are adjusted based on the output of the video discriminator.

3. The video synthesis method according to claim 1, wherein, The motion domain dataset corresponds to motion trajectory vectors, which are used to synthesize each individual data frame corresponding to the generated synthetic video.

4. The video synthesis method according to claim 1, wherein, The pre-trained image generator is configured to receive the initial latent code and the output from the motion generator to generate the synthetic video.

5. The video synthesis method according to claim 1, wherein, The pre-trained image generator is pre-trained using a primary dataset, which includes at least one of a content dataset or real images.

6. The video synthesis method according to claim 5, wherein, The generators corresponding to the motion generator and the pre-trained image generator are trained using a secondary dataset that is different from the main dataset.

7. The video synthesis method according to claim 1, wherein, The motion generator is configured to receive the initial latent code to predict successive latent codes.

8. The video synthesis method according to claim 1, wherein, The motion generator is implemented using two long short-term memory neural networks.

9. The method according to claim 1, wherein, The first dataset is a video dataset, and The second dataset is an image dataset.

10. The method according to claim 9, wherein, The first domain corresponds to the captured object of the animal's face. Wherein, the second field corresponds to the object capturing human facial expressions, and The pre-trained image generator is trained based on the video dataset of the animal faces, while the motion generator parameters are learned using the image dataset of human facial expressions.

11. A system comprising: processor; as well as A memory for storing instructions that, when executed by the processor, configure the system to perform operations including: Access the main generative adversarial network (GAN), which includes a pre-trained image generator, a motion generator comprising multiple neural networks, and a video discriminator; An updated GAN is generated based on the main GAN by performing operations, the operations including The input data for identifying the updated GAN includes the initial latent code and motion domain dataset, as well as... The motion generator is trained based on the input data; and Synthetic videos are generated based on the main GAN and the input data. The pre-trained image generator is trained using a first dataset from a first domain and a second dataset from a second domain to learn motion generator parameters. The first and second domains correspond to different types of captured objects, and the first and second datasets correspond to different types of data. The motion generator learns motion patterns from the second domain to synthesize time-consistent video frames in the first domain, thereby enabling the generation of realistic video in the first domain using motion extracted from the second domain.

12. The system according to claim 11, wherein, The GANs that generate updates based on the main GANs also include: The weights of multiple neural networks in the main GAN are adjusted based on the output of the video discriminator.

13. The system according to claim 11, wherein, The motion domain dataset corresponds to motion trajectory vectors, which are used to synthesize each individual data frame corresponding to the generated synthetic video.

14. The system according to claim 11, wherein, The pre-trained image generator is configured to receive the initial latent code and the output from the motion generator to generate the synthetic video.

15. The system according to claim 11, wherein, The pre-trained image generator is pre-trained using a primary dataset, which includes at least one of a content dataset and real images.

16. The system according to claim 15, wherein, The generators corresponding to the motion generator and the pre-trained image generator are trained using a secondary dataset that is different from the primary dataset.

17. The system according to claim 11, wherein, The motion generator is configured to receive the initial latent code to predict successive latent codes.

18. The system according to claim 11, wherein, The motion generator is implemented using two long short-term memory neural networks.

19. A non-transitory computer-readable storage medium, the computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform an operation, the operation comprising: Access the main generative adversarial network (GAN), which includes a pre-trained image generator, a motion generator comprising multiple neural networks, and a video discriminator; An updated GAN is generated based on the main GAN by performing operations, the operations including: The input data for identifying the updated GAN includes the initial latent code and motion domain dataset, as well as... The motion generator is trained based on the input data; and a synthetic video is generated based on the main GAN and the input data. The pre-trained image generator is trained on a first dataset of a first domain and simultaneously learns motion generator parameters using a second dataset of a second domain. The first and second domains correspond to different types of captured objects, and the first and second datasets correspond to different types of data. The motion generator learns motion patterns from the second domain to synthesize time-consistent video frames in the first domain, thereby enabling the generation of realistic video in the first domain using motion extracted from the second domain.

20. The computer-readable storage medium according to claim 19, wherein, The GANs that generate updates based on the main GANs also include: The weights of the multiple neural networks of the main GAN are adjusted based on the output of the video discriminator.

Images