Video-controlled hybrid-style video generation through generative ai

Abstract

A method comprising receiving one or more of a guiding video, a set of image examples, a set of region masks associated with image examples, generating hybrid inputs based on the guiding video, the set of image examples, and the set of region masks, processing the hybrid inputs using a multi-style diffusion model to generate a set of multi-style latent features, generating multi-style output frames based on the multi-style latent features and the hybrid inputs, obtaining camera view parameters from the guiding video, computing, using a neural radiance fields (NeRF) model, synthesized video frames based on the camera view parameters and the multi-style output frames, generating a hybrid-style video from the synthesized video frames, where the hybrid-style video maintains an original style of the guiding video and applies visual styles from the image examples to the one or more target objects, and displaying the hybrid-style video on a display device.

Description

Atty. Docket No.4502-84101 (6000710PCT02) Video-Controlled Hybrid-Style Video Generation through Generative AI CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No.63 / 660,809 filed on June 17, 2024, which is hereby incorporated by reference. TECHNICAL FIELD

[0002] The present disclosure describes techniques for generating video content. More specifically, this disclosure describes techniques for generating video content using generative artificial intelligence (GAI). BACKGROUND

[0003] Generating video content through GAI remains one of the most complex challenges in the field of content creation. In particular, controlled video generation that produces desired visual content presents significant technical difficulties. Existing text-to-video generation engines, such as Sora or CustomVideo, face the innate difficulty of using text prompts to accurately describe the desired generation target, such as the scene composition, scene motion, visual details, and related elements. SUMMARY

[0004] The disclosed embodiments provide techniques for generating a hybrid-style free-view video with controls over both the semantic scene content and desired styles of various objects. In an embodiment, the semantic scene content is controlled using a guiding video input, and the desired styles of specific objects are controlled by image examples of the objects. In an embodiment, a neural radiance fields (NeRF) model, built upon hybrid-style video frames, generates the hybrid-style free- view video.

[0005] A first aspect relates to a method implemented by a computing device, comprising: receiving one or more of a guiding video, a set of image examples, a set of region masks associated with image examples, generating hybrid inputs based on the guiding video, the set of image examples, and the set of region masks; processing the hybrid inputs using a multi-style diffusion model to generate a set of multi-style latent features; generating multi-style output frames based onAtty. Docket No.4502-84101 (6000710PCT02) the multi-style latent features and the hybrid inputs; obtaining camera view parameters from the guiding video, computing, using a NeRF model, synthesized video frames based on the camera view parameters and the multi-style output frames; generating a hybrid-style video from the synthesized video frames, where the hybrid-style video maintains an original style of the guiding video and applies visual styles from the image examples to the one or more target objects; and displaying the hybrid-style video on a display device.

[0006] Optionally, in any of the preceding aspects, another implementation of the aspect provides that each image frame comprises one or more of a grayscale image, a color image, or a color image with associated depth information.

[0007] Optionally, in any of the preceding aspects, another implementation of the aspect provides that each image example represents a desired visual style of a corresponding target object.

[0008] Optionally, in any of the preceding aspects, another implementation of the aspect provides that each region mask identifies spatial regions corresponding to one or more target objects in the image frames of the guiding video.

[0009] Optionally, in any of the preceding aspects, another implementation of the aspect provides that generating the hybrid inputs comprises selecting, for each target object and each image frame, a matching image example from the set of image examples based on viewpoint similarity; and generating the hybrid inputs for each target object and each image frame by masking a target object region in the image frame and concatenating the masked frame with the selected matching image example.

[0010] Optionally, in any of the preceding aspects, another implementation of the aspect provides that generating the set of multi-style latent features comprises applying a diffusion-based model to embed the hybrid inputs into a class-conditioned multi-style latent space that encodes style variations across the one or more target objects.

[0011] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the restoration model reconstructs the multi-style output frames to match resolution and channel structure of the guiding video.

[0012] Optionally, in any of the preceding aspects, another implementation of the aspect provides that obtaining the camera view parameters comprises estimating the camera view parameters of a three-dimensional scene from different camera views using a structure-from-motion algorithm.Atty. Docket No.4502-84101 (6000710PCT02)

[0013] Optionally, in any of the preceding aspects, another implementation of the aspect provides receiving metadata associated with the guiding video, wherein the metadata includes camera views for each image frame of the guiding video; and obtaining the camera view parameters from the metadata.

[0014] Optionally, in any of the preceding aspects, another implementation of the aspect provides that each synthesized video frame comprises photorealistic representations of the one or more target objects to corresponding image examples.

[0015] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the NeRF model comprises a volumetric ray tracing pipeline to map three- dimensional spatial coordinates and view directions to volumetric density and color values using a neural function.

[0016] Optionally, in any of the preceding aspects, another implementation of the aspect provides training the NeRF model to synthesize video frames that are style-consistent with the multi- style output frames for the one or more target objects.

[0017] Optionally, in any of the preceding aspects, another implementation of the aspect provides that training the NeRF model comprises initializing the NeRF model; and for each training iteration: receiving a set of training camera view parameters and a set of training multi- style latent features; generating a set of training synthesized frames based on the set of training camera view parameters and the corresponding set of training multi multi-style latent features; computing a loss function based on the training synthesized frames; calculating a gradient of the loss function; and updating model weights of the NeRF model based on the gradient of the training loss via backpropagation.

[0018] Optionally, in any of the preceding aspects, another implementation of the aspect provides performing a data update by modifying training guiding video frames with the training synthesized frames.

[0019] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the loss function comprises one or more of a mean square error (MSE) loss, an average loss, a perceptual loss, or an adversarial loss.

[0020] Optionally, in any of the preceding aspects, another implementation of the aspect provides updating a subset of the model weights during each training iteration.Atty. Docket No.4502-84101 (6000710PCT02)

[0021] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the multi-style diffusion model and the restoration model are trained based on a set of multi-style image pairs with associated masks.

[0022] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the multi-style diffusion model comprises a multi-style embedding model and / or a reverse diffusion model.

[0023] Optionally, in any of the preceding aspects, another implementation of the aspect provides training the multi-style diffusion model comprises: generating a training masked hybrid input based on a training guiding image, a training target image, and a training region mask; computing, using the multi-style embedding model, an embedded training joint latent feature based on the training masked hybrid input; generating a training multi-style latent feature based on the embedded training joint latent feature; processing, using the restoration model, the training multi- style latent feature and the training masked hybrid input to obtain a training multi-style output; and updating the multi-style embedding model based on the training multi-style output.

[0024] Optionally, in any of the preceding aspects, another implementation of the aspect provides generating the training multi-style latent feature using the reverse diffusion model based on the embedded training joint latent feature and a random noise; and updating the reverse diffusion model based on the training multi-style latent feature.

[0025] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the reverse diffusion model comprises a convolutional diffusion networks or a diffusion transformer.

[0026] A second aspect relates to a computing device, comprising: a memory configured to store instructions; and one or more processors coupled to the memory and configured to execute the instructions to cause the computing device to perform the method in any of the disclosed embodiments.

[0027] A third aspect relates to a computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computer- executable instructions when executed by one or more processor of a computing device, cause the computing device to perform the method in any of the disclosed embodiments.

[0028] A fourth aspect relates to a computing device, comprising: a storage means configured to store instructions; and one or more processing means coupled to the storage means andAtty. Docket No.4502-84101 (6000710PCT02) configured to execute the instructions to cause the computing device to perform the method in any of the disclosed embodiments.

[0029] For the purpose of clarity, any one of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create a new embodiment within the scope of the present disclosure.

[0030] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0032] FIG.1 is a schematic diagram of an overall workflow for generating a hybrid-style free- view video using a NeRF model according to an embodiment of the disclosure.

[0033] FIG.2 is a schematic diagram of a learning method for the NeRF Model according to an embodiment of the disclosure.

[0034] FIG. 3 is a schematic diagram of a learning method for a multi-style diffusion module and a restoration module according to an embodiment of the disclosure.

[0035] FIG.4 is a method implemented by a computing device according to an embodiment of the disclosure.

[0036] FIG.5 is a schematic diagram of a network apparatus according to an embodiment of the disclosure. DETAILED DESCRIPTION

[0037] It should be understood at the outset that although an illustrative implementation of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.Atty. Docket No.4502-84101 (6000710PCT02)

[0038] Disclosed herein are various systems and methods for generating hybrid-style videos based on a guiding video and image examples of desired styles of target objects in the generated video with associated masks. The guiding video serves as the basis to define the dynamic semantic development in the video, such as scene composition, scene content, scene movements, etc. For each target object in the guiding video with a desired presenting style in the generated video, the exemplar images with associated masks provides detailed visual description about the desired appearance of the object in the generated video. Different target objects can have different desired presenting styles, and the final generated video integrates these different styles into the guiding video while following the semantic and scene development defined by the guiding video.

[0039] Great success has been achieved for AI Generated Content (AIGC) by using a wide range of image generative models, including generative adversarial networks (GAN), diffusion models, and auto-regressive (AR) models. The goal is to enable fast and accessible high-quality content creation. Various methods have been developed to allow for efficient manipulation of the generated content using different types of prompt inputs, such as using text descriptions and / or spatial / spatiotemporal compositions like sketches or segmentation maps. However, it is innately difficult to use prompt inputs to accurately describe detailed aspects of the image content, such as size, shape, color, location of various objects, scene composition, etc. It is difficult to control the generated output to have any specific desired attributes or visual details.

[0040] Image-to-image transformation has been largely used for transferring image styles, such as the StyleGAN method. Given an input image, a style generator generates a synthesized output image with a desired style. The style generator is trained to encode the style information from a set of example images, and the style encoded latent feature is injected into the generation model to combine with the embedding feature of the input, the style-transferred output usually preserves the high-level semantic attributes of the input image. However, it is in general difficult to control the properties of generated images, which requires both disentangling features or properties in images and adding controls over these properties to the style generator. As a result, existing GAN-based image-to-image style transfer methods are applied to two scenarios: transferring the global style of the image, e.g., changing from photos to oil paintings; or transferring specific features of an object category with well-defined high-level attributes like human faces, e.g., change skin color, makeup, or hair style.Atty. Docket No.4502-84101 (6000710PCT02)

[0041] Recent works use pretrained text-to-image diffusion models like Imagen to create variations of images, or to manipulate specific image regions. Text-guided image-to-image transformation gives large flexibility to generate novel image content. However, same as general text-to-image generation, due to the lack of description power of using text to define visual content, it is innately difficult to control the generated output to have any specific desired visual details.

[0042] Video generation is one of the most difficult problems among all GAI-based content generation problems. Controlled video generation that produces desired visual content is even harder. Existing text-to-video generation engines like Sora or CustomVideo face the innate difficulty of using text prompts to accurately describe the desired generation target, such as the scene composition, scene motion, visual details and so on.

[0043] Instead of using text descriptions, using an input video as a guiding example to accurately illustrate and define scene composition, semantic content, scene motion, etc. is much more efficient. Also, using image examples to illustrate and define the details of objects in the generated video enables the system to control the generation results. Controlled video generation through GAI enables great flexibility in video content production, which is however not approached by other methods.

[0044] The disclosed techniques offer a novel framework for generating a hybrid-style free-view video with controls over both the semantic scene content and desired styles of various objects. In an embodiment, the semantic scene content is controlled using a guiding video input, and the desired styles of specific objects are controlled by image examples of the objects. In an embodiment, a NeRF model, built upon hybrid-style video frames, generates the hybrid-style free-view video.

[0045] The disclosure is related to generating video content using GAI. The disclosed approach introduces a system framework that takes as input a guiding video, one or multiple semantic masks indicating the target objects in one or multiple frames in the guiding video, and one or multiple exemplar object images corresponding to each semantic mask, and then generates as output a hybrid- style video where each masked object in the video is altered to follow the style of the corresponding exemplar object images, and the remaining part of the generated video maintains the original style as the guiding video. The proposed system provides the capability for controlled generation of video content where the scene composition and semantic content is faithful to the guiding video, while changing the object of interests in the video into different designed styles that are explicitly described by image examples.Atty. Docket No.4502-84101 (6000710PCT02)

[0046] The disclosed techniques have a wide range of applications across industries where controlled and high-fidelity video content generation is essential. For example, in the entertainment and media sector, the framework embodied by the disclosed techniques can be used for post- production editing in films and animation, enabling creators to restyle specific characters or objects without re-shooting or manually re-animating scenes. For example, in fashion and retail, the framework enables virtual try-on experiences by replacing clothing or accessories on models with styled alternatives from exemplar images. For example, in the gaming industry, the framework supports efficient asset customization and dynamic cutscene generation by applying distinct visual themes to characters or objects within pre-rendered videos. Additionally, the disclosed techniques can be employed in augmented and virtual reality environments for overlaying personalized content that blends seamlessly with the original scene. In marketing and social media, the framework offers a tool for generating engaging, personalized, and style-consistent content tailored to specific audiences or branding requirements. The framework’s ability to selectively stylize objects while maintaining scene integrity makes the framework a powerful solution for any application requiring precision-driven, creative manipulation of video content.

[0047] FIG. 1 is a schematic diagram of an overall workflow 100 (a.k.a., framework) for generating a hybrid-style free-view video using a NeRF model according to an embodiment of the disclosure. In an embodiment, the overall workflow 100 is implemented by or on a personal computer (PC), a smart phone, a smart tablet, or some other computing device used to play games or consume entertainment.

[0048] As shown in FIG. 1, the overall framework 100 (a.k.a., system) is given an inputcomprising (1) a guiding video represented by a sequence of T image frames ^^, … , ^^, (2) a set ofimage examples ^^^ , ⋯ ^ ^^^ , ⋯, ^^^ , ⋯ ^^^^ , and (3) a corresponding set of region masks^^^, ⋯ ^^^ , ⋯ , ^^^, ⋯target objects represented in the image examples andimage frames.

[0049] In an embodiment, each image frame ^^may be represented in various formats, including but not limited to: 1-channel (e.g., gray scale image), 3-channel (e.g., red green blue (RGB) color image), and / or 4-channel (e.g., red green blue–depth (RGBD) image with color and associateddepth), etc. In an embodiment, the guiding video ^^, … , ^^ that captures three-dimensional (3D)scene from different camera views, the corresponding camera views ^^, … , ^^ for the frames^^, … , ^^ are either provided directly (e.g., as metadata associated with the video) or can beAtty. Docket No.4502-84101 (6000710PCT02) computed in a computing view module 102, which employs 3D reconstruction techniques such as structure-from-motion to recover camera poses based on scene geometry and image correspondences.

[0050] In an embodiment, in the set of image examples, each ^^^ , ⋯ ^ ^^^ (^^ ≥ 1) illustrates thedesired presenting style of a target j-th object in the guiding video. In an embodiment, in the region masks, each ^^^ , ⋯ ^^^ comprises a set of masks marking a spatial region of the target j-th object in the T frames of the guiding video. The system 100 generates a new video in which the target j-th object is rendered with the appearance style consistent with the corresponding image examples ^^ , ⋯ ^^ . In an^ , ⋯ ^^ ^^^embodiment, the image examples ^^ ^^have the exemplary appearance of the looking from different views. In some alternative embodiments, the system may operatewith only a single image example per object, corresponding to a single viewpoint.

[0051] In an embodiment, the system 100 further comprises an alignment and masking module 104 configured to process the guiding video ^ ^ ^ ^^, … , ^^, the image examples ^^ , ⋯ ^^^ , ⋯, ^^ , ⋯ ^^^^ , and the region masks ^^, ⋯ ^^ ^ ^^ , ⋯ , ^^, ⋯ ^^ to produce a set of^^ ^ embodiment,target j-th object and each guiding frame ^^^image examples ^^ , ⋯ ^^ ^ , the alignment and masking module 104 identifies the best matching image example ^ ^ ^ , whose view point of the target j-th object is the most similar to the view point of the target j-th object in ^ ^ ^ . Once the best example is selected, a hybrid input ^ ^ is generated by concatenating two components: (1) a m^^ asked guiding frame ^^⊗ ^ ^ ^ , where the pixels in output of ^^^ ⊗ ^^ ^ corresponding to the marked region of the target j-th in ^ ^ are zeroed out toappearance, and (2) the matching image ^ ^ ^ , where ^ ^ ^ is the opposite operation. This combination ensures that each hybrid input capturescontext without the object, paired with a stylized reference for how the object should appear. This process is applied to every object across all frames, resulting in a comprehensive set of hybrid inputs ^^ ^ ^ ^^ , ⋯ , ^^ , ⋯ , ^^ , ⋯ , ^^ .

[0052] In anfurther comprises a multi-style diffusion module 106 configured to process the set of hybrid inputs ^^ ^ ^ ^^, ⋯ , ^^ , ⋯ , ^^ , ⋯ , ^^ to generate a set ofmulti-style latent features ^ , ⋯ , ^^ ^^, ⋯ , ^^, ⋯ , ^ .latent feature inAtty. Docket No.4502-84101 (6000710PCT02) learned class-guided multi-style joint embedding space for the object class of the target j-th object. This embedding space is trained to capture the relationship between object class, appearance style, and viewpoint, thereby encoding complex style variations across different object instances and camera perspectives.

[0053] In an embodiment, the system 100 further comprises a restoration module 108configured to generate multi-style outputs ^^^, ⋯ , ^^^ based on the multi-style latent features^^^, ⋯ , ^^^ , ⋯ , ^^^ , ⋯ , ^^^ and the hybrid inputs ^^^, ⋯ , ^^^ , ⋯ , ^^ ^^ , ⋯ , ^^ . Each output image ^^^^ dimensions, i.e., ^^have 1-channel (e.g., gray scale image), 3-channel (e.g., RGB color image), or 4-channel (e.g. RGBD image with color and associated depth), etc.

[0054] In an embodiment, the system 100 further comprises a NeRF generation module 110configured to compute a set of synthesized frames ^^^, ⋯ , ^^^ based on a NeRF model ^^(^, , ^),the camera views ^ , … , ^^, and the multi-style outputs ^^^, ⋯ , ^^^. In angeneration module 110 may use NeRF-based such as the NeRF model denotedas ^^(^, , ^), that combines a volumetric ray tracer with a neural mapping from spatial coordinatesto color and volumetric density.

[0055] To generate a new view (i.e., a synthesized frame), the NeRF generation module 110 performs volumetric ray tracing: for each pixel in a target output frame, a ray is cast from the virtual camera center through the image plane and into the 3D scene. Along this ray, a series of 3D samplepoints are evaluated using the neural mapping function ^^(^, , ^), which outputs the volumetricdensity values ^ and RGB color values c. The densities and colors values of many sampled 3D points are fed into the volumetric ray tracer to render the output image.

[0056] Learning the NeRF model

[0057] FIG. 2 is a schematic diagram of an overall workflow 200 (a.k.a., framework) of a learning method for the NeRF model according to an embodiment of the disclosure. In an embodiment, the detailed workflow 200 outlines the detailed steps involved in the embodiment ofthe learning process to derive the NeRF model ^^(^, , ^). In the diagram, the multi-style diffusionmodule is considered predefined and remains fixed.the restoration module is also assumed to be available. In an embodiment, the restoration module can either remain fixed or befine-tuned jointly with the NeRF model ^^(^, , ^).Atty. Docket No.4502-84101 (6000710PCT02)

[0058] As shown in FIG. 2, the overall framework 200 (a.k.a., system) is given an inputcomprising (1) a training guiding video represented by a sequence of image frames ^!"^ , ⋯ , ^^!"#$, (2) a set of training image examples ^!",^^ , ⋯ ^ !",^^#$^ , ⋯, ^!",^^ , ⋯ ^!",^ ^#$ ^ , and (3) a set of training region masks ^!",^^ , ⋯ ^ !",^^#$ , ⋯ , ^!",^^ , ⋯ ^!",^ ^#$image examples.^!" ^^#$ and ^^, … , ^^ are images capturing the same 3D sceneusing the same or different cameras from different views, where the corresponding camera trainingviews ^!"^ , ⋯ , ^ !"^#$ for the frames ^!"^ , ⋯ , ^^!"#$are either provided directly (e.g., as metadata associated with the video) or can be computed in the computing view module 202, which employs 3D reconstruction techniques such as structure-from-motion. In an embodiment, the training guidingvideo ^!"^ , ⋯ , ^ !"^#$ is the same with the guiding video input ^^, … , ^^, and the training and the testinference are performed over the same video sequence. In an embodiment, when the training guidingvideo ^!"^ , ⋯ , ^ !"^#$ is the same as the guiding video ^^, … , ^^ , the camera training views^!" , ⋯ , !"^ ^^#$ is the same as the camera views ^^, … , ^^.

[0060] In an embodiment, in the set of!",^image examples ^^ , ⋯ ^ !",^^#$^ , ⋯, ^!",^^ , ⋯ ^!",^ ^#$ ^ in the training stage, each ^!",^!",^!^ , ⋯ ^ "^#$ ^ (^^ ≥ 1) illustratesatraining target j-th object in the training guiding video ^!"^ , ⋯ , ^^!"#$. In an embodiment, the training target j-th object is the same object in the captured 3D scene as the target j-th object in the guidingvideo ^ , … , ^ in FIG. 1. In an embod !",^ !",^^ ^ iment, the training image examples ^^ , ⋯ ^^#$^ , ⋯,^!",^^ , ⋯ ^!",^ ^#$ ^are the same as image examples ^^ ^ ^^ , ⋯ ^^^ , ⋯, ^^ , ⋯ ^^^^ . In an embodiment, in the set of training region masks ^!",^, ⋯ ^ !",^, ⋯ , ^!",^, ⋯ ^!",^ , each!",^, ⋯ ^!",^^^#$ ^ ^#$^^ ^#$comprises a set of masks marking a spatialobject in the Ttraining guidingvideo ^!"^ , ⋯ , ^ !"^#$ . In an embodiment, when the training guiding video ^!"^ , ⋯ , ^^!"#$is the same asthe guiding video ^ !",^ !",^ !",^!",^ ^, … , ^^ , the training region masks ^^ , ⋯ ^^#$ , ⋯ , ^^ , ⋯ ^^#$is thesame as the region masks ^^ ^ ^ ^^, ⋯ ^^ , ⋯ , ^^, ⋯ ^^.

[0061] In ancomprises an alignment and masking module204 configured to process the training guiding video ^!"^ , ⋯ , ^^!"#$, the training image examples ^!",^^ , ⋯ ^ !",^^#$^ , ⋯, ^!",^!",^!",^ !",^ !",^!",^ ^, ⋯ ^^#$ , and the training region masks ^, ⋯ ^ #$ , ⋯ , ^ , ⋯ ^ #$^ ^ ^ ^ ^, toAtty. Docket No.4502-84101 (6000710PCT02) compute a set of training hybrid inputs ^!",^^ , ⋯ , ^ !",^^#$ , ⋯ , ^!",^^ , ⋯ , ^!",^ ^#$. In an embodiment, for the target j-th object and each training mask ^!",^ ^ , among ^ aining image examples ^!the ^!"tr",^^ , ⋯ ^^#$ ^ , the alignment and masking module 204the best matching training image example ^!",^ ^ , whose view point of the target j-th object is the most similar to the view point of the target j-th object in ^!",^ ^ . Once the best example is selected, a training hybrid input ^!",^!",^^is generated by concatenating a masked guiding frame ^^ ⊗ ^!",^ ^ and the matching training image example ^!",^ ^ , where ^!",^ ^ is the opposite operation. Inthe pixels in the output of ^!",^^ ⊗to the marked region of the target j-th object in ^!",^ ^ are set to zeros. In an embodiment, the system 200 further comprises a multi-style diffusion module 206 configured to process the set of the training hybrid inputs ^!",^^ , ⋯ , ^ !",^^#$ , ⋯ , ^!",^^ , ⋯ , ^!",^ ^#$to generate a set of training multi-style latent features ^!",^^ ,^!",^ ^ is an embedding latent feature in the learnedfor the object class of the target j-th object.

[0063] In an embodiment, the system 200 further comprises a restoration module 208 configuredto generate a training multi-style outputs ^^!"^ , ⋯ , ^^^!"#$based on the training multi-style latent features ^!",^^ , ⋯ , ^ !",^^#$ , ⋯ , ^!",^!",^ ^, ⋯ , ^^#$and the training hybrid inputs output image ^^!"corresponds to the original

[0064] In an embodiment, the system 200 further comprises a NeRF generation module 210configured to compute a set of training synthesized frames ^^!"^ , ⋯ , ^^^!"#$based on a current NeRFmodel ^^^ (^, , ^) at the i-th training iteration, the training camera views ^!"^ , ⋯ , ^!^"#$, and thetraining multi-style outputs ^^!"^ , ⋯ , ^^^!"#$.

[0065] In an embodiment, the system 200 further comprises a compute loss module 212configured to compute a loss function &(^!"^ , ⋯ , ^ !"^#$ , ^^!"^ , ⋯ , ^^^!"#$) . For example, for each pair consisting of a training input frame ^^!"and its corresponding training synthesized frame ^^^!"aAtty. Docket No.4502-84101 (6000710PCT02) distortion loss &(^!"^ , ^^^!") (e.g., MSE loss) can be calculated. The overall loss across all the training frames may be then computed as average loss as: &'^!" !"^ , ⋯ , ^^#$ , ^^!"^ , ⋯ , ^^ !"^#$ ( ) ^^#$∑^#$!"^+^ &(^^ , ^^^!") .

[0066] In an or generative adversarialnetworks (GANs)- the gradient of the lossfunction &'^!"^ , ⋯ , ^ !"^#$ , ^^!"^ , ⋯ , ^^^!"#$( can be calculated and backpropagated through a the model weights of the NeRF model ^^^ (^, , ^) .Optionally, the backpropagation module 214 can also update the neural network model weights of the restoration module 208, which is indicated with a dotted outline in FIG. 2.

[0067] In an embodiment, the system 200 further comprises a data update module 216configured to add the training synthesized frames ^^!"^ , ⋯ , ^^^!"#$to the training guiding video^!"^ , ⋯ , ^^!"#$. For example, each synthesized frame replace the original input frame^^!"to serve as the new training guiding video frame or be added to the dataset as an additional training guiding video frame. Following this, the system 200 advances to the next training iteration i+1.

[0068] In an embodiment, the model update and data update can be performed together or at different frequencies. For example, the system 200 can perform one model update and / or data update using the total loss of all the ,!"frames, the loss of each training pair &(^!"^ , ^^^!"), or the total loss of a batch of training pairs. Also, the weight parameters of the NeRF model (and optionally weight parameters of the restoration module 208) can be updated all together or in parts. This disclosure does not put any restrictions on the methods for the model update and data update.

[0069] Learning the multi-style diffusion module & the restoration module

[0070] FIG.3 is a schematic diagram of an overall workflow 300 (a.k.a., framework or system) of a learning method used to obtain a multi-style diffusion module 302 and a restoration module 304 according to an embodiment of the disclosure. In an embodiment, multi-style diffusion module 302configured to either generate a set of multi-style latent features ^^ ^ ^ ^^, ⋯ , ^^ , ⋯ , ^^ , ⋯ , ^^ based on thehybrid inputs ^^ ^ ^ ^^, ⋯ , ^^ , ⋯ , ^^ , ⋯ , ^^ during the inferencetraining⋯ , ^ !",^^#$ , ⋯ , ^!",^^ , ⋯ , ^!",^ ^#$based on the training hybrid inputs ^!",^, ⋯ , ^ !",^^ , ⋯ , ^!",^^ ,,(as shown in FIG. 2). Inmay comprise two main processing modules, a multi-Atty. Docket No.4502-84101 (6000710PCT02) style embedding module 306 and a reverse diffusion module 308. The reverse diffusion module 308 is optional and can be omitted in some implementations, and the corresponding processing flow is marked by dotted lines in FIG. 3.

[0071] In an embodiment, the restoration module 304 configured to either generate the multi-style output ^^ , ⋯ , ^^ based on the multi-style latent featu ^ ^ ^ ^^ ^ res ^^, ⋯ , ^^ , ⋯ , ^^, ⋯ , ^^ and thehybrid inputs ^^, ⋯ , ^^^ , ⋯ , ^^^ , ⋯ , ^^^ during the inferenceaining ^!"the tr ^ ⋯ , ^^^!"#$based on the training multi-style latent^!",^ !^ , ⋯ , ^^#$ ⋯ , ^ ⋯ , ^ ",^^#$and the training hybrid inputs ^^ , ⋯ , ^ !",^^#$ , ⋯ , ^!",^^ , ⋯ , ^!",^ ^#$during the training stage (as shown in FIG. 2).

[0072] In an embodiment, the multi-style diffusion module 302 and the restoration module 304 are trained using a large dataset of multi-style image pairs with associated region masks. In an embodiment, during training, the system 300 receives a training guiding image input ^!", a training target image ^!", and a training region mask ^!". In an embodiment, the masked regions indicated by the region mask ^!"of the training target image ^!"and the training guiding image input ^!"are the same object with different appearance styles. In an embodiment, ^!"and ^!"have the samenumber of channels as the guiding video frames ^^, … , ^^, which the multi-style diffusion module302 will ultimately process. The masked objects in ^!"and ^!"usually represent the same object (i.e., the same geometry and structure layout), differing only in styles attributes such as color, texture, or rendering style (e.g., oil painting versus photorealistic).

[0073] In an embodiment, the system 300 further comprises a masking module 310 configured to generate a training masked hybrid input ^!"based on the training guiding image input ^!", the training target Image ^!", and the training Region Mask ^!". In an embodiment, the training maskedhybrid input ^!" is a concatenated tensor of a masked guiding input ^!" ⊗ ^!" and a masked targetinput ^!" ⊗ ^!", where pixels corresponding to the marked region indicated by ^!" in ^!" and ^!"are set to zeros. In an embodiment, the multi-style embedding module 306 is configured to compute an embedded training joint latent ^̂!"using ^!", which is a latent feature representation in a multi- style embedding space for the object with the style of ^!"and the style of ^!". In an embodiment, when the reverse diffusion module 308 is used, it generates a training multi-style latent feature ^!"based on the training joint latent ^̂!"and a random noise sample n (e.g., a gaussian random noise), using a reverse diffusion method. In an embodiment, when the reverse diffusion module 308 is notAtty. Docket No.4502-84101 (6000710PCT02) used, the embedded training joint latent ^̂!"is directly used as the training multi-style latent feature ^!". After that, the restoration module 304 computes a training multi-style output ^^!"based on training multi-style latent feature ^!"and the training masked hybrid input ^!".

[0074] The multi-style embedding module 306 can be implemented using various neural network architectures, including convolutional neural networks (CNNs) or vision transformers. Similarly, the reverse diffusion module 308 can be realized using architectures such as convolutional diffusion networks or diffusion transformers. This disclosure does not impose any constraints on the specific network architectures used in these modules.

[0075] In an embodiment, the system 300 further comprises a compute multi-style loss module312 configured to generate a multi-style loss &(^!" , ^!" , ^!" , ^^!") using the training guiding imageinput ^!", the training target image ^!", the training region mask ^!", and the training multi-style output ^^!". The gradient of the multi-style loss is then calculated and backpropagated through the network using a backpropagation module 314 to update the weight parameters of the multi-stylediffusion module 302 and the restoration module 304. The multi-style loss &(^!" , ^!" , ^!" , ^^!")can be defined in various ways. In an embodiment, it is formulated as a weighted combination of multiple loss components, including: (1) the pixel-level distortion between the masked target input^!" ⊗ ^!" and the masked multi-style output ^^!" ⊗ ^!" , & !".^ / 0^(^ ⊗ ^!" , ^^!" ⊗ ^!"), such asthe L1 or L2-norm between ^!" ⊗ ^!" and ^^!" ⊗ ^!" aiming at enforcing the target objectindicated by the masked region in the generated ^^!"to match the style of ^!"; (2) the pixel-leveldistortion between the unmasked guiding input ^!" ⊗ ^!" and the unmasked multi-style output^^!" ⊗ ^!", & (^!" ⊗ ^!" , ^^!" ⊗ ^!"), such as the L1 o ^!" !".^ / 0^ r L2-norm between ^ ⊗ ^ and^^!" ⊗ ^!" aiming at enforcing the remaining region in the generated ^^!" other than the markedtarget object to preserve the style of the training guiding input of ^!". Additional loss components may be incorporated to enhance performance. For instance, a GAN loss can be included to encourage realism in the generated output ^^!", promoting more natural and visually plausible results.

[0076] In an embodiment, the updates of the multi-style diffusion module 302 and the restoration module 304 can be performed either jointly or at different update frequencies. Furthermore, the weight parameters of these modules can be updated in their entirety or partially (i.e., layer-wise or submodule-wise). This disclosure does not impose any constraints on the strategies used for updating the model parameters.Atty. Docket No.4502-84101 (6000710PCT02)

[0077] In an embodiment, the training dataset used to learn the multi-style diffusion module 302 and the restoration module 304 consists of multi-style image pairs with corresponding masks across various object categories. These categories include the target object classes to which the trained model will be applied during inference. For each object category, the training data includes multiple pairs of images exhibiting different style combinations, which encompass the guiding and target styles that the model will encounter during inference. A distinct set of weight parameters may be trained for the multi-style diffusion module 302 for each individual object category or for a group of categories. Similarly, separate sets of parameters may be learned for the restoration module 304 on a per-category or grouped-category basis.

[0078] Inference with the multi-style diffusion module & the restoration module

[0079] In an embodiment, during the inference stage, the multi-style diffusion module 302 andthe restoration module 304 generate the multi-style outputs ^^^, ⋯ , ^^^ based on the hybrid inputs^^^, ⋯ , ^^^ , ⋯ , ^^^ , ⋯ , ^^^ . Each output frame ^^^ has multiple objects, each rendered in a desiredby the image examples ^^^ , ⋯ ^ ^^^ , ⋯, ^^^ , ⋯ ^^^. Furthermore, the j-th target object in ^ corresponding to ^ ^ follows the^^^. In an embodiment,desired styles are iteratively merged into the guiding video input ^^, … , ^^.

[0080] Furthermore, let ^^^1^ ^ denote a previous restored version of the multi-style output ^^^after j-1 style iteration steps. In the j-th iteration, the system uses ^^^1^ ^ to replace the original guiding video frame ^ , and co ^ ^ mpute the corresponding hybrid input ^^targeting at the j-th object. This hybrid input is formed by concatenating the masked replaced guiding frame ^^^1^^ ⊗ ^^ ^ and the matching image example ^ ^ whose viewpoint of the target j-th object is theto viewpoint of the target j-th object in ^ ^ ^ among the ^^image examples ^^^ , ⋯ ^^ ^^. The multi-style embedding module 306 then computes an embedded joint latent ^ ^ ^ ^̂ from ^^. After that, when the reverse diffusion module 308 is used, it generates the multi-style latent Feature ^ ^ ^ based on the joint latent ^̂!"and a random noise n (e.g., a gaussian random noise). When the reverse diffusion module 308 is not used, the embedded joint latent ^̂ ^ is used directly as the multi-style latent fea ^ ^ ture ^^. Finally, the restoration module 304 computes the restored multi-style output ^^^^ based on the style latent feature ^ ^ ^ ^ and the masked hybrid input ^^.Atty. Docket No.4502-84101 (6000710PCT02)

[0081] FIG.4 is a method 400 implemented by a computing device according to an embodiment of the disclosure. In an embodiment, the computing device is a computer, a smart phone, a smart tablet, or other device configured to play games or display video content. In an embodiment, the method 400 is implemented during gaming or when video content is being consumed by a user.

[0082] In block 402, the computing device receives one or more of a guiding video, a set of image examples, a set of region masks associated with the image examples. In an embodiment, the guiding video comprises a sequence of image frames. In an embodiment, each image frame comprises one or more target objects.

[0083] In an embodiment, each image frame comprises one or more of a grayscale image, a color image, or a color image with associated depth information. In an embodiment, each image example represents a desired visual style of a corresponding target object. In an embodiment, each region mask identifies spatial regions corresponding to one or more target objects in the image frames of the guiding video.

[0084] In block 404, the computing device generates hybrid inputs based on the guiding video, the set of image examples, and the set of region masks. In an embodiment, the computing device generates the hybrid inputs by selecting, for each target object and each image frame, a matching image example from the set of image examples based on viewpoint similarity; and generating the hybrid inputs for each target object and each image frame by masking a target object region in the image frame and concatenating the masked frame with the selected matching image example.

[0085] In block 406, the computing device processes the hybrid inputs using a multi-style diffusion model to generate a set of multi-style latent features. In an embodiment, the computing device generates the set of multi-style latent features by applying a diffusion-based model to embed the hybrid inputs into a class-conditioned multi-style latent space that encodes style variations across the one or more target objects.

[0086] In block 408, the computing device generates, using a restoration model, multi-style output frames based on the multi-style latent features and the hybrid inputs. In an embodiment, the restoration model reconstructs the multi-style output frames to match resolution and channel structure of the guiding video.

[0087] In block 410, the computing device obtains camera view parameters from the guiding video. In an embodiment, the computing device obtains the camera view parameters by estimating the camera view parameters of a three-dimensional scene from different camera views using aAtty. Docket No.4502-84101 (6000710PCT02) structure-from-motion algorithm. In an embodiment, the computing device receives metadata associated with the guiding video, wherein the metadata includes camera views for each image frame of the guiding video; and obtains the camera view parameters from the metadata.

[0088] In block 412, the computing device computes, using a NeRF model, synthesized video frames based on the camera view parameters and the multi-style output frames. In an embodiment, each synthesized video frame comprises photorealistic representations of the one or more target objects to corresponding image examples. In an embodiment, the NeRF model comprises a volumetric ray tracing pipeline to map three-dimensional spatial coordinates and view directions to volumetric density and color values using a neural function.

[0089] In block 414, the computing device generates a hybrid-style video from the synthesized video frames, wherein the hybrid-style video maintains an original style of the guiding video and applies visual styles from the image examples to the one or more target objects. In an embodiment, the hybrid-style video is displayed on a display, screen, or monitor of the computing device for the benefit and enjoyment of the user.

[0090] FIG. 5 is a schematic diagram of a computing device 500 (e.g., a personal computer, smart phone, smart tablet, handheld gaming device, etc.) according to an embodiment of the disclosure. The computing device 500 is suitable for implementing the disclosed embodiments as described herein. The computing device 500 comprises ingress ports / ingress means 510 (a.k.a., upstream ports) and receiver units (Rx) / receiving means 520 for receiving data; a processor, logic unit, or central processing unit (CPU) / processing means 530 to process the data; transmitter units (Tx) / transmitting means 540 and egress ports / egress means 550 (a.k.a., downstream ports) for transmitting the data; and a memory / memory means 560 for storing the data. The computing device 500 may also comprise optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress ports / ingress means 510, the receiver units / receiving means 520, the transmitter units / transmitting means 540, and the egress ports / egress means 550 for egress or ingress of optical or electrical signals.

[0091] The processor / processing means 530 is implemented by hardware and software. The processor / processing means 530 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor / processing means 530 is in communication with the ingress ports / ingress means 510, receiver units / receiving means 520,Atty. Docket No.4502-84101 (6000710PCT02) transmitter units / transmitting means 540, egress ports / egress means 550, and memory / memory means 560. The processor / processing means 530 comprises a hybrid-style video generation module 570. The hybrid-style video generation module 570 is able to implement the methods disclosed herein. The inclusion of the hybrid-style video generation module 570 therefore provides a substantial improvement to the functionality of the computing device 500 and effects a transformation of the computing device 500 to a different state. Alternatively, the hybrid-style video generation 570 is implemented as instructions stored in the memory / memory means 560 and executed by the processor / processing means 530.

[0092] The computing device 500 may also include input and / or output (I / O) devices or I / O means 580 for communicating data to and from a user. The I / O devices or I / O means 580 may include output devices such as a display for displaying video data, speakers for outputting audio data, etc. The I / O devices or I / O means 580 may also include input devices, such as a keyboard, mouse, trackball, etc., and / or corresponding interfaces for interacting with such output devices. The memory / memory means 560 comprises one or more disks, tape drives, and solid-state drives and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. The memory / memory means 560 may be volatile and / or non-volatile and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0093] Embodiments of the present disclosure provide at least the following technical advantages.

[0094] a. A novel system to generate a hybrid-style free-view video with controls over both the semantic scene content and desired styles of various objects. The semantic scene content is controlled by the guiding video input, and the desired styles of specific objects are controlled by image examples of the objects. A NeRF model built upon hybrid-style video frames generates the hybrid-style free-view video.

[0095] b. A novel function that transforms a guiding video input into a hybrid-style free-view video by using image examples of target styles of various objects and corresponding semantic object masks in the guiding video input. The masked objects of the guiding video frames are transferred to the desired styles of the corresponding image examples, and the remaining region retains the originalAtty. Docket No.4502-84101 (6000710PCT02) style of the guiding video, resulting in a hybrid-style output video. The NeRF model is built upon the hybrid-style output video to generate the free-view hybrid-style video.

[0096] While several embodiments have been provided in the present disclosure, it may be understood that the disclosed systems and methods might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.

[0097] In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims

Atty. Docket No.4502-84101 (6000710PCT02) CLAIMS What is claimed is:

1. A method implemented by a computing device, comprising: receiving one or more of a guiding video, a set of image examples, a set of region masks associated with the image examples, wherein the guiding video comprises a sequence of image frames, and wherein each image frame comprises one or more target objects; generating hybrid inputs based on the guiding video, the set of image examples, and the set of region masks; processing the hybrid inputs using a multi-style diffusion model to generate a set of multi- style latent features; generating, using a restoration model, multi-style output frames based on the multi-style latent features and the hybrid inputs; obtaining camera view parameters from the guiding video; computing, using a neural radiance fields (NeRF) model, synthesized video frames based on the camera view parameters and the multi-style output frames; generating a hybrid-style video from the synthesized video frames, wherein the hybrid-style video maintains an original style of the guiding video and applies visual styles from the image examples to the one or more target objects; and displaying the hybrid-style video on a display device.

2. The method of claim 1, wherein each image frame comprises one or more of a grayscale image, a color image, or a color image with associated depth information.

3. The method of any of claims 1-2, wherein each image example represents a desired visual style of a corresponding target object.

4. The method of any of claims 1-3, wherein each region mask identifies spatial regions corresponding to one or more target objects in the image frames of the guiding video.Atty. Docket No.4502-84101 (6000710PCT02) 5. The method of any of claims 1-4, wherein generating the hybrid inputs comprises: selecting, for each target object and each image frame, a matching image example from the set of image examples based on viewpoint similarity; and generating the hybrid inputs for each target object and each image frame by masking a target object region in the image frame and concatenating the masked frame with the selected matching image example.

6. The method of any of claims 1-5, wherein generating the set of multi-style latent features comprises applying a diffusion-based model to embed the hybrid inputs into a class-conditioned multi-style latent space that encodes style variations across the one or more target objects.

7. The method of any of claims 1-6, wherein the restoration model reconstructs the multi-style output frames to match resolution and channel structure of the guiding video.

8. The method of any of claims 1-7, wherein obtaining the camera view parameters comprises estimating the camera view parameters of a three-dimensional scene from different camera views using a structure-from-motion algorithm.

9. The method of any of claims 1-7, further comprising: receiving metadata associated with the guiding video, wherein the metadata includes camera views for each image frame of the guiding video; and obtaining the camera view parameters from the metadata.

10. The method of any of claims 1-9, wherein each synthesized video frame comprises photorealistic representations of the one or more target objects to corresponding image examples.

11. The method of any of claims 1-10, wherein the NeRF model comprises a volumetric ray tracing pipeline to map three-dimensional spatial coordinates and view directions to volumetric density and color values using a neural function.Atty. Docket No.4502-84101 (6000710PCT02) 12. The method of any of claims 1-11, further comprising training the NeRF model to synthesize video frames that are style-consistent with the multi-style output frames for the one or more target objects.

13. The method of any of claims 1-12, wherein training the NeRF model comprises: initializing the NeRF model; and for each training iteration: receiving a set of training camera view parameters and a set of training multi-style latent features; generating a set of training synthesized frames based on the set of training camera view parameters and the corresponding set of training multi-style latent features; computing a loss function based on the training synthesized frames; calculating a gradient of the loss function; and updating model weights of the NeRF model based on the gradient of the training loss via backpropagation.

14. The method of any of claims 1-13, further comprising performing a data update by modifying training guiding video frames with the training synthesized frames.

15. The method of any of claims 1-14, wherein the loss function comprises one or more of a mean square error (MSE) loss, an average loss, a perceptual loss, or an adversarial loss.

16. The method of any of claims 1-15, further comprising updating a subset of the model weights during each training iteration.

17. The method of any of claims 1-16, wherein the multi-style diffusion model and the restoration model are trained based on a set of multi-style image pairs with associated masks.

18. The method of any of claims 1-17, wherein the multi-style diffusion model comprises a multi-style embedding model and / or a reverse diffusion model.Atty. Docket No.4502-84101 (6000710PCT02) 19. The method of any of claims 1-18, wherein training the multi-style diffusion model comprises: generating a training masked hybrid input based on a training guiding image, a training target image, and a training region mask; and computing, using the multi-style embedding model, an embedded training joint latent feature based on the training masked hybrid input; generating a training multi-style latent feature based on the embedded training joint latent feature; processing, using the restoration model, the training multi-style latent feature and the training masked hybrid input to obtain a training multi-style output; and updating the multi-style embedding model based on the training multi-style output.

20. The method of any of claims 1-19, further comprising: generating the training multi-style latent feature using the reverse diffusion model based on the embedded training joint latent feature and a random noise; and updating the reverse diffusion model based on the training multi-style latent feature.

21. The method of any of claims 1-20, wherein the multi-style embedding model comprises a convolutional neural network or a vision transformer.

22. The method of any of claims 1-21, wherein the reverse diffusion model comprises a convolutional diffusion network or a diffusion transformer.

23. A computing device, comprising: a memory configured to store instructions; and one or more processors coupled to the memory and configured to execute the instructions to cause the computing device to perform a method according to any of claims 1-22.

24. A computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computer-executable instructions whenAtty. Docket No.4502-84101 (6000710PCT02) executed by one or more processor of a computing device, cause the computing device to perform a method according to any of claims 1-22.

25. A computing device, comprising: a storage means configured to store instructions; and one or more processing means coupled to the storage means and configured to execute the instructions to cause the computing device to perform a method according to any of claims 1-22.

Images