Video-guided free-view video generation with robust object control by gaussian editing

Abstract

A method comprises receiving a guiding video comprising image frames, camera parameters, camera views, and region masks identifying objects of interest across the image frames; generating a Gaussian splatting (GS) model representing a three-dimensional (3D) scene based on the image frames, the camera parameters, the camera views, and the region masks; receiving object-centric instructions specifying operations to be performed on the objects of interest or on novel objects not present in the guiding video; selecting, based on the object-centric instructions, an editing method; applying the editing method to the GS model to generate an edited GS model that reflects the operations on the objects of interest; rendering a synthesized video frame from a target camera view using the edited GS model; and generating a free-view video from the synthesized video frame, wherein the free-view video is spatio-temporally consistent with the guiding video and reflects the object-centric instructions.

Description

Video-Guided Free-View Video Generation with Robust Object Control by Gaussian EditingCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 711,537 filed on October 24, 2024, which is hereby incorporated by reference.TECHNICAL FIELD

[0002] The present disclosure describes techniques for generating video content. More specifically, this disclosure describes techniques for generating video content using generative artificial intelligence (GAI) and Gaussian splatting (GS).BACKGROUND

[0003] Generative Artificial Intelligence (Al) is a form of Al that learns patterns from massive datasets and then uses this knowledge to create new, original content like text, images, audio, video, and even code. Powered by deep learning models, generative Al works by encoding information into a mathematical representation and then decoding it to generate novel outputs when prompted by a user.SUMMARY

[0004] The disclosed embodiments provide techniques for generating a free-view video based on a guiding video and an object-centric instruction so that an object of interest in the generated video presents desired characteristics following the instruction. In an embodiment, the guiding video serves as the basis to define the dynamic semantic development in the video, such as scene composition, scene content, scene movements, etc. The object of interest can be an existing object in the guiding video or a novel object to be added into the generated video. The object-centric instruction describes the desired operation of the object of interest in the generated video, chosen from a list of predefined object-centric operations such as removing an existing object from the generated video, adding a novel object into the generated video, changing the appearance (e.g., color, texture, style) of an existing object, replacing an existing object by a different object, repositioning an existing object to another location, etc.

[0005] A first aspect relates to a method implemented by a computing device, comprising: receiving a guiding video comprising a sequence of image frames representing a three-dimensional (3D) scene from different viewpoints; receiving camera parameters and camera views associated with the guiding video; receiving region masks identifying objects of interest across the sequence of image frames; generating, using a semantic-aware three-dimensional (3D) Gaussian splatting (GS) model, a GS model representing the 3D scene based on the sequence of image frames, the camera parameters, the camera views, and the region masks; receiving object-centric instructions specifying operations to be performed on the objects of interest or on novel objects not present in the guiding video; selecting, based on the object-centric instructions, an editing method; applying the editing method to the GS model to generate an edited GS model that reflects the operations on the objects of interest; rendering a synthesized video frame from a target camera view using the edited GS model; generating a free-view video from the synthesized video frame, wherein the free- view video is spatio-temporally consistent with the guiding video and reflects the object-centric instructions; and displaying the free-view video on a display device.

[0006] Optionally, in any of the preceding aspects, another implementation of the aspect provides that each image frame comprises one or more of a grayscale image, a color image, or a color image with associated depth information.

[0007] Optionally, in any of the preceding aspects, another implementation of the aspect provides obtaining the camera views and the camera parameters using a structure-from-motion algorithm.

[0008] Optionally, in any of the preceding aspects, another implementation of the aspect provides obtaining the region masks using a semantic anything model (SAM).

[0009] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the target camera view is a novel view not present in the guiding video.

[0010] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the GS model comprises a set of 3D Gaussians representing the 3D scene, wherein each 3D Gaussian comprises a position vector, a scale vector, a rotation quaternion, an opacity value, a color vector, or a semantic mask vector for splatting-based rendering.

[0011] Optionally, in any of the preceding aspects, another implementation of the aspect provides rendering the synthesized image comprises projecting each 3D Gaussian into the target camera view and blending projected two-dimensional (2D) Gaussians using alpha-blending.

[0012] Optionally, in any of the preceding aspects, another implementation of the aspect provides receiving the guiding video, the region masks, and the camera views to generate an initial 3D GS model using a Gaussian splatting method; and assigning semantic labels to Gaussians in the initial 3D GS model using an inverse semantic projection model to generate the semantic-aware 3D GS model.

[0013] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the object-centric instructions comprise removing an existing object, adding a novel object, changing appearance of an object, replacing the existing object by a different object, or repositioning the existing object to another location.

[0014] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the object-centric instructions comprise direct editing instructions or inversion-based editing instructions.

[0015] Optionally, in any of the preceding aspects, another implementation of the aspect provides applying, for each direct editing instruction, a direct edit process to the GS model to generate an initial edited GS model; and performing a meta tuning process on the initial edited GS model to generate the edited GS model.

[0016] Optionally, in any of the preceding aspects, another implementation of the aspect provides applying, for each inversion-based editing instruction, an inversion-based edit process to generate an initial edited GS model; and performing a meta tuning process on the initial edited GS model to generate the edited GS model.

[0017] Optionally, in any of the preceding aspects, another implementation of the aspect provides performing the meta tuning process comprises rendering, for each camera view, a pre-edit 2D image using a previously edited GS model; rendering, for each camera view, a post-edit 2D image using a newly edited GS model; applying a 2D inpainting model to the post-edit 2D image to produce an inpainted image; computing a tuning loss between the inpainted image and the preedit 2D image based on object-specific region masks; updating the GS model using the tuning loss and Gaussian splat rendering losses; and iteratively performing the meta tuning process across the camera views to obtain the edited GS model.

[0018] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the tuning loss comprises a perceptual loss or an adversarial loss.

[0019] Optionally, in any of the preceding aspects, another implementation of the aspect provides applying the inversion-based edit process comprises computing, using an inversion-based 2D embedding model, a 2D embedder based on the sequence of image frames, the camera parameters, the region masks, and the camera views; computing, using an object-centric 2D adaptation model, a 2D adaptor based on the 2D embedder and the inversion-based editing instructions; generating, using a diffusion generation model, modified 2D frames by applying the 2D adaptor to the sequence of image frames and replacing the region masks corresponding to target objects with edited content; generating an initial edited 3D GS model based on the modified 2D frames, the camera parameters, and the region masks; and performing a meta tuning on the initial edited GS model to generate the edited GS model.

[0020] Optionally, in any of the preceding aspects, another implementation of the aspect provides initializing the 2D embedder using a pre-trained text-to-image generation model or a diffusion-based model.

[0021] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the 2D adaptor comprises adaptive parameters added to the 2D embedder.

[0022] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the adaptive parameters comprise low-rank adaptation (LoRA) layers or feature transformation layers.

[0023] A second aspect relates to a computing device, comprising: a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory or the storage means and configured to execute the instructions to cause the computing device to perform the method in any of the disclosed embodiments.

[0024] A third aspect relates to a computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computerexecutable instructions when executed by one or more processors of a computing device, cause the computing device to perform the method in any of the disclosed embodiments.

[0025] For the purpose of clarity, any one of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create a new embodiment within the scope of the present disclosure.

[0026] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0027] For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0028] FIG. 1 is a schematic diagram of an overall workflow for generating a free- view video using GAI and GS according to an embodiment of the disclosure.

[0029] FIG. 2 is a schematic diagram of an overall workflow of a semantic-aware 3D Gaussian splat module according to an embodiment of the disclosure.

[0030] FIG. 3 is a schematic diagram of an overall workflow of semantic-aware 3D GS edit module according to an embodiment of the disclosure.

[0031] FIG. 4 is a schematic diagram of a workflow of meta tuning module according to an embodiment of the disclosure.

[0032] FIG. 5 is a schematic diagram of an overall workflow of semantic-aware 3D GS edit module according to an embodiment of the disclosure.

[0033] FIG. 6 is a method implemented by a computing device according to an embodiment of the disclosure.

[0034] FIG. 7 is a schematic diagram of a network apparatus according to an embodiment of the disclosure.DETAILED DESCRIPTION

[0035] It should be understood at the outset that although an illustrative implementation of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0036] Disclosed herein are various systems and methods for generating a free-view video based on a guiding video and an object-centric instruction so that an object of interest in the generated video presents desired characteristics following the instruction. The guiding video serves as the basis to define the dynamic semantic development in the video, such as scene composition, scene content,scene movements, etc. The object of interest can be an existing object in the guiding video or a novel object to be added into the generated video. The object-centric instruction describes the desired operation of the object of interest in the generated video, chosen from a list of predefined objectcentric operations such as removing an existing object from the generated video, adding a novel object into the generated video, changing the appearance (e.g., color, texture, style) of an existing object, replacing an existing object by a different object, repositioning an existing object to another location, etc. Depending on different types of operations, different types of signals are given as the object-centric instructions. Multiple object-centric instructions can be given to control the generation process of multiple objects of interests, and the final generated video integrates these object-centric operations into the guiding video while following the semantic and scene development defined by the guiding video.

[0037] Great success has been achieved for Al Generated Content (AIGC) by using a wide range of image generative models, including generative adversarial networks (GAN), diffusion models, and auto-regressive (AR) models. The goal is to enable fast and accessible high-quality content creation. Various methods have been developed to allow for efficient manipulation of the generated content using different types of prompt inputs, such as using text descriptions and / or spatial / spatiotemporal compositions like sketches or segmentation maps. However, it is innately difficult to use prompt inputs to accurately describe detailed aspects of the image content, such as size, shape, color, location of various objects, scene composition, etc. It is difficult to control the generated output to have any specific desired attributes or visual details.

[0038] Recent works use pretrained text-to-image diffusion models like Imagen to create variations of images, or to manipulate specific image regions. Text-guided image-to-image transformation gives large flexibility to generate novel image content. However, same as general text-to-image generation, due to the lack of description power of using text to define visual content, it is innately difficult to control the generated output to have any specific desired visual details.

[0039] Video generation is one of the most difficult problems among all GAI-based content generation problems. Controlled video generation that produces desired visual content is even harder. Existing text-to-video generation engines like Sora or CustomVideo face the innate difficulty of using text prompts to accurately describe the desired generation target, such as the scene composition, scene motion, visual details and so on. Instead of using text descriptions, using an input video as a guiding example to accurately illustrate and define scene composition, semantic content,scene motion, etc. is much more efficient. Also, using image examples to illustrate and define the details of objects in the generated video enables the system to control the generation results.

[0040] In addition, how to achieve spatial-temporal consistency of both geometry and texture in the generated video is an open challenge. It is innately difficult to use text descriptions to adequately describe the desired spatial-temporal consist properties of the generated video, especially in terms of geometric accuracy. Using an input video as a guiding example, accurate 3D information of the video scene can be computed, which functions as a proxy to largely alleviate the problem of generating spatial-temporal consist content.

[0041] In terms of 3D scene manipulation, generative methods that uses radiance fields like Neural Radiance Fields (NeRF) face more challenges than traditional 3D reconstruction methods. Object-level manipulation such as object insertion, removal, or reposition, is especially hard. This is because NeRF models the synthesis process of an existing 3D scene without explicit 3D modeling, and it is very hard to explicitly manipulate specific objects in the learned implicit model. Recently 3D Gaussian Splatting has been developed to model the synthesis process of a 3D scene with explicit 3D Gaussians, which provides an opportunity for 3D object editing in radiance-field-based generative methods.

[0042] The disclosure is related to generating video content using GAI and GS. The disclosed approach introduces a system framework that takes as input a guiding video and a set of user instructions of target editing effects of individual objects in the generated video, and then generates an output free-view video where the scene composition and 3D scene structure are defined by the guiding video, and individual objects in the output generated video follows the user instructions in a spatial-temporal consistent way. The proposed system provides the capability for a variety of types of object control, such as removing objects, inserting novel objects, transferring styles of objects, and repositioning of objects.

[0043] The disclosed techniques have a wide range of applications across industries requiring precise, high-fidelity video content generation. For example, in entertainment and media sector, the framework embodied by the disclosed techniques can be used for efficient post-production editing by restyling characters or objects without reshooting or reanimation. For example, in fashion and retail, the framework supports virtual try-on experiences by replacing garments or accessories in video with styled alternatives. In gaming, the framework allows dynamic asset customization and cutscene generation with varied visual themes. Additionally, the disclosed techniques can beemployed in augmented and virtual reality environments for seamless integration of personalized content into existing scenes. In marketing and social media, it supports the creation of engaging, brand-aligned video content tailored to target audiences. The ability to selectively stylize video elements while preserving scene integrity makes the framework a powerful tool for controlled, creative video manipulation.

[0044] FIG. 1 is a schematic diagram of an overall workflow 100 (a.k.a., framework) for generating free-view video using GAI and GS according to an embodiment of the disclosure. In an embodiment, the overall workflow 100 is implemented by or on a personal computer (PC), a smart phone, a smart tablet, or some other computing device used to play games or consume entertainment.

[0045] As shown in FIG. 1, the overall framework 100 (a.k.a., system) is given an input comprising a guiding video represented by a sequence of T image frames Xlt... , XT. In an embodiment, each image frame Xtmay be represented in various formats, including but not limited to: 1-channel (e.g., gray scale image), 3-channel (e.g., red green blue (RGB) color image), and / or 4- channel (e.g., red green blue-depth (RGBD) image with color and associated depth), etc. In an embodiment, camera parameters Ccamof the camera capturing the guiding video input Xlt... , XTis also provided to the system. For example, for pinhole camera, Ccamcan be the camera intrinsic matrix where are focal length along the two axes of the imagingplane, x0, y0are principle point offsets of two axes of the imaging plane, and s is the axis skew.

[0046] In an embodiment, the guiding video Xlt... , XTthat captures three-dimensional (3D) scene from different camera views, the corresponding camera views v1, ... , vTfor the frames are either provided directly (e.g., as metadata associated with the video) or can becomputed in a computing view module, which employs 3D reconstruction techniques such as structure-from-motion to recover camera poses based on scene geometry and image correspondences.

[0047] In an embodiment, a set of region masksare provided to the system, each consisting of a set of masks marking the j-th object of interest in the Tframes of the guiding video. These masks can be obtained as ground-truth meta data (e.g., groundtruth segmentation masks in 3D rendered synthetic video or depth-guided segmentation during capture) or by segmentation methods like segment anything model (SAM). This disclosure doesnot put any restriction on how additional inputs, i.e., region masks, the camera views vlt... , vTand the camera parameters Ccamare obtained.

[0048] In an embodiment, the system 100 comprises a semantic-aware 3D Gaussian splat module 102 configured to compute a GS modelto represent the 3D scene of the guiding video based on the guiding video input s, ... , XT, the camera parameters Ccam, the camera views vlt... , vT, and the region masksThen, the system 100 is provided with a set of object-centric user instructions ins1, ... , insn, where each insj describes the desired operation to be performed on either a / -th object among the k objects of interest masked by or a novel object not existing in the guiding video.

[0049] In an embodiment, the system 100 further comprises an edit method selection module 104 configured to select the appropriate editing method for the target objects indicated in the user instructions and a semantic-aware 3D GS edit module 106 configured to edit the GS Model according to the selected editing method to generate an edited GS model

[0050] In an embodiment, the system 100 further comprises a semantic-aware rendering module 108 configured to compute, based on a target camera view voutand using the edited GS modelthe Synthesized Xout, which is spatial-temporally consistent with the 3D scene structure of the guiding video and the objects of interests present the desired characteristics following the corresponding user instructions. The target camera view voutcan be an arbitrary free view, i.e., can be the same as or different from the original camera views v , ... , vTof the guiding video.

[0051] Semantic-aware 3D Gaussian splat

[0052] In this disclosure, letdenote the set of 3D gaussians representing the 3D scene structure. In general Gaussian splatting, each Gaussianis defined by a set of parameters:where are the mean, scale, and rotation quaternion,is the opacity, andis the color. During rendering, to render an image with view vout, the 3D Gaussian is transformed into the image coordinates and projected onto the imageplane of the target 2D image to obtain a 2D Gaussianbased on pi. Si, qt, and vout. The rendered color of a pixel x in the target 2D image is computed using point-based alpha-blending on each ray:where N is the total number of Gaussians, andis the Gaussian response of overthe pixel.

[0053] In an embodiment, each Gaussian includes an additional parameteris the semantic Gaussian mask forand the j-th semantic label,where k is the total number of semantic objects corresponding to objects marked by the region masksand one additional semantic label is assigned to the background regions not marked by any masks.

[0054] FIG. 2 is a schematic diagram of an overall workflow 200 (a.k.a., framework) of semantic-aware 3D Gaussian splat module according to an embodiment of the disclosure. As shown in FIG. 2, the overall framework 200 (a.k.a., system) comprises a 3D Gaussian splat module 202 configured to compute an initial 3D GS Modelbased on guiding video input Xlt... , XTand the camera Views vlt... , vT, and using Gaussian splatting methods.

[0055] In an embodiment, the system 200 further comprises an inverse semantic projection module 204 configured to un-project the 2D semantic labels back to the initial 3D GS Modelbased on the region Masksand the camera views vlt... , vT. In an embodiment, for each pixel xtEXtwith semantic label lj masked by, the inverse semantic projection module 204 tries to locate the 3D Gaussian 0r, that accounts for xt, so that label Z,- can be associated with The inversesemantic projection module 204 can use different methods to do so.

[0056] In one embodiment, a depth map Dtis provided in Xt. For example, for the case the input Xtis a 4-channel RGBD image comprising both color and depth. A depth value dtis associated with the pixel xt. Then, the 3D Gaussian whose center location ptis closest to the depth dtcan be located as 0n. In another embodiment, when ground-truth depth information is not provided, the inverse semantic projection module 204 can make use of the corresponding 3D point of pixel xtin the 3D space. Using the camera parameters Ccamand the camera view vt, the pixel coordinate p(xt) = (u, v) and the 3D coordinates of the corresponding 3D point phas a fixed relation:, where K is the camera intrinsicmatrix obtained from the camera parameters Ccam, and E is the camera extrinsic parameters obtained from the camera view vt. That is, pixel xtcorresponds to a beam of ray in the 3D space. In terms of semantic label lj of xt, it is projected to the surface points in the 3D space. Duringrendering, the opacity is accumulated from near to far. Let on(xt) denote the accumulated opacity of xtafter adding the / / -th Gaussian, =1, , IV, when on(t) is greater than a cutoff threshold, it is assumed to reach a surface point, and this / / -th 3D Gaussian can be located as can be given as a hyperparameter. is theprojected 2D Gaussian of the 3D Gaussian based on pt, sitqt,vt, and is the Gaussianresponse of over pixel xt.

[0057] In an embodiment, for each semantic label lj, a total weight can be computed forGaussian to indicate how possiblecarries the semantic labelis a weight computed based on each input Xtand mask. For example,where has semantic label lj or not. nxis the located id of the 3D Gaussianfor pixel xt. In an embodiment, the Gaussianis assigned to the semantic label lj if WtJis greater than a threshold 14 , and entry mLj = 1 in m;. Otherwise, for allsemantic labelsand the Gaussian is considered to belong tobackground unknown category. The threshold can be given as a hyperparameter.

[0058] In other embodiments, the inverse semantic projection module 204 can use a neural network to predict the semantic label of 3D gaussians. Let T denote the neural network parameters, and using the initial 3D GS Modelas input, outputs m1, ■■■ , mN. The finalGaussians•■• , 7V form the GS model <This disclosure does not put any restrictions on methods or the network models the inverse semantic projection module 204 uses.

[0059] In an embodiment, the system 200 optionally comprises a GS update module 206 configured to iteratively update the initial GS Modele.g., by performing splitting and pruning, etc., and then perform inverse semantic projection to update the GS Modelso that the original gaussian splat loss can be optimized by considering semantic label un-projection. The optional processing flow is marked by dotted lines in FIG. 2.

[0060] Semantic-aware 3D GS edit

[0061] Given the object-centric user instructions ins^ ... , insn, the edit method selection module first selects the appropriate procedures to perform editing accordingly. In the preferred embodiment, the edit method selection module categorizes the instruction insj into two predefined categories: a direct editing instruction or an inversion-based editing instruction. For a direct editing instruction, the semantic-aware 3D GS edit module will use the process described in FIG. 3. For an inversion-based editing instruction, the semantic-aware 3D GS edit module will use the process described in FIG. 3. The predefined editing categories are empirically determined as prior knowledge. For example, the direct editing instructions include changing style (e.g. color and texture appearances) of an object without changing the geometry structure, removing an existing object, repositioning / rescaling / rotating an existing object. The inversion-based editing instruction include inserting a new object, replacing an existing object with another novel object, changing geometry structure of an existing object. This disclosure does not put restrictions on the specific types of direct editing or inversion-based editing instructions.

[0062] FIG. 3 is a schematic diagram of an overall workflow of semantic-aware 3D GS edit module 300 according to an embodiment of the disclosure. As shown in FIG. 3, the overall framework 300 (a.k.a., system) comprises a direct Gaussian edit module 302. Assume that there are nddirect editing instruction ins , ... , insnd, the direct gaussian edit module 302 first process the original GS Modelaccording to the instructions ins1, ... , insndto generate an initial edited GS ModelpxThe system further comprises a meta tuning module 304 in which a meta tuning process will compute the final edited GS model cpxv, M by tuning andcomputing an adaptor Ax( , M, ins), based on the guiding video input Xlt... , XT, the camera views vlt... , vT, the region masks and the camera parameters Ccam.

[0063] In an embodiment, for the instruction inSj of removing the / -th object, if a 3D Gaussian ©i is assigned to the target object and the total number of objectsis assigned to is less than a predefined threshold, it is removed from the GS Model. For the instruction insj of repositioning / rescaling / rotating the / -th object, if a 3D Gaussian 0 is assigned to the target object and the total number of objectsis assigned to is less than a predefined threshold, the position Pt, scale st, and rotation ofis direct edited to the desired location / scale / rotation. For theinstruction insj of changing the style of the / -th object, if a 3D Gaussian;is assigned to the target object and the total number of objects is assigned to is less than a predefined threshold, thecolor ctand opacity atofis either directly edited to the desired values if the instruction insj provided the target values. Sometimes the instruction insj does not provide specific values, but instead provide an example color or texture pattern, and the color q and opacityof will betuned to match the example through the meta tuning module 304.

[0064] In an embodiment, the main function of the meta tuning module 304 is to readjust the 3D Gaussian representation to remove artifacts and improve rendering performance. Without loss of generality, let the adaptor Axv, M, ins) contain the readjusted part of the 3D Gaussian model. For example, depending on the type of editing, parametersM) can be fixed or partly fixed, and the unfixed parameters can be tuned. New 3D Gaussians can be created whose parameters are learned during this tuning process. Therefore, Ax( , M, ins) contains all the adjusted parameters, i.e., the adjusted parameters fromand newly created parameters.

[0065] In an embodiment, the direct editing instruction can be conducted one after another where different target objects are edited one after another, or a group of objects can be edited together. For example, if the instruction is to switch the position of two objects, or to edit two close by objects, they can be edited together. The meta tuning process can be carried out after editing each object or can be carried out after many objects being edited initially.

[0066] FIG. 4 is a schematic diagram of a workflow 400 of meta tuning module according to an embodiment of the disclosure. As shown in FIG. 4, the overall framework 400 (a.k.a., system) comprises a meta tuning module. Assume that the current target is to conduct editing instructions of / objects and letdenote the current edited GS Model with the previousnd— J objects edited already. The tuning process is to obtain the edited GS Modelwhere the additional / objects are edited. The system 400 comprises a first rendering module 402 configured to, for each camera view vt, render a 2D image based on equation (1)usingThe tuning process is iterative. The system 400 further comprises a second rendering module 404 configured to, using the current <px(v, M, new) and for each camera view vt, t = 1, render a 2D image Xbased on equation (1) too. The system 400 further comprises an inpainting module 406 configured to, using a 2D inpainting method, compute an inpainted imagebased onThe system 400 further comprises an compute tuning loss module 408 configured to compute a lossbetween the inpaintedand X based on theregion mask ... , , where is the mask for the Zy-th target object in Xt. For example, forregions not masked by any. should match with and a distortion lossMflr) (e.g., LI or L2 loss) can be computed between pixels of and in these regions, where is pixel- wise masking operation. Overall perceptual qualitylike Frechet inception distance (FID) score of the inpainted can also be included into the loss.Adversarial loss from GAN can also be added to improve the realist of the inpainted. The system 400 further comprises a 3D Gaussian splat module 410. The overallis then combined with the original Gaussian splat loss to update the adjustable parameters in modelnew) by using the inpaintedas model input for camera view vtthrough the 3D GS Splat module 410. After many iterations through different camera views the fina can be obtained,which is then used to replace Then, the system 400 moves on to process the nextbatch of editing instructions, and after all the instructions are processed, the final, new) will be used as the edited GS model

[0067] Semantic-aware 3D GS edit with inversion-based instruction

[0068] FIG. 5 is a schematic diagram of an overall workflow of semantic-aware 3D GS edit module 500 according to an embodiment of the disclosure. Assume that there are nLinversion-based editing instruction ins1:..., insnAs shown in FIG. 5, the overall framework 500 (a.k.a., system) comprises an inversion-based 2D embedding module 502. Based on the guiding video input the camera views the region masks and the cameraparameters Ccam. inversion-based 2D embedding module 502 first compute an embedder mThe system 500 further comprises an object-centric 2D adaptation module 504 configured to compute an 2D adaptor 6x( , M, ins) based on the guiding video input XltXT, the camera views vlt... , vT, the region masksthe camera parameters Ccam, the embedder and the inversion-based editing instruction ins^ insn. . The system 500 furthercomprises a diffusion generation module 506. The 2D Adaptor is used to compute agenerated input based on the guiding video inputand the region masksin the diffusion generation module 506. The system 500 further comprisesa semantic-aware 3D Gaussian splat module 508 configured to compute an edited initial GS model based on the camera viewsT, the region masks thecamera parameters Ccam, and the generated input. The system 500 further comprises ameta tuning module 510 configured to compute the final edited GS Model (px( , M) based on the edited initial GS model, the guiding video input Xlt... ,XT, the camera views vlt... , vT, the region masksand the camera parameters Ccam.

[0069] Compared to the previous framework 300 for direct Gaussian edit, the main difference is how to obtain the edited initial GS model, and the meta tuning module is the same as FIG. 4. Similar to the previous direct Gaussian edit case, the inversion-based instructions can be conducted one after another where different target objects are edited one after another, or a group of objects can be edited together. The meta tuning process can be carried out after editing each object or can be carried out after many objects being edited initially.

[0070] Without loss of generality, assume that the current target is to conduct editing instructions of J objects Jand letP denote the current edited GS Model with the previous n, — J objects edited already. Initially,before conducting any editing instruction. The current task of the semantic-aware 3D GS edit module is to obtain the edited GS Modelnew) where the additional J objects are edited. The embeddernxy, M) of the infusion-based 2D embedding module 502 is initialized as a pre-trained image generation model, such as a pre-trained text-to-image generation model. Based on each original input Xt, this (nx(y, M) computes a generated input Xen, which is expected to match the original input Xtover the region of the target objects to edit, and a distortion lossLI or L2 loss) can be computed between pixels of and Xtin these targeted regionsmasked bythe mask for the Zy-th target object in Xt. Gradient of this loss can be computed and backpropagated to update the model parameters of . Optionally,a text description describing the target objects in Xtcan be used as input to a>x(y, M) for computing too. Such a text description can be a user input or can be generated automatically through image-to-text inversion using a vision-language model like CLIP. After many iterations through different original inputs, the updated embedder mx(y, M) can be seen as a 2D image generation model that focus on generating 2D images of the target objects of interest. Note that a different embedder can be tuned from the pre-trained image generation model for each instruction inSj, or all different instructions ins1, ... , insncan share a same embedder

[0071] In an embodiment, the object-centric 2D adaptation module 504 further configured to compute 2D Adaptor based on the updated embedder Further, the 2DAdaptor) has the same backbone structure as embedder , with objectcentric adaptive parametersadded. For example, the LoRA layers or feature transformation layers can be attached to each layer in where parameters inare fixed and the added adaptive parameters Ax(v, M, ins) are learned for the object-centric 2D adaptation module 504.

[0072] In an embodiment, for inversion-based instructions, each inSj usually consists of additional information other than a text description of the editing goal. For example, when the type of editing is to change the style of an existing object or to insert a new object into the scene, one or multiple visual examples are provided in inSj to illustrate the desired style of the existing object to edit or the target object to insert denote a set of image examples in forinSj illustrating the desired presenting style of the target object or the new target object. Different from the same embedderan individual 2D adaptoris usually learned for each individual instruction inSj. Further, for each original input Xt, the 2D adaptor / computes a modified / based on both input Xtand the text instruction in inSj. Then a best matching image example is found, and the target object region in and areexpected to match. can be one of , or can be synthesized based onusing single-view 3D object reconstruction methods or multi-view diffusion models. That is, a distortion loss (e.g., LI or L2 loss) can be computed betweenpixels of X andin these targeted regions masked by Gradient of this loss can becomputed and backpropagated to update the model parameters ofAfter many iterations through different original inputs, the updatedcan be seen as a 2D image editing model that focus on generating 2D images of the target objects of interest with the desired editing effect.

[0073] In an embodiment, for learning both embedder and the 2D adaptor, besides the distortion scores over the target region of interest, other loss functionslike the overall perceptual quality (e.g., FID score) can also be included into the loss. Adversarial loss from GAN can also be added to improve the realist of the generated x or modified x

[0074] In an embodiment, the diffusion generation module 506 further configured to compute, using the final updated 2D adaptor^, for each camera view vt, and based on the original input and each instruction inSj, the generated input . The diffusion generationmodule 506 basically performs inference through( by using Xtas input, which computes the modifiedAfter performing inferences for all J objects indicated by instruction the generated inputtis obtained from x by replacingeach target object region in the original ^ by the corresponding object region in

[0075] Semantic-aware rendering

[0076] In an embodiment, after obtaining edited GS Model , given an arbitrary targetcamera view , the semantic-aware rendering module 108 configured to compute the synthesized To improve spatial-temporally consistency with the 3D scene structure of the guiding video, in the preferred embodiment, the rendering process of Equation (1) uses the edited GS Modelto render regions over the current target J objects, and uses the original GS Modeto render the remaining regions. Further, for the j-th object with label lj, the final rendered color( ) of a pixel x is a normalized combination of the rendered color fromwhere is the 2D Gaussian of the z-th Gaussiann of the z-th the semanticGaussian mask for 0ewand the Ij-th semantic label, and m! ™ is the semantic Gaussian mask for and the j semantic label.

[0077] FIG. 6 is a method 600 implemented by a computing device according to an embodiment of the disclosure. In an embodiment, the computing device is a computer, a smart phone, a smarttablet, or other device configured to play games or display video content. In an embodiment, the method 600 is implemented during gaming or when video content is being consumed by a user.

[0078] In block 602, the computing device receives a guiding video comprising a sequence of image frames representing a three-dimensional (3D) scene from different viewpoints. In an embodiment, each image frame comprises one or more of a grayscale image, a color image, or a color image with associated depth information

[0079] In block 604, the computing device receives camera parameters and camera views associated with the guiding video. In an embodiment, the computing device obtains the camera views by estimating the camera views using a structure-from-motion algorithm. In an embodiment, the computing device receives metadata associated with the guiding video, wherein the metadata includes the camera views for each image frame of the guiding video; and obtains the camera views from the metadata.

[0080] In block 606, the computing device receives region masks identifying objects of interest across the sequence of image frames. In an embodiment, the computing device obtains the region masks using a semantic anything model (SAM).

[0081] In block 608, the computing device generates, using a semantic-aware 3D Gaussian splatting (GS) model, a GS model representing the 3D scene based on the sequence of image frames, the camera parameters, the camera views, and the region masks. In an embodiment, the GS model comprises a set of 3D Gaussians representing the 3D scene, and wherein each 3D Gaussian comprises a position vector, a scale vector, a rotation quaternion, an opacity value, a color vector, or a semantic mask vector for splatting-based rendering.

[0082] In block 610, the computing device receives object-centric instructions specifying operations to be performed on the objects of interest or on novel objects not present in the guiding video. In an embodiment, the object-centric instructions comprise removing an existing object, adding a novel object, changing appearance of an object, replacing the existing object by a different object, or repositioning the existing object to another location. In an embodiment, the object-centric instructions comprise direct editing instructions or inversion-based editing instructions.

[0083] In block 612, the computing device selects, based on the object-centric instructions, an editing method.

[0084] In block 614, the computing device applies the editing method to the GS model to generate an edited GS model that reflects the operations on the objects of interest. In anembodiment, the computing device applies, for each direct editing instruction, a direct edit process to the GS model to generate an initial edited GS model; and performs a meta tuning process on the initial edited GS model to generate the edited GS model. In an embodiment, the computing device applies, for each inversion-based editing instruction, an inversion-based edit process to generate an initial edited GS model; and performs a meta tuning process on the initial edited GS model to generate the edited GS model.

[0085] In block 616, the computing device renders rendering a synthesized video frame from a target camera view using the edited GS model. In an embodiment, rendering the synthesized image comprises performing point-based alpha by projecting each 3D Gaussian into a 2D Gaussian in the target camera view. In an embodiment, the target camera view is a novel view not present in the guiding video.

[0086] In block 618, the computing device generates a free-view video from the synthesized video frame, wherein the free-view video is spatio-temporally consistent with the guiding video and reflects the object-centric instructions. In an embodiment, the free-view video is displayed on a display, screen, or monitor of the computing device for the benefit and enjoyment of the user. In an embodiment, the free-view video is stored in a non-transitory computer-readable storage medium and / or transmitted to a remote device over a network for display or further processing.

[0087] FIG. 7 is a schematic diagram of a computing device 700 (e.g., a personal computer, smart phone, smart tablet, handheld gaming device, etc.) according to an embodiment of the disclosure. The computing device 700 is suitable for implementing the disclosed embodiments as described herein. The computing device 700 comprises ingress ports / ingress means 710 (a.k.a., upstream ports) and receiver units (Rx) / receiving means 720 for receiving data; a processor, logic unit, or central processing unit (CPU) / processing means 730 to process the data; transmitter units (Tx) / transmitting means 740 and egress ports / egress means 750 (a.k.a., downstream ports) for transmitting the data; and a memory / memory means 760 for storing the data. In an embodiment, the receiver units (Rx) / receiving means 720 comprise a discrete circuit, integrated circuit, chip set, package, hardware module, electronic device, or other structure capable of receiving signals. In an embodiment, the transmitter units (Tx) / transmitting means 740 comprise a discrete circuit, integrated circuit, chip set, package, hardware module, electronic device, or other structure capable of transmitting signals. The computing device 700 may also comprise optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress ports / ingress means710, the receiver units / receiving means 720, the transmitter units / transmitting means 740, and the egress ports / egress means 750 for egress or ingress of optical or electrical signals.

[0088] The processor / processing means 730 is implemented by hardware and software. The processor / processing means 730 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor / processing means 730 is in communication with the ingress ports / ingress means 710, receiver units / receiving means 720, transmitter units / transmitting means 740, egress ports / egress means 750, and memory / memory means 760. The processor / processing means 730 comprises a free-view video generation module 770. The free-view video generation module 770 is able to implement the methods disclosed herein. The inclusion of the free-view video generation module 770 therefore provides a substantial improvement to the functionality of the computing device 700 and effects a transformation of the computing device 700 to a different state. Alternatively, the free-view video generation module 770 is implemented as instructions stored in the memory / memory means 760 and executed by the processor / processing means 730.

[0089] The computing device 700 may also include input and / or output (I / O) devices or I / O means 780 for communicating data to and from a user. The I / O devices or I / O means 780 may include output devices such as a display for displaying video data, speakers for outputting audio data, etc. The I / O devices or I / O means 780 may also include input devices, such as a keyboard, mouse, trackball, etc., and / or corresponding interfaces for interacting with such output devices. The memory / memory means 760 comprises one or more disks, tape drives, and solid-state drives and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. The memory / memory means 760 may be volatile and / or non-volatile and may be read-only memory (ROM), random-access memory (RAM), ternary content-addressable memory (TCAM), and / or static RAM (SRAM).

[0090] Embodiments of the present disclosure provide at least the following technical advantages.

[0091] A. A novel free-view video generation method that allows controls over both the semantic scene content and desired styles of individual objects. The semantic scene content is controlled by a guiding video input, and the desired styles of specific objects are controlled by userinstructions, which comprise image examples and / or text descriptions of the objects depending on the target of control. A 3D representation in the form of 3D Gaussians is learned to model the desired 3D scene, which generates video frames of the modeled scene from arbitrary views.

[0092] A novel system that transforms a guiding video input into a free-view video by editing the 3D representation of the scene captured by the guiding video according to user instructions. The system allows a variety of controls over individual objects, such as changing the spatial position, pose, and styles of existing objects, removing existing objects, inserting new objects, and all other controls that can be derived from these operations. Depending on the types of controls, image examples and / or text descriptions can be used as instructions. The edited 3D representation promotes spatial-temporal consistency for generated video, and allows continuous content creation, i.e., by further editing the edited 3D representation with new instructions generate new video with new desired objects.

[0093] While several embodiments have been provided in the present disclosure, it may be understood that the disclosed systems and methods might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.

[0094] In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims

CLAIMSWhat is claimed is:

1. A method implemented by a computing device, comprising: receiving a guiding video comprising a sequence of image frames representing a three- dimensional (3D) scene from different viewpoints; receiving camera parameters and camera views associated with the guiding video; receiving region masks identifying objects of interest across the sequence of image frames; generating, using a semantic-aware 3D Gaussian splatting (GS) model, a GS model representing the 3D scene based on the sequence of image frames, the camera parameters, the camera views, and the region masks; receiving object-centric instructions specifying operations to be performed on the objects of interest or on novel objects not present in the guiding video; selecting, based on the object-centric instructions, an editing method; applying the editing method to the GS model to generate an edited GS model that reflects the operations on the objects of interest; rendering a synthesized video frame from a target camera view using the edited GS model; generating a free- view video from the synthesized video frame, wherein the free-view video is spatio-temporally consistent with the guiding video and reflects the object-centric instructions; and displaying the free-view video on a display device.

2. The method of claim 1, wherein each image frame comprises one or more of a grayscale image, a color image, or a color image with associated depth information.

3. The method of any of claims 1 -2, further comprising obtaining the camera views and the camera parameters using a structure-from-motion algorithm.

4. The method of any of claims 1-3, further comprising obtaining the region masks using a semantic anything model (SAM).

5. The method of any of claims 1 -4, wherein the target camera view is a novel view not present in the guiding video.

6. The method of any of claims 1-5, wherein the GS model comprises a set of 3D Gaussians representing the 3D scene, and wherein each 3D Gaussian comprises a position vector, a scale vector, a rotation quaternion, an opacity value, a color vector, or a semantic mask vector for splatting-based rendering.

7. The method of any of claims 1-6, wherein rendering the synthesized video frame comprises projecting each 3D Gaussian into the target camera view and blending projected two-dimensional (2D) Gaussians using alpha-blending.

8. The method of any of claims 1-7, further comprising: receiving the guiding video, the region masks, and the camera views to generate an initial 3D GS model using a Gaussian splatting method; and assigning semantic labels to Gaussians in the initial 3D GS model using an inverse semantic projection model to generate the semantic-aware 3D GS model.

9. The method of any of claims 1-8, wherein the object-centric instructions comprise removing an existing object, adding a novel object, changing appearance of an object, replacing the existing object by a different object, or repositioning the existing object to another location.

10. The method of any of claims 1-9, wherein the object-centric instructions comprise direct editing instructions or inversion-based editing instructions.

11. The method of any of claims 1-10, further comprising applying, for each direct editing instruction, a direct edit process to the GS model to generate an initial edited GS model; and performing a meta tuning process on the initial edited GS model to generate the edited GS model.

12. The method of any of claims 1-11, further comprising applying, for each inversion-based editing instruction, an inversion-based edit process to generate an initial edited GS model; and performing a meta tuning process on the initial edited GS model to generate the edited GS model.

13. The method of any of claims 1-12, wherein performing the meta tuning process comprises: rendering, for each camera view, a pre-edit 2D image using a previously edited GS model; rendering, for each camera view, a post-edit 2D image using a newly edited GS model; applying a 2D inpainting model to the post-edit 2D image to produce an inpainted image; computing a tuning loss between the inpainted image and the pre-edit 2D image based on object-specific region masks; updating the GS model using the tuning loss and Gaussian splat rendering losses; and iteratively performing the meta tuning process across the camera views to obtain the edited GS model.

14. The method of any of claims 1-13, wherein the tuning loss comprises a perceptual loss or an adversarial loss.

15. The method of any of claims 1-14, wherein performing the meta tuning process comprises: computing, using an inversion-based 2D embedding model, a 2D embedder based on the sequence of image frames, the camera parameters, the region masks, and the camera views; computing, using an object-centric 2D adaptation model, a 2D adaptor based on the 2D embedder and the inversion-based editing instructions; generating, using a diffusion generation model, modified 2D frames by applying the 2D adaptor to the sequence of image frames and replacing the region masks corresponding to target objects with edited content; generating an initial edited 3D GS model based on the modified 2D frames, the camera parameters, and the region masks; and performing the meta tuning process on the initial edited GS model to generate the edited GS model.

16. The method of any of claims 1-15, further comprising initializing the 2D embedder using a pre-trained text-to-image generation model or a diffusion-based model.

17. The method of any of claims 1-16, wherein the 2D adaptor comprises adaptive parameters added to the 2D embedder.

18. The method of any of claims 1-17, wherein the adaptive parameters comprise low-rank adaptation (LoRA) layers or feature transformation layers.

19. A computing device, comprising: a memory configured to store instructions; and one or more processors coupled to the memory and configured to execute the instructions to cause the computing device to perform a method according to any of claims 1-18.

20. A computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computer-executable instructions when executed by one or more processors of a computing device, cause the computing device to perform a method according to any of claims 1-18.21 . A computing device, comprising: a storage means configured to store instructions; and one or more processing means coupled to the storage means and configured to execute the instructions to cause the computing device to perform a method according to any of claims 1-18.

22. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause a computing device to perform operations according to any of claims 1- 18.

Images