Video object retiming via hierarchical neural rendering

Abstract

A computer-implemented method for decomposing a video into a plurality of layers (212, 213) that can be recomposed with modified relative timing includes obtaining video data comprising a plurality of image frames (201) depicting one or more objects. For each frame of the plurality of frames, the computer-implemented method includes generating one or more object maps that describe a respective position of at least one object of the one or more objects within the image frame. For each frame of the plurality of frames, the computer-implemented method includes inputting the image frame and the one or more object maps into a machine-learned layer Tenderer model (220). For each frame of the plurality of frames, the computer-implemented method includes receiving, as output from the machine-learned layer Tenderer model, a background layer that accounts for a background of the video data and one or more object layers respectively associated with one of the one or more object maps. The object layers include image data that accounts for the at least one object and one or more artifact effects attributable at least in part to the at least one object, such that the one or more object layers and the background layer can be recomposed with modified relative timing.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to video processing. More specifically, the present disclosure relates to systems and methods for separating a video into multiple layers that can be recombined with modified relative timing. BACKGROUND

[0002] By manipulating the timing of a person's movement in a video, various effects that can change the suggested perception of events recorded in the video can be achieved. For example, in movies, altering time by speeding up, slowing down, or synchronizing a person's motion is often used to dramatize or de-emphasize certain movements or events in a video. For example, by freezing the motion of some of the people in a video full of action while allowing others to move, a viewer's attention can be focused on a particular person of interest.

[0003] To date, motion retiming has been primarily studied in the context of character animation and has been used as a tool to edit a character's motion to match a desired duration or target speed at a given time. However, these animation-focused techniques do not directly apply to the domain of natural realistic videos. For example, in the field of character animation, a major challenge can be retiming the motion of a set of joints where there is a spatio-temporal correlation between them. However, similarly, manipulating the timing of a person in a video requires not only modifying the person's self-motion, but also modifying all the various elements in the scene related to them - shadows, reflections, a dancer's flowing dress, or splashing water.

[0004] As such, a video processing system seeking to retime a person within a video should ensure that all related events in the scene follow and respect the timing change. Moreover, unlike character animation, a system seeking to retime natural realistic videos does not have any realistic-valued 3D model of the scene over time; therefore, rendering photo-realistic, high-quality retiming effects in a video is more challenging. SUMMARY

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0006] One example aspect of the present disclosure relates to a computer-implemented method for decomposing a video into a plurality of layers that can be recombined with modified relative timing. The computer-implemented method can include obtaining, by a computing system comprising one or more computing devices, video data. The video data can include a plurality of image frames depicting one or more objects. For each frame of the plurality of frames, the computer-implemented method can include generating, by the computing system, one or more object maps. Each object map of the one or more object maps can describe a respective position of at least one object of the one or more objects within the image frame. For each frame of the plurality of frames, the computer-implemented method can include inputting, by the computing system, the image frame and the one or more object maps into a machine-learned layer renderer model. For each frame of the plurality of frames, the computer-implemented method can include receiving, by the computing system, as output from the machine-learned layer renderer model, a background layer illustrative of a background of the video data and one or more object layers respectively associated with one object map of the one or more object maps. Each of the one or more object layers can include, for each image frame, image data illustrative of the at least one object and one or more artifacts attributable at least in part to the at least one object, such that the one or more object layers and the background layer can be recombined with modified relative timing.

[0007] Another example aspect of the present disclosure relates to a computing system configured to decompose video data into multiple layers. The computing system can include one or more processors and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations can include obtaining, by a computing system comprising one or more computing devices, video data. The video data includes a plurality of image frames depicting one or more objects. For each frame of the plurality of frames, the operations can include generating, by the computing system, one or more object maps. Each object map of the one or more object maps can describe a respective position of at least one object of the one or more objects within the image frame. For each frame of the plurality of frames, the operations can include inputting, by the computing system, the image frame and the one or more object maps into a machine-learned layer renderer model. For each frame of the plurality of frames, the operations can include receiving, by the computing system, as output from the machine-learned layer renderer model, a background layer illustrative of a background of the video data and one or more object layers respectively associated with one object map of the one or more object maps. Each object layer of the one or more object layers can include, for each image frame, image data illustrative of the at least one object and one or more artifacts attributable at least in part to the at least one object, such that the one or more object layers and the background layer can be recomposed with modified relative timing.

[0008] Other aspects of the present disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0009] These and other features, aspects, and advantages of various embodiments of the present disclosure will be better understood when read with reference to the following description and appended claims in conjunction with the accompanying drawings. The accompanying drawings illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related principles. BRIEF DESCRIPTION OF DRAWINGS

[0010] A detailed discussion of embodiments with reference to the drawings will be made in the description, which makes apparent to those of ordinary skill in the art the implementation of the embodiments.

[0011] FIG. 1A FIG. C depicts an example application of a layered neural rendering technique in accordance with example embodiments of the present disclosure.

[0012] FIG. 2 An example processing pipeline for performing a layered neural rendering technique in accordance with example embodiments of the present disclosure is depicted.

[0013] FIG. 3An example layer decomposition using a layered neural rendering technique according to an example embodiment of the present disclosure is described.

[0014] FIG. 4 Example images and UV diagrams are depicted according to exemplary embodiments of the present disclosure.

[0015] FIG. 5 Example ternary diagrams and images are depicted according to exemplary embodiments of the present disclosure.

[0016] FIG. 6 The manual editing of the initial mask according to an example embodiment of this disclosure is depicted.

[0017] FIG. 7 An automatic motion alignment technique according to an example embodiment of the present disclosure is described.

[0018] FIG. 8 Example results of layered versus non-layered rendering according to an example embodiment of this disclosure are depicted.

[0019] FIG. 9 Example retiming results according to example embodiments of this disclosure are depicted.

[0020] FIG. 10 Example retiming results according to example embodiments of this disclosure are depicted.

[0021] FIG. 11A A block diagram of an example computing system according to an example embodiment of the present disclosure is depicted.

[0022] FIG. 11B A block diagram of an example computing device according to an example embodiment of the present disclosure is depicted.

[0023] FIG. 11C A block diagram of an example computing device according to an example embodiment of the present disclosure is depicted.

[0024] FIG. 12 A block diagram of an example layer renderer system according to an example embodiment of the present disclosure is depicted.

[0025] FIG. 13 A block diagram of an example layer renderer system according to an example embodiment of the present disclosure is depicted.

[0026] FIG. 14 A flowchart is depicted illustrating an example method, according to an example embodiment of the present disclosure, for decomposing a video into multiple layers capable of being recombined with modified relative timing. Detailed Implementation

[0027] SUMMARY

[0028] In general, the present disclosure is directed to systems and methods that enable retiming people or other objects in ordinary natural videos, such as manipulating and editing the timing of different motions of various individuals in a video. Example implementations of the present disclosure are able to align different motions in time, change the speed of certain actions (e.g., speed up / slow down, or completely "freeze" an object), or "erase" selected objects from a video. These effects can be used to enhance the way a video observer perceives and interacts with the proposed dynamic world through ordinary videos. For example, by freezing certain motions in a video full of action, and letting only one object move, example implementations of the present disclosure are able to focus the viewer's attention on that particular object.

[0029] Example implementations of the present disclosure are able to computationally implement these effects via a specialized learning-based hierarchical video representation. Each frame in a video can be decomposed into separate RGBA layers, representing the appearance of different objects in the video. One valuable property of the proposed model is that it is able to disentangle not only the direct motion of each object in the input video, but also each object's relation to the scene changes it generates (e.g., shadows, reflections, and / or motion of loose clothing), which are collectively referred to herein as "trace effects."

[0030] These layers can be individually retimed and recombined into a new video. This can allow a video processing system to generate realistic, high-quality renderings of retiming effects for real-world videos depicting complex actions involving many different individuals, including dancing, trampoline jumping, or group running.

[0031] More specifically, the input to the proposed method can be an ordinary natural video with multiple objects moving, and the output can be a realistic re-rendering of the video where the timing of the object movements can be modified. The proposed system and method support various retiming effects, including aligning motions of different objects, changing the speed of certain actions (e.g., speeding up / slowing down or completely "freezing" an object). In addition, the proposed technology is also able to "erase" selected objects from a video.

[0032] Some example implementations of the present disclosure focus on temporal warping, but do not perform generative actions. That is, the pose or other configuration of each object (e.g., person) in the proposed output exists in some frame of the input, and no new unseen poses or viewpoints are generated.

[0033] FIG. 1A FIG. 1 illustrates one example application of the technology described herein. In particular, FIG. 1A FIG. 2 illustrates multiple frames of an original video, where each of multiple children jumps into a pool at different times. FIG. 1B FIG. 3 illustrates multiple retiming frames of a computationally retimed video. As in FIG. 1BAs can be seen in the re-timed video frames, the jumps of children I and III align in time with the jump of child II, such that they all jump into the pool together. Child II remains constant in the input and output frames.

[0034] Example embodiments of the present disclosure can produce this and other re- timing effects in natural, ordinary videos. The proposed techniques can leverage a novel deep neural network that learns a hierarchical decomposition of the input video. FIG. 1C Examples of this decomposition are shown in FIG. 1. For example, the proposed model can not only disentangle the motion of objects in different layers, but also capture various scene elements (e.g., trace effects) related to those objects (e.g., water splashes when the children hit the water, shadows, reflections). When objects are re-timed, the trace effects can be automatically re-timed with the objects due to the learned associations with the objects. This can allow users (e.g., users of video editing programs) to create realistic and faithful re-renderings of videos for a wide variety of re-timing effects.

[0035] As one example, a core of the proposed techniques can be a novel deep neural network-based model that learns a hierarchical decomposition of the input video. More specifically, each frame of the original video can be decomposed into a set of layers, each layer consisting of an RGB color image and an opacity mask a (collectively, “RGBA”). Example embodiments of the present disclosure can include designing and training the proposed model such that each RGBA layer over time can be associated with a particular object in the video (e.g., a single person or a group of people predefined by the user to be considered as a group). This allows the proposed model to not only disentangle the motion of objects at different layers, but also relate the objects to changes in the scene related to the objects (e.g., trace effects).

[0036] Given the decomposition layers that define each object, the original frames of the video can be easily reconstructed (e.g., using standard back-to-front compositing). For example, re-timing effects can be produced by simple manipulation of the layers (removing, copying, or interpolating particular layers) without additional training or processing. This can allow re-timing effects to be implemented using simple video editing software.

[0037] Example embodiments of the present disclosure will be discussed in greater detail. For ease of illustration, example embodiments of the present disclosure are described herein with respect to re-timing of people or groups of people in videos. However, the proposed techniques are equally applicable to re-timing of other objects in videos — including, for example, animals, inanimate objects (e.g., cars or other vehicles, traffic signs, etc.), and / or any other object. Thus, further references to people or groups of people should be interpreted as scalable to any form of object.

[0038] Example techniques

[0039] Given an input video with multiple people with natural movements, the present disclosure provides a model suitable for supporting various retiming effects, including aligning different people's motions in time or changing the speed of certain people (e.g., speeding up / slowing down, or freezing them completely). However, high-quality rendering of such effects poses several fundamental challenges. For example, one challenge can be motion consistency. Example implementations of the present disclosure can isolate not only the motion of the subjects, but also all the motion caused by the subjects (e.g., trace effects). For example, in FIGS. 1A-1C , each child causes splashes when hitting the water. Other common cases of trace effects include reflections, shadows, or objects attached to the subjects, such as loose clothing. As another example, a challenge can be occlusion and de-occlusion. Retiming motions can often cause new occlusions and de-occlusions in the scene. Rendering scene content in de-occluded regions and maintaining correct depth ordering between subjects can help achieve high-quality, realistic effects. Another example challenge can be temporal coherence. Achieving temporally coherent results can be challenging - small errors in alignment between frames, such as subtle misalignments, can manifest as noticeable visual artifacts when frames are viewed as a video.

[0040] Example implementations of the present disclosure can address these challenges via a deep neural network that decomposes the input video V into layers. In some implementations, the proposed model can be trained in a self-supervised manner only on the input video, i.e., without any external data, and without any manual annotation. In the following sections, example implementations of the present disclosure describe the proposed layered video representation, neural rendering model, training scheme, and data preprocessing.

[0041] Example layered video representation

[0042] Given an input video V, the proposed goal can be to decompose each frame I t ∈ V into a set of N layers:

[0043]

[0044] wherein can be a color (RGB) image, and can be an opacity map (mask). All frames of the i-th layer can be associated with a person i in the video. Example implementations of the present disclosure can add an additional background layer that is not associated with any person or object, which can include learning the background color and / or the background scene.

[0045] Given this layered representation and a desired retiming ot representing the front-to-back reversal of the ordering of the layers, each frame of the video can be rendered by the standard "over" operator (e.g., ordered covering). Example implementations of the present disclosure represent this operation by the following:

[0046]

[0047] Example implementations of the present disclosure assume a synthetic order o t may be known, but is time-varying, i.e., the depth ordering between people can change throughout the video.

[0048] A key property of this representation can be that retiming effects can be achieved by simple operations on individual layers. For example, removing person i from frame t can be done simply by removing the ith layer from the composition (e.g., by setting to zero in equation 2). Similarly, generating a video that can freeze person i at time t0can be achieved by copying all frames before t0and after t0 Example operations include person removal, motion freezing, and "photo finish" effects.

[0049] Some existing video retiming tools only provide UV coordinates for fully visible regions. In contrast, the proposed method can be able to generate full-body UVs for occluded people. For example, the proposed method can first include training a network on an external dataset of keypoint and UV map pairs. Then, the method can include inputting a complete skeleton of an occluded person. This can allow creating editing effects that require de-occluded people.

[0050] Estimating a set of layers from a normal real-world video may be a difficult and / or under-constrained problem. For example, there can be many possible decompositions that can provide an accurate reconstruction of the original frame I t To produce realistic high-quality renderings of retiming effects, each layer should therefore not only disentangle each person and their own motion accurately, but also capture all motion and / or other trace effects caused by that person in the scene. Furthermore, the information captured by each layer set may desirably extend beyond the frame I tvisible content. This can support image inpainting of people or background content that can be de-occluded when re-timing motion. Thus, in some embodiments, it can be beneficial to impose additional constraints and priors in order to steer the technical solution towards a desired decomposition. Example implementations of the present disclosure can do so indirectly through the proposed model design. For example, the structure of the convolutional neural network as a prior can pull the result towards a decomposition that naturally seems reasonable. Additionally and / or alternatively, this can be done directly through the training scheme and loss according to example aspects of the present disclosure.

[0051] Example layered neural rendering

[0052] The proposed model can combine a neural rendering approach (e.g., an integration of traditional elements from graphics rendering with the latest advances in deep learning) in a novel human-specific hierarchical video representation, which example implementations of the present disclosure refer to as hierarchical neural rendering.

[0053] FIG. 2 An example pipeline 200 is illustrated in FIG. 2. At a high level, the proposed pipeline 200 can first include estimating the geometry of each person in a video (e.g., using a combination of existing off-the-shelf tools and proposed own tools). For example, the pipeline can include obtaining a video 201 comprising a plurality of image frames. Additionally, the pipeline can include generating one or more object maps 202 (e.g., UV maps) describing the geometry. For example, each object map 202 of the one or more object maps 202 can describe a respective position of at least one object of the one or more objects within an image frame. The object maps 202 can be used to sample from a depth texture map 204 by a resampler 210, which are then individually passed to a neural renderer 220 to produce RGBA layers 212 (including color layers and / or opacity or alpha layers 213). When composited (e.g., at 230), the layers can reconstruct the original video frames 240. To produce editing effects, example implementations of the present disclosure can use the RGBA layers 212 predicted by the neural renderer 220 during training and modify the original video by adjusting the layers 212 (e.g., turning layers on / off, changing their speed, and playing different layers with different temporal offsets).

[0054] As an example, FIG. 2An example machine learning layer rendering model 220 is shown. The layer rendering model is able to decompose each frame of a video into a set of RGBA layers (color and opacity). Each layer can be associated with a particular person in the video (either a single person or a group of people pre-defined by the user). For example, the person or object can be defined by a UV map. The layers can be computed in separate forward passes by feeding the neural renderer with depth texture maps corresponding to individual layers. In particular, example embodiments of the invention utilize a single depth texture map T i representing each person in the video, and the scene background can be represented with a depth texture map T B . Given a pre-computed UV map, those depth texture maps are re-sampled and composited to form the input to the neural renderer. The set of estimated layers can then be composited in reverse order to reconstruct the original frame. Retiming effects can be achieved via simple manipulation of the layers.

[0055] The background can be represented with a single texture map T 0 for the entire video. Sampling from the background can be performed according to a UV map In the case where the camera is static, can be the same xy coordinate grid for all frames. If the camera is in motion, camera tracking can be used to determine the background. For example, if a homography transformation estimated from camera tracking is available, can be the result of the homography transformation xy coordinate grid of frame t.

[0056] Example embodiments of the present disclosure can utilize a single human texture atlas T i (204) and a frame-by-frame object map (e.g., UV coordinate map) (202) to represent each person in the video, which maps pixels (e.g., each pixel) in the human region in frame I t . Example embodiments of the present disclosure can use the parameterization of the SMPL model, which can be estimated from input images using existing methods (e.g., DensePose). This representation can provide a unified parameterization of the person over time and a convenient model for appearance and texture. Additionally, a UV map of the background can be placed behind the UV map of each person to provide the context of the background to the renderer network. This can allow the neural renderer to learn the effects that a person can have on their environment, such as shadows or trampoline deformation.

[0057] To help the network converge, example embodiments of the present disclosure can initially train the neural network to predict an alpha mask that matches the UV ternary map. For example, as FIG. 5As depicted, the gray pixels represent the dilated region around the person's UV, for which the weight of the matching loss can be reduced. This can allow the network flexibility to match body variations, hair, and clothes that are not represented by the UV map. Thus, the final model prediction can be a more accurate segmentation mask, which can include relevant regions such as shadow or other trace effects.

[0058] In some embodiments, example implementations of the present disclosure can replace the classical RGB texture map with a learnable high-dimensional texture map that can encode more powerful and richer appearance information. The depth texture map can then be encoded into RGB values using a neural rendering network. More specifically, the proposed neural rendering network can receive the sampled depth texture map corresponding to the ith layer at time instance t as input. Then, the neural rendering network can output a time-varying color image and an opacity map for the layer, respectively.

[0059] Example implementations of the present disclosure can control the mapping between the person in the video and the output layer by controlling the input sampled texture (204). For example, to associate layer i with person i in the video, example implementations of the present disclosure can resample T i (texture map of person i) by resampler 210 using to allow the model to also capture the correlation between the person and the motion in the scene related to it, example implementations of the present disclosure can synthesize the sampled texture on the sampled background texture. This synthesized depth texture can form the input to the proposed neural rendering network for layer i.

[0060] Each resampled texture layer and the background resampled texture can be fed to the neural rendering network in separate feed-forward channels (e.g., as shown in FIG. 2 By controlling and unmixing the input to the neural rendering network, example implementations of the present disclosure can indirectly guide the decomposition. The decomposed layers can then be reconstructed (e.g., using Equation 2) with the desired layer manipulation.

[0061] The neural renderer can be able to substantially "correct" any imperfections in the input geometry to accurately reconstruct the original video. These imperfections can include any trace effects that can not be represented by the input UV map, such as shadows, reflections, loose clothing or hair, etc., or any other suitable trace effects. To accurately reconstruct these scene elements, the neural renderer can learn to relate them to the input geometry. The hierarchical nature of the neural renderer can be beneficial for this aspect. For example, if the entire scene geometry can be given to the neural renderer as input once, it can learn spurious correlations, or learn to associate a certain element (e.g. a shadow) with a particular configuration of a person, rather than the individual person alone. However, in some cases, by splitting the input into layers and feeding each of them separately, example embodiments of the present disclosure can effectively limit the observation capabilities of the network, forcing it to place each missing scene element in the layer with the strongest signal. As an example, the strongest signal related to a person's shadow should be the person that can be casting it.

[0062] Additionally and / or alternatively, in some embodiments, the system according to example aspects of the present disclosure can support manual editing of the initial masks. For example, in some difficult cases, assigning effects to layers can be incorrect. Thus, a user can encourage better solutions by manually editing the masks (e.g., expanding them).

[0063] Example training

[0064] Example embodiments of the present disclosure can include learning the optimal parameters θ of the neural renderer and the set of latent textures

[0065] One property of the decomposition of the learning can be to allow for accurate reconstruction of the original video. Formally, let:

[0066]

[0067] where, is the output layer for frame t, and o t is the synthesis order.

[0068] In some cases, the individual reconstruction loss can not be sufficient to make the optimization converge from random initialization. Thus, some example embodiments of the present disclosure can guide the training by encouraging the learned alpha maps to match the person segment associated with layer i. To this end, example embodiments of the present disclosure can apply the following loss:

[0069]

[0070] where, can be the UV map The derived tri-graph (see FIG. 2 ), and D() can be a distance metric.

[0071] In some cases, E mask Can be used only for guiding the model and can be turned off as optimization progresses.

[0072] Example embodiments of the present disclosure can further apply a regularization loss to the opacity To encourage sparsity in space. This loss can be defined as a mix of L1 and an approximation of L0:

[0073]

[0074] where Φ0(x) = 2 · Sigmoid(5x) - 1 smoothly penalizes non-zero values of the alpha map, while γ controls the relative weight between terms.

[0075] The proposed total loss can then be given by:

[0076] E total = E recon + γ m E mask + βE reg , (6)

[0077] where γ m and β control the relative weight of terms.

[0078] Example high-resolution refinement and detail transfer

[0079] Example embodiments of the present disclosure can include a multi-scale approach. This approach can include first training the proposed model on a down-sampled version of the original video using Equation 6. Then, example embodiments of the present disclosure can up-sample the results to the original resolution using a separate lightweight refinement network consisting of several residual blocks operating separately on each RGBA layer. The refinement network can receive as input the RGBA output of the neural renderer, the texture sampled by the UV input to the neural renderer, and / or a bilinear up-sampling of the final feature maps output by the neural renderer before the RGBA output layer (e.g., to the desired output size) concatenated. The RGBA output of the refinement network can then be composited in the same way as the output of the neural renderer. In some embodiments, the up-sampling network can be trained using only the L1 reconstruction loss.

[0080] In some implementations, example implementations of the present disclosure can avoid the additional cost of training with perceptual and adversarial losses by directly transferring high-resolution details from the original video in the post-processing step. The residual between the neural renderer output and the video can define the details to be transferred. The amount of residual to transfer to each layer can be determined by the transmittance map

[0081]

[0082] where Comp α represents the alpha channel of the composite produced by the neural renderer. The final layer color can be defined by

[0083]

[0084] where Cnr can be the color produced by the neural renderer. Given this transfer, the up-sampling network can only need to refine the predicted alpha matte and produce reasonable colors in occluded regions where the true value high-frequency details are not available.

[0085] Example UV preprocessing

[0086] Video retiming effects can generally de-occlude people that are partially or even completely occluded in the original frames. In some implementations, such as in implementations where each person has a single texture map that can be learned jointly for the entire video, example implementations of the present disclosure can render the de-occluded content (e.g., as long as example implementations of the present disclosure can sample from it correctly). To this end, example implementations of the present disclosure can ensure that all UV maps represent the full body of each individual as learnable from the video frames, even in the presence of occlusions. In some embodiments, image inpainting and / or other estimation techniques can be used to fill in missing texture.

[0087] In some implementations, UV maps can be determined using direct estimation (e.g., DensePose) and / or keypoint estimation (e.g., AlphaPose). Direct estimation of UV maps from images using existing tools such as DensePose can not produce UV maps in occluded regions. Additionally, direct estimation can suffer from unstable errors in visible regions. In contrast, keypoint estimators such as AlphaPose can tend to be more robust in the presence of occlusions. Additionally, keypoints can be easily manually corrected when the estimator fails.

[0088] ​To produce complete, clean UV maps, example embodiments of the present disclosure can combine direct UV estimation and keypoint estimation. This can include training a neural network to predict UV maps from keypoints. To train the proposed keypoint-to-UV model, example embodiments of the present disclosure can use a dataset curated to contain only single-person video frames. Another example dataset includes approximately 10 minutes of filmed video of a single person performing various poses. Example embodiments of the present disclosure then generate approximately 20K keypoint-UV training examples by running AlphaPose and DensePose on the original frames.

[0089] To predict full-body UV maps in the presence of occlusions, example embodiments of the present disclosure can first estimate keypoints (e.g., using AlphaPose), track keypoints (e.g., using PoseFlow), and then manually clean up keypoints when estimation or tracking fails. The keypoint-to-UV network can then process these keypoints to generate UV maps of the occluded person.

[0090] Removing the layer decomposition component of the proposed model results in a lack of generalization capabilities. While both models can reconstruct the original video, the non-hierarchical model can struggle to disentangle people (e.g., as evidenced by the failure of the segmentation of the girl wearing a white dress in the Edit (Freeze) column) and / or disentangle their relevance (e.g., as evidenced by the incomplete removal of the avatar when people are removed in the Edit (Remove) column). FIG. 8 FIG. 8

[0091] Example user interaction There are several points in the proposed pipeline where users can apply edits to control or improve layer decomposition, including, for example, assigning people to layers, ordering of layers over time, keypoint cleaning, and mask editing.

[0092] Multiple people can sometimes be difficult to separate with a consistent depth ordering (e.g., the dancing pair in the dancehall sequence depicted in In these cases, the user can choose to collect multiple people onto the same layer. People on the same layer can not be independently retimed. Interactions between them (collisions, self-shadows, etc.) can be faithfully reconstructed.

[0093] FIG. 3 In cases where the depth ordering of people can be well-defined but changes over time (e.g., in the trampoline sequence depicted in In some embodiments, the neural renderer and neural textures are shared across layers, so changing the layer order does not change the training scheme.

[0094] FIG. 3 In some embodiments, the neural renderer and neural textures are shared across layers, so changing the layer order does not change the training scheme.

[0095] ​In some cases, manual clean-up can be necessary to correct errors in the predicted key points. For example, some existing key point detectors can still fail due to motion blur, occlusions, noise, etc. Example embodiments of the present disclosure can include manually cleaned-up key points in particularly blurry or occluded areas (e.g., using a rotoscoping tool).

[0096] For some particularly difficult videos, it can be necessary to refine the initial mask m t by manual editing to encourage a particular decomposition. Since the initial mask is refined by optimization, this editing can be loose.

[0097] Example camera tracking

[0098] When the input video contains a moving camera, example embodiments of the present disclosure can include first estimating the camera motion using a feature-based tracking algorithm. For example, example embodiments of the present disclosure can model the camera motion at each frame using homography transformations, which example embodiments of the present disclosure can robustly estimate from matching ORB features between frames. When stabilizing small camera motion or natural hand-shake, example embodiments of the present disclosure can compute homographies between each frame and a single reference frame (which can work better than tracking the camera over time), then use them to stabilize the input video. When the video contains large camera motion or substantial translation, example embodiments of the present disclosure can estimate homographies between consecutive frames over time, use them to record all frames relative to a common coordinate system, then apply that coordinate system transformation to the background UV map to preserve the original camera motion. For example, re-timing a layer from frame t to frame can be achieved by transforming the layer to the common coordinate system using the transformation at t, then applying the inverse transformation at to the layer.

[0099] Example implementation details

[0100] Details for generating one or more example implementations of the described results are presented below. These details are provided as one example implementation only and do not necessarily define any limitations of the implementation according to the example aspects of the present disclosure. The example implementations of the present disclosure first train a neural renderer for 2K epochs on an input video downsampled along the smaller dimension (352x256 for the disco sequence, 448x256 for all other videos) to size 256. The example implementations of the present disclosure apply augmentation for 75% of the time by rescaling the size of the frame with a uniform random scaling factor in the range [1, 1.25] and then performing random cropping. Next, the example implementations of the present disclosure freeze the existing parameters and train an additional upsampling network to augment the low resolution results. This lightweight network can be trained for 500 epochs with an L1 reconstruction loss without any data augmentation. The final output of the upsampling network has dimensions that are twice the size of the low resolution output (704x512 for the disco sequence, 896x512 for all other videos). The example implementations of the present disclosure implement the proposed networks in PyTorch and use an Adam optimizer with a learning rate of le-3. Depending on the video length and the number of prediction layers, the total training time on 4 NVIDIA Tesla P100 GPUs is between 6 and 12 hours.

[0101] Example experimental results

[0102] The example implementations of the present disclosure test the proposed method on many real-world videos, many of which are captured by handheld mobile phone cameras. The videos depict a wide range of human motion (e.g., dancing, jumping, running) with many people moving simultaneously and across complex natural environments. Representative frames from these videos are shown in FIG. 9 .

[0103] Example layer decomposition

[0104] Several of the proposed layer decompositions are visualized in FIG. 3 . For some videos (e.g., the disco scene depicted in FIG. 3 and the FIG. 3In contrast to the previous examples, the proposed model successfully disentangles the people into layers for all videos. These layers capture fine details such as loose hair and clothing (e.g., the white dress of the foreground dancer in the dancehall scene), or objects attached to people (e.g., the child's balloon in the bounce scene). This can be achieved according to example aspects of the present disclosure despite initializing the proposed model with coarse person UV maps that do not explicitly represent these elements. This ability of the proposed model to accurately segment the person regions also enables FIG. 3 to be shown more closely in

[0105] Furthermore, the predicted layers can be people in the scene that have other non-obvious visual changes related to them - e.g., shadows cast on the floor by different dancers (e.g., in the dancehall scene of FIG. 3 ), complex mirroring of two people crossing each other (e.g., in the mirroring scene of FIG. 3 ), surface deformations (e.g., in the trampoline scene of FIG. 3 ), or water splashes caused by the motion of people (e.g., in the splash pool scene of Retiming and editing results example ).

[0106] FIGS. 7-10

[0107] After obtaining the disentangled layers, example embodiments of the present disclosure can produce various retiming and editing effects via simple operations on the layers. Example embodiments of the present disclosure show several such retiming results in FIG. 9 .

[0108] FIG. 9 Example frames of videos depicting multiple people performing similar actions (e.g., in the splash scene, the bounce scene, and the kids running scene) but their motions are not synchronized are depicted. For example, in the splash scene, the children jump into the pool one after another. In the bounce scene, the periodic motions of the children are independent. In both examples, example embodiments of the present disclosure retimes the people to align their motions. For the splash, a few manually defined alignment points are sufficient to align the children's jumps. In the bounce, due to the periodic nature of the motion, the alignment can be performed automatically using a related optimization warp (e.g., a variation of dynamic time warping).

[0109] As shown, the proposed layer disentanglement includes not only the individuals but also the trace effects related to the people, so as example embodiments of the present disclosure can retimes the layers of the people, all the related trace effects (e.g., water splashes when they hit the water, trampoline deformations when they bounce on it) automatically follow them in order to produce retimed videos that appear natural.

[0110] Example implementations of the present disclosure are also able to use the proposed method to "freeze" a person at a certain point in time while letting the other people move as in the original video. This can create an effect that encourages the viewer to focus their attention on the moving person while ignoring the remaining motion in the original video. For example, in the dance hall scene depicted in FIG. 9 , example implementations of the present disclosure freeze the dancing couple in the back throughout the video while the couple in the front remains moving. Here, the shadows and reflections on the floor also move realistically with the moving couple while the shadows of the background couple remain static. In addition, the disoccluded regions of the back couple are realistically rendered.

[0111] For example, in the children running scene depicted in FIG. 4 , example implementations of the present disclosure show how the proposed model can scale to multiple layers to produce complex retiming effects involving many people. Example implementations of the present disclosure retimes the original video where the children cross a faint finish line on the ground at different times to produce a "finish line photography" video where all the children cross the finish line together. Example implementations of the present disclosure can achieve this result, for example, by slowing down the layers of children running out of the screen. Even though this sequence involves many individuals, the proposed model is able to obtain clean masks for each child. In addition, when processing significant motion blur present in the input video, it is able to realistically inpaint the disoccluded people and large disoccluded regions in the background.

[0112] As mentioned, in addition to retiming effects, the proposed method can also support easy removal of people in a video due to the layered representation of example implementations of the present disclosure. For example, FIG. 8 and FIG. 4 depict a scene showing the removal of people in a video containing two people walking through a path in front of a window. Here, for example, the proposed model is able to perform several non-obvious tasks. The model is able to fully disocclude the person walking in the back. For example, as shown in FIG. 10 , the model is able to generate object maps (UV maps) that capture the portions of the depicted person walking in the back. The model is able to properly associate each person with their reflection and shadow. When the two reflections overlap, the model is able to unmix the two reflections even though none of these elements are explicitly represented by the model. Traditional video editing pipelines can have difficulty achieving these results. For example, the reflections can have to be tracked with the people to perform proper removal. Additionally, the person in the back can have to be manually inpainted at the point where they are disoccluded by the person in the front. Thus, one advantage of the proposed method can be that example implementations of the present disclosure can achieve the same results with significantly less manual work by simply inputting the UVs of each person in separate layers and "turning on and off" those layers.

[0113] FIG. 10 Examples are depicted in accordance with aspects of the present disclosure, where the original camera motion can be preserved in the proposed retiming results. For example, as depicted, the girl on the left can be duplicated with a short temporal offset between her copies. The girl in blue on the right can be frozen from the previous focus region of the camera, while the camera can pan as in the original video. Example layered vs. non-layered neural rendering

[0114] FIG. 8

[0115] Another example aspect of the present disclosure can include a non-hierarchical model (e.g., a model that takes as input a single sampled texture map representing all people in the frame as well as the background and directly outputs an RGB reconstruction of the frame). For example, a non-hierarchical model can be simpler than a hierarchical model. An example non-hierarchical model is compared to an example hierarchical model. This baseline follows a deferred neural rendering approach.

[0116] FIG. 8 A comparison between the reconstruction and editing results produced by the example hierarchical model and the example non-hierarchical model is depicted. It can be seen that the non-hierarchical model can reconstruct the original frame quite well, even with missing information and noise in the UV map. However, when performing edits, the non-hierarchical model can fail to generalize to new syntheses of people. This is evidenced, for example, by the significant visual artifacts in the dancehall editing example depicted in Correction For example, to produce the editing effects, the non-hierarchical model can need to generalize to new UV syntheses of people in configurations that it has never seen during training. This can cause challenges in producing photorealistic-looking results based on L1 reconstruction loss alone. The proposed hierarchical approach can avoid this generalization problem because edits can be performed as post-processing on the predicted layers (e.g., the same outputs produced during training). Moreover, when the input to the model can be a synthesis of all people in the frame rather than separate UVs, the model can more easily reconstruct the original frame without having to capture meaningful correlations. For example, it can not be necessary to disentangle separately moving parts of the scene. This can be exemplified in the mirror editing results, where the non-hierarchical model struggles to learn the correct relationships between different people and their mirrors.

[0117] Another benefit of the hierarchical model can be that example implementations of the present disclosure can grab high-resolution details from the input video. Non-hierarchical neural rendering models can not support this and, in addition, can require extensive training time and adversarial losses to achieve comparable visual quality.

[0118] FIG. 6

[0119] ​While the proposed layers can generally be successfully decomposed, in some particularly difficult cases, manual correction can be required. For example, in some cases, artifacts can be present in the decomposed layers. One example embodiment of an artifact observed by the present disclosure can be in the background layer in the splashing scene depicted in FIG. 6 FIG. 6. As depicted in Example model architecture , artifacts can be seen in areas of the model where real values appearance is never observed (e.g., those areas are always occluded by people). In these cases, an image inpainting network (e.g., a separate image inpainting network) can be utilized to correct the artifacts. Advantageously, backgrounds can generally be easier to image inpaint than similar human deformable objects. The proposed model can address these deformable objects to provide full image inpainting support. For example, an image inpainting step can be added to the layers (e.g., using a neural network or other image inpainting system).

[0120] Additionally, some time-varying aspects of the background can become associated with the layers. For example, in the dance hall scene, purple light appears above the head of the background girl as the light color changes throughout the video. Thus, as a signal that changes over time, the light is included in one of the foreground layers. In this case, the purple light can be in the layer of the foreground couple in front, and then can be rendered on top of the frozen background couple. Such artifacts can be avoided if some flexibility is given in the representation of the background. For example, the background can be allowed to change over time. The representation capability of the time-varying background must be balanced with the footprint effect. For example, example embodiments of the present disclosure can desirably still be able to represent the footprint effect caused by moving people, such as clothing and shadows.

[0121] Layer

[0122] Example aspects of the present disclosure can utilize a mask error E mask . For example, the mask can be a ternary map with values in [0, 0.5, 1] where the uncertain region is produced by a morphological dilation of the binary UV mask. For a ternary map m, let b0be the binary mask of pixels where m = 0, where b 0.5 and b1are defined identically. In some embodiments, the foreground can be weighted up while the uncertain region can be weighted down to avoid bad local minima. The final mask error function is:

[0123] D(m, a) = 5||b1h(l.0 - a) ||1+ 0.2||b 0.5 ha a ||1+||b0h a ||1 (9)

[0124] where h is the Hadamard (element-wise) product.

[0125] An example training schedule defined by a training epoch e is given below:

[0126]

[0127] The schedule can include a large initial loss on the masking term to force the optimization towards a reasonable solution. Then, the schedule can relax the loss to allow the optimization to introduce effects that are not present in the mask.

[0128] An example network architecture is given below. In some or all networks, padding is mode “same”. As used herein, “bn” refers to batch normalization, “in” refers to instance normalization, “convt” refers to convolution transpose, “leaky” refers to a leaky RELU with a slope of -0.2, “skipk” refers to a skip connection to layer k, and “resblock” denotes a residual block consisting of a conv, instance norm, RELU, conv, instance norm.

[0129] An example neural renderer architecture is given below:

[0130] Output channels Stride Activation 4x4conv leaky 64 2 4x4conv, bn leaky 128 2 4x4conv, bn leaky 256 2 4x4conv, bn leaky 256 2 4x4conv, bn leaky 256 2 4x4conv, bn leaky 256 1 4x4conv, bn leaky 256 1 skip5, 4x4conv t, bn relu 256 2 skip4, 4x4conv t, bn relu 256 2 skip3, 4x4conv t, bn relu 128 2 skip2, 4x4conv t, bn relu 64 2 skip1, 4x4conv t, bn relu 64 2 4x4conv tanh 4 1 Layer

[0131] An example up-sampling network architecture is given below. The up-sampling network can predict a residual image that is added to the bilinear up-sampled composite neural renderer output. The example up-sampling network architecture is as follows:

[0132] Output channels Stride Activation 3x3conv, in relu 64 1 3x3resblock relu 64 1 3x3resblock relu 64 1 3x3resblock relu 64 1 3x3conv none 4 1 Example devices and systems

[0133] The keypoint-to-UV network architecture can be the same as the neural renderer architecture, with the final layer replaced by two heads, including a final convolutional layer with 25 output channels to predict body part and background classifications and a convolutional layer with 48 output channels to regress UV coordinates for each of the 24 body parts. The body part classifier can be trained with a cross-entropy loss and the predicted UV coordinates can be trained with an LI loss. The regression loss on the UV coordinates can be considered for a body part if the pixel is within the particular part, as defined by the ground truth UV map.

[0134] FIG. 11A

[0135] FIG. 2 A block diagram of an example computing system 100 that performs layered neural rendering in accordance with example embodiments of the present disclosure is depicted. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled by a network 180.

[0136] The user computing device 102 can be any type of computing device such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0137] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as

[0138] In some implementations, the user computing device 102 can store or include one or more layer renderer models 120. For example, the layer renderer models 120 can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and / or linear models. The neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. See, e.g., FIGS. 12-13 and FIG. 2 Example layer renderer models 120 are discussed.

[0139] In some implementations, the one or more layer renderer models 120 can be received from the server computing system 130 over the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 can implement multiple parallel instances of a single layer renderer model 120 (e.g., to perform parallel layered neural rendering across multiple instances of the layer renderer model 120).

[0140] More specifically, the layer renderer model 120 is capable of receiving video data as input and decomposing the video data into one or more layers, each layer being associated with and primarily containing video data depicting one or more specified objects. For example, the layer renderer model is capable of generating one or more object maps (e.g., by the object map generation model) and packing texture data to a depth texture map based on the object maps. Additionally, the layer renderer model is capable of associating a trail effect in the video data with an object that caused or otherwise affected the trail effect and including the trail effect in a layer depicting the respective object.

[0141] Additionally or alternatively, one or more layer renderer models 140 can be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the layer renderer models 140 can be implemented by the server computing system 140 as part of a web service (e.g., a layered neural rendering service). Thus, one or more models 120 can be stored and implemented at the user computing device 102, and / or one or more models 140 can be stored and implemented at the server computing system 130.

[0142] The user computing device 102 can also include one or more user input components 122 that receive user input. For example, the user input components 122 can be touch-sensitive components (e.g., a touch-sensitive display screen or a touchpad) that are sensitive to touch by a user input object (e.g., a finger or a stylus). The touch-sensitive components can be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0143] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as

[0144] In some implementations, the server computing system 130 includes or is otherwise implementable by one or more server computing devices. In cases where the server computing system 130 includes multiple server computing devices, such server computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0145] As noted above, the server computing system 130 can store or otherwise include one or more machine-learned layer renderer models 140. For example, the models 140 can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed-forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. See, e.g., Goodfellow, et al., Deep Learning, MIT Press, 2016, which is incorporated by reference herein in its entirety. FIGS. 12-13 and FIG. 11A Example models 140 are discussed.

[0146] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 via interaction with a training computing system 150 that can be communicatively coupled with the network 180. The training computing system 150 can be separate from the server computing system 130 or can be part of the server computing system 130.

[0147] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as

[0148] The training computing system 150 can include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over multiple training iterations.

[0149] In some implementations, performing backpropagation of errors can include performing truncated backpropagation through time. The model trainer 160 can perform various generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization capabilities of the model being trained.

[0150] In particular, the model trainer 160 can train the layer renderer model 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, an external dataset of keypoint and UV map pairs. Additionally and / or alternatively, the training data 162 can include video data. As one example, the training data 162 for the object graph generation model can include a dataset curated to contain only single-person video frames. Another example dataset includes approximately 10 minutes of filmed video of a single person performing a wide variety of poses.

[0151] In some implementations, the training examples can be provided by the user computing device 102 if the user has provided consent. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some cases, this process can be referred to as personalizing the model.

[0152] The model trainer 160 includes computer logic in the form of software or firmware instructions. The software or firmware instructions configure the general purpose processor to operate as an inference engine. The software or firmware instructions can be stored in the memory 164 and executed by the general purpose processor 162.

[0153] Network 180 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or a combination thereof, and can include any number of wired or wireless links. Typically, communication over network 180 can be carried over any type of wired and / or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0154] FIG. 11B The illustration shows an example computing system that can be used to implement this disclosure. Other computing systems can also be used. For example, in some embodiments, user computing device 102 can include a model trainer 160 and a training dataset 162. In such embodiments, model 120 can be trained and used locally at user computing device 102. In some such embodiments, user computing device 102 can implement model trainer 160 to personalize model 120 based on user-specific data.

[0155] FIG. 11B A block diagram is depicted of an example computing device 10 that can perform layered neural rendering in addition to other functions according to an example embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0156] The computing device 10 includes multiple applications (e.g., applications 1 to N). Each application contains its own machine learning library and one or more machine learning models. For example, each application can include machine learning models. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0157] like FIG. 11C As shown, each application can communicate with multiple other components of the computing device—such as one or more sensors, a scene manager, a device status component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application can be application-specific.

[0158] FIG. 11C A block diagram depicts an example computing device 50 implemented according to an example embodiment of the present disclosure. The computing device 50 can be a user computing device or a server computing device.

[0159] The computing device 50 includes a plurality of applications (e.g., applications 1 through N). Each application is capable of communicating with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like. In some implementations, each application is capable of communicating with the central intelligence layer (and the model(s) stored therein) using an API (e.g., a common API across all applications).

[0160] The central intelligence layer includes a plurality of machine-learned models. For example, as shown, a respective machine-learned model (e.g., model ) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model ) for all applications. In some implementations, the central intelligence layer can be included within or otherwise implemented by an operating system of the computing device 50. FIG. 11C

[0161] The central intelligence layer is capable of communicating with a central device data layer. The central device data layer can be a centralized data store for the computing device 50. As shown, the central device data layer can communicate with a plurality of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. Example model arrangement In some implementations, the central device data layer is capable of communicating with each device component using an API (e.g., a private API).

[0162] FIG. 12

[0163] FIG. 13 A block diagram of an example layer renderer system 1200 according to example embodiments of the present disclosure is depicted. In some implementations, the layer renderer model 1200 can be trained to receive a set of input data 1204 describing video data and, as a result of receiving the input data 1204, provide output data 1206 including decomposed video layers. Thus, in some implementations, the layer renderer system 1200 can include a layer renderer model 1202, which can be operable to decompose video data into one or more layers, including, for example, a background layer and / or one or more object layers.

[0164] FIG. 12 A block diagram of an example layer renderer system 300 according to example embodiments of the present disclosure is depicted. The layer renderer system 300 can be similar to the layer renderer system 200, except that the layer renderer model 302 is capable of receiving a set of input data 304 describing video data and, as a result of receiving the input data 304, providing output data 306 including decomposed video layers. Example method ​The layer renderer system 1200 can be configured to generate a layer 1206 based on the input data 1204 and the object graph 304. For example, the layer renderer model 1202 can be configured to generate the layer 1206 based on the input data 1204 and the object graph 304. As one example, the layer renderer model 1202 can be trained (e.g., on a training dataset of layers and corresponding video data) to generate the layer 1206 based on the input data 1204 and the object graph 304. The layer 1206 can be generated by identifying a key point in the video data and determining a layer based on the key point. The layer 1206 can be provided as an output to the layer renderer system 1200.

[0165] FIG. 14

[0166] FIG. 6 A flow diagram depicting an example method performed in accordance with example embodiments of the present disclosure is depicted. Although the method 600 is depicted as a series of acts, it is contemplated that the method 600 can be implemented in various ways, such as with one or more of the layer renderer systems 300, 1200. Additional disclosure The method of the present disclosure is not limited to the particular illustrated order or arrangement of steps, but can be practiced in various ways, including with various orders, arrangements, combinations, and / or eliminations of the steps of the method 600 without departing from the scope of the present disclosure.

[0167] At 602, a computing system comprising one or more computing devices can obtain video data. The video data can comprise a plurality of image frames depicting one or more objects. For example, the video data can be a video captured by a mobile device, a video camera, or any other suitable video capture device. The video data can be stored in any suitable manner. For example, the video data can be stored in a computer-readable memory in any suitable format, such as a digital file format (e.g., an.mp4 file format, a.wav file format, etc.).

[0168] At 604, the computing system can generate one or more object graphs, wherein each object graph of the one or more object graphs can describe a respective position of at least one object of the one or more objects within an image frame. For example, the one or more object graphs can comprise UV maps. The object graph(s) can indicate an overall shape of an object, but can omit trace effects (e.g., flowing clothing, particles or other environmental elements affected by the object, shadows, reflections, etc.). In some embodiments, for example, generating the one or more object graphs can comprise identifying, by the computing system, one or more key points, and obtaining, by the computing system, one or more UV maps based on the key points. The object graphs can be identified based at least in part on the video data (e.g., manually and / or by the computing system).

[0169] At 606, the computing system can input the image frame and the one or more object maps into a machine-learned layer renderer model. At 608, the computing system can receive, as output from the machine-learned layer renderer model, a background layer that accounts for a background of the video data and one or more object layers respectively associated with one of the one or more object maps. For example, the object layers can account for the one or more objects and / or the trail effects produced by the one or more objects interacting with the environment.

[0170] ​

[0171] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, and actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0172] While the subject matter has been described in detail with respect to various specific embodiments of the subject matter, each example is presented exclusively by way of explanation, and not by way of limitation. Those skilled in the art can readily produce alterations, modifications, and equivalents to such embodiments, after obtaining an understanding of the foregoing descriptions. Therefore, the disclosure is not limited to the embodiments described herein, but includes all modifications equivalent within the scope of the subject matter. For example, a feature shown or described as part of one embodiment can be used with another embodiment to produce a yet further embodiment. Thus, the disclosure can be intended to cover such alterations, modifications, and equivalents.

Claims

1. A computer-implemented method for decomposing a video into a plurality of layers that can be recombined with modified relative timing, the computer-implemented method comprising: obtaining, by a computing system comprising one or more computing devices, video data comprising a plurality of image frames depicting one or more objects; and for each image frame of the plurality of image frames: generating, by the computing system, one or more object maps, wherein each object map of the one or more object maps describes a respective position of at least one object of the one or more objects within the image frame; inputting, by the computing system, the image frame and the one or more object maps into a machine-learned layer renderer model; and receiving, by the computing system from the machine-learned layer renderer model, a background layer that accounts for a background of the video data and one or more object layers respectively associated with one object map of the one or more object maps; wherein, for each image frame, each object layer of the one or more object layers comprises image data that accounts for the at least one object and one or more artifacts attributable at least in part to the at least one object, such that the one or more object layers and the background layer can be recombined with modified relative timing.

2. The computer-implemented method of claim 1, wherein, inputting, by the computing system, the image frame and the one or more object maps into the machine-learned layer renderer model comprises iteratively and individually inputting each object map of the one or more object maps into the machine-learned layer renderer model and iteratively and individually receiving, by the computing system from the machine-learned layer renderer model, each object layer of the one or more object layers corresponding to the one or more object maps.

3. The computer-implemented method of claim 1, wherein, the background layer and the one or more object layers comprise one or more color channels and an opacity mask.

4. The computer-implemented method of claim 1, wherein, the machine-learned layer renderer model comprises a neural network.

5. The computer-implemented method of claim 1, wherein, the machine-learned layer renderer model has been trained based at least in part on a reconstruction loss, a masking loss, and a regularization loss.

6. The computer-implemented method of claim 5, wherein, the training is performed on a downsampled video and then upsampled.

7. The computer-implemented method of claim 1, wherein, the one or more object maps comprise one or more texture maps.

8. The computer-implemented method of claim 1, wherein, the one or more object maps comprise one or more resampled texture maps.

9. The computer-implemented method of claim 8, wherein, obtaining, by the computing system, one or more object maps comprises: obtaining, by the computing system, one or more UV maps, each UV map of the one or more UV maps indicating the at least one object of the one or more objects depicted within the plurality of image frames; obtaining, by the computing system, a background depth texture map and one or more object depth texture maps; and resampling, by the computing system, the one or more object depth texture maps based at least in part on the one or more UV maps.

10. The computer-implemented method of claim 9, wherein, generating, by the computing system, the one or more UV maps comprises: identifying, by the computing system, one or more keypoints; and obtaining, by the computing system, one or more UV maps based on the one or more keypoints.

11. The computer-implemented method of any of claims 1-10, further comprising: transmitting, by the computing system, high resolution details of the video data in post-processing steps subsequent to receiving the background layer and the one or more object layers.

12. A computing system configured to decompose video data into multiple layers, the computing system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: obtaining video data comprising a plurality of image frames depicting one or more objects; and for each image frame of the plurality of image frames: generating one or more object maps, wherein each object map of the one or more object maps describes a respective position of at least one object of the one or more objects within the image frame; inputting the image frame and the one or more object maps into a machine-learned layer renderer model; and receiving, from the machine-learned layer renderer model, a background layer illustrative of a background of the video data and one or more object layers respectively associated with one object map of the one or more object maps; wherein, for each image frame, each object layer of the one or more object layers comprises image data illustrative of the at least one object and one or more artifacts attributable at least in part to the at least one object, such that the one or more object layers and the background layer can be recombined with modified relative timing.

13. The computing system of claim 12, wherein, inputting the image frame and the one or more object maps into the machine-learned layer renderer model comprises iteratively and individually inputting each object map of the one or more object maps into the machine-learned layer renderer model and iteratively and individually receiving, by the computing system from the machine-learned layer renderer model, each object layer of the one or more object layers corresponding to the one or more object maps.

14. The computing system of claim 12, wherein, the background layer and the one or more object layers comprise one or more color channels and an opacity mask.

15. The computing system of claim 12, wherein, the machine-learned layer renderer model comprises a neural network.

16. The computing system of claim 12, wherein, the machine-learned layer renderer model has been trained based at least in part on a reconstruction loss, a masking loss, and a regularization loss.

17. The computing system of claim 16, wherein, the training is performed on down-sampled video and then up-sampled.

18. The computing system of claim 12, wherein, the one or more object maps comprise one or more texture maps.

19. The computing system of claim 12, wherein, the one or more object maps comprise one or more re-sampled texture maps.

20. The computing system of claim 12, wherein, obtaining one or more object maps comprises: obtaining one or more UV maps, each UV map of the one or more UV maps indicating the at least one object of the one or more objects depicted within the plurality of image frames; obtaining a background depth texture map and one or more object depth texture maps; and re-sampling the one or more object depth texture maps based at least in part on the one or more UV maps.

21. The computing system of claim 20, wherein, obtaining the one or more UV maps comprises: identifying one or more key points; and generating one or more UV maps based on the one or more key points.

22. The computing system of any one of claims 12 to 21, wherein, The operations further include: transmitting high resolution details of the video data in a post-processing step subsequent to receiving the background layer and the one or more object layers.

Images