Calculating image of controllable dynamic scene

Abstract

To compute an image of a dynamic 3D scene comprising a 3D object, a description of a deformation of the 3D object is received, the description comprising a cage of primitive 3D elements and associated animation data from a physical engine or an articulated object model. For a pixel of the image, the method calculates a ray animated according to the animation data from a virtual camera through the pixel into the cage, and calculates a plurality of samples on the ray. Each sample is a 3D position and view direction in one of the 3D elements. The method calculates a transformation of the sample into a canonical cage. For each transformed sample, the method queries a learned radiation field parameterization of the 3D scene to obtain color values and opacity values. A volume rendering method is applied to the color values and the opacity values to produce pixel values of the image.

Description

Background Art

[0001] A dynamic scene is an environment in which one or more objects are moving; in contrast to a static scene in which all objects are stationary. An example of a dynamic scene is a person's face moving as the person speaks. Another example of a dynamic scene is a spinning propeller of an aircraft. Another example of a dynamic scene is a standing person moving his arms.

[0002] In conventional computer graphics, computing synthetic images of dynamic scenes is a complex task because a rigged three-dimensional (3D) model of the scene and its dynamics is required. Obtaining such a rigged 3D model is complex and time-consuming, and involves manual work.

[0003] Synthetic images of dynamic scenes are used for various purposes, such as computer games, movies, video communications, etc.

[0004] The embodiments described below are not limited to implementations that solve any or all disadvantages of known apparatuses for computing synthetic images of dynamic scenes. Summary of the invention

[0005] A simplified summary of the disclosure is given below in order to provide the reader with a basic understanding. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Its sole purpose is to present a selection of concepts disclosed herein in a simplified form as a prelude to a more detailed description that is presented later.

[0006] In various examples, there are ways to compute images of dynamic scenes in a controllable manner, so that a user or an automated process can easily control how to animate the dynamic scenes. Optionally, the images are computed in real time (such as at 30 frames per second or more) and are realistic, that is, the images have properties that roughly match the properties of empirical images and / or videos.

[0007] In various examples, there is a computer-implemented method for computing an image of a dynamic 3D scene including 3D objects. The method includes receiving a description of a deformation of the 3D object, the description including a cage of primitive 3D elements and associated animation data from a physics engine or an articulated object model. For a pixel of the image, the method computes a ray from a virtual camera through the pixel into the cage, animated according to the animation data, and computes a plurality of samples on the ray. Each sample is a 3D position and a view direction in one of the 3D elements. The method computes a transformation of the sample into a canonical version of the cage to produce a transformed sample. For each transformed sample, the method queries a learned radiation field parameterization of the 3D scene to obtain a color value and an opacity value. A volume rendering method is applied to the color value and the opacity value to produce a pixel value of the image.

[0008] Many of the attendant features will be more readily appreciated as the same becomes better understood by reference to the following detailed description considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The present specification will be better understood by reading the following detailed description in conjunction with the accompanying drawings, in which:

[0010] Figure 1 is a schematic diagram of an image animator for computing images of a controllable dynamic scene;

[0011] Figure 2 Shows the use of Figure 1 The image animator calculates the deformation depiction and three images of the human head;

[0012] Figure 3 Shows the chair and uses Figure 1 The image animator calculates the image of the broken chair;

[0013] Figure 4 Is Figure 1 A flowchart of an example method performed by an image animator;

[0014] Figure 5 is a schematic diagram of a ray in a deformed cage, the ray being transformed to a canonical cage, a volume lookup, and a volume rendering;

[0015] Figure 6 is a flow chart of the sampling method;

[0016] Figure 7 is a flow chart of a method of computing an image depicting a person with their mouth open;

[0017] Figure 8is a flowchart of a method for training a machine learning model and computing a cache;

[0018] Fig. 9 An exemplary computing-based device in which embodiments of an animator for computing images of a controllable dynamic scene are implemented is illustrated.

[0019] In the drawings, like reference numerals are used to denote like parts throughout the accompanying figures. DETAILED DESCRIPTION

[0020] The detailed description provided below in conjunction with the accompanying drawings is intended as a description of this example, and is not intended to represent the only form in which this example is constructed or utilized. The description sets forth the functions of the example and the sequence of operations for constructing and operating the example. However, the same or equivalent functions and sequences may be implemented by different examples.

[0021] The techniques described in this article use radiation fields and volume rendering methods. The radiation field parameterization represents the radiation field (referred to as the field) as a function from a five-dimensional (5D) space to a four-dimensional (4D) space, where the value of the radiation is known for each pair of 3D points and 2D view directions in the field. The radiation value is composed of a color value and an opacity value. The radiation field parameterization can be a trained machine learning model, such as a neural network, a support vector machine, a random decision forest, or other machine learning model, which learns the association between the radiation value and a pair of 3D points and the view direction. In some cases, the radiation field parameterization is a cache of associations between radiation values ​​and 3D points, where the association is obtained from a trained machine learning model.

[0022] Volume rendering methods compute an image from the radiance field for a particular camera viewpoint by examining the radiance values ​​of points along the rays that form the image. Volume rendering software is well known and commercially available.

[0023] As mentioned above, composite images of dynamic scenes are used for various purposes, such as computer games, movies, video communications, telepresence, etc. However, it is difficult to generate composite images of dynamic scenes in a controllable manner; that is, it is difficult to easily and accurately control how the scene is animated. Precise control is desired for many applications, such as applications where a composite image of an avatar of a person in a video call is to accurately depict the facial expressions of a real person. Precise control is also desired for video game applications, where the image of a particular chair is to be broken in a realistic manner. These examples of video calls and video games are not intended to be limiting, but rather illustrate the use of the present technology. The technology can be used to capture any scene that is static or dynamic, such as an object, a vegetation environment, a person, or other scene.

[0024] Enrollment is another problem that arises when generating synthetic images of dynamic scenes. Enrollment is the situation where a radiation field parameterization is created for a specific 3D scene, such as a specific person or a specific chair. Some approaches to enrollment use a large number of training images that depict a specific 3D scene over time and from different viewpoints. Among other things, enrollment is time consuming and presents computationally heavy difficulties.

[0025] Being able to generate composite images of dynamic scenes in real time, such as during a video call where an avatar of the caller is to be created, is becoming increasingly important. However, real-time operation is difficult to achieve due to complex calculations and computational burden.

[0026] Generalization ability is an ongoing problem. Trained radiation field parameterizations are often difficult to generalize to facilitate computation of images of 3D scenes that differ from those used during training of the radiation field parameterizations.

[0027] Alternatives to using implicit deformation methods based on learned functions are “black-box” to content creators, they require large amounts of training data to generalize meaningfully, and they do not produce realistic extrapolations beyond the training data.

[0028] The present technology provides a precise way to control how images of dynamic scenes are animated. A user or an automated process can specify parameter values, such as volumetric blend shapes and skeleton values ​​of cages applied to primitive 3D elements. In this way, a user or an automated process can precisely control the deformation of 3D objects to be depicted in a composite image. In other examples, a user of the automated process can use animation data from a physics engine to precisely control the deformation of 3D objects to be depicted in a composite image. A blend shape is a mathematical function that changes the parameter values ​​of a parameterized 3D model when applied to the 3D model. In an example, where the 3D model is a 3D model of a person's head, there may be hundreds of blend shapes, each of which changes the 3D model based on facial expressions or identity characteristics.

[0029] In some examples, the present techniques reduce the registration burden. The registration burden is reduced by using a reduced amount of training images, such as training image frames from only one or only two moments in time.

[0030] In some examples, the present technique can operate in real time (such as at 30 frames per second or more). This is achieved by using optimization when computing the transformation of sample points to the canonical space used by the radiation field parameterization.

[0031] In some cases, the present technique operates with good generalization capabilities. By creating scenes animatable using parameters from a chosen facial model or physics engine, the technique is able to use the model dynamics from the facial model or physics engine to animate scenes beyond the training data in a physically meaningful way to generalize well.

[0032] Figure 1 1 is a schematic diagram of an image animator 100 for computing a composite image of a dynamic scene. In some cases, the image animator 100 is deployed as a web service. In some cases, the image animator 100 is deployed at a personal computer or other computing device that communicates with a head-mounted computer 114, such as a head-mounted display device. In some cases, the image animator 100 is deployed in a companion computing device to the head-mounted computer 114.

[0033] The image animator 100 includes a radiation field parameterization 102, at least one processor 104, a memory 106, and a volume renderer 108. In some cases, the radiation field parameterization 102 is a neural network, or a random decision forest, or a support vector machine, or other type of machine learning model. It has been trained to predict pairs of color values ​​and opacity values ​​and view directions for three-dimensional points in a canonical space of a dynamic scene, and more details about the training process are given later in this document. In some cases, the radiation field parameterization 102 is a cache of associations between three-dimensional points in a canonical space and color values ​​and opacity values.

[0034] The volume renderer 108 is a well-known computer graphics volume renderer that takes pairs of color and opacity values ​​for three-dimensional points along a ray and computes an output image 116 .

[0035] The image animator 100 is configured to receive queries from a client device such as a smartphone 122, a computer gaming device 110, a head mounted computer 114, a movie creation device 120, or other client device. The queries are sent from the client device to the image animator 100 via the communication network 124.

[0036] The query from the client device includes a specified viewpoint of a virtual camera, specified values ​​of intrinsic parameters of the virtual camera, and a deformation description 118. The composite image will be calculated by the image animator 100 as if it had been captured by the virtual camera. The deformation description describes the desired dynamic content of the scene in the output image 116.

[0037] The image animator 100 receives the query and, in response, generates a composite output image 116 which it sends to the client device. The client device uses the output image 116 for one of a variety of useful purposes, including, but not limited to: generating a virtual webcam stream, generating a video of a computer video game, generating a hologram for display by a mixed reality head mounted computing device, generating a movie. The image animator 100 is capable of computing a composite image of a dynamic 3D scene on demand for a particular specified desired dynamic content and a particular specified viewpoint. In the example, the dynamic scene is the face of a person speaking. The image animator 100 is capable of computing a composite image of the face from multiple viewpoints and with any specified dynamic content. Non-limiting examples of specified viewpoints and dynamic content are: plan view, eyes closed, face tilted upward, smiling; perspective view, eyes open, mouth open, angry expression. Note that because the machine learning used to create the radiation field parameterization 102 is capable of generalization, the image animator 100 is capable of computing a composite image for viewpoints and deformation descriptions that are not present in the training data used to train the radiation field parameterization 102. Reference is made below Figure 2 and Figure 3 Other examples of dynamic scenes are given and include generic objects such as chairs, cars, trees, and full human bodies. By using the deformation description, the dynamic scene content depicted in the generated synthetic image can be controlled. In some cases, the deformation description is obtained using a physics engine 126 so that a user or automated process can apply physical rules to break the 3D objects depicted in the synthetic output image 116 or apply other physical rules to depict animations such as bouncing, waving, swinging, dancing, rotating, spinning, or other animations. The deformation description can be created by applying physical simulations to a cage of 3D primitive elements using finite element methods to produce elastic deformation or breaking. In some cases, such as when a human avatar is created, the deformation description is obtained using a face or body tracker 124. By selecting viewpoints and intrinsic camera parameter values, the characteristics of the synthetic output image can be controlled.

[0038] The image animator operates in an unconventional manner to enable the generation of synthetic images of dynamic scenes in a controllable manner.Many alternative methods of using machine learning to generate synthetic images have little or no control over what is depicted in the generated synthetic image.

[0039] Image animator 100 improves the functionality of an underlying computing device by enabling the computation of composite images of dynamic scenes in a manner that is controllable by the content and viewpoint of the dynamic scene.

[0040] Alternatively or additionally, the functionality of the image animator 100 is at least partially performed by one or more hardware logic components. For example, but not limited to, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), graphics processing units (GPUs).

[0041] In other examples, the functionality of the image animator 100 is located at the client device, or shared between the client device and the cloud.

[0042] Figure 2 Shows the use of Figure 1 The image animator 100 calculates a deformation description 200 of a person's head and three images 204, 206, 208, each image showing the person's head animated in a different way, such as with the mouth open or closed. The deformation description 200 is a cage of primitive 3D elements, which are in Figure 2 In the example of , a tetrahedron is used, but in some examples other primitive 3D elements are used, such as a sphere or a cuboid. Figure 2 In the example of , a cage of tetrahedrons is extended from the surface mesh of a person's head so as to include a volume around the head that can be used to represent the person's hair and any headgear worn by the person. In the case of a generic object such as a chair, the volume around the object in the cage is useful because modeling the volume with a volume rendering method results in a more realistic image, and the cage only needs to approximate the mesh; this reduces the complexity of the cage for objects with many parts (a cage for a plant does not need to have a different part for each leaf, it only needs to cover all leaves) and allows the same cage to be used for objects of the same type with similar shapes (different chairs can use the same cage). The cage can be intuitively deformed and controlled by the user, physically based simulation, or conventional automatic animation techniques like fused shapes. The human face is a particularly difficult case due to the non-trivial combination of rigid and (visco)elastic motion, and the present technique performs well for human faces, as described in more detail below. Once a radiance field has been trained using the present technique, it can be generalized to arbitrary geometric deformations that can be represented using a cage of 3D primitives constructed from their density. This opens new possibilities for using volumetric models in gaming or AR / VR contexts, where the user's manipulation of the environment is not known a priori.

[0043] In an example, the deformation description 200 is referred to as a volumetric three-dimensional deformable model (Vol3DMM), which is a parametric 3D facial model that uses a skeleton and blended shapes to animate a surface mesh of a person's head and the volume around the mesh.

[0044] A user or automated process can specify values ​​for the parameters of the Vol3DMM model that are used to animate the Vol3DMM model to create images 204 to 208, as described in more detail below. Different values ​​for the parameters of the Vol3DMM model are used to generate each of the three images 204 to 208. The Vol3DMM model along with the parameter values ​​is an example of a deformation description.

[0045] Vol3DMM animates a volumetric mesh with a sequence of volumetric fused shapes and a skeleton. It is a generalization of the Parametric Three-Dimensional Deformable Model (3DMM) model, which animates a mesh with a skeleton and fused shapes to a parametric model to animate the volume around the mesh.

[0046] The skeleton and blend shapes of Vol3DMM are defined by extending the skeleton and blend shapes of the parametric 3DMM facial model. The skeleton has four bones: the root bone that controls the rotation, the neck bone, the left eye bone and the right eye bone. To use this skeleton in Vol3DMM, the linear blend skin weights are extended from the vertices of the 3DMM mesh to the vertices of the tetrahedron by nearest vertex lookup, that is, each tetrahedron vertex has the skin weight of the nearest vertex in the 3DMM mesh. Volumetric blend shapes are created by extending the 224 expression blend shapes and 256 identity blend shapes of the 3DMM model into the volume surrounding its template mesh: the i-th volumetric blend shape of Vol3DMM is created as a tetrahedral embedding of the mesh of the i-th 3DMM blend shape. To create the tetrahedral embedding, a single volume structure is created from the common mesh and an accurate embedding is created that takes into account the facial geometry and facial deformations: it avoids inter-tetrahedral penetration between the upper and lower lips, it defines a volume support that covers the hair, and has higher resolution in areas that are subject to more deformations. In the example, the exact number of bones or fused shapes are inherited from the specific instance of the chosen 3DMM model, but the technique can be applied to different 3DMM models using fused shapes and / or skeletons for modeling faces, bodies or other objects.

[0047] As a result of this construction, the Vol3DMM is controlled and posed using the same identity, expression and pose parameters α, β, θ of the 3DMM face model. This means that it can be animated using a face tracker built on the 3DMM face model by varying α, β, θ, and more importantly, generalizes to any expression representable by the 3DMM face model, as long as there is a good fit of the face model to the training frames. During training, the tetrahedral mesh of the Vol3DMM is posed using the parameters α, β, θ to define the physical space, while the canonical space is defined for each object by posing the Vol3DMM with the identity parameter α and setting β, θ to zero for a neutral pose. In the example, the decomposition of identity, expression and pose is inherited from the specific instance of the selected 3DMM model. However, the techniques for training and / or animation adapt to different decompositions by constructing a corresponding Vol3DMM model for the specific 3DMM model selected.

[0048] Figure 3 The chair 300 is shown and used Figure 1 3. A composite image 302 of a chair shattering is computed using an image animator of a computer program product. In this case, the deformation description includes a cage surrounding the chair 300, wherein the cage is formed by primitive 3D elements such as tetrahedrons, spheres, or cuboids. The deformation description also includes information such as rules from a physics engine about how the object behaves when it shatters.

[0049] Figure 4 Is Figure 1 Flowchart of an example method performed by an image animator of . The input 400 to the method includes a deformation description, a camera viewpoint, and camera parameters. The camera viewpoint is the viewpoint of the virtual camera for which the synthetic image is to be generated. The camera parameters are lens and sensor parameters, such as image resolution, field of view, focal length. The type and format of the deformation description depends on the type and format of the deformation description used in the training data when the radiation field parameterization was trained. Figure 7 To describe the training process. Figure 4 Involving test time operations after the radiation field parameterization has been learned. In some cases, the deformation description is a vector of concatenated parameter values ​​of a parameterized 3D model of an object in a dynamic scene, such as a Vol3DMM model. In some cases, the deformation description is one or more physics-based rules from a physics engine to be applied to a cage of primitive 3D elements encapsulating the 3D object to be rendered and extending into a volume around the 3D object.

[0050] In some examples, the input 400 includes default values ​​for some or all of the deformation description, the viewpoint, and intrinsic camera parameters. In some cases, the input 400 comes from a user or from a gaming device or other automated process. In an example, the input 400 is made according to a game state from a computer game or according to a state received from a mixed reality computing device. In an example, a face or body tracker 420 provides the value of the deformation description. The face or body tracker is a trained machine learning model that takes as input the captured sensor data depicting at least a portion of a person's face or body, and predicts the value of a parameter of a 3D face model or a 3D body model of the person. The parameter is a shape parameter, a pose parameter, or other parameter.

[0051] The deformation description includes a cage 418 of primitive 3D elements. The cage of primitive 3D elements represents the 3D object to be depicted in the image and the volume extending from the 3D object. In some cases, such as when the 3D object is a person's head or body, the cage includes a volume mesh with multiple volume fusion shapes and a skeleton. In some cases where the 3D object is a chair or other 3D object, the cage is calculated from the learned radiation field parameterization by calculating the mesh according to the density of the learned radiation field volume using Marching Cubes and calculating the tetrahedral embedding of the mesh. The cage 418 of primitive 3D elements is a deformed version of the canonical cage. That is, in order to produce a modified version of the scene, the method starts by deforming the canonical cage into a desired shape, which is the deformation description. The method is agnostic about the way in which the deformed cage is generated and what kind of object is deformed.

[0052] Using a cage to control and parameterize volume deformations enables deformations to be represented and applied to scenes in real time, can represent both smooth and discontinuous functions and allows intuitive control by changing the geometry of the cage. This geometric control is compatible with machine learning models, physics engines, and art generation software, thereby allowing good extrapolation or generalization to configurations not observed in training.

[0053] In the case where the cage is formed of tetrahedrons, using a collection of tetrahedrons is equivalent to a piecewise linear approximation of the motion field. Graphics processing unit (GPU) accelerated ray tracing allows the cage representation to be fast enough to be queried in milliseconds, even with highly complex geometries. The cage representation using tetrahedrons can reproduce hard object boundaries by construction and can be edited in off-the-shelf software since it consists only of points and triangles.

[0054] At operation 402, the dynamic scene image generator computes a plurality of rays, each ray being associated with a pixel of the output image 116 to be generated by the image animator. For a given pixel (x, y position in the output image), the image animator computes a ray from the virtual camera through the pixel into the deformation description comprising the cage. To compute the ray, the image animator uses the geometry and selected values ​​of the intrinsic camera parameters and the camera viewpoint. The rays are computed in parallel where possible to give efficiency, since one ray is computed for each pixel.

[0055] For each ray, the image animator samples multiple points along the ray. In general, the more points sampled, the better the quality of the output image. Rays are randomly selected and samples are drawn within specified boundaries obtained from scene knowledge 416. In the example, the specified boundaries are calculated based on training data that has been used to train the machine learning system. The boundaries indicate the size of the dynamic scene so that one or more samples are taken from the area of ​​the ray that is in the dynamic scene. In order to calculate the boundaries based on the training data, standard image processing techniques are used to examine the training images. The boundaries of the dynamic scene can also be manually specified by an operator, or the boundaries can be automatically measured using a depth camera, a global positioning system (GPS) sensor, or other position sensor.

[0056] Each sample is assigned the index of the 3D primitive element of the deformed cage within which the sample falls.

[0057] At operation 406, the image animator transforms the samples from the deformation description cage to a canonical cage. The canonical cage is a version of the cage that represents the 3D object in a static or other specified original state, such as where the parameter values ​​are zero. In an example where the 3D object is a person's head, the canonical cage represents the person's head looking straight ahead at the virtual camera, with eyes open and mouth closed, and a neutral expression.

[0058] In the case where the primitive 3D element is a tetrahedron, the sample to canonical cage transformation is computed using barycentric coordinates as described below. Using barycentric coordinates is a particularly efficient way to compute the transformation.

[0059] In the example where the cage uses tetrahedrons, a point p in the deformed space is mapped to a point P in the canonical space using barycentric coordinates defined for both the canonical tetrahedron X={X1, X2, X3, X4} and the deformed tetrahedral cell x={x1, x2, x3, x4}.

[0060] A tetrahedron, a basic building block, is a four-sided pyramid. The undeformed "rest" positions of its four constituent points are defined as:

[0061] X={X1,X2,X3,X4}(2)

[0062] And use lower case to represent the deformed state x = {x1, x2, x3, x4}. Because the tetrahedron is a simplex, it is possible to refer to the set X or x and use the barycentric coordinates (λ1, λ2, λ3, λ4) to represent the points falling inside them.

[0063] Although the input point can be restored as: If p falls inside the tetrahedron, its stationary position P in the canonical space is obtained as:

[0064]

[0065] In the case where the primitive 3D element is a sphere or a cuboid, an affine transformation is used instead to compute the sample-to-canonical cage transformation, which is sufficiently expressive for large rigid moving parts of the motion field.

[0066] From each camera, shoot a ray into physical space, detect the tetrahedron x0 incident on each sample p along the ray and calculate its barycentric coordinates such that:

[0067]

[0068] In the case where the 3D element is a tetrahedron, an optimization is optionally used to calculate the transformation at operation 406 by optimizing the primitive point lookup. The optimization includes calculating the transformation P of the sample by setting P equal to the normalized distance between the previous intersection point and the next intersection point of the tetrahedron on the ray, multiplied by the sum of the barycentric coordinates of the vertex at the previous intersection point multiplied by the canonical coordinates of the vertex on the four vertices of the tetrahedron, plus one minus the normalized distance, multiplied by the sum of the barycentric coordinates of the vertex at the next intersection point multiplied by the canonical coordinates of the vertex on the four vertices of the tetrahedron. This optimization was found to give a significant improvement in processing time, such that Figure 4 Real-time operation of the process is possible at more than 30 frames per second (ie, more than 30 images are calculated per second in the case where the processor is a single RTX3090 (trademark) graphics processing unit).

[0069] Operation 407 is optional and includes rotating the view direction of at least one of the rays. In this case, for one of the transformed samples, rotating the view direction of the rays of the sample is done before querying the learned radiation field. Computing the rotation R of the view direction for a small portion of the primitive 3D element and propagating the value of R to the remaining tetrahedrons via nearest neighbor interpolation has been found to give good results in practice.

[0070] For each sampling point, the dynamic scene image generator queries 408 the radiation field parameterization 102. The radiation field parameterization has been trained to produce color values ​​and density values ​​given a point in a canonical 3D cage and an associated viewing direction. In response to each query, the radiation field parameterization produces a pair of values ​​comprising color and opacity at the sampling point in the canonical cage. In this way, the method computes multiple color values ​​and opacity values ​​410 for 3D points in the canonical cage and the viewing direction using the applied deformation description.

[0071] In an example, the learned radiation field parameterization 102 is a cache of associations between 3D points and view directions and color and opacity values ​​in a canonical version of the cage obtained by querying a machine learning model trained using training data including images of the dynamic scene from multiple viewpoints. By using a cache of values ​​rather than directly querying the machine learning model, significant speedups are achieved.

[0072] The radiation field is a function of the color c and density σ that is queried to obtain at that location in space. Typically, the image plane is obtained via volume rendering using the emission-absorption form of the volume rendering equation The color of the pixel on:

[0073]

[0074] where δi = (pi + 1 - pi) represents the distance between samples (a total of N) along the straight line, and the projection rate Ti is defined as It is usually modeled by a multi-layer perceptron (MLP), an explicit voxel grid, or a combination of both. In addition to the sample position p, It is also conditioned on the direction b of the ray, which allows it to model view-dependent effects such as specular reflections.

[0075] For each ray, a volume rendering 412 method is applied to the color values ​​and opacity values ​​calculated along the ray to produce pixel values ​​for the output image. Any well-known computer graphics method for volume ray tracing is used. In cases where real-time operation is desired, hardware accelerated volume rendering is used.

[0076] The output images are stored 414 or inserted into a virtual web camera stream or used for telepresence, gaming or other applications.

[0077] Figure 5is a schematic diagram of rays in a deformed cage 500, rays transformed to a canonical cage 502, volume lookup 504, and volume rendering 506. To render a single pixel, a ray is cast from the camera center through the pixel into the scene in its deformed state. Multiple samples are generated along the ray, and each sample is then mapped to the canonical space using the deformation Mj of the corresponding tetrahedron j. The transformed sample position p′ is then used j The volume representation of the scene is queried with the direction of the ray rotated based on the rotation of the jth tetrahedron. The resulting per-sample opacity and color values ​​are then merged using volume rendering as in Eq.

[0078] The density and color at each point in the scene are functions of both the sample position and the view direction. If the sample position is moved, but the view direction remains the same, then the light reflected from elements of the scene will look the same for each deformation. To alleviate this problem, the view direction of each sample is rotated using a rotation between the canonical tetrahedron and its deformed equivalent:

[0079] v′=Rv,

[0080] U,E,V=SVD((X-cX) T (X′-v′X)),

[0081] R=UVT,

[0082] where cX, c′X are the centroids of the canonical and deformed states of the tetrahedron that a given sample falls into. With this scheme, the direction from which light is reflected at each point of the scene will match the deformation caused by the tetrahedral mesh. Note, however, that the reflected light will represent the scene in its canonical pose.

[0083] In practice, it is inefficient to compute R for every sample or even every tetrahedron in the scene, since it requires computing a singular value decomposition (SVD). Instead, a random scheme is employed, where R is computed for a small fraction ρ of tetrahedrons and propagated to the remaining tetrahedrons via nearest neighbor interpolation. In the experiments described in this paper, ρ = 0.05.

[0084] More details are now given on the example of primitive point finding.

[0085] For complex meshes, checking each tetrahedron for association with each input point is difficult given the complexity of the tetrahedron midpoint test. For non-self-intersecting tetrahedral meshes, the notion of a point "in front of" or "behind" a particular triangle is uniquely determined by the winding order of the triangle vertices. Determining which tetrahedron a point belongs to is equivalent to shooting rays from the point in random directions, evaluating the triangle at the first intersection point, and checking which side of the triangle the sample is on. This uniquely identifies the tetrahedron, since each triangle can belong to at most two tetrahedrons. In particular, when hardware acceleration is available, these queries are very efficient in terms of storage and computation.

[0086] In the example, the same acceleration is applied to any triangular shape to combine a tetrahedron with a triangular rigid shift shape, which does not need to be filled with tetrahedrons, but can be treated as a unit in terms of deformation. Second, the number of required tetrahedral midpoint tests is reduced by noting that many samples along a single ray can fall into the same element. Knowing the previous and next middle parts, a simple depth test determines which tetrahedral samples fall into them. The barycentric coordinates are linear, and therefore the barycentric interpolation is obtained by interpolating the values ​​at the previous intersection point and the next intersection point within each element. To do this, equation (3) is rewritten as:

[0087]

[0088] Wherein, the superscripts 1 and 2 refer to the previous intersection point and the next intersection point, and a is the normalized distance between the two intersection points, which defines the point at which the method interpolates.

[0089] Due to this modification, each point value remains stable even if the "wrong" side of a triangle is queried (or the incorrect triangle altogether) due to lack of numerical precision. An important side effect of this per-ray is that it integrates naturally with ray marching schemes for rendering, as opposed to the per-point formulation of tetrahedron index lookups. In the latter, rays are terminated based on transmittance, which the reformulated tetrahedron lookup algorithm naturally allows for.

[0090] Figure 6 is a flow chart of a method for sampling. The method includes querying the learned radiance field of a 3D scene to obtain color values ​​and opacity values, using only one radiance field network 600 and increasing the size 602 of the sampling bin.

[0091] Volume rendering typically involves sampling the depth along each ray. In an example, there is a sampling strategy that enables capturing thin structures and fine details, as well as improving sampling boundaries. The method gives improved quality at a fixed sampling count.

[0092] Some schemes represent a scene with two multi-layer perceptions (MLPs): "coarse" and "fine". First, Nc samples are evaluated by the coarse network to obtain a coarse estimate of the opacity along the ray. These estimates then guide a second round of Nf samples placed around the location where the opacity value is maximum. The fine network is then queried at both the coarse sample locations and the fine sample locations, resulting in Nc evaluations in the coarse network and Nc+Nf evaluations in the fine network. During training, the two MLPs are optimized independently, but only samples from the fine MLP contribute to the final pixel color. The inventors have recognized that the first Nc samples evaluated in the coarse MLP are not used to render the output image and are therefore effectively wasted.

[0093] To improve efficiency, we avoid querying the fine network at the location of the coarse samples, and instead utilize a single MLP network to reuse the output from the first round of coarse samples.

[0094] The simple change of using one network instead of two leads to artifacts where, for example, Figure 6 As illustrated in 606, 608 of , the area around the fragments of the rays that have been assigned high weights can be clipped. Clipping can occur because the bin placement used to draw the fine samples treats the density as a step function at the sample location, rather than a point estimate of a smooth function. Therefore, the size of each importance sampling bin 610 is doubled, allowing the importance samples to cover the entire range between the coarse samples, as in Figure 6 As shown in 612 and 614.

[0095] Figure 7 is a flow chart of a method for computing an image depicting a person with their mouth open. Optionally using Figure 7 A method in which only one or two time instances are used in the training images. If many time instances are available in the training images, then no Figure 7 In the method, the cage represents a human face and includes a mesh of the interior of a mouth 700, a first plane representing the upper teeth of the human, and a second plane 702 representing the lower teeth of the human. The method includes checking 704 whether one of the samples falls within the interior of the mouth and calculating a transformation 708 of the sample using information about the first plane and the second plane. The transformed sample is used to query 710 the radiation field, and the method is as follows Figure 4 Do as in.

[0096] In the example, a separate deformation model is defined for the interior of the mouth, bounded by closed triangle primitives, and animated by two rigidly moving planes, one for each row of teeth.

[0097] It operates in a minimal data training regime and uses a single frame to train the model. The animation is then driven by an "a priori" known animation model Vol3DMM in the case of animating the face. Therefore, the cage geometry model makes the primitives non-self-intersecting (to allow real-time rendering) and is driven using Vol3DMM. In the special case of the inside of the mouth, completely filling the cavity with tetrahedrons is not an appropriate choice, because the rendered teeth will deform as the mouth opens. This will result in unrealistic appearance and movement. The alternative of placing rigid deforming tetrahedrons around the teeth would require a very high accuracy geometry estimate.

[0098] Instead, a different primitive is chosen for the interior of the mouth. First, the interior of the mouth is filled with tetrahedra, as if it were treated the same as the rest of the head, and second, the indices of the outer triangles of the tetrahedra corresponding to the interior of the mouth are recorded, effectively forming a surface mesh for the interior of the mouth. This surface mesh moves with the rest of the head and is used to determine which samples fall within the interior of the mouth, but is not used to deform them back to canonical space. GPU accelerated ray tracing supports both tetrahedral and triangle defined primitives, allowing changes to the primitives driving the animation.

[0099] To model the deformation, two planes are used, one placed just below the top teeth and one placed just above the bottom teeth. These two planes, along with the underlying volumetric 3D deformable model of the face, move rigidly (i.e., they both remain flat). The teeth are assumed to move rigidly with these planes and it is decided not to support the tongue, so the space between the planes is assumed to be empty.

[0100] Using the surface mesh that defines the entire mouth and these two planes, animate the interior of the mouth using the following steps.

[0101] (1) Detect the primitive in which each sample falls and check whether it is an inside-mouth primitive.

[0102] (2) For each sample within the mouth interior primitive, use the signed distance to the upper and lower planes to determine whether it falls into the upper or lower mouth.

[0103] (3) Calculate the coordinates of the sample in the canonical space by 1) calculating the coordinates of the sample relative to the relevant plane, 2) finding the position of the plane in the canonical space, and 3) assuming that the relative coordinates of the sample relative to the relevant plane remain unchanged.

[0104] In the example, the canonical pose is that of a closed mouth, i.e., one in which the teeth overlap (the top of the bottom teeth is below the bottom of the upper teeth). As a result, the upper and lower mouth regions partially overlap in the canonical space. Therefore, the colors and densities learned in the canonical space must be averages of the corresponding regions in the upper and lower mouths. To overcome this obstacle, the canonical regions for the upper and lower mouth interiors are placed outside the tetrahedral cage, on its left and right sides. Together with the assumption of an empty space between the two planes, this placement results in a bijective mapping of the sample from the mouth interior in the deformation space to the canonical space, allowing correct learning of the radiation field for this region.

[0105] Figure 8 1 is a flow chart of a method for training a machine learning model and computing a cache for use in the image animator 100. Training data 800 comprising images of a scene (static or dynamic) taken from many viewpoints is accessed. It is possible to use a collection of images of static scenes as training data. It is also possible to use a sequence in which each image represents a scene in a different state.

[0106] Figure 8 The description is first made for the case where the images are of a scene from multiple different viewpoints obtained at the same time instance or two time instances, so that the amount of training data required for registration is relatively low. By using a single time instance or two time instances, the accuracy of the ground truth parameter values ​​where the face tracker is used to calculate the deformation description is improved. This is because the face tracker introduces errors and if it is used on frames at many time instances, there are more errors.

[0107] The training data image is a real image such as a photograph or a video frame. The training data image may also be a synthetic image. Tuples of values ​​are extracted 601 from the training data image, where each tuple is a deformation description, a camera viewpoint, a camera intrinsic parameter, and a color of a given pixel.

[0108] In from Figure 3 In the example of a chair, the training data includes images of the chair taken from many different known viewpoints at the same time. The images are synthetic images generated using computer graphics techniques. A tuple of values ​​is extracted from each training image, where each tuple is a deformation description, a camera viewpoint, a camera intrinsic parameter, and the color of a given pixel. The deformation description is a cage determined 802 by placing a cage of primitive 3D elements around the chair and extending from the chair using known image processing techniques. A user or an automated process such as a computer game triggers the physics engine to deform the cage using the rules of physics so as to break the chair when it falls under gravity or crush the chair when it is subjected to pressure from another object.

[0109] To form training data, samples are acquired along rays in the cage 804 by shooting rays from the viewpoint of the camera that captured the training images. Samples are acquired along the rays, as shown in FIG. Figure 4 as described. Each sample is assigned an index of one of the 3D primitive elements of the cage according to the element within which the sample falls. The samples are then transformed 806 to a standard cage, which is a version of the cage in a static position. The transformed samples are used to calculate output pixel colors using volume rendering. The output pixel colors are compared to the ground truth output pixel colors of the training images, and a loss function is used to evaluate the difference or error. The loss function output is used to perform back propagation to train 808 the machine learning model and output a trained machine learning model 810. The training process is repeated for many samples until convergence is reached.

[0110] The resulting trained machine learning model 810 is used to compute and store a cache 812 of associations between 3D positions and view directions and color and opacity values ​​in the canonical cage. This is done by querying the trained machine learning model 810 for ranges of 3D positions and storing the results in a cache.

[0111] In from Figure 2 In the example of a face of a person, the training data includes images of the person's face taken from many different known viewpoints at the same time. Associated with each training data image are the values ​​of the parameters of the Vol3DMM model of the person's face and head. The parameters include the pose (position and orientation) of the eyes and the bones of the neck and jaw, as well as fused shape parameters that specify characteristics of human facial expressions such as eyes closed / open, mouth closed / open, smile / no smile, etc. The images are real images of people captured using one or more cameras with known viewpoints. A 3D model is fitted to each image using any well-known model fitting process, thereby searching for the values ​​of the parameters of the 3DMM model used to generate the Vol3DMM to find a set of values ​​that enables the 3D model to describe the observed real image. The values ​​of the parameters found are then used to label the real image and are the values ​​of the deformation description. Each real image is also labeled with the known camera viewpoint of the camera used to capture the image.

[0112] Then, execute Figure 8 The process from operation 802 to operation 812 is as follows.

[0113] The machine learning model is trained 808 using a training objective that attempts to minimize the difference between the colors produced by the machine learning model and the colors given in the ground truth training data.

[0114] In some examples involving facial animation, a sparsity loss is optionally applied in the volume around the head and inside the mouth.

[0115] The sparsity loss allows handling incorrect background reconstructions, as well as mitigating issues caused by non-occlusions in the inner region of the mouth. In the example, the Cauchy loss is used,

[0116]

[0117] where i indexes the ray ri shot from the training camera, and k indexes the samples tk along each of the rays. N is the number of samples to which the loss is applied, λs is a scalar loss weighting hyperparameter, and σ is the opacity returned by the radiance field parameterization. To ensure that the space is uniformly covered by the sparsity loss, it is applied to the “coarse” samples. Other sparsity-inducing losses such as l1 or weighted least squares also work. The sparsity loss is applied in two regions: in the volume around the head and inside the mouth. Applied to the volume around the head, the sparsity loss prevents opaque regions from appearing in areas where there is not enough multi-view information to eliminate the foreground from the background in 3D. To detect these areas, the loss is applied to (1) samples that fall in tetrahedral primitives, since this is the area rendered at test time, and (2) samples that belong to rays that fall in the background in the training image, as detected by the 2D face segmentation network for the training image. The sparsity loss is also applied to coarse samples that fall within the volume inside the mouth. This prevents the creation of opaque regions inside the mouth in areas that are not seen at training time, and are therefore unsupervised, but become unoccluded at test time.

[0118] The sparsity loss in the mouth interior ensures that there is no unnecessary density inside the mouth interior. However, the colors behind the areas occluded at the training frame remain undefined, leading to unpleasant artifacts when these areas are occluded at test time. The solution in this paper is to override the color and density of the last sample along each ray falling in the mouth interior, which allows setting the color of the unoccluded areas at test time to match the learned colors in the visible areas between the teeth at training time.

[0119] The present technique has been empirically tested for a first application and a second application.

[0120] In the first experiment, a physics-based simulation was used to control the deformation of a static object (aircraft propeller) that was subject to complex topological changes, and a photo-realistic image of the process was rendered for each step of the simulation. This experiment shows the representation power of the deformation description and the ability to render images from physical deformations that are difficult to capture with a camera. A data set of propellers that were subjected to continuous compression and rotation was synthesized. For these two types of deformations, 48 ​​time frames were rendered for 100 cameras. For this technique, only the first frame that can be considered as a stationary state is trained, but a coarse tetrahedral mesh describing the motion of the sequence is supplied. In the first application, the average peak signal-to-noise ratio of the interpolation of each other frame (not visible in training) of the present technique is 27.72, compared to 16.63 for the alternative scheme that does not use a cage and uses position encoding on the time signal. Compared with 12.78 for the alternative technique, the peak signal-to-noise ratio of the present technique over time extrapolation (the second half of the frame, not visible in training) is 29.87 for the present technique. The present technique computes the image in a first application in approximately 6 ms at a 512x512 resolution frame, as opposed to approximately 30 s for the alternative technique.

[0121] In a second experiment, a fused shape based face tracker was used to compute photo-realistic animations of a human head avatar in real time. The avatar was trained using 30 images of the object taken from different viewpoints at the same moment. Therefore, for each avatar, the method only sees a single facial expression and pose. To animate the head avatar, control parameters of a parametric 3DMM face model extending from a surface mesh to the volume around it are used. The resulting parametric volumetric face model is called Vol3DMM. Building on the parametric face model allows generalization to facial expressions and poses that were not visible at training time, and using a face tracker built on top of it for real-time control. A key benefit of the method is that hair, accessories and other elements are captured by the cage. The proposed scheme can be applied to the whole body.

[0122] In the second experiment, multi-view facial data was acquired using a camera rig that captured synchronized video from 31 cameras at 30 frames per second. The cameras were located 0.75-1m from the subject, with the viewpoint spanning 270° around its head and mostly focused on the frontal view within ±60 frames. The illumination was non-uniform. All images were downsampled to 512×512 pixels and color corrected to have consistent color features across cameras. The camera pose and intrinsic parameters were estimated using a standard motion-to-structure pipeline.

[0123] For the second experiment, speech sequences with natural head motion for four subjects were captured. Half of the subjects additionally performed various facial expressions and head rotations. To train the model for each subject, the face tracking results from the face tracker and images from multiple cameras at a single time instance (frame) were used. The frames were selected to meet the following criteria: 1) the salient areas of the teeth are visible, and the bottom of the upper teeth are above the top of the lower teeth to place a plane between them, 2) the subject is looking forward and some of the whites of the eyes are visible on both sides of the iris, 3) the face fit for the frame is accurate, 4) the texture of the face is not too wrinkled (e.g., in the nasolabial folds) due to the open mouth. When a single frame satisfying 1-4 is not available, two frames are used: a frame in which the user has a neutral expression looking forward that satisfies 2-4 to train everything except the inside of the mouth, and a frame in which the mouth is open and satisfies 1 and 3 to train the inside of the mouth.

[0124] The present technique was found to provide 00.1 dB better PSNR than the baseline technique and provide a 10% improvement in Learned Perceptual Image Patch Similarity (LPIPS).The baseline technique uses an explicit grid and does not have a cage extending beyond the face.

[0125] Fig. 9 Illustrated are various components of an exemplary computing-based device 900 implemented as any form of computing and / or electronic device, and in which embodiments of an image animator are implemented in some examples.

[0126] The computing-based device 900 includes one or more processors 914, which are microprocessors, controllers, or any other suitable type of processor for processing computer-executable instructions to control the operation of the device so as to generate a synthetic image of a dynamic scene in a controllable manner. In some examples, such as where a system-on-chip architecture is used, the processor 914 includes one or more fixed function blocks (also referred to as accelerators) that are implemented in hardware (rather than software or firmware). Figures 4 to 8 900 . Platform software including an operating system 908 or any other suitable platform software is provided at the computing-based device to enable application software 910 to execute on the device. Data storage 922 saves output images, values ​​of face tracker parameters, values ​​of physics engine rules, intrinsic camera parameter values, viewpoints, and other data. Animator 902 including radiation field parameterization 904 and volume renderer 906 is present at the computing-based device 900 .

[0127] Computer executable instructions are provided using any computer-readable medium that can be accessed by the computing-based device 900. Computer-readable media include, for example, computer storage media, such as memory 912 and communication media. Computer storage media such as memory 912 include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, etc. Computer storage media include, but are not limited to: random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices, or any other non-transmission medium for storing information accessed by a computing device. In contrast, communication media embody computer-readable instructions, data structures, program modules, etc. in modulated data signals (such as carrier waves) or other transmission mechanisms. As defined herein, computer storage media do not include communication media. Therefore, computer storage media should not be interpreted as propagating signals themselves. Although computer storage media (memory 912 ) is shown as being within computing-based device 900 , it will be appreciated that in some examples storage is distributed or remotely located and accessed via a network or other communications link (eg, using communications interface 916 ).

[0128] The computing-based device 900 has an optional capture device 918 to enable the device to capture sensor data, such as images and video. The computing-based device 900 has an optional display device 920 for displaying output images and / or parameter values.

[0129] Alternatively or in addition to other examples described herein, examples include any combination of the following:

[0130] Clause A. A computer-implemented method of computing an image of a dynamic 3D scene including 3D objects, the method comprising:

[0131] receiving a description of a deformation of the 3D object, the description comprising a cage of primitive 3D elements and associated animation data from a physics engine or an articulated object model;

[0132] For a pixel of the image, calculating a ray from a virtual camera through the pixel into the cage animated according to the animation data, and calculating a plurality of samples on the ray, each sample being a 3D position and a view direction in one of the 3D elements;

[0133] computing a transformation of the sample into a canonical version of the cage to produce a transformed sample;

[0134] For each transformed sample, querying a learned radiance field parameterization of the 3D scene to obtain a color value and an opacity value;

[0135] A volume rendering method is applied to the color values ​​and the opacity values ​​to generate pixel values ​​for the image.

[0136] Clause B. The method of clause A, wherein the cage of primitive 3D elements represents the 3D object and a volume extending from the 3D object.

[0137] Clause C. The method of clause B, wherein the cage comprises a volume mesh having a plurality of volume fused shapes and a skeleton.

[0138] Clause D. A method according to clause B, wherein the cage is computed according to the learned radiation field parameterization by computing a density mesh according to the learned radiation field parameterization using a marching cubes method and computing a tetrahedral embedding of the mesh.

[0139] Item E. A method according to any of the preceding items, further comprising: calculating a transformation P of the sample by setting P equal to the normalized distance between a previous intersection point and a next intersection point of a tetrahedron on the ray, multiplied by the sum of the barycentric coordinates of the vertices at the previous intersection point on the four vertices of the tetrahedron multiplied by the canonical coordinates of the vertices, plus one minus the normalized distance, multiplied by the sum of the barycentric coordinates of the vertices at the next intersection point on the four vertices of the tetrahedron multiplied by the canonical coordinates of the vertices.

[0140] Clause F. The method of clause A, further comprising: for one of the transformed samples, rotating a view direction of rays of the sample before querying the learned radiation field parameterization.

[0141] Clause G. The method of clause F, comprising: computing a rotation R of the view direction for a small portion of the primitive 3D element, and propagating the value of R to the remaining tetrahedrons via nearest neighbor interpolation.

[0142] Clause H. A method according to any preceding clause, wherein the canonical version of the cage is a cage with specified parameter values ​​of an articulated object model or specified parameters of a physics engine.

[0143] Clause I. A method according to any preceding clause, wherein the canonical version of the cage represents a face with a closed mouth.

[0144] Clause J. A method according to any preceding clause, wherein the learned radiation field parameterization is a cache of associations between 3D points in the canonical version of the cage and color values ​​and opacity values, the associations being obtained by querying a machine learning model trained using training data comprising images of the dynamic scene from multiple viewpoints.

[0145] Clause K. A method according to any preceding clause, wherein the images of the dynamic scene from multiple viewpoints are obtained at the same time instance or at two moments in time.

[0146] Clause L. A method according to clause K, wherein the cage represents a face of a person and includes a mesh of the interior of a mouth, a first plane representing an upper row of teeth of the person, and a second plane representing a lower row of teeth of the person.

[0147] Clause M. The method according to clause L comprises: checking whether one of the samples falls within the interior of the mouth, and using information about the first plane and the second plane to calculate the transformation of the sample.

[0148] Item N. A method according to any preceding item, comprising: using only one radiation field network and increasing the number of sampling bins during the process of querying the learned radiation field parameterization of the 3D scene to obtain color values ​​and opacity values ​​for each transformed sample.

[0149] Clause O. An apparatus comprising: at least one processor; a memory storing instructions that, when executed by the at least one processor, perform a method for computing an image of a dynamic 3D scene including a 3D object, the method comprising:

[0150] receiving a description of a deformation of the 3D object, the description comprising a cage of primitive 3D elements and associated animation data from a physics engine or an articulated object model;

[0151] For a pixel of the image, calculating a ray from a virtual camera through the pixel into the cage animated according to the animation data, and calculating a plurality of samples on the ray, each sample being a 3D position and a view direction in one of the 3D elements;

[0152] computing a transformation of the sample into a canonical version of the cage to produce a transformed sample;

[0153] For each transformed sample, querying a learned radiance field parameterization of the 3D scene to obtain a color value and an opacity value;

[0154] A volume rendering method is applied to the color values ​​and the opacity values ​​to generate pixel values ​​for the image.

[0155] Clause P. A computer-implemented method of computing an image of a dynamic 3D scene including 3D objects, the method comprising:

[0156] receiving a description of a deformation of the 3D object;

[0157] For a pixel of the image, calculating a ray from a virtual camera through the pixel to the description and calculating a plurality of samples on the ray, each sample being a 3D position and a view direction in one of the 3D elements;

[0158] computing a transformation of the sample into a canonical space to produce a transformed sample;

[0159] for each transformed sample, querying a cache of associations between 3D points in the canonical space and color and opacity values;

[0160] A volume rendering method is applied to the color values ​​and the opacity values ​​to generate pixel values ​​for the image.

[0161] Item Q. The method according to Item P further includes one or more of the following: storing the image, transmitting the image to a computer game application, transmitting the image to a remote presentation application, inserting the image into a virtual web camera stream, transmitting the image to a head-mounted display.

[0162] Clause R. A method according to clause P or Q, comprising: using an object tracker to detect parameter values ​​of a model of a 3D object depicted in a video, and using the detected parameter values ​​and the model to calculate the description of the deformation of the 3D object.

[0163] Clause S. A method according to any one of clauses P to R, comprising specifying the description using a physics engine.

[0164] Clause T. A method according to any one of clauses P to S, wherein the 3D primitive element is any one of the following: a tetrahedron, a sphere, a cuboid.

[0165] The term 'computer' or 'computing-based device' is used herein to refer to any device having processing capabilities that enable it to execute instructions. Those skilled in the art will recognize that such processing capabilities are incorporated into many different devices, and thus the terms 'computer' and 'computing-based device' each include personal computers (PCs), servers, mobile phones (including smartphones), tablet computers, set-top boxes, media players, game consoles, personal digital assistants, wearable computers, and many other devices.

[0166] In some examples, the methods described herein are performed by software in a machine-readable form (e.g., in the form of a computer program including computer program code elements) on a tangible storage medium, the computer program code elements being adapted to perform all operations of one or more of the methods described herein when the program is run on a computer, and wherein the computer program may be embodied on a computer-readable medium. The software is adapted to be executed on a parallel processor or a serial processor, so that the method operations may be performed in any suitable order or simultaneously.

[0167] Those skilled in the art will recognize that the storage device for storing program instructions is optionally distributed on the network. For example, the remote computer can store examples of processes described as software. Local or terminal computers can access the remote computer and download part or all of the software to run the program. Alternatively, the local computer can download fragments of the software as needed, or execute some software instructions at the local terminal and the remote computer (or computer network). Those skilled in the art will also recognize that, by utilizing conventional techniques known to those skilled in the art, all or part of the software instructions can be executed by dedicated circuits such as digital signal processors (DSPs), programmable logic arrays, etc.

[0168] As will be apparent to one skilled in the art, any range or device value given herein may be expanded or altered without losing the effect sought.

[0169] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0170] It will be understood that the benefits and advantages described above may relate to one embodiment or may relate to several embodiments. The embodiments are not limited to those that solve any or all of the problems described or those that have any or all of the benefits and advantages described. It will also be understood that reference to an item 'a' refers to one or more of those items.

[0171] The operations of the methods described in this article may be performed in any suitable order, or may be performed simultaneously where appropriate. In addition, individual blocks may be deleted from any method without departing from the scope of the subject matter described in this article. Aspects of any of the examples described above may be combined with aspects of any of the examples in the other examples described to form additional examples without losing the effect sought.

[0172] The term 'comprising' is used herein to mean including the identified method blocks or elements, but such blocks or elements do not comprise an exclusive list and the method or apparatus may contain additional blocks or elements.

[0173] The term 'subset' is used herein to refer to a proper subset, such that the subset of a set does not include all elements of the set (ie, at least one of the elements of the set is missing from the subset).

[0174] It will be understood that the above description is given as an example only, and various modifications may be made by those skilled in the art. The above description, examples and data provide a complete description of the structure and use of exemplary embodiments. Although various embodiments have been described above with a certain degree of particularity or with reference to one or more individual embodiments, those skilled in the art may make many changes to the disclosed embodiments without departing from the scope of this specification.

Claims

1. A computer-implemented method of computing an image of a dynamic 3D scene including 3D objects, the method comprising: receiving a description of a deformation of the 3D object, the description comprising a cage of primitive 3D elements and associated animation data from a physics engine or an articulated object model; For a pixel of the image, calculating a ray from a virtual camera through the pixel into the cage animated according to the animation data, and calculating a plurality of samples on the ray, each sample being a 3D position and a view direction in one of the 3D elements; computing a transformation of the sample into a canonical version of the cage to produce a transformed sample; For each transformed sample, querying a learned radiance field parameterization of the 3D scene to obtain a color value and an opacity value; A volume rendering method is applied to the color values ​​and the opacity values ​​to generate pixel values ​​for the image.

2. The method according to claim 1, wherein: The cage of primitive 3D elements represents the 3D object and the volume extending from the 3D object.

3. The method according to claim 2, wherein: The cage includes a volume mesh having a plurality of volume fused shapes and a skeleton.

4. The method according to claim 2, wherein: The cage is computed according to the learned radiation field parameterization by computing a mesh according to a density of the learned radiation field parameterization using a marching cubes method and computing a tetrahedral embedding of the mesh.

5. The method according to any preceding claim, further comprising: The transformation P of the sample is calculated by setting P equal to the normalized distance between the previous intersection point and the next intersection point of the tetrahedron on the ray, multiplied by the sum of the barycentric coordinates of the vertices multiplied by the canonical coordinates of the vertices at the previous intersection point on the four vertices of the tetrahedron, plus one minus the normalized distance, multiplied by the sum of the barycentric coordinates of the vertices multiplied by the canonical coordinates of the vertices at the next intersection point on the four vertices of the tetrahedron.

6. The method according to any preceding claim, further comprising: For one of the transformed samples, a view direction of rays of the sample is rotated before querying the learned radiation field parameterization.

7. The method according to claim 6, comprising: A rotation R of the view direction for a small portion of the primitive 3D element is calculated, and the value of R is propagated to the remaining tetrahedrons via nearest neighbor interpolation.

8. A method according to any preceding claim, wherein: The canonical version of the cage is the cage with specified parameter values ​​for an articulated object model or specified parameters for a physics engine.

9. A method according to any preceding claim, wherein: The canonical version of the cage shows a face with a closed mouth.

10. A method according to any preceding claim, wherein: The learned radiation field parameterization is a cache of associations between 3D points in the canonical version of the cage and color values ​​and opacity values, the associations being obtained by querying a machine learning model trained using training data comprising images of the dynamic scene from multiple viewpoints.

11. A method according to any preceding claim, wherein: The images of the dynamic scene from multiple viewpoints are acquired at the same time instance or at two moments in time.

12. The method according to claim 11, wherein: The cage represents a person's face and includes a mesh of an interior of a mouth, a first plane representing an upper row of teeth of the person, and a second plane representing a lower row of teeth of the person.

13. The method according to claim 12, comprising: It is checked whether one of the samples falls within the interior of the mouth, and the transformation of the sample is calculated using information about the first plane and the second plane.

14. A method according to any preceding claim, comprising: During the process of querying the learned radiance field parameterization of the 3D scene to obtain color and opacity values ​​for each transformed sample, only one radiance field network is used and the number of sampling bins is increased.

15. An apparatus comprising: at least one processor; A memory storing instructions which, when executed by the at least one processor, perform a method for calculating an image of a dynamic 3D scene including 3D objects, the method comprising: receiving a description of a deformation of the 3D object, the description comprising a cage of primitive 3D elements and associated animation data from a physics engine or an articulated object model; For a pixel of the image, calculating a ray from a virtual camera through the pixel into the cage animated according to the animation data, and calculating a plurality of samples on the ray, each sample being a 3D position and a view direction in one of the 3D elements; computing a transformation of the sample into a canonical version of the cage to produce a transformed sample; For each transformed sample, querying a learned radiance field parameterization of the 3D scene to obtain a color value and an opacity value; A volume rendering method is applied to the color values ​​and the opacity values ​​to generate pixel values ​​for the image.

16. A computer-implemented method of computing an image of a dynamic 3D scene including 3D objects, the method comprising: receiving a description of a deformation of the 3D object; For a pixel of the image, calculating a ray from a virtual camera through the pixel to the description and calculating a plurality of samples on the ray, each sample being a 3D position and a view direction in one of the 3D elements; computing a transformation of the samples into a canonical space to produce transformed samples; for each transformed sample, querying a cache of associations between 3D points in the canonical space and color and opacity values; A volume rendering method is applied to the color values ​​and the opacity values ​​to generate pixel values ​​for the image.

17. The method of claim 16, further comprising one or more of: storing the image, transmitting the image to a computer game application, transmitting the image to a telepresence application, inserting the image into a virtual webcam stream, transmitting the image to a head mounted display.

18. A method according to claim 16 or claim 17, comprising: An object tracker is used to detect parameter values ​​of a model of a 3D object depicted in the video, and the description of the deformation of the 3D object is calculated using the detected parameter values ​​and the model.

19. A method according to any one of claims 16 to 18, comprising specifying the description using a physics engine.

20. The method according to any one of claims 16 to 19, wherein: The 3D primitive element is any one of the following: tetrahedron, sphere, cuboid.

Images