Deformable neural radiance fields

Abstract

Image synthesis techniques using neural radiance fields (NeRFs) involve generating a deformation model of the motion experienced by a subject in a non-rigidly deforming scene. For example, when an image synthesis system uses NeRFs, the system takes multiple poses of the subject as input for training data. Unlike traditional NeRFs, the technical solution first represents the position of the subject from various viewpoints in an observation frame. Then, the technical solution involves deriving a deformation model, i.e., a mapping between the observation frame and a canonical frame in which the subject's motion is taken into account. This mapping is achieved using a latent deformation code for each pose determined using a multilayer perceptron (MLP). The NeRF is then derived from the position and projection ray direction in the canonical frame using another MLP. A new pose for the subject may then be derived using the NeRF.

Description

[Technical Field]

[0001] Reference to related applications This application is a non-provisional application claiming priority to U.S. Provisional Patent Application No. 63 / 198,841, filed on November 16, 2020, titled "Deformable Neural Radiance Fields," and the entire content of that application is referenced by public citation.

[0002] Technical field This specification relates to image synthesis using a neural radiance field (NeRF). [Background technology]

[0003] background Some computers configured to render computer graphics objects can render objects in a specified view, assuming multiple existing views. For example, in the case of several depth and color images captured from a camera of a scene containing such computer graphics objects, the goal may be to synthesize a new view of the scene from different viewpoints. This scene may be a real-world scene whose view is captured using physical color and depth sensors, or it may be a composite scene whose view is captured using rendering algorithms such as rasterization. In the case of a real-world scene, many depth sensing techniques exist, such as time-of-flight sensors, structured light-based sensors, and stereo or multi-view stereo algorithms. Such techniques may include visible or infrared sensors that use passive or active lighting patterns whose patterns can change over time. [Overview of the project]

[0004] overview In one schematic configuration, the method may include the step of acquiring image data representing multiple images, each of which includes an image of a scene within an observation frame, and the scene includes a non-rigid deformable object as seen from its respective viewpoint. The method may also include the step of generating a deformation model based on the image data, which describes the movement of the non-rigid deformable object while the image data was being generated, and which is represented by a mapping between a position in the observation frame and a position in the canonical frame. The method may further include the step of generating a deformable neural radiance field (D-NeRF) based on the position and field of view direction of projected rays passing through a position in the canonical frame. The D-NeRF provides the mapping between the position and field of view direction to the color and optical density at each position in the observation frame, which enables viewing the non-rigid deformable object from a new viewpoint.

[0005] In another schematic context, the computer program product comprises a non-temporary storage medium and includes codes that, when executed by a processing circuit of a computing device, cause the processing circuit to execute a method. The method may include the step of acquiring image data representing a plurality of images. Each of the plurality of images includes an image of a scene in an observation frame, and the scene includes a non-rigid deformable object as seen from its respective viewpoint. The method may also include the step of generating a deformation model based on the image data. The deformation model describes the movement of the non-rigid deformable object while the image data was being generated, and the deformation model is represented by a mapping between a position in the observation frame and a position in the canonical frame. The method may further include the step of generating a deformable neural radiance field (D-NeRF) based on the position and field of view direction of projected rays passing through the position in the canonical frame. The D-NeRF provides the mapping between the position and field of view direction with color and optical density at each position in the observation frame. The color and optical density at each position within the observation frame allow the non-rigid deformation object to be viewed from a new perspective.

[0006] In another schematic configuration, the electronic device comprises a memory and a control circuit coupled to the memory. The control circuit may be configured to acquire image data representing a plurality of images. Each of the plurality of images includes an image of a scene in an observation frame, and the scene includes a non-rigid deformable object as seen from its respective viewpoint. The control circuit may also be configured to generate a deformation model based on the image data. The deformation model describes the movement of the non-rigid deformable object while the image data was being generated, and the deformation model is represented by a mapping between a position in the observation frame and a position in the canonical frame. The control circuit may further be configured to generate a deformable neural radiance field (D-NeRF) based on the position and field of view direction of projected light rays passing through that position in the canonical frame. The D-NeRF provides the mapping between the position and field of view direction to the color and optical density at each position in the observation frame. The color and optical density at each position in the observation frame allow the non-rigid deformable object to be viewed from a new viewpoint.

[0007] Details of one or more implementations are described in the attached drawings and the following description. Other features will become apparent from the following description, the attached drawings, and the attached claims. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an exemplary electronic environment for realizing the technical solutions described herein. [Figure 2] Figure 1 shows an exemplary system architecture for generating a deformable neural radiance field within the electronic environment. [Figure 3] Figure 1 is a flowchart illustrating an exemplary method for implementing a technical solution within the electronic environment shown. [Figure 4A] This diagram shows an example posture of the subject within the observation frame. [Figure 4B]This diagram shows an example of a person's posture in a canonical frame. [Figure 4C] This figure shows an example of interpolation between a keyframe without elastic regularization (top) and a keyframe with elastic regularization (bottom). [Figure 5] This figure shows examples of computer devices and mobile computer devices that may be used with the circuits described herein. [Modes for carrying out the invention]

[0009] Detailed explanation Conventional methods for synthesizing new views of a scene employ the Neural Radiance Field (NeRF). In this method, the static scene is represented as a continuous 5-dimensional function that outputs the radiance emitted in each direction (θ,ψ) at each point (x,y,z) in space, and the density at each point acts like a differential opacity that controls how much radiance is accumulated by light rays passing through (x,y,z). This method represents the 5-dimensional function by optimizing a fully connected deep neural network without convolutional layers, often called a multilayer perceptron (MLP), to regress from a single 5D coordinate (x,y,z,θ,ψ) to a single volume density and view-dependent RGB color.

[0010] To render this 5-dimensional function, or NeRF, we can: 1) propagate camera rays through the scene to generate a sampling set of 3D points; 2) use these points and their corresponding 2D field of view directions as input to a neural network to generate an output set of color and density; and 3) use conventional volume rendering techniques to accumulate these colors and densities into a 2D image. Since this process is naturally distinguishable, gradient descent can be used to optimize the NeRF by minimizing the error between each observed image and the corresponding view rendered from our representation. Minimizing this error across multiple views facilitates the network's prediction of a coherent model of the scene by assigning high volume density and accurate color to locations containing the true underlying scene content.

[0011] NeRF generally excels at compositing views in scenes containing rigid, inanimate objects. In contrast, NeRF is not as good at compositing views in scenes containing people (more generally, non-rigid objects that tend to move). Technical problems arise when modeling people using handheld cameras. These technical problems are difficult because 1) they are non-rigid and cannot remain completely still, and 2) they contain troublesome materials such as hair, glasses, and earrings, which violate the assumptions used in most reconstruction methods. For example, if NeRF is used to composite a view of a person taking a picture of themselves (i.e., a selfie) with a cell phone camera, the composite view may contain artifacts resulting from inaccuracies caused by NeRF's inability to handle non-rigid and troublesome materials.

[0012] Unlike conventional approaches to solving the above technical problems, the technical solution to the above technical problems involves generating a motion deformation model that an object experiences in a non-rigid deformation scene that defines how the object is distorted by movement. For example, when an image synthesis system uses NeRF, the system takes in multiple poses of an object as input for training data. Unlike conventional NeRF, the technical solution first represents the position of the object from various viewpoints in the observation frame. Next, the technical solution involves deriving a deformation model, i.e., a mapping between the observation frame and the canonical frame in which the object's movement is considered. This mapping is achieved using pose-specific latent deformation codes determined using a multi-layer perceptron (MLP). NeRF is then derived from the position and projection ray direction within the canonical frame using another MLP. Next, a new pose for the object can be derived using NeRF.

[0013] The technical advantage of the above technical solution is that the technical solution accurately predicts a synthetic view of a new scene without artifacts resulting from not considering the movement of the object.

[0014] In some implementations, the deformation model is conditioned on frame-specific latent codes, and the latent codes encode the state of the scene within the frame.

[0015] In some implementations, the deformation model includes a rotation, a pivot point corresponding to the rotation, and a translation. In some implementations, the rotation is encoded as a pure logarithmic quaternion. In some implementations, the deformation model includes (i) a similarity transformation for the difference between the position and the pivot point, (ii) the pivot point, and (iii) the translation.

[0016] In some implementations, the deformation model includes a multilayer perceptron (MLP) within a neural network. In some implementations, the elastic loss function component for the MLP is based on the norm of the matrix representing the deformation model. In some implementations, the matrix is ​​the Jacobian of the deformation model with respect to its position within the observation frame. In some implementations, the elastic loss function component is based on the singular value decomposition of the matrix representing the deformation model. In some implementations, the elastic loss function component is based on the logarithm of the singular value matrix resulting from the singular value decomposition. In some implementations, the elastic loss function component is composed of rational functions to generate a robust elastic loss function.

[0017] In some implementations, the background loss function component requires specifying points in the scene as static points with a motion penalty. In some implementations, the background loss function component is based on the difference between static points and the mapping of static points in the observation frame to a canonical frame according to a deformation model. In some implementations, the step of generating the deformation model includes applying position coding to position coordinates in the scene to generate a periodic function of position, which has a frequency that increases with training iterations for the MLP. In some implementations, the periodic function of the position coding is multiplied by a weight indicating whether the training iterations include a particular frequency.

[0018] NeRF is a continuous volume representation.

[0019]

number

[0020] Therefore, NeRF, when combined with volume rendering technology, can represent scenes with photographic realism. For this reason, NeRF is built to address the challenges of capturing people realistically, like a photograph.

[0021] The NeRF training procedure assumes a 3D scene and relies on the fact that two intersecting rays of light from two different cameras should produce the same color. Ignoring specular reflection and transmission, this assumption holds true for all scenes with static structure. Unfortunately, it has been found that people are not capable of remaining completely still. This can be verified as follows: If you try to film a selfie while remaining completely still, you will find that your gaze naturally follows the camera, and even parts that you think are still are moving relative to the background.

[0022] Understanding these limitations, NeRF is extended to allow for the reconstruction of non-rigid deformation scenes. This is done by using a canonical template of the scene instead of directly projecting rays through NeRF. This template contains the relative structure and appearance of the scene, and rendering will use the non-rigid transformation version of this template. Alternatively, deformations can be modeled on a template and per-frame, but these deformations are defined on mesh points and voxel grids, respectively, while being modeled as continuous functions using MLP.

[0023] The transformation from observation to canonical is adopted for every frame i ∈ {1, ..., n], where n is the number of observed frames. This is a mapping T that maps all observation space coordinates x to canonical space coordinates x′. i Define :x→x′. In practice, the deformation field is the latent code ω learned frame by frame. i A single MLP T:(x,ω i )→x′ is used to model all time steps. The latent code for each frame models the state of the scene in that frame. Assuming a canonical space radiance field F and an observation-canonical mapping T, the observation space radiance field is,

[0024]

number

[0025] It can be evaluated as follows: During rendering, rays and sample points are simply projected into the observation frame, and then a deformation field is used to map the sampled points to points on the template.

[0026] Figure 1 shows an exemplary electronic environment 100 in which the improved technology described above can be realized. As shown in Figure 1, the exemplary electronic environment 100 includes a computer 120.

[0027] The computer 120 includes a network interface 122, one or more processing units 124, and memory 126. The network interface 122 includes, for example, an Ethernet® adapter for converting electronic and / or optical signals received from a network into an electronic format usable by the computer 120. The set of processing units 124 includes one or more processing chips and / or assemblies. The memory 126 includes both volatile memory (e.g., RAM) and one or more non-volatile memory such as ROM, disk drives, or solid-state drives. Together, the set of processing units 124 and the memory 126 form a control circuit configured and arranged to perform various methods and functions as described herein.

[0028] In some embodiments, one or more components of the computer 120 may include a processor (e.g., a processing unit 124) configured to process instructions stored in memory 126. Examples of instructions, as shown in Figure 1, include an image acquisition manager 130, a deformation model manager 140, and a template NeRF manager 150. Furthermore, memory 126 is configured to store various types of data described for each manager that uses such data, as shown in Figure 1.

[0029] The image acquisition manager 130 is configured to acquire image data 132 to be input to the deformation model manager 140. In some implementations, the image acquisition manager 130 receives the image data 132 via the network interface 122, i.e., over a network. In some implementations, the image acquisition manager 130 receives the image data 132 from local storage (e.g., disk drive, flash drive, SSD, etc.).

[0030] Image data 132 represents multiple images of scenes 134(1), 134(2), ..., 134(N). For example, a user may generate images 134(1), 134(2), ..., 134(N) by recording images of themselves, i.e., "selfies," from various viewpoints 136(1), 136(2), ..., 136(N) using a mobile phone camera.

[0031] Modeling people with a handheld camera is particularly difficult due to both 1) their non-rigidity, which prevents them from remaining completely still, and 2) the troublesome materials such as hair, glasses, and earrings, which violate the assumptions used in most reconstruction methods. To model non-rigid deformation scenes, NeRF can be generalized by introducing an additional component: a canonical NeRF model that acts as a template for all observations, supplemented by a deformation field for each observation that distorts the 3D points in the reference frame of observation, making it the reference frame for the canonical model.

[0032] The deformation model manager 140 is configured to generate a deformation model that provides a mapping between coordinates in the observation space of the scene and coordinates in the canonical space to which the NeRF model is applied. For this purpose, the deformation model manager 140 is configured to generate observation frame position data 141 and latent deformation code data 142.

[0033] The observation frame data 141 represents the coordinates of a point in the observation frame, i.e., the coordinate frame of images 134(1), 134(2), ..., 134(N). For example, the observation frame data 141 may represent a point x within a volume representing the extent of the space in which the image data 132 was generated. The observation frame can be visualized in Figure 2.

[0034] Figure 2 shows an exemplary system architecture 200 for generating a deformable neural radiance field. Figure 2 shows the observation frame 220 as a set of points in a three-dimensional volume. Figure 2 also shows the camera viewpoint 210 to which the ray 212 is directed.

[0035] The latent deformation code data 142 represents the latent deformation code, denoted by the symbol ω in Figure 2. Each image 134(1), ..., 134(N) is associated with its own latent deformation code. The image-specific latent deformation code associated with the scene images models the state of the scene in that image. In some implementations, image-specific latent deformation codes are learned. Each image-specific latent deformation code has a small number of dimensions, and in some implementations, each latent deformation code has 8 dimensions.

[0036] The deformation model manager 140 is also configured to generate deformation models based on observation frame data 141 and latent deformation code data 142. In some implementations, the deformation model is derived using a neural network. In some implementations, the neural network does not include convolutional layers. As shown in Figure 2, the deformation model is derived using a multilayer perceptron (MLP) 230. As shown in Figure 1, the deformation model is derived using deformation field MLP data 143.

[0037] The deformation field MLP data 143 represents values ​​that define the deformation field MLP. An exemplary deformation field MLP in the context of the technical solutions described herein has six layers (one input, one output, and four hidden layers). In this example, the size of the hidden layers (i.e., the number of nodes) is 128, and there is a skip connection in the fourth layer, and the Softplus activation function log(1+e x ) is also included. The deformed field MLP data 143 further includes loss function data 144 and coarse-density data 145.

[0038] Deformation models introduce ambiguities that can make optimization more difficult. For example, an object moving backward may be visually equivalent to an object shrinking in size, with a great deal of resolution in between. These ambiguities lead to a lack of constraints on optimization, resulting in unrealistic deformations and artifacts. Therefore, prior art is introduced to provide more reasonable solutions.

[0039] The loss function data 144 represents the loss function components used to determine the node values ​​of the deformed field MLP (i.e., MLP230 in Figure 2) for each training iteration. As shown in Figure 2, the loss function components for MLP230 include elastic loss data 144(1) and background loss data 144(2).

[0040] The elastic loss data 144(1) represents the value of the elastic loss function used to determine the deformation model. Modeling non-rigid deformation using elastic energy, which measures the local deformation deviation from rigid motion, is common in geometrical processing and physical simulations. Such energy has been widely used for the reconstruction and tracking of non-rigid scenes and objects. Therefore, elastic energy is a suitable candidate for such methods. While elastic energy has been most commonly used for discretized surfaces, such as meshes, a similar concept can be applied to situations of continuous deformation fields included in the deformation model.

[0041] constant latent sign ωi In this case, the continuous deformation field T is

[0042]

Number

[0043] a non - linear mapping to. Nevertheless, such a non - linear mapping can be approximated in matrix form.

[0044] In some embodiments, the non - linear mapping is also differentiable. In this case,

[0045]

Number

[0046] represents a suitable linear approximation of the transformation at that point. Therefore, the local behavior of the deformation model is controllable by the Jacobian of T. Note that, unlike other methods using discretized surfaces, this continuous / differentiable formulation enables the direct calculation of the Jacobian of this mapping via automatic differentiation of the deformation field MLP.

[0047] The deviation of the Jacobian J from the rigid - body transformation T There are several ways to penalize. Considering the singular - value decomposition of the Jacobian J T =UΣV T several methods penalize the deviation from the nearest rotation as

[0048]

Number

[0049] where R = VU T and · F is the Frobenius norm. In some embodiments, the elastic loss component is J TBased on the singular values, the elastic loss component includes a measure of the deviation of the singular value matrix Σ from identity I. The logarithm of the singular values ​​was found to work better, giving equal weight to the stretching of the same factor. Thus, the point x is derived from the deviation of the log-singular values ​​from zero. i The elastic loss component in this case is penalized as follows:

[0050]

number

[0051] In the formula, log represents the matrix logarithm. In some implementations, the elastic loss component is remapped to a more robust loss function. For example, while humans are mostly rigid, there are some movements that can break our assumptions about local rigidity, such as facial expressions that locally stretch or contract the skin. The elastic energy defined above can then be remapped using a robust loss component.

[0052]

number

[0053] In the equation, ρ(·) is the robust error function of the Geman-McClure system realized with the hyperparameter c=0.03, and w i This is the weight. The net robust loss component L across multiple points. elastic-r This is the weighted average of the robust loss components at each of those points. The robust loss component reduces the impact of outliers during training by reducing the loss gradient to zero when the argument value is large.

[0054] Background loss data 144(2) represents the values ​​of the background loss function used to determine the deformation model. The deformation field T is unconstrained, and therefore everything can move freely. In some implementations, a regularization term is added to prevent the background from moving. Assuming a set of 3D points in the scene that are known to be static, a penalty can be imposed on any deformation at these points. For example, by aligning the camera using 3D structure from motion reconstruction from multi-view images, a set of 3D feature points that behave rigidly across at least several sets of observations is generated. These static 3D points {x1…,x K Assuming this is the case, the movement will be:

[0055]

number

[0056] A penalty is imposed for this. In addition to preventing background points from moving, this regularization also has the advantage of aligning the observation coordinate frame with the canonical coordinate frame.

[0057] The coarse-density data 145 represents coarse-density deformation regularization. The core component of the NeRF architecture is position coding. A similar concept is adopted for deformation field MLP.

[0058]

number

[0059] The hyperparameter m controls the number of frequency bands (and therefore the highest frequency) used in the mapping. This has been shown to control the smoothness of the network. A higher value of m allows for the modeling of higher frequency details, but this can result in NeRF overfitting and modeled image noise in the 3D structure.

[0060] It has been observed that optimizing NeRF together with the deformation field results in an optimization problem that tends to yield local minima. In the early stages of training, neither NeRF nor the deformation field contains meaningful information. Using a large value for m means that the deformation field may overfit an incomplete NeRF template. For example, if an object rotates its head sideways, a network using a large m will often choose to encode the change in appearance using the view-direction component of the NeRF while keeping the head in a forward position. On the other hand, using a small value for m, the network will not be able to model deformations that require high-frequency details such as facial expressions or moving strands of hair.

[0061] It has been found that the position coding used in NeRF has a convenient interpretation with respect to the neural tangent kernel (NTK) of the NeRF MLP. As a result, a static interpolation kernel is obtained in which m controls the adjustable "bandwidth" of the interpolation kernel. When the number of frequencies is small, a wide kernel is obtained that results in underfitting the data, while when the number of frequencies is large, a narrow kernel is obtained that results in overfitting the data. Taking this into consideration, a method has been proposed to smoothly anneal the NTK bandwidth by introducing a parameter α that windowes the frequency band of the position coding. The weights are, for each frequency band j of the position coding,

[0062]

number

[0063] Defined as, where linear annealing of parameter α∈[0,m] can be interpreted as sliding a partially cut-off, incomplete Hann window across the frequency band (where the left side is fixed to 1 and the right side to 0). Then, position coding is,

[0064]

number

[0065] In this equation, t is the current training iteration, and N is a hyperparameter indicating when α should reach the maximum number of frequencies m.

[0066]

number

[0067] Along these lines, the deformation field MLP data 143 also includes SE(3) transformation data 146. The SE(3) transformation data 146 is encoded in the MLP to represent the transformation field as described above. The SE(3) transformation data 146 includes rotation data 147 representing the rotation q, rotation point data 148 representing the rotation point s, and translation data representing the translation t. In some implementations, the rotation data 147, rotation point data 148, and translation data 149 are represented in quaternion form. In some implementations, the rotation data 147, rotation point data 148, and translation data 149 are represented in another form, such as matrix form.

[0068] Template NeRF Manager 150 is

[0069]

number

[0070] It is configured to generate a 5-dimensional representation of the image. In some implementations, the appearance code Ψ is used for each image. i A modulator is provided to modulate the color output and process appearance changes between input frames, such as exposure and white balance. As shown in Figure 2, the canonical frame 240 is visualized. Points along the ray 212 in the observation frame 220 are mapped to points along the curve 242 in the canonical frame 240 using the deformation field MLP 230. Each position and ray / camera viewpoint is mapped to color and density along with appearance codes by the NeRF MLP 250.

[0071] As shown in Figure 1, the template NeRF manager 150 is configured to generate canonical frame position data 151, orientation data 152, latent appearance code data 153, and template NeRF MLP data 154, and to output output data 160 including color data 162 and density data 163. The canonical frame position data 152 represents the position in the canonical frame mapped from the observation frame using a deformation field MLP (e.g., MLP 230 in Figure 2). The orientation data 152 represents the rays, camera angles, or direction cosines passing through each point in the canonical frame. The latent appearance code data 153 represents the latent appearance code for each image.

[0072] The template NeRF MLP data 154 represents values ​​that define the NeRF MLP. An exemplary NeRF MLP in the context of the technical solutions described herein has six layers (one input, one output, and four hidden layers). In this example, the size of the hidden layers (i.e., the number of nodes) is 128, there is a skip connection in the fourth layer, and there is a ReLU activation function. The deformed field MLP data 154 further includes color loss function data 155 and density data 156.

[0073] The color loss function data 155 represents the values ​​of the color loss function defined as follows: For each optimization iteration, a batch of camera rays is randomly sampled from the entire set of pixels in the dataset, and then N from the coarse network. c Samples and N from a dense network c +N f Hierarchical sampling is then performed to query the samples. A volume rendering procedure is used to render the color of each ray from both sets of samples. Color loss is the total squared error between the rendered pixel color and the true pixel color for both coarse and dense renderings.

[0074]

number

[0075] The coarse-density data 156 represents the same coarse-density transformation regularization as coarse-density data 145. However, in the case of NeRF MLP, the sine and cosine are not weighted.

[0076] Returning to the elastic loss function, the deformation field T can behave freely within the open space because the object moving relative to the background requires non-rigid deformation somewhere in the space. Therefore, in some implementations, the elastic loss function is weighted at each point by its contribution to the rendered view, as follows:

[0077] The 5-dimensional neural radiance field represents the scene as volume density and directional radiance at any point in space. The color of any light ray passing through the scene is rendered using principles derived from conventional volume rendering.

[0078]

number

[0079] The function T(t) is t n The cumulative transmittance along a ray from to t, i.e., the light ray without colliding with other particles to t n This represents the probability of moving from point A to point B. By rendering a view from our continuous neural radiance field, it becomes necessary to estimate this integral C(r) for the camera rays traced through each pixel of the desired virtual camera.

[0080] This continuous integral is estimated numerically using quadrature. Deterministic quadrature, typically used to render discretized voxel grids, would effectively limit the resolution of our representation because the MLP can only be queried in a fixed set of discrete positions. Instead, we use a layered sampling method. In this case, [t n ,t fThe sample is divided into M bins arranged at equal intervals, and then one sample is randomly drawn from each bin.

[0081]

number

[0082] A discrete set of samples is used to estimate the integral, but stratified sampling allows for the representation of continuous scenes. This is because, as a result of stratified sampling, the MLP is evaluated at continuous positions throughout the optimization process. We estimate C(r) using these samples as follows.

[0083]

number

[0084] A rendering strategy that densely evaluates a neural radiance field network at M query points along each camera ray is inefficient, as free space and closed regions that do not contribute to the rendered image are still repeatedly sampled. Therefore, instead of representing the scene with a single network, two networks, one "sparse" and the other "dense," are optimized simultaneously. Firstly, N c A set of locations is sampled using stratified sampling, and the "coarse" network at these locations is evaluated as shown in equations (13), (14), and (15). Based on the output of this "coarse" network, more informative sampling is obtained for each point along each ray. In this case, the sample is biased towards the relevant parts of the volume. For this reason, first, the alpha composite color is given in equation (14),

[0085]

number

[0086] Output data 160 represents the output of the NeRF MLP (i.e., MLP250 in Figure 2). Output data 160 includes color data 162 and density data 164. As described above (e.g., equation (11)), the color data 162 depends on position and ray angle, while the density data 164 depends only on the ray angle. From the color data 162 and density data 164, it may also be possible to obtain a view of the scene from any viewpoint.

[0087] The system 200 shown in Figure 2 may be optimized as a unit, that is, MLP230 and MLP250 may be identified by a single loss function which is the sum of the loss components described above.

[0088]

number

[0089] In the formula, λ and μ are weights. In some implementations, λ = μ = 10 -3 That is the case. Figure 3 is a flowchart illustrating an exemplary method 300 for generating a deformable NeRF. Method 300 may be performed by a software configuration described in association with Figure 1, which resides in the memory 126 of computer 120 and is executed by a set of processing units 124, or by a software configuration which resides in the memory of a computing device different from computer 120 (for example, remotely from computer 120).

[0090] In 310, the image acquisition manager 130 acquires image data (e.g., image data 132) representing multiple images (e.g., images 134(1), ..., 134(N)), each of which includes an image of a scene within an observation frame (e.g., observation frame position data 141), and the scene includes non-rigid deformation objects as seen from each viewpoint (e.g., viewpoints 136(1), ..., 136(N)).

[0091] In 320, the deformation model manager 140 generates a deformation model (e.g., deformation field MLP data 143) based on image data, the deformation model describes the movement of a non-rigid deformation object while the image data was being generated, and the deformation model is represented by a mapping between the position in the observation frame and the position in the canonical frame (e.g., canonical frame position data 151).

[0092] In 330, the template NeRF manager 150 generates a deformable neural radiance field (D-NeRF) based on the position and field of view direction of projected rays passing through the positions in the canonical frame, and the D-NeRF provides a mapping between position and observation direction to color (e.g., color data 162) and optical density (e.g., density data 164) at each position in the observation frame, and the color and optical density at each position in the observation frame enable viewing of non-rigid deformable objects from a new viewpoint.

[0093] The effects of the deformation field on the object are shown in Figures 4A, 4B, and 4C. Figure 4A shows an exemplary posture 400 of the human object in the observation frame. Figure 4B shows an exemplary posture 450 of the human object in the canonical frame. In both Figures 4A and 4B, the object is shown with insets showing orthographic projections in the forward and left directions. Note that in Figure 4A (observation frame), there is displacement from right to left and displacement from front to back between the observation model and the canonical model, and these displacements are modeled by the deformation field for this observation.

[0094] Figure 4C shows an exemplary interpolation between a keyframe without elastic regularization (whiteout) (top) and a keyframe with elastic regularization (whiteout) (bottom). Figure 4C shows a new figure synthesized without elastic regularization and a new figure synthesized with elastic regularization by linear interpolation of the observed deformation codes. Without elastic regularization, the intermediate state shows distortion, for example, the spacing between facial features changes from the original image.

[0095] Figure 5 shows examples of general-purpose computer devices 500 and general-purpose mobile computer devices 550 that may be used in conjunction with the technologies described herein.

[0096] As shown in Figure 5, computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 550 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are illustrative and not intended to limit the examples of the invention described and / or claimed herein.

[0097] The computing device 500 includes a processor 502, memory 504, a storage device 506, a high-speed interface 508 connected to memory 504 and a high-speed expansion port 510, and a low-speed bus 514 and a low-speed interface 512 connected to the storage device 506. Each of the components 502, 504, 506, 508, 510, and 512 may be interconnected using various buses and implemented on a common motherboard or otherwise as appropriate. The processor 502 can process instructions to be executed within the computing device 500, including instructions stored in memory 504 or the storage device 506, for displaying graphical information relating to a GUI on an external input / output device such as a display 516 coupled to the high-speed interface 508. In other implementations, multiple processors and / or multiple buses may be used as appropriate, along with multiple memories and multiple types of memory. Multiple computing devices 500 may also be connected, each device providing a required portion of the operation (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0098] Memory 504 stores information within the computing device 500. In one implementation, memory 504 is one or more volatile memory units. In another implementation, memory 504 is one or more non-volatile memory units. Memory 504 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0099] The storage device 506 can provide high-capacity storage to the computing device 500. In one implementation, the storage device 506 may be or include an array of devices, including computer-readable media such as floppy disk devices, hard disk devices, optical disk devices, or tape devices, or flash memory or other similar solid-state memory devices, or devices in a storage area network or other configuration. A computer program product can be tangibly embodied on an information carrier. A computer program product may also include instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer-readable or machine-readable medium such as memory 504, the storage device 506, or memory on the processor 502.

[0100] The high-speed controller 508 manages bandwidth-intensive operations for the computing device 500, while the low-speed controller 512 manages less bandwidth-intensive operations. Such function assignments are merely examples. In one implementation, the high-speed controller 508 is coupled to memory 504 and display 516 (e.g., via a graphics processor or accelerator), and to a high-speed expansion port 510 that can accept various expansion cards (not shown). In this implementation, the low-speed controller 512 is coupled to storage device 506 and low-speed expansion port 514. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth®, Ethernet®, Wireless Ethernet®), may be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, or networking devices such as a switch or router, for example, via a network adapter.

[0101] The computing device 500 can be implemented in several different forms, as shown in the figure. For example, the computing device 500 may be implemented as a standard server 520, or multiple times within a group of such servers. Alternatively, the computing device 500 may be implemented as part of a rack server system 524. In addition, the computing device 500 may be implemented in a personal computer such as a laptop computer 522. Alternatively, components from the computing device 500 may be combined with other components in a mobile device (not shown), such as device 550. Each such device may include one or more computing devices 500, 550, and the entire system may consist of multiple computing devices 500, 550 communicating with each other.

[0102] The computing device 550 includes, among other components, a processor 552, memory 564, input / output devices such as a display 554, a communication interface 566, and a transceiver 568. The device 550 may also include storage devices such as a microdrive or other devices to provide additional storage. Each of the components 550, 552, 564, 554, 566, and 568 are interconnected using various buses, and some of these components may be implemented on a common motherboard or in other appropriate manner.

[0103] The processor 552 can execute instructions, including those stored in memory 564, within the computing device 450. The processor may be implemented as a chipset of chips including multiple separate analog and digital processors. The processor can provide coordination with other components of the device 550, such as control of the user interface, applications run by the device 550, and wireless communication by the device 550.

[0104] The processor 552 may communicate with the user via a control interface 558 and a display interface 556 coupled to the display 554. The display 554 may be, for example, a thin-film transistor liquid crystal display (TFT LCD) or an organic light-emitting diode (OLED) display, or other suitable display technology. The display interface 556 may include appropriate circuitry for driving the display 554 to present graphical and other information to the user. The control interface 558 may receive commands from the user and translate such commands to present them to the processor 552. In addition, an external interface 562 communicating with the processor 552 may be provided to enable short-range communication between device 550 and other devices. The external interface 562 may, for example, provide wired communication in some implementations, or wireless communication in other implementations, and multiple interfaces may be used.

[0105] Memory 564 stores information within the computing device 550. Memory 564 can be implemented as one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 574 may be provided and connected to the device 550 via an expansion interface 572, which may include, for example, a Single In Line Memory Module (SIMM) card interface. Such an expansion memory 574 may provide additional storage space for the device 550 or store applications or other information for the device 550. Specifically, the expansion memory 574 may include instructions for executing or supplementing the processes described above, as well as secure information. For example, the expansion memory 574 may be provided as a security module for the device 550 and may be programmed with instructions that enable the secure use of the device 550. In addition, a secure application may be provided via the SIMM card along with additional information, such as by placing identification information on the SIMM card in a hack-proof manner.

[0106] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one implementation, the computer program product is tangibly embodied in an information carrier. When executed, the computer program product includes instructions that perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as memory 564, extended memory 574, or memory on the processor 552, which may be received, for example, via a transceiver 568 or an external interface 562.

[0107] Device 550 can communicate wirelessly via a communication interface 566, which may include digital signal processing circuitry as needed. The communication interface 566 may, among other things, provide communication under various modes or protocols such as GSM® voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA®, CDMA2000, or GPRS. Such communication may be performed, for example, via a radio frequency transceiver 568. In addition, short-range communication may be performed using Bluetooth®, WiFi, or other such transceivers (not shown). Furthermore, a Global Positioning System (GPS) receiver module 570 may provide device 550 with additional navigation-related and location-related radio data that may be used as appropriate by applications running on device 550.

[0108] Device 550 may also communicate in a speech-recognizable manner using an audio codec 560, which can receive speech information from a user and convert it into usable digital information. Similarly, the audio codec 560 may generate audible sound for the user, for example, through a speaker in the handset of device 550. Such sound may include sounds from a voice telephone call, recorded sounds (e.g., voice messages, music files, etc.), or sounds generated by an application running on device 550.

[0109] The computing device 550 can be implemented in several different forms, as shown in the figure. For example, the computing device 550 may be implemented as a mobile phone 580. The computing device 550 may also be implemented as part of a smartphone 582, a personal digital assistant, or other similar mobile device.

[0110] Various implementations of the systems and technologies described herein can be implemented in digital electronic circuits, integrated circuits, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive and transmit data and instructions to and from a storage system, at least one input device, and at least one output device.

[0111] These computer programs (also known as programs, software, software applications, or code) include machine instructions for programmable processors and may be implemented in high-level procedural and / or object-oriented programming languages, as well as in / or assembly / machine languages. The terms “machine-readable medium” and “computer-readable medium” as used herein refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0112] To provide user interaction, the systems and technologies described herein can be implemented on a computer having a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) that the user can use to provide input to the computer. User interaction can be performed using other types of devices, for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback or tactile feedback), and input from the user may be received in any form, including acoustic input, voice input or tactile input.

[0113] The systems and technologies described herein can be implemented in a computing system that includes backend components (e.g., as data servers), middleware components (e.g., application servers), or frontend components (e.g., a client computer having a graphical user interface or web browser that enables a user to interact with an implementation of the systems and technologies described herein), or any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0114] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between a client and a server arises from computer programs that run on each computer and have a client-server relationship with each other.

[0115] Returning to Figure 1, in some implementations, memory 126 can be any type of memory, such as random access memory, disk drive memory, or flash memory. In some implementations, memory 126 can be implemented as multiple memory components (e.g., multiple RAM components or disk drive memory) associated with the components of the compression computer 120. In some implementations, memory 126 can be database memory. In some implementations, memory 126 can be or contain non-local memory. For example, memory 126 can be or contain memory shared by multiple devices (not shown). In some implementations, memory 126 can be associated with a server device (not shown) in a network and can be configured to function for the components of the compression computer 120.

[0116] The components of the compression computer 120 (e.g., modules, processing units 124) can be configured to operate on one or more platforms (e.g., one or more similar or different platforms) which may include one or more types of hardware, software, firmware, operating systems, runtime libraries, etc. In some implementations, the components of the compression computer 120 may be configured to operate within a cluster of devices (e.g., a server farm). In such implementations, the functions and processing of the components of the compression computer 120 may be distributed among some of the devices in the cluster.

[0117] The components of computer 120 may be or include any type of hardware and / or software configured to process attributes. In some implementations, one or more of the components shown in Figure 1 of computer 120 may be or include hardware-based modules (e.g., digital signal processors (DSPs), field-programmable gate arrays (FPGAs), memory), firmware modules, and / or software-based modules (e.g., modules of computer code, sets of computer-readable instructions executable in a computer). For example, in some implementations, one or more of the components of computer 120 may be or include software modules configured for execution by at least one processor (not shown). In some implementations, the functionality of the components may be contained in modules and / or components different from those shown in Figure 1.

[0118] Although not illustrated, in some implementations, components (or parts thereof) of computer 120 may be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, or one or more server / host devices. In some implementations, components (or parts thereof) of computer 120 may be configured to operate within a network. Therefore, components (or parts thereof) of computer 120 can be configured to function within various types of network environments, which may include one or more devices and / or one or more server devices. For example, the network may be or include a local area network (LAN), a wide area network (WAN), etc. The network may be or include a wireless network, and / or may be or include a wireless network implemented using, for example, a gateway device, a bridge, a switch, etc. The network may include one or more segments and / or parts based on various protocols such as Internet Protocol (IP) and / or proprietary protocols. The network may include at least a portion of the Internet.

[0119] In some embodiments, one or more components of the computer 120 may be or include a processor configured to process instructions stored in memory. For example, the depth image manager 130 (and / or part thereof), viewpoint manager 140 (and / or part thereof), ray projection manager 150 (and / or part thereof), SDV manager 160 (and / or part thereof), aggregation manager 170 (and / or part thereof), root finding manager 180 (and / or part thereof), and depth image generation manager 190 (and / or part thereof) may be combinations of a processor and memory configured to execute instructions related to a process for realizing one or more functions.

[0120] While numerous embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of this specification.

[0121] Furthermore, when an element is referred to as being on another element, connected to another element, electrically connected to another element, coupled to another element, or electrically coupled to another element, it will be understood that the element may be directly on another element, connected to another element, or coupled to another element, or there may be one or more intervening elements. In contrast, when an element is referred to as being directly on another element, directly connected to another element, or directly coupled to another element, there are no intervening elements. The expressions directly on, directly connected to, or directly coupled may not be used throughout the detailed description, but elements shown as directly on, directly connected to, or directly coupled may be referred to as such. The claims of this application may be amended to describe exemplary relationships described herein or shown in the figures.

[0122] While some features of the described embodiments are illustrated as described herein, those skilled in the art will likely envision many modifications, alternatives, alterations, and equivalents. It should be understood that the appended claims are intended to encompass all such modifications and alterations so that they fall within the scope of the embodiments. These are presented only as examples and not limitations, and various changes in form and detail are possible. Any part of the apparatus and / or method described herein may be combined in any combination, except for mutually exclusive combinations. The embodiments described herein may include various combinations and / or partial combinations of the functions, components, and / or features of the various embodiments described.

[0123] Furthermore, the logical flow shown in the diagram does not require a specific order or sequence to achieve the desired result. In addition, other steps may be provided, or steps may be removed from the described flow, and other components may be added to or removed from the described system. Accordingly, other embodiments are within the scope of the appended claims.

Claims

1. A computer-implemented method for generating a system for synthesizing a new view of a scene, comprising: The method includes acquiring image data representing a plurality of images, each of the plurality of images including an image of a scene, the scene including an object viewed from each of a plurality of points, the method further comprising: generating a deformation model based on the image data, the deformation model describing movements made by the object while the image data was being generated, the deformation model mapping a position of each point in the scene to a position distorted by the movements made by the object, the method further comprising: and generating a neural radiance field model (NeRF) that outputs the new view from a viewpoint of the scene based on the position and viewing direction of each of the plurality of points of the scene, the positions of which are distorted by the movement performed by the object described by the deformation model.

2. The method of claim 1 , wherein the deformation model is modeled for multiple time steps using latent deformation codes associated with frames that provide a view of the scene from the viewpoint.

3. The method of claim 1 , wherein the deformation model includes a rotation, a center point corresponding to the rotation, and a translation.

4. The method of claim 3 , wherein the rotation is encoded as the logarithm of a quaternion.

5. The method of claim 3 , wherein the deformation model includes a sum of (i) a similarity transformation for the difference between the position of each of the points of the scene and the center point, (ii) the center point, and (iii) the translation.

6. The method of claim 1 , wherein the deformation model comprises a multilayer perceptron (MLP) in a neural network.

7. The method of claim 6 , wherein a loss function for the MLP is based on a norm of a matrix representing the deformation model.

8. The method of claim 7 , wherein the matrix is ​​a Jacobian.

9. The method of claim 7 , wherein the loss function is based on the norm of a second of three matrices obtained by singular value decomposition of the matrix representing the deformation model.

10. The method of claim 9 , wherein the loss function is based on the logarithm of the norm of the second of the three matrices obtained by the singular value decomposition.

11. The method of claim 7 , wherein the loss function comprises a rational function.

12. The method of claim 6 , wherein a background loss function component for the MLP includes designating points in the scene as static points with a penalty for motion.

13. The method of claim 12 , wherein the background loss function components are based on differences between static points and points to which the static points are mapped according to the deformation model.

14. 7. The method of claim 6, wherein generating the deformation model comprises applying a position encoding to position coordinates in the scene to generate a periodic function of position, the periodic function having a frequency that increases with training iterations for the MLP.

15. The method of claim 14, wherein the step of generating the deformation model includes multiplying the periodic function of the position encoding by a weight indicating whether a training iteration includes a particular frequency.

16. A computer program which, when executed by processing circuitry of a computing device, causes the processing circuitry to perform a method according to any one of claims 1 to 15.

17. 1. An electronic device comprising: A memory for storing the computer program of claim 16; and control circuitry coupled to the memory, the control circuitry configured to execute the computer program stored in the memory.

Images