View synthesis robust to unconstrained image data

Abstract

To provide a system and a method for synthesizing novel views of complex scenes (e.g., outdoor scenes).SOLUTION: In some implementations, a system and a method can include or use a machine-learned model capable of learning from unstructured and / or unconstrained collections of imagery such as, for example, "in the wild" photographs. In particular, example implementations of the present disclosure can learn a volumetric scene density and radiance represented by a machine-learned model, such as one or more multilayer perceptrons (MLPs).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] Related Applications This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 059,322, filed July 31, 2020. U.S. Provisional Patent Application No. 63 / 059,322 is incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to systems and methods for modeling a scene, for example to facilitate performing viewpoint synthesis of the scene. More particularly, the present disclosure relates to machine learning models that provide improved robustness against ubiquitous real-world phenomena in uncontrolled images, such as variable illumination and temporary occlusions. [Background technology]

[0003] The task of view synthesis aims to generate a new view of a particular environment (e.g., a scene, object, or subject) starting from several photographs taken from a given viewpoint. As an example, given several images of one particular subject captured from a particular point with a particular camera setting and orientation, a view synthesis system tries to generate a synthetic image as if it were captured from a virtual camera placed at a different point and with the given settings.

[0004] Although several approaches to view synthesis leverage neural brightness fields to effectively learn volumetric scene density and brightness from images captured in controlled environments, these approaches cannot model many ubiquitous real-world phenomena in uncontrolled images, such as variable illumination and temporary occlusions. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Representing Scenes as Neural Radiance Fields for View Synthesis by Mildenhall et al. (arXiv:2003.08934v1) Summary of the Invention [Means for solving the problem]

[0006] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned through practice of the embodiments.

[0007] An exemplary aspect of the present disclosure is directed to a computing system for generating a synthetic image of a scene. The computing system includes one or more processors and one or more non-transitory computer-readable media, which collectively store a machine-learned viewpoint synthesis model including a base, a static content portion that models static content in the scene, and a temporary content portion that models temporary content in the scene, and instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations include obtaining a position in a three-dimensional space, processing data describing the position with the base of the machine-learned viewpoint synthesis model to generate a static opacity and a latent representation, processing the latent representation with the static content portion of the machine-learned viewpoint synthesis model to generate a static color, processing the latent representation with the temporary content portion of the machine-learned viewpoint synthesis model to generate a temporary opacity and a temporary color, and performing volume rendering to generate a synthetic pixel color for a synthetic pixel of the synthetic image from the static opacity, the static color, the temporary opacity, and the temporary color.

[0008] Another exemplary aspect of the present disclosure is directed to a computer-implemented method for viewpoint synthesis with user-specifiable characteristics. The method includes obtaining a desired location in three-dimensional space and a user-specified generative embedding by a computing system including one or more computing devices, where the generative embedding encodes one or more visual characteristics of a generated synthetic image. The method includes processing data describing the location with a base of a machine-learned viewpoint synthesis model to generate an opacity and a latent representation by the computing system. The method includes processing the latent representation and the generative embedding by the computing system with a content portion of the machine-learned viewpoint synthesis model to generate a color. The method includes performing volume rendering by the computing system to generate a synthetic pixel color for a synthetic pixel of the synthetic image from the opacity and color, where the synthetic image exhibits the one or more visual characteristics encoded by the generative embedding.

[0009] Another exemplary aspect of the present disclosure is directed to one or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more processors, cause a computing system to perform operations. The operations include obtaining, by the computing system, a location in three-dimensional space and one or more camera parameters associated with an existing training image. The operations include processing, by the computing system, data describing the location and the one or more camera parameters using a base of a machine-learned viewpoint synthesis model to generate an opacity and a latent representation. The operations include processing, by the computing system, the latent representation using a content portion of the machine-learned viewpoint synthesis model to generate a color. The operations include performing, by the computing system, volume rendering to generate a synthetic pixel color for a synthetic pixel of the synthetic image from the opacity and the color. The operations include evaluating a loss function that compares the synthetic pixel color to a ground truth pixel color for a training pixel included in the existing training image. The operations include modifying one or more values ​​of the camera parameters based at least in part on the loss function.

[0010] Another exemplary aspect of the present disclosure is directed to a computing system for generating a synthetic image of a scene. The computing system includes one or more processors and one or more non-transitory computer-readable media, which collectively store a machine-learned viewpoint synthesis model including a base and a static content portion that models static content in the scene, the base and the static portion trained together with a temporary content portion that models temporary content in the scene, and instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations include obtaining a position in a three-dimensional space, processing data describing the position with the base of the machine-learned viewpoint synthesis model to generate a static opacity and a latent representation, processing the latent representation with the static content portion of the machine-learned viewpoint synthesis model to generate a static color, and performing volume rendering to generate a synthetic pixel color for a synthetic pixel of the synthetic image from the static opacity and the static color.

[0011] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0012] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain associated principles.

[0013] Detailed descriptions of embodiments directed to persons skilled in the art are set forth herein and refer to the accompanying figures. [Brief description of the drawings]

[0014] [Figure 1]FIG. 1 is a block diagram of an example process for training a machine-learned view synthesis model and then performing view synthesis using the machine-learned view synthesis model according to an example embodiment of the present disclosure. [Diagram 2] FIG. 1 is a block diagram of an example process for training a machine-learned view synthesis model according to an example embodiment of the present disclosure. [Diagram 3] FIG. 1 is a block diagram of an example process for using a machine-learned view synthesis model according to an example embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram of an exemplary machine-learned view synthesis model in accordance with an exemplary embodiment of the present disclosure. [Figure 5A] FIG. 1 is a block diagram of an exemplary computing system in accordance with an exemplary embodiment of the present disclosure. [Figure 5B] FIG. 2 is a block diagram of an exemplary computing device in accordance with an exemplary embodiment of the present disclosure. [Figure 5C] FIG. 2 is a block diagram of an exemplary computing device in accordance with an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Reference numbers that are repeated among the figures identify like features in various implementations.

[0016] overview In general, the present disclosure is directed to systems and methods for synthesizing novel perspectives of complex scenes (e.g., outdoor scenes). In some implementations, the systems and methods can include or use machine-learned models that can learn from unstructured and / or unconstrained collections of images, such as "wild" photographs. In particular, example implementations of the present disclosure can learn volumetric scene density and brightness represented by machine-learned models, such as one or more multilayer perceptrons (MLPs) or other neural networks or other machine-learned models.

[0017] More specifically, while some techniques, such as those described in NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (arXiv:2003.08934v1) by Mildenhall et al. (hereafter "NeRF"), work well with images captured in controlled environments, they cannot model many ubiquitous real-world phenomena in uncontrolled images, such as variable illumination and temporary occlusions.

[0018] Specifically, NeRF is built on the assumption of 3-D consistency, which assumes that two photos of the same scene should match. However, in uncontrolled environments (e.g., "wild" photos), two images of the same scene often have inconsistencies, including inconsistencies in lighting or other visual characteristics and / or the presence / absence of temporal content / items (e.g., two photos of a landmark may each have a different set of temporal tourists positioned in front of the landmark).

[0019] NeRF learns a model for each scene described by a (learned) function. This function takes in a position (x, y, z) and a viewing direction (theta, phi) to produce an RGB color and opacity (sigma). To produce a render, a ray is traced through each pixel in the camera and the color is integrated weighted by the opacity accumulated along the length of the ray. The NeRF model is trained to minimize the squared error between the predicted RGB values ​​and the ground truth image. Thus, to train NeRF, a photograph and camera parameters are required. The photograph describes what color each pixel should be, and the camera parameters describe where in 3-D space the rays emanating from that pixel should lie.

[0020] This disclosure provides several techniques to improve the NeRF model or similar models. These additions allow the proposed model to be more robust to cluttered real-world photographs.

[0021] In particular, according to one aspect of the present disclosure, during training of a machine-learned viewpoint synthesis model on a set of training images, each training image may be assigned a generative embedding, which may serve to encode visual characteristics of the image, such as lighting characteristics (e.g., white balance), content characteristics (e.g., time of day, weather, season), style characteristics (e.g., photo editing software settings), etc.

[0022] Thus, in some implementations, a small generative embedding (e.g., a vector of 8-32 floats) may be assigned to each training image (e.g., indexed by image ID). The use of such generative embeddings allows the model to account for variations in individual images, such as white balance and stylistic post-processing.

[0023] These generative embeddings can be learned / updated as part of the model. For example, the generative embeddings can be treated as model parameters and updated during model training (e.g., during or as a result of backpropagation of a loss function). Through such a process, generative embeddings for images with similar properties can be moved closer to each other.

[0024] Then, during viewpoint synthesis, desired generative embeddings that encode desired characteristics (e.g., desired lighting characteristics, content characteristics, style characteristics, etc.) may be specified by a user and fed to the model. In response, the model may generate a synthetic image having the desired characteristics encoded by the desired generative embeddings. In one example, the desired generative embeddings may be obtained by interpolating two or more generative embeddings learned for two or more of the training images selected by the user (e.g., because they exhibit / represent the desired characteristics).

[0025] Thus, the model can use generative embedding to capture differences in image characteristics that do not reflect actual differences in the underlying scene. For example, differences in post-processing settings for images of landmarks do not actually reflect differences in the landmarks themselves. This allows the trained model to better handle inconsistencies between training images, which results in the model being more robust to unconstrained sets of training images.

[0026] According to another example aspect, some implementations of the present disclosure incorporate uncertainty directly into the volume rendering pipeline of the model. For example, in addition to static opacity and static color, the model can also predict an uncertainty value "β", a temporal opacity, and a temporal color. The temporal data can be integrated to be performed on the static data to obtain a learned weight multiplier for each ray. Intuitively, this allows the model to deprioritize reconstructing "difficult" parts of the image that are inconsistent with other photos (e.g., often corresponding to temporary and inconsistent occlusions in the image).

[0027] Thus, exemplary implementations of the present disclosure have a novel model architecture that includes a base, a static content part that models static content in a scene, and a temporary content part that models temporary content in a scene. In some implementations, the loss function on which the model is trained can provide a reduced amount of loss for examples in which the difference between the predicted synthesis color and the ground truth color arises from the temporary content modeled by the viewpoint synthesis model. Thus, the model has the ability to deviate from the ground truth image when the ground truth image shows temporary content or is possibly associated with high uncertainty. This allows the trained model to better handle discrepancies between training images, which results in the model being more robust to a set of unconstrained training images.

[0028] According to another exemplary aspect, some implementations of the present disclosure can directly fine-tune the camera parameters of each training image in the model itself. Exemplary camera parameters include orientation, location, focal length, principal point, skew, radial distortion, tangential distortion, and / or various camera intrinsics. This allows the model to adjust the camera parameters (e.g., within some threshold or tolerance) to perfectly match the scene being reconstructed, and does not rely on sparse image features that are commonly used in 3-D registration. In other words, by allowing the camera parameters for each training image to be modified, some amount of noise present in the training dataset can be removed from the dataset. This allows the trained model to better handle inconsistencies between training images, which results in the model being more robust to a set of unconstrained training images.

[0029] The systems and methods of the present disclosure provide several technical effects and benefits. As an example, the systems and methods described herein enable models to be more robust to uncontrolled images. This results in improved model performance when synthesizing viewpoints of scenes based on uncontrolled training datasets. Thus, the systems and methods of the present disclosure improve the realism, consistency, and user controllability of viewpoint synthesis systems.

[0030] Another exemplary technical effect and benefit of providing a model with an explicit mechanism for understanding and / or correcting inconsistencies in a training dataset is that the model can converge to an optimal solution faster (e.g., with fewer training iterations and / or fewer training images). Faster convergence can result in savings in computational resources, such as reduced processor usage, memory usage, and / or bandwidth usage.

[0031] Thus, the present disclosure provides systems and methods that solve problems associated with unconstrained images and / or provide additional user controllability. Exemplary experimental data included in U.S. Provisional Patent Application No. 63 / 059,322 demonstrates the effectiveness of the proposed techniques through comprehensive synthetic and real-world experiments. The exemplary experiments apply exemplary implementations of the systems described herein to multiple landmarks using both high-resolution captures and internet photographs, resulting in photorealistic reconstructions and results that significantly surpass previous work.

[0032] Although example implementations of the present disclosure are described with respect to a single frame of an image, the systems and methods of the present disclosure may be equally applied to videos or other multi-frame datasets. For example, a video showing a "flyover" of a modeled scene may be synthesized. The frames of the video may be temporally and stylistically consistent (e.g., via consistent use of the same desired generative embeddings for all of the frames of the synthesized video).

[0033] In addition, the synthesized image frames may be of the same or different resolution as the training images on which the model is trained. For example, a model may be trained on low-resolution images and then used to generate high-resolution synthetic images.

[0034] Color data used and / or generated by the systems and methods of the present disclosure (e.g., input data colors, static colors, temporary colors, composite colors, etc.) may be represented in any color space, including, for example, RGB spaces (e.g., RGB, RGBA), CIE spaces, HSV and HSL spaces, CMYK spaces, grayscale, and / or other color spaces.

[0035] Exemplary embodiments of the present disclosure will now be described in further detail with reference to the figures. Example Techniques

[0036] This section describes an exemplary implementation of the proposed approach, which in some implementations may be referred to as NeRF-W. The model described is well designed for reconstructing 3-D scenes from "in the wild" photo collections, and improves upon the NeRF model described by Mildenhall et al.

[0037] At its core, NeRF relies on multi-view consistency: every point in 3-D space must look the same in all (unobstructed) viewpoints. This condition is relaxed depending on the viewing direction, but significant variations in color are not something NeRF can capture. In particular, NeRF generally handles color variations by correlating these variations with viewing angle. So while a single image may look normal, a validation view or set of multiple images from different viewing angles (e.g., a fly-through video) will not match.

[0038] When photographs are captured in a controlled setting by a single person, the NeRF assumptions are satisfied and photo-realistic reconstruction is achievable. However, unconstrained photo collections, such as amateur photographs of famous landmarks, present many challenges that NeRF cannot capture.

[0039] As an example, unconstrained images may show the same scene but in different weather: the photos are taken at different times under variable lighting conditions. In outdoor photos, the time of day and the weather directly affect the color of all objects in the scene. Furthermore, the sky itself changes over time.

[0040] As another example, unconstrained images may show the same scene but with different post-processing, where post-processing of the photo including exposure and white balance adjustments is not constrained at all and further affects the color of all objects in the scene.

[0041] As yet another example, unconstrained images may show the same scene but with different temporary objects, and because the photographs were not captured at a single point in time, the temporary objects, including people, flags, and foliage, vary from image to image.

[0042] In the discussion that follows (and also in the summary above), the present disclosure proposes several enhancements that are directly designed to address these phenomena.

[0043] FIG. 1 illustrates a block diagram of an example process for training a machine-learned view synthesis model and then performing view synthesis using the machine-learned view synthesis model, according to an example embodiment of the present disclosure.

[0044] With reference to FIG. 1, the training dataset 12 may include existing training images showing a scene. As described herein, the training images are unconstrained and may exhibit various inconsistencies with one another. As shown at 14, a computing system may perform a model optimization or training process on the training dataset 12 to generate a machine-learned viewpoint synthesis model 16 (see, e.g., FIG. 2). After training, a location 18 of a desired synthetic image may be provided to the model 16. In response, the model 16 may generate a synthetic image 20 showing the scene from the location 18 (see, e.g., FIG. 3).

[0045] 2 illustrates a block diagram of an example process for training a machine-learned view synthesis model according to an example embodiment of the present disclosure. In some implementations, the process illustrated in FIG. 2 may be performed for each pixel of each training image.

[0046] 2, training positions 22 of existing training images may be provided to the machine-learned view synthesis model 24. The positions 22 may include the location and orientation of a camera that took the training images. Additionally, according to aspects of the present disclosure, in some implementations, one or more camera parameters 25 for the training images and / or training image embeddings 26 for the training images may be provided to the machine-learned view synthesis model 24. By way of example, the additional camera parameters 25 may include focal length, principal point, skew, radial distortion, tangential distortion, and / or various camera intrinsics. The training image embeddings 26 may be generative embeddings assigned to the training images.

[0047] In particular, central to the challenges presented by "in the wild" images is the concept of image-to-image color variation; while the 3-D geometry of the scene is assumed to be identical across all images, little color consistency is expected due to variations in camera settings such as lighting and exposure.

[0048] To solve this problem, in some implementations, each image in the training set is assigned a unique embedding.

[0049]

number

[0050] 26. These embeddings

[0051]

number

[0052] can be optimized during training together with the parameters of the model.

[0053] 2, the machine-learned viewpoint synthesis model 24 can process the input data to generate opacity and color data 27. For example, in some implementations, the machine-learned viewpoint synthesis model 24 can generate only a single set of opacity and color data 27, or in other implementations, the machine-learned viewpoint synthesis model 24 can generate both a static set of opacity and color data for static content of the scene and a temporary set of color and opacity data 27 for temporary content of the scene.

[0054] As an example, the differential opacity σ(r) and color c(r,d) 27 may be predicted by a multi-layer perceptron (MLP) or other model (e.g., some other form of neural network or other machine-learned model) given the 3-D location r(t) and gaze direction d. This MLP or other model may be explicitly designed to ensure that gaze direction d does not affect the differential opacity σ. For example, the basis of the model may be to predict opacity only from location, while color may be predicted from both location and viewing angle / gaze direction. Again, in some implementations, the input of this MLP may be the embedding e (g) can be increased by (c, σ) = MLP(r(t), d, e (g) ) where e (g) is the generative embedding that corresponds to the image to be rendered. Similar to the viewing direction d, some example implementations use a generative embedding e (g) does not affect the differential opacity σ.

[0055]

number

[0056] By increasing the,r,, our model is able to directly vary the color and lighting of a scene based on the image identity without modifying its 3-D geometry.

[0057] At 28, volume rendering techniques may be performed to generate a composite pixel color from the opacity and color data 27. For example, for a single set of opacity and color data 27, the composite pixel color may be obtained by integrating along rays emanating from the camera.

[0058]

number

[0059] According to another exemplary aspect, GLO can capture variable lighting and post-processing, but cannot model variations in 3-D geometry. Thus, some implementations of the present disclosure feature a double-headed model that includes both a static part that models static content and a temporal part that models temporal content. Unlike implementations that create a single tuple (σ, c) for a position in 3-D space, the proposed model with both static and temporal heads has one “static” object (σ, c) that is common to all images. s , c s ), and another for “temporary” objects (σt , c t ), we create two heads for the vectors, β, and β, respectively. These quantities can be combined with a modified version of the volume rendering equation presented in Equation 1. Furthermore, the proposed model outputs an uncertainty estimate β, which is used to modulate the loss function for each pixel. Figure 4 provides an illustration of the architecture of the proposed model.

[0060] 4, a block diagram of an exemplary machine-learned viewpoint synthesis model is provided in accordance with an exemplary embodiment of the present disclosure. The model can include a base 34, a static portion 36, and a temporary portion 38.

[0061] 3-D point r(t) 40, GLO embedding e (g) 48, and Uncertainty Embedding e (u) 52, the model shown in Figure 4 calculates the differential opacity σ s , σ t 44, 54, color c s , c t 45, 56, and the differential uncertainty β 58. Position embedding, gaze direction, and nonlinearities are omitted for clarity. As above, additional camera parameters 42 may be optionally provided as well.

[0062] In some implementations, the base 34 of the exemplary proposed model includes an MLP applied to the 3-D points r(t) 40. Other models may be used as well. The MLP is a function of the differential opacity σ s 44 and outputs the latent representation z46. The latent representation z46 is employed in two ways. First, it is used to represent the color c s The static part 36 (which may include, for example, a four-layer MLP or other model) for z 46 and viewing direction d to create 45. The second is the temporal differential opacity σ t 54, Temporary Color C t 56, and the uncertainty value β58, to create three quantities: the embedding for each image, e (u)52 (e.g., it may include a second 4-layer MLP or other model). In some implementations, the l1 regularization is t can be applied to.

[0063] 2 and 4 together, to produce a color at 28 for a single pixel from both static and transient data 27, the following modification of the volume rendering equation:

[0064]

number

[0065]

number

[0066] may be used.

[0067] The color c in Equation 1 is σ s c s +σ t c t and the accumulated opacity σ is s +σ t Note that, , is replaced by the sum of

[0068] In some implementations, such as the example shown in Figure 4, the model is allowed to emit an uncertainty estimate β 58. At training time, the accumulated opacity equations can be used to obtain the uncertainty for the corresponding predicted color C(r).

[0069]

number

[0070] As an example, the loss for a single pixel with ground truth color y may therefore be given by:

[0071]

number

[0072] Intuitively, a larger value of β allows the model to de-weight outliers - typically transient or moving objects such as people, grass, or clouds. The logarithmic term prevents β from growing unboundedly and can be derived directly from the likelihood of a normal distribution. In some implementations, the hyperparameter β is set to 0 to prevent the model from concentrating most of the loss on a small number of pixels. min ≧0 may be used.

[0073] More generally, and referring again to Figure 2, a loss function 30 may evaluate the difference between the synthetic pixel colors generated at 28 and the ground truth pixel colors 32 of existing training images. For example, the squared error between pixel colors expressed in RGB or some other color scheme may be used.

[0074] The loss function 30 may be backpropagated to train the machine-learned viewpoint synthesis model 24. Additionally, in some implementations, the training image embeddings 26, the training positions 22, and / or the camera parameters 25 may be updated based on the loss function 30 (e.g., by continuing to backpropagate the loss through and past the model 24) as well.

[0075] At test time, the model 24 can be used to render a common static geometry for all photos in the training set. In some implementations, the images are t , c t , and β may be rendered by omitting all of them.

[0076] As an example, Figure 3 illustrates an exemplary use of a machine-learned viewpoint synthesis model after training. In particular, a desired position 40 (e.g., location and orientation) for a synthetic image of a scene is provided. Optionally, desired camera parameters 42 and / or a desired generative embedding 44 may be provided as well.

[0077] The machine-learned viewpoint synthesis model 24 can process the input to generate opacity and color data 27 (e.g., a single set of opacity and color data, or both static and temporary opacity and color data, or simply static opacity and color data). Volume rendering 28 can be performed on the opacity and color data (e.g., only static data) to generate synthetic pixel colors for pixels of the synthetic image.

[0078] The process shown in FIG. 3 may be performed for each pixel of the composite image. Exemplary Devices and Systems

[0079] 5A illustrates a block diagram of an exemplary computing system 100 according to an exemplary embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled via a network 180.

[0080] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0081] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0082] In some implementations, the user computing system 102 can store or include one or more machine-learned models 120. For example, the machine-learned models 120 can be or otherwise include various machine-learned models, such as neural networks (e.g., deep neural networks) or other types of machine-learned models including nonlinear and / or linear models. The neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Exemplary machine-learned models 120 are discussed with reference to FIGS. 1-4.

[0083] In some implementations, one or more machine-learned models 120 may be received from the server computing system 130 over the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single machine-learned model 120 (e.g., to perform parallel view synthesis across multiple examples of the same or different scenes).

[0084] Additionally or alternatively, one or more machine-learned models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine-learned models 140 may be implemented by the server computing system 140 as part of a web service (e.g., a view synthesis service). Thus, one or more models 120 may be stored and implemented at the user computing device 102 and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0085] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may be useful for implementing a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0086] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0087] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0088] As described above, the server computing system 130 may store or otherwise include one or more machine-learned models 140. For example, the models 140 may be or otherwise include a variety of machine-learned models. Exemplary machine-learned models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feed-forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Exemplary models 140 are discussed with reference to FIGS. 1-4.

[0089] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 by interacting with a training computing system 150 that is communicatively coupled via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0090] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operably connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0091] The training computing system 150 may include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored in the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation. For example, a loss function may be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters for several training iterations.

[0092] In some implementations, performing the error backpropagation may include performing abbreviated temporal backpropagation. The model trainer 160 can implement some generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0093] In particular, model trainer 160 can train machine learned models 120 and / or 140 based on a set of training data 162. Training data 162 can include, for example, unconstrained image data such as “in the wild” photographs.

[0094] In some implementations, if the user provides consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 against user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0095] The model trainer 160 includes computer logic used to provide the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some implementations, the model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium, such as a RAM, a hard disk, or an optical or magnetic medium.

[0096] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over network 180 may be carried over any type of wired and / or wireless connections, using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0097] 5A illustrates one exemplary computing system that may be used to implement the present disclosure. Other computing systems may be used as well. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training dataset 162. In such implementations, the model 120 may be both trained and used locally on the user computing device 102. In some such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0098] 5B illustrates a block diagram of an exemplary computing device 10 for performing according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0099] The computing device 10 includes several applications (e.g., applications 1-N). Each application includes its own machine learning library and machine-learned models. For example, each application may include a machine-learned model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0100] 5B, each application may communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application may communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0101] 5C illustrates a block diagram of an exemplary computing device 50 for performing according to an exemplary embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0102] Computing device 50 includes several applications (e.g., applications 1-N). Each application communicates with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0103] The central intelligence layer includes several machine-learned models. For example, as shown in FIG. 5C, a respective machine-learned model (e.g., model) may be provided to each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine-learned model. For example, in some implementations, the central intelligence layer may provide a single model (e.g., single model) to all of the applications. In some implementations, the central intelligence layer is included in or otherwise implemented by the operating system of the computing device 50.

[0104] The central intelligence layer can communicate with a central device data layer, which can be a centralized repository of data for computing device 50. As shown in FIG. 5C, the central device data layer can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API). Additional Disclosures

[0105] The technology discussed herein refers to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among components. For example, the processes described herein may be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0106] Although the present subject matter has been described in detail with respect to various specific exemplary embodiments thereof, each example is provided as an explanation, not a limitation of the present disclosure. Upon understanding the above, a person skilled in the art can easily create modifications, variations, and equivalents of such embodiments. Thus, the present disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the present subject matter as would be readily apparent to a person skilled in the art. For example, a feature shown or described as part of one embodiment may also be used with another embodiment to produce a further embodiment. Thus, it is intended that the present disclosure covers such modifications, variations, and equivalents. [Explanation of symbols]

[0107] 10. Computing Devices 12 Training Data Set 14 Model optimization or training process 16 Machine-learned view synthesis model 18 Composite Images 20 Composite Image 22 Training positions 24 Machine-learned viewpoint synthesis model 25 Camera parameters 26 Training Image Embedding 27 Opacity and Color Data 28 Volume Rendering 30 Loss Function 32 ground truth pixel colors 34 Base 36 Static part 38 Temporary part 40 3-D points 42 Additional Camera Parameters 44 Differential opacity 46 colors 48 GLO embedding 50 Computing Devices 52 Embedding Uncertainty 54 Differential Opacity 56 colors 58 Differential Uncertainty 100 Computing Systems 102 User Computing Devices 112 processors 114 Memory 116 Data 118 Command 120 Machine Learning Models 122 User Input Components 130 Server Computing System 132 processors 134 Memory 136 Data 138 Command 140 Machine Learning Models 150 Training Computing System 152 processors 154 Memory 156 Data 158 Command 160 Model Trainer 162 training data set 180 Network

Claims

1. 1. A computing system for generating a synthetic image of a scene, comprising: one or more processors; and one or more non-transitory computer-readable media, the one or more non-transitory computer-readable media comprising: a machine-learned viewpoint synthesis model including a base, a static content portion that models static content in the scene, and a temporary content portion that models temporary content in the scene; and instructions, collectively, that when executed by the one or more processors cause the computing system to perform operations, the operations including: Obtaining a position in three-dimensional space; processing the data describing the positions with the base of the machine-learned viewpoint synthesis model to generate static opacity and latent representations; processing the latent representation with the static content portion of the machine-learned view synthesis model to generate static colors; processing the latent representation with the temporal content portion of the machine-learned view synthesis model to generate a temporal opacity and a temporal color; and performing volume rendering to generate a composite pixel color for a composite pixel of a composite image from the static opacity, the static color, the temporary opacity, and the temporary color.

2. the locations in three-dimensional space include capture locations relative to existing training images; The operation includes: evaluating a loss function that compares the synthetic pixel colors to ground truth pixel colors for training pixels in the existing training images; and modifying one or more parameter values ​​for one or more parameters of the machine-learned view synthesis model based at least in part on the loss function.

3. processing the latent representation with the temporal content portion of the machine-learned view synthesis model further produces an uncertainty value; The computing system of claim 2 , wherein the loss function includes a pixel loss term that provides a negative correlation between a loss magnitude and the uncertainty value.

4. The operation includes:

4. The computing system of claim 1, further comprising: inputting an uncertainty embedding into the temporal content portion of the machine-learned viewpoint synthesis model alongside the latent representation to generate the temporal opacity and the temporal color.

5. The operation includes:

5. The computing system of claim 2, 3 or 4, further comprising inputting generative embeddings into the static content portion of the machine-learned viewpoint synthesis model alongside the latent representation to generate the static colors.

6. The generative embedding is associated with the existing training images; 6. The computing system of claim 5, wherein the operations further include modifying one or more values ​​of the generative embedding based at least in part on the loss function.

7. The operation includes: inputting one or more camera parameters into the base of the machine-learned viewpoint synthesis model in parallel with the position to generate the static opacity and the latent representation; and modifying one or more of the camera parameters based at least in part on the loss function.

8. A computing system according to claim 1 , further comprising: a computing system configured to perform each of said operations for each respective pixel in said composite image.

9. The computing system of claim 1 , wherein the position includes a location and an orientation.

10. 10. The computing system of claim 1, 8 or 9, wherein the positions include new positions not included in a training set on which the machine-learned view synthesis model was trained.

11. 11. The computing system of claim 1, wherein the base portion of the machine-learned viewpoint synthesis model, the static content portion, and the temporary content portion each include a respective multi-layer perceptron.

12. 1. A computer-implemented method for view synthesis with user-specifiable properties, comprising: obtaining, by a computing system including one or more computing devices, a desired location in three-dimensional space and a user-specified generative embedding, the generative embedding encoding one or more visual characteristics of a synthetic image to be generated; processing, by the computing system, the data describing the positions using a machine-learned viewpoint synthesis model basis to generate opacity and latent representations; processing, by the computing system, the latent representation and the generative embedding using a content portion of the machine-learned view synthesis model to generate color; and performing, by the computing system, volume rendering to generate a synthetic pixel color for a synthetic pixel of the synthetic image from the opacity and the color, wherein the synthetic image exhibits the one or more visual characteristics encoded by the generative embedding.

13. the opacity produced by the base includes a static opacity; processing, by the computing system, the latent representation and the generative embedding using the content portion of the machine-learned view synthesis model to generate the color; processing, by the computing system, the latent representation and the generative embedding using a static content portion of the machine-learned view synthesis model to generate static colors; processing, by the computing system, the latent representation using a temporal content portion of the machine-learned view synthesis model to generate a temporal opacity and a temporal color; 13. The computer-implemented method of claim 12, wherein performing volume rendering by the computing system to generate the composite pixel color for the composite pixel of the composite image from the opacity and the color comprises performing volume rendering by the computing system to generate the composite pixel color for the composite pixel of the composite image from the static opacity, the static color, the temporary opacity, and the temporary color.

14. 14. The computer-implemented method of claim 13, further comprising inputting, by the computing system, an uncertainty embedding into the temporal content portion of the machine-learned viewpoint synthesis model alongside the latent representation to generate the temporal opacity and the temporal color.

15. 15. The computer-implemented method of claim 12, wherein the generative embedding comprises an interpolated embedding generated by interpolating respective image embeddings associated with two or more existing images selected by a user.

16. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations including: obtaining, by the computing system, a training embedding associated with a location in three-dimensional space and an existing training image, the training embedding encoding one or more visual characteristics of the existing training image; processing, by the computing system, the data describing the positions using a machine-learned viewpoint synthesis model basis to generate opacity and latent representations; processing, by the computing system, the latent representation and the training embeddings using a content portion of the machine-learned view synthesis model to generate color; performing, by the computing system, volume rendering to generate a composite pixel color for a composite pixel of a composite image from the opacity and the color; evaluating a loss function that compares the synthetic pixel colors to ground truth pixel colors for training pixels in the existing training images; and modifying one or more values ​​of the training embeddings based at least in part on the loss function.

17. the opacity produced by the base comprises a static opacity; processing, by the computing system, the latent representation and the training embeddings with the content portion of the machine-learned view synthesis model to generate the color; processing, by the computing system, the latent representations and the training embeddings using a static content portion of the machine-learned view synthesis model to generate static colors; processing, by the computing system, the latent representation using a temporal content portion of the machine-learned viewpoint synthesis model to generate a temporal opacity and a temporal color; 17. The one or more non-transitory computer-readable media of claim 16, wherein performing volume rendering by the computing system to generate the composite pixel color for the composite pixel of the composite image from the opacity and the color comprises performing volume rendering by the computing system to generate the composite pixel color for the composite pixel of the composite image from the static opacity, the static color, the temporary opacity, and the temporary color.

18. processing the latent representation with the temporal content portion of the machine-learned view synthesis model further produces an uncertainty value; 20. The one or more non-transitory computer-readable media of claim 17, wherein the loss function includes a pixel loss term that provides a negative correlation between a loss magnitude and the uncertainty value.

19. The operation, inputting one or more camera parameters associated with a camera that captured the existing training images into the base of the machine-learned viewpoint synthesis model along with the position to generate the opacity and the latent representation; and modifying one or more of the camera parameters based at least in part on the loss function.

20. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations including: obtaining, by the computing system, one or more camera parameters associated with a position in three-dimensional space and with existing training images; processing, by the computing system, data describing the position and the one or more camera parameters using a basis of a machine-learned viewpoint synthesis model to generate opacity and latent representations; processing, by the computing system, the latent representation using a content portion of the machine-learned view synthesis model to generate color; performing, by the computing system, volume rendering to generate a composite pixel color for a composite pixel of a composite image from the opacity and the color; evaluating a loss function that compares the synthetic pixel colors to ground truth pixel colors for training pixels in the existing training images; and modifying one or more values ​​of the camera parameters based at least in part on the loss function.

21. 1. A computing system for generating a synthetic image of a scene, comprising: one or more processors; and one or more non-transitory computer-readable media, the one or more non-transitory computer-readable media comprising: a machine-learned view synthesis model including a base and a static content portion that models static content in the scene, the base and static portion being trained together with a temporary content portion that models temporary content in the scene; and instructions, collectively, that when executed by the one or more processors cause the computing system to perform operations, the operations including: Obtaining a position in three-dimensional space; processing the data describing the positions with the base of the machine-learned viewpoint synthesis model to generate static opacity and latent representations; processing the latent representation with the static content portion of the machine-learned view synthesis model to generate static colors; and performing volume rendering to generate a composite pixel color for a composite pixel of a composite image from the static opacity and the static color.

Images