Robust viewpoint compositing for unconstrained image data

Abstract

To provide a system and a method for synthesizing novel views of complex scenes (e.g., outdoor scenes).SOLUTION: In some implementations, a system and a method can include or use a machine-learned model capable of learning from unstructured and / or unconstrained collections of imagery such as, for example, "in the wild" photographs. In particular, example implementations of the present disclosure can learn a volumetric scene density and radiance represented by a machine-learned model, such as one or more multilayer perceptrons (MLPs).SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Related applications This application claims priority and benefit of U.S. Provisional Patent Application No. 63 / 059,322, filed on 31 July 2020. U.S. Provisional Patent Application No. 63 / 059,322 is incorporated herein by reference in its entirety.

[0002] This disclosure generally relates to systems and methods for modeling scenes to facilitate, for example, the performance of scene viewpoint synthesis. More specifically, this disclosure relates to machine learning models that provide improved robustness to ubiquitous real-world phenomena in uncontrolled images, such as variable lighting and transient occlusions. [Background technology]

[0003] The task of viewpoint synthesis aims to generate a new viewpoint (e.g., scene, object, or subject) of a particular environment, starting from several photographs taken from a given viewpoint. As an example, given several images of a particular subject captured from a specific point with specific camera settings and orientation, a viewpoint synthesis system attempts to generate a composite image that appears as if it were captured from a virtual camera placed at different points and having a given setting.

[0004] While several viewpoint synthesis techniques leverage neural luminance fields to effectively learn volumetric scene density and luminance from images captured in controlled environments, these techniques cannot model many ubiquitous real-world phenomena in uncontrolled images, such as variable lighting and transient occlusions. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Representing Scenes as Neural Radiance Fields for View Synthesis by Mildenhall et al. (arXiv:2003.08934v1) [Overview of the project] [Means for solving the problem]

[0006] Aspects and advantages of the embodiments of this disclosure are partially described in the following description, or can be learned from the description, or can be learned through the implementation of the embodiments.

[0007] An exemplary aspect of the present disclosure relates to a computing system for generating a composite image of a scene. The computing system comprises one or more processors and one or more non-temporary computer-readable media, the one or more non-temporary computer-readable media which collectively store a machine-learned viewpoint composite model comprising a base, a static content section for modeling static content in the scene, and a temporary content section for modeling temporary content in the scene, and instructions that, when executed by one or more processors, cause the computing system to perform an action. The action includes obtaining a position in three-dimensional space, processing data describing the position using the base of the machine-learned viewpoint composite model to generate static opacity and latent representations, processing the latent representation using the static content section of the machine-learned viewpoint composite model to generate static color, processing the latent representation using the temporary content section of the machine-learned viewpoint composite model to generate temporary opacity and temporary color, and performing volume rendering to generate composite pixel colors for composite pixels in the composite image from the static opacity, static color, temporary opacity, and temporary color.

[0008] Another exemplary aspect of this disclosure relates to a computer implementation method for viewpoint synthesis having user-specifiable properties. The method includes the step of obtaining a desired location in three-dimensional space and user-specified generative embeddings by a computing system including one or more computing devices, the generative embeddings encoding one or more visual properties of the resulting composite image. The method includes the step of processing data describing the location using the base of a machine-learned viewpoint synthesis model to generate opacity and latent representations by the computing system. The method includes the step of processing the latent representations and generative embeddings by the computing system using the content portion of a machine-learned viewpoint synthesis model to generate color. The method includes the step of performing volume rendering by the computing system to generate composite pixel colors for composite pixels of the composite image from the opacity and color, the composite image exhibiting one or more visual properties encoded by the generative embeddings.

[0009] Another exemplary aspect of the Disclosure relates to one or more non-temporary computer-readable media for storing instructions together, which, when executed by one or more processors, cause a computing system to perform an action. The action includes the computing system obtaining a position in three-dimensional space and one or more camera parameters associated with an existing training image. The action includes the computing system processing data describing the position and one or more camera parameters using the base of a machine-learned viewpoint synthesis model to generate opacity and latent representations. The action includes the computing system processing the latent representations using the content portion of the machine-learned viewpoint synthesis model to generate colors. The action includes the computing system performing volume rendering to generate composite pixel colors for composite pixels in a composite image from the opacity and colors. The action includes evaluating a loss function that compares the composite pixel colors to ground truth pixel colors for training pixels contained in an existing training image. The action includes modifying one or more values ​​of camera parameters based at least in part on the loss function.

[0010] Another exemplary aspect of the present disclosure relates to a computing system for generating a composite image of a scene. The computing system comprises one or more processors and one or more non-temporary computer-readable media, the one or more non-temporary computer-readable media which collectively store a machine-learned viewpoint synthesis model having a base and a static content section that models static content in a scene, the base and static section being trained together with a temporary content section that models temporary content in a scene, and instructions that cause the computing system to perform an action when executed by one or more processors. The action includes obtaining a position in three-dimensional space, processing data describing the position using the base of the machine-learned viewpoint synthesis model to generate static opacity and latent representation, processing the latent representation using the static content section of the machine-learned viewpoint synthesis model to generate static color, and performing volume rendering to generate composite pixel colors for composite pixels in the composite image from the static opacity and static color.

[0011] Other aspects of this disclosure cover a variety of systems, apparatus, non-temporary computer-readable media, user interfaces, and electronic devices.

[0012] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. The appended drawings incorporated herein and forming part thereof illustrate exemplary embodiments of this disclosure and, together with this description, are useful for illustrating the relevant principles.

[0013] A detailed description of embodiments intended for those skilled in the art is provided herein, and this specification refers to the accompanying figures. [Brief explanation of the drawing]

[0014] [Figure 1]A block diagram of an exemplary process for training a machine-learned view synthesis model according to an exemplary embodiment of the present disclosure and then performing view synthesis using the machine-learned view synthesis model. [Figure 2] A block diagram of an exemplary process for training a machine-learned view synthesis model according to an exemplary embodiment of the present disclosure. [Figure 3] A block diagram of an exemplary process for using a machine-learned view synthesis model according to an exemplary embodiment of the present disclosure. [Figure 4] A block diagram of an exemplary machine-learned view synthesis model according to an exemplary embodiment of the present disclosure. [Figure 5A] A block diagram of an exemplary computing system according to an exemplary embodiment of the present disclosure. [Figure 5B] A block diagram of an exemplary computing device according to an exemplary embodiment of the present disclosure. [Figure 5C] A block diagram of an exemplary computing device according to an exemplary embodiment of the present disclosure.

DETAILED DESCRIPTION

[0015] Reference numerals repeated across multiple figures identify the same features in various implementations.

[0016] SUMMARY Generally, the present disclosure is directed to systems and methods for synthesizing novel views of complex scenes (e.g., outdoor scenes). In some implementations, the systems and methods can include or use a machine-learned model that can learn, for example, from unstructured and / or unconstrained collections of images such as “wild” photographs. In particular, exemplary implementations of the present disclosure can learn volumetric scene density and luminance represented by a machine-learned model such as one or more multilayer perceptrons (MLPs), other neural networks, or other machine-learned models.

[0017] More specifically, several techniques, such as those described in Mildenhall et al.'s NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (arXiv:2003.08934v1), (hereafter "NeRF"), are effective for images captured in a controlled environment. However, these techniques cannot model many ubiquitous real-world phenomena in uncontrolled images, such as variable lighting and temporary occlusions.

[0018] Specifically, NeRF is built on the assumption of 3-D consistency, which assumes that two photographs taken of the same scene should match. However, in uncontrolled environments (e.g., "wild" photographs), two images of the same scene often have inconsistencies, including inconsistencies in lighting characteristics or other visual characteristics and / or the presence / absence of temporary content / items (e.g., each of two photographs of a landmark may have a different set of temporary travelers positioned in front of the landmark).

[0019] NeRF learns a scene-by-scene model described by a (learned) function. This function takes location (x, y, z) and line of sight direction (theta, phi) to create RGB color and opacity (sigma). To generate a render, rays passing through each pixel in the camera are tracked, and the colors, weighted by the opacity accumulated along the length of the rays, are integrated. The NeRF model is trained to minimize the squared error between the predicted RGB values ​​and the ground truth image. Therefore, photographic and camera parameters are needed to train NeRF. The photograph indicates what color each pixel should be, and the camera parameters indicate where in 3-D space the rays diverging from that pixel reside.

[0020] This disclosure provides several techniques for improving the NeRF model or a similar model. These additions enable the proposed model to become more robust to cluttered real-world photographs.

[0021] In particular, according to one aspect of this disclosure, while training a machine learning-prepared viewpoint synthesis model on a set of training images, each training image may be assigned a generative embedding. The generative embedding can serve to encode the visual characteristics of the image, such as lighting characteristics (e.g., white balance), content characteristics (e.g., time of day, weather, season), and style characteristics (e.g., photo editing software settings).

[0022] Therefore, in some implementations, each training image may be assigned a small generative embedding (e.g., a vector of 8-32 floats) (e.g., indexed by the image ID). The use of such generative embeddings allows the model to account for variations in individual images, such as post-processing of white balance and stylistics.

[0023] These generative embeddings can be learned / updated as part of the model. For example, generative embeddings can be treated as model parameters and updated during model training (e.g., during or as a result of backpropagation of the loss function). Through such a process, generative embeddings for images with similar properties can be moved closer to each other.

[0024] Next, during viewpoint synthesis, desired generative embeddings that encode desired characteristics (e.g., desired lighting characteristics, content characteristics, style characteristics, etc.) may be specified by the user and supplied to the model. In response, the model can generate a composite image having the desired characteristics encoded by the desired generative embeddings. In one example, the desired generative embeddings may be obtained by interpolating two or more generative embeddings learned for two or more training images selected by the user (for example, because they exhibit / represent the desired characteristics).

[0025] Therefore, the model can use generative embeddings to capture differences in image properties that do not reflect actual differences within the underlying scene. For example, differences in post-processing settings for a landmark image do not actually reflect differences in the landmark itself. This allows the trained model to handle inconsistencies between training images better, which in turn makes the model more robust to unconstrained sets of training images.

[0026] In another exemplary embodiment, some implementations of the present disclosure directly incorporate uncertainty into the model's volume rendering pipeline. For example, in addition to static opacity and static color, the model can also predict an uncertainty value "β", transient opacity, and transient color. The transient data can be integrated so as to be done on the static data to obtain a learned weight multiplier for each ray. Intuitively, this allows the model to lower the priority of reconstructing the "hard" parts of the image that do not match other photographs (e.g., often corresponding to transient and inconsistent occlusions in the image).

[0027] Accordingly, exemplary implementations of this disclosure have a novel model architecture comprising a base, a static content section for modeling static content in a scene, and a transient content section for modeling transient content in a scene. In some implementations, the loss function on which the model is trained may result in a reduced amount of loss for instances where the difference between the predicted composite color and the ground truth color arises from transient content modeled by the viewpoint composite model. Thus, the model has the ability to deviate from the ground truth image when the ground truth image represents transient content or, in some cases, is associated with high uncertainty. This allows the trained model to better handle inconsistencies between training images, which in turn makes the model more robust to unconstrained sets of training images.

[0028] In another exemplary embodiment, several implementations of this disclosure may directly fine-tune the camera parameters of each training image within the model itself. Exemplary camera parameters include orientation, location, focal length, principal point, skew, radial distortion, tangential distortion, and / or various camera intrinsics. This allows the model to adjust the camera parameters (e.g., within several thresholds or tolerances) to perfectly match the reconstructed scene and to be independent of the sparse image features commonly used in 3-D registration. In other words, by allowing the camera parameters for each training image to be modified, some amount of noise present in the training dataset can be removed from the dataset. This allows the trained model to better handle inconsistencies between training images, which results in the model being more robust to unconstrained sets of training images.

[0029] The systems and methods of this disclosure offer several technical effects and benefits. For example, the systems and methods described herein enable models to become more robust to uncontrolled images. This results in improved model performance when synthesizing viewpoints of a scene based on uncontrolled training datasets. Thus, the systems and methods of this disclosure improve the realism, consistency, and user controllability of viewpoint synthesis systems.

[0030] Another exemplary technical effect and benefit of providing a model with a clear mechanism for identifying and / or correcting inconsistencies within the training dataset is that the model can converge to the optimal solution faster (e.g., with fewer training iterations and / or requiring fewer training images). Faster convergence can result in savings of computational resources, such as reduced processor usage, memory usage, and / or bandwidth usage.

[0031] Accordingly, this disclosure provides a system and method for solving problems related to unconstrained images and / or providing additional user controllability. Exemplary experimental data contained in U.S. Provisional Patent Application No. 63 / 059,322 demonstrate the effectiveness of the proposed technique through comprehensive artificial and real-world experiments. The exemplary experiments apply an exemplary implementation of the system described herein to multiple landmarks using both high-resolution captures and internet photographs, resulting in photorealistic reconstructions and results that significantly surpass previous works.

[0032] While the exemplary implementations of this disclosure are described in relation to a single frame of an image, the systems and methods of this disclosure may equally apply to video or other multi-frame datasets. For example, a video showing "flying over" a modeled scene may be synthesized. The frames of the video may be temporally and stylistically consistent (for example, through the consistent use of the same desired generative embedding for all frames of the synthesized video).

[0033] In addition, the frames of the synthesized image may have the same or different resolution as the training images on which the model is trained. For example, a model may be trained on low-resolution images and then used to generate high-resolution synthesized images.

[0034] Color data used and / or generated by the systems and methods of this disclosure (e.g., input data colors, static colors, temporary colors, composite colors, etc.) may be represented in any color space, including, for example, RGB space (e.g., RGB, RGBA), CIE space, HSV and HSL space, CMYK space, grayscale, and / or other color spaces.

[0035] Next, exemplary embodiments of this disclosure will be described in more detail with reference to the figures. Exemplary Techniques

[0036] This section describes an exemplary implementation of the proposed method, sometimes referred to as NeRF-W in some implementations. The model described is well-designed for reconstructing 3-D scenes from "wild" photobooks and improves upon the NeRF model described by Mildenhall et al.

[0037] At its core, NeRF relies on multi-view consistency, meaning that all points in 3-D space must appear the same from all (unobstructed) viewpoints. While this condition is relaxed depending on the viewing direction, significant color variations are not something NeRF can capture. In particular, NeRF generally handles color variations by associating them with the viewing angle. Therefore, a single image may look normal, but verification views or sets of multiple images from different viewing angles (e.g., fly-through video) may not match.

[0038] When a photograph is captured in a controlled setting by a single person, the assumption of NeRF is satisfied, and realistic reconstruction is achievable. However, unconstrained photobooks, such as amateur photographs of famous landmarks, present many challenges that NeRF cannot grasp.

[0039] For example, unconstrained images may show the same scene but in different weather conditions. Photographs are taken at different times under variable lighting conditions. In outdoor photography, time and weather directly affect the colors of all objects in the scene. Furthermore, the sky itself changes over time.

[0040] As another example, an unconstrained image may show the same scene but with different post-processing, and post-processing of a photograph, including exposure and white balance adjustments, is not constrained at all and further affects the color of all objects in the scene.

[0041] As yet another example, unconstrained images may depict the same scene but with different temporary objects, and since photographs are not captured at a single point in time, temporary objects, including people, flags, and branches and leaves, vary from image to image.

[0042] In the following description (and also in the summary above), this disclosure proposes several enhancements that are directly designed to address these phenomena.

[0043] Figure 1 shows a block diagram of an exemplary process for training a machine learning-based viewpoint synthesis model and then performing viewpoint synthesis using the machine learning-based viewpoint synthesis model, according to an exemplary embodiment of the present disclosure.

[0044] Referring to Figure 1, the training dataset 12 may include existing training images representing a scene. As described herein, the training images are not constrained and may exhibit various inconsistencies with one another. As shown in 14, the computing system can perform a model optimization or training process on the training dataset 12 to generate a machine-learned viewpoint synthesis model 16 (see, for example, Figure 2). After training, a desired location 18 for a synthesized image may be provided to the model 16. In response, the model 16 can generate a synthesized image 20 representing a scene from location 18 (see, for example, Figure 3).

[0045] Figure 2 shows a block diagram of an exemplary process for training a machine learning-prepared viewpoint synthesis model according to an exemplary embodiment of the present disclosure. In some implementations, the process shown in Figure 2 may be performed for each pixel of each training image.

[0046] Referring to Figure 2, the training position 22 of existing training images may be provided to the machine-learned viewpoint synthesis model 24. The position 22 may include the location and orientation of the camera that took the training images. In addition, according to aspects of this disclosure, in some implementations, one or more camera parameters 25 for the training images and / or training image embeddings 26 for the training images may be provided to the machine-learned viewpoint synthesis model 24. For example, the additional camera parameters 25 may include focal length, principal point, skew, radial distortion, tangential distortion, and / or various camera intrinsics. The training image embeddings 26 may be generative embeddings assigned to the training images.

[0047] In particular, at the heart of the challenges presented by "wild" images is the concept of color variation from image to image, where the 3-D geometry of the scene is assumed to be identical across all images, while little color consistency can be expected due to variations in camera settings such as lighting and exposure.

[0048] To solve this problem, in some implementations, each image in the training set has a unique embedding.

[0049]

number

[0050] 26 can be assigned. These embeddings

[0051]

number

[0052] This can be optimized during training in conjunction with the model parameters.

[0053] Referring again to Figure 2, the machine learning-prepared viewpoint synthesis model 24 can process the input data to generate opacity and color data 27. For example, in some implementations, the machine learning-prepared viewpoint synthesis model 24 can generate only a single set of opacity and color data 27, or in other implementations, it can generate both a static set of opacity and color data for the static content of the scene and a temporary set of color and opacity data 27 for the temporary content of the scene.

[0054] As an example, differential opacity σ(r) and color c(r,d)²⁷ can be predicted by a multilayer perceptron (MLP) or other model (e.g., a neural network or some other form of other machine-learned model) given a 3-D location r(t) and line of sight d. This MLP or other model can be explicitly designed to ensure that the line of sight d does not affect the differential opacity σ. For example, the base of the model can predict opacity from location only as long as color can be predicted from both location and field of sight / line of sight. Again, in some implementation forms, the input to this MLP is an embedded e (g) It can increase. (c, σ) = MLP(r(t), d, e (g) ) Here e (g) This is a generative embedding corresponding to the rendered image. Similar to the gaze direction d, some exemplary implementations use generative embedding e (g) This ensures that it does not affect the differential opacity σ. Embedding the MLP input

[0055]

number

[0056] By increasing this, the proposed model can directly change the color and lighting of a scene based on the image's identity without modifying its 3-D geometry.

[0057] In 28, volume rendering techniques may be used to generate a composite pixel color from opacity and color data 27. For example, for a single set of opacity and color data 27, the composite pixel color may be obtained by integrating along the light rays diverging from the camera.

[0058]

number

[0059] In another exemplary embodiment, GLO can capture variable lighting and post-processing, but cannot model variations in 3-D geometry. Therefore, some implementations of this disclosure feature a double-head model that includes both a static part for modeling static content and a transient part for modeling transient content. Unlike implementations that create a single tuple (σ, c) for positions in 3-D space, the proposed model having both static and transient heads, one of which is a “static” object (σ) common to all images. s , c s ) is opposed to another, which is a "temporary" object specific to a particular image (σt and c t Create two heads for ( ). These amounts can be combined with a modified version of the volume rendering equation presented in Equation 1. Further, the proposed model outputs an uncertainty estimate β that is used to modulate the loss function for each pixel. FIG. 4 provides a diagram of the architecture of the proposed model.

[0060] Specifically, referring now to FIG. 4, a block diagram of an exemplary machine-learned view synthesis model according to an exemplary embodiment of the present disclosure is provided. The model can include a base 34, a static portion 36, and a temporal portion 38.

[0061] Given the 3-D point r(t) 40, the GLO embedding e (g) 48, and the uncertainty embedding e (u) 52, the model shown in FIG. 4 produces the differential opacity σ s , σ t 44, 54, the color c s , c t 45, 56, and the differential uncertainty β 58. The position embedding, the viewing direction, and the non-linearity are omitted for clarity. As described above, additional camera parameters 42 can be optionally provided as well.

[0062] In some implementations, the base 34 of the exemplary proposed model includes an MLP applied to the 3-D point r(t) 40. Other models may be used as well. This MLP outputs the differential opacity σ s 44 and the latent representation z 46. The latent representation z 46 is employed in two ways. The first is the static portion 36 for z 46 and the viewing direction d to create the color c s 45, similar to the MLP of NeRF (e.g., it can include a 4-layer MLP or other model). The second is the per-image embedding e for creating the three amounts of the temporal differential opacity σ t 54, the temporal color c t 56, and the uncertainty value β 58, as described below. (u)The temporary part 38 for z46 can be increased by 52 (for example, it may include a second 4-layer MLP or other model). In some implementations, l1 regularization is used to promote dilution σ t It can be applied to this.

[0063] Referring together to Figures 2 and 4, the following variation of the volume rendering formula is used to create a color for a single pixel in 28 from both static and transient data 27:

[0064]

number

[0065]

number

[0066] It can be used.

[0067] In Equation 1, color c is σ s c s +σ t c t The accumulated opacity σ is replaced by a linear combination of σ s +σ t Please note that it will be replaced by the sum of [the specified values].

[0068] In some implementations, such as the example shown in Figure 4, the model is allowed to emit an uncertainty estimate β58. During training time, the accumulated opacity formula can be used to obtain the uncertainty for the corresponding predicted color C(r).

[0069]

number

[0070] As an example, the loss for a single pixel having ground truth color y can therefore be given by the following formula.

[0071]

number

[0072] Intuitively, a larger value of β allows the model to reduce the weight of outliers—generally transient or moving objects such as people, grass, or clouds. The logarithmic term prevents β from growing indefinitely and can be directly derived from the likelihood of a normal distribution. In some implementations, the hyperparameter β is used to prevent the model from concentrating a large portion of the loss on a small number of pixels. min ≥0 may be used.

[0073] More generally, referring again to Figure 2, the loss function 30 can evaluate the difference between the composite pixel color generated in 28 and the ground truth pixel color 32 of the existing training image. For example, the squared error between pixel colors represented in RGB or some other color scheme may be used.

[0074] The loss function 30 can be backpropagated to train a machine learning-prepared viewpoint synthesis model 24. In addition, in some implementations, the training image embeddings 26, training positions 22, and / or camera parameters 25 can also be updated based on the loss function 30 (for example, by continuing to backpropagate the loss through and around the model 24).

[0075] During the test, Model 24 can be used to render a common static geometry for all photos in the training set. In some implementations, the images are σ t , c t It can be rendered by omitting all of β.

[0076] As an example, Figure 3 shows an exemplary use of a trained machine learning-based viewpoint synthesis model. Specifically, a desired position 40 (e.g., location and orientation) for the synthesized image of the scene is provided. Optionally, desired camera parameters 42 and / or desired generative embeddings 44 may also be provided.

[0077] The machine learning-prepared viewpoint synthesis model 24 can process the input to generate opacity and color data 27 (for example, a single set of opacity and color data, or both static and transient opacity and color data, or simply static opacity and color data). Volume rendering 28 can be performed on the opacity and color data (for example, static data only) to generate composite pixel colors for the pixels of the composite image.

[0078] The process shown in Figure 3 can be performed for each pixel of the composite image. Exemplary devices and systems

[0079] Figure 5A shows a block diagram of an exemplary computing system 100 according to an exemplary embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150, which are communicably coupled via a network 180.

[0080] The user computing device 102 may be any type of computing device, such as a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0081] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple processors connected operably. The memory 114 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 114 can store data 116 and instructions 118 executed by the processors 112 to cause the user computing device 102 to perform operations.

[0082] In some implementations, the user computing system 102 may store or contain one or more machine learning models 120. For example, the machine learning models 120 may be a variety of machine learning models, including neural networks (e.g., deep neural networks) or other types of machine learning models including nonlinear and / or linear models, or may otherwise contain such models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long-short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Exemplary machine learning models 120 are discussed with reference to Figures 1 to 4.

[0083] In some implementations, one or more machine learning models 120 may be received from a server computing system 130 via a network 180, stored in user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single machine learning model 120 (for example, to perform parallel viewpoint synthesis across multiple examples of the same or different scenes).

[0084] As an addition or alternative, one or more machine learning models 140 may be included in, or otherwise stored and implemented by, a server computing system 130 that communicates with a user computing device 102 according to a client-server relationship. For example, a machine learning model 140 may be implemented by the server computing system 140 as part of a web service (e.g., a viewpoint synthesis service). Thus, one or more models 120 may be stored and implemented in the user computing device 102, and / or one or more models 140 may be stored and implemented in the server computing system 130.

[0085] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may be useful for implementing a virtual keyboard. Other exemplary user input components include a microphone, a conventional keyboard, or other means by which the user can provide user input.

[0086] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple processors connected operably. The memory 134 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 134 can store data 136 and instructions 138 executed by the processors 132 to cause the server computing system 130 to perform operations.

[0087] In some implementations, the server computing system 130 includes one or more server computing devices, or is otherwise implemented by one or more server computing devices. In cases where the server computing system 130 includes multiple server computing devices, such server computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or any combination thereof.

[0088] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, the models 140 may be various machine learning models or otherwise include them. Exemplary machine learning models include neural networks or other multilayer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Exemplary models 140 are discussed with reference to Figures 1 to 4.

[0089] The user computing device 102 and / or the server computing system 130 can train models 120 and / or 140 by interacting with a training computing system 150 which is communicatively connected via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0090] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple processors operably connected. The memory 154 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 executed by the processors 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes one or more server computing devices, or is otherwise implemented by server computing devices.

[0091] The training computing system 150 may include a model trainer 160 that trains machine-learned models 120 and / or 140 stored in the user computing device 102 and / or server computing system 130 using various training or learning techniques, such as backpropagation. For example, a loss function may be backpropagated through the model to update one or more parameters of the model (for example, based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters over several training iterations.

[0092] In some implementations, performing backpropagation may include performing shortened temporal backpropagation. The model trainer 160 may implement several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0093] In particular, the model trainer 160 can train the machine-learned models 120 and / or 140 based on the training data set 162. The training data 162 may include unconstrained image data, such as "wild" photographs.

[0094] In some implementations, if the user gives consent, training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some cases, this process is referred to as model personalization.

[0095] The model trainer 160 includes computer logic used to provide desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that control a general-purpose processor. For example, in some implementations, the model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer executable instructions stored on a RAM hard disk or a tangible computer-readable storage medium such as an optical or magnetic medium.

[0096] Network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or any combination thereof, and may include any number of wired or wireless links. Generally, communication over Network 180 can be carried over any type of wired and / or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encoding or formatting (e.g., HTML, XML), and / or protection methods (e.g., VPN, Secure HTTP, SSL).

[0097] Figure 5A shows one exemplary computing system that may be used to implement the present disclosure. Other computing systems may be used similarly. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training dataset 162. In such implementations, the model 120 can be both trained locally on the user computing device 102 and used. In some such implementations, the user computing device 102 may implement the model trainer 160 based on user-specific data to personalize the model 120.

[0098] Figure 5B shows a block diagram of an exemplary computing device 10 that operates according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0099] The computing device 10 includes several applications (for example, applications 1 to N). Each application includes its own machine learning library and pre-trained models. For example, each application may include a pre-trained model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and so on.

[0100] As shown in Figure 5B, each application can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0101] Figure 5C shows a block diagram of an exemplary computing device 50 that operates according to an exemplary embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0102] The computing device 50 contains several applications (for example, applications 1 through N). Each application communicates with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, and a browser application. In some implementations, each application can communicate with the central intelligence layer (and the models stored within it) using an API (for example, a common API across all applications).

[0103] The central intelligence layer contains several machine learning models. For example, as shown in Figure 5C, each machine learning model (e.g., Model) may be provided to each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model (e.g., SingleModel) to all applications. In some implementations, the central intelligence layer is included in the operating system of the computing device 50, or otherwise implemented by the operating system.

[0104] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized repository of data for the computing device 50. As shown in Figure 5C, the central device data layer can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API). Additional disclosure

[0105] The technologies discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions performed and the information transmitted to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among their components. For example, the processes described herein may be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0106] While the subject matter has been described in detail with respect to various specific exemplary embodiments thereof, each example is given for illustrative purposes only and not as a limitation of the disclosure. Those skilled in the art, understanding the foregoing, will readily be able to create modifications, variations, and equivalents of such embodiments. Therefore, the disclosure does not preclude the inclusion of such modifications, variations, and / or additions to the subject matter, as will be readily apparent to those skilled in the art. For example, a feature shown or described as part of one embodiment may also be used in conjunction with another embodiment to bring about further embodiments. Thus, the disclosure is intended to cover such modifications, variations, and equivalents. [Explanation of symbols]

[0107] 10 Computing Devices 12 training datasets 14. Model Optimization or Training Process 16 Machine Learning-Prepared Viewpoint Synthesis Model 18 Composite image 20 Composite Images 22 Training positions 24 Machine Learning-Prepared Viewpoint Synthesis Model 25 Camera Parameters 26. Training response embedding 27. Opacity and color data 28 Volume Rendering 30 Loss Function 32 Ground Truth Pixel Colors 34 Base 36 Static part 38 Temporary part 40 3-D points 42 Additional camera parameters 44 Differential opacity 46 colors 48 GLO embedded 50 Computing Devices 52. Embedding Uncertainty 54 Differential Opacity 56 colors 58 Differential Uncertainty 100 Computing Systems 102 User Computing Devices 112 processors 114 memory 116 data 118 Command 120 machine learning models 122 User Input Components 130 Server Computing Systems 132 processors 134 memory 136 data 138 Command 140 machine learning models 150 Training Computing Systems 152 processors 154 memory 156 data 158 Command 160 Model Trainers 162 training data sets 180 Networks

Claims

1. A computing system configured to perform viewpoint synthesis, One or more processors, A machine learning-prepared viewpoint synthesis model is configured to receive and process the position of a desired composite image in order to generate a composite image of a scene from the position of the desired composite image, The aforementioned machine learning-prepared viewpoint synthesis model includes a neural luminance field model, The machine learning-based viewpoint synthesis model comprises a static content unit that models static content within the scene and a temporary content unit that models temporary occlusions within the scene. The aforementioned neural luminance field model is trained with a set of training images representing the scene, The set of training images comprises unrestrained images including the temporary obstruction, A machine learning-prepared viewpoint synthesis model, One or more non-temporary computer-readable storage media, which, when executed by the one or more processors, allows the computing system to: The position of the desired composite image is obtained, To generate the desired composite image, the machine learning-prepared viewpoint synthesis model is used to process the position of the desired composite image. The composite image is provided as output. A non-temporary computer-readable storage medium that stores a set of commands to execute actions. A computing system equipped with [the following features].

2. The machine learning-trained viewpoint synthesis model further comprises a base, and the machine learning-trained viewpoint synthesis model is used to process the position of the desired synthesized image. Processing data describing the position using the base of the machine learning-prepared viewpoint synthesis model to generate static opacity and latent representation, The latent representation is processed using the static content portion of the machine learning-prepared viewpoint synthesis model in order to generate static colors, Processing the latent representation using the temporary content portion of the machine learning-prepared viewpoint synthesis model to generate temporary opacity and temporary color, Performing volume rendering to generate a composite pixel color for the composite pixels of the composite image from the static opacity, the static color, the temporary opacity, and the temporary color. The computing system according to claim 1, including the following:

3. The computing system according to claim 1, wherein the set of training images comprises photographs generated under several different conditions, the set of training images shows inconsistent lighting in the scene, the set of training images was taken at different times, and the set of training images comprises an outdoor scene.

4. One or more non-temporary computer-readable storage media, which, when executed by one or more processors, are used by a computing system. The computing system acquires one or more camera parameters associated with a position in three-dimensional space and an existing training image, and the one or more camera parameters include the camera orientation, the camera's focal length, the camera's principal point, the camera's skew, the camera's radial or tangential distortion, or the camera's camera characteristics. The computing system processes the position in the three-dimensional space and the one or more camera parameters using the machine learning-prepared viewpoint synthesis model described in any one of claims 1 to 3 in order to generate opacity and color, The computing system performs volume rendering to generate composite pixel colors for composite pixels in a composite image from the opacity and color, The loss function is evaluated by comparing the ground truth pixel color with the composite pixel color for the training pixels contained in the existing training image, Modifying one or more values ​​of the one or more camera parameters based at least partially on the loss function. One or more non-temporary computer-readable storage media that store a set of instructions for performing an action comprising the above.

5. The machine learning-prepared viewpoint synthesis model comprises a neural luminance field model, and the computing system processes the position in the three-dimensional space and one or more camera parameters using the machine learning-prepared viewpoint synthesis model to generate the opacity and the color. The computing system processes the position and one or more camera parameters using the base of a machine learning-prepared viewpoint synthesis model to generate the opacity and latent representation, The computing system processes the latent representation using the static content portion of the machine learning-prepared viewpoint synthesis model in order to generate the color. One or more non-temporary computer-readable storage media according to claim 4, comprising:

6. One or more non-temporary computer-readable storage media according to claim 4, wherein one or more camera parameters comprises the orientation of the camera, and one or more camera parameters comprises the focal length of the camera.

7. One or more non-temporary computer-readable storage media according to claim 6, wherein the one or more camera parameters comprise the radial distortion or tangential distortion of the camera, and the one or more camera parameters comprise the camera-specificity of the camera.

8. A computer implementation method for training a machine learning-based viewpoint synthesis model with unconstrained image data, A step of acquiring multiple unconstrained training images representing a scene, wherein each of the unconstrained training images includes a different temporary occlusion. For each of the aforementioned multiple unconstrained training images, A step to obtain a position in 3D space, The position in the three-dimensional space includes a capture position associated with the unconstrained training image, The aforementioned unrestrained training images include unrestrained photographs captured by the user. Steps and A step of processing location-indicating data using the machine learning-prepared viewpoint synthesis model to generate one or more opacities and one or more colors, wherein the machine learning-prepared viewpoint synthesis model comprises a neural luminance field model, The steps include: performing volume rendering to generate a composite pixel color for the composite pixels of a composite image from the one or more opacities and the one or more colors; The steps include evaluating a loss function that compares the ground truth pixel color with the composite pixel color for training pixels included in the unconstrained training image, The steps include modifying one or more values ​​of one or more parameters of the machine learning-prepared viewpoint synthesis model based at least partially on the loss function, and A computer implementation method comprising the following features.

9. The computer implementation method according to claim 8, comprising a base unit of the machine learning-prepared viewpoint synthesis model, a static content unit for modeling static content in the scene, and a temporary content unit for modeling temporary occlusions in the scene.

10. A computing system for generating composite images of scenes, One or more processors, One or more non-temporary computer-readable storage media, A machine learning-trained viewpoint synthesis model trained by the method described in claim 8 or 9, An instruction, when executed by one or more processors, to the computing system Accessing an existing training image associated with the aforementioned scene, wherein the existing training image encodes ground truth pixels for the training pixels, The method involves generating static colors based on descriptive data and the machine-learned viewpoint synthesis model, wherein the descriptive data indicates a position in three-dimensional space associated with the scene. To generate a temporary color based on the descriptive data and the machine learning-prepared viewpoint synthesis model, The process involves generating a composite pixel color for the composite pixels of a composite image representing the aforementioned scene, wherein the composite pixel color is generated based on the static color, the temporary color, and the machine learning-prepared viewpoint synthesis model. Evaluating a loss function that compares the ground truth pixel color with the composite pixel color, Training the machine learning-prepared viewpoint synthesis model based on the loss function mentioned above. Commands that cause an action including and A non-temporary computer-readable storage medium comprising A computing system equipped with [the following features].

11. The aforementioned operation, To generate a latent representation based on the descriptive data and the machine learning-prepared viewpoint synthesis model, Further generating the static color based on the aforementioned latent representation, Further generating the temporary color based on the aforementioned latent expression The computing system according to claim 10, further comprising:

12. The aforementioned operation, To generate static opacity based on the descriptive data and the machine learning-prepared viewpoint synthesis model, The method involves generating a temporary opacity based on a latent representation and the machine-learned viewpoint synthesis model, wherein the latent representation is generated based on the descriptive data and the machine-learned viewpoint synthesis model, Performing volume rendering to generate the aforementioned composite pixel color, further based on the aforementioned static opacity and temporary opacity. The computing system according to claim 10, further comprising:

13. The aforementioned machine learning-prepared viewpoint synthesis model includes a base, and the operation is, The latent representation is generated based on the base of the machine learning-prepared viewpoint synthesis model that processes the descriptive data, To generate the static opacity based on the base of the machine learning-prepared viewpoint synthesis model that processes the descriptive data. The computing system according to claim 12, further comprising the above.

14. The machine learning-prepared viewpoint synthesis model includes a static content section that models the static content within the scene, and the operation is, The composite pixel color is generated based on the static content portion of the machine learning-prepared viewpoint synthesis model that processes the latent representation. The computing system according to claim 13, further comprising:

15. The machine learning-prepared viewpoint synthesis model includes a temporary content section that models temporary occlusions within the scene, and the operation is, The temporary opacity is generated based on the temporary content portion of the machine learning-prepared viewpoint synthesis model that processes the latent representation, To generate the temporary color based on the temporary content portion of the machine learning-prepared viewpoint synthesis model that processes the latent representation. The computing system according to claim 13, further comprising:

16. Processing the latent representation using the temporary content portion of the machine learning-prepared viewpoint synthesis model further generates uncertainty values. The loss function includes a pixel loss term that gives a negative correlation between the magnitude of the loss and the uncertainty value. The computing system according to claim 15, further comprising:

17. The aforementioned operation, To generate the aforementioned temporary opacity and temporary color, uncertainty embedding is input to the temporary content portion of the machine learning-trained viewpoint synthesis model alongside the latent representation. The computing system according to claim 15, further comprising:

18. The operation further includes inputting a generative embedding into the static content portion of the machine-learned viewpoint synthesis model alongside the latent representation in order to generate the static color, The generative embedding is associated with the existing training image, and the operation is, Further includes modifying one or more values ​​of the generative embedding based at least partially on the loss function, The computing system according to claim 14, further comprising the above.

19. A computer implementation method, A step of accessing an existing training image associated with a scene, wherein the existing training image encodes ground truth pixels for training pixels, A step of generating static colors based on descriptive data and a machine learning-trained viewpoint synthesis model trained by the method described in claim 8 or 9, wherein the descriptive data indicates a position in three-dimensional space associated with the scene; A step of generating a temporary color based on the descriptive data and the machine learning-prepared viewpoint synthesis model, A step of generating a composite pixel color for a composite pixel of a composite image showing the scene, wherein the composite pixel color is generated based on the static color, the temporary color, and the machine learning-prepared viewpoint synthesis model. A step of evaluating a loss function that compares the ground truth pixel color with the composite pixel color, A step of training the machine learning-prepared viewpoint synthesis model based on the loss function. A computer implementation method comprising the following features.

20. A computer implementation method, Steps include obtaining the position in 3D space associated with the scene, A step of generating static colors based on descriptive data and a machine learning-trained viewpoint synthesis model trained by the method described in claim 8 or 9, wherein the descriptive data indicates a position in the three-dimensional space, A step of generating a temporary color based on the descriptive data and the machine learning-prepared viewpoint synthesis model, A step of performing volume rendering to generate a composite pixel color for composite pixels in a composite image, wherein the composite pixel color is generated based on the static color, the transient color, and the machine learning-prepared viewpoint composite model. A computer implementation method comprising the following features.

21. A computer implementation method for gaze synthesis using adjustable visual characteristics, A computing system comprising one or more computer devices obtains a position and embedding in three-dimensional space, wherein the embedding encodes one or more visual characteristics of the composite image to be generated. The computing system provides a first portion of a viewpoint synthesis model trained by the method described in claim 8 or 9 to generate opacity, The computing system performs the steps of generating color data based on the embedding using a second portion of the viewpoint synthesis model, A step of rendering the composite pixel color for the composite pixels of the composite image based on the opacity and color data. Equipped with, A computer implementation method wherein the composite image exhibits one or more visual characteristics encoded by the embedding.

22. A computer implementation method for calculating parameter values ​​for a viewpoint synthesis model using unconstrained image data, comprising the steps of: acquiring a plurality of unconstrained image data comprising unconstrained photographs in a scene, wherein each of the unconstrained images includes a different temporary occlusion; For each of the aforementioned multiple unconstrained images, The steps include obtaining the capture position associated with each of the aforementioned images, The steps include processing data indicating the capture position using a viewpoint synthesis model trained by the method described in claim 8 or 9 in order to generate one or more opacities and one or more colors, A step of rendering the composite pixel color for the composite pixels of the composite image based on the one or more opacities and the one or more colors, The steps include: modifying one or more parameter values ​​for one or more parameters of the viewpoint synthesis model based on a loss function that compares the reference pixels in each of the aforementioned images with the composite pixel color of the reference image; A computer implementation method comprising the following features.

23. A computer implementation method, A computing system comprising one or more processors acquires viewpoint data indicating the viewpoint of a scene from a given location. The computing system inputs the viewpoint data into a viewpoint synthesis model trained by the method described in claim 8 or 9, The viewpoint synthesis model comprises parameters having values ​​based on a set of reference images representing the scene, The input image set comprises unconstrained images representing static content within the scene and temporary occlusions within the scene. The viewpoint synthesis model comprises a static content section that models the static content within the scene, The viewpoint synthesis model includes a temporary content section that models the temporary occlusions within the scene. Steps and The computing system generates a composite image of the viewpoint of the scene from the position, based on processing the viewpoint data using the viewpoint synthesis model. A computer implementation method comprising the following features.

24. A computer implementation method, A computing system comprising one or more processors obtains one or more camera parameters associated with a location and a reference image, wherein the one or more camera parameters include the orientation of the camera, the focal length of the camera, the principal point of the camera, the skew of the camera, the radial or tangential distortion of the camera, or the camera-specific properties of the camera. The computing system includes the steps of processing the position and the one or more camera parameters using a viewpoint synthesis model trained by the method described in claim 8 or 9 in order to generate opacity and color, The computing system performs the steps of rendering the composite pixel color for the composite pixels of the composite image based on the opacity and the color, A step of modifying one or more values ​​of one or more camera parameters based on a loss function that compares the reference pixel color for a reference pixel in the reference image with the composite pixel color. A computer implementation method comprising the following features.

25. A computer implementation method, A step of accessing an input image associated with a scene, wherein the input image encodes ground truth pixels for ground truth pixels, A step of generating static colors based on a position in three-dimensional space associated with the scene using a viewpoint synthesis model trained by the method described in claim 8 or 9, Using the aforementioned viewpoint synthesis model, the step of generating a temporary color based on the position, Using the viewpoint synthesis model, the steps include generating a composite pixel color for the composite pixels of the composite image representing the scene, based on the static color and the temporary color. A computer implementation method comprising the following features.

Images