Large scene neural view synthesis

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for rendering a new image, the new image depicting a scene from a perspective of a camera at a new camera viewpoint.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 285,980, filed December 3, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This manual relates to the use of neural networks to synthesize images. Background Technology

[0004] A neural network is a machine learning model that uses one or more learning operations to predict the output of a received input. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from the received input based on the current values ​​of the corresponding parameter set. Summary of the Invention

[0005] This specification describes a system implemented as a computer program on one or more computers in one or more locations, which synthesizes images of scenes in an environment.

[0006] Throughout the specification, “scene” may refer to, for example, a real-world environment or a simulated environment (e.g., a simulation of a real-world environment, such that the simulated environment is a synthetic representation of a real-world scene).

[0007] An entity's "embedding" can refer to a representation of an entity as an ordered set of numerical values, such as a vector, matrix, or other numerical tensor.

[0008] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages.

[0009] Some existing neural image rendering techniques can perform photorealistic reconstructions and novel view synthesis using only a set of camera images of a given scene. However, these existing techniques are typically only suitable for small-scale or object-centric reconstructions, such as, at most, the size of a single room or building. Applying these techniques to large environments often results in significant artifacts and low visual fidelity due to the limited model capacity.

[0010] However, large-scale environment reconstruction has enabled several important use cases in fields such as autonomous driving and aerial surveying. One example is mapping, where high-fidelity maps of the entire operational domain are created to serve as powerful priors for various problems including robot localization, navigation, and collision avoidance. Furthermore, large-scale scene reconstruction can be used for closed-loop robot simulation and / or to generate synthetic training data for perception algorithms.

[0011] Autonomous driving systems are typically evaluated by resimulating previously encountered scenarios; however, any deviation from a recorded encounter can alter the vehicle's trajectory, necessitating high-fidelity rendering of a new view along the altered path. Beyond basic view synthesis, the robustness of these tasks could be increased if the view synthesis model could also modify ambient lighting conditions such as camera exposure, weather, or time of day, which could be used to further enhance the simulated scene.

[0012] Reconstructing such large-scale environments introduces additional challenges, including the presence of transient objects (cars and pedestrians), limitations on model capacity, and memory and computational constraints. Furthermore, training data for such large environments is highly unlikely to be collected under consistent conditions in a single capture. Instead, data for different parts of the environment may need to originate from different data collection efforts, introducing variations in both scene geometry (e.g., construction work and parked cars) and appearance (e.g., weather conditions and time of day).

[0013] The described technology addresses these challenges of generating accurate reconstructions and synthesizing new views in large-scale environments, such as large-scale scenarios encountered in urban driving scenarios.

[0014] Specifically, the described technique divides a large environment into separately trained view synthesis models, each corresponding to a sub-region of a given scene. These view synthesis models are then dynamically rendered and combined at inference time. Modeling these models independently allows for maximum flexibility, scaling to arbitrarily large environments, and provides the ability to update or introduce new regions in a piecewise manner without retraining the entire environment. To compute the target camera viewpoint, only a subset of the view synthesis models is rendered and then synthesized based on their geographic location relative to the camera of the target view. Therefore, the image synthesis process for any given viewpoint remains computationally efficient, despite being able to handle any viewpoint within a large scene.

[0015] In some implementations, the technique includes appearance embedding to address environmental variations between training images as described above.

[0016] In some implementations, the technique includes learned pose refinement to account for pose errors in the training data used for the view synthesis model.

[0017] In some implementations, these techniques are combined with exposure adjustment to provide the ability to modify the exposure during inference, i.e., to provide composite images that appear to be taken by a camera with a specified exposure level.

[0018] In some implementations, to allow for more seamless synthesis of multiple synthetic images from multiple different models, the technique includes appearance matching, which optimizes the appearance embedding of different models to achieve visual alignment between them.

[0019] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0020] Figure 1 This is a block diagram of an example image rendering system.

[0021] Figure 2 This shows an example of rendering an image of the scene.

[0022] Figure 3 An example view composite model is shown.

[0023] Figure 4 This is a flowchart illustrating an example process for rendering a new image using an image rendering system.

[0024] The same reference numerals and names in each of the figures indicate the same elements. Detailed Implementation

[0025] Figure 1 This is a block diagram of an example image rendering system 100 capable of rendering (“compositing”) a new image 108 depicting scene 125 from the camera’s perspective at a new camera viewpoint 126. The image rendering system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and techniques described below are implemented.

[0026] An “image” can typically be represented, for example, as an array of “pixels,” where each pixel is associated with a corresponding point in the image (i.e., a corresponding point in the camera’s image plane) and corresponds to a corresponding vector representing one or more numerical values ​​of image data at that point. For example, a two-dimensional (2D) RGB image can be represented by a 2D pixel array, where each pixel is associated with a corresponding three-dimensional (3D) vector representing the values ​​of the intensity of red, green, and blue at the point corresponding to the pixel in the image.

[0027] Throughout the instruction manual, "scene" can refer to, for example, a real-world environment or a simulated environment. For example, as... Figure 1 As shown, scene 125 may include various geometric objects in the simulated environment.

[0028] Typically, by utilizing improved view compositing techniques, System 100 can accurately render new images of large-scale scenes. For example, System 100 can render new images of urban driving scenes spanning multiple city blocks, such as an entire multi-block neighborhood in a dense town or city.

[0029] A camera "viewpoint" can refer to, for example, the camera's position and / or orientation within scene 125. The camera's position can be represented, for example, as a three-dimensional vector indicating the camera's spatial location within scene 125. The camera's orientation can be represented, for example, as a three-dimensional vector defining the direction the camera is facing, such as the camera's yaw, pitch, and roll.

[0030] For example, system 100 can synthesize new images as part of generating a computer simulation of a real-world environment navigating by a simulated autonomous vehicle and other agents. For instance, synthesized images can ensure that the simulation includes images similar to those encountered in the real-world environment, but captures novel views of scenes not available in images of the real-world environment. More generally, simulations can be part of testing control software for a real-world autonomous vehicle before it is deployed, part of training one or more machine learning models that will later be deployed on the autonomous vehicle, or both. As a specific example, the synthesized new images can be used to construct high-fidelity maps of the entire operating domain to serve as priors for testing software against various problems, including robot localization, navigation, and collision avoidance.

[0031] As another example, synthetic images can be used to augment training datasets that are used to train one or more machine learning models that will later be deployed on autonomous vehicles. That is, the system can generate synthetic images from novel viewpoints and use these synthetic images to improve the robustness of training datasets used to train one or more machine learning models, such as computer vision models. Examples of computer vision models include image classification models, object detection models, and so on.

[0032] As another example, a composite image can be generated and displayed to the user in the user interface to allow the user to view the environment from different perspectives and camera viewpoints. In one example, the image rendering system 100 can be used as part of a software application (e.g., referred to as a "street view" application for convenience) that provides the user with access to an interactive panorama showing the physical environment, such as the environment near a street. In response to a user request to view the physical environment from the camera's perspective in a new camera viewpoint, the street view application can provide the user with a rendered image of the environment generated by the image rendering system 100. As described below, the image rendering system can render a new image of the environment based on, for example, a collection of existing images of the environment previously captured by cameras mounted on vehicles traversing the environment.

[0033] In another example, the image rendering system 100 can be used to render images of a virtual reality environment, for example, implemented in a virtual reality headset or helmet. For instance, in response to a request received from a user to view the virtual reality environment from a different perspective, the image rendering system can render a new image of the virtual reality environment from the desired perspective and provide it to the user.

[0034] The image rendering system 100 can use multiple view composition models 140 to render a new image 108 of the scene 125.

[0035] Each view synthesis model 140 corresponds to a specific sub-region of scene 125 in the environment and is configured to receive input from a camera viewpoint within the specified sub-region and generate a synthesized image of the scene from the camera viewpoint as output. Typically, each view synthesis model 140 includes the same neural network with the same architecture, but the neural networks have different parameters due to the training of the models.

[0036] More specifically, each view synthesis model 140 has been trained, i.e., trained by system 100 or different training systems on different subsets of the set of training images 102. In particular, each view synthesis model 140 has been trained on a set of training images 120 taken from a camera viewpoint within the corresponding sub-region. Because each view synthesis model 140 corresponds to a different, potentially overlapping, sub-region, the training system can train the models 140 independently, and the presence of different models among the multiple models 140 can be modified by adding or removing new models 140 without retraining any other models among the maintained set of models.

[0037] The following is for reference. Figure 3 Describe an example technique used to train the view synthesis model 140.

[0038] At a high level, once model 140 is trained, system 100 is able to generate new images 108 by selecting a subset of multiple views from model 140 based on the new camera viewpoint 126.

[0039] Then, system 100 processes the corresponding input for the specified new camera viewpoint 126 using each model 140 in the subset to generate a synthetic image of the scene as output from the new camera viewpoint 126. Thus, system 100 generates a corresponding synthetic image for each model 140 in the subset.

[0040] Then, system 100 combines the synthetic images generated by the view synthesis model 140 in the subset to generate a final synthetic image of the scene from the new camera viewpoint 126.

[0041] Figure 2 An example of generating a new image is shown in the figure.

[0042] Figure 2 An example of the operation of system 100 is shown.

[0043] exist Figure 2 In the example, system 100 maintains three models 140. Each model 140 has a corresponding sub-region defined by a corresponding origin position 202 and radius 204, that is, such that the sub-region of each model 140 includes all points within the corresponding radius 204 of the origin position 202.

[0044] More generally, assuming that each point in the scene is in at least one sub-region for model 140, system 100 can divide the scene into sub-regions in any suitable manner.

[0045] For example, in an urban driving scenario, system 100 can place a model 140 at each intersection, which has a sub-region covering the intersection itself and 75% of any connecting streets, until it converges to the next intersection. This results in a 50% overlap between any two adjacent blocks on the connecting street segments, as described below, which makes appearance alignment easier. Following this process means that the block size is variable; additional blocks can be introduced as connectors between intersections if necessary.

[0046] As another example, that is, in Figure 2 In the example shown, system 100 instead places model 140 at uniform distances along a single street segment, and defines the size of each sub-region as a sphere around the corresponding origin 202.

[0047] exist Figure 2 In the example shown, the system receives a new camera viewpoint 210. (As...) Figure 2 As shown, the new camera viewpoint 210 specifies the position and orientation (“attitude”) of the camera in the scene.

[0048] from Figure 2 It can be seen that the new camera viewpoint 210 is within the corresponding sub-region of each of the three models 140. Therefore, the system 100 initially selects all three models 140 in a subset. However, in Figure 2 In the example, based on the corresponding visibility estimate 222 generated by system 100 for each of models 140, system 100 discards 220 of the models 140 from the subset. Typically, the visibility estimate of a given model estimates the degree to which a point along a ray projected from a new viewpoint is visible in the training images used to train the view synthesis model 140. Figure 2 In the example, visibility estimation 222 includes a corresponding visibility score ranging from zero to one for each pixel in the image taken from the target viewpoint 210. Because the discarded model 220 has a low visibility estimate (such as...), Figure 2The visibility 222 of the discarded model 220 is all black (i.e., all zero), which proves that system 100 discards model 220 from the subset.

[0049] The following is for reference. Figure 3 and Figure 4 The generation of visibility estimates and the determination of whether to discard the model are described in more detail 140.

[0050] For each of the two models 140 that were not discarded, the system 100 then generates a corresponding image 230 of the scene from the new viewpoint. Although Figure 2 Image 230 generated from discarded model 220 is also shown, but this is not necessary and can be omitted to improve the computational efficiency of the view compositing process. Figure 2 It can be seen that the image 230 generated by the discarded model 220 is blurry and may not add any valuable information to the image generated by the other models 140.

[0051] The following is for reference. Figure 3 This describes an example technique for generating images of a scene from a new viewpoint using a view synthesis model.

[0052] Then, system 100 combines the two new images 230 to generate a final image 240 of the scene from the new viewpoint.

[0053] The following is for reference. Figure 4 A more detailed description of the combination of images from multiple models 140.

[0054] Therefore, system 100 can accurately generate new images of a scene from any given viewpoint by combining different images of the scene generated by different models in model 140.

[0055] Figure 3 An example of the operation of one of the view synthesis models 140 is shown. As mentioned above, each model 140 will generally have the same architecture and will generate synthesized images in the same way. However, because each model 140 corresponds to a different sub-region of the scene and is trained on images associated with the corresponding sub-region, each model 140 will generally have different parameters after training, and therefore, given the same camera viewpoint, different models 140 will generate different images.

[0056] like Figure 3 As shown, model 140 includes a first neural network 300 and a second neural network 350.

[0057] First neural network 300(f) σThe first neural network 300 is configured to receive a first input including data representing the coordinates of a point x in the scene, and to process the first input to generate an output including (i) the volume density σ of the point x and (ii) a feature vector. For example, the first neural network 300 may be a multilayer perceptron (MLP) that processes the coordinates x to generate the output.

[0058] As a specific example, a point in a scene can be represented as a three-dimensional vector, for example, a spatial coordinate x.

[0059] Typically, the volume density at a point in a scene can characterize any appropriate aspect of the scene at that point. In one example, the volume density at a point in a scene can characterize the probability that a ray traveling through the scene will terminate at that point x in the scene.

[0060] Specifically, model 140 can be configured such that the volume density σ is generated independently of the viewing direction d, and therefore varies only as a function of points in the scene. This can promote volume consistency across different viewing angles within the same scene.

[0061] In some cases, volume density can have values ​​such as σ ≥ 0, where a zero value can represent, for example, a negligible probability that a ray will terminate at a particular point, perhaps indicating that there is no object in the scene at that point. On the other hand, a large positive value of volume density may indicate that there is an object in the scene at that point, and therefore a high probability that a ray will terminate at that location.

[0062] Second neural network 350(f) c The second neural network 350 is configured to receive inputs including a feature vector (generated by neural network 300) and data representing the viewing direction d, and to process the second input to generate color as output. For example, the second neural network 350 could also be an MLP that processes the feature vector d to generate color as output.

[0063] The color generated by the second neural network 350 as output for a given viewing direction d and point x is the radiation emitted at that point in the scene in that viewing direction, for example, RGB, where R is the emitted red, G is the emitted green, and B is the emitted blue.

[0064] Optionally, the "second" input to the second neural network 350 may also include additional information.

[0065] As an example, the input may also include appearance embeddings characterizing the target appearance of the synthesized image. Including appearance embeddings allows model 140 to consider appearance-changing factors, i.e., factors that could cause two images taken from the same point and viewing direction to have different appearances. Two examples of these factors are changing weather conditions and changing lighting conditions, such as time of day. Specifically, during training, the training system can be trained to incorporate appearance embeddings by generating latent optimizations to optimize the corresponding per-image appearance embedding for each training image. After training, system 100 can use these appearance embeddings to interpolate between different appearance-changing factors observed during training.

[0066] As another example, the second input may include the target camera exposure information for the synthesized image. That is, training images for model 140 can be captured at a wide range of exposure levels, which could potentially affect training if left unchecked. By including camera exposure information during training, model 140 can compensate for visual differences caused by different exposure levels. Therefore, after training, model 140 can generate images that appear to have been taken by a camera with the target exposure level by including the target exposure level as part of the second input. As an example, the exposure information can be represented as γ. PE (shutter speed x analog gain / t, where γ) PE It is a sinusoidal positional code with a fixed number of levels, such as 2, 4, or 6, and t is a scaling factor, such as equal to 250, 700, 1000, or 1500.

[0067] In some implementations, model 140 may also use sinusoidal position encoding γ. PE Let x and d be the inputs.

[0068] Typically, γ PE Each component z of a given input can be represented as a vector:

[0069] γ PE =[sin(2 0 z),cos(2 0 z)…,sin(2 L-1 z),cos(2 L-1 z),

[0070] Where L is the number of encoding levels.

[0071] Using this encoding scheme allows neural networks 300 and 350 to represent higher frequency details.

[0072] In some other implementations, model 140 uses sinusoidal position coding to represent d, while using integrated position coding to represent x.

[0073] Specifically, the system can use projected pixel footprints to sample frustums along the light rays instead of points (as described below). To feed these frustums into the MLP, model 140 approximates each of them as having a parameter μ. i , Σ i A Gaussian distribution is used, and its application is performed with parameter μ. i , Σ i The expected value of the input Gaussian is used to replace the positional encoding γ. PE That is, the integrated position encoding of a given frustum is derived from having parameter μ. i , Σ i Encoding γ of points sampled in a Gaussian distribution PE The expected value. Therefore, in these implementations, each point x along the ray sampling (described below) represents the expected value sampled from the truncated cone that has already been sampled from the ray.

[0074] Given a camera viewpoint as input, model 140 is able to generate synthetic images using neural networks 300 and 350.

[0075] More specifically, each pixel in the synthesized image (i.e., the synthesized image captured by the camera at the new camera viewpoint 126) can be associated with a ray projected from the camera's image plane into the scene at the new camera viewpoint 126. Given a camera viewpoint, the direction and position of the ray corresponding to a pixel in the new image can be computed as a predefined function of camera parameters, such as the camera's position and orientation, focal length, etc. In some implementations, potential inaccuracies in the camera's pose, provided to the system, are considered. Specifically, the system can jointly learn pose offset parameters defining the learned pose refinement with neural networks 300 and 350, and then use the pose offset parameters to adjust the provided pose when determining the direction and position of the ray corresponding to a pixel. For example, the pose offset parameters may include a position offset and a 3×3 residual rotation matrix.

[0076] The ray r(t) for a given pixel can be represented as:

[0077] r(t) = o + td, where t is the distance along the ray, o is the origin of the ray, for example, as specified by the new camera viewpoint, and d is the viewing direction corresponding to the pixel.

[0078] To generate the color of a given pixel in an image, model 140 can sample multiple points along the light rays from the new camera viewpoint and along the viewing direction corresponding to the pixel. For example, model 140 can randomly sample a distance t along the light rays to generate a sampling point r(t) for each sampling distance t.

[0079] For each sampling point, model 140 can generate a first input including data representing the coordinates of the sampling point, and process the first input using a first neural network in the view synthesis model to generate an output including the volume density and feature vector of the sampling point. Then, model 140 can generate a second input including the feature vector and data representing the viewing direction corresponding to the pixel (and optionally, target appearance embedding and target exposure level information), and process the second input using a second neural network in the view synthesis model to generate the color of the sampling point as the output.

[0080] Therefore, Model 140 obtains the corresponding color and corresponding volume density for each sampling point.

[0081] Then, model 140 uses the color and volume density of the sampled points to generate the final color of the pixel. For example, the system can accumulate the color of the sampled points using weights calculated based on the corresponding volume density of the sampled points. As a specific example, when there are N sampled points, the final output color c of a given pixel... out It can be equal to:

[0082] Where c i The color is calculated for point i. T i =exp(-∑ j<i Δ j σ j ), and Δ i =t i -t i-1 .

[0083] In some implementations, model 140 can be achieved by adjusting the weights w i Instead of directly using the distance t of random sampling to generate the final set of points for calculating the output color, the points are iteratively resampled as a probability distribution to better concentrate the samples in high-density regions.

[0084] System 100 or different training systems train each model 140, i.e., train a first neural network 300 and a second neural network 350 on training images taken from a viewpoint in a corresponding sub-region of model 140. Specifically, the training system can train the neural networks to minimize a discriminative rendering loss, which measures the error between a synthetic image of a scene from a given viewpoint generated by model 140 as described above and a training image of the scene taken from the given viewpoint. An example of such a loss function is described in Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields, ICCV, 2021. Another example of this loss function is described in Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the Field: Neural radiance fields for unconstrained photo collections, CVPR, 2021.

[0085] In some implementations, model 140 also includes a third neural network 370, which system 100 uses to compute a visibility estimate of model 140 for a new camera viewpoint. The visibility estimate estimates the degree to which points along rays projected from the new viewpoint are visible in training images used to train view synthesis model 140.

[0086] More specifically, the third neural network 370 is configured to receive a third input including data representing the coordinates of a point x in the scene and data representing the viewing direction d, and to process the third input to output an estimated transmittance of the point from the viewing direction. For example, the third neural network 370 may be an MLP.

[0087] Transmittance represents the visibility of a point from a specific input camera viewpoint: a point in free space or on the surface of the first intersecting object—that is, a point on the surface of the first object intersecting the light rays projected from the camera viewpoint to that point—will have a transmittance close to 1, and a point inside or behind the first visible object will have a transmittance close to 0. If a point is visible from some viewpoints but not from others, the regressed transmittance value will be the average across all training cameras and will be between zero and one, indicating that the point is partially observed. To train the third neural network 370, the system can train the neural network 370 to regress the T calculated above from the volume density generated by the first neural network. i The transmittance value is matched by using the output generated by the first neural network as supervision for training the third neural network 370.

[0088] To compute a visibility estimate for model 140 for a given new viewpoint, the system can sample multiple point-viewing direction pairs, for example, corresponding to different pixels in an image that will be generated from the given new viewpoint using model 140. For example, the sampled pairs can be all or a subset of the sampled pairs described above for generating the image.

[0089] Then, for each sample pair, the system can use a third neural network 370 in the view synthesis model 140 to process a third input including data representing the coordinates of points in the pair and data representing the viewing direction in the pair, to generate an estimated transmittance for the sample pair and determine a visibility estimate from the estimated transmittance of multiple points. For example, the system can compute the visibility estimate as the average of the estimated transmittance of multiple points.

[0090] The third neural network 370 can operate independently of the first neural network 350 and the second neural network 370, so the system can use the visibility estimate calculated by the third neural network 370 to determine whether to use the corresponding model 140 when generating a new image from a given new camera viewpoint.

[0091] The following is for reference. Figure 4 A more detailed description of the use of visibility estimation.

[0092] Figure 4 This is a flowchart of an example process 400 for rendering a new image. For convenience, process 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, an image rendering system appropriately programmed according to this specification, such as... Figure 1 System 100 can execute process 400.

[0093] As described above, the system maintains multiple view composition models. Each view composition model corresponds to a specific sub-region of the scene in the environment and is configured to receive input from a camera viewpoint in the specified sub-region and generate a composite image of the scene from the camera viewpoint as output.

[0094] The system receives input specifying a new camera viewpoint (step 402).

[0095] The system selects a subset of multiple view synthesis models based on the new camera viewpoint (step 404).

[0096] Typically, the system selects each view synthesis model with a corresponding sub-region that includes the new camera viewpoint to include in the subset.

[0097] Optionally, the system can then determine whether any of the selected view composition models should be removed from the subset.

[0098] For example, the system can compute a corresponding visibility estimate for each selected model—that is, for each view synthesis model having a corresponding sub-region including the new camera viewpoint—which estimates the visibility of a point along a ray projected from the new viewpoint in the training images used to train the view synthesis model. (See above reference.) Figure 3 An example technique for generating visibility estimates is described.

[0099] The system then removes any view synthesis models from the subset that have corresponding visibility estimates below the visibility threshold. This avoids using any view synthesis models that are unlikely to produce meaningful output from the new camera viewpoint.

[0100] For each view synthesis model in the subset, the system processes the corresponding input specifying the new camera viewpoint to generate a synthesized image of the scene from the new camera viewpoint as output (step 406).

[0101] In other words, the system uses a view synthesis model to process the corresponding inputs that specify a new camera viewpoint, in order to generate a synthesized image of the scene from the new camera viewpoint as output.

[0102] As described above, in some implementations, each view synthesis model includes a first neural network and a second neural network, and the first and second neural networks are used to generate a synthesized image given the corresponding inputs to the view synthesis model.

[0103] The above is for reference only. Figure 3 The generation of synthetic images using a first neural network and a second neural network is described.

[0104] As described above, in some implementations, the second neural network in each model also receives target camera exposure information, target appearance embedding, or both as input. That is, the corresponding input to each model also includes target camera exposure information, target appearance embedding, or both.

[0105] When the second neural network also receives target camera exposure information as input, the system can provide the same target exposure information to each model, making the generated images consistent with each other. For example, the system can receive the target camera exposure level as input, or it can randomly select a camera exposure level from a set of possible camera exposure levels. Therefore, by adjusting the target camera exposure information, the system can generate images that appear to have been taken by cameras with different exposure levels.

[0106] When the second neural network also receives the target appearance embedding as input, in some implementations, the system can provide the same target appearance embedding to each model, making the generated images consistent with each other. For example, the system can receive the target appearance embedding as input, or it can randomly select the appearance embedding from a set of possible appearance embeddings.

[0107] However, these embeddings (“code”) are randomly initialized during the training of each view synthesis model, and therefore the same code often results in different appearances when fed into different view synthesis models. This can be undesirable when synthesizing images, as it can lead to inconsistencies between views.

[0108] Therefore, in some other implementations, the system receives the target appearance embedding of a first view synthesis model in a subset, and generates corresponding appearance embeddings of other view synthesis models in the subset based on the target appearance embedding of the first view synthesis model.

[0109] For example, a user can provide a target appearance embedding that matches the appearance embedding from one of the training images used to train the first view synthesis model, so that the system generates an image taken under the same conditions. As another example, a user can provide a target appearance embedding that is a weighted sum of the appearance embeddings from multiple training images used to train the first view synthesis model, so that the system generates an image taken under conditions that are a combination of conditions from the multiple training images. As yet another example, a user can “search” for a target appearance embedding with the quality of interest by having the first model render multiple different images with different appearance embeddings and then selecting the appearance embedding that results in generating an image of the desired quality.

[0110] The system then uses the appearance embedding generated for that model to generate images for each of the other view-synthesized models. Thus, by adjusting the appearance embedding of the models, the system can generate images that appear to have been taken at different times of day, under different weather conditions, or under other external conditions that might affect the appearance of the camera images.

[0111] To generate a target appearance embedding for a given model, the system first selects a 3D matching location between the given model and neighboring models for which appearance embeddings have already been generated. For example, the system can select a matching location with visibility predictions that have thresholds for more than two models.

[0112] Given a matching location, the system freezes the model weights and optimizes only the appearance embedding of the given model to reduce the L2 loss between rendering corresponding regions at the matching location. Because the model weights are frozen, the system can perform this optimization quickly and computationally efficiently, requiring fewer than 100 iterations to converge. The system then uses the optimized appearance embedding as the target appearance embedding for the given model. This process aligns most global and low-frequency attributes of the scene, such as time of day, color balance, and weather, between the two models, allowing for successful synthesis of images generated from both models.

[0113] The optimized appearance is iteratively propagated through the scene, starting from the first view synthetic model (the "root" model). If multiple models around a given model have already been optimized, the system considers each of them when calculating the loss, i.e., by including the corresponding L2 loss of each of the multiple models in the optimization.

[0114] The system combines the synthetic images generated by the view synthesis models in the subset to generate a final synthetic image of the scene from the new camera viewpoint (step 408).

[0115] For example, the system can determine the corresponding weights for each view synthesis model in the subset, and then generate the final synthesized image by interpolating between synthesized images generated by the view synthesis models in the subset according to the corresponding weights of the view synthesis models in the subset. That is, for each pixel, the system interpolates between the color outputs of pixels in the synthesized image according to the corresponding weights of the corresponding view synthesis model.

[0116] As an example, the system can determine the corresponding weights for each model based on the distance between the new camera viewpoint and the center (“origin”) of the corresponding sub-region of the scene. As a specific example, the system can assign weights w to the i-th model. i Calculate w i ∝distance(c,x i ) -p Where c is the new camera viewpoint position, x iis the position of the center of the corresponding sub-region of the i-th model, and p is a constant value that affects the blending rate between images.

[0117] In some implementations, the system performs the above-described process 400 only when the viewpoint is located in a region of the environment that is a corresponding sub-region of multiple view composition models included in the view composition model. That is, when the viewpoint is located in a region of the environment that has only a single corresponding model, the system uses a single view composition model to generate a composite image, for example, without checking visibility and without combining the output images as described above.

[0118] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a specific operation or action, this means that software, firmware, hardware, or a combination thereof have been installed on the system that causes the system to perform the operation or action in operation. For one or more computer programs to be configured to perform a specific operation or action, this means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0119] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control of the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0120] The term "data processing apparatus" refers to data processing hardware and encompasses all types of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0121] A computer program, also referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as a portion of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple harmonizing files, for example, a file storing one or more modules, subroutines, or code portions. A computer program can be deployed to execute on a single computer or on multiple computers located at one site or distributed across multiple sites and interconnected via a data communication network.

[0122] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0123] The processes and logic flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by dedicated logic circuitry, such as FPGAs or ASICs, or by a combination of dedicated logic circuitry and one or more programmable computers.

[0124] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to them, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0125] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0126] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device running a messaging application, such as a smartphone, and receiving response messages in return from the user.

[0127] The data processing apparatus used to implement machine learning models may also include, for example, dedicated hardware accelerator units for handling the common and computationally intensive parts of machine learning training or production, namely inference and workloads.

[0128] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework or the Jax framework.

[0129] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server, or middleware components, such as an application server, or frontend components, such as a client computer having a graphical user interface, web browser, or app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0130] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data, such as HTML pages, to a user device, for purposes such as displaying data to a user interacting with the device and receiving user input from it, the device acting as a client. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0131] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0132] Similarly, although operations are depicted in the accompanying drawings and described in a specific order in the claims, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or to perform all of the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0133] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method executed by one or more computers, the method comprising: Multiple view synthesis models are maintained, wherein each view synthesis model corresponds to a corresponding sub-region of the scene of the environment and is configured to receive input from a camera viewpoint in a specified corresponding sub-region and generate a synthesized image of the scene from the camera viewpoint as output, and wherein each view synthesis model has been independently trained on a corresponding set of training data, the corresponding set of training data including images captured from the viewpoint in the corresponding sub-region of the scene. Obtain input for the specified new camera viewpoint; Based on the new camera viewpoint, a subset of the multiple view synthesis models is selected; For each view synthesis model in the subset, the corresponding input specifying a new camera viewpoint is processed to generate a synthesized image of the scene from the new camera viewpoint as output; and The composite images generated by the view synthesis models in the subset are combined to generate a final composite image of the scene from the new camera viewpoint.

2. The method according to claim 1, wherein, A subset of multiple view synthesis models based on the new camera viewpoint selection includes: Each view synthesis model having a corresponding sub-region including the new camera viewpoint is selected to be included in the subset.

3. The method according to claim 2, further comprising: For each view synthesis model having a corresponding sub-region including the new camera viewpoint, a corresponding visibility estimate is determined, which estimates the degree to which a point along a ray cast from the new viewpoint is visible in the training images used to train the view synthesis model. as well as Remove any view synthesis models from the subset that have a corresponding visibility estimate below the visibility threshold.

4. The method according to claim 1, wherein, Combining the synthetic images generated by the view synthesis models in the subset to generate a final synthetic image of the scene from the camera viewpoint includes: Determine the corresponding weights for each view composition model in the subset; and The final composite image is generated by interpolating between composite images generated by the view synthesis models in the subset according to the corresponding weights for the view synthesis models in the subset.

5. The method according to claim 4, wherein, Determining the corresponding weights for each view composition model in the subset includes, for each view composition model: The corresponding weights are determined based on the distance between the new camera viewpoint and the center of the corresponding sub-region of the scene.

6. The method according to claim 3, wherein, Each view composition model includes: A first neural network is configured to receive a first input including data representing the coordinates of a point in the scene, and to process the first input to generate an output including the volume density and feature vector of that point; and The second neural network is configured to receive a second input including feature vectors and data representing the viewing direction, and to process the second input to generate color as output.

7. The method according to claim 6, wherein, Processing the input corresponding to a specified new camera viewpoint to generate a composite image of the scene from the new camera viewpoint as output includes, for each pixel in the image: Sample multiple points along the light rays from the new camera viewpoint and along the viewing direction corresponding to the pixel; For each sampling point: Generate a first input, which includes data representing the coordinates of the sampling point; The first neural network in the view synthesis model is used to process the first input to generate an output that includes the volume density and feature vectors of the sampled points; Generate a second input, the second input including the feature vector and data representing the viewing direction corresponding to the pixel; and The second neural network in the view synthesis model is used to process the second input to generate the color of the sampled point as the output; as well as The color of the pixel is generated using the color and volume density of the sampling point.

8. The method according to claim 7, wherein, For each view synthesis model, the second input includes a corresponding appearance embedding that characterizes the target appearance of the synthesized image.

9. The method according to claim 8, further comprising: Receive the target appearance embedding of the first view synthesis model in the subset; as well as The corresponding appearance embeddings for other view synthesis models in the subset are generated based on the target appearance embedding for the first view synthesis model.

10. The method of claim 8, further comprising: Receive the target appearance embedding; as well as Set the corresponding appearance embedding of the view synthesis model in the subset as the target appearance embedding.

11. The method according to claim 7, wherein, For each view synthesis model, the second input includes data representing the target camera exposure information about the synthesized image.

12. The method according to claim 6, wherein, Each view synthesis model includes a third neural network configured to receive a third input, the third input including data representing the coordinates of the point in the scene and data representing the viewing direction, and to process the third input to output an estimated transmittance of the point from the viewing direction, wherein, for each view synthesis model having a corresponding sub-region including the new viewpoint, determining the corresponding visibility estimate includes: For each of the multiple point-viewing-direction pairs, a third neural network in the view synthesis model is used to process a third input, including data representing the coordinates of the points in the pair and data representing the viewing direction in the pair, to generate an estimated transmittance; and The visibility estimate is determined from the estimated transmittance at the plurality of points.

13. The method according to claim 12, wherein, The visibility estimate is the average of the estimated transmittance.

14. A system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to perform operations of a corresponding method according to any one of claims 1-13.

15. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operation of a corresponding method according to any one of claims 1-13.

Images