Rendering novel images of a scene using a geometric shape recognition neural network conditioned on latent variables

Abstract

Methods, systems, and apparatus including computer programs encoded on computer storage media are provided for rendering a new image depicting a scene from a viewpoint of a camera at a new camera location. In one aspect, the method includes receiving a plurality of observations characterizing a scene, generating latent variables representing the scene from the plurality of observations characterizing the scene, conditioning a scene representation neural network with the latent variables representing the scene, where the scene representation neural network conditioned on the latent variables representing the scene defines a geometric model of the scene as a three-dimensional (3D) radiance field, and rendering a new image depicting the scene from a viewpoint of a camera at the new camera location using the scene representation neural network conditioned on the latent variables representing the scene.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 145,782, filed February 4, 2021, which is incorporated herein by reference in its entirety.

[0002] This specification relates to processing data using machine learning models. [Background technology]

[0003] A machine learning model receives input and generates an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models, which generate an output based on the received input and the values ​​of the parameters of the model.

[0004] Some machine learning models are deep models that use multiple layers of the model to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to the received input to generate an output. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Mildenhall et al., “NeRF: Representing scenes as neural radiance fields for view synthesis”, arXiv:2003.08934v2 (2020) Summary of the Invention [Means for solving the problem]

[0006] This specification describes an image rendering system implemented as a computer program on one or more computers at one or more locations that is capable of rendering new images depicting a scene from the viewpoint of a camera at a new camera location.

[0007] Throughout this specification, a "scene" can refer to, for example, a real-world environment, or a simulated environment (e.g., a simulation of a real-world environment, e.g., where the simulated environment is a synthetic representation of a real-world scene).

[0008] An "embedding" of an entity may refer to a representation of the entity as an ordered set of numbers, for example, a vector, matrix, or other tensor of numbers.

[0009] In one aspect, a system and method are described for rendering a new image depicting a scene from the perspective of a camera at a new camera location. The system receives an image of the scene and a plurality of observations characterizing the scene, each observation comprising data identifying the location of the camera that captured the image of the scene, and generates from the observations latent variables that represent the scene.

[0010] The system conditions a (trained) scene representation neural network with latent variables representing a scene. The scene representation neural network defines a model of the scene as a three-dimensional (3D) radiance field. The scene representation neural network is configured to receive a representation of a spatial location within the scene and a representation of an observation direction, and to process the representation of the spatial location within the scene, the representation of the observation direction, and the latent variables representing the scene to generate an output that defines the radiance emitted at that spatial location within the scene in that observation direction.

[0011] The system renders a new image depicting the scene from the viewpoint of the camera at the new camera location by projecting radiance values ​​from a model of the scene as a 3D radiance field onto the image plane of the camera at the new camera location.

[0012] As described below, a 3D radiance field defines, for example, the directional radiance emitted from every spatial location in a scene. A 3D radiance field can therefore be a vector field that represents a scene as radiance (emitted light) at each 3D position and across each direction in 3D in the 3D space of the scene, for example as a viewing direction dependent emitted color, and optionally including a volumetric density, as described below.

[0013] This model can be referred to as a geometric model because it is based on a geometric representation, i.e., (3D) position and, as suggested by radiance, direction. In an implementation, rather than simply generating images conditioned on latent variables, the scene representation neural network models the scene based on (3D) spatial locations and directions, and then uses these to generate outputs that define the radiance of (i.e., along) the observation direction. This model is then used to render a new image by projecting radiance values, accumulated or integrated, e.g., along rays to the camera, e.g., to pixels in the image plane, onto the image plane of the camera at a new camera location. This form of representation models the 3D space of the scene rather than an image from any particular viewpoint, and therefore can ensure consistency across different viewpoints as well as generalization to new viewpoints. It is not necessary for the scene representation neural network to have any particular architecture; this advantage comes from the type of representation being processed and the output being generated. This approach can also reduce the amount of computation and training required, because the scene representation neural network does not need to learn how to render images.

[0014] According to a first aspect, there is provided a method, performed by one or more data processing devices, for rendering a new image depicting a scene from the perspective of a camera at a new camera location, comprising: receiving a plurality of observations characterizing the scene, each observation comprising (i) an image of the scene, and (ii) data identifying the location of a camera that captured the image of the scene; generating latent variables representing the scene from the plurality of observations characterizing the scene; and conditioning a scene representation neural network on the latent variables representing the scene, wherein the scene representation neural network conditioned on the latent variables representing the scene defines a geometric model of the scene as a three-dimensional (3D) radiance field. to generate an output defining a radiance emitted at the spatial location in the scene in the observation direction; and rendering a new image depicting the scene from the viewpoint of a camera at the new camera location by projecting radiance values ​​from a geometric model of the scene as a 3D radiance field onto an image plane of the camera at the new camera location using a scene representation neural network conditioned on the latent variables representing the scene.

[0015] In some implementations, generating latent variables representing the scene from a plurality of observations characterizing the scene includes generating parameters of a probability distribution over a space of latent variables from the plurality of observations characterizing the scene, and sampling latent variables representing the scene from the space of latent variables according to the probability distribution over the space of latent variables.

[0016] In some implementations, generating parameters of a probability distribution over a space of latent variables from a plurality of observations characterizing the scene includes: generating a respective representation for each observation; processing each representation for each observation using an encoding neural network to generate a corresponding embedding for each observation; and generating parameters of a probability distribution over a space of latent variables from the embeddings of the multiple observations. Includes:

[0017] In some implementations, generating parameters of a probability distribution over a space of latent variables from an embedding of the multiple observations includes averaging embeddings of the multiple observations, where the parameters of the probability distribution over the space of latent variables are based on the average of the embeddings of the multiple observations.

[0018] In some implementations, generating parameters of a probability distribution over the space of latent variables from an embedding of the multiple observations includes initializing current parameters of a current probability distribution over the space of latent variables; for each time step in the series of time steps, sampling a current latent variable from the space of latent variables according to the current probability distribution over the space of latent variables; conditioning a scene representation neural network on the current latent variables; rendering an image depicting the scene from the perspective of a camera at a target camera location using the scene representation neural network conditioned on the current latent variables; determining a gradient of an objective function with respect to the current parameters of the current probability distribution over the space of latent variables, wherein the objective function depends on (i) the rendered image depicting the scene from the perspective of a camera at the target camera location and (ii) a target image of the scene captured from a camera at the target camera location; and updating the current parameters of the current probability distribution over the space of latent variables using (i) the gradient of the objective function and (ii) the embedding of the multiple observations.

[0019] In some implementations, the latent variable representing the scene comprises multiple latent subvariables.

[0020] In some implementations, the scene representation neural network comprises a series of one or more update blocks, each configured to receive a current joint embedding of a spatial location in the scene and an observation direction, and to update the current joint embedding of the spatial location in the scene and the observation direction using attention across one or more of a plurality of latent subvariables of the latent variable.

[0021] In some implementations, the attention is multi-head query-key-value attention.

[0022] In some implementations, processing representations of spatial locations within the scene, representations of observation directions, and latent variables representing the scene to generate an output defining a radiance emitted in the observation direction at the spatial locations within the scene includes generating a joint embedding of the spatial locations within the scene and the observation direction from the representations of the spatial locations within the scene and the representations of the observation direction, updating the joint embedding using each update block in a series of one or more update blocks, and generating an output defining a radiance emitted in the observation direction at the spatial locations within the scene from the updated joint embedding generated by the last update block in the series of update blocks.

[0023] In some implementations, each latent partial variable comprises multiple channels, each update block is assigned a respective latent partial variable, and for each update block, updating the current joint embedding using attention across one or more of the multiple latent partial variables of the latent variable includes updating the current joint embedding using attention across only the latent partial variables assigned to that update block.

[0024] In some implementations, rendering the new image includes identifying, for each pixel of the new image, a ray that corresponds to that pixel and is cast into the scene from the image plane of the camera at the new camera location; determining, for each of a plurality of spatial locations on the ray, a radiance emitted in the direction of the ray at that spatial location on the ray using a scene representation neural network conditioned on latent variables that represent the scene; and rendering a color of the pixel in the new image based on the radiance emitted in the direction of the ray at the plurality of spatial locations on the ray.

[0025] In some implementations, the method further includes determining, for each of a plurality of spatial locations on the ray, a volume density of the scene at that spatial location that characterizes the likelihood that the ray will stop at that spatial location, and rendering a color of a pixel in the new image based on both the radiance emitted in the direction of the ray at the plurality of spatial locations on the ray and the volume density at the plurality of spatial locations on the ray.

[0026] In some implementations, determining, for each of a plurality of spatial locations on the ray, the radiance emitted at that spatial location in the direction of the ray and the volume density at that spatial location includes providing a representation of the spatial location on the ray and a representation of the ray direction to a scene representation neural network conditioned on latent variables representing the scene to generate an output defining the radiance emitted at that spatial location in the direction of the ray and the volume density at that spatial location.

[0027] In some implementations, rendering the color of a pixel in the new image based on both the radiance emitted in the direction of the ray at multiple spatial locations on the ray and the volume density at multiple spatial locations on the ray includes accumulating the radiance emitted in the direction of the ray at multiple spatial locations on the ray and the volume density at multiple spatial locations on the ray.

[0028] In some implementations, the scene representation neural network has a plurality of neural network parameters, and the scene representation neural network is trained to determine post-trained values ​​of the neural network parameters from initial values ​​of the neural network parameters before being used to render new images of the scene, and training the scene representation neural network includes, for each of a plurality of other scenes, conditioning the scene representation neural network with latent variables representing the other scenes, rendering one or more images each depicting the other scenes from the viewpoint of a camera at a location within the other scenes using the scene representation neural network conditioned on the latent variables representing the other scenes, and updating current values ​​of the neural network parameters of the scene representation neural network using gradients of an objective function that depend on the images of the other scenes rendered using the scene representation neural network conditioned on the latent variables representing the other scenes.

[0029] According to another aspect, one or more non-transitory computer storage media are provided that store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the methods described herein.

[0030] According to another aspect, a system is provided that includes one or more computers and one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the methods described herein.

[0031] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0032] The image rendering system enables rendering of new images of a scene, i.e., depictions of the scene from the perspective of a camera at a new camera location, based on a set of existing images of the scene and data defining the respective locations of the cameras that captured each existing image. The image rendering system therefore enables more efficient use of resources, such as memory resources and camera resources, by allowing new images of a scene to be synthetically rendered on demand from a set of existing images. More specifically, without the image rendering system, generating new images of a scene may require physically capturing the new images using a camera, which requires the use of camera resources and may be impractical for practical reasons. Furthermore, synthetically rendering new images of a scene on demand reduces memory resource usage, for example, by freeing up memory space that would otherwise be required to store new images before they are needed.

[0033] The image rendering system described herein can render new images of a scene using a scene representation neural network that, when conditioned on latent variables that represent the scene, defines a geometric model of the scene. Explicitly incorporating the geometric scene model enables the image rendering system to render new images of the scene with greater geometric accuracy and realism than, for example, a system that renders new images without incorporating an explicit geometric model. For example, incorporating the geometric scene model enables the image rendering system to more effectively render new images from camera locations that are significantly different from the locations of the cameras that captured existing images of the scene.

[0034] The image rendering system trains parameter values ​​of a scene representation neural network (i.e., one that implements a geometric scene model) using a set of images captured from multiple different scenes. The image rendering system can then use the scene representation neural network to render images of new scenes without retraining the parameter values ​​of the scene representation neural network on images captured from the new scenes. Specifically, rather than retraining the scene representation neural network on images captured from the new scenes, the image rendering system conditions the scene representation neural network on latent variables that represent the new scenes. Thus, the geometric scene model learns a structure shared across scenes, and information about a particular scene is encoded in the latent variables. Conditioning the scene representation neural network on latent variables that represent new scenes allows the image rendering system to avoid retraining the scene representation neural network for each new scene, thereby reducing the consumption of computational resources (e.g., memory and computing power).

[0035] Furthermore, the image rendering system can generate latent variables that effectively represent new scenes and enable accurate rendering of new images using significantly fewer images of new scenes than would be required to retrain the scene representation neural network. Thus, by conditioning the scene representation neural network with latent variables that represent new scenes (e.g., rather than retraining the scene representation neural network for each new scene), the image rendering system enables more efficient use of resources.

[0036] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0037] [Figure 1] FIG. 1 is a block diagram of an exemplary image rendering system. [Figure 2] FIG. 1 illustrates an example of a volume rendering of an image of a scene. [Figure 3] FIG. 2 is a block diagram of a more detailed exemplary image rendering system. [Figure 4] FIG. 1 is a block diagram of an exemplary scene representation neural network included within an image rendering system. [Figure 5] FIG. 1 is a flow diagram of an exemplary process for rendering a new image using an image rendering system. [Figure 6] FIG. 1 is a flow diagram of an exemplary process for training an encoding neural network and a scene representation neural network included in an image rendering system. [Figure 7] FIG. 1 illustrates experimental results obtained using an image rendering system. [Figure 8]FIG. 1 illustrates experimental results obtained using an image rendering system. DETAILED DESCRIPTION OF THE INVENTION

[0038] Like reference numbers and designations in the various drawings indicate like elements.

[0039] 1 is a block diagram of an exemplary image rendering system 100 capable of rendering a new image 108 depicting a scene 125 from the perspective of a camera at a new camera location 126d. Image rendering system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.

[0040] An "image" can generally be represented, for example, as an array of "pixels," where each pixel is associated with a respective spatial location in the image (i.e., a respective spatial location in the image plane of the camera) and corresponds to a respective vector of one or more numerical values ​​that represent the image data at that spatial location. For example, a two-dimensional (2D) RGB image can be represented by a 2D pixel array, where each pixel is associated with a respective three-dimensional (3D) vector of values ​​that represent the intensities of red, green, and blue colors at the spatial location corresponding to that pixel in the image.

[0041] Throughout this specification, a "scene" can refer to, for example, a real-world environment or a simulated environment. For example, as shown in FIG. 1, a scene 125 can include various geometric objects in a simulated environment. A camera "location" can refer to, for example, the spatial location and / or orientation of a camera. The spatial location of a camera can be expressed, for example, as a three-dimensional vector indicating the spatial position of the camera. The orientation of a camera can be expressed, for example, as a three-dimensional vector that defines the direction in which the camera is oriented, e.g., the yaw, pitch, and roll of the camera.

[0042] The image rendering system 100 can render a new image 108 of a scene 125 based on multiple observations 102 that characterize the scene 125. Each observation can include (i) an image of the scene 125 and (ii) data identifying the location of a camera that captured the image of the scene 125. For example, as shown in Figure 1, each image depicts the scene 125 from a different observation perspective, e.g., from the perspective of a camera at a respective camera location. For example, a first image depicts the scene 125 from the perspective of a camera at location 126a, a second image depicts the scene 125 from the perspective of a camera at location 126b, and a third image depicts the scene 125 from the perspective of a camera at location 126c.

[0043] Some or all of the images included in the plurality of observations 102 may depict the scene 125 from different viewpoints. In general, the plurality of observations 102 may include images depicting the scene 125 from any number of different viewpoints, for example, 1, 5, 50, 100, or 1000 different viewpoints.

[0044] The image rendering system 100 can process the plurality of observations 102 and the data defining the new camera location 126d to render a new image 108 that depicts the scene 125 from the perspective of the camera at the new camera location 126d (e.g., at a new orientation and / or spatial location of the camera within the scene 125). In some implementations, the new location 126d of the camera can be different from any of the camera locations 126a, 126b, 126c associated with images included in the plurality of observations 102. In other words, the system 100 can render a new image 108 that depicts the scene 125 from an entirely new observational perspective.

[0045] The image rendering system 100 can render a novel image 108 based on latent variables 104 that represent a scene 125. A "latent variable" can generally refer to an embedding in a latent space. In general, the latent variables 104 can implicitly represent features specific to the scene 125, such as the location and type of objects depicted in the scene 125, the color and lighting in the scene 125, or any other suitable feature of the scene 125. In one particular example, the latent variables can be a 128-dimensional vector of numerical values. As another particular example, the latent variables can be, for example, a vector of size [H z ,W z ,D z ] can be a three-dimensional array with

[0046] The image rendering system 100 can generate the latent variables 104 using an encoding neural network 110. For example, the encoding neural network 110 can be configured to process a plurality of observations 102, including images depicting a scene 125 from different observation viewpoints and data identifying corresponding camera locations 126 a, 126 b, 126 d, to generate parameters of a probability distribution over the space of latent variables.

[0047] After generating the parameters of the probability distribution using the encoding neural network 110, the image rendering system 100 can sample latent variables 104 representing the scene 125 from the space of latent variables according to the probability distribution. The system 100 can then use the latent variables 104 to condition a scene representation neural network 120, which can then be used to render a new image 108 of the scene 125 from the perspective of a camera at a new camera location 126d. In general, "conditioning" the scene representation neural network 120 with the latent variables 104 representing the scene 125 can refer to providing the latent variables 104 as inputs to the scene representation neural network, for example, to be processed along with other inputs to the scene representation neural network 120.

[0048] Once conditioned on latent variables 104 representing the scene 125, the scene representation neural network 120 can define a geometric model of the scene 125, for example, as a three-dimensional radiance field that defines the directional radiance emitted at every spatial location within the scene 125. More specifically, by processing an observation direction 107 and a spatial location 106 within the scene 125 (along with the conditioning latent variables 104), the scene representation neural network 125 can output a value that defines the corresponding radiance emitted at that spatial location 106 within the scene 125 in that observation direction 107. The radiance emitted at a spatial location within the scene in a observation direction can characterize, for example, the amount of light passing through that spatial location within the scene 125 in that observation direction. In other words, after being conditioned on the latent variables 104, the scene representation neural network 120 can generate a corresponding emitted radiance each time it is “queried” with a particular spatial location 106 and observation direction 107 within the scene 125.

[0049] Furthermore, each pixel in the new image 108 (i.e., as captured by the camera at the new camera location 126d) can be associated with a ray that is cast from the image plane of the camera at the new camera location 126d into the scene 125. As described in more detail below with reference to FIG. 2, for each pixel in the new image, the image rendering system 100 can use the scene representation neural network 120 conditioned on the latent variables 104 to generate multiple values ​​of directional emission radiance along the ray that corresponds to that pixel. The image rendering system 100 can then render the new image 108, e.g., on a pixel-by-pixel basis, by projecting the radiance values ​​from a geometric model of the scene onto the image plane of the camera at the new camera location 126d, e.g., by using volume rendering, as described in more detail below with reference to FIG. 2. The new image 108 may depict the scene 125 from the perspective of the camera at the new camera location 126d (e.g., from an entirely new observation perspective).

[0050] As a specific example, a spatial location 106 within a scene 125 can be represented, for example, as a three-dimensional vector x of spatial coordinates. An observation direction 107 within the scene 125 can be represented, for example, as a three-dimensional unit vector d. As another example, an observation direction can be represented using a two-dimensional vector (θ, φ) in a spherical coordinate system. As shown in equation (1), a scene representation neural network G conditioned on a latent variable z representing the scene 125 and having a set of parameters θ can process a representation (x, d) to generate an emitted radiance, for example (r, g, b), in that observation direction at that spatial location within the scene, where r is the emitted red color, g is the emitted green color, and b is the emitted blue color.

[0051]

number

[0052] In some implementations, the output of the scene representation neural network 120 may further include a volumetric density σ of the scene 125 at that spatial location x. In general, the volumetric density at a spatial location in a scene may characterize any suitable aspect of the scene 125 at that spatial location. In one example, the volumetric density at a spatial location in a scene may characterize the likelihood that a ray of light traveling through the scene will end up at that spatial location x in the scene 125. Specifically, the scene representation neural network may be configured such that the volumetric density σ is generated independently of the observation direction d and thus varies only as a function of the spatial location within the scene 125. This may promote volumetric consistency across different observation viewpoints of the same scene 125.

[0053] In some cases, the volume density may have a value, e.g., σ≧0, where a value of zero may represent, for example, that there is a negligible chance that a ray of light will come to rest at a particular spatial location, and may indicate, for example, that there is likely no object at that spatial location in the scene 125. On the other hand, a large positive value of the volume density may indicate that there is likely an object at that spatial location in the scene 125, and therefore that there is a high probability that the ray will come to rest at that location.

[0054] As explained above, the latent variables 104 may capture features specific to the scene 125, such as the location and type of objects depicted in the scene 125, the color and lighting in the scene 125, or any other suitable feature of the scene 125. That is, the latent variables 104 may be understood as a semantic representation of the scene.

[0055] Scene representation neural network 120 can be trained using a collection of images captured within each corresponding scene, and thus, after training, the parameters of neural network 120 can capture or store shared information between different scenes, such as texture and shape, properties of common elements, or any other features shared between different scenes.

[0056] Explicitly incorporating a geometric model of scene 125 as a three-dimensional radiance field may enable image rendering system 100 to render new images of scene 125 with greater geometric accuracy and realism than, for example, a system that renders new images without incorporating an explicit geometric model. For example, incorporating a geometric scene model (i.e., in the form of a radiance field represented by a scene representation neural network) may enable image rendering system 100 to more effectively render new images from camera locations that are significantly different from the locations of the cameras that captured the images contained in the plurality of observations 102.

[0057] The encoding neural network 110 and the scene representation neural network 120 may each have any suitable neural network architecture that enables them to perform their described functions. For example, they may include any suitable number of suitable neural network layers (e.g., convolutional layers, fully connected layers, recurrent layers, attention layers, etc.) connected in any suitable configuration (e.g., as a linear sequence of layers). As a specific example, the encoder neural network 110 may include a series of one or more encoder blocks and an output block. Each encoder block may include, for example, one or more convolutional neural network layers, a batch normalization neural network layer, and a residual connection that combines inputs to the encoder block with outputs from the encoder block. The output block may include, for example, one or more batch normalization neural network layers and one or more convolutional neural network layers. An exemplary architecture of the scene representation neural network 120 is described in more detail below with reference to FIG. 4.

[0058] In some implementations, the image rendering system 100 may include a training engine that can jointly train the scene representation neural network 120 and the encoding neural network 110 on a set of training data over multiple training iterations. An exemplary process for training the scene representation neural network 120 and the encoding neural network is described in more detail below with reference to FIG.

[0059] After training, the scene representation neural network 120 can be conditioned with latent variables that represent any scene. After being conditioned with latent variables that represent a particular scene, the image rendering system 100 can use the neural network 120 to render one or more images depicting that particular scene from different observation viewpoints. That is, after generating latent variables 104 that represent a scene from a set of observations of the scene, the image rendering system 100 can generate any desired number of new images of the scene using the scene representation neural network conditioned on the latent variables without regenerating the latent variables.

[0060] Image rendering system 100 can be used in any of a variety of possible applications, some examples of possible applications of image rendering system 100 are described in more detail below.

[0061] In one example, the image rendering system 100 can be used as part of a software application (e.g., for convenience referred to as a "street view" application) that provides a user with access to an interactive panorama showing a nearby physical environment, e.g., a street. In response to a user request to view the physical environment from the perspective of a camera at a new camera location, the street view application can provide the user with a rendered image of the environment generated by the image rendering system. The image rendering system can render a new image of the environment based on a collection of existing images of the environment previously captured, e.g., by cameras mounted on vehicles that have passed through the environment.

[0062] In another example, the image rendering system 100 can be used to process a collection of one or more images (e.g., x-ray images) of a biomolecule (e.g., a protein) along with data defining the location of the imaging sensor that captured the image of the biomolecule, thereby rendering a new image of the biomolecule from a new perspective.

[0063] In another example, the image rendering system 100 may be used to render images of a virtual reality environment, such as that implemented in a virtual reality headset or helmet. For example, in response to receiving a request from a user to view the virtual reality environment from a different viewpoint, the image rendering system may render a new image of the virtual reality environment from the desired viewpoint and provide it to the user.

[0064] The techniques described herein can be extended to sequences of images, for example, video.

[0065] 2 shows an example of volume rendering of an image 215 of a scene 250 using a radiance field. Volume rendering can be used by an image rendering system (such as system 100 of FIG. 1) to render an image 215 that depicts the scene 250 from the perspective of a camera at a new camera location 216. Specifically, the image rendering system can render the image 215 using a scene representation neural network 240 that defines a geometric model of the scene as a three-dimensional radiance field 260.

[0066] As described above with reference to FIG. 1, a scene representation neural network 240 (e.g., neural network 120 of FIG. 1) included in an image rendering system, when conditioned on latent variables representing scene 250 (e.g., latent variables 104 representing scene 125 of FIG. 1), can define a geometric model of scene 250, for example as a three-dimensional radiance field 260 that defines the directional emission radiance at different spatial locations within scene 250.

[0067] The inputs of the scene representation neural network may include representations of spatial locations within the scene 250 and representations of observation directions within the scene 250. As shown in FIG. 2 , the observation direction may be, for example, the direction of a ray 210 having a particular orientation and projected from the image plane of the camera at the new camera location 216 through the scene 250. Thus, the representation of the observation direction may be, for example, a three-dimensional unit vector d that defines the orientation of the ray 210. A spatial location 230 within the scene 250 (e.g., indicated by a circle on the ray 210) may be represented, for example, as a three-dimensional vector x of spatial coordinates. In general, each pixel in the image 215 generated by the camera at the new camera location 216 may be associated with a ray 210 projected from the camera's image plane into the scene 250, and with multiple spatial locations 230 on the ray 210, each corresponding to a particular spatial location within the scene 250. (The direction and position of the ray corresponding to a pixel in the new image may be calculated as a predefined function of camera parameters, such as the camera's position and orientation, the camera's focal length, etc.).

[0068] The image rendering system can query a scene representation neural network 240, conditioned on latent variables representing a scene 250, with a particular spatial location 230 and observation direction 210 within the scene 250. The scene representation neural network 240 can generate a corresponding output that defines the radiance (r, g, b) emitted in that direction 210 at that spatial location 230 within the scene 250. In some implementations, the neural network 240 can also generate a volumetric density σ at that spatial location 230 within the scene 250 that characterizes the likelihood that, for example, a ray 210 will end up at that spatial location 230 within the scene 250. After generating the radiance values ​​(r, g, b) using the scene representation neural network 240, the image rendering system can render a new image 215 of the scene 250 by projecting these values ​​onto the image plane of a camera at a new camera location 216.

[0069] As a particular example, for each pixel in the new image corresponding to the camera at the new camera location, the image rendering system may identify a corresponding ray 210 that corresponds to that pixel and is projected from the camera's image plane into scene 250. The image rendering system may use a neural network 240 to determine a radiance value (r, g, b) for each of a plurality of spatial locations 230 (e.g., represented as circles in FIG. 2 ) along ray 210, e.g., at predefined intervals along the ray. The system may then render the color of the pixel in the new image based on those radiances (r, g, b), e.g., by accumulating the radiances (r, g, b) determined for different spatial locations on ray 210. The image rendering system may repeat this process for each pixel in the new image captured by the camera at new camera location 216, thereby rendering the color of every pixel in new image 215.

[0070] In some implementations, for each spatial location on the ray 210, the image rendering system can also use the neural network 240 to determine the volumetric density σ of the scene 250 at that spatial location. The image rendering system can render the color of each pixel in the new image based on both (i) the radiance value and (ii) the volumetric density at each point along the ray corresponding to that pixel. The graphs included in Figure 2 show an example of the volumetric density σ varying as a function of spatial location 230 in the scene 250, with each graph being specific to a particular observation direction (e.g., "ray 1" or "ray 2").

[0071] To render the new image, the system may render the color of each pixel in the new image 215 by accumulating the radiance emitted in the direction of the ray 210 at multiple spatial locations 230 on the ray 210 and the volume density σ at multiple spatial locations 230 on the ray 210. The system may accumulate the radiance and volume density along the ray corresponding to a pixel using any suitable accumulation technique. For example, the system may accumulate the radiance and volume density by scaling each radiance value along the ray by the corresponding volume density and then summing the scaled radiance values. Other techniques for accumulating the radiance and volume density along the ray corresponding to a pixel are described in Mildenhall et al., "NeRF: Representing scenes as neural radiance fields for view synthesis," arXiv:2003.08934v2 (2020), which also describes the correspondence between volume density and alpha values.

[0072] Explicitly incorporating geometric model 260 of scene 250 allows an image rendering system to render new images of scene 250 with greater geometric accuracy and realism than, for example, a system that renders new images without incorporating an explicit geometric model. For example, incorporating geometric model 260 allows the system to more effectively render new images from camera locations that are significantly different from the locations of the cameras that captured existing images of the scene. As a particular example, the system may use neural network 240 conditioned on latent variables representing scene 250 to render another new image 220 of scene 250 from the perspective of a camera at an entirely different camera location 225 (e.g., shown as perpendicular to camera location 216).

[0073] The image rendering system will now be described in more detail with reference to FIG.

[0074] 3 is a block diagram of a more detailed example image rendering system 300 (e.g., image rendering system 100 of FIG. 1 ). Image rendering system 300 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.

[0075] 1, image rendering system 300 can include encoding neural network 310 and scene representation neural network 320. System 300 can process a plurality of observations 302, each observation including an image of a scene and data identifying the location of a camera that captured the image of the scene, to generate latent variables 304 that represent the scene. System 300 can condition scene representation neural network 320 on latent variables 304 and use scene representation neural network 320 to render a new image 308 that depicts the scene from the perspective of a camera at a new camera location.

[0076] The image rendering system 300 can use an encoding neural network 310 to process multiple observations 302 and generate parameters of a probability distribution over a space of latent variables. The system 300 can then sample latent variables representing a scene from the space of latent variables according to the probability distribution. The encoding neural network 110 can generate the parameters of the probability distribution in any of a variety of ways. In one example, the encoding neural network 110 can combine embeddings of multiple observations 102, generated, for example, by processing the multiple observations 102 using the encoding neural network 110.

[0077] More specifically, before generating parameters for a probability distribution over the space of latent variables, the system may process multiple observations 302 to generate a respective representation for each observation 302 .

[0078] For each pixel in an image depicting a scene, the system can use the camera parameters (e.g., field of view, focal length, camera spatial location, camera orientation, etc.) of the camera that captured the image to determine the parameters (e.g., orientation and position parameters) of the ray that corresponds to that pixel and that is projected from the camera's image plane into the scene. The ray that is projected into the scene for each pixel can have an orientation that is expressed, for example, as a three-dimensional unit vector d. The system can generate a representation of each observation by concatenating, for each pixel in the image included in the observation, data that defines the corresponding ray orientation for that pixel. In other words, each pixel in the image can have an associated five- or six-dimensional feature vector (c, d), where, for example, c represents the RGB color of the pixel and d represents the orientation of the ray that corresponds to the pixel.

[0079] As a particular example, image I of the scene k and the corresponding camera location c that captured the image of the scene. k For an observation k that defines k can be generated as follows: C k =concat(I k ,map_to_rays(c k )) (2) where "concat" is the concatenation operator and "map_to_rays" is the operator described above that determines the orientation of each ray corresponding to each pixel in the image. After generating representations of the multiple observations 302, for example according to equation (2), the system may provide each representation of each observation as an input to an encoding neural network 310. The encoding neural network 310 may be configured to process the representations of the observations to generate an embedding of the observations. An "embedding" of an observation may refer to a representation of the observation as an ordered set of numeric values, for example, a vector, matrix, or other tensor of numeric values.

[0080] The system can use the embeddings of multiple observations 302 to generate parameters of a probability distribution over the space of latent variables. For example, the system can process the respective embeddings of each observation to generate a combined embedding, and then process the combined embedding using one or more neural network layers to generate parameters of a probability distribution over the space of latent variables. A probability distribution q over the space of latent variables z, with parameter λ, conditioned on a set of observations C is given by q λ (z|C). In one example, the system uses an embedding of multiple observations, h k By averaging the embeddings, we can generate a combined embedding, h, e.g.

[0081]

number

[0082] where N is the total number of observations and h k is the embedding of observation k and h is the mean of the embedding. The system 100 then computes a probability distribution q over the space of latent variables z. λ The parameter λ of (z|C) can be generated as follows: λ=MLP(h) (4) However, "MLP" can refer to Multi-Layer Perceptron.

[0083] The image rendering system 300 generates a probability distribution q with parameters λ determined by an encoding neural network 310 from the space of latent variables. λ The latent variables representing the scene can be sampled according to (z|C). In one example, the probability distribution can be parameterized, for example, as a multidimensional Gaussian distribution, where each dimension is parameterized by a respective mean and standard deviation parameter.

[0084] The system 300 can then condition the image rendering neural network 320 with the latent variables, e.g., as described above with reference to FIG. 2, and render using a rendering engine a new image depicting the scene from the camera's perspective at the new camera location.

[0085] An exemplary scene representation neural network 320 included within the image rendering system 300 is described in more detail below.

[0086] 4 is a block diagram of an exemplary scene representation neural network 400 included within an image rendering system (e.g., scene representation neural network 120 of FIG. 1 ). Scene representation neural network 400 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below are implemented.

[0087] As described above with reference to FIGS. 1 and 3 , an encoding neural network included in an image rendering system can process a plurality of observations, each including an image of a scene and data identifying the location of a camera that captured the image, to generate parameters of a probability distribution over a space of latent variables. The system can sample latent variables representing the scene according to the probability distribution. The system can then condition a scene representation neural network 400 on the latent variables, causing the scene representation neural network to define a 3D geometric model of the scene as a 3D radiance field. More specifically, when conditioned on the latent variables, the scene representation neural network can process representations of spatial locations within the scene and representations of observation directions to generate data defining radiance emitted at spatial locations within the scene in the observation direction and, optionally, volumetric density at the spatial locations within the scene. The system uses the 3D radiance field defined by the scene representation neural network conditioned on the latent variables to render a new image of the scene from the viewpoint of a camera at a new camera location, e.g., using volume rendering techniques.

[0088] In some implementations, the latent variable representing the scene may include multiple (e.g., a set of) latent subvariables, e.g., variables that can be combined to form a latent variable. For example, a latent variable z may have a size [H z ×W z ×D z ]. The array can be divided into multiple subarrays, each subarray defining a respective subvariable of the latent variable. In one particular example, the array can be an array having a channel dimension (e.g., D z ) into subarrays along the channel dimensions, e.g., such that each subarray corresponds to a proper subset of the channel dimensions (i.e., a subset of less than all of the channel dimensions) and has size [H z ×W z ×D'z ], but D' z <D z Each latent subvariable has a latent embedding {(h, w, :):1 ≤ h ≤ H z ,1≦w≦W z}, where each latent embedding has dimensionality D' z It has.

[0089] In such cases, the scene representation neural network 400 can use one or more update blocks to process representations of spatial locations and observation directions within a scene to generate an output that defines the radiance emitted in the observation direction at that spatial location within the scene. Specifically, the scene representation neural network 400 can include a series of one or more update blocks (e.g., "BAB" in FIG. 4), each configured to update an input embedding using attention across a corresponding latent subvariable, e.g., by applying an attention mechanism to the corresponding latent subvariable to generate a combined latent embedding that is used to update the input embedding. Many different attention mechanisms can be used. The attention mechanism can be a query-key-value attention mechanism, for example, in which the combined latent embedding is calculated as a weighted sum of values, where the weight of each value is calculated by a compatibility function between the query and the corresponding key, e.g., a dot product or a scaled dot product, and the input embedding is mapped to the query and each latent embedding contained in the latent subvariable is mapped to a key and a value. The attention mechanism can implement multi-head attention, i.e., multiple different attention mechanisms can be applied in parallel.

[0090] For example, to update the input embeddings using attention across the latent partial variables, the update block can generate a respective attention weight for each latent embedding contained in the latent partial variables and generate a combined latent embedding based on the latent embeddings contained in the latent partial variables and the corresponding attention weights. As a particular example, the update block can generate the combined latent embedding as a weighted sum of the latent embeddings in the latent partial variables, e.g., by multiplying each latent embedding in the latent partial variables by its corresponding weight and summing the weighted latent embeddings. The update block can then use the combined latent embedding to update the input embeddings, e.g., by replacing the input embeddings with the combined latent embedding, by adding the combined latent embedding to the input embedding, or in any other suitable manner.

[0091] In some implementations, the update block can perform a query-key-value (QKV) attention operation, e.g., updating an input embedding using attention across latent embeddings in latent partial variables using a query (Q) embedding, a key (K) embedding, and a value (V) embedding. Specifically, each update block can include (i) a query sub-network, (ii) a key sub-network, and (iii) a value sub-network. For each input embedding, the query sub-network can be configured to process the input embedding to generate a respective query embedding (Q) for the input embedding. The key sub-network can be configured to process each latent embedding contained in the latent partial variable corresponding to the update block to generate a respective key embedding (K). Similarly, the value sub-network can be configured to process each latent embedding contained in the latent partial variable corresponding to the update block to generate a respective value embedding (V).

[0092] Each update block can then update the input embeddings using the query embedding (Q), key embedding (K), and value embedding (V). Specifically, each update block can generate attention weights for each latent embedding in the corresponding latent partial variable, e.g., as an inner (e.g., dot) product of the query embedding (Q) and each of the key embeddings (K). Based on the set of latent embeddings and the attention weights in the latent partial variables, each update block can generate a combined latent embedding, e.g., as a linear combination of the value embeddings (V) weighted with their corresponding attention weights. Finally, each update block can update the input embeddings using the combined latent embeddings.

[0093] As a particular example, the scene representation neural network 400 may generate an embedding circ(x) of a representation of a spatial location x=(x1, x2, x3) in a scene, for example, as follows:

[0094]

number

[0095] The scene representation neural network 400 can use one or more update blocks to update the embedding circ(x), e.g., using attention over the latent partial variables as described above. The neural network 400 can then process the updated embedding using a multi-layer perceptron neural network layer (e.g., "MLP" in FIG. 4) to generate an output that defines the volumetric density σ at that spatial location within the scene.

[0096] Additionally, the scene representation neural network 400 can generate an embedding circ(d) of the representation of the observation direction, e.g., using a procedure similar to that described with reference to equation (5). The neural network 400 can then process the embedding circ(d) using a linear neural network layer (e.g., "Linear" in FIG. 4) and generate a joint embedding by combining (e.g., concatenating) the output from the linear layer with the updated embedding of circ(x) generated by the update block.

[0097] The scene representation neural network 400 can use each update block in the series of update blocks to update the joint embedding, e.g., using attention over the set of latent partial variables as described above. After the joint embedding is updated by the final update block in the series of update blocks, the neural network 400 can process the joint embedding using multi-layer perceptron neural network layers to generate an output that defines the radiance (e.g., (r, g, b) in FIG. 4) emitted in the observation direction at that spatial location in the scene.

[0098] As explained above, the latent variable can be an array, and the array can be divided into multiple subarrays, where each subarray can define a respective subvariable of the latent variable. In one particular example, the array can be a channel dimension (e.g., D z), e.g., such that each subarray corresponds to a proper subset of the channel dimensions (i.e., a subset of less than all of the channel dimensions). That is, when a latent variable comprises an array, the channels of the latent subvariable can correspond to the channels of the latent variable. In some implementations, each update block of the scene representation neural network 400 can be assigned a respective latent subvariable (whether each latent subvariable comprises multiple channels). In such a case, each update block can update its input embedding using attention over only the latent embedding of the latent subvariable assigned to that update block.

[0099] The scene representation neural network 400 and the update block may have any suitable neural network architecture that enables them to perform their described functions. For example, in addition to including an attention neural network layer, the update block may further include any other suitable neural network layers (e.g., convolutional layers, fully connected layers, recurrent layers, attention layers, etc.) of any suitable number (e.g., 2, 5, or 10 layers) connected in any suitable configuration (e.g., as a linear series of layers). The scene representation neural network 400 may include any number of update blocks and any number of other neural network blocks configured to perform any suitable operations.

[0100] An exemplary process for rendering a new image using the image rendering system is now described in more detail.

[0101] 5 is a flow diagram of an exemplary process 500 for rendering a new image using an image rendering system. For convenience, process 500 is described as being performed by a system of one or more computers located at one or more locations. For example, an image rendering system suitably programmed in accordance with this specification, such as system 100 of FIG. 1, may perform process 500.

[0102] The system receives a plurality of observations characterizing a scene 502. Each observation may include (i) an image of the scene, and (ii) data identifying the location of a camera that captured the image of the scene.

[0103] The system generates latent variables representing the scene from multiple observations characterizing the scene (504). In some implementations, the system can use the multiple observations to generate parameters of a probability distribution over the space of latent variables. For example, the system can generate a representation for each observation, process each representation using an encoding neural network to generate a corresponding embedding for each observation, and generate the parameters of the probability distribution from the embeddings.

[0104] As a particular example, the system may, for example, average the embeddings and then process the average embeddings through one or more neural network layers to generate parameters of a probability distribution over the space of latent variables. For example, the system may generate a combined embedding h as described above and process it using an MLP to generate parameters of the probability distribution.

[0105] As another specific example, the system can generate parameters of a probability distribution over a space of latent variables using an iterative process. More specifically, the system can initialize the current parameters of the probability distribution to default (e.g., predefined or random) values. After initializing the current parameters, for each time step in the series of time steps, the system can sample a current latent variable from the space of latent variables, e.g., according to a current probability distribution over the space of latent variables conditioned on one of a plurality of observations 302 characterizing the scene. The current probability distribution may be conditioned on multiple, e.g., all, observations by averaging embeddings of the multiple, e.g., all, observations as described above. The system can then condition a scene representation neural network on the current latent variables and render an image depicting the scene from the viewpoint of a camera at a target camera location using the scene representation neural network conditioned on the current latent variables. The target camera location can be the location of the camera at one of the plurality of observations 302 characterizing the scene.

[0106] At each time step, the system may determine the gradient of an objective function with respect to the current parameters of the probability distribution over the space of latent variables, for example, the objective function described below with reference to equation (6). Finally, at each time step, the system may update the current parameters of the probability distribution over the space of latent variables using (i) the gradient of the objective function with respect to the current parameters of the probability distribution, and (ii) an embedding of multiple training observations, i.e., generated using an embedding neural network.

[0107] For example, the system may process one of the multiple observations 302, e.g., the observation corresponding to the target camera location, using an encoding neural network to generate an embedding E(C) of the observation. The system may then calculate the current parameters λ of the probability distribution at time step t+1.t+1 of,

[0108]

number

[0109] can be determined as, where λ t are the parameters of the probability distribution at time step t,

[0110]

number

[0111] is, for example, λ as defined in equation (6) below. t denotes the gradient of the objective function L with respect to , and f(·) denotes a recurrent "refinement" neural network, e.g., a long short-term memory (LSTM), followed by scaling the output of the LSTM with the parameter λ. t is a linear layer for mapping L to L. Thus, a refinement neural network can be a neural network that processes the embedding of the observations and the gradient of the objective function with respect to the current parameters of the probability distribution to generate updates to the current parameters. In some cases, the gradient of L can be calculated by processing only a subset of the pixels in the image of the observations.

[0112] After generating the parameters of the probability distribution over the space of latent variables, the system can sample latent variables representing the scene from the space of latent variables according to the probability distribution.

[0113] The system conditions a scene representation neural network with latent variables representing the scene (506). The scene representation neural network can define a geometric model of the scene as a three-dimensional (3D) radiance field. In some implementations, the scene representation neural network can receive representations of spatial locations within the scene and representations of observation directions. The neural network can process these representations and the latent variables representing the scene to generate an output that defines the radiance emitted at spatial locations within the scene in the observation direction.

[0114] The system renders a new image depicting the scene from the viewpoint of the camera at the new camera location using a scene representation neural network conditioned on the latent variables representing the scene (508). For example, the system can project radiance values ​​from a geometric model of the scene as a 3D radiance field onto the image plane of the camera at the new camera location.

[0115] In some implementations, the system can render a new image based on rays corresponding to each pixel in the image. Specifically, the system can identify a ray that corresponds to the pixel and is cast into the scene (in the direction the camera is facing) from the image plane of the camera at the new camera location. For each of multiple spatial locations on the ray, the system can determine the radiance emitted in the direction of the ray at that spatial location on the ray using a scene representation neural network conditioned on latent variables representing the scene. The system can then render the color of the pixel in the new image based on the radiance emitted in the direction of the ray at the multiple spatial locations on the ray.

[0116] In some implementations, for each of a plurality of spatial locations on the ray, the system can determine a volumetric density of the scene at that spatial location, which characterizes the likelihood that the ray will stop at that spatial location. The system can then render the color of a pixel in the new image based on both the radiance emitted in the direction of the ray at the plurality of spatial locations on the ray and the volumetric density at the plurality of spatial locations on the ray.

[0117] In some implementations, determining the radiance emitted in the direction of the ray at that spatial location and the volumetric density at that spatial location may include providing representations of the spatial location on the ray and the direction of the ray to a scene representation neural network conditioned on latent variables representing the scene to generate an output defining the radiance emitted in the direction of the ray at that spatial location and the volumetric density at that spatial location.

[0118] In some implementations, rendering the color of a pixel in the new image based on both the radiance emitted in the direction of the ray at multiple spatial locations on the ray and the volume density at multiple spatial locations on the ray may include accumulating the radiance emitted in the direction of the ray at multiple spatial locations on the ray and the volume density at multiple spatial locations on the ray.

[0119] In general, the scene representation neural network has a set of scene representation neural network parameters, and the encoding neural network has a set of encoding neural network parameters. The scene representation neural network and the encoding neural network can be jointly trained to determine post-training values ​​of their corresponding neural network parameters from initial values ​​of their corresponding neural network parameters. For example, the system can train the scene representation neural network and the encoding neural network on multiple observations from multiple scenes.

[0120] FIG. 6 is a flow diagram of an exemplary process 600 for training an encoding neural network and a scene representation neural network included in an image rendering system (eg, system 100 of FIG. 1).

[0121] As described above with reference to Figure 1, the image rendering system may include a training engine capable of training the scene representation neural network and the encoding neural network on a set of training data over multiple training iterations. In some implementations, the training engine may jointly train the scene representation neural network and the encoding neural network, for example, by backpropagating the gradient of an objective function through the scene representation neural network into the encoding neural network.

[0122] The training data can include a set of training examples, where each training example includes a set of observations of a particular scene, e.g., images depicting the scene from different observation viewpoints, and data identifying the corresponding locations of the cameras. Each set of scene observations can be divided into (i) a set of training observations and (ii) a set of target observations.

[0123] The system can train the encoding neural network and the scene representation neural network for each training example in the training data. For convenience, each step in Figure 6 describes training the encoding neural network and the scene representation neural network for a particular training example.

[0124] The system processes multiple training observations to generate latent variables representing a scene 602. An exemplary technique for processing multiple observations to generate latent variables representing a scene using an encoding neural network is described above with reference to steps 502-504 of Figure 5.

[0125] The system conditions (604) a scene representation neural network with latent variables that represent the scene.

[0126] The system generates a respective predicted image corresponding to each target observation using (i) a scene representation neural network conditioned on the latent variables and (ii) a camera location specified by the target observation (606). In some cases, rather than generating a pixel value for each pixel in the predicted image, the system generates a respective probability distribution over a set of possible pixel values ​​for each pixel in the predicted image. An exemplary process for rendering predicted images using a scene representation neural network that defines a 3D geometric model of the environment is described above with reference to FIG. 2.

[0127] The system determines 608 the gradient of an objective function that depends on the predicted image of the scene rendered using the scene representation neural network, for example, using backpropagation techniques. The objective function may be any suitable objective function, for example, a variational objective function (e.g., an evidence lower bound function). For example, the objective function L evaluated for a target image defined by the target observations may be: L=E z~q [logp θ (I|z,c)]-KL(q λ (z|C)||p(z)) (6) where p θ (I|z,c) characterizes the error between the predicted image and the target image, and q λ (z|C) denotes the probability distribution over the space of latent variables, p(z) denotes the predefined prior distribution over the space of latent variables, and KL(q λ (z|C)||p(z)) is q λ indicates the divergence (e.g., Kullback-Leibler divergence) between (z|C) and p(z). In some implementations, p θ (I|z,c) defines the product of likelihood of the value of each pixel in the target image under the probability distribution generated for that pixel using the scene representation neural network. θ (I|z,c) may represent the error between the predicted image and the target image, for example, the L2 error.

[0128] In some implementations, the system can maintain two instances of the scene representation neural network 320, one based on a coarse set of 3D points and the other based on a finer set of 3D points. These can be conditioned on the same latent variables, and their outputs can be combined to render an image. In that case, during training, equation (6) adds an additional likelihood term (logp θ (I)).

[0129] The training engine can update the parameter values ​​of the scene representation neural network and the encoding neural network using gradients, for example, using any suitable gradient descent optimization algorithm, such as Adam or RMSprop.

[0130] In some cases, the system can iteratively optimize parameters of the probability distribution. More specifically, the system can initialize current parameters of the probability distribution to default (e.g., predefined or random) values. After initializing the current parameters, for each time step in the series of time steps, the system can sample current latent variables from the space of latent variables according to the current probability distribution over the space of latent variables. The system can then condition a scene representation neural network on the current latent variables and render an image depicting the scene from the viewpoint of a camera at a target camera location (defined by the target observation) using the scene representation neural network conditioned on the current latent variables.

[0131] At each time step, the system may determine the gradient of an objective function, e.g., the objective function described with reference to Equation (6), with respect to the current parameters of the probability distribution over the space of latent variables. Finally, at each time step, the system may update the current parameters of the probability distribution over the space of latent variables using (i) the gradient of the objective function with respect to the current parameters of the probability distribution, and (ii) an embedding of multiple training observations, i.e., generated using an embedding neural network. For example, the system may determine the current parameters λ of the probability distribution at time step t+1. t+1 of,

[0132]

number

[0133] can be determined as, where λ t are the parameters of the probability distribution at time step t, E(C) denotes the average embedding of multiple training observations,

[0134]

number

[0135] is λ t where f(·) denotes the gradient of the objective function with respect to , e.g., the "refinement" neural network described above. The parameters of the refinement neural network can be trained together with the parameter values ​​of the encoding neural network.

[0136] After determining the optimized parameters of the probability distribution over the latent space, the system can sample the latent variables according to the optimized probability distribution and then proceed to perform steps 604-610 using the latent variables sampled from the optimized probability distribution.

[0137] 7 shows experimental results obtained using the image rendering system described herein. Specifically, graph 702 plots the mean squared error (MSE) of images generated by (i) the image rendering system described herein (red line) and (ii) an alternative image rendering system (blue line). The MSE is plotted against the "number of views," which represents the number of observations used by the image rendering system to generate latent variables used, for example, to condition a scene representation neural network. Graph 704 shows a histogram of the MSE of images generated by the image rendering system described herein, and graph 706 shows a histogram of the MSE of images generated by the alternative image rendering system. It can be seen that the image rendering system described herein significantly outperforms the alternative image rendering system.

[0138] FIG. 8 compares the predictive accuracy of images generated using (i) the image rendering system described herein ("II+ATT") and (ii) an alternative image rendering system ("C-AR-VAE"). It can be seen that the image rendering system described herein achieves a lower MSE for generating images from "within-distribution" viewpoints and "out-of-distribution" viewpoints. A viewpoint of a new image can be referred to as "in-distribution" if, for example, the new image is drawn from the same distribution as the viewpoints of a set of existing images of a scene used to generate latent variables for conditioning the scene representation neural network. Conversely, a viewpoint of a new image can be referred to as "out-of-distribution" if the new image is drawn from a different distribution than the viewpoints of the set of existing images of a scene, for example, if the viewpoint of the new image is outside the support of the training data distribution. Diagram 804 shows a diagram contrasting an within-distribution viewpoint with an out-of-distribution viewpoint.

[0139] The term "configured" is used herein with respect to systems and computer program components. A system of one or more computers is configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform the operation or action. One or more computer programs are configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0140] The embodiments and functional operations of the present subject matter described herein can be implemented as digital electronic circuitry, as tangibly embodied computer software or firmware, as computer hardware including the structures disclosed herein and their structural equivalents, or as a combination of one or more of them. The embodiments of the present subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial-access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated, propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to an appropriate receiver device for execution by the data processing apparatus.

[0141] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus can also be or include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus can optionally also include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0142] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, or declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data, such as in one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple linked files, such as files containing one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0143] The term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines may be installed and running on the same one or more computers.

[0144] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0145] A computer suitable for running a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, and / or is operatively coupled to receive data from or transfer data to them. However, a computer need not have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0146] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0147] To enable interaction with a user, embodiments of the present subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can input to the computer. Other types of devices can also be used to enable interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device being used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, such as a smartphone, running a messaging application and receiving a response message in return from the user.

[0148] Data processing devices for implementing machine learning models may also include dedicated hardware accelerator units, for example for handling common computationally intensive parts of machine learning training or production, i.e., inference, workloads.

[0149] The machine learning model can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0150] Embodiments of the present subject matter described herein can be implemented within a computing system that includes a back-end component, e.g., a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, web browser, or app through which a user can interact with an implementation of the present subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0151] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., HTML pages, to a user device, e.g., for the purpose of displaying data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of a user interaction, may be received from the device at the server.

[0152] While this specification contains many specific details of implementations, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as working in a particular combination, and even initially claimed as such, one or more features from a claimed combination can, in some cases, be deleted from that combination, and the claimed combination may be directed to a subcombination or variations of the subcombination.

[0153] Similarly, although operations are depicted in the figures or described in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequence shown, or that all of the operations shown be performed, to achieve desirable results. Under certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0154] Specific embodiments of the present subject matter have been described above. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0155] 100 Image rendering system, encoding neural network 102 Observation Results 104 Conditioning Latent Variables 106 Spatial Location 107 Observation Direction 108 new images 110 Encoding Neural Network, Encoder Neural Network 120 Scene Representation Neural Network 125 Scenes, Scene Representation Neural Networks 126a Camera Location 126b Camera Location 126c Camera Location 126d New camera location, new location 210 Ray, Observation Direction 215 new images 216 New Camera Locations 220 new images 225 Camera Locations 230 Spatial Location 240 Scene Representation Neural Network 250 scenes 260 3D Radiance Field, Geometric Model 300 Image Rendering System 302 Observation Results 304 Latent Variables 310 Encoding Neural Networks 320 Scene representation neural network, image rendering neural network 400 Scene Representation Neural Network 500 processes 600 processes 702 graphs 704 graphs 706 graphs 804 Figures

Claims

1. 1. A method, implemented by one or more data processing devices, for rendering a new image depicting a scene from the viewpoint of a camera at a new camera location, comprising: receiving a plurality of observations characterizing the scene, each observation comprising (i) an image of the scene, and (ii) data identifying the location of a camera that captured the image of the scene; generating latent variables representing the scene from the plurality of observations characterizing the scene; conditioning a scene representation neural network with the latent variables representing the scene, wherein the scene representation neural network conditioned on the latent variables representing the scene defines a geometric model of the scene as a three-dimensional (3D) radiance field; receiving a representation of a spatial location within the scene and a representation of an observation direction; and processing the representation of the spatial location within the scene, the representation of the observation direction, and the latent variables representing the scene to generate an output defining a radiance emitted in the observation direction at the spatial location within the scene; and rendering the new image depicting the scene from the viewpoint of the camera at the new camera location by projecting radiance values ​​from the geometric model of the scene as a 3D radiance field onto an image plane of the camera at the new camera location using the scene representation neural network conditioned on the latent variables representing the scene; A method comprising:

2. generating the latent variables representing the scene from the plurality of observations characterizing the scene, generating parameters of a probability distribution over a space of latent variables from the plurality of observations characterizing the scene; sampling the latent variables representing the scene from the space of latent variables according to the probability distribution over the space of latent variables; 2. The method of claim 1, comprising:

3. generating parameters of the probability distribution over the space of latent variables from the plurality of observations characterizing the scene, generating a respective representation of each observation; for each observation, processing the representation of the observation using an encoding neural network to generate an embedding of the observation; generating the parameters of the probability distribution over the space of latent variables from the embedding of the plurality of observations; 3. The method of claim 2, comprising:

4. generating the parameters of the probability distribution over the space of latent variables from the embedding of the plurality of observations, averaging the embeddings of the plurality of observations, wherein the parameters of the probability distribution over the space of latent variables are based on the average of the embeddings of the plurality of observations.

4. The method of claim 3, comprising:

5. generating the parameters of the probability distribution over the space of latent variables from the embedding of the plurality of observations, initializing current parameters of a current probability distribution over the space of latent variables; For each time step in the sequence of time steps, sampling a current latent variable from the space of latent variables according to the current probability distribution over the space of latent variables; conditioning the scene representation neural network on the current latent variables; Rendering an image depicting the scene from the viewpoint of a camera at a target camera location using the scene representation neural network conditioned on the current latent variables; determining a gradient of an objective function with respect to the current parameters of the current probability distribution over the space of the latent variables, the objective function depending on (i) the rendered image depicting the scene from the viewpoint of the camera at the target camera location, and (ii) a target image of the scene captured from the camera at the target camera location; updating the current parameters of the current probability distribution over the space of latent variables using (i) the gradient of the objective function, and (ii) the embedding of the plurality of observations; 4. The method of claim 3, comprising:

6. The method of claim 1 , wherein the latent variable representing the scene comprises a plurality of latent subvariables.

7. The scene representation neural network comprises a series of one or more update blocks, each update block comprising: receiving a current joint embedding representing the spatial location within the scene and the observation direction; updating the current joint embedding using attention across one or more of the plurality of latent subvariables of the latent variable; The method of claim 6, configured to:

8. The method of claim 7 , wherein the attention is a multi-headed query-key-value attention.

9. processing the representation of the spatial location within the scene, the representation of the observation direction, and the latent variables representing the scene to generate an output defining the radiance emitted in the observation direction at the spatial location within the scene, generating, from the representation of the spatial location within the scene and the representation of the observation direction, the joint embedding of the representation of the spatial location within the scene and the representation of the observation direction; updating the joint embedding using each update block in the series of one or more update blocks; generating the output defining the radiance emitted in the observation direction at the spatial location within the scene from the updated joint embedding generated by a last update block in the series of update blocks; 9. The method of claim 7 or 8, comprising:

10. each latent subvariable comprising a plurality of channels, each update block being assigned a respective latent subvariable, and for each update block, updating the current joint embedding using attention across one or more of the plurality of latent subvariables of the latent subvariable; updating the current joint embedding using attention over only the latent subvariables assigned to the update block; The method according to any one of claims 7 to 9, comprising:

11. The step of rendering the new image comprises, for each pixel of the new image: identifying a ray cast into the scene from the image plane of the camera at the new camera location that corresponds to the pixel; determining, for each of a plurality of spatial locations on the ray, a radiance emitted in a direction of the ray at the spatial location on the ray using the scene representation neural network conditioned on the latent variables representing the scene; rendering a color of the pixel in the new image based on the radiance emitted in the direction of the ray at the plurality of spatial locations on the ray; The method according to any one of claims 1 to 10, comprising:

12. for each of the plurality of spatial locations on the ray, determining a volume density of the scene at the spatial location that characterizes the likelihood that the ray will come to rest at that spatial location; rendering the color of the pixel in the new image based on both the radiance emitted in the direction of the ray at the plurality of spatial locations on the ray and the volume density at the plurality of spatial locations on the ray; 12. The method of claim 11, further comprising:

13. determining, for each of the plurality of spatial locations on the ray, the radiance emitted in the direction of the ray at the spatial location and the volume density at the spatial location, providing a representation of the spatial location on the ray and a representation of the direction of the ray to the scene representation neural network conditioned on the latent variables representing the scene to generate an output defining the radiance emitted in the direction of the ray at the spatial location and the volumetric density at the spatial location.

13. The method of claim 12, comprising:

14. Rendering the color of the pixel in the new image based on both the radiance emitted in the direction of the ray at the plurality of spatial locations on the ray and the volumetric density at the plurality of spatial locations on the ray, accumulating the radiance emitted in the direction of the ray at the plurality of spatial locations on the ray and the volume density at the plurality of spatial locations on the ray.

14. The method of claim 12 or 13, comprising:

15. the scene representation neural network has a plurality of neural network parameters, the scene representation neural network is trained to determine post-trained values ​​of the neural network parameters from initial values ​​of the neural network parameters before being used to render the new image of the scene, and training the scene representation neural network includes, for each of a plurality of other scenes: conditioning the scene representation neural network with latent variables representing the other scenes; rendering one or more images, each depicting the other scene from the viewpoint of a camera at a location within the other scene, using the scene representation neural network conditioned on the latent variables representing the other scene; updating current values ​​of the neural network parameters of the scene representation neural network using gradients of an objective function that depends on the images of the other scenes rendered using the scene representation neural network conditioned on the latent variables representing the other scenes; 15. The method of any one of claims 1 to 14, comprising:

16. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 15.

17. one or more computers; one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 15; A system comprising:

Images