Three-dimensional diffusion model

Abstract

Systems and methods for performing new view synthesis on a three-dimensional (3D) scene using a machine-learned diffusion model are provided. Example implementations of the proposed models may be referred to as "3D diffusion models" or 3DiM. The model described herein may be or may include an image-to-image diffusion model that inputs one or more (e.g., single) reference views and one or more (e.g., single) relative gestures and generates a target view. Thus, the machine-learned diffusion model described herein can perform new view synthesis from as few as a single image.

Description

[0001] Priority declaration

[0002] This application is based on and claims priority to U.S. Provisional Application No. 63 / 403,650, filed on September 2, 2022, which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to machine learning and more particularly to systems and methods for performing new view synthesis for a three-dimensional scene using a machine-learned diffusion model. Background Art

[0004] Diffusion Probabilistic Models (DPMs), also referred to as "diffusion models" for short, have recently emerged as a powerful class of generative models that achieve state-of-the-art performance on audio and image synthesis, with improved training stability compared to adversarial methods and the ability to perform likelihood computation, enabling further applications such as compression and density estimation. Diffusion models have achieved impressive empirical results on a variety of image-to-image translation tasks, not limited to text-to-image, super-resolution, inpainting, colorization, uncropping, and artifact removal. Summary of the invention

[0005] Various aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0006] An example aspect of the present disclosure relates to a computer-implemented method for performing new view synthesis of a three-dimensional scene using a machine-learned diffusion model. The method can be performed for each of one or more iterations. The method includes obtaining an input by a computing system including one or more computing devices, the input including data describing an input posture. The method includes processing the input by the computing system using a machine-learned diffusion model to generate a synthetic image of the three-dimensional scene from the input posture. The machine-learned diffusion model includes multiple denoising steps, and the multiple denoising steps are configured to receive multiple conditional images respectively. Processing the input by the computing system using the machine-learned diffusion model to generate a synthetic image includes: for each of at least two of the multiple denoising steps, the computing system accesses an image collection containing multiple images, the multiple images depicting the three-dimensional scene from multiple postures; and the computing system samples the sampled image from the image collection as a conditional image for such denoising step.

[0007] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0008] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0010] Figure 1A Depicted is a graphical diagram of an example process for performing pose-conditioned image-to-image training in accordance with an example embodiment of the present disclosure.

[0011] Figure 1B and Figure 1C Depicted is a graphical diagram of an example process of performing new view synthesis in multiple iterations according to an example embodiment of the present disclosure.

[0012] Figure 2 Depicted is a graphical diagram of an example diffusion model architecture for performing new view synthesis according to an example embodiment of the present disclosure.

[0013] Figure 3A Depicted is a block diagram of an example computing system that performs new view synthesis according to an example embodiment of the present disclosure.

[0014] Figure 3B A block diagram of an example computing device is depicted according to an example embodiment of the present disclosure.

[0015] Figure 3C A block diagram of an example computing device is depicted according to an example embodiment of the present disclosure.

[0016] Repeated reference numerals among the various drawings are intended to identify like features in the various implementations. DETAILED DESCRIPTION

[0017] Overview

[0018] In general, the present disclosure relates to systems and methods for performing new view synthesis for three-dimensional (3D) scenes using machine-learned diffusion models. Example implementations of the proposed models may be referred to as "3D diffusion models" or 3DiMs. The models described herein may be or may include image-to-image diffusion models that take as input one or more (e.g., a single) reference views and one or more (e.g., a single) relative poses and generate a target view. Thus, the machine-learned diffusion models described herein may perform new view synthesis from as little as a single image.

[0019] More specifically, one particular image-to-image translation problem to which the proposed diffusion model can be applied is that of novel view synthesis. In the novel view synthesis task, given a collection of images of a known 3D scene, the task is to infer the scene depicted in the set of images from a new viewpoint.

[0020] According to one aspect of the present disclosure, the example systems and methods described herein may include and / or utilize an image-to-image diffusion model to perform a new view synthesis task. The image-to-image diffusion model may have been trained on pairs of images of the same scene, where the relative pose between the two images is assumed to be known. Specifically, the image-to-image diffusion model may be trained to denoise a second image conditioned on a first (noise-free) image (and optionally multiple conditional images) and the relative pose between the two.

[0021] According to another aspect of the present disclosure, the proposed technology can overcome 3D inconsistencies in images synthesized by the model at inference time by sampling frames in a manner similar to an autoregressive model. Specifically, during the "backward diffusion" process of each individual frame (e.g., which can also be referred to as "denoising" or inference), the conditional frame of each denoising step can be sampled (e.g., randomly or "stochastically") from an image collection, where the image collection contains a set of previously generated synthetic frames and / or initial reference frames. Therefore, multiple different images sampled from an initially given or previously generated set of frames can guide the generation of the next synthetic image. For example, the diffusion model can be conditioned on a random frame in each denoising step. This allows efficient mixing of previously generated views or reference views of the scene, thereby improving the 3D consistency between the synthetic image and the images in the image collection.

[0022] Example experiments show that the proposed random conditional sampler can produce more 3D consistent results compared to a simple sampling process conditioned on only a single previous frame. For example, some experiments compare 3DiM with previous work on the SRN ShapeNet dataset, showing that 3DiM achieves higher fidelity in scene videos generated from a single view while still maintaining 3D consistency.

[0023] Another aspect of the present disclosure relates to a new evaluation method that can be referred to as a 3D consistency score. In the 3D consistency score, a neural radiance field (NeRF) model can be trained on the output of a diffusion model. The performance of the trained NeRF model can then be evaluated to measure the 3D consistency of the diffusion model (e.g., as a proxy for the diffusion model). Thus, the 3D consistency score can capture 3D consistency numerically by training a neural field on the model output.

[0024] Another example aspect provides improvements for 3D new view synthesis (e.g., applicable to UNet architectures). For example, according to one aspect, at least one of the multiple denoising steps of a machine-learned diffusion model may include at least one block that uses shared weights to process both a current intermediate image of such denoising step and a conditional image of such denoising step. Additionally or alternatively, at least one of the multiple denoising steps of a machine-learned diffusion model may include one or more frame cross-attention blocks, wherein information mixing between the current intermediate image of such denoising step and the conditional image of such denoising step is limited to the one or more frame cross-attention blocks (e.g., does not occur in the denoising step except within the frame cross-attention blocks). These improvements can help provide high-quality results in new view synthesis.

[0025] The present disclosure provides a variety of technical effects and benefits. As an example, the proposed 3D diffusion model is geometry-free. Therefore, the proposed technology can provide improved performance in "few-shot" or "single-shot" settings. Therefore, the performance of the computing system on the new view synthesis task can be improved in these settings.

[0026] Furthermore, the proposed model does not rely on a hypernetwork or test-time optimization to achieve novel view synthesis. Thus, the proposed technique is a simpler end-to-end approach that may require less processor usage and / or memory usage to execute compared to current state-of-the-art methods.

[0027] As another example, the proposed technique allows a single model to easily scale to a large number of scenes. Thus, a single trained model can be used to perform view synthesis for multiple different scenes, for example, compared to a NeRF model. This "reuse" of a trained model (e.g., compared to training multiple different models) can reduce the training that needs to be performed. Performing less training reduces the consumption of computational resources such as memory usage, processor usage, and network bandwidth.

[0028] Example embodiments of the present disclosure will be discussed in further detail.

[0029] Example pose conditional diffusion model

[0030] Consider the problem of synthesizing new views given a small number of images from a probabilistic perspective. Given a 3D scene A complete description of , for any pose p, the view x at pose p (p) Depend on Completely determined, that is, given However, the example implementation does not have The form q(x 1 , ..., xm |}x m+1 , ..., x n ), where the views are no longer conditionally independent.

[0031] A concrete example is as follows: given the back of a person's head, there are multiple plausible views of the front of their head. An image-to-image model that samples only frontal views given the back should indeed produce different outputs for each frontal view, but these outputs are not guaranteed to be consistent with each other, especially if the model learns the data distribution well.

[0032] Similarly, given a single view of an apparently small object, the pose itself is ambiguous: is it small and close, or just far away? Therefore, given the ambiguity inherent in the few-shot setting, a sampling scheme is needed where the generated views can rely on each other to achieve 3D consistency.

[0033] This is in contrast to the NeRF approach, where the query ray is These methods attempt to learn a single scene The richest possible representation of , and 3DiM completely avoids learning The difficulty of generating models.

[0034] Example image-to-image diffusion model using pose conditioning

[0035] Given a pose p from a common scene 1 , p 2 The data distribution q(x) of the view pairs at ∈SE(3) 1 , x 2 ), the example implementation defines an isotropic Gaussian process that adds more and more noise to the data samples as the signal-to-noise ratio λ decreases:

[0036]

[0037] where σ(·) is the sigmoid function. An example implementation can apply the reparameterization trick and sample from these marginal distributions via

[0038]

[0039] Then, given a pair of views, the example implementation learns to reverse this process in one of the two frames by minimizing the following objective, which produces better sample quality than maximizing the true evidence lower bound (ELBO):

[0040]

[0041] where ε θ is a neural network whose task is to, given different (clean) frames x 1 In the case of frame is denoised, and λ is the logarithmic signal-to-noise ratio. To make the notation clearer for the examples presented, some descriptions in this paper slightly abuse notation and will from now on simply be written as

[0042] Example 3D consistency achieved via random conditions

[0043] Figure 1A Depicts an example of pose-conditioned image-to-image training, including example training inputs and outputs for a pose-conditioned image-to-image diffusion model. Given two frames from a common scene and their poses (R, t), the task is to remove the noise added to one of the two frames. (*) In practice, one trains a neural network proposed in the example to predict the Gaussian noise ε used to corrupt the original view --- the predicted view is still just a linear combination of the noisy input and the predicted ε.

[0044] Figure 1B and Figure 1C Depicted is an example random conditional sampler - the example sampling process shown has two main components: 1) autoregressive generation of multiple frames, and 2) a denoising process used to generate each frame. When generating a new frame, the example implementation randomly selects the previous frame as the conditional frame in each denoising step. Some example implementations omit the pose inputs in the figure to avoid graphics overload, but it should be understood that these pose inputs are recalculated in each step depending on the conditional view being sampled.

[0045] Ideally, the example implementation would use the chain rule decomposition to model the 3D scene frame proposed by the example:

[0046]

[0047] This decomposition is ideal because it accurately models the distribution without making any conditional independence assumptions. Each frame is generated by autoregression conditional on all previous frames. However, it was found that this solution performs poorly.

[0048] Due to memory limitations, some example implementations can only be conditioned on a limited number of frames in practice (eg, k-Markov models). It was also found that as example implementations increase the maximum number of input frames k, the sample quality becomes worse.

[0049] Therefore, to achieve the best sample quality, some example implementations use a minimum value of k=2 (i.e., image-to-image model). Using k=2, example implementations can still achieve approximate 3D consistency. Some example implementations do not use a frame-based Markov sampler, but instead exploit the iterative nature of diffusion sampling by changing the conditional frame in each denoising step.

[0050] Random Conditioning. This section now details a novel random conditional sampling procedure proposed by the examples, which allows the example implementations to generate 3D consistent samples from 3DiM. Some example implementations select conditional views of a set of static scenes. Start with k = 1 or very small. The example implementation then generates a new frame by running a modified version of the standard denoising diffusion inverse process for a step of λ min =λ T <λ T-1 <...<λ 0 =λ max :

[0051]

[0052]

[0053] where i~Uniform({1,...,k}) is resampled in each denoising step. In other words, each individual denoising step is based on The input view is conditioned on different random views of the set containing the input view and previously generated samples.

[0054] Once the example implementation has finished running the sampling chain and generated the final x k+1 , the example implementation can simply combine it with Add and repeat this process (if the example implementation wants to sample more frames). Given enough denoising steps, the random condition allows each generated frame to be guided by all previous frames. See Figure 1B and Figure 1C In practice, some example implementations use 256 denoising steps, which has been found to be sufficient to achieve high sample quality and approximate 3D consistency. As is often the case in the literature, in some implementations the first (noisiest) sample is just a Gaussian sample, i.e., And in the last step λ 0 In the example implementation, noise-free sampling is performed.

[0055] The random condition can also be interpreted as a simple approximation to true autoregressive sampling, which works well in practice. True autoregressive sampling requires the form However, this would strictly require multi-view training data, while the ultimate goal of the example implementation is to achieve novel view synthesis with only two training views per scene.

[0056] Example X-UNet

[0057] Figure 2 Depicted example X-UNet architectures - example implementations modify a typical UNet architecture to accommodate 3D new view synthesis. Some example implementations share the same UNet weights between two input frames, a clean conditional view and a denoised target view. Some example implementations add a cross-attention layer to mix information between input and output views.

[0058] The 3DiM model can benefit from a neural network architecture that takes both the conditional and noisy frames as input. A natural approach is to concatenate the two images along the channel dimension and use a standard UNet architecture. This “Concat-UNet” has achieved significant success in previous work on image-to-image diffusion models.

[0059] However, in some early experiments, the example implementations found that Concat-UNet produced very poor results - there were severe 3D inconsistencies and lack of alignment with the conditional image. It is assumed that it is difficult to learn complex nonlinear image transformations that rely solely on self-attention given limited model capacity and training data. Therefore, some example implementations introduced the proposed X-UNet, whose core changes are: (1) sharing parameters to process each of the two views, and (2) using cross attention between the two views. Some example implementations show that the proposed X-UNet architecture is very effective for 3D new view synthesis.

[0060] This section now describes X-UNet in detail. Some example implementations use UNet with residual blocks and self-attention. Some example implementations share weights across two input frames for all convolutional and self-attention layers, but there are several key differences:

[0061] 1. Some example implementations let each frame have its own noise level (recall that the input to the DDPM residual block is the feature map and the positional encoding of the noise level). Some example implementations use λ for the clean frame max In contrast, some previous methods denoise multiple frames simultaneously, with the same noise level for each frame.

[0062] 2. Some example implementations modulate each UNet block via FiLM, but some example implementations use the sum of pose and noise level position encodings instead of separate noise level embeddings. In one example, additionally, the pose encodings proposed by the examples differ in that they have the same dimension as the frame - they are camera rays.

[0063] 3. Some example implementations do not focus over "time" after each self-attention layer (in the case presented in the example, this only requires two attention weights), but instead define a cross-attention layer and have the feature map of each frame call this layer to query the feature map of another frame.

[0064] Example illustration of novel view synthesis using diffusion model

[0065] Figure 1B and Figure 1C Depicted is a graphical diagram of an example process of performing new view synthesis in multiple iterations according to an example embodiment of the present disclosure.

[0066] Figure 1B and Figure 1C The process shown involves using a machine learning diffusion model 12 in multiple iterations. Specifically, Figure 1B Two iterations are shown, an initial iteration at t = 1 and a subsequent iteration at t = 2. Although only two iterations are shown, any number of iterations may be performed.

[0067] A new view synthesis task may begin by receiving one or more reference images, such as reference image 20. Reference image 20 may depict a 3D scene from a reference view. The 3D scene may be a real-world scene, or a synthetic or virtual scene. Reference image 20 may be a real-world (e.g., "natural environment") image, or a synthetic image (e.g., synthesized by model 12 or some generative model or process). Reference image 20 (and any other provided reference images) may be added to image set 18.

[0068] Next, in order to generate a new synthetic image 16 of the 3D scene, an input may be provided to the diffusion model 12. The input may include information about an input pose 14 from which the new synthetic image 16 should depict the 3D scene. The input pose 14 may be represented as raw pose information (e.g., a set of nine degrees of freedom values). In another example, the input pose 14 may be represented as a relative change in pose relative to one or more reference poses (e.g., a pose of the reference image 20). In another example, the input pose 14 can be encoded in the same dimension as the frame and represent a relative camera ray.

[0069] The input to the diffusion model 12 may also include an initial image 15. In some examples, the initial image 15 may include only random noise.

[0070] The diffusion model 12 may process the input to generate a composite image 16. The composite image 16 may depict a 3D scene from the input pose 14. Specifically, the machine-learned diffusion model 12 may include multiple denoising steps. Figure 1B Three denoising steps 22a, 22b and 22n are depicted. However, any number of steps may be used. For example, the model 12 may include hundreds of denoising steps.

[0071] Each denoising step may be configured to receive a current intermediate image and a conditional image. Except for the initial denoising step 22a which receives the initial image 15, the current intermediate image of each denoising step 22b-22n may be an image output by a previous sequential denoising step. Each denoising step 22a-22n may denoise the current intermediate image conditioned on the conditional image to produce a next intermediate image for the next sequential denoising step, except that the final denoising step 22n may output a final composite image 16.

[0072] For example, Figure 1B As shown, the denoising step 22b can be configured to receive the intermediate image 26 and the conditional image 24b output by the previous denoising step 22a. Because in the example shown, there is only a single image 20 in the image set 18 at t=1, the reference image 20 is used as the conditional image for all denoising steps 22a-n. The final denoising step 22n can output the composite image 16.

[0073] According to one aspect of the present disclosure, the synthesized image 16 may be added to the image set 18 so as to be sampled as a conditional image in subsequent iterations. Figure 1C As shown, at t=2, the image set may now include the reference image 20 and the composite image 16 generated at t=1.

[0074] like Figure 1C As shown, the diffusion model 12 may again be used to generate another composite image 66 of the 3D scene (eg, based on inputs including another input gesture 54 and another initial image 55 ).

[0075] However, according to one aspect of the present disclosure, for each of the plurality of denoising steps 22a-n, an image may be sampled (e.g., randomly) from the image set 18 to serve as a conditional image for the denoising step. This allows efficient blending of previously generated views or reference views of the scene, thereby improving the 3D consistency between the synthesized image and the images in the image set 18.

[0076] Specifically, if Figure 1C As shown: for the denoising step 22a, the composite image 16 can be sampled as the conditional image 64a; for the denoising step 22b, the reference image 20 can be sampled as the conditional image 64b; and for the denoising step 22n, the composite image 16 can be sampled as the conditional image 64n.

[0077] In each denoising step, the conditional image can be sampled using various sampling techniques. As an example, pure random sampling can be performed. As another example, weighted random sampling (e.g., random sampling with weighted probability) can be performed. The weight of each image in the set can be based on the distance between the input pose 54 and the corresponding pose of the corresponding image in the image set 18. For example, images in the set 18 with poses that are more similar (closer) to the input pose 54 can be given greater weights, making them more likely to serve as conditional images. This can improve local 3D consistency. As another example, the weight of a reference image (e.g., image 20) may be greater than that of a synthesized image (e.g., image 16) to increase the likelihood of using a reference real image to adjust the new synthesized image (but while still achieving 3D consistency via blending). In another example, a defined blending plan can be used to sample the conditional image.

[0078] At the end of iteration t=2, the synthesized image 66 may be added to the image set 18. Additional iterations may then be performed. In this manner, any number of synthesized images representing new views of a scene may be generated while exhibiting 3D consistency via the blending of conditional images.

[0079] Figure 2 A graphical diagram depicts an example diffusion model architecture for performing new view synthesis according to an example embodiment of the present disclosure. For example, Figure 2 The architecture shown can be used to implement one or more of the denoising steps. That is, Figure 2 The architecture shown represents a single denoising step and can be repeated for any number of denoising steps.

[0080] Specifically, Figure 2 The architecture shown is similar to the typical UNet architecture, but contains modifications to that UNet architecture that can use BigGAN-style residual blocks followed by self-attention at feasible resolutions.

[0081] Specifically, according to one aspect, the architecture can share most weights between two input frames, e.g., the current intermediate image 202 and the conditional image 204. Thus, some or all blocks can use shared weights to process both the current intermediate image 202 of such a denoising step and the conditional image 204 of such a denoising step. For example, weights can be shared across two input frames for all convolutional layers and self-attention layers.

[0082] The architecture may also include a frame cross attention block. According to another aspect, information mixing between the current intermediate image 202 for such a denoising step and the conditional image 204 for such a denoising step (e.g., between their forward processing flows) may be limited to one or more frame cross attention blocks.

[0083] Example Apparatus and Systems

[0084] Figure 3A A block diagram of an example computing system 100 that performs new view synthesis according to an example embodiment of the present disclosure is depicted. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled via a network 180.

[0085] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smart phone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0086] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 executed by the processor 112 to cause the user computing device 102 to perform operations.

[0087] In some implementations, the user computing device 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be or may otherwise include various machine learning models such as the diffusion model described above. Figure 2 Discuss example machine learning model 120.

[0088] In some implementations, one or more machine learning models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single machine learning model 120 (e.g., to perform parallel new view synthesis across multiple instances of a scene, reference image, etc.).

[0089] Additionally or alternatively, one or more machine learning models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine learning model 140 may be implemented by the server computing system 140 as part of a web service (e.g., a new view synthesis service). Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0090] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other devices by which a user can provide user input.

[0091] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0092] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0093] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, the model 140 may be or may otherwise include a variety of machine learning models. Example machine learning models include diffusion models as described herein. Figure 2 Discuss example model 140 .

[0094] User computing device 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with training computing system 150 communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0095] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc. and combinations thereof. The memory 154 can store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes one or more server computing devices or is otherwise implemented by one or more server computing devices.

[0096] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques such as, for example, error back propagation. For example, a loss function may be back propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters in multiple training iterations.

[0097] In some implementations, performing error back-propagation may include performing truncated back-propagation through time.The model trainer 160 may perform a variety of generalization techniques (eg, weight decay, dropout, etc.) to improve the generalization capabilities of the model being trained.

[0098] Specifically, model trainer 160 may train machine learning models 120 and / or 140 based on training data set 162. In some implementations, if the user has provided consent, the training examples may be provided by user computing device 102. Thus, in such implementations, model 120 provided to user computing device 102 may be trained by training computing system 150 based on user-specific data received from user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0099] The model trainer 160 includes computer logic for providing the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some implementations, the model trainer 160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more computer executable instruction sets stored in a tangible computer readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0100] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 180 may be conducted via any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0101] Figure 3A An example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training data set 162. In such implementations, the model 120 may be both trained and used locally at the user computing device 102. In some such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0102] Figure 3B Depicted is a block diagram of an example computing device 10 performing in accordance with an example embodiment of the present disclosure. Computing device 10 may be a user computing device or a server computing device.

[0103] Computing device 10 includes multiple applications (e.g., application 1 to application N). Each application contains its own machine learning library and machine learning model. For example, each application can include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0104] like Figure 3B As shown, each application can communicate with multiple other components of the computing device (such as, for example, one or more sensors, a context manager, a device state component, and / or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to the application.

[0105] Figure 3C Depicted is a block diagram of an example computing device 50 performing in accordance with an example embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0106] The computing device 50 includes a plurality of applications (e.g., Application 1 to Application N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0107] The central intelligence layer includes multiple machine learning models. For example, Figure 3C As shown, a corresponding machine learning model may be provided for each application, and the corresponding machine learning model may be managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model for all applications. In some implementations, the central intelligence layer is included in the operating system of the computing device 50 or is otherwise implemented by the operating system.

[0108] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for the computing device 50. Figure 3C As shown, the central device data layer can communicate with multiple other components of the computing device (e.g., such as one or more sensors, context managers, device state components, and / or additional components). In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0109] Additional disclosure

[0110] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for multiple possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system, or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0111] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation rather than limitation of the present disclosure. Those skilled in the art may easily produce changes, modifications, and equivalents to such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, variations, and / or additions to the subject matter that would be readily apparent to those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to encompass such changes, variations, and equivalents.

Claims

1. A computer-implemented method for performing new view synthesis of a three-dimensional scene using a machine-learned diffusion model, the method comprising: For each of one or more iterations: obtaining, by a computing system including one or more computing devices, input including data describing an input gesture; processing the input by the computing system using the machine-learned diffusion model to generate a composite image of the three-dimensional scene from the input gesture; wherein the machine-learned diffusion model comprises a plurality of denoising steps, wherein the plurality of denoising steps are configured to receive a plurality of conditional images respectively; and Wherein processing the input by the computing system using the machine-learned diffusion model to generate the composite image comprises: for each of at least two of the plurality of denoising steps: accessing, by the computing system, a collection of images comprising a plurality of images depicting the three-dimensional scene from a plurality of poses; as well as A sampled image is sampled by the computing system from the image set as the conditional image for such a denoising step.

2. The computer-implemented method of any preceding claim, further comprising: The composite image is added by the computing system to the image set for sampling as a conditional image in a subsequent iteration.

3. A computer-implemented method as claimed in any preceding claim, wherein for at least one of the one or more iterations, the set of images comprises at least one previously generated composite image previously generated in a previous iteration.

4. A computer-implemented method as claimed in any preceding claim, wherein the image collection contains only a single reference real image of the three-dimensional scene.

5. The computer-implemented method of any preceding claim, wherein sampling, by the computing system, the sampled image from the image collection comprises randomly sampling, by the computing system, the sampled image from the image collection.

6. A computer-implemented method as described in any preceding claim, wherein at least one of the multiple denoising steps of the machine-learned diffusion model includes at least one block that uses shared weights to process both a current intermediate image of such denoising step and the conditional image of such denoising step.

7. A computer-implemented method as described in any preceding claim, wherein at least one of the multiple denoising steps of the machine-learned diffusion model includes one or more frame cross-attention blocks, and wherein information mixing between a current intermediate image of such denoising step and the conditional image of such denoising step is limited to the one or more frame cross-attention blocks.

8. The computer-implemented method of any preceding claim, further comprising: evaluating, by the computing system, a loss function that compares the synthesized image of the three-dimensional scene from the input pose to a ground truth image of the three-dimensional scene from the input pose; as well as One or more values ​​or one or more parameters of the machine-learned diffusion model are modified based on the loss function.

9. The computer-implemented method of any preceding claim, further comprising: evaluating, by the computing system, a three-dimensional consistency of the machine-learned diffusion model; Wherein evaluating the three-dimensional consistency of the machine-learned diffusion model comprises: training a neural radiance field model on the image set and the composite image; and evaluating the performance of the trained neural radiation field model on one or more performance metrics; Wherein the performance of the trained neural radiation field model is indicative of the three-dimensional consistency of the machine-learned diffusion model.

10. A computing system configured to perform the method of any one of claims 1 to 9.

11. One or more non-transitory computer-readable media collectively storing instructions that, when executed by a computing system, cause the computing system to perform the method of any one of claims 1 to 9.

Images