Diffusion-guided three-dimensional reconstruction
Through the multi-step process of cumulative score sampling loss and diffusion neural network based on the decoder, the problems of insufficient quality and high resource consumption in image editing in the prior art are solved, and higher quality image editing and 3D reconstruction are achieved, and the resources are Low consumption.
Patent Information
- Application Number
- CN202410754813.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-12
- Filing Date
- 2024-06-12
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art is difficult to generate high-quality enhanced images in image editing, especially when processing noisy, occluded or truncated images, and the computing and storage resources are consumed more, resulting in insufficient stability and refinement of 3D reconstruction.
Using a decoder-based cumulative fraction sampling (DASS) loss and diffusion neural network, the image spatial gradient is calculated through a multi-step diffusion process, iteratively updates the potential representation, generates more stable and less noise images, and uses 2D diffusion prior to guide the image Optimize to achieve more realistic 3D reconstruction.
Improves the quality of 2D rendered images and the robustness of 3D reconstruction, can accurately estimate the camera viewpoint, posture, shape and texture of objects, consumes less computing and storage resources, and can accurately edit even when the original image is occluded or truncated. .
Smart Images

Figure CN118918241A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 472,560, filed on June 12, 2024. The disclosure of the prior application is considered part of and incorporated by reference into the disclosure of the present application. Technical Field
[0003] This specification relates to processing images using neural networks. Background Art
[0004] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict the output of a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input according to the current values of a corresponding set of parameters. Summary of the invention
[0005] This specification describes a system implemented as a computer program on one or more computers that performs image editing based on a decoder-based accumulated fractional sampling (DASS) loss.
[0006] In some implementations, the system uses the DASS loss and a diffusion neural network to directly optimize the input image to apply an edit to the input image, such as to enhance the input image or to denoise the input image. In other words, the system generates an output image by editing the input image using the DASS loss (i.e., unconditionally or conditional on the text description).
[0007] In some other implementations, the system uses a DASS loss and a diffusion neural network to adjust a differentiable renderer that generates an input image, i.e., the differentiable renderer is adjusted by backpropagating image spatial gradients through (parameters of) the differentiable renderer. Once adjusted, the system or another system can use the differentiable renderer to edit the input image, i.e., generate an output image that is an edited version of the input image.
[0008] Particular embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. This specification describes a novel diffusion-based scoring technique to (i) optimize an image, e.g., a two-dimensional (2D) rendered image, or (ii) optimize a parametric model (e.g., a differentiable renderer including one or more NeRF neural networks) to generate an enhanced rendered image.
[0009] A system using the diffusion-based scoring technique described in the present specification can utilize a 2D diffusion prior as a guide to generate an enhanced 2D rendered image of an object instance, the enhanced 2D rendered image having improved quality in both reconstruction of 3D joint shapes and texture of the object instance. Specifically, the system computes image space gradients from a multi-step diffusion process during which a diffusion neural network iteratively computes denoised outputs in a latent space to denoise the 2D rendered image, and incorporates the image space gradients into the optimization of the rendered image by using the denoised image as a target image.
[0010] Unlike some existing systems that backpropagate latent space gradients through an image encoder neural network to update a rendered image, the described system employs a decoder-based multi-step approach to compute more stable and less noisy gradient updates for rendered image optimization, and can therefore generate enhanced 2D rendered images with more robust, more detailed, and more realistic (from both input and novel viewpoints) 3D reconstructions of various object instances, even though the optimization process consumes less computational and memory resources. Advantageously, the system can accurately estimate the camera viewpoint, pose, shape, and texture of an object instance even in the presence of occlusions or truncations in the original image depicting the object instance.
[0011] The details of one or more embodiments of the subject matter of this specification are set forth in the following drawings and the description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 is a diagram of an example neural network system.
[0013] Figure 2 is an example illustration of the operations performed by a neural network system to edit an image using the DASS loss.
[0014] Figure 3 is a diagram of another example neural network system.
[0015] Figure 4 is an example diagram of the operations performed by a neural network system to tune a differentiable renderer using a DASS loss.
[0016] Figure 5 is a flow chart of an example process for editing an input image.
[0017] Figure 6 yes Figure 5 A flowchart of sub-steps of one of the steps of a process.
[0018] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION
[0019] Figure 1 is a diagram of an example neural network system 100. Neural network system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.
[0020] The neural network system 100 is a system that generates an output image by editing an input image using a decoder-based cumulative fractional sampling (DASS) loss (ie, either unconditionally or conditioned on a text description).
[0021] like Figure 1 As shown, in some implementations, the neural network system 100 uses a decoder-based cumulative fractional sampling (DASS) loss and a diffusion neural network 120 to directly optimize the input image 102 by applying edits to the input image 102.
[0022] In other words, the neural network system 100 generates the output image 112 by editing the input image 102 across multiple update iterations using a decoder-based accumulated fractional sampling (DASS) loss calculated using the diffusion neural network 120.
[0023] Figure 1 Four example input images 102 are shown. The input images 102 are noisy images each depicting an animal. The animals depicted in the input images 102 have a blurred appearance caused by noisy textures, dim lighting, occlusion, or truncation present in the input images 102.
[0024] Figure 1 Also shown are four example output images 112. Each output image 112 may be generated by neural network system 100 by processing a corresponding input image 102. Each output image 112 is an enhanced (e.g., optimized) version of the corresponding input image 102 that includes a higher quality depiction of the animal depicted in the corresponding input image 102.
[0025] That is, the output image 112 and the input image 102 depict the same object instance (the same animal), but the quality of the output image 112 is higher than that of the input image 102 because the output image 112 depicts the object instance (animal) with a clearer appearance and includes less noise, less occlusion, less truncation, etc.
[0026] The diffusion neural network 120 may be any suitable diffusion neural network that can perform a diffusion process in a latent space (e.g., in a latent space of lower dimensionality than the image space (or pixel space)). That is, the image on which the diffusion neural network 120 operates is the latent image, and the values of the pixels of the image are the learned latent values rather than the color values.
[0027] Specifically, the diffusion neural network 120 may have been trained to receive a denoised input including an intermediate latent representation of the input image 102, and process the input to generate a denoised output of the intermediate latent representation. In some implementations, the denoised input also includes a time step that defines a noise level of the intermediate latent representation. In some implementations where the diffusion neural network 120 is configured as a diffusion neural network conditioned on text, the denoised input also includes data derived (generated) from a text prompt.
[0028] The diffusion neural network 120 is associated with the image encoder neural network 110 to encode the image into a latent space, and is associated with the image decoder neural network 130, which receives an input including a latent representation of the image and decodes the latent representation to reconstruct the image. For example, the image encoder neural network 110 and the image decoder neural network 130 may have been jointly trained for an image reconstruction objective (e.g., a VAE objective, a VQ-GAN objective, or a VQ-VAE objective).
[0029] The diffusion neural network 120 may generally have any suitable neural network architecture.
[0030] For example, the diffusion neural network 120 may have a convolutional neural network architecture, such as a U-Net architecture having a plurality of convolutional blocks. As another example, the diffusion neural network 120 may have a Transformer neural network architecture having a set of self-attention blocks (and / or a set of cross-attention blocks), and the denoised input is processed by the set of self-attention blocks (and / or the set of cross-attention blocks) to generate a denoised output. As another example, the diffusion neural network 120 may have a diffusion transformer (DiT). As another example, the diffusion neural network 120 may have a universal visual transformer (UViT) architecture.
[0031] Figure 2 Is Figure 1 An example illustration 200 of the operations performed by the neural network system 100 to edit an image using the DASS loss. Figure 2 In the example of , the neural network system 100 uses the DASS loss calculated using the diffusion neural network 120 to apply editing to the input image 202.
[0032] The neural network system 100 encodes the input image 202 into a latent space using the image encoder neural network 110 , i.e., processes the input image 202 to generate an initial latent representation of the input image 202 .
[0033] The neural network system 100 then generates a target image 211 by updating the initial latent representation using the diffusion neural network 120 within n update iterations to generate an updated latent representation of the input image 202 .
[0034] At each update iteration, the diffusion neural network 120 processes the denoised input to generate a denoised output from which an updated intermediate latent representation can be inferred, the denoised input comprising (i) the intermediate latent representation, (ii) the time steps defining the noise level of the intermediate latent representation, and optionally, when the diffusion neural network is configured as a text-conditioned diffusion neural network, (iii) data derived from the text prompt.
[0035] The noise level is usually dependent on the time step. For example, the noise level can be determined by calculating a noise scheduling function for time step t (e.g., a linear scheduling function, a sigmoid scheduling function, a cosine scheduling function defined with reference to time step t).
[0036] The text prompt generally describes the output image that the neural network system 100 is to generate based on the input image 202. Figure 2 In the example of the embodiment, the text prompt is: "A photo of * (photo of *)", where * represents the type of animal, for example, zebra, tiger, giraffe, elephant, kangaroo or penguin. In other examples, different text prompts may also be used.
[0037] In some implementations, n may be any positive integer greater than or equal to two, for example, a positive integer between two and sixteen or between three and ten. If the update iteration is the first of n update iterations, the intermediate potential representation is the initial potential representation. For any subsequent update iterations, the intermediate potential representation is the updated intermediate potential representation that has been generated in the immediately preceding update iteration.
[0038] As part of processing the denoised input at a first update iteration, some implementations of the neural network system 100 may generate a noisy image from the input image 202, e.g., by sampling noise from a noise distribution (e.g., a Gaussian noise distribution) and then adding the sampled noise to at least a subset of pixels in the input image 202 (e.g., background pixels). Foreground pixels refer to pixels in the input image 202 that are part of an object instance, and background pixels refer to pixels in the input image 202 that are not part of an object instance. Foreground pixels and background pixels may be determined, for example, by processing the input image 202 using an instance / semantic segmentation neural network or another segmentation algorithm. Thus, the diffusion neural network processes (i) an initial latent representation generated from the noisy image, (ii) a time step, and optionally (iii) data derived from a textual prompt to generate a denoised output for the first update iteration.
[0039] For example, the denoised output generated by the diffusion neural network 120 may define an estimate of the noise that has been added to the latent representation of the target image in the latent space to obtain the intermediate latent representation in the latent space, and thus the neural network system 100 may generate an updated intermediate latent representation by removing the estimated noise from the intermediate latent representation. In this way, at each update, the updated intermediate latent representation will generally be a less noisy version of the intermediate latent representation.
[0040] At each update iteration, the neural network system 100 evaluates a latent space objective function that measures the difference between (i) an estimate of the noise defined by the denoised output of the intermediate latent representation and (ii) the known noise included in the intermediate latent representation.
[0041] For example, the latent space objective function can be of the following form:
[0042]
[0043] where z t is an intermediate latent representation at a given time step t, which includes known noise ∈ that depends on the given time step t, y represents the data derived (generated) from the text prompt, φ represents the diffusion neural network, ∈ φ represents the estimate of the noise defined by the denoised output generated by the diffusion neural network, ω t is a constant multiplier that depends on a given time step t and is optionally used to balance between textual guidance and classifier-free guidance of the unconditional diffusion neural network.
[0044] At each update iteration, the neural network system 100 calculates the corresponding gradient of the latent space objective function with respect to the latent value included in the intermediate latent representation, and then determines an update to the intermediate latent representation based on the calculated gradient of the latent space objective function.
[0045] For example, the gradient calculated by the neural network system 100 may be The intermediate latent representation is updated based on the calculated gradient of the latent space objective function. Figure 2 If it is marked as As shown by the dotted arrow.
[0046] After n update iterations have been performed, neural network system 100 processes the updated intermediate latent representation generated in the last update iteration of the n update iterations using image decoder neural network 130 to generate target image 211. That is, neural network system 100 generates target image 211 based on the updated final latent representation generated as a result of performing n update iterations on input image 202. As explained below, target image 211 is generated in order to calculate the gradient of the image space objective function for updating input image 202.
[0047] The neural network system 100 calculates the corresponding gradient of the image space objective function with respect to the color values of the input image 202. The image space objective function includes a decoder-based accumulated fractional sampling (DASS) loss term that measures the difference between (i) the input image 202 obtained before n update iterations and (ii) the target image 211 generated as a result of n update iterations.
[0048] For example, the image space objective function can be of the following form:
[0049]
[0050] Where x is the input image 202, represents the accumulation (e.g., sum) of the gradient of the latent space objective function with respect to the latent value over n update iterations, Z is the initial latent representation of the input image 202 generated by the image encoder neural network 110, and D represents the image decoder neural network 130.
[0051] Specifically, the neural network system 100 may calculate an approximation of the gradient of the image-space objective function instead of calculating the gradient of the image-space objective function by explicitly calculating the partial derivative of the image-space objective function with respect to the color value of the input image 202 .
[0052] For example, the approximation of the gradient of the image-space objective function including the DASS loss term can be calculated as follows:
[0053]
[0054] Where x′ is the target image 211,
[0055] The neural network system 100 determines one or more updates to the input image 202 based on the calculated gradient of the image-space objective function, and then applies the updates to the input image 202 to generate an output image.
[0056] For example, neural network system 100 can approximate the gradient by The approximation of the gradient is added to or subtracted from the input image 202 to generate the output image. The input image 202 is updated based on the calculated gradient of the image space objective function. Figure 2 As shown by the dashed arrow labeled “image gradient”.
[0057] The generated output image may be used in any of a variety of ways. For example, neural network system 100 may then provide the output image as an edited version of input image 202 for presentation on a computer, such as to a user who submitted input image 202. Additionally or alternatively, neural network system 100 may store the output image in a repository for future use.
[0058] Editing images using the DASS loss term is different from and improves upon existing methods of updating input images by backpropagating image space gradients through an image encoder neural network. As an example of existing methods, the fractional distillation sampling (SDS) method (described in Poole, Ben et al., "Dreamfusion: Text-to-3d using 2d diffusion", arXiv preprint arXiv:2209.14988 (2022)) involves computing the gradient of an image space objective function in the following form:
[0059]
[0060] in Denote the partial derivatives of the image-space objective function with respect to the color values of the input image, and then backpropagate the gradients through the image encoder neural network at each of n update iterations.
[0061] Although the SDS method is effective (due to the rich generative priors of the diffusion neural network from which it is sampled), the image space gradients back-propagated through the image encoder neural network in the SDS method are usually noisy, resulting in undesirable artifacts on 3D shape and texture. In addition, the SDS method requires additional computational and memory resource consumption for gradient back-propagation, which limits the training batch size and thus reduces stability.
[0062] Advantageously, as mentioned above Figure 2 As described above, since the neural network system 100 applies updates to the input image 202 based on (an approximation of) the gradient of the image space objective function including the DASS loss term, the neural network system 100 only needs to perform a single forward pass through the image encoder neural network 110 and the image decoder neural network 130. Therefore, the neural network system 100 consumes approximately half the memory resource consumption when editing the input image compared to a system using the SDS approach.
[0063] Figure 3 is a diagram of another example neural network system 300. Neural network system 300 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.
[0064] The neural network system 300 is a system that uses a decoder-based cumulative fractional sampling (DASS) loss to adjust a differentiable renderer that generates an input image. Once adjusted, the neural network system 300 or another system can use the differentiable renderer to edit the input image, i.e., generate an output image that is an edited version of the input image.
[0065] like Figure 3 As shown, in some implementations, the neural network system 300 includes a diffusion neural network 320, an adjustment engine 340, and a differentiable renderer 350. The adjustment engine 340 within the neural network system 300 uses a decoder-based cumulative fractional sampling (DASS) loss generated using the diffusion neural network 320 to adjust parameters of the differentiable renderer 350.
[0066] The differentiable renderer 350 may be or include a three-dimensional (3D) rendering engine configured to receive a set of initial images 301 each depicting an object instance, and apply a 3D rendering process to generate two-dimensional (2D) rendered images of object instances in a scene as input images 302. The object instances may be any of a variety of types of objects, including landmarks, landscape or location features, vehicles, tools, food, clothing, devices, animals, humans, to name a few examples. In the 3D rendering process, the differentiable renderer 350 may generate one or more 3D models 354 of the object instances.
[0067] 3D model 354 may include one or more features of the object instance represented by the model. In some examples, the 3D model may specify the identity of the object instance. The 3D model may also specify the physical features of the object instance, such as size, weight, mechanical elements, color, material, grip features, etc.
[0068] There are many ways in which the differentiable renderer 350 can generate one or more 3D models 354 based on a set of initial images 301. In some implementations, the 3D model 354 may include a 3D model that defines a 3D point cloud, voxel, or mesh representation of an object instance generated by the differentiable renderer 350 from a set of initial images 301 that each depict the object instance. The 3D model may directly define a 3D geometry using various 3D shapes such as cubes, spheres, and cones, or define faces of a 3D geometry using 2D shapes such as points, circles, rectangles, and triangles.
[0069] In these implementations, the differentiable renderer 350 may process a set of initial images 301 using one or more point cloud, voxel, or mesh prediction neural networks to generate output data defining a 3D model. Examples of point cloud, voxel, or mesh prediction neural networks that can be used to generate such 3D models include those described in the paper “Mesh R-CNN” by Gkioxari et al., published in the IEEE / CVF International Conference on Computer Vision Proceedings in 2019 and the paper “MarrNet: 3D shape reconstruction via 2.5D sketches” by Wu Jiajun et al., published in the arXiv preprint, arXiv:1711.03129 in 2017.
[0070] In some implementations, the 3D model 354 may include a 3D model that defines a neural radiance field (NeRF) representation (e.g., a neural surface representation) of one or more parts (or components) of the object instance generated by the differentiable renderer 350 from the set of initial images.
[0071] In these implementations, the differentiable renderer 350 may process a set of initial images 301 using one or more NeRF neural networks to generate output data defining a NeRF representation. For example, to define the NeRF representation, the output data may include one or more of the following: density data describing an estimate of the density of an object instance, 3D shape data describing the form of an outer surface of the object instance, texture data describing characteristics of the outer surface of the object instance.
[0072] Examples of NeRF neural networks that can be used to generate such 3D models include those described in “Hi-lassie: High-fidelity articulated shape and skeleton discovery from sparse image ensemble” by Chun-Han Yao et al., published in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, and “NeRS: Neural reflectance surfaces for sparse-view 3D reconstruction in the wild” by Jason Zhang et al., published in NeurIPS,34, 2021.
[0073] After having generated the one or more 3D models 354, the differentiable renderer 350 may then generate a 2D rendered image of the object instance as the input image 302 according to the 3D model 354 by applying one or more differentiable rendering operations. In other words, the differentiable renderer 350 may generate the 2D rendered image without applying any discretization rendering operations including a discrete sampling operation (referred to as rasterization), which prevents gradients from flowing into the 3D model 354.
[0074] For example, the differentiable renderer 350 may apply at least some of the differentiable operations described in “Soft rasterizer: A differentiable renderer for image-based 3D reasoning” by Shichen Liu et al., published in CVPR, pages 7708 to 7717, 2019 - e.g., transformations, color calculations, probability calculations, and aggregation operations, the contents of which are incorporated herein by reference.
[0075] As discussed further below, an adjustment engine 340 within the neural network system 300 then uses the decoder-based cumulative fractional sampling (DASS) loss generated using the diffusion neural network 320 to adjust the parameters of the differentiable renderer 350 that generates the input image 302 through self-supervised learning. Because the operation is differentiable, the adjustment engine 340 is also able to adjust the parameters of the one or more 3D models 354, as well as the neural network that generates the one or more 3D models 354. This makes the adjustment more efficient.
[0076] Once adjusted, the neural network system 300 or another system may use the differentiable renderer 350 to edit the input image 302 , i.e., generate an output image 312 that is an enhanced (e.g., optimized) version of the input image 302 .
[0077] For example, the differentiable renderer 350 may generate an enhanced 2D rendered image of the object instance as the output image 312. That is, the output image 312 and the input image 302 are 2D rendered images of the same object instance, but the quality of the output image 312 is higher than that of the input image 302 because the output image 312 depicts the object instance with a clearer appearance and includes less noise, less occlusion, less truncation, etc.
[0078] In some implementations, the differentiable renderer 350 may generate multiple 2D rendered images each having a different view of the same object instance. For example, multiple 2D rendered images may be generated relative to multiple different camera poses in a scene including the object instance, thereby obtaining multiple views of the object instance.
[0079] In some implementations, the differentiable renderer 350 may generate a 3D model of the object instance to have a specific (e.g., user-specified) pose, a specific (e.g., user-specified) motion, a specific (e.g., user-specified) texture, or a combination thereof. In these implementations, by using the 3D model, the differentiable renderer 350 may generate a 2D rendered image of the object instance having a specific pose, a specific motion, a specific texture, or a combination thereof.
[0080] Figure 4 Is Figure 3 An example illustration 400 of the operations performed by the neural network system 300 to adjust the differentiable renderer 350 using the DASS loss. Figure 4 In the example of , the neural network system 300 uses the DASS loss and the diffusion neural network 320 to adjust the differentiable renderer 350 that generates the input image, i.e., the differentiable renderer is adjusted by back-propagating the image space gradient through the differentiable renderer (e.g., the parameters of the differentiable renderer).
[0081] like Figure 4 As shown on the left side of , the differentiable renderer 350 can generate a neural radiance field (NeRF) representation of each of one or more parts of the object instance based on a set of initial images depicting the object instance by using a corresponding NeRF model. The NeRF models used to generate the NeRF representation can each be implemented as a corresponding neural network with multiple parameters, for example, a multi-layer perceptron (MLP) model, a convolutional neural network, etc.
[0082] Specifically, the NeRF model can be configured to receive an input comprising or derived from the set of initial images, and process the input to generate an output comprising one or more of: (i) a set of volume density values defining a density estimate of a portion of an object instance, (ii) a set of 3D shape values defining the form of an outer surface of a portion of an object instance (e.g., x, y, z values mapped from 3D surface points (u, v) on a unit sphere), or (iii) a set of albedo values defining a color estimate of a portion of an object instance (e.g., r, g, b values mapped from 3D surface points (u, v) on a unit sphere).
[0083] For each portion of the object instance, the differentiable renderer 350 may then generate a corresponding NeRF representation based on the set of volume density values, the set of 3D shape values, the set of albedo values, or some combination thereof.
[0084] The differentiable renderer 350 may generate a set of 2D rendered images of the object instance according to the NeRF representation by applying one or more differentiable rendering operations. For example, the operations may include the operations described in the reference “Soft rasterizer: A differentiable renderer for image-based 3D reasoning” mentioned above.
[0085] like Figure 4 As shown on the right side of , the neural network system 300 then uses the DASS loss and the diffusion neural network 320 to adjust the differentiable renderer 350 through self-supervised learning.
[0086] Adjusting the differentiable renderer 350 involves: for each 2D rendered image in the set of 2D rendered images that have been generated by using the differentiable renderer 350, generating an enhanced 2D rendered image corresponding to the 2D rendered image by using the diffusion neural network 320, and then adjusting parameters of the differentiable renderer 350 based on optimizing an image space objective function that includes at least a decoder-based accumulated fractional sampling (DASS) loss term.
[0087] The neural network system 300 can be implemented by executing the above reference Figure 2 Operations similar to those described are used to generate an enhanced 2D rendered image corresponding to the 2D rendered image (where the input image 202 is a 2D rendered image generated by using the differentiable renderer 350, and the target image 211 is the corresponding enhanced 2D rendered image).
[0088] More specifically, for each 2D rendered image in the group, the neural network system 300 first processes the 2D rendered image using the image encoder neural network to generate an initial latent representation of the 2D rendered image, updates the initial latent representation within n update iterations by using the diffusion neural network 320 to generate an updated latent representation of the 2D rendered image, and then processes the updated intermediate latent representation generated in the last update iteration of the n update iterations using the image decoder neural network to generate an enhanced 2D rendered image that is an enhanced version of the 2D rendered image.
[0089] The neural network system 300 then determines one or more updates to the parameters of the differentiable renderer 350 by back-propagating through the color values of the 2D rendered image that has been generated by the differentiable renderer 350 to calculate the corresponding gradients of the image space objective function with respect to the parameters of the differentiable renderer 350 .
[0090] Then, the neural network system 300 proceeds to update the parameters of the differentiable renderer 350 based on applying an update rule (e.g., an Adam update rule, an Rmsprop update rule, or a stochastic gradient descent (SGD) update rule) to the corresponding gradients. For example, the neural network system 300 may update the parameters of the NeRF model included in the differentiable renderer 350. As described above, each NeRF model may be implemented as a corresponding neural network, and thus updating the differentiable renderer involves updating the parameters of the corresponding neural network.
[0091] Specifically, for each 2D rendered image in the group, the image space objective function includes a decoder-based accumulated fractional sampling (DASS) loss term that measures the difference between (i) the 2D rendered image obtained before n update iterations and (ii) the enhanced 2D rendered image generated due to n update iterations.
[0092] Optionally, in some implementations, the image space objective function further includes one or more additional loss terms. For example, the image space objective function may include a texture reconstruction loss term that measures the difference between (i) an enhanced 2D rendered image of an object instance and (ii) a rendered RGB image of the object instance that has been generated based on shape and texture estimates of the object instance.
[0093] Furthermore, in this example, differences in the texture reconstruction loss term can be masked by foreground contour estimates of object instances that have been generated using a trained neural network (e.g., the Visual Transformer (ViT) neural network described in “Emerging properties in self-supervised vision transformers,” Mathilde Caron et al., 2021, ICCV, pp. 9650–9660).
[0094] For example, the texture reconstruction loss term can be calculated as follows:
[0095]
[0096] in represents the enhanced 2D rendered image of the jth part of the object instance generated as a result of n update iterations, and represents foreground contour estimation; is the rendered RGB image; and ⊙ represents the element-wise product.
[0097] As another example, the image space objective function may also include a contour loss term that measures the difference between the foreground contour estimate of the object instance and the (pseudo) reference true foreground contour of the object instance. As another example, the image space objective function may also include a semantic consistency loss term that forces the aggregated 3D point features to be closer to similar pixel feature projections in the 2D rendered image. As another example, the image space objective function may also include a partial rotation loss term that limits the angular offset of the object instance from its static pose. As another example, the image space objective function may also include a mesh regularization loss that promotes a smooth 3D surface by pulling each vertex toward the center of its neighbors. As another example, the image space objective function may also include a surface normal loss that promotes adjacent faces to have similar normals. More details of these loss terms that may be included in the image space objective function alone or in combination are described in more detail in the reference "Hi-lassie: High-fidelityarticulated shape and skeleton discovery from sparse image ensemble" mentioned above.
[0098] For example, in some implementations, the image space objective function can be defined as the weighted sum of all these loss terms:
[0099] in
[0100] where α is a vector with predetermined values that define the respective weights assigned to the various loss terms.
[0101] Figure 5 is a flow chart of an example process 500 for editing an input image. For convenience, process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system (e.g., Figure 1 The neural network system 100 or Figure 3 The neural network system 300 depicted in FIG. 1 may perform process 500 .
[0102] The system obtains an input image (step 502). For example, the input image may be obtained as a user upload. As another example, the input image may be a 2D rendered image of an object instance that has been generated by a differentiable renderer from a set of initial images depicting the object instance (e.g., a set of in-the-wild images, a set of noisy web images, etc.).
[0103] The system encodes the input image into a latent space using an image encoder neural network, i.e., processes the input image using the image encoder neural network to generate an initial latent representation of the input image (step 504).
[0104] The system updates the initial latent representation within n update iterations using a diffusion neural network to generate an updated latent representation of the input image (step 506), as described below with reference to Figure 6 Further discussion.
[0105] Figure 6 yes Figure 5 Flowchart of sub-steps 602 to 606 of step 506 of the process of FIG. Sub-steps 602 to 606 may be performed at each update iteration within n update iterations. By repeatedly performing iterations of sub-steps 602 to 606, the system may generate an updated potential representation of the input image.
[0106] The system processes a denoised input using a diffusion neural network to generate a denoised output of an intermediate latent representation (step 602), the denoised input including (i) the intermediate latent representation, (ii) time steps defining a noise level of the intermediate latent representation, and optionally (iii) data derived from a textual prompt. For example, the denoised output generated by the diffusion neural network may define an estimate of noise that has been added to a latent representation of a target image in a latent space to obtain an intermediate latent representation in the latent space.
[0107] If the update iteration is the first of n update iterations, the intermediate latent representation is the initial latent representation. For any subsequent update iterations, the intermediate latent representation is the updated intermediate latent representation that has been generated in the immediately preceding update iteration.
[0108] The system calculates the corresponding gradient of the latent space objective function with respect to the latent value included in the intermediate latent representation (step 604). The latent space objective function measures the difference between (i) the denoised output of the intermediate latent representation and (ii) the known noise included in the intermediate latent representation. For example, the known noise can be determined based on the time step and the time-dependent noise schedule.
[0109] The system determines an update to the intermediate latent representation based on the calculated gradient of the latent space objective function (step 606). For example, the system can update the intermediate latent representation by adding the gradient to the intermediate latent representation or subtracting the gradient from the intermediate latent representation to generate an updated intermediate latent representation.
[0110] The system processes the updated final latent representation generated as a result of the n update iterations using the image decoder neural network to generate the target image (step 508). The updated final latent representation is the updated intermediate latent representation generated in the last update iteration of the n update iterations.
[0111] The system calculates the corresponding gradient of the image space objective function with respect to the color values (pixel values) of the input image (step 510). The image space objective function includes a decoder-based cumulative fractional sampling (DASS) loss term that measures the difference between (i) the input image obtained before n update iterations and (ii) the target image generated due to the n update iterations. In some cases, the system may calculate an approximation of the corresponding gradient of the image space objective function instead of explicitly calculating the partial derivatives of the image space objective function with respect to the color values of the input image. The target image of the DASS loss term can be generated based on the accumulation of the gradient of the latent space objective function with respect to the latent value within n update iterations. For example, the accumulated gradient can be subtracted from (or, as appropriate, added to) the initial latent representation of the input image, and the result can be provided to the decoder to generate the target image.
[0112] In some cases where the input image is a 2D rendered image that has been generated by a differentiable renderer, the image space objective function may include one or more additional loss terms, such as a texture reconstruction loss term and other loss terms mentioned above. In general, including those additional loss terms may make the tuning of the differentiable renderer more efficient, for example in terms of computational resource consumption.
[0113] The system determines one or more updates to the input image based on the calculated gradient of the image-space objective function, and then applies the one or more updates to the input image to generate an output image, step 512. For example, the system can generate the output image by adding an approximation of the gradient to the color values of the input image or subtracting the approximation of the gradient from the color values of the input image.
[0114] In some cases where the input image is a 2D rendered image that has been generated by a differentiable renderer, the system may also update the differentiable renderer based on back-propagating the corresponding gradient of the image space objective function through the color values of the 2D rendered image to the parameters of the differentiable renderer. For example, the system may update parameters of one or more neural radiance field (NeRF) models included in the differentiable renderer, where each NeRF model may be implemented as a corresponding neural network, so updating the differentiable renderer involves updating the parameters of the corresponding neural network.
[0115] In some of these cases, after the differentiable renderer has been updated by performing process 500, the system may use the updated differentiable renderer to generate one or more three-dimensional (3D) models of the object instance based on the input image, and then generate multiple 2D rendered images based on the 3D models. For example, the 3D model may include a 3D point cloud, voxel, or mesh representation of the object instance. As another example, the 3D model may include a NeRF representation of one or more parts of the object instance. Such a 3D representation may have a user-specified pose, a user-specified motion, a user-specified texture, or a combination thereof.
[0116] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform specific operations or actions, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the system to perform those operations or actions in operation. For one or more computer programs to be configured to perform specific operations or actions, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.
[0117] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical, or electromagnetic signal), which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.
[0118] The term "data processing equipment" refers to data processing hardware and includes all types of equipment, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The equipment may also be or further include a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to the hardware, the equipment may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
[0119] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script or code, may be written in any form of programming language (including compiled or interpreted languages or declarative or procedural languages); and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a data communications network.
[0120] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and may be stored on a storage device in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed in a different way.
[0121] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines may be installed and run on the same one or more computers.
[0122] The processes and logic flows described in this specification may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by a dedicated logic circuit system (e.g., an FPGA or ASIC) or by a combination of a dedicated logic circuit system and one or more programmed computers.
[0123] A computer suitable for executing a computer program may be based on a general or special microprocessor or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory may be supplemented by or incorporated into a dedicated logic circuit system. Typically, the computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or be operatively coupled to receive data from the mass storage device or transfer data to the mass storage device or both. However, the computer does not need to have such a device. In addition, the computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.
[0124] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (such as EPROM, EEPROM and flash memory devices), magnetic disks (such as internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks.
[0125] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smart phone running a messaging application) and receiving a response message from the user in return.
[0126] A data processing device used to implement a machine learning model may also include, for example, a dedicated hardware accelerator unit for processing general-purpose and computationally intensive portions of machine learning training or production (i.e., inference, workloads).
[0127] Machine learning models can be implemented and deployed using a machine learning framework (e.g., TensorFlow or JAX framework).
[0128] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.
[0129] A computing system may include a client and a server. The client and server are usually remote from each other and typically interact through a communication network. The relationship of client and server is due to computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a user device, for example, for displaying data to a user interacting with the device acting as a client and receiving user input from it. Data generated at the user device, for example, due to user interaction, can be received from the device at the server.
[0130] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but should be interpreted as descriptions of features that may be unique to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may involve a sub-combination or a variant of a sub-combination.
[0131] Similarly, although operations are depicted in the drawings and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a continuous order, or that all of the operations shown be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
[0132] Certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes shown in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.
Claims
1. A method performed by one or more computers, the method comprising: Get the input image; processing the input image using an image encoder neural network to generate an initial latent representation of the input image; The initial latent representation is updated within n update iterations to generate an updated latent representation of the input image, the updating comprising at each update iteration: processing a denoised input including an intermediate latent representation derived from the initial latent representation using a diffusion neural network to generate a denoised output of the intermediate latent representation; computing a gradient of a latent space objective function that measures the difference between (i) the denoised output of the intermediate latent representation and (ii) known noise included in the intermediate latent representation; and determining an update to the intermediate latent representation based on the calculated gradient of the latent space objective function; processing the updated latent representation resulting from the n update iterations using an image decoder neural network to generate a target image; computing a gradient of an image-space objective function, the image-space objective function comprising a decoder-based accumulated fractional sampling (DASS) loss term that measures the difference between (i) the input image obtained before the n update iterations and (ii) the target image generated as a result of the n update iterations; as well as One or more updates to the input image are determined based on the calculated gradient of the image-space objective function.
2. The method of claim 1, wherein n is an integer value greater than or equal to two.
3. The method of claim 2, wherein n is an integer value between three and ten.
4. A method as claimed in any one of claims 1 to 3, wherein the denoised output comprises a noise estimate of the intermediate latent representation.
5. The method of any one of claims 1 to 4, wherein the diffusion neural network is a pre-trained text-to-image diffusion neural network operating on a latent image.
6. The method of any one of claims 1 to 5, wherein the input image is a 2D rendered image of a target object instance, and wherein obtaining the image comprises: Obtaining an initial image of the target object instance; as well as The 2D rendered image of the target object instance is generated from at least the initial image and by using a differentiable renderer.
7. The method of claim 6, wherein generating the 2D rendered image of the target object instance comprises: Sample random camera poses; as well as The 2D rendered image is generated using the differentiable renderer relative to the sampled random camera pose.
8. The method of claim 7, wherein determining the one or more updates to the 2D rendered image comprises: The differentiable renderer is updated based on back-propagating the gradient of the image-space objective function through the 2D rendered image to the differentiable renderer.
9. The method of any one of claims 7 to 8, wherein the differentiable renderer comprises: a neural radiance field (NeRF) model configured to generate (i) a set of volume density values defining a density estimate of the target object instance and (ii) a set of albedo values defining a color estimate of the target object instance; as well as A 2D renderer is configured to generate the 2D rendered image based on the set of volume density values and the set of albedo values.
10. The method of claim 9, wherein the NeRF model is a NeRF multi-layer perceptron (MLP) model, and wherein updating the differentiable renderer comprises updating parameter values of the NeRF MLP model.
11. The method of any one of claims 6 to 10, wherein the image space objective function further comprises a texture reconstruction loss term, the texture reconstruction loss term measuring the difference between (i) an enhanced image of the target object instance generated from the input image using the diffusion neural network and (ii) a rendered RGB image of the target object instance generated based on shape and texture estimation of the target object instance.
12. The method of claim 11, wherein the difference in the texture reconstruction loss term is masked by a foreground contour estimate of the target object instance that has been generated using a trained Visual Transformer (ViT) neural network.
13. The method of any one of claims 11 to 12, wherein generating the enhanced image from the input image using the diffusion neural network comprises adding Gaussian noise to background pixels in the input image.
14. The method according to any one of claims 6 to 13, further comprising: A three-dimensional (3D) representation of the target object instance is generated based on the input image using the updated differentiable renderer.
15. The method of claim 14, wherein the 3D representation comprises a 3D voxel grid representation or a 3D mesh representation.
16. The method of any one of claims 14 to 15, wherein the 3D representation of the target object instance has a user-specified pose, a user-specified motion, a user-specified texture, or a combination thereof.
17. A system implemented by one or more computers, comprising: One or more computers and one or more storage devices storing instructions which, when executed by the one or more computers, are operable to cause the one or more computers to perform the operations of the corresponding method as claimed in any one of claims 1 to 16.
18. A computer storage medium encoded with a computer program, the program comprising instructions operable when executed by a data processing apparatus to cause the data processing apparatus to perform the operations of a corresponding method as claimed in any one of claims 1 to 16.
Citation Information
Cited By
High-resolution three-dimensional reconstruction method of fusion diffusion model
CN120976443A
A high-resolution three-dimensional reconstruction method fusing a diffusion model
CN120976443B
