Automatic decoding of potential 3d diffusion models

Abstract

Systems and methods are provided for generating static and articulated 3D assets, the core of which includes a 3D automatic decoder. The 3D automatic decoder framework embeds attributes learned from the target dataset into potential space, which may then be decoded into volumetric representations to render view-consistent appearances and geometries. Appropriate intermediate volume potential spaces are then identified and robust normalization and anti-normalization operations are performed to learn 3D diffusion from 2D images or monocular videos of rigid or articulated objects. These methods are sufficiently flexible, may use existing camera supervision or do not use camera information at all, but rather effectively learn camera information during training. The results generated are proved to be superior to the most advanced alternatives in various reference datasets and indices, including multi-view image datasets of synthetic objects, real field videos of moving persons, and real video datasets of large-scale static objects.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Application Serial No. 18 / 211,149, filed June 16, 2023, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The examples described in this paper typically involve the generation of static and articulated 3D images, and in particular, methods and systems for automatically decoding latent 3D diffusion models to embed attributes learned from a target dataset in the latent space into a volumetric representation for rendering. Background Technology

[0004] The generation of photorealism is undergoing rapid development. Just over a year ago, the latest improvements in image quality, composition, stylization, resolution, scale, and manipulability were unimaginable. The richness of online images (often containing text, tags, labels, and sometimes even pixel-level segmentation) has greatly accelerated this process. The emergence and development of the Denoising Diffusion Probability Model (DDPM) has propelled these advancements in image synthesis and other fields such as audio and video.

[0005] However, the world is 3D, composed of both static and dynamic objects. Its geometric and temporal properties pose challenges to generative methods. First, the available data primarily consists of images and monocular video. For a limited number of object categories, 3D meshes with corresponding multi-view images or videos can be used, often acquired through tedious capture processes or manually created by artists. Second, unlike convolutional neural networks (CNNs), there is currently no widely accepted 3D or four-dimensional (4D) representation suitable for generating 3D geometry and appearance. Therefore, with a few exceptions, most existing 3D generative methods are limited to a narrow range of object categories applicable to available data and common geometric representations. Moving, articulated objects, such as humans, further complicate the problem, as the representation must also support deformation. Attached Figure Description

[0006] In accompanying drawings that are not necessarily drawn to scale, the same figures may depict similar parts in different views. Some non-limiting examples are shown in the drawings, in which:

[0007] Figure 1A This is a diagram illustrating the complete pipeline of a two-stage approach for automatically decoding potential 3D diffusion models in a sample configuration.

[0008] Figure 1BThis is a flowchart illustrating a method for automatically decoding a latent 3D diffusion model, which embeds attributes learned from a target dataset in the latent space into a volumetric representation of the object for rendering in an example configuration.

[0009] Figure 2 This is a graph showing the impact of diffusion resolution and the number of sampling steps on the sampling quality and inference time of the sampling method.

[0010] Figure 3 This is a diagram depicting samples generated from a model trained using monocular videos from MVImgNet.

[0011] Figure 4 It is a diagram depicting samples generated from a model trained using rendered images from the Objaverse.

[0012] Figure 5 This is a graph comparing three diffusion models trained with the same time, resources, and number of parameters, used in the automatic decoder at 4... 3 8 3 and 16 3 Diffusion is performed at three resolutions, of which 8 3 The model demonstrates an optimal tradeoff between quality and training speed, making it the best choice for training on large-scale 3D datasets.

[0013] Figure 6 It is a block diagram of a machine within which instructions (e.g., software, programs, applications, applets, or other executable code) can be executed to cause the machine to perform one or more methods discussed herein.

[0014] Figure 7 This is a block diagram illustrating a software architecture in which the examples described herein can be implemented. Detailed Implementation

[0015] The 3D autodecoder framework embeds attributes learned from the target dataset into a latent space for generating static and articulated 3D assets. The framework can decode into volumetric representations to render view-consistent appearance and geometry. Appropriate intermediate volumetric latent spaces are then identified, and robust normalization and denormalization operations are implemented to learn 3D diffusion from 2D images or monocular videos of rigid or articulated objects. Results outperform state-of-the-art alternatives on various benchmark datasets and metrics, including multi-view image datasets of synthetic objects, real-world video of moving people, and real-world video datasets of large-scale static objects.

[0016] The following description aims to design and train a denoising diffusion probabilistic model (DDPM) for efficient use on datasets of various sizes. The systems and methods described are general enough to handle rigid and articulated objects and versatile enough to learn diverse 3D geometries and appearances from multi-view images and monocular videos of static and dynamic objects. Identifying the pose of objects in such data is crucial for learning useful 3D representations. The systems and methods described are therefore designed to robustly learn poses during training, using ground truth poses, poses estimated using motion structures, or not using input pose information at all. The systems and methods described are scalable enough to be trained on large datasets of diverse objects in single or multi-class categories, suitable for synthesizing a wide range of realistic content.

[0017] Traditional diffusion methods consist of two phases. During the first phase, the autodecoder learns a rich latent space. In order to generate new samples, the diffusion process is trained during the second phase to explore this latent space. Training an image-to-image autodecoder requires many images. Similarly, training a 3D autodecoder requires a large amount of 3D data, which is scarce. Existing systems have used synthetic datasets such as ShapeNet, DiffRF, SDFusion, and similar datasets, and are therefore limited to domains where such data is available.

[0018] Unlike these methods, the system and method described in this paper use a volumetric autodecoder to learn the latent space of diffuse sampling. Unlike prior art autodecoder-based methods, the autodecoder maps one-dimensional (1D) vectors to each object in the training set and therefore does not require 3D supervision. The autodecoder uses rendering consistency as supervision to learn 3D representations from two-dimensional (2D) observations. Following unsupervised volumetric animation (UVA), the resulting 3D representations support the modeling of joint portions required for non-rigid objects.

[0019] Learning such a rich latent 3D space using an autodecoder presents several challenges, which the system and method described in this paper address. First, the autodecoder does not have a definite “bottleneck.” Starting with a 1D embedding, the autodecoder upsamples latent features at multiple resolutions until it finally reaches the output radiosity and density volume. Here, each intermediate volume representation can potentially become a “bottleneck.” Second, autoencoder-based methods typically normalize the bottleneck by imposing KL divergence constraints, meaning that diffusion is performed in a normalized space.

[0020] To identify the optimal size, an exhaustive layer-by-layer search can be performed. However, such a search is computationally very expensive, requiring hundreds of computationally costly experiments. Instead, the system and method described in this paper employ robust normalization and denormalization operations that can be applied to any layer of a pre-trained and fixed autodecoder. These operations compute robust statistics to perform layer normalization, thus allowing the diffusion process to be trained at any intermediate resolution of the autodecoder. The inventors have found that at relatively low resolutions, the space is compact and provides the necessary regularization for the geometry, causing the training data to contain only sparse observations of each object. Deeper layers, on the other hand, operate more as upsamplers.

[0021] The generality and scalability of the system and method described in this paper are demonstrated on a variety of tasks involving the synthesis of rigid and articulated 3D objects. The model was initially trained using multi-view images and a camera. It was then scaled up to hundreds of thousands of different objects trained on the real-world MVImgNet dataset, exceeding the capabilities of existing 3D diffusion methods. Finally, the model was trained on a subset of CelebroV-Text consisting of approximately 44,000 high-quality human motion video sequences.

[0022] Neural Radiation Fields (NeRF) enable high-quality novel viewpoint synthesis (NVS) of rigid scenes learned from 2D images. Volumetric neural rendering using NeRFss methods has been successfully applied to various tasks, including generating objects suitable for 3D-aware NVS. Inspired by the rapid development of Generative Adversarial Models (GANs) for generating 2D images and videos, subsequent works in the prior art have extended this to 3D content generation with neural rendering techniques. Such works have shown promising results in this task, but they suffer from limited multi-view consistency across arbitrary viewpoints and struggle to generalize to multi-class image datasets.

[0023] Existing techniques for estimating waveform generation gradients, known as Pi-GAN, employ neural rendering with periodic activation functions for generation with view-consistent rendering. However, this approach requires accurate estimation of the camera pose distribution on the dataset, limiting its applicability to free-viewpoint videos. EG3D and EpiGRAF use a three-plane representation of a 3D scene created by a generator-discriminator framework based on StyleGAN2. However, these techniques require pose estimation from keypoints (e.g., facial features) for training, again limiting the field of view.

[0024] Existing methods primarily generate content within a single object category, with limited variation in shape and appearance. A notable exception is 3DGP, which generalizes to ImageNet. However, 3DGP's reliance on monocular depth prediction limits its ability to generate forward-facing scenes. These limitations also hinder these methods from handling deformable, articulated objects. As explained below, the system and method described in this paper are applicable to both deformable and rigid objects and cover a wider range of perspectives.

[0025] Denoising diffusion probabilistic models (DDPM) represent the generation process as learning denoising on data that is progressively corrupted by a series of diffusion steps. Traditional techniques that improve training objectives, architectures, and sampling processes have made rapid progress in generating high-quality data across various data domains. However, these techniques have primarily demonstrated effectiveness in tasks where samples from the target domain are fully observable, rather than in tasks where only partial observations of the dataset's content are possible.

[0026] One such domain is 3D data, which, for most real-world content, is primarily observed in 2D images. Several traditional methods have shown promising initial results in this domain. DiffRF proposes reconstructing the NeRF volume of each object for a synthetic dataset and then applying diffusion training to it within the U-Net framework. However, DiffRF requires reconstructing many object volumes and is limited to low-resolution volumes due to the high computational cost of diffusion training. In contrast, the framework described below operates in the latent space of the autodecoder and effectively shares knowledge learned from all training data, thus achieving low-resolution, latent 3D diffusion. 3D autodecoders have previously been used to generate 3D shapes, but this approach requires ground-value 3D supervision and focuses only on shape generation, with textures added using off-the-shelf methods. In contrast, the framework described below can learn to generate surface appearances and corresponding geometries without such ground-value 3D supervision.

[0027] Recently, other existing methods have proposed using large-scale, pre-trained text-to-image 2D diffusion models for 3D generation. The key idea behind these methods is to use the 2D diffusion model to evaluate the rendering quality of randomly sampled viewpoints, and then use this information to optimize the 3D-aware representation of the content. However, compared to the methods described below, such methods require a more expensive optimization process to generate each new object.

[0028] This disclosure addresses the aforementioned and other limitations in the prior art by providing systems, methods, and instructions on a computer-readable medium to implement a method for training a three-dimensional (3D) diffusion model, thereby embedding attributes of two-dimensional (2D) images learned from a target dataset into a latent space using an autodecoder. The system includes a volumetric autodecoder (G) that learns embedding vectors from a library of embedding vectors corresponding to objects in the training dataset to generate latent 3D feature volumes and decodes these latent 3D feature volumes into a 3D voxel mesh representing the density and radiosity of the object's shape and appearance. In an example configuration, the autodecoder includes a first part G1 and a second part G2. The system also uses the volumetric autodecoder to train the 3D diffusion model. The 3D diffusion model operates in a 3D latent space obtained from the first part G1, which is volumetrically rendered using a voxel mesh and supervised by two-dimensional (2D) reconstruction from training images in the training dataset to extract structural and appearance attributes from the training dataset. In an example configuration, the second part G2 of the volumetric autodecoder generates a 3D representation of the object based on the structural and appearance attributes extracted from the training dataset.

[0029] Before decoding the upsampled latent 3D feature volumes into a 3D voxel mesh, the volume autodecoder can progressively upsample the latent 3D feature volumes to the desired resolution. The volume autodecoder can also perform robust normalization on the latent 3D feature volumes before training the 3D diffusion model. Robust normalization may involve using the median m as the distribution center of the latent 3D feature volumes and approximating the scale of the latent 3D feature volumes using the normalized interquartile range (IQR). These features can be normalized to normalized features before using the features F from the latent 3D feature volumes for diffusion through the 3D diffusion model. Then, during inference, before generating the 3D representation of the object, the features F can be denormalized by the second part G2 based on the structural and appearance attributes extracted from the training dataset. .

[0030] In the sample configuration, the embedding vectors are learned by a volumetric autodecoder, and the volumetric autodecoder can provide at least four residual blocks at each resolution in the autodecoder, and at the second-level resolution of the autodecoder 8. 3 and the third level resolution 16 3 Use a self-attention layer.

[0031] In the sample configuration, objects can be in a canonical pose, and the voxel grid can be trained using the ground truth pose, a pose estimated using motion structure estimation, or a pose learned from the training dataset during training. The canonical pose can include a canonical voxel representation of a density grid, which is a discrete representation of the density field, and a canonical representation of the red, green, and blue (RGB) radiation fields. Density values ​​and RGB values ​​can be trilinearly interpolated from the 3D voxel grid after decoding.

[0032] In further sample configurations, the background of training images in the training dataset can be removed before training the 3D diffusion model to improve training efficiency.

[0033] When an object is an articulated non-rigid object, its shape and local motion can be modeled based on its dynamic pose and the corresponding non-rigid deformation of local regions. A differentiable Perspective-n-Point algorithm can estimate the camera pose for each part of the non-rigid object and progressively refine the estimated camera pose during training using a combination of learned 3D keypoints for each part of the non-rigid object and the predicted corresponding 2D projections in each image. These parts can be combined with reasonable deformations using a learned volumetric linear blending skinning (LBS) algorithm, which incorporates skinning weights for each part of the non-rigid object estimated during 3D diffusion model training.

[0034] In a further configuration, each object in the training set can be represented by an embedding vector consisting of a concatenation of smaller embedding vectors. A deterministic mapping from each training object index to its corresponding concatenated embedding vector can be implemented using a hash function, where for object index k, the corresponding embedding index is:

[0035]

[0036] For those with 2 r A table of entries, where w and a are heuristic hash parameters used to reduce the number of collisions while maintaining an appropriate table size. The target non-rigid object can also be decomposed into... Each region contains one image. Each point and its corresponding One projection. A point can be shared among all non-rigid objects, and the non-rigid objects can be aligned in the learned gauge space to allow motion transfer between non-rigid objects.

[0037] In a further configuration, training may include extracting textual descriptions of objects in the training dataset by providing prompts and a first view of the object, along with a question requesting a description of the object's shape and color, for use in the inference phase to identify the object.

[0038] A detailed description of a method for automatically decoding a latent 3D diffusion model will now be described with reference to Figures 1 through 7. While this specification provides a detailed description of possible implementations, it should be noted that these details are exemplary only and in no way limit the scope of the subject matter of the invention.

[0039] The method and system for automatically decoding a latent 3D diffusion model in the exemplary configuration implements a two-stage approach. In the first stage, an autodecoder G containing a library of embedding vectors corresponding to objects in the training dataset is learned. These vectors are first processed to create a low-resolution latent 3D feature volume, then progressively upsampled, and finally decoded into a voxelized representation of the generated object shape and appearance. A voxel mesh trained on this volume is used using volumetric rendering techniques, and 2D reconstruction supervision is performed from the training images.

[0040] During the second phase, the automatic decoder G was divided into two parts. An autodecoder is then used to train a 3D diffusion model operating in a compact, 3D latent space derived from G1. Using structural and appearance properties extracted from the autodecoder training dataset, this 3D diffusion process allows voxel meshes to be used efficiently to generate diverse and realistic 3D content.

[0041] Figure 1A The complete pipeline 100 of the two-stage method in the example configuration is shown. Stage 1 (Auto-decoding of Volumes) trains an autodecoder 110 with two generative parts G1 112 and G2 114. The autodecoder 110 learns to assign a 1D embedding 120 to each training set object, which is processed by G1 into a latent volume space 130. The features of the latent volumes generated by G1 112 are normalized to the same range by robust normalization 140 to allow for a uniform set of diffusion hyperparameters across all datasets and all sampled volumes of the trained autodecoder 110. G2 114 decodes these volumes in the latent volume space 130 into a larger radiative volume 150 suitable for rendering. In Stage 2 (Latent 3D Diffusion), the parameters of the autodecoder 110 are frozen. The uniform diffusion hyperparameter set in the sampling volume 160 is then used to train a 3D denoising diffusion process (e.g., 3D U-Net) 170 to generate a 3D noise volume 180 from which objects are rendered via G2 at inference time.

[0042] During inference, G1 112 is not used because the amount of 3D noise 180 generated during phase 2 is randomly sampled, 3D U-Net 170 is used for iterative denoising at 185, robust denormalization is performed at 190, and decoding is performed via G2 114 for rendering, resulting in the generated radiative volume 195.

[0043] The following describes the volumetric autodecoding architecture, training procedure, and reconstruction loss of the autodecoder 110, as well as the training and sampling strategy for 3D diffusion in the latent space 130 of the autodecoder 110.

[0044] Automatic Decoder Architecture

[0045] In the sample configuration, a 3D voxel mesh is used to represent the 3D structure and appearance of the object. The object is assumed to be in its canonical pose, thus decoupling the 3D representation from the camera pose. This decoupling is desirable for learning a compact representation of the object and can also serve as a constraint to learn meaningful 3D structure from 2D images without direct 3D supervision. Specifically, the canonical voxel representation can be represented by a density mesh. Composition, density grid is a resolution Discrete representation of the density field, and This represents the RGB radiation field. Similar to NeRF, volumetric rendering can be used to integrate the radiance and opacity values ​​of rays along each view. However, unlike the original NeRF, instead of using a multilayer perceptron (MLP) to compute these local values, the density and RGB values ​​can be trilinearly interpolated from the decoded voxel mesh.

[0046] 3D voxel meshes for density and radiance ( and This can be generated by a volumetric autodecoder G110 trained using rendering supervision from 2D images. This method directly generates... and Instead of intermediate representations such as feature volume or triplanes, it renders more efficiently and ensures consistency across multiple views. It's worth noting that feature volume and triplanes involve running an MLP process for each sample point, which can require significant computational and memory overhead during training and inference.

[0047] The Autodecoder 110 is trained as a large-scale generative network spanning various object categories from large-scale multi-view or monocular video datasets. The architecture of Autodecoder 110 is tuned to support large-scale datasets, which presents a challenge in designing an Autodecoder 110 architecture capable of generating high-quality 3D content across various categories. To represent each of approximately 300,000 objects in the largest dataset, a very high-capacity Autodecoder 110 is required. To this end, a relatively basic decoder is extended to support various shapes and appearances in the target dataset by increasing the length of the embedding vectors learned by the decoder of Autodecoder 110 from 64 to 1024. The number of residual blocks per resolution in Autodecoder 110 can also be increased from 1 to 4. Furthermore, to reconcile the reconstructed appearance of objects, additional layers can be added at the second and third levels (resolution 8). 3 and 16 3 Introducing a self-attention layer in ).

[0048] Automatic Decoder Training

[0049] The autodecoder 110 is trained from image data using an analytical synthesis method, with the primary objective of minimizing the difference between the rendered image and the training image. The RGB color image C is rendered using volumetric rendering. Furthermore, a 2D occupancy mask O is rendered to monitor the contours of objects.

[0050] Pyramid perceptual loss is used as the primary reconstruction loss on the rendered image:

[0051] (1)

[0052] in , These are the resolutions RGB rendered images and training images; It is the i-th layer of the pre-trained VGG-19 network; and the operator Downsample the image to the pyramid level. The resolution.

[0053] Since the method described in this paper is interested in modeling individual objects within a sample configuration, the background is removed across all considered datasets. However, if the object's color is black (corresponding to a lack of density), the network can make the object semi-transparent. To improve the overall shape of the reconstructed object, a foreground-supervised loss can be used. Using a binary foreground mask (estimated or synthesized via an off-the-shelf matting method such as Segment Anything, depending on the dataset), the L1 loss is applied to the rendered occupancy map to match the mask loss corresponding to the image, as follows:

[0054] (2)

[0055] in These are the inference occupancy mask and the truth occupancy mask, respectively.

[0056] Because the resulting autodecoder 110 has a large capacity, generating a volume incurs greater overhead compared to rendering images based on this volume (which consists primarily of trilinear sampling of voxel cubes). Therefore, instead of rendering a single view for the canonical representation of the target object in each batch, four views are rendered for each object in the batch. This technique does not incur significant overhead and effectively increases the batch size by four times. As an added benefit, this technique improves the overall quality of the generated results by significantly reducing batch variance. Table 2 below ablates this technique and key architectural design choices to show their impact on sample quality.

[0057] For articulated, non-rigid objects, such as videos of human subjects, the object's shape and local motion can be modeled based on its dynamic pose and corresponding non-rigid deformations in local regions. This assumes that these sequences can be decomposed into a set... Smaller rigid components (e.g., The pose of the object can be estimated to ensure consistent alignment in a canonical 3D space. During training, the camera pose of each part is estimated and progressively refined using a combination of learned 3D keypoints for each part of the depicted object and the corresponding predicted 2D projections in each image. This estimation can be performed via a differentiable Perspective-n-Point (PnP) algorithm.

[0058] To incorporate these components with appropriate deformation, a learned volumetric linear blending skin (LBS) algorithm can be employed. (Voxenon mesh) Skin weights can be introduced to represent the skin weights of each deformation component. Since there is no prior knowledge of the contents or distribution of the object parts, the skin weights of each part are also estimated during training.

[0059] Potential 3D diffusion

[0060] The diffusion model architecture used in the system and method described in this paper extends existing diffusion techniques in 2D space to a potential 3D space. Its 2D operations (including convolution and self-attention layers) are implemented in a 3D decoder space. In the text conditionalization experiments, a similar cross-attention layer was used after the self-attention layer.

[0061] One of the key observations using the method described in this paper is that the features F in the latent space of the 3D autodecoder 110 have a bell-shaped distribution, which eliminates the need to impose any form of prior on it. Operating in the latent space without priors enables training a single autodecoder 110 for each possible latent diffusion resolution. However, the feature distribution F has a very long tail. This is assumed because the final density values ​​inferred by the network do not have any natural boundaries and can therefore fall within any range. In fact, networks are encouraged to make such predictions because they have the clearest boundaries between surfaces and blank areas. However, in order to allow all datasets and all trained autodecoders 110 to have a uniform set of diffusion hyperparameters, their features are normalized to the same range. This is equivalent to calculating the center and scale of the distribution. It is worth noting that due to the very long tail of the feature distribution, typical mean and standard deviation statistics will be severely biased. Therefore, a robust alternative based on the quantiles of the feature distribution is proposed, which centers the distribution on the median m and approximates its scale using the normalized interquartile range (IQR) of the normal distribution: Before using feature F for diffusion, these features are normalized to During inference, when the final volume is produced, the features are denormalized to... This method is referred to as robust normalization in this paper.

[0062] During inference, these methods rely on sampling methods from the Expression Diffusion Model (EDM) within the general framework described by Karras et al. in “Elucidating the DesignSpace of Diffusion-Based Generative Models” (Arxiv.org / abs / 2206.00364), with some minor modifications. Regardless of the experiments, the hyperparameters of the EDM matching the dataset distribution were fixed at 0.5, and feature statistics were modified in the feature processing step. Classifier-free guidance was also introduced for text conditionalization experiments. The study found that setting the weights to 3 yielded good results on all datasets.

[0063] Figure 1BThis is a flowchart illustrating a method 200 for automatically decoding a latent 3D diffusion model, which embeds attributes learned from a target dataset in the latent space into a volumetric representation of an object for rendering in an example configuration. As shown, the method includes processing a one-dimensional (1D) embedding vector 120 of an autodecoder (G) 110 at 210, the autodecoder 110 including a library of embedding vectors corresponding to objects in the training dataset to generate a latent 3D feature volume 130. At 220, the latent 3D feature volume 130 is progressively upsampled to the desired resolution. At 230, robust normalization 140 is performed on the latent 3D feature volume 130 to normalize features from the latent 3D feature volume 130 before decoding the latent 3D feature volume 130 into a 3D voxel mesh 150 representing the density and radiosity of the object's shape and appearance at 240.

[0064] At position 250, the automatic decoder 110 is divided into a first part G1 and a second part G2, wherein... Then, at 260, the autodecoder 110 is used to train a 3D diffusion model 170 operating in the 3D latent space obtained from the first part G1. The 3D diffusion model 170 uses volume rendering of a 3D voxel mesh 160 and uses two-dimensional (2D) reconstruction supervision from training images in the training dataset to extract the structure and appearance properties of the objects from the training dataset.

[0065] During inference, at position 270, the second part G2 of the autodecoder 110 uses structural and appearance attributes extracted from the training dataset to generate a 3D representation of the object. At position 280, robust denormalization is used to denormalize the object's features F to complete the generation of the object's 3D representation.

[0066] result

[0067] As shown below, the above method has been evaluated on multiple different datasets for both unconditional and conditional settings. The design choices in the autodecoder 110 and diffusion are also ablated below.

[0068] The datasets used to evaluate these methods are as follows. These methods are primarily evaluated on synthetic rendering datasets of 3D objects. However, results are also provided for challenging dynamic human object video datasets and static object video datasets. The datasets include:

[0069] ABO tables. These methods were evaluated on object rendering in a subset of the Tables dataset from the Amazon Berkeley objects (ABO) dataset, which consists of 1,676 training sequences, each with 91 renders, for a total of 152,516 renders.

[0070] PhotoShape Chairs. Images from the Chairs subset of the PhotoShape dataset were used, totaling 3,115,200 frames, with 200 renders for each of the 15,576 Chairs models.

[0071] The Objaverse dataset contains approximately 800,000 publicly available 3D models. A manually filtered subset of approximately 30,000 unique objects was used, varying with the object geometry and appearance. Six images were rendered for each training object, totaling approximately 1.8 million frames.

[0072] MVImgNet. For this dataset, approximately 6.5 million frames were used from 219,188 videos of real-world objects across 239 categories, averaging 30 frames per category. Grounded Segment Anything was used for background removal, and then filtering was applied to remove objects that failed to be segmented. This process yielded 206,990 usable objects.

[0073] CelebV-Text. The CelebV-Text dataset consists of approximately 70,000 high-quality celebrity video sequences captured in real-world outdoor environments, under lighting, with action and poses. They typically depict the head, neck, and upper body regions, but include more challenging pose and motion variations than previous datasets (e.g., VoxCeleb). A robust video matting framework was used to obtain masks for foreground supervision. Some sample filtering was used to ensure sufficient video quality and continuity for training. This produced approximately 44,400 unique videos, averaging about 373 frames per video, totaling approximately 16.6 million frames.

[0074] For training, camera parameters used to render each synthetic object dataset and estimated parameters provided for real video sequences in MVImgNet were used. These parameters were tuned to center the content and scale it to the render volume. For human videos in CelebrV-Text, an additional pose estimator and an autodecoder G110 were trained to predict the pose per frame for each joint region, ensuring all objects are aligned in canonical space. Notably, to create dynamic 3D videos, pose sequences transmitted from real videos of another person in the dataset were used.

[0075] Synthetic datasets. Results for the ABO tables and the PhotoShape Chairs dataset are shown in Table 1 below. Compared to other datasets, the results for the single-class, synthetic rendering dataset are relatively small, indicating that the method described in this paper performs well even when dealing with smaller, more homogeneous data. Ten views of 1000 samples from each dataset are rendered, and the Frèchet initial distance (FID) and kernel initial distance (KID) are shown compared to 10 ground truth images randomly selected from each training sequence. Results are shown in Table 1 compared to GAN-based and more recent diffusion-based methods. As shown, the published method significantly outperforms state-of-the-art methods on both metrics on the Tables dataset and achieves better or comparable results on the Chairs dataset. KID score multiplied by 10 3 .

[0076] Table 1

[0077]

[0078] Large-scale datasets. The aforementioned large-scale datasets were tested: MVImgNet, CelebrV-Text, and Objaverse. For each dataset, five images were rendered from random poses for each of the 10,000 generated samples. Table 2 below reports a comparison of the FID and KID of these experiments with five ground truth images for each of the 10,000 training objects. Since no previous method has been shown to generalize to such large-scale datasets, the model described in this paper is compared with the 1D latent space of the Direct Sampling-Based Autodecoder 110 architecture (using noise vectors generated from a standard normal distribution). This 3D generation method demonstrates good performance.

[0079] The disclosed method was also evaluated with different numbers of diffusion steps (16, 32, and 64). The results are shown in Table 2. Qualitative results show significant improvements in the fidelity and quality of geometry and texture. When identity is sampled directly in the 1D latent space, the normals and depth are noticeably less sharp, indicating the presence of spurious density in the sampled volume. Table 2 further supports this observation: both FID and KID are significantly lower than those obtained through direct sampling and generally improve with additional steps. KID score multiplied by 10... 3 .

[0080] Table 2

[0081]

[0082] An ablation study was conducted on the key design choices for the autodecoder 110 architecture and training. Starting with the final version, each part was subtracted. The model was then trained on the PhotoShape Chairs dataset, rendering four images for each of approximately 15,500 object embeddings.

[0083] Table 3 below provides the Peak Signal-to-Noise Ratio (PSNR) and the learned Perceptual Image Patch Similarity (LPIPS) reconstruction metrics. The final version of the process was found to significantly outperform the infrastructure and training process. While the largest improvement came from increasing the embedding size, simply removing multi-frame training resulted in a significant deterioration in quality for each metric. Interestingly, removing the self-attention layers slightly increased PSNR and decreased LPIPS. This is likely due to the increased training complexity introduced by these layers, which may be unnecessary for datasets of this size. Significant improvements in this feature were observed for large-scale datasets. Reducing the depth of the residual convolutional blocks and reducing the embedding size both resulted in a significant deterioration in overall quality, especially the latter. This suggests that even on smaller datasets, the additional capacity provided by these components has an impact.

[0084] Table 3

[0085]

[0086] In Table 3, "-" indicates that the component has been removed. The top row depicts the results of the unmodified architecture and training procedure as each element is removed sequentially. LPIPS results are multiplied by 10.

[0087] Ablation was also performed on the diffusion process to evaluate the choice of the number of diffusion steps (16, 32, and 64) and the resolution of the automatic decoder performing the diffusion (4). 3 8 3 and 16 3 The impact of this is noted. For these variants, PhotoShape Chairs followed generative quality training and evaluation protocols, except that randomness was disabled in sampling during inference to achieve more consistent performance on these tests. Each model was trained using roughly the same amount of time and computation. Figure 2 The effects of diffusion resolution and number of sampling steps on sample quality and inference time of the sample configuration method are shown.

[0088] exist Figure 2 In the process, the results obtained from diffusion in an earlier or later automatic decoder stage are compared with those published with a resolution of 8. 3There are significant differences in the results between the methods. Overfitting the training dataset at the lowest resolution layer leads to a significant quality degradation when new objects are processed via diffusion. Training at higher resolutions requires substantial resources, limiting convergence achievable within a reasonable timeframe. The number of sampling steps has a smaller, more variable impact. Increasing inference time from 16 to 32 steps can reasonably improve results, but the greatest improvement is seen at 16 resolution in 64 steps. 3 Each sample requires more than 30 seconds. The selected diffusion resolution is 8. 3 It achieves optimal results, allowing for high sample quality in 64 steps with only about 8 seconds of computation, but provides reasonable results in 32 steps with about 4 seconds.

[0089] Finally, the diffusion model was trained using text-conditional methods. For MVImgNet and Objaverse, the text was generated using an off-the-shelf captioning system. The qualitative results for MVImgNet and Objaverse are as follows: Figure 3 and Figure 4 As shown. Figure 3 The image shows samples generated from a model trained using monocular video from MVImgNet, while... Figure 4 The image shows samples generated from a model trained using rendered images from Objavirse. Figure 3 and Figure 4 In the diagram, each object is shown in three views, along with the normals for each view. The rightmost view, along with the results of the text conditions, also displays the depth. Truth captions were generated using MiniGPT-4. It can be seen that, in all cases, the disclosed method generates objects with reasonable geometry that generally follow the cues. However, some details may be lost. The disclosed model seems to have learned to ignore certain details from the text cues, as MiniGPT-4 often produces the illusion of inconsistencies with the object's appearance details. A better captioning system should help mitigate this problem in the future.

[0090] While performing flexible 3D content generation for arbitrary content domains without 3D supervision presents inherent challenges, the methods described in this paper demonstrate that it is possible with the right approach. By leveraging the inherent capability of autodecoders to synthesize content within a domain without corresponding encoded input, the disclosed method learns representations of the structure and appearance of a wide variety of complex content suitable for generating high-fidelity 3D objects using only 2D supervision. The latent volumetric representation facilitates 3D diffusion modeling for both conditional and unconditional generation, while enabling view-consistent rendering of the synthesized objects. The results demonstrate that this generalizes well to a wide range of domains and datasets, from relatively small, single-class, synthetic rendering to large-scale, multi-class real-world datasets. It also supports the challenging task of generating articulated moving objects from video. These problems are addressed within a single framework.

[0091] The disclosed methods focus on images and videos depicting a key person or object in the foreground. Generating or composing more complex, multi-object scenes is a challenging task and an interesting direction for future development. Because these methods are trained using multiple views or video sequences of each object in the dataset, they do not support single-image datasets. Learning the appearance and geometry of diverse content from such limited data for controlled 3D generation and animation is extremely challenging, especially for articulated objects. However, it may be feasible for such purposes to reduce or eliminate the multi-image requirement when learning to generate additional object categories by using general knowledge about shape, motion, and appearance extracted from the dataset used in this paper. This would allow for the generation of content learned from potentially infinitely large and diverse image datasets.

[0092] Volume rendering

[0093] As described above, learnable volumetric rendering is used to generate the final image from the final decoded volume. Given the camera intrinsic and extrinsic parameters of the target image, and the radiant field volume generated by the automatic decoder 110, for each pixel in the image, light rays pass through this volume, sampling color and density values ​​by traveling along the light rays. Color is calculated using integral radiance. It has a near boundary and a far boundary. and :

[0094] (3)

[0095] in , It is the density and RGB values ​​of the radiation field volume sampled along these rays, and yes and The cumulative transmittance between.

[0096] To monitor the outline of objects, the 2D occupancy map O also uses volume equations for rendering:

[0097] (4)

[0098] During training and inference, 128 points on these rays are sampled for radiation field rendering.

[0099] Jointed animation

[0100] Because the techniques described in this paper are flexibly designed to support both rigid and articulated objects, different approaches can be used for pose supervision to better handle each case.

[0101] For articulated objects, learnable 3D keypoints can be used during training. The set of images and their predicted 2D projections in each image in an extended version of the Perspective-n-Point (PnP) algorithm. To estimate the pose. However, in order to handle articulated animation, instead of using these points to learn the individual pose in each image, it is assumed that the target object can be decomposed into... Each region contains one image. Points and their corresponding Each point is a projection. These points are shared across all objects and aligned in the learned canonical space, allowing for realistic generation and motion transformations between these objects. This allows for the learning of each frame's... Each posture defines each region. The posture relative to its posture in the normative posture of learning.

[0102] Successfully reconstructing the training images for each object involves learning the appropriate canonical locations of 3D keypoints in each region to predict the 2D projections of these keypoints in each frame, as well as the pose that best matches the 3D points and 2D projections of these regions. This information is then used in a volumetric rendering framework to appropriately sample from the canonical space so that the appearance and pose of the object remain consistent and appropriate throughout the video sequence. Using this approach, this information can be learned along with the auto-decoder parameters for articulated objects, using reconstruction and foreground supervision losses designed for rigid object datasets.

[0103] As mentioned above, to better handle the non-rigid shape deformation corresponding to this articulated motion, volumetric linear hybrid skin (LBS) can be employed. This allows for the learning of each component in the gauge space based on the spatial correspondence between these two spaces. Weights of sampling points in the deformable space:

[0104] (5)

[0105] in It is the estimated pose of part p relative to the camera (where (This is an estimated camera pose relative to the standard volume). It is a 3D point that deforms to correspond to the current pose; It is the corresponding point when aligned in the standard volume; and Position in volume Sampled components The learned LBS weights are used to define this correspondence. It is worth noting that inverse LBS weights following Human NeRF can be used to compute approximate solutions, avoiding the overcomputation used by direct solutions.

[0106] Therefore, for non-rigid objects, in addition to the density and color volume required to integrate equations 3 and 4 above, the autodecoder 110 learns to generate a volume containing the constituent parts of the object. The volume of each LBS weight in a locally rigid region By each point in the deformation space Points in the norm space Establishing a correspondence between them allows for a complete description of the deformation. This correspondence is established using Linear Blend Skin (LBS), as shown below:

[0107] (6)

[0108] in is the weight assigned to each part p. Intuitively, LBS weights divide the object into distinct parts. As an example, a point with an LBS weight of 1.0 on the left will always move according to the left-hand transformation. Unfortunately, during volumetric rendering, canonical points are typically queried using points in deformation space, which requires solving equation (6) to obtain The cost of this procedure is too high, so defining inverse LBS weights can be used instead. An approximate solution, so that:

[0109] (7)

[0110] Among them, weight Defined as follows:

[0111] (8)

[0112] This approximation has an intuitive explanation: given a deformation point, using the inverse... Project it onto the canonical pose and check if it corresponds to part p in the canonical pose.

[0113] In the sample test configuration, for Each of the regions was assigned 3D keypoints were used. For these tests, a fixed camera interior function with a field of view of 0.175 radians was assumed. The camera pose was estimated during training, and training was accelerated using the learned 3D keypoints and their predicted 2D projections for each estimated region of the depicted object (implemented using the differentiable Perspective-n-Point (PnP) algorithm from PyTorch3D). Since this was sufficient for objects with standard canonical shapes performing non-rigid motion in continuous video sequences (e.g., faces), this approach was used for testing on the CelebroV-Text dataset. While theoretically this approach could be used for pose estimation of rigid objects (with only 1 component) in each view, it may be less reliable for rigid object datasets, which contain sparse, multi-view images from randomly sampled, non-continuous camera poses depicting content with drastic changes in shape and appearance (e.g., multi-class object datasets described below). Therefore, for these objects, known ground truth or estimated camera poses were used as input to either the synthetic rendering or the real image, respectively. Those skilled in the art will recognize that category-independent object or camera pose estimation can also be achieved using predefined keypoints from sparse images without arbitrary objects or scenes.

[0114] Architectural features

[0115] For the aforementioned volumetric autodecoder 110 architecture, given an embedding vector e of size 1024, a fully connected layer can be used, followed by a reshaping operation to transform it into a 4-dimensional vector with 512 features per unit. 3 The volume is then processed. This is followed by a series of four 3D residual blocks, each upsampling the volume resolution for each dimension and halving the features of each cell, resulting in a final resolution of 64. 3 And 32 features. These blocks consist of two... The convolutional path consists of four convolutional blocks, each of which undergoes batch normalization in the main path, while the residual path consists of four convolutional blocks. The convolutional layers are followed by a Corrected Linear Activation Unit (ReLU) function. After the first block in these layers, an 8-bit array with 256 features per unit for training the diffusion network can be obtained. 3 Each volume is represented. Self-attention layers are applied in this block and subsequent blocks. After the final upsampling block, final batch normalization is applied, followed by... Convolution to produce the final output used in the volumetric renderer. density and RGB color features .

[0116] For non-rigid objects, this architecture produces Output channels, where the latter set of channels has the corresponding region in the canonical space. LBS weights for local rigid components. The unsupervised 2D keypoint predictor uses a U-Net architecture, which is optimized for downsampling. The input image is processed to predict the position of keypoints corresponding to each 3D keypoint, which are used to determine the pose of the camera relative to each region of the object when aligned in the canonical volume space.

[0117] For the basic diffusion model architecture, the ablation diffusion model (ADM) can be used, which is a U-Net architecture originally designed for 2D image synthesis. Preprocessing enhancements for this model can be incorporated. Since this architecture was originally designed for 2D, all convolutional and normalization operations, as well as the attention mechanism, are applicable to 3D. For the cross-attention mechanism used in conditioned reflex experiments, the latent space cross-attention mechanism is also extended to the 3D latent space.

[0118] ablation for potential volumetric resolution of diffusion

[0119] Three diffusion models were trained for the same amount of time, resources, and parameters to perform diffusion at three resolutions in the autodecoder 110: 4 3 8 3 and 16 3 The study found that 4 3 Individual models (even those that train faster) often fail to converge to something meaningful and produce partial results. (From 16) 3 Most samples generated by the model are of acceptable quality. However, many samples also exhibit spurious density values. For example... Figure 5 As shown, 8 3 The diffusion model produced the best results, and its fast training speed makes it suitable for large-scale training.

[0120] Preparing text embeddings for text-driven generation

[0121] The model was trained on three datasets for text-conditional image generation: CelebrV-Text, MVImgNet, and Objaverse. The latter two datasets provide the object category for each sample, but they do not provide text descriptions. Using MiniGPT4, the model generates images by providing a hint, a first view of each object, and a question: <imagehere>Describe this in one sentence <hint>Describe its shape and color. Keep it concise, using only one sentence. Extract the description using the given description. For MVImgNet, this hint is "class name," while for Objaverse, it's "asset name." The assigned description can then be used during the inference phase to identify the object to be rendered.

[0122] Using text-image pairs from these three datasets, an 11 billion-parameter T5 model was used to extract a series of text embedding vectors. These vectors were 1024-dimensional. During training, the length of the embedding sequence was fixed at 32 elements. Longer sentences were pruned, and smaller sentences were padded with zeros.

[0123] Hash embedding

[0124] Each object in the training set is encoded by an embedding vector. However, due to the use of multi-view datasets of various sizes, with up to approximately 300,000 unique objects from multiple categories, storing a separate embedding vector for each object depicted in the training images is cumbersome. Therefore, a technique can be used to efficiently utilize a significantly reduced number of embeddings (no more than approximately 32,000 embeddings required for any evaluation) while allowing the generation of effective content from large-scale datasets. This is achieved by concatenating smaller embedding vectors to create more combinations of the unique embedding vectors used during training. Regarding the embedding vector length... The input embedding vector used for the object to be decoded. It is a smaller embedding vector A cascade, where each vector is derived from having Selected from an ordered keybook of entries, where each entry contains a length of... of A set of embedding vectors:

[0125] (9)

[0126] in It is used to extract from location In the final vector The index set is selected from 10 possible codebook entries. This approach allows for an exponentially greater number of embedding vector combinations during training than must be provided in the learned embedding vector library.

[0127] However, although it is possible to randomly select a location for each location. Vector at position index Instead of accessing the corresponding codebook entry, the inventors used a deterministic mapping from each training object index to its corresponding concatenated embedding vector. This function is implemented using a hash function employing a multiplication approach, enabling fast indexing with efficient bitwise operations. For object indices... The corresponding embedding index is:

[0128] (10)

[0129] The table has 2 r 1 entry. w and a are heuristic hash parameters used to reduce the number of collisions while maintaining an appropriate table size. 32 is used for w, while a must be between 32 and 32. and Odd numbers between. Each smaller codebook has its own value for a:

[0130] (11)

[0131] in It is the index of the codebook. Research found that the impact of this method on the overall speed and quality of the training and synthesis processes is negligible. It was also found that using this hash embedding method can reduce the model storage requirements for this dataset by approximately 75%.

[0132] Processing platform

[0133] Figure 6 This is a schematic representation of machine 600, in which instructions 610 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause machine 600 to perform one or more of the methods discussed herein. For example, instructions 610 can cause machine 600 to perform one or more of the methods described herein. Instructions 610 transform a general, unprogrammed machine 600 into a specific machine 600 programmed to perform the described and illustrated functions in the manner described. Machine 600 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 600 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 600 may include, but is not limited to, server computers, client computers, personal computers (PCs), tablets, laptops, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web appliances, network routers, network switches, bridges, or any machine capable of sequentially or otherwise executing instructions 610, which specify the actions to be taken by machine 600. Furthermore, although only a single machine 600 is shown, the term "machine" should also be considered as a collection of machines that individually or jointly execute instructions 610 to perform one or more of the methods discussed herein. Machine 600 may, for example, implement Figure 1A The two-stage pipeline 100. In some examples, machine 600 may also include client and server systems, where certain operations of a particular method or algorithm are performed on the server side, and certain operations of a particular method or algorithm are performed on the client side.

[0134] Machine 600 may include processor 604, memory 606, and input / output (I / O) units 602, which may be configured to communicate with each other via bus 640. In one example, processor 604 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof may be included, such as processor 608 and processor 612 executing instructions 610. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although Figure 6 Multiple processors 604 are shown, but machine 600 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0135] Memory 606 includes main memory 614, static memory 616, and memory cells 618, all of which are accessible to processor 604 via bus 640. Main memory 606, static memory 616, and memory cells 618 store instructions 610 for one or more of the methods or functions described herein. During execution of instructions 610 by machine 600, instructions 610 may also reside wholly or partially in main memory 614, static memory 616, machine-readable medium 620 within memory cells 618, at least one of processor 604 (e.g., within the processor's cache), or any suitable combination thereof.

[0136] I / O component 602 may include a wide variety of components to receive input, provide output, generate output, transmit information, exchange information, capture measurements, and so on. The specific I / O component 602 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O component 602 may include... Figure 6 Many other components are not shown. In various examples, I / O component 602 may include user output component 626 and user input component 628. User output component 626 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistive mechanisms), and other signal generators, etc. User input component 628 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide the position and force of a touch or touch gesture), and audio input components (e.g., microphones), etc.

[0137] In a further example, I / O component 602 may include biometric component 630, motion component 632, environmental component 634 or position component 636, and a range of other components. For example, biometric component 630 includes components for detecting expressions (e.g., hand gestures, facial expressions, vocal expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brainwaves), and identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 632 includes accelerometer components (e.g., accelerometers), gravity sensor components, and rotation sensor components (e.g., gyroscopes).

[0138] Any biometric data collected by biometric component 630 is captured and stored only with the user's approval and deleted upon the user's request. Furthermore, such biometric data may be used for very limited purposes, such as authentication. To ensure the limited and authorized use of biometric information and other personally identifiable information (PII), access to such data is restricted to authorized personnel, if permitted. Any use of biometric data may be strictly limited to authentication purposes, and biometric data may not be shared or sold to any third party without the user's explicit consent. Additionally, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of this sensitive information.

[0139] Environmental component 634 includes, for example, one or more cameras (with still image / photograph and video capabilities), lighting sensor components (e.g., a photometer), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., a barometer), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., an infrared sensor for detecting nearby objects), gas sensors (e.g., a gas detection sensor for detecting hazardous gas concentrations to ensure safety or measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0140] The position component 636 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude is derived), and an orientation sensor component (e.g., a magnetometer), etc.

[0141] Communication can be implemented using a wide variety of technologies. I / O component 602 also includes communication component 638, operable to couple machine 600 to network 622 or device 624 via appropriate coupling or connection. For example, communication component 638 may include a network interface component or another suitable device that interfaces with network 622. In further examples, communication component 638 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components that provide communication via other modalities. Device 624 may be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0142] Furthermore, the communication component 638 can detect identifiers or include components operable for detecting identifiers. For example, the communication component 638 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes (such as Universal Product Code (UPC) barcodes), multi-dimensional barcodes (such as Quick Response (QR) codes), Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCCRSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying tag audio signals). Additionally, various information can be derived via the communication component 638, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detection of NFC beacon signals that can indicate a specific location, etc.

[0143] Various memories (e.g., main memory 614, static memory 616, and memory of processor 604) and storage units 618 may store one or more sets of instructions and data structures (e.g., software) embodying or used by one or more of the methods or functions described herein. When executed by processor 604, these instructions (e.g., instruction 610) cause various operations to be performed on the disclosed examples.

[0144] Instruction 610 can be sent or received via network 622, using a transmission medium, via a network interface device (e.g., the network interface component included in communication component 638), and using any of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, instruction 610 can be sent or received using a transmission medium via coupling to device 624 (e.g., peer-to-peer coupling).

[0145] Figure 7 This is a block diagram 700 illustrating a software architecture 704 that can be installed on one or more devices in the apparatus described herein. The software architecture 704 is supported by hardware, such as machine 702 (see...). Figure 6 This includes a processor 720, memory 726, and I / O components 738. In this example, the software architecture 704 can be conceptualized as a layered stack, where each layer provides specific functionality. The software architecture 704 includes layers such as an operating system 712, libraries 710, frameworks 708, and applications 706. Operationally, application 706 invokes API call 750 through the software stack and receives messages 752 in response to API call 750.

[0146] Operating system 712 manages hardware resources and provides general services. Operating system 712 includes, for example, kernel 714, services 716, and drivers 722. Kernel 714 acts as an abstraction layer between hardware and other software layers. For example, kernel 714 provides memory management, processor management (e.g., scheduling), component management, network and security settings, and other functions. Services 716 can provide other public services to other software layers. Drivers 722 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 722 may include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Power drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), Wi-Fi® drivers, audio drivers, and power management drivers, etc.

[0147] Library 710 provides the general-purpose low-level infrastructure used by application 706. Library 710 may include system libraries 718 (e.g., the C standard library), which provide functions such as memory allocation, string manipulation, and mathematical functions. Additionally, library 710 may include API libraries 724, such as media libraries (e.g., libraries for supporting the rendering and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codecs, Joint Picture Experts Group (JPEG or JPG) or Portable Web Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for rendering graphical content in two-dimensional (2D) and three-dimensional (3D) on a display), database libraries (e.g., SQLite for providing various relational database functions), web libraries (e.g., WebKit for providing web browsing capabilities), etc. Library 710 may also include a wide variety of other libraries 728 to provide many other APIs to application 706.

[0148] Framework 708 provides general-purpose high-level infrastructure for use by application 706. For example, framework 708 provides various graphical user interface (GUI) functions, advanced resource management, and advanced location services. Framework 708 can provide a wide range of other APIs that can be used by application 706, some of which may be specific to a particular operating system or platform.

[0149] In one example, application 706 may include a home application 736, a contacts application 730, a browser application 732, a book reader application 734, a location application 742, a media application 744, a messaging application 746, a game application 748, and various other applications, such as third-party applications 740. Application 706 is a program that performs the functions defined in the program. One or more applications 706 may be generatively constructed in various ways using various programming languages, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a particular example, third-party application 740 (e.g., an application developed by an entity other than a platform-specific vendor using the Android™ or iOS™ Software Development Kit (SDK)) may be mobile software operating on a mobile operating system such as iOS™, Android™, Windows® Phone, or another mobile operating system. In this example, third-party application 740 may invoke API calls 750 provided by operating system 712 to facilitate the functions described herein.

[0150] "Carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions to be executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0151] "Client device" refers to any machine that connects to a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablets, ultrabooks, netbooks, laptops, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0152] "Communication network" refers to one or more parts of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless local area network (WLAN), wide area network (WAN), wireless wide area network (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a common old-style telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, a network or a part of a network may include a wireless or cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other types of cellular or wireless coupling. In this example, the coupling can implement any of a variety of data transmission technologies, such as Single Carrier Radio Transmission (1xRTT), Evolved Data Optimization (EVDO), General Packet Radio Service (GPRS), Enhanced Data Rate GSM Evolution (EDGE), the 3rd Generation Partnership Project (3GPP) including 3G, 4G networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other standards defined by various standards-setting organizations, other remote protocols, or other data transmission technologies.

[0153] A "component" refers to a device, physical entity, or logic having boundaries defined by function or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularization of specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processes. A component can be an encapsulated functional hardware unit designed for use with other components, and part of a program that typically performs a specific function related to that function. Components can constitute software components (e.g., code embodied on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit capable of performing operations and can be configured or arranged in some physical manner. In various examples, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or a group of processors) of a computer system can be configured by software (e.g., an application or application portion) to perform certain operations described herein. Hardware components can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic permanently configured to perform certain operations. A hardware component can be a dedicated processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). Hardware components may also include programmable logic or circuitry temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific part of a machine) tailored specifically to perform the configured functions, and is no longer a general-purpose processor. It should be understood that the decision to implement a hardware component mechanically in dedicated and permanently configured circuitry or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time factors. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include tangible entities, i.e., entities that physically construct, permanently configure (e.g., hardwire), or temporarily configure (e.g., programmable) to operate or perform certain operations described herein. Considering examples where hardware components are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each of the hardware components at any given point in time. For example, in the case where a hardware component includes a general-purpose processor configured by software as a dedicated processor, that general-purpose processor may be configured at different times as correspondingly different dedicated processors (e.g., including different hardware components). The software accordingly configures one or more specific processors, for example, constituting a specific hardware component at one time and a different hardware component at different times. The hardware components can provide information to other hardware components and receive information from other hardware components. Therefore, the hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between these hardware components can be achieved, for example, by storing and retrieving information in a memory structure accessible to the multiple hardware components. For example, one hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Another hardware component can then later access that memory device to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., collections of information). The various operations of the example methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components whose operations are to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors.

[0154] Similarly, the methods described herein can be implemented at least in part by a processor, where a specific processor or one or more processors are examples of hardware. For example, at least some operations of a method can be performed by one or more processors or processor-implemented components. Furthermore, one or more processors can also operate to support the performance of related operations in a "cloud computing" environment or as "Software as a Service" (SaaS). For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The performance of certain operations may be distributed among processors, residing not only within a single machine but also deployed across multiple machines. In some examples, the processor or processor-implemented component may be located in a single geographic location (e.g., in a home environment, office environment, or server farm). In other examples, the processor or processor-implemented component may be distributed across multiple geographic locations.

[0155] "Computer-readable storage medium" refers to both machine storage media and transmission media. Therefore, the term includes storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" refer to the same thing and are used interchangeably in this disclosure.

[0156] "Machine storage medium" refers to one or more storage devices and media (e.g., centralized or distributed databases and associated caches and servers) that store executable instructions, routines, and data. Therefore, this term should be understood to include, but is not limited to, solid-state memory, optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include non-volatile memory, including, for example, semiconductor storage devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "computer storage medium," and "device storage medium" explicitly exclude carrier waves, modulated data signals, and other such media, at least some of which are included in the term "signal medium."

[0157] "Non-transitory computer-readable storage medium" refers to a tangible medium capable of storing, encoding, or carrying instructions that can be executed by a machine.

[0158] "Signal medium" means any intangible medium capable of storing, encoding, or carrying instructions executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be understood to include any form of modulated data signal and carrier wave, etc. The term "modulated data signal" refers to a signal whose characteristics are set or altered in such a way as encoding information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and are used interchangeably in this disclosure.< / hint> < / imagehere>

Claims

1. A method for training a three-dimensional (3D) diffusion model using an automatic decoder to embed attributes of a two-dimensional (2D) image learned from a target dataset into a latent space, the method comprising: The embedding vectors of the automatic decoder (G), which includes a library of embedding vectors corresponding to objects in the training dataset, are processed to generate a latent 3D feature volume. The automatic decoder decodes the latent 3D feature volume into a 3D voxel mesh representing the density and radiosity of the object's shape and appearance; The automatic decoder is divided into a first part G1 and a second part G2; The automatic decoder is used to train a 3D diffusion model operating in a 3D latent space obtained from the first part G1, the 3D latent space being volumetrically rendered using the 3D voxel mesh, and supervised by two-dimensional (2D) reconstruction from training images in the training dataset to extract structural and appearance attributes from the training dataset. as well as Using the second part G2 and the structural and appearance attributes extracted from the training dataset, a 3D representation of the object is generated.

2. The method according to claim 1, further comprising: The potential 3D feature volume is progressively upsampled before being decoded into the 3D voxel mesh.

3. The method according to claim 1, further comprising: Before training the 3D diffusion model, a robust normalization is performed on the latent 3D feature volume. The robust normalization includes using the median m as the distribution center of the latent 3D feature volume, using the normalized interquartile range (IQR) to approximate the scale of the latent 3D feature volume, and normalizing the feature F from the latent 3D feature volume to the difference between the feature F and the median m divided by the IQR before using the feature F from the latent 3D feature volume to diffuse through the 3D diffusion model.

4. The method according to claim 3, further comprising: During inference, before generating the 3D representation of the object, the second part G2 denormalizes the feature F into the sum of the median m and the product between the normalized feature and the IQR based on the structural and appearance attributes extracted from the training dataset.

5. The method according to claim 1, further comprising: The embedding vector is learned by the automatic decoder.

6. The method according to claim 1, wherein, Decoding by the automatic decoder includes providing at least four residual blocks at each resolution in the automatic decoder, and at the second-level resolution 8 of the automatic decoder. 3 and the third level resolution 16 3 Use a self-attention layer.

7. The method according to claim 1, wherein, The object is in a normal pose, and training the 3D voxel mesh includes training the 3D voxel mesh using a ground truth pose, a pose estimated using motion structure, or a pose learned from the training dataset during training.

8. The method according to claim 7, wherein, The canonical pose includes a canonical voxel representation of the density grid and a canonical representation of the red, green, and blue (RGB) radiation fields. The density grid is a discrete representation of the density field. It also includes trilinear interpolation of the density values ​​and RGB values ​​from the 3D voxel grid after decoding.

9. The method according to claim 1, further comprising: Remove the background of the training images in the training dataset before training the 3D diffusion model.

10. The method according to claim 1, wherein, The object is an articulated non-rigid object, and the modeling also includes the object's shape and local motions from dynamic postures, as well as the corresponding non-rigid deformations of local regions.

11. The method according to claim 10, further comprising: The camera pose of each part of the non-rigid object is estimated using a differentiable Perspective-n-Point algorithm, and the estimated camera pose is progressively refined during training using a combination of learned 3D keypoints for each part of the non-rigid object and the corresponding predicted 2D projections in each image. The learned volumetric linear blending skin (LBS) algorithm is used to combine the parts with reasonable deformations, the algorithm having skin weights of the individual parts of the non-rigid object estimated during the training of the 3D diffusion model.

12. The method according to claim 1, further comprising: Each object in the training set is represented by a concatenated embedding vector comprising a smaller embedding vector, wherein representing each object includes: a deterministic mapping from each training object index to its corresponding concatenated embedding vector using a hash function, wherein for object index k, the corresponding embedding index is: For those with 2 r A table of entries, where w and a are heuristic hash parameters used to reduce the number of collisions while maintaining an appropriate table size.

13. The method according to claim 1, further comprising: Decompose the target non-rigid object into Each region contains one image. Each point and its corresponding A projection, shared among all non-rigid objects. The system uses points to align the non-rigid objects in the learned canonical space, allowing motion transfer between the non-rigid objects.

14. The method according to claim 1, wherein, The training includes extracting textual descriptions of objects from the training dataset by providing prompts and a first view of the object, along with questions requesting a description of the object's shape and color, for use in identifying the object during the inference phase.

15. A system for rendering objects by embedding attributes learned from a target dataset in a latent space into a volumetric representation of the object, the system comprising: The volume autodecoder (G) learns embedding vectors from a library of embedding vectors corresponding to objects in the training dataset to generate latent 3D feature volumes and decodes the latent 3D feature volumes into a 3D voxel grid of density and radiosity representing the shape and appearance of the objects. The autodecoder includes a first part G1 and a second part G2. and A 3D diffusion model trained on the latent representation by the volumetric autodecoder operates in the 3D latent space obtained from the first part G1 using volumetric rendering of the voxel mesh, and is supervised by two-dimensional (2D) reconstruction from training images in the training dataset to extract structural and appearance attributes from the training dataset. The second part G2 of the volume autodecoder generates a 3D representation of the object based on the structural and appearance attributes extracted from the training dataset.

16. The system according to claim 15, wherein, Before training the 3D diffusion model, the volume autodecoder progressively upsamples the latent 3D feature volumes and performs robust normalization on the upsampled latent 3D feature volumes. The robust normalization includes taking the median m as the distribution center of the latent 3D feature volumes and using the normalized interquartile range (IQR) to approximate the scale of the latent 3D feature volumes. Before using the feature F from the latent 3D feature volumes for diffusion through the 3D diffusion model, the feature F is normalized to the difference between feature F and the median m divided by the IQR. During inference, before generating the 3D representation of the object, the feature F is denormalized by the second part G2 based on structural and appearance attributes extracted from the training dataset to the sum of the median m and the product of the normalized feature and the IQR.

17. The system according to claim 15, wherein, The volumetric autodecoder provides at least four residual blocks for each resolution in the volumetric autodecoder, and includes a second-level resolution of 8. 3 and the third level resolution 16 3 The self-attention layer.

18. The system according to claim 15, wherein, The object is an articulated non-rigid object, wherein the volumetric autodecoder models the shape of the object and local motions from dynamic poses, as well as corresponding non-rigid deformations of local regions. The volumetric autodecoder includes a differentiable Perspective-n-Point algorithm and a learned Volumetric Linear Blending Skin (LBS) algorithm. The differentiable Perspective-n-Point algorithm estimates the camera pose of each part of the non-rigid object and progressively refines the estimated camera pose during training using a combination of 3D keypoints learned for each part of the non-rigid object and the predicted corresponding 2D projections in each image. The learned Volumetric Linear Blending Skin (LBS) algorithm uses skinning weights of each part of the non-rigid object estimated during training of the 3D diffusion model to combine the parts with reasonable deformations.

19. The system according to claim 15, wherein, The 3D diffusion model represents each object in the training set through a concatenated embedding vector comprising smaller embedding vectors, and also includes an encoder that encodes each object using a deterministic mapping from each training object index to its corresponding concatenated embedding vector, and using a hash function, where for object index k, the corresponding embedding index is: For those with 2 r A table of entries, where w and a are heuristic hash parameters used to reduce the number of collisions while maintaining an appropriate table size.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform a method for training a three-dimensional (3D) diffusion model, the method embedding attributes of a two-dimensional (2D) image learned from a target dataset in the future into a latent space using an automatic decoder by performing operations including: The embedding vectors of the automatic decoder (G), which includes a library of embedding vectors corresponding to objects in the training dataset, are processed to generate a latent 3D feature volume. The automatic decoder decodes the latent 3D feature volume into a 3D voxel mesh representing the density and radiosity of the object's shape and appearance; The automatic decoder is divided into a first part G1 and a second part G2; A 3D diffusion model operating in a 3D latent space obtained from the first part G1 is trained using the autodecoder, the latent space being volumetrically rendered using the voxel mesh, and supervised by 2D reconstruction from training images in the training dataset to extract structural and appearance attributes from the training dataset; and Using the second part G2 and the structural and appearance attributes extracted from the training dataset, a 3D representation of the object is generated.

Images