Generation of volumetric representation from text

Abstract

Methods and apparatus for generating a volumetric representation from text. In one example, the method includes converting the text into a reference image using a first neural network. The method also includes generating a plurality of alpha channels for a plurality of layers of a multiplane image corresponding to the reference image using a second neural network. Each of the layers is at a respective depth determined by the second neural network based on the reference image. The method also includes generating a respective plurality of texture channels for each of the plurality of layers based on the reference image with a third neural network. Operations for generating the respective plurality of texture channels include inpainting a respective occluded area of a layer of the plurality of layers.

Description

GENERATION OF VOLUMETRIC REPRESENTATION FROM TEXT 1. Cross-Reference to Related Applications

[0001] This application claims the benefit of priority from US Provisional Application Ser. No. 63 / 637,832, filed on 23 April 2024, and EP Patent Application No. EP 24176812.6, filed on 18 May 2024, each of which is incorporated by reference herein in its entirety. 2. Field of the Disclosure

[0002] Various example embodiments generally relate to a generative artificial intelligence (AI) system and method for creating images from a text input. 3. Background

[0003] Text-to-image generation is a task in computer vision and natural language processing where the goal is to generate an image that corresponds to a given textual description. This task typically involves converting the text input into a meaningful representation and then using this representation to generate an image that matches the description. Example applications for generating photo-realistic images from text include photo-editing, computer-aided design, gaming, and the like. BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS

[0004] Disclosed herein are various embodiments for generating a volumetric representation from a text prompt. In one example, a generative model is configured to leverage some pre-existing neural networks (such as Stable Diffusion and Adaptive MPI), which are interconnected with an occlusion mapper and an inpainting neural network to convert a text prompt into a corresponding multiplane image (MPI). The generative model is trained using score distillation sampling, which does not require external training data with ground truth. In some examples, we randomly sample novel camera positions for each training iteration and compute the rendered image corresponding to the new viewpoint. This image is then encoded into a latent space, where it undergoes diffusion using a set of predefined Gaussian noise functions. Subsequently, a pretrained latent diffusion model is used to denoise the polluted latent feature map and predict the added noise. The error in the noise prediction is used as a loss function, which is minimized to find optimal parameters for thenon-locked parts of the generative model. In effect, the latent diffusion model operates to judge whether the rendered image is photorealistic and sufficiently coherent with the input text prompt.

[0005] In at least some examples, in response to a text prompt, the above-indicated generative model produces a corresponding photorealistic, coherent MPI that contains high-quality supporting data in the areas of occlusion. This MPI beneficially offers enhanced image rendering options from a broader range of viewpoints compared to those of at least some alternative methods. In some cases, further enhancements to the generative model can be obtained by adjusting the alpha channels and eliminating artifacts in or near the occlusion areas.

[0006] According to one example, a method for generating a volumetric representation from text comprises: with a first neural network, converting the text into a reference image; with a second neural network, generating a plurality of alpha channels for a plurality of layers of an MPI corresponding to the reference image, each of the layers being at a respective depth determined by the second neural network based on the reference image; and with a third neural network, generating a respective plurality of texture channels for each of the plurality of layers based on the reference image, wherein said generating the respective plurality of texture channels includes inpainting a respective occluded area of a layer of the plurality of layers.

[0007] According to another example, an apparatus for generating a volumetric representation from text comprises: at least one processor; and at least one memory including program code; and wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: with a first neural network, convert the text into a reference image; with a second neural network, generate a plurality of alpha channels for a plurality of layers of an MPI corresponding to the reference image, each of the layers being at a respective depth determined by the second neural network based on the reference image; and with a third neural network, generate a respective plurality of texture channels for each of the plurality of layers based on the reference image, wherein said generating the respective plurality of texture channels includes inpainting a respective occluded area of a layer of the plurality of layers.

[0008] According to yet another example embodiment, provided is a non-transitory computer- readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising the above method for generating a volumetric representation from text.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Other aspects, features, and benefits of various disclosed embodiments will become more fully apparent, by way of example, from the following detailed description and the accompanying drawings, in which:

[0010] FIGS. 1A-1B are block diagrams illustrating a framework for text-to-image generation according to some examples.

[0011] FIG. 2 pictorially illustrates a volumetric representation using a multiplane image (MPI) according to one example.

[0012] FIG. 3 is a block diagram illustrating a forward diffusing process and a backward denoising process of a diffusion model that can be used in the framework of FIGS. 1A-1B according to some examples.

[0013] FIG. 4 is a block diagram illustrating a latent diffusion model that can be used in the framework of FIGS. 1A-1B according to some examples.

[0014] FIG. 5 is a block diagram illustrating a U-Net that can be used in the latent diffusion model of FIG. 4 according to some examples.

[0015] FIG. 6 is a block diagram illustrating a workflow used to implement the framework of FIGS. 1A-1B according to some examples.

[0016] FIG. 7 is a block diagram illustrating a text-to-image module used in the workflow of FIG. 6 according to some examples.

[0017] FIG. 8 is a block diagram illustrating an image-to-alpha module used in the workflow of FIG. 6 according to some examples.

[0018] FIG. 9 is a block diagram illustrating a gated convolution kernel for an inpainting module used in the workflow of FIG. 6 according to some examples.

[0019] FIGS. 10A-10B include a block diagram illustrating training of the generative model used in the workflow of FIG. 6 according to some examples.

[0020] FIG. 11 is a block diagram illustrating a computing device according to various examples. DETAILED DESCRIPTION

[0021] In some cases, a multiplane image (MPI) can be used to represent a three-dimensional (3D) scene. An MPI stores fronto-parallel planes of a scene at a discretely sampled range of depths from the reference coordinate frame. Information stored in each plane includes the information on texture (e.g., in terms of the R, G, B values) and transparency (in terms of the alpha (A) channel). Each plane represents a part of the 3D scene’s content at the corresponding depth. The planes can be warped and rendered to form a novel viewpoint seen from another camera pose, e.g., to provide a more immersive viewing experience. One challenge associated with an MPI representation is how to address the occluded regions on the plane with high depth magnitudes. These occluded regions, if reconstructed inaccurately, may result in artifacts in the rendered images from at least some novel viewpoints.

[0022] In some examples, a diffusion probabilistic model can be used for the image generation task. In some cases, the diffusion probabilistic model can be adapted to provide solutions for converting text to images, thereby significantly broadening the practical use of diffusion probabilistic models. To accurately represent a 3D environment, different options, such as mesh, neural field, and MPI, can be implemented. Among these volumetric representations, the MPI representation better lends itself to efficient rendering and transmission. Since an MPI can be considered as a combination of multiple two-dimensional (2D) images, MPI approaches can benefit from the use of many conventional, well-developed 2D algorithms.

[0023] Some embodiments disclosed herein leverage the diffusion probabilistic model to generate the RGB channels for an MPI from an input text prompt. The above-mentioned occlusion problem is addressed by training an inpainting network to fill in the occluded area with generated realistic content. Some examples utilize the pre-trained Stable Diffusion model to guide accurate representation of a 3D scene. In some examples, there is no need for external training data with ground truth, such as MPIs of the corresponding 3D scene. Rather, for each input text prompt, the Score Distillation Sampling (SDS) loss can be used to evaluate and supervise the realism of the warp and rendered scene from the MPI.

[0024] FIGS. 1A-1B are block diagrams illustrating a framework 100 for text-to-image generation according to some examples. The framework 100 is conceptually divided into two parts: (i) an MPI generation part 100a and (ii) a novel-view rendering part 100b.

[0025] FIG. 1A is a block diagram illustrating the MPI generation part 100a according to one example. The MPI generation part 100a employs a generative model 110 configured to take a text prompt 102 as an input and provide ^ layers of a corresponding MPI 112 as an output. The number ^ is a hyperparameter that can be specified by the user. In some examples, the number D is set to^ = 16. In some examples, the generative model 110 includes the following components:i. A Stable Diffusion model that maps the text prompt 102 to a 2D image. ii. An MPI-generation model that produces the alpha channels from the 2D image. iii. A thresholding component that converts the alpha channels to occlusion maps. iv. An inpainting model that generates realistic content within the occluded area on the RGB channels of the MPI 112. In some examples, the MPI-generation model can be an adaptive MPI model. In some other examples, other suitable MPI-generation models can also be used.

[0026] FIG. 1B is a block diagram illustrating the novel-view rendering part 100b according to one example. The novel-view rendering part 100b applies warping and rendering operations 120 to the MPI 112 to generate a novel view 122. In the notation used in FIG. 1B, ^ denotes the canonical view which is set to be the source view. For a novel target viewpoint ^^, the MPI 112 is warped to adjust the viewing angle and to provide a rendered 2D image for the novel view 122. In the example shown, the novel view 122 corresponds to the novel target viewpoint ^^.

[0027] Some embodiments provide the following features representing significant improvements over the current state of the art: • a novel generative MPI method that enables straightforward access to 3D scene generation. In various examples, non-professional users can relatively quickly get a 3D scene representation by providing a relatively simple text prompt. • a novel way to address the occlusion problem in the generative MPI using a diffusion probabilistic model. This approach significantly improves the quality of the image rendered from the MPI and expands the available viewing angles. In at least some examples, this approach does not rely on external training data.

[0028] An MPI comprises multiple image planes, with each of the image planes being a “snapshot” of the 3D scene at a certain depth with respect to the camera position. Information stored in each plane includes the texture information (e.g., represented by the R, G, B values) and transparency information (e.g., represented by the alpha (A) values). Herein, the acronyms R, G, B stand for red, green, and blue, respectively. In some examples, the three texture components can be (Y, Cb, Cr), or (I, Ct, Cp), or another functionally similar set of values. In different examples, an MPI can be generated in a number of different ways. For example, two or more input images from two or more cameras located at different known viewpoints can be co-processed to generate a corresponding multiplane image. Alternatively, a multiplane image can be generated using a source image captured by a single camera.

[0029] FIG. 2 pictorially illustrates a 3D-scene representation using an MPI 200 according to one example. The MPI 200 has D planes or layers (P0, P1, …, P(D−1)), where D is an integer greater than one. In the example shown, the planes (layers) are indexed such that the most remote layer, from the reference camera position (RCP) s, is indexed as the 0-th layer and is at a distance (or depth) d0 from the RCP along the Z dimension of the 3D scene. The index is incremented by one for each next layer located closer to the RCP. The plane (layer) that is the closest to the RCP has the index value (D−1) and is at a distance (or depth) dD−1 from the RCP along the Z dimension. Each of the planes (P0, P1, …, P(D−1)) is orthogonal to a base plane 202 which is parallel to the XZ-coordinate plane. The RCP is at a vertical height h above the base plane 202. The XYZ triad shown in FIG. 2 indicates the general orientation of the multiplane image 200 and the planes P0, P1, …, P(D−1) with respect to the X, Y, and Z dimensions of the 3D scene. In various examples, the number D can be 32, 16, 8, or any other suitable integer greater than one.

[0030] In mathematical terms, the MPI 200 corresponding to the RCP ^ contains a set of ^RGBA pairs {(^(^^ (^^ ^^^^ , ^^ ^|^^^ }. Given the MPI 200, one can render it to obtain an imageposition that is different from the RCP s (also see FIG. 1B). An example MPI-rendering algorithm (often referred to as the “MPI viewer”) that can be used for this purpose may include the steps of warping and compositing. Other suitable MPI viewers may also be used. The rendered image (e.g., 122, FIG. 1B) can be viewed on a suitable display device.

[0031] During the warping step of the MPI-rendering algorithm, each layer (^(^^, (^^^ ^^ ^ iswarped from the source view ^ to a novel target view ^. Assuming that theof thesource and target are represented by (^^, ^^^ and (^^, ^^^, respectively, the transformation of thecolor and alpha channels can be expressed as: ^(^^^ = ^^,^(^^, ^^^ (1)The warping function ^^,^can^^^ ^ = ^(! −#$%&'^(^^^^^^^ ^ (3)where^and^are the and target position,respectively; R and t are the n is the normal vector (0,0,1*+; and a is the distance to a plane that is fronto-parallel to the source camera at the depth ,^^.

[0032] During the compositing step of the MPI-rendering algorithm, the MPI viewer operates to render a novel view (e.g., 122, FIG. 1B) from the warped color and alpha channels. In some examples, the corresponding operations are based on the following equations for composition: -(^^ = ^(^^ ∙ ∏^^^ ( (^^^ ^ 0^^1^ 1 − ^0 ^ (4)where -(^^^ representsrendered image at the^.

[0033] FIG. 3 is a block diagram illustrating a forward diffusing (7(85|85^^^) process and a backward denoising (9(85^^|85^) process of a diffusion model that can be used in the framework 100 according to some examples. Diffusion models represent a class of generative models that can generate diverse high-resolution images. Different from previous deep generative models, such as the generative adversarial networks (GANs) and variational autoencoders (VAEs), the diffusion model aims to decompose the image generation process into many small denoising steps that start from a Gaussian noise image.

[0034] Given an image :^sampled from the real data distribution 7(:^, e.g., :^~7(:^, the forward diffusion process is defined by a Markov chain that gradually adds small amounts of Gaussian noise in steps. Suppose the variance for the Gaussian noise at each step is defined by a schedule {<^, ⋯ , <+}, where <5 ∈ (0, 1^, ? = 1, ⋯ , @. Then, the posterior can be derived by thefollowing chain rule:7(:^:+|:^^ = ∏+5^^ 7^:5|:5^^^ (6)where 7^:^:+^ denotes the A diffusion can be by a Gaussian process:7^:5|:5^^^ = B^:5; D1 − <5:5^^, <52^ (7)After defining E5 = 1 − step :5 can be sampledvia reparameterization as :5~7^:5|:^^ = B^:5; DEF5:^, ^1 − EF5^2^ (8)

[0035] When @ isTherefore, the data generation can be achieved when the deep neural network 9Hmanages to estimate the reverse distribution 7^:5^^|:5^. Here, I indicates the learnable parameters used in the network. According to the sampling theory, the denoising process 7^:5^^|:5^will also be Gaussian for sufficiently small <5. As such, the denoising process sets 9Hto be Gaussian and parameterizes the mean and variance as follows: 9H^:5^^|:5^ = B^:5^^; KH^:5, ?^, ∑H^:5, ?^^ (9)In some examples,model only predicts themean KH^:5, ?^. By minimizing the Kullback-Leibler (KL) divergence between 7^:5^^|:5, :^^ and9H^:5^^|:5^, the following relationship is obtained: KH^: ^ OM5, ?^ = N:5 − QH^:5, ?^R (10)In an examplenoise Q from :5, and the loss function used for training represents the mean square error for the estimation of the noise, e.g., expressed as follows: ℒ= T5,UV ~W&5&,X~^^,Y^(‖Q − QH^:5, ?^‖^^* (11)

[0036] In some examples, for better control over the generative content, diffusion models configured to accept a conditional input may be preferred. Generating images corresponding to an input text prompt may benefit from the use of such models. Among the candidate algorithms, the Stable Diffusion model tends to achieve state-of-the-art results in terms of the image quality and sampling time.

[0037] FIG. 4 is a block diagram illustrating a latent diffusion model (LDM) 400 according to some examples. Therein, 8 denotes pixel-space variables, and [ denotes latent-space variables. TheLDM 400 is configured to use conditional inputs received from a conditioning block 406 as indicated in FIG. 4.

[0038] The LDM 400 includes a vector-quantized variational autoencoder (VQ-VAE) 410 configured to transform an image 408 from a pixel space 402 to a latent space 404. A diffusing process 420 operates to map the output, [^, of the VQ-VAE 410 to the latent variable, [+. A denoising process 428, which learns a transition from the distribution of the latent code [ to a unit normal distribution. In some examples, an implementation of the denoising process 428 is based on a denoising diffusion probabilistic model (DDPM). In the example shown, the DDPM is implemented using a sequence of denoising decoders 430 approximated by a diffusion process viavariational inference. The observed variable is [^ = [, and the latent variables [^, [^, … , [+ are usedwith the sequence of the denoising decoders 430 as indicated in FIG. 4. A decoder 440 operates to transform the [^back to the pixel space 402, thereby generating an output image 442. In some examples, the processes 420 and 428 can be implemented as described above in reference to FIG. 3.

[0039] FIG. 5 is a block diagram illustrating an architecture of a U-Net 500 that can be used in denoising decoders 430 according to some examples. In the example shown, the U-Net 500 is a neural network having a U-shaped topology that includes a down-sampling branch 502 and an up- sampling branch 504 connected by a bottleneck 506 as indicated in FIG. 5. Skip connections between the branches (502, 504) indicated by the horizontal arrows are added at different scales of the latent features. In some examples, self-attention layers are inserted between convolution layers. In some examples, implementation of the U-Net 500 may benefit from certain features described in Ronneberger, O., Fischer, P., and Brox, T., “U-net: Convolutional networks for biomedical image segmentation,” Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241, which is incorporated herein by reference in its entirety.

[0040] Referring back to FIG. 4, the denoising process 428 is configured to take the noisy latent vector ]+and the encoded conditional information generated by the conditioning block 406 as inputs. For the text to 2D image task, the conditional information refers to the text prompt 102 (see FIG. 1A) specifically. In some examples, the text prompt 102 is tokenized as a vector with fixed length. Then the resulting token vector is further encoded by an encoder 432, ^H. An encoded feature 434 generated by the encoder 432 is inserted to every layer of the U-Net 500 to guide thedenoising process 428. As indicated in FIG. 3, the denoising process 428 goes through the iterative stages 430. Therefore, the noisy latent feature ]+goes through the denoising U-Net 500 @ times to get a clean latent vector ], which is mapped to the pixel space 402 by the decoder 440, ^. It should be noted that the diffusion process 420 of the LDM 400 is only used for training.

[0041] FIG. 6 is a block diagram illustrating a workflow 600 used to implement the framework 100 according to some examples. In the example shown, the workflow 600 includes: (i) a text-to- image module 610; (ii) an image-to-alpha module 620; (iii) an alpha-to-occlusion module 630; (iv) an inpainting module 640; and (v) a loss function module 650. Each of the modules 610-650 is described in more detail below.

[0042] Briefly, the workflow 600 has the following functional features. The _ denotes the input text prompt 102, and 2^^^denotes the image generated from the text prompt by the text-to-image module 610. The image 2^^^is then used as the reference image at the source view s. Consistent with the aboveworkflow 600 refers to the alpha channel and color channels of the MPI at the source view ^ as ^^^^ ^^^^ and ^^, where 4 is the layer index. The total number of layers in the MPI is ^. The occlusion masks for each layer derived from the alpha channels are represented using the same notation format. As such, `^^^^ stands for the occlusion mask for the 456layer. For the warping and rendering operationswarping function denoted as ^^,^aoperates to shift the viewpoint from ^ to ^^. The resulting novel rendering is denoted as 2^^a^. The model is optimized using the score distillation sampling (SDS) loss ℒ . Inshown, only network of the inpainting module 640 is trained, whereas the networks of the modules 610 and 620 are pretrained and frozen. In FIG. 6, these features are indicated by the locked and unlocked lock icons, with the locked icon indicating that the corresponding model is frozen, and the unlocked icon indicating that the corresponding model is updated during training. The solid arrows mark the feed- forward process whereas the dashed arrows show the gradient path in backward propagation.

[0043] FIG. 7 is a block diagram illustrating the text-to-image module 610 used in the workflow 600 according to some examples. The module 610 operates to transform the text prompt 102 into acorresponding 2D image 2^^^. The dimension of the 2D color image 2^^^ is c × e × 3. In someexamples, the height H and the width W are c = e = 512. Asindicated above, the module610 can be implemented using a pre-trained Stable Diffusion text-to-image model that includes blocks 710-740 connected as indicated in FIG. 7.

[0044] To accommodate widely different input texts 102, _, the module 610 employs the tokenizer block 710 configured to map _ to a corresponding token vector 712 having a fixed length. This mapping process is often referred to as tokenization. The token vector 712 is further encoded by the text-encoder block 720 to obtain a latent feature ]ijGWk. The latent feature ]ijGWkcan qualitatively be understood as containing the information expressed by the text prompt 102 in the form that can be leveraged to guide the image generation via the latent diffusion module 730. As indicate above, the latent diffusion block 730 using the U-Net-based architecture illustrated in FIGS. 3-5. The input to the block 730 is a noisy latent feature ]+that follows the Gaussian distribution,e.g., ]+~B^0, l^. The guidance ]ijGWk is inserted to every layer of the U-Net 500 as explainedpreviously. The latent feature ]+is iteratively denoised, step by step, e.g., by going through the U-Net 500 @ times. In some examples, the number T is @ = 50, and there is a fixed variance schedule^<^, ⋯ , <+^ specified for each denoising step. After the @56 iteration, the latent diffusion modelto output a clean latent vector ], which can be decoded in the decoder block 740 to a realistic 2D image 2^^^that matches the text prompt 102. In some examples, a respective pre-trained model is used forof the subnetworks of the blocks 710-740, with all parameters thereof having been frozen in the preceding training phase which is not a part of the workflow 600.

[0045] FIG. 8 is a block diagram illustrating the image-to-alpha module 620 used in the workflow 600 according to some examples. The module 620 receives the 2D image 2^^^generated by the module 610 as an input and generates as outputs the Alpha channels (^^^^) and a disparityvector (m). In one example, the alpha channels have the dimension of c16, and thedisparity vector has the size of 1 × 16. The Alpha channels play an imporatnt role in the warp andrender operations 120 since they represent the opacity at each depth. The Alpha channels are also related to the disparity map for the input image.

[0046] In the module 620, the image 2^^^is processed using a depth-estimation algorithm 802 to generate a corresponding depth map 804. In some examples, the depth-estimation algorithm 802 can be the DepthDPT algorithm describe, e.g., in Ranftl, René, Alexey Bochkovskiy, and Vladlen Koltun, “Vision transformers for dense prediction,” Proceedings of the IEEE / CVF international conference on computer vision, 2021, pp. 12179-12188, which is incorporated herein by reference in its entirety. In the module 620, the depth map 804 is leveraged as an additional input channel along with the image 2^^^. In the example shown, the module 620 includes a color prediction network 810and a layer depth determination network 820. The layer depth determination network 820 operates to determine the placement of each MPI layer by finding the optimal ^^on the normalized disparity histogram. In some examples, implementation of the layer depth determination network 820 may benefit from certain features of the plane adjustment network (PAN) described in Han, Yuxuan, Ruicheng Wang, and Jiaolong Yang, “Single-view view synthesis in the wild with learned adaptive multiplane images,” ACM SIGGRAPH 2022 Conference Proceedings, Article No. 14, pp. 1-8, which is incorporated herein by reference in its entirety.

[0047] In some examples, the networks 810 and 820 may be implemented using some features of the adaptive MPI network described in Han, Yuxuan, Ruicheng Wang, and Jiaolong Yang, “Single-view view synthesis in the wild with learned adaptive multiplane images,” ACM SIGGRAPH 2022 Conference Proceedings, Article No. 14, pp. 1-8. The adaptive MPI network canbe used to acquire the alpha channels ^^^^^ , ⋯ , ^^^^^^^ and the corresponding disparity vectorindicating the layer depths (^ , ⋯ , ^ *+ for t ^^^^ ^^^ he MPI corresponding to the image 2 . In oneexample, the overall workflow of the module 620 includes: (i) computing themap 804 of the image 2^^^with the DepthDPT algorithm 802; and (ii) using both the image 2^^^and the depth map 804 as to the adaptive MPI network 810 / 820 to get an output setthe ^ alpha channels ^^^^and the disparity vector m.

[0048] back to FIG. 6, the alpha-to-occlusion module 630 receives Alpha channels (^^^^) from the module 620 as inputs and generates an occlusion mask (`^^^) of dimensionc × e × 16 as an output. One goal of the workflow 600 is to generate RGB channels that haverealistic imagery in the occluded area so that the corresponding MPI can provide better rendering results at new viewpoints. Thus, it needs to identify the regions that are potentially blocked in each layer. This can be accomplished in the module 630 by manipulating the alpha channels acquired by the module 620. Thus, the input of the module 630 includes the alpha channels acquired from the module 620, and the output includes the occlusion masks of each layer.

[0049] The alpha channel represents the opacity of every pixel in an image. In an MPI, it is particularly useful for specifying occlusion areas. For the 456layer, the ray presence is defined by: n^^^ ^^^ ^^^^ = ∏G^^ ^1 − ^ ^ (12)where 1 − ^^^^G indicates the transparency of each previous layer. The cumulative product oftransparencies represents how much light is still available for the 456layer. Here, to define “occlusion,” we use a hard threshold. In some examples, this hard threshold is selected to be 0.05.With this threshold value, if n^^^^ < 0.05, then 95% of the light has been absorbed by the previous^4 − 1^ layers, and we consider the corresponding pixel to be “occluded.” In this manner, a binaryocclusion map for the 456layer can be produced by thresholding n^^^^ as follows: 1^^^^ , n < 0.05` ^^^ = q ^ (13)≥ 0.05

[0050] The inpaintingthe image (2^^^) from the module 610 and the occlusion mask (`^^^) from the module 630. The output of the module 640 includes color channels(^^^^) of× e × 3 × 16, which correspond to the RGB channels of the MPI that theworkflow 600 is configured to generate. To fill in the occlusion area in each layer specified by `^^^^ , the module 640 employs an inpainting network 642 that needs to be trained. The network 642 to take the reference image 2^^^and the occlusion masks `^^^as inputs and produce the ^ layers of RGB channels. In onethe network 642 hasNet backbone with the vanilla convolution kernel replaced by the gated convolution kernel. The latter kernel is described in Yu, Jiahui, et al., “Free-form image inpainting with gated convolution.” Proceedings of the IEEE / CVF international conference on computer vision, 2019, pp. 4471-4480, which is incorporated herein by reference in its entirety. One reason why the vanilla convolutions are not well-fitted for the task of inpainting is that they treat all pixels equally. However, for the inpainting task, not all pixels are valid, which may cause ambiguity during training and induce visual artifacts, such as color discrepancy and blurriness, during testing. The gated convolution is a mechanism that helps the model to learn the soft mask automatically from data.

[0051] FIG. 9 is a block diagram illustrating a gated convolution kernel 900 used in the inpainting module 640 according to some examples. In the kernel 900, soft gating blocks 910 are arranged along a paralleled pathway to the pathway including independent learnable kernels 920. Hereafter, we denote the convolution kernel for soft gating as eiwhile that for the feature as es. Inone example, the size of both kernels is selected to be 3 × 3. In practice, we apply themask`^^^ on the masked RGB image 2^^^ to form a m t^^^ ^^^ ^^^^ asked RGB image 2^ = u1 − `^ v ^ 2 , where^ is the element-wise multiplication of two matrices; and 1 − `(^^^ represents the un-occludedregion of the 456 layer. The input l has the dimension of (xy?zℎ, zℎy||}4, c, e*, which is formedby concatenating 2t(^^^ and `(^^^ along the channel axis. In some examples, xy?zℎ = 16, since theinpainting is on each layer of MPI (4 = 0, ⋯ , ^ − 1^. The value of zℎy||}4 iszℎy||}4 = 5, includes three channels of 2t(^^, one channel of `(^^^ and one channel of ones.Regarding the last channel of ones, the gated convolution kernels in the above-cited Yupaper enables additional flexibility where the user can provide a map of sketches to indicate approximate structures for the inpainting results (for example, with sketches providing house-like edges will cause the inpainting network to inpaint a house-like structure in the missing regions). Considering that in the workflow 600 main supervision comes from the evaluation of the realism of the warped and rendered novel scenes and, as such, do not significantly rely on such sketch guidance, we set the corresponding plane to ones (hence, the channel of ones).

[0052] Suppose the coordinate of a pixel is represented by (8, ~^, then the convolution for thetwo pathways of the kernel 900 can be written as: ^y?^|^U,^ = ∑ ∑ ei ∙ l (14)^}y?^^}U,^ = ∑ ∑ es ∙ l (15)Then the convolution results are merged (as represented by dash lines 912, FIG. 9) as follows: ^^?9^?U,^ = ^^^(^}y?^^}U,^^ ^ ^^^^^^^(^y?^|^U,^^ (16)where ELU stands for the exponential linear unit, which serves as an activation function. In one example, the ELU is defined as follows: ^^^(8^ = ^ 8, 8 > 0− (17)where E is a constant. InThe sigmoid function used in Eq. (16) is defined as follows: ^^^^^^^(8^ = ^^1k^ (18)The sigmoid function applies a transform that confines the value of 8 to the interval (0,1*.

[0053] In various examples, implementations of the inpainting module 640 may benefit from the pertinent features of the DeepFillv2 model disclosed in the above-cited Yu paper and modifications thereof described above in reference to FIG. 9.

[0054] The loss function module 650 receives the MPI 648 composited from (i) the color channels (^^^^) generated with the inpainting module 640 and (ii) the alpha channels (^^^^) generated with the image-to-alpha module 620. The module 650 also receives vectors(^) that contain the MPI layer depth information. As an output, the module 650 the backward gradient that supervises weights for the inpainting module 640.

[0055] In some examples, ground truth data for training the model may not be available. Similar with the neural field, for each input text prompt, the inpainting network 642 needs to be trained to remember a specific scene. In some examples, the pre-trained text-to-image Stable Diffusion model can be used to compute score distillation sampling (SDS) loss ℒb^b. Additional description of the SDS loss ℒb^bis provided in Poole, Ben, et al., “Dreamfusion: Text-to-3d using 2d diffusion,” arXiv preprint arXiv:2209.14988 (2022), which is incorporated herein by reference in its entirety.

[0056] FIGS. 10A-10B include a block diagram illustrating a process 1000 of training of the generative model 1001 used in the workflow 600 according to some examples. FIG. 10B also shows an example structure of the loss function module 650 used in the workflow 600 and the process 1000. The inputs to the module 650 include the tokenized text prompt _ and the novel rendering image 2^^a^and the output will be the SDS loss gradient. Hereafter, we denote the generative model 1001 as ^Hwhere I represents the learnable parameters. Given an input textprompt, the model 1001 will produce an MPI 1048 including color channels ^^^^ ^^^^ , ⋯ , ^^^^ , alphachannels ^^^^^ , ⋯ , ^^^^^^^ , and the disparity vector (^^, ⋯ , ^^^^*+. For each iteration, we randomlysample a^^and apply the warp and composite function ^^,^ato get a corresponding novel image 2^^a^rendered from the MPI 1048. Intuitively, the SDS loss ℒb^bserves to judge whether the novel rendering image looks realistic. In other words, the loss is large when 2^^a^is far from a realistic data distribution.

[0057] The original DreamFusion is configured to use the Imagen, which is an image space diffusion model for text-to-image generation. The Imagen is described in Saharia, Chitwan, et al., “Photorealistic text-to-image diffusion models with deep language understanding,” Advances in Neural Information Processing Systems 35 (2022), pp. 36479-36494, which is incorporated herein by reference in its entirety. For our purposes, we modified the DreamFusion by replacing the Imagen with the latent diffusion model Stable Diffusion. As such, in the example shown in FIGS.10A-10B, the module 650 has a network structure that is analogous to the network structure described above in reference to FIG. 7 with the decoder 740 having been removed.

[0058] The rendered image 2^^a^is encoded to the latent space ]^^a^by an image encoder 1060. The resulting representation passes through a diffusion process 1070 with a known varianceschedule ^<^, ⋯ , <+^. As the Gaussian noise added in each diffusion step of the diffusion process1070 is independent, the sum of these noises is still Gaussian. In other words, we canregard theto be ]^a + ^, where ^~B^0, l^. Then, the latent diffusion U-Net of thediffusion process 1070 outputs an estimate of the Gaussian noise ^^after the @ steps of denoising.The error in the noise prediction ^^^ − ^^ represents the loss function. In one example, wecustomize the SDS loss gradient as follows: ∇Hℒb^b = T5,^ ^^^?^^^^ − ^^ ^i^^_^^H ^ (19)where ? is a time step value defined by:^?^ = 1 − ∏G^^ ^1 − <G^ (20)We use ^H^_^ to represent the MPI 1048 generated by the model ^H. In FIGS. 10A-10B, the dashed arrows indicate the gradient path in the backward propagation. In the example shown, the blocks 710, 720, 1060, and 1070 are frozen, and the warp and composite function ^^,^ais differentiable.

[0059] We also note is that the module 650 is only used in the training stage. In the inference stage, the model-bypass connection is removed, which disables the module 650. The trained generative model 1001 will output the MPI 1048 as described above in reference to FIGS. 6-9. The trained generative model 1001 will also output the disparity vectors (m) to indicate the layer depths in the MPI 1048, which is needed to properly configure the warp and composite function ^^,^afor novel view generation (also see FIG. 6).

[0060] In some examples, during training we randomly sample a new viewpoint ^^to create a novel rendering for each iteration. The sampling is performed within a range for the rotation angleand translation ranges for the 8, ~ and [ axes. Suppose that the coordinate for the source camera is^0,0,0^. We then set the range for translation along the 8 axis to be (−5,5*, for translation along the ~ axis to be (−2,2*, for translation along the [ axis is (−1.2,1*. The range for the rotation angle in 8is (−7°, 8°*, for the rotation angle in ~ is (−15°, 11°*, and for the rotation angle in [ is (−7°, 8°*. Tocompute the warp and composite function, the intrinsic parameters for the source and target camera^and^are also used. In some examples, we assume that the cameras are the same i.e.,^= ^. In such examples, we use the following camera intrinsic parameters: 1372.42 0 256^ = ^ = ^ 0 1372.42 256 ^ (21)0 0 1

[0061] In somemodel 1001, the workflow 600, and the training process 1000 are coded in Python 3.10.12 with PyTorch 2.0.4. To get access to the pre-trained Stable Diffusion model, the diffusers 0.17.1 are used. The model 1001 is trained on a single NvidiaA100 GPU with a 40 GB memory. The number of layers in the MPI 1048 is ^ = 16, and the imagesize is 512 × 512. The number of iterations is set to be 15,000. For the process 1000, we use theAdam optimizer with an initial learning rate of 5 × 10^ , and it is set to decay by 0.5 every 500iterations.

[0062] FIG. 11 is a block diagram illustrating a computing device 1100 according to various examples. In some examples, the computing device 1100 is configured to implement the generative model 1001. In some examples, two or more instances of the computing device 1100 can be used in the workflow 600 and / or the process 1000.

[0063] The computing device 1100 of FIG. 11 is illustrated as having a number of components, but any one or more of these components may be omitted or duplicated, as suitable for the application and setting. In some embodiments, some or all of the components included in the computing device 1100 may be attached to one or more motherboards and enclosed in a housing. In some embodiments, some of those components may be fabricated onto a single system-on-a-chip (SoC) (e.g., the SoC may include one or more electronic processing devices 1102 and one or more storage devices 1104). Additionally, in various embodiments, the computing device 1100 may not include one or more of the components illustrated in FIG. 11, but may include interface circuitry for coupling to the one or more components using any suitable interface (e.g., a Universal Serial Bus (USB) interface, a High-Definition Multimedia Interface (HDMI) interface, a Controller Area Network (CAN) interface, a Serial Peripheral Interface (SPI) interface, an Ethernet interface, a wireless interface, or any other appropriate interface). For example, the computing device 1100 may not include a display device 1110, but may include display device interface circuitry (e.g., a connector and driver circuitry) to which an external display device 1110 may be coupled.

[0064] The computing device 1100 includes a processing device 1102 (e.g., one or more processing devices). As used herein, the terms “electronic processor device” and “processing device” interchangeably refer to any device or portion of a device that processes electronic data from registers and / or memory to transform that electronic data into other electronic data that may be stored in registers and / or memory. In various embodiments, the processing device 1102 may include one or more digital signal processors (DSPs), application-specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GPUs), server processors, or any other suitable processing devices.

[0065] The computing device 1100 also includes a storage device 1104 (e.g., one or more storage devices). In various embodiments, the storage device 1104 may include one or more memory devices, such as random-access memory (RAM) devices (e.g., static RAM (SRAM) devices, magnetic RAM (MRAM) devices, dynamic RAM (DRAM) devices, resistive RAM (RRAM) devices, or conductive-bridging RAM (CBRAM) devices), hard drive-based memory devices, solid-state memory devices, networked drives, cloud drives, or any combination of memory devices. In some embodiments, the storage device 1104 may include memory that shares a die with the processing device 1102. In such an embodiment, the memory may be used as cache memory and include embedded dynamic random-access memory (eDRAM) or spin transfer torque magnetic random-access memory (STT-MRAM), for example. In some embodiments, the storage device 1104 may include non-transitory computer readable media having instructions thereon that, when executed by one or more processing devices (e.g., the processing device 1102), cause the computing device 1100 to perform any appropriate ones of the methods disclosed herein below or portions of such methods.

[0066] The computing device 1100 further includes an interface device 1106 (e.g., one or more interface devices 1106). In various embodiments, the interface device 1106 may include one or more communication chips, connectors, and / or other hardware and software to govern communications between the computing device 1100 and other computing devices. For example, the interface device 1106 may include circuitry for managing wireless communications for the transfer of data to and from the computing device 1100. The term “wireless” and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate data via modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although insome embodiments they might not. Circuitry included in the interface device 1106 for managing wireless communications may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.11 family), IEEE 802.16 standards, Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as “3GPP2”), etc.). In some embodiments, circuitry included in the interface device 1106 for managing wireless communications may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. In some embodiments, circuitry included in the interface device 1106 for managing wireless communications may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). In some embodiments, circuitry included in the interface device 1106 for managing wireless communications may operate in accordance with Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. In some embodiments, the interface device 1106 may include one or more antennas (e.g., one or more antenna arrays) configured to receive and / or transmit wireless signals.

[0067] In some embodiments, the interface device 1106 may include circuitry for managing wired communications, such as electrical, optical, or any other suitable communication protocols. For example, the interface device 1106 may include circuitry to support communications in accordance with Ethernet technologies. In some embodiments, the interface device 1106 may support both wireless and wired communication, and / or may support multiple wired communication protocols and / or multiple wireless communication protocols. For example, a first set of circuitry of the interface device 1106 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second set of circuitry of the interface device 1106 may be dedicated to longer- range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some other embodiments, a first set of circuitry of the interface device 1106 may be dedicated to wireless communications, and a second set of circuitry of the interface device 1106 may be dedicated to wired communications.

[0068] The computing device 1100 also includes battery / power circuitry 1108. In various embodiments, the battery / power circuitry 1108 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1100 to an energy source separate from the computing device 1100 (e.g., to AC line power).

[0069] The computing device 1100 also includes a display device 1110 (e.g., one or multiple individual display devices). In various embodiments, the display device 1110 may include any visual indicators, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display.

[0070] The computing device 1100 also includes additional input / output (I / O) devices 1112. In various embodiments, the I / O devices 1112 may include one or more data / signal transfer interfaces, audio I / O devices (e.g., microphones or microphone arrays, speakers, headsets, earbuds, alarms, etc.), audio codecs, video codecs, printers, sensors (e.g., thermocouples or other temperature sensors, humidity sensors, pressure sensors, vibration sensors, etc.), image capture devices (e.g., one or more cameras), human interface devices (e.g., keyboards, cursor control devices, such as a mouse, a stylus, a trackball, or a touchpad), etc.

[0071] Depending on the specific embodiment, various components of the interface devices 1106 and / or I / O devices 1112 can be configured to output suitable control signals, receive suitable control / telemetry signals, and receive and transmit data streams. In some examples, the interface devices 1106 and / or I / O devices 1112 include one or more analog-to-digital converters (ADCs) for transforming received analog signals into a digital form suitable for operations performed by the processing device 1102 and / or the storage device 1104. In some additional examples, the interface devices 1106 and / or I / O devices 1112 include one or more digital-to-analog converters (DACs) for transforming digital signals provided by the processing device 1102 and / or the storage device 1104 into an analog form suitable for being transmitted through a communication channel.

[0072] According to an example embodiment disclosed above, e.g., in the summary section and / or in reference to any one or any combination of some or all of FIGs. 1-11, provided is an apparatus for generating a volumetric representation from text, the apparatus comprising: at least one processor; and at least one memory including program code; and wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: with a first neural network, convert the text into a reference image; with a second neuralnetwork, generate a plurality of alpha channels for a plurality of layers of an MPI corresponding to the reference image, each of the layers being at a respective depth determined by the second neural network based on the reference image; and with a third neural network, generate a respective plurality of texture channels for each of the plurality of layers based on the reference image, wherein said generating the respective plurality of texture channels includes inpainting a respective occluded area of a layer of the plurality of layers.

[0073] According to another example embodiment disclosed above, e.g., in the summary section and / or in reference to any one or any combination of some or all of FIGs. 1-11, provided is a method for generating a volumetric representation from text, the method comprising: with a first neural network, converting the text into a reference image; with a second neural network, generating a plurality of alpha channels for a plurality of layers of an MPI corresponding to the reference image, each of the layers being at a respective depth determined by the second neural network based on the reference image; and with a third neural network, generating a respective plurality of texture channels for each of the plurality of layers based on the reference image, wherein said generating the respective plurality of texture channels includes inpainting a respective occluded area of a layer of the plurality of layers.

[0074] In some embodiments of the above method, the method further comprises determining the respective occluded area based on the plurality of alpha channels, wherein the determined respective occluded area is provided as an input to the third neural network together with the reference image.

[0075] In some embodiments of any of the above methods, the determining includes computing a set of occlusion masks corresponding to the plurality of layers.

[0076] In some embodiments of any of the above methods, the computing comprises: computing a pixelwise cumulative product of transparencies; and comparing said product with a threshold value.

[0077] In some embodiments of any of the above methods, the converting comprises: with a tokenizer, tokenizing the text into a token vector; with a text encoder, mapping the token vector to a latent space to obtain a corresponding latent feature; with a latent diffusion model, generating a latent vector by iteratively denoising a noisy latent feature with guidance provided by thecorresponding latent feature; and with a decoder, decoding the latent vector to obtain the reference image.

[0078] In some embodiments of any of the above methods, the latent diffusion model is implemented using a U-net.

[0079] In some embodiments of any of the above methods, the second neural network includes a layer depth determination network configured to set the respective depths based on a disparity histogram of the reference image.

[0080] In some embodiments of any of the above methods, the layer depth determination network includes a plane adjustment network.

[0081] In some embodiments of any of the above methods, said generating the plurality of alpha channels comprises: computing a depth map of the reference image; and with an MPI-generation network configured to receive the depth map and the reference image as inputs, computing a disparity vector for the MPI and the plurality of alpha channels.

[0082] In some embodiments of any of the above methods, the MPI-generation network includes an adaptive MPI network.

[0083] In some embodiments of any of the above methods, the third neural network includes a series of gated convolution kernels.

[0084] In some embodiments of any of the above methods, a gated convolution kernel of the series includes a feature convolution block and a soft gating block; and wherein the inpainting comprises merging convolution results of the feature convolution block and the soft gating block.

[0085] In some embodiments of any of the above methods, the merging comprises: applying an ELU activation function to the result of the feature convolution block; and applying a sigmoid function to the result of the soft gating block.

[0086] In some embodiments of any of the above methods, the third neural network is trained using an SDS loss function.

[0087] In some embodiments of any of the above methods, parameters of the first and second neural networks are locked when the third neural network is being trained.

[0088] In some embodiments of any of the above methods, the method further comprises generating an output image corresponding to a target viewpoint of the volumetric representation by applying corresponding warping and rendering operations to the MPI.

[0089] In some embodiments of any of the above methods, the method further comprises: with an image encoder, mapping the output image to a latent space to obtain a corresponding latent vector in a latent space; with a tokenizer, tokenizing the text into a token vector; with a text encoder, mapping the token vector to the latent space to obtain a corresponding latent feature; with a latent diffusion model, subjecting the latent vector to an iterative diffusion process with guidance provided by the corresponding latent feature to generate a predicted noise; and changing one or more parameters of the third neural network based on a difference between the predicted noise and an expected noise.

[0090] In some embodiments of any of the above methods, the method further comprises randomly selecting the target viewpoint from a range defined by a fixed range of rotation angles and a fixed translation range representing camera views.

[0091] In some embodiments of any of the above methods, the expected noise is Gaussian noise.

[0092] In some embodiments of any of the above methods, the target viewpoint is different from a viewpoint corresponding to the reference image.

[0093] A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising any one of the above methods.

[0094] With regard to the processes, systems, methods, heuristics, etc. described herein, it should be understood that, although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes could be practiced with the described steps performed in an order other than the order described herein. It further should be understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be omitted. In other words, the descriptions of processesherein are provided for the purpose of illustrating certain embodiments and should in no way be construed so as to limit the claims.

[0095] Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent upon reading the above description. The scope should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the application is capable of modification and variation.

[0096] All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by those knowledgeable in the technologies described herein unless an explicit indication to the contrary is made herein. In particular, use of the singular articles such as “a,” “the,” “said,” etc. should be read to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0097] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments incorporate more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in fewer than all features of a single disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

[0098] While this disclosure includes references to illustrative embodiments, this specification is not intended to be construed in a limiting sense. Various modifications of the described embodiments, as well as other embodiments within the scope of the disclosure, which are apparent to persons skilled in the art to which the disclosure pertains are deemed to lie within the principle and scope of the disclosure, e.g., as expressed in the following claims.

[0099] Some embodiments may be implemented as circuit-based processes, including possible implementation on a single integrated circuit.

[0100] Some embodiments can be embodied in the form of methods and apparatuses for practicing those methods. Some embodiments can also be embodied in the form of program code recorded in tangible media, such as magnetic recording media, optical recording media, solid state memory, floppy diskettes, CD-ROMs, hard drives, or any other non-transitory machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the patented invention(s). Some embodiments can also be embodied in the form of program code, for example, stored in a non- transitory machine-readable storage medium including being loaded into and / or executed by a machine, wherein, when the program code is loaded into and executed by a machine, such as a computer or a processor, the machine becomes an apparatus for practicing the patented invention(s). When implemented on a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits.

[0101] Unless explicitly stated otherwise, each numerical value and range should be interpreted as being approximate as if the word “about” or “approximately” preceded the value or range.

[0102] The use of figure numbers and / or figure reference labels in the claims is intended to identify one or more possible embodiments of the claimed subject matter in order to facilitate the interpretation of the claims. Such use is not to be construed as necessarily limiting the scope of those claims to the embodiments shown in the corresponding figures.

[0103] Although the elements in the following method claims, if any, are recited in a particular sequence with corresponding labeling, unless the claim recitations otherwise imply a particular sequence for implementing some or all of those elements, those elements are not necessarily intended to be limited to being implemented in that particular sequence.

[0104] Reference herein to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the disclosure. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor areseparate or alternative embodiments necessarily mutually exclusive of other embodiments. The same applies to the term “implementation.”

[0105] Unless otherwise specified herein, the use of the ordinal adjectives “first,” “second,” “third,” etc., to refer to an object of a plurality of like objects merely indicates that different instances of such like objects are being referred to, and is not intended to imply that the like objects so referred-to have to be in a corresponding order or sequence, either temporally, spatially, in ranking, or in any other manner.

[0106] Unless otherwise specified herein, in addition to its plain meaning, the conjunction “if” may also or alternatively be construed to mean “when” or “upon” or “in response to determining” or “in response to detecting,” which construal may depend on the corresponding specific context. For example, the phrase “if it is determined” or “if [a stated condition] is detected” may be construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event].”

[0107] Also, for purposes of this description, the terms “couple,” “coupling,” “coupled,” “connect,” “connecting,” or “connected” refer to any manner known in the art or later developed in which energy is allowed to be transferred between two or more elements, and the interposition of one or more additional elements is contemplated, although not required. Conversely, the terms “directly coupled,” “directly connected,” etc., imply the absence of such additional elements.

[0108] As used herein in reference to an element and a standard, the term compatible means that the element communicates with other elements in a manner wholly or partially specified by the standard and would be recognized by other elements as sufficiently capable of communicating with the other elements in the manner specified by the standard. The compatible element does not need to operate internally in a manner specified by the standard.

[0109] The functions of the various elements shown in the figures, including any functional blocks labeled as “processors” and / or “controllers,” may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not beconstrued to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing software, random access memory (RAM), and nonvolatile storage. Other hardware, conventional and / or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.

[0110] As used in this application, the terms “circuit,” “circuitry” may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry); (b) combinations of hardware circuits and software, such as (as applicable): (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions); and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.” This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0111] It should be appreciated by those of ordinary skill in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0112] “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” in this specification is intended to introduce some example embodiments, with additional embodiments being described in “DETAILED DESCRIPTION” and / or in reference to one or more drawings. “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0113] Various aspects of the present disclosure may be appreciated from the following Enumerated Example Embodiments (EEEs): EEE 1. A method for generating a volumetric representation from text, the method comprising: with a first neural network, converting the text into a reference image; with a second neural network, generating a plurality of alpha channels for a plurality of layers of a multiplane image (MPI) corresponding to the reference image, each of the layers being at a respective depth determined by the second neural network based on the reference image; and with a third neural network, generating a respective plurality of texture channels for each of the plurality of layers based on the reference image, wherein said generating the respective plurality of texture channels includes inpainting a respective occluded area of a layer of the plurality of layers. EEE 2. The method of EEE 1, further comprising determining the respective occluded area based on the plurality of alpha channels, wherein the determined respective occluded area is provided as an input to the third neural network together with the reference image. EEE 3. The method of EEE 2, wherein the determining includes computing a set of occlusion masks corresponding to the plurality of layers. EEE 4. The method of EEE 3, wherein the computing comprises: computing a pixelwise cumulative product of transparencies; and comparing said product with a threshold value. EEE 5. The method of any one of EEEs 1 to 4, wherein the converting comprises: with a tokenizer, tokenizing the text into a token vector;with a text encoder, mapping the token vector to a latent space to obtain a corresponding latent feature; with a latent diffusion model, generating a latent vector by iteratively denoising a noisy latent feature with guidance provided by the corresponding latent feature; and with a decoder, decoding the latent vector to obtain the reference image. EEE 6. The method of EEE 5, wherein the latent diffusion model is implemented using a U net. EEE 7. The method of any one of EEEs 1 to 6, wherein the second neural network includes a layer depth determination network configured to set the respective depths based on a disparity histogram of the reference image. EEE 8. The method of any one of EEEs 1 to 7, wherein said generating the plurality of alpha channels comprises: computing a depth map of the reference image; and with an MPI-generation network configured to receive the depth map and the reference image as inputs, computing a disparity vector for the MPI and the plurality of alpha channels. EEE 9. The method of any one of EEEs 1 to 8, wherein the third neural network includes a series of gated convolution kernels. EEE 10. The method of EEE 9, wherein a gated convolution kernel of the series includes a feature convolution block and a soft gating block; and wherein the inpainting comprises merging convolution results of the feature convolution block and the soft gating block. EEE 11. The method of EEE 10, wherein the merging comprises: applying an exponential linear unit (ELU) activation function to the result of the feature convolution block; and applying a sigmoid function to the result of the soft gating block.EEE 12. The method of EEE 9, wherein the third neural network is trained using a score distillation sampling (SDS) loss function. EEE 13. The method of EEE 12, wherein parameters of the first and second neural networks are locked when the third neural network is being trained. EEE 14. The method of any one of EEEs 1 to 13, further comprising generating an output image corresponding to a target viewpoint of the volumetric representation by applying corresponding warping and rendering operations to the MPI. EEE 15. The method of EEE 14, further comprising: with an image encoder, mapping the output image to a latent space to obtain a corresponding latent vector in a latent space; with a tokenizer, tokenizing the text into a token vector; with a text encoder, mapping the token vector to the latent space to obtain a corresponding latent feature; with a latent diffusion model, subjecting the latent vector to an iterative diffusion process with guidance provided by the corresponding latent feature to generate a predicted noise; and changing one or more parameters of the third neural network based on a difference between the predicted noise and an expected noise. EEE 16. The method of EEE 15, further comprising randomly selecting the target viewpoint from a range defined by a fixed range of rotation angles and a fixed translation range representing camera views. EEE 17. The method of EEE 15 or EEE 16, wherein the expected noise is Gaussian noise. EEE 18. The method of EEE 14, wherein the target viewpoint is different from a viewpoint corresponding to the reference image.EEE 19. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising the method of any one of EEEs 1-18. EEE 20. An apparatus for generating a volumetric representation from text, the apparatus comprising: at least one processor; and at least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: with a first neural network, convert the text into a reference image; with a second neural network, generate a plurality of alpha channels for a plurality of layers of a multiplane image (MPI) corresponding to the reference image, each of the layers being at a respective depth determined by the second neural network based on the reference image; and with a third neural network, generate a respective plurality of texture channels for each of the plurality of layers based on the reference image, wherein said generating the respective plurality of texture channels includes inpainting a respective occluded area of a layer of the plurality of layers.REFERENCES Ronneberger, O., Fischer, P., and Brox, T., “U-net: Convolutional networks for biomedical image segmentation,” Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241, 8 pages. Ranftl, René, Alexey Bochkovskiy, and Vladlen Koltun, “Vision transformers for dense prediction." Proceedings of the IEEE / CVF international conference on computer vision, 2021, 10 pages. Han, Yuxuan, Ruicheng Wang, and Jiaolong Yang, “Single-view view synthesis in the wild with learned adaptive multiplane images,” ACM SIGGRAPH 2022 Conference Proceedings, Article No. 14, pp. 1-8, 8 pages. Yu, Jiahui, et al. “Free-form image inpainting with gated convolution.” Proceedings of the IEEE / CVF international conference on computer vision, 2019, pp. 4471-4480, 10 pages. Poole, Ben, et al., “Dreamfusion: Text-to-3d using 2d diffusion,” arXiv preprint arXiv:2209.14988 (2022), 18 pages. Saharia, Chitwan, et al., “Photorealistic text-to-image diffusion models with deep language understanding,” Advances in Neural Information Processing Systems 35 (2022), pp. 36479-36494, 16 pages.

Claims

CLAIMS What is claimed is:

1. A method for generating an image from text, the method comprising: with a first neural network, converting the text into a reference image; with a second neural network, generating a plurality of alpha channels for a plurality of layers of a multiplane image (MPI) corresponding to a volumetric representation of the reference image, each of the layers being at a respective depth determined by the second neural network based on the reference image; determining a respective occluded area of each layer of the plurality of layers based on the plurality of alpha channels; with a third neural network, generating a respective plurality of texture channels for each of the plurality of layers based on the reference image, wherein said generating the respective plurality of texture channels includes inpainting the respective occluded area of each layer of the plurality of layers; and generating an output image corresponding to a target viewpoint of the volumetric representation by applying corresponding warping and rendering operations to the MPI.

2. The method of claim 1, further comprising: providing the determined respective occluded area as an input to the third neural network together with the reference image.

3. The method of claim 2, wherein the determining includes computing a set of occlusion masks corresponding to the plurality of layers.

4. The method of claim 3, wherein the computing comprises: computing a pixelwise cumulative product of transparencies; and comparing said product with a threshold value.

5. The method of any one of claims 1 to 4, wherein the converting comprises: with a tokenizer, tokenizing the text into a token vector;with a text encoder, mapping the token vector to a latent space to obtain a corresponding latent feature; with a latent diffusion model, generating a latent vector by iteratively denoising a noisy latent feature with guidance provided by the corresponding latent feature; and with a decoder, decoding the latent vector to obtain the reference image.

6. The method of claim 5, wherein the latent diffusion model is implemented using a U net.

7. The method of any one of claims 1 to 6, wherein the second neural network includes a layer depth determination network configured to set the respective depths based on a disparity histogram of the reference image.

8. The method of any one of claims 1 to 7, wherein said generating the plurality of alpha channels comprises: computing a depth map of the reference image; and with an MPI-generation network configured to receive the depth map and the reference image as inputs, computing a disparity vector for the MPI and the plurality of alpha channels.

9. The method of any one of claims 1 to 8, wherein the third neural network includes a series of gated convolution kernels.

10. The method of claim 9, wherein a gated convolution kernel of the series includes a feature convolution block and a soft gating block; and wherein the inpainting comprises merging convolution results of the feature convolution block and the soft gating block.

11. The method of claim 10, wherein the merging comprises: applying an exponential linear unit (ELU) activation function to the result of the feature convolution block; and applying a sigmoid function to the result of the soft gating block.

12. The method of claim 9, wherein the third neural network is trained using a score distillation sampling (SDS) loss function.

13. The method of claim 12, wherein parameters of the first and second neural networks are locked when the third neural network is being trained.

14. The method of any of claims 1-13, further comprising: with an image encoder, mapping the output image to a latent space to obtain a corresponding latent vector in a latent space; with a tokenizer, tokenizing the text into a token vector; with a text encoder, mapping the token vector to the latent space to obtain a corresponding latent feature; with a latent diffusion model, subjecting the latent vector to an iterative diffusion process with guidance provided by the corresponding latent feature to generate a predicted noise; and changing one or more parameters of the third neural network based on a difference between the predicted noise and an expected noise.

15. The method of claim 14, further comprising randomly selecting the target viewpoint from a range defined by a fixed range of rotation angles and a fixed translation range representing camera views.

16. The method of claim 14 or claim 15, wherein the expected noise is Gaussian noise.

17. The method of any of claims 1-16, wherein the target viewpoint is different from a viewpoint corresponding to the reference image.

18. A non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising the method of any one of claims 1-17.

20. An apparatus for generating an image from text, the apparatus comprising: at least one processor; andat least one memory including program code, wherein the at least one memory and the program code are configured to, with the at least one processor, cause the apparatus at least to: with a first neural network, convert the text into a reference image; with a second neural network, generate a plurality of alpha channels for a plurality of layers of a multiplane image (MPI) corresponding to a volumetric representation of the reference image, each of the layers being at a respective depth determined by the second neural network based on the reference image; determine a respective occluded area of each layer of the plurality of layers based on the plurality of alpha channels; with a third neural network, generate a respective plurality of texture channels for each of the plurality of layers based on the reference image, wherein said generating the respective plurality of texture channels includes inpainting the respective occluded area of each layer of the plurality of layers; and generate an output image corresponding to a target viewpoint of the volumetric representation by applying corresponding warping and rendering operations to the MPI.

Images