Multi-view 3D diffusion

Abstract

Embodiments of the present disclosure relate to multi-view 3D diffusion. An image generation system is described. The system includes a neural network model configured to perform a diffusion process to generate a set of multi-view images from the same input cues. The multi-view image set has the same topic from different view orientations. The neural network model includes a self-attention layer configured to associate pixels across the set of multi-view images.

Description

Background Art

[0001] 3D content creation is an important step in the pipeline of the modern gaming and media industries, but it is a labor-intensive task that requires trained designers to spend hours or days to create a single 3D asset. Therefore, a system that can generate 3D content by ordinary users in a simplified manner is of great value. Existing 3D generation methods can be classified into three types: traditional template-based generation pipelines, 3D generation models, and 2D lifting methods. Template-based pipelines (such as MetaHuman) are the most mature methods for generating 3D content because they focus on adjusting parameters on predefined templates. However, they are generally limited to certain types of common objects or subjects (e.g., faces), and it may be difficult to generalize their outputs to other types of objects without predefined templates. With the support of deep learning methods, researchers in academia have been training neural networks as 3D generation models, including 3D variational autoencoders, 3D generation adversarial models, and 3D diffusion models. However, due to the limited availability of 3D models and high data complexity, these models may also have difficulty generalizing their outputs to generate arbitrary objects.

[0002] 2D lifting methods have been developed to generate 3D assets using pre-trained 2D prior models. For example, the Dreamfusion and magic3D systems use 2D diffusion models for 3D generation. These models are trained on large 2D image datasets, have a good understanding of real-world objects and have excellent generalization capabilities. However, since these models only have 2D knowledge of objects, they provide single-view supervision, which often results in assets generated by these methods having multi-view Figure 1 In other words, a 3D object generated from a single view of a 2D image may be inconsistent when viewed from different directions. For example, during the score refinement process, problems caused by insufficient multi-view knowledge or insufficient 3D perception of 2D diffusion models can include the multi-faceted Janus problem, where the system tends to generate repeated content described by textual cues, as well as drifting content from different views.

[0003] Embodiments are described with respect to these and other general considerations.Furthermore, although relatively specific problems have been discussed, it should be understood that the embodiments should not be limited to solving the specific problems identified in the context. Summary of the invention

[0004] Aspects of the present disclosure relate to an image processor for image generation.

[0005] In some aspects, an image generation system is provided. The image generation system includes a neural network model configured to perform a diffusion process to generate a set of multi-view images from the same input prompt. The set of multi-view images has the same subject from different view orientations. The neural network model includes a self-attention layer configured to associate pixels across the set of multi-view images.

[0006] In some aspects, the neural network model includes a view encoder configured to generate, as input, respective view embeddings representing different view orientations.

[0007] In some aspects, each view embedding is combined with the corresponding diffusion time step as the residual of the neural network model.

[0008] In some aspects, the neural network model further comprises a criss-cross attention layer configured to receive text embeddings representing the same input prompt, wherein the text embeddings are combined with the view embeddings.

[0009] In some aspects, the view encoder is a multi-layer perceptron.

[0010] In some aspects, the neural network model is trained using a plurality of 3D image sets, each 3D image in the 3D image set having the same subject and a different view orientation.

[0011] In some aspects, the diffusion time step is shared between each 3D image within the set of 3D images.

[0012] In some aspects, the neural network model is further trained using multiple separate 2D images having different subjects from each other.

[0013] In some aspects, view embedding is omitted for multiple separate 2D images.

[0014] In some aspects, the neural network model is based on a pre-trained 2D diffusion model for transfer learning and is fine-tuned using multiple individual 2D images and multiple sets of 3D images.

[0015] In other aspects, a method for training an image processor having a neural network model is provided. The method includes generating a first training batch including a plurality of 3D image sets. Each 3D image within the 3D image set has the same subject and a different view orientation. The method also includes generating corresponding view embeddings representing the different view orientations as inputs to the neural network model. The method also includes training a neural network model of the image processor for multi-view image diffusion using the first training batch and the view embeddings.

[0016] In some aspects, training the neural network model of the image processor includes using a self-attention layer of the neural network model to associate pixels in images across the collection of 3D images.

[0017] In some aspects, training the neural network model of the image processor includes combining the respective view embeddings with the corresponding diffusion time steps as a residual of the neural network model.

[0018] In some aspects, each 3D image within the set of 3D images corresponds to the same input prompt, and training the neural network model includes: generating a text embedding representing the same input prompt, wherein the text embedding is combined with the view embedding; and providing the text embedding to a criss-cross attention layer of the neural network model.

[0019] In some aspects, generating the corresponding view embeddings includes generating the corresponding view embeddings using a multi-layer perceptron.

[0020] In some aspects, generating a first training batch including a plurality of 3D image sets includes: for each 3D image set, generating 3D images such that their view orientations have the same elevation angle at evenly distributed azimuth angles.

[0021] In some aspects, generating the first training batch further comprises generating a plurality of separate 2D images having different subjects from one another.

[0022] In some aspects, training the neural network model further includes: pre-training the neural network model as a 2D diffusion model for transfer learning; and fine-tuning the neural network model using the plurality of 3D image sets and the plurality of individual 2D images.

[0023] In some aspects, fine-tuning the neural network model includes: receiving a plurality of identity text / image pairs of a subject; and fine-tuning parameters of the neural network model using a parameter-preserving loss.

[0024] In some aspects, training the neural network model includes sharing diffusion time steps between each 3D image in the collection of 3D images.

[0025] This summary is intended to introduce a series of concepts in a simplified form that are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0027] Figure 1 A block diagram illustrating an example of an image generation system according to example embodiments is shown.

[0028] Figure 2A An example self-attention pattern according to the prior art is shown.

[0029] Figure 2B , Figure 2C and Figure 2D An example image showing the multi-view problem when using the 2D lifting method.

[0030] Figure 3 A method for multi-viewing according to an example embodiment is shown. Figure 3 Block diagram of an example image processor for D diffusion.

[0031] Figure 4 An example self-attention pattern for 3D cross-frame self-attention according to an example embodiment is shown.

[0032] Figure 5 A flowchart of an example method for training an image processor having a neural network model according to an example embodiment is shown.

[0033] Figure 6 is a block diagram illustrating example physical components of a computing device that can be used to practice aspects of the present disclosure.

[0034] Figure 7 and Figure 8 is a simplified block diagram of a computing device that can be used to practice aspects of the present disclosure.

[0035] Fig. 9 A block diagram is shown of an example process for fine-tuning a neural network model according to an example embodiment. DETAILED DESCRIPTION

[0036] In the following detailed description, reference is made to the accompanying drawings that form a part of this description, in which specific embodiments or examples are shown by way of illustration. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the present disclosure. The embodiments may be put into practice as methods, systems, or devices. Therefore, the embodiments may be implemented in hardware, completely in software, or in the form of an implementation combining software and hardware. Therefore, the following detailed description should not be considered restrictive, and the scope of the present disclosure is defined by the appended claims and their equivalents.

[0037] The present disclosure describes various examples of an image generation system having a neural network model and a method for training the image generation system. The image generation system uses the neural network model to generate an output image, such as a 3D image or even a 3D model. In general, the image generation system provides a method for improving multi-viewing. Figure 1More specifically, the image generation system generates multi-view geometrically consistent images from a given text or single-view image input. In various aspects, instead of directly using a pre-trained 2D diffusion prior for 3D generation, the diffusion model can be converted into a multi-view diffusion model, where the diffusion model learns the joint distribution of multiple views of the same subject. Advantageously, the diffusion model inherits both the generality of the 2D prior and the multi-view distribution learned from the rendered data. Figure 1 The invention relates to a method for improving the stability of a multi-view image by applying the diffusion model to 2D lifting compared to a conventional 2D lifting method. An example image generation system may include a neural network model configured to perform a diffusion process to generate a set of multi-view images from the same input prompt. The set of multi-view images has the same subject from different view orientations (e.g., front view orientation, side view orientation, etc.). The neural network model includes a self-attention layer configured to associate pixels in the set of multi-view images.

[0038] This and many other embodiments of a computing device are described herein. For example, Figure 1 A block diagram of an example of an image generation system 100 according to an example embodiment is shown. In general, the image generation system 100 is configured to simultaneously generate a set of multi-view images that are consistent with each other. In some examples, the image generation system 100 uses a similar architectural design of a diffusion model, but it is adapted for multi-view image generation. This allows transfer learning using a pre-trained diffusion model, and the resulting multi-view diffusion model can inherit the versatility of the diffusion model. In some examples, the pre-trained diffusion model is a 2D diffusion model.

[0039] System 100 includes a computing device 110 configured to train a neural network model, such as neural network model 118 of image processor 112, or neural network model 128, using source images 130. Computing device 110 includes image processor 112 configured to process images to train neural network model 118. In some examples, system 100 may also include a data repository 120 communicatively coupled to computing device 110 via network 140. In some examples, computing device 110 includes a first neural network model 118 for diffusion model and a second neural network model (not shown) for 3D model generation, as described below.

[0040] Typically, source image 130 is an image representing an object or group of objects, a person or group of people, or other suitable subject matter. In some examples, part or all of source image 130 is generated by a 2D or 3D rendering module, and a snapshot or screen shot is captured based on the output of the rendering module or the content on a display (e.g., a computer monitor or a smart phone screen). In other examples, source image 130 is an image captured by a digital camera, a digital image processor, an image capture module (e.g., of a webcam or smart phone), or other suitable image capture device.

[0041] Source images 130 include at least some images that form different sets of multi-view images with the same subject from different view orientations. For example, a first set of multi-view images may have a bulldog wearing a pirate hat as a subject, and the multi-view image may have four images with a front view orientation, a left view orientation, a right view orientation, and a rear view orientation, respectively. A second set of multi-view images may have a different subject (e.g., a red sports car) and four images with a front view orientation, a left view orientation, a right view orientation, and a rear view orientation.

[0042] In the examples described herein, each multi-view image set typically has one subject, four images, and four view orientations, where each image has a different view orientation. In addition, each image is cropped or processed to typically place the subject in the center of the image. In other examples, several view image sets may have different numbers of subjects (e.g., two or more), different numbers of images (e.g., two, three, five, six, etc.), different numbers of view orientations (e.g., two view orientations, three view orientations, etc.). In some examples, a first image set may have four images of a first subject, a second image set may have seven images of a second subject, and so on. In other examples, different image sets may have different view orientations. In addition, different view orientations may have any suitable combination of coordinates (e.g., x, y, z coordinates) and orientations (e.g., polar angles θ and azimuth angles φ) in 3D space.

[0043] Computing device 110 may be any type of computing device, including a smartphone, a mobile computer, or a mobile computing device (e.g., devices, laptops, notebooks, tablets (such as Apple iPad TM), netbook, etc.), or a fixed computing device such as a desktop computer or PC (personal computer). Computing device 110 can be configured to communicate with a social media platform, a cloud processing provider, a software-as-a-service provider, or other suitable entity, for example using social media software and a suitable communication network. Computing device 110 can be configured to execute one or more software applications (or "apps") and / or services and / or manage hardware resources (e.g., processor, memory, etc.) that can be used by a user of computing device 110.

[0044] exist Figure 1 In the example shown in , the image processor 112 includes a text encoder 116 and a view encoder 114. In other examples, the text encoder 116 and the view encoder 114 may be formed as a combined processor. In some examples, at least some portions of the text encoder 116 and the view encoder 114 may be combined with a neural network model 118, for example, by including a neural network processor or other suitable processor configured to implement a neural network model. In other words, the neural network model 118 may be integrated with the text encoder 116 and the view encoder 114 and implemented together with or as a neural network processor. In some examples, the neural network model 118 is omitted from the computing device 110, and a neural network model 128 is used instead.

[0045] The neural network model 118 is trained using the image processor 112 and is configured to process input prompts to provide a multi-view image set of a subject. In some examples, the neural network model 118 is implemented at least in part using a variational autoencoder. The neural network model 128 is generally similar to the neural network model 118, but is stored at a location remote from the computing device 110 (e.g., stored at the data repository 120). Typically, the input prompt is a text input describing the desired subject or condition of the generated image. The text input can be in a natural language format. In other words, the prompt can be written in a conversational manner, which is easy for users (e.g., ordinary users of social media platforms) to understand even without special training of computers or neural network models. In some examples, the prompt can combine natural language with other information in a suitable text data format (such as a text-based element vector). Typically, the text encoder 116 is configured to encode the text input into a vector (e.g., a text embedding) as an input to the neural network model 118 (e.g., an input to a cross-attention layer). Similarly, the view encoder 114 is configured to encode the view orientation into a vector or matrix, for example, as an input to the neural network model 118.

[0046] The image processor 112 can be similar to stable diffusion, DALL-E, Imagen, Midjourney, or other suitable text-to-image processors, with additional features as described herein. In the examples described herein, the image processor 112 is a diffusion model with a noise processor configured to iteratively denoise a noisy image through several layers. Image denoising is based on a hint to adjust the noise processor, as described below.

[0047] The computing device 100 may also include an object modeler 119, which may be configured to generate a suitable 3D object or model based on the set of multi-view images, as described below.

[0048] Data repository 120 may include one or more storage mechanisms of any type, including disks (e.g., in a hard drive), optical disks (e.g., in an optical drive), magnetic tapes (e.g., in a tape drive), memory devices (such as RAM devices, ROM devices, etc.), and / or any other suitable type of storage medium. For example, data repository 120 may store neural network model 128 and / or source images 130 (e.g., images used to train neural network model 118 and / or 128). In some examples, data repository 120 provides source images 130 to image processor 112 for use in training neural network model 118 and / or neural network model 128. In some examples, one or more data repositories 120 may be co-located (e.g., housed in one or more nearby buildings along with associated components (such as backup power, redundant data communications, environmental controls, etc.) to form a data center, or may be arranged in other ways. Thus, in one embodiment, one or more data repositories 120 may be a data center in a distributed set of data centers.

[0049] The network 140 may include one or more networks (such as a local area network (LAN), a wide area network (WAN), an enterprise network, the Internet, etc.), and may include one or more wired and / or wireless portions. The computing device 110 and the data repository 120 may include at least one wired or wireless network interface that enables communication with each other (or with intermediate devices, such as a web server or a database server) via the network 140. Examples of such network interfaces include, but are not limited to, an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth® interface, a cellular network ... TM An interface, or a near field communication (NFC) interface. Examples of network 140 include a local area network (LAN), a wide area network (WAN), a personal area network (PAN), the Internet, and / or any combination thereof.

[0050] Figure 2A An example self-attention pattern 202 according to the prior art is shown. Figure 2A In the example, the first image 210 and the second image 212 are processed by a prior art 2D diffusion model with a self-attention layer that is trained only for a single image. In other words, the training does not take into account that features of a subject may appear in multiple images with different viewpoint orientations.

[0051] Figure 2B , Figure 2C and Figure 2D An example image of a multi-view problem using the 2D lifting method is shown. Figure 2B and Figure 2C In , five images are provided that illustrate the multifaceted Janus problem, where the system tends to generate repetitive content described by the text prompt. For example, Figure 2B and Figure 2C Different views of the eagle are shown, but the eagle's beak 220 is repeated in some images (i.e., in the third image and the fourth image). Figure 2D , five images showing drifting content from different views are provided, where the system tends to generate a portion 250 of a subject (i.e., the edge of a waffle) that drifts from one subject (a waffle) to another subject 260 (i.e., a piece of fried chicken).

[0052] The multi-faceted problem can arise for a variety of reasons. For some objects (such as a razor blade), the object may be barely visible from certain view orientations (e.g., looking at the edge of a razor blade). As another example, an animal as an object may have a portion (e.g., a long tail) that is self-occluded by the animal's body and not visible from some angles. When a person evaluates these objects, the person may observe the object from different angles in a joint manner, remembering the angles seen previously, but the 2D diffusion model does not coordinate the different angles, so it may tend to generate repetitive content. To overcome this lack of multi-view awareness of objects, the image generation system 100 is configured to provide self-attention to multiple views of the same object, as described herein.

[0053] Figure 3 FIG. 1 is a block diagram showing an example of an image processor 300 according to an example embodiment. The image processor 300 generally corresponds to the image processor 112 and generates an image with improved multi-view. Figure 1The image processor 300 is configured to generate an output image set 304 based on a suitable prompt 342. As described above, in some examples, the image processor 300 may correspond to a stable diffusion neural network model that performs a denoising process based on the prompt 342 to generate an output image set 304. Typically, the prompt 342 may be a title or sentence of an image, a text-based description of an image (or image group), or other suitable text prompts for image generation. The prompt 342 may be a text field, a string, or other characters that combines natural language with other information in a suitable text data format (such as a text-based element vector). Typically, the prompt 342 is used to adjust the image processor 300, thereby affecting the output image 304. The prompt 342 may represent the desired image style and / or theme of the output image 304.

[0054] exist Figure 3 In the example shown in , the image processor 300 is based on the UNet text-to-image diffusion model, but is adapted to allow a multi-view image set 302 to be provided as a batch during training and to change the self-attention layer from 2D to 3D to allow connections between different view orientations of images within the same batch. In addition, the image processor 300 generates a view embedding for encoding different view orientations of images within the batch. Typically, the image processor 300 includes an encoder 302, a noise processor 310, a view encoder 314, a text encoder 316, an embedding processor 306, and a decoder 322. The encoder 302 and the decoder 322 can operate as a variational autoencoder. For example, the encoder 302 can generate a noisy image 312 by introducing noise to the input image. The introduced noise can be Gaussian noise, pseudo-random noise, or other suitable noise.

[0055] The noise processor 310 includes a plurality of processing layers 317, 318, 319 that iteratively denoise the noisy image 312 to generate a denoised image 315. The decoder 322 processes the denoised image 315 to generate an output image 304. In some examples, the noise processor 310 is implemented as a UNet model, further adapted as described herein. Typically, the noise processor 310 attempts to predict the noise introduced into the input image 302, and the noise processor 310 is trained using a suitable loss function (e.g., L2 reconstruction loss). In some examples, the decoder 322 includes an upsampling processor or a neural network model that is configured to generate a high-resolution image (e.g., 512x512 or larger) from a low-resolution image (e.g., 64x64 or smaller).

[0056] As described above, the noise processor 310 includes a plurality of processing layers (or blocks) 317, 318, and 319 that iteratively denoise the noisy image 312 to generate a denoised image 315. The processing layers may include a plurality of instances of a residual layer 317, a self-attention layer 318, and a cross-attention layer 319 arranged in an interlaced pattern. Although not described in detail for clarity, Figure 3 Only three layers are shown in FIG. 3 , but the noise processor 310 may have more layers. For example, the noise processor 310 may have more types of layers (e.g., normalization layers) or more instances of residual layers, attention layers, etc.

[0057] See also Figure 4 As an example, the self-attention layer 318 of the noise processor 310 can be trained to have a self-attention pattern 400 for 3D cross-frame self-attention. More specifically, the self-attention layer 318 models the long-range dependencies between different pixels in an image set, where pixels of features from different images (with different view orientations) within the image set can be connected to each other.

[0058] The view encoder 314 is configured to generate view embeddings as input to the noise processor 310, representing different view orientations within the image set. In some examples, the view encoder 314 is implemented as a two-layer multi-perceptron. To provide the view embeddings to the noise processor 310, the embedding processor 306 can add each view embedding to the corresponding time embedding (i.e., the time step 320 for diffusion) as a residual, or alternatively, the embedding processor 306 can append the view embedding to the text embedding (i.e., the text encoder 316 from the prompt 342) for the cross-attention layer 319. In the diffusion model, the noise level is controlled by the time step t, and in the case of the noise processor 310, the same time step t is shared between the multi-view images in the image set. In some scenarios where the view orientation is less entangled with the prompt 342, it may be preferable to combine the view embedding with the time step. Advantageously, by adding the view embedding to the time step, the view embedding can be easily omitted for processing 2D images without view information, and the noise processor 310 can use the same weights for individual 2D images and multi-view image sets.

[0059] The text encoder 316 is configured to process the prompt 342 to generate a coding prompt or text embedding (not shown), which adjusts at least some processing layers of the noise processor 310. In some examples, the text encoder 316 is a pre-trained transformer language model that transforms the text prompt into a coding prompt (e.g., an embedding space). In one such example, the text encoder 316 is a CLIP ViT-L / 14 text encoder that generates the coding prompt.

[0060] The training of the image processor 300 may be performed using only a set of multi-view images having the same subject, or using a set of multi-view images having the same subject and a separate 2D image. In some examples, the set of images and the separate 2D images are used to allow transfer learning. In one example, the image processor 310 is based on a pre-trained 2D diffusion model for transfer learning, and is fine-tuned using a training batch having a plurality of separate 2D images and a plurality of sets of 3D images. As an example, a training batch may have 70% of its images from a set of 3D images, and 30% of its images may be from 2D images.

[0061] By rendering a set of multi-view images for each subject from a real 3D dataset (such as Objaverse), the computing device 110 can generate a set of 3D images suitable for training the image processor 112 (image processor 300). Advantageously, rendering the set of 3D images from a known model allows for known truth values ​​for view orientation and consistent scenes (i.e., not animated or moving subjects). Relevant factors for generating training batches include the choice of view orientation, the number of images, the image resolution, and joint training with an original text-to-image dataset (e.g., a dataset from the Large Scale Artificial Intelligence Open Network). By jointly training the image processor 112 on multi-view images and images from the text-to-image dataset, the image processor 112 can achieve improved consistency and generality of the output image set 304.

[0062] Typically, a multi-view image set (e.g., input image 302) is generated using different view orientations uniformly distributed at the same elevation angle. The elevation angle can be randomly or pseudo-randomly selected between 0 and 30 degrees. In some examples, adding elevation angles outside this range will reduce the generation quality. Relatively more view orientations will also increase the difficulty of convergence, so the multi-view image set 302 includes four view orientations. In other examples, more or fewer view orientations can be used, for example, depending on the processing resources available for training.

[0063] In some examples, image processor 300 is configured to generate multiple views of the same scene from different angles. Figure 1 The image processor 300 can generate multiple images of an object based on the text prompt, and these images can be used to generate a suitable 3D asset or model. In some examples, the 3D generation is performed using a multi-view score distillation process. In other examples, the 3D generation is performed using a suitable 3D reconstruction technique. For example, the score distillation and / or 3D reconstruction can be performed by the object modeler 119.

[0064] Fractional distillation can be used to promote 2D priors into 3D content. For example, in dreamfusion, a random NeRF (neural radiance field) is initialized to represent 3D objects. At each step during optimization, a random view is rendered from NeRF and supervised by a 2D image diffusion model via fractional distillation sampling (SDS). Specifically, random noise is added to the rendered image. Given a noisy image, the diffusion model predicts noise, and its difference from the true noise is used as the gradient of the rendered image. This method can be thought of as an approximation of minimizing the KL divergence (Kullback-Leibler, a generalization of squared distance) between two distributions: the first distribution comes from NeRF, and the second distribution comes from the diffusion prior. However, this approach supervises text-image correspondences for a single view at a time and therefore does not have suitable multi-view. Figure 1 Even if the view orientation is explicitly mentioned in the text input (e.g., “front view”, “back view”), multi-view Figure 1 Consistency is also an issue.

[0065] The object modeler 119 can be configured to generate four views of a given NeRF from a 4-view set of fake multi-view images at each training step. These four images are then provided to a multi-view diffusion model (e.g., image processor 300) for score distillation. In this way, the image processor 300 jointly supervises the text-image correspondence and multi-view Figure 1 The consistency is improved to produce more coherent 3D content. In a similar way to the training phase, during SDS, the same time step t is shared between different views. The ground-truth view orientation used for rendering is input into the view encoder 314 to generate a suitable view embedding.

[0066] In some examples, the multi-view diffusion of the image processor 300 can be conditioned on the input image. In addition, a multi-view image set conditioned on the image can be used for 3D generation. The image-conditioned multi-view diffusion can be achieved by replacing the prompt 342 of the image processor 300 with the image condition. In some examples, the image condition is provided using the image embedding in the cross-attention layer 319. Alternatively, in other examples, the image is added to the input space together with the noise image (e.g., the noise image 312). Note that the image condition here can be a single input image or a multi-view image set. Therefore, two different pipelines can be used to process the image condition. The first pipeline for a single-view image can include a single concept image generated with a suitable image diffusion model (e.g., stable diffusion), and the concept image is used as an input to the image processor 300 without view embedding. For the second pipeline for multi-view images, a multi-view concept image is generated using the image processor 300 with the prompt 342, and the multi-view concept image is further used as the input of the image processor 300 with view embedding.

[0067] Figure 4 An example self-attention pattern 400 for 3D cross-frame self-attention according to an embodiment is shown. As described above, the self-attention layer 318 of the noise processor 310 can be trained to have the self-attention pattern 400 for 3D cross-frame self-attention. Figure 4 As shown in , the self-attention layer 318 can associate pixels of a first image 410 of a multi-view image set with pixels of a second image 412 of the multi-view image set. Thus, after training with multi-view images of the same subject, the self-attention layer 318 can provide improved consistency between multi-view images, as described above.

[0068] Figure 5 1 shows a flow chart of an example method 500 for training an image processor having a neural network model according to an example embodiment. Unless otherwise indicated, the technical processes shown in these figures will be performed automatically. In any given embodiment, some steps of the process may be repeated, possibly with different parameters or data. The steps in the embodiment may also be performed in the same manner as in the example embodiment. Figure 5 The steps of method 500 may be performed in a different order from the top-to-bottom order set forth in the flowchart. The steps may be performed sequentially, in a partially overlapping manner, or completely in parallel. Therefore, the order in which the steps of method 500 are performed may vary from one execution of the process to another execution of the process. Steps may also be omitted, combined, renamed, regrouped, performed on one or more machines, or otherwise deviate from the illustrated flow, provided that the performed process is operable and complies with at least one claim. Figure 5The steps of may be performed by computing device 120 (eg, via image processor 112 and / or neural network model 118), image processor 300, or other suitable computing devices.

[0069] Method 500 begins at step 502. At step 502, a first training batch is generated that includes a set of 3D images. Each 3D image in the set of 3D images has the same subject and a different view orientation. In one example, computing device 110 generates the first training batch to include Figure 3 The multi-view image set 302 shown in FIG. 3 and additional sets (not shown for clarity). Figure 3 As shown in , the subject of the image set 302 is a bulldog wearing a pirate hat, and the multi-view image has four images, and the four images have a front view orientation, a left view orientation, a right view orientation, and a rear view orientation. In some examples, the first training batch may also include multiple individual 2D images with different subjects from each other.

[0070] At step 504, corresponding view embeddings are generated as inputs for the neural network model. The corresponding view embeddings represent different view orientations. For example, the view encoder 314 generates a view embedding for each image in the multi-view image set 302. In some examples, generating the corresponding view embeddings is performed using a multi-layer perceptron.

[0071] At step 506, a neural network model of an image processor is trained using the first training batch and the view embeddings for multi-view image diffusion. For example, one or more of the noise processor 310, the encoder 302, and the decoder 322 may be trained using the first training batch. In some examples, step 506 includes using a self-attention layer of the neural network model to associate pixels in images within the set of 3D images. For example, the self-attention layer 318 of the noise processor 310 may associate pixels in images within the set of images 302. Typically, when training the neural network model, diffusion time steps are shared between each 3D image within the set of 3D images.

[0072] Step 506 may also include combining the corresponding view embeddings with the corresponding diffusion time steps as the residual of the neural network model. For example, the view embeddings from the view encoder 314 may be combined with the time steps 320 by the embedding processor 306 and provided to the noise processor 310 (e.g., to the cross-attention layer).

[0073] In some examples, each 3D image in the set of 3D images corresponds to the same input prompt. In other words, the multi-view image set 302 can be generated using the same instance of the prompt 342 so that each image has the same subject, where the prompt can be a description of the subject. Therefore, training the neural network model can include generating a text embedding representing the same input prompt and providing the text embedding to a cross-attention layer of the neural network model, where the text embedding is combined with the view embedding. For example, when the computing device 110 generates a set of 3D images suitable for training the image processor 112 by rendering a multi-view image set for each subject from a separate 3D data set, a description of the subject from the 3D data set can be used as the prompt 342. For each set of 3D images, 3D images with view orientations having the same elevation angle with a uniform distribution in azimuth can be generated.

[0074] In some examples, method 500 also includes pre-training the neural network model as a 2D diffusion model for transfer learning. For example, the image processor 300 can be trained as a 2D diffusion model using a 2D prior. In addition, training the neural network model can also include fine-tuning the neural network model using multiple 3D image sets and multiple individual 2D images. In some examples, fine-tuning includes receiving multiple identity text / image pairs of a subject and fine-tuning parameters of the neural network model using a parameter preservation loss (described below).

[0075] In some examples, method 500 also includes generating a second training batch similar to the first training batch and fine-tuning image processor 300 using the second training batch.

[0076] Figure 6 and Figure 7 and the associated descriptions provide a discussion of a variety of operating environments in which aspects of the present disclosure may be practiced. Figure 6 and Figure 7 The devices and systems illustrated and discussed are for purposes of example and explanation only and are not intended to limit the vast array of computing device configurations that may be used to practice aspects of the present disclosure, as described herein.

[0077] Figure 6is a block diagram illustrating the physical components (e.g., hardware) of a computing device 600, with which aspects of the present disclosure may be practiced. The computing device components described below may have computer-executable instructions for implementing an image processor application 620 on a computing device (e.g., computing device 110 or 120), including computer-executable instructions of the image processor application 620 that may be executed to implement the methods disclosed herein. In a basic configuration, the computing device 600 may include at least one processing unit 602 and a system memory 604. Depending on the configuration and type of the computing device, the system memory 604 may include, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of these memories. The system memory 604 may include an operating system 605 and one or more program modules 606 suitable for running the image processor application 620, such as those related to image processing. Figure 1 and one or more components of FIG. 2 , in particular an image processor 621 (e.g., corresponding to image processor 112 or image processor 300), a text encoder 622 (e.g., corresponding to text encoder 116 or text encoder 316), a view encoder 623 (e.g., corresponding to view encoder 114 or view encoder 314), and a neural network model 624 (e.g., corresponding to neural network model 118 or neural network model 128).

[0078] For example, operating system 605 may be adapted to control the operation of computing device 600. In addition, examples of the present disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application program, and are not limited to any particular application or system. Figure 6 608. The computing device 600 may have additional features or functionality. For example, the computing device 600 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be used to store data. Figure 6 This is illustrated by a removable storage device 609 and a non-removable storage device 610.

[0079] As stated above, several program modules and data files may be stored in system memory 604. When executed on processing unit 602, program module 606 (e.g., image processor application 620) may perform processing including, but not limited to, the various aspects described herein. Other program modules that may be used in accordance with various aspects of the present disclosure (particularly program modules for training image processors) may include image processor 621, hint processor 622, enhancement processor 623, and / or neural network model 624.

[0080] In addition, examples of the present disclosure may be practiced in circuits including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits using microprocessors, or on a single chip containing electronic components or microprocessors. For example, aspects of the present disclosure may be practiced via a system on a chip (SOC), wherein Figure 6 Each or many components illustrated in the figure can be integrated into a single integrated circuit. Such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functionalities, all of which are integrated (or "burned") into a chip substrate as a single integrated circuit. When operated via the SOC, the functionality described herein regarding the ability to switch protocols for the client can be operated via application-specific logic integrated with other components of the computing device 700 on a single integrated circuit (chip). The examples of the present disclosure may also be practiced using other technologies capable of performing logical operations (e.g., such as AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, the examples of the present disclosure may be practiced in a general-purpose computer or any other circuit or system.

[0081] The computing device 600 may also have one or more input devices 612, such as a keyboard, a mouse, a pen, an audio or voice input device, a touch or sliding input device, etc. Output device(s) 614 may also be included, such as a display, a speaker, a printer, etc. The aforementioned devices are examples, and other devices may be used. The computing device 600 may include one or more communication connections 616, thereby allowing communication with other computing devices 650. Examples of suitable communication connections 616 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuitry; universal serial bus (USB), parallel and / or serial ports.

[0082] The term computer-readable medium used herein may include computer storage media. Computer storage media may include volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, or program modules. System memory 604, removable storage device 609, and non-removable storage device 610 are all examples of computer storage media (e.g., memory storage devices). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage device, cassette, tape, disk storage device or other magnetic storage device, or any other product that can be used to store information and can be accessed by computing device 600. Any such computer storage medium may be part of computing device 600. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0083] Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or changed in a manner that encodes information in the signal. By way of example and not limitation, communication media may include wired media such as a wired network or direct-wired connection and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0084] Figure 7 A mobile computing device 700 is illustrated, for example, a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a notebook computer, etc., with which examples of the present disclosure may be practiced. In some aspects, the client may be a mobile computing device. Figure 7, illustrates one aspect of a mobile computing device 700 for implementing these aspects. In a basic configuration, the mobile computing device 700 is a handheld computer with input elements and output elements. The mobile computing device 700 typically includes a display 705 and one or more input buttons 710, which allow a user to enter information into the mobile computing device 700. The display 705 of the mobile computing device 700 can also be used as an input device (e.g., a touch screen display). If included, the optional side input element 715 allows more user input. The side input element 715 can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, the mobile computing device 700 can include more or fewer input elements. For example, in some examples, the display 705 may not be a touch screen. In yet another alternative example, the mobile computing device 700 is a portable telephone system such as a cellular phone. The mobile computing device 700 may include a front camera 730. The mobile computing device 700 may also include an optional keyboard 735. The optional keyboard 735 may be a physical keyboard or a "soft" keyboard generated on the touch screen display. In various examples, output elements include a display 705 for displaying a graphical user interface (GUI), a visual indicator 720 (e.g., a light emitting diode), and / or an audio transducer 725 (e.g., a speaker). In some aspects, the mobile computing device 700 incorporates a vibration sensor for providing tactile feedback to the user. In another aspect, the mobile computing device 700 incorporates input and / or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port) for sending signals to or receiving signals from external devices.

[0085] Figure 8 802 is a block diagram illustrating an architecture of one aspect of a mobile computing device. That is, mobile computing device 700 can be combined with system (e.g., architecture) 802 to implement some aspects. In one example, system 802 is implemented as a "smart phone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, game, and media client / player). In some aspects, system 802 is integrated into a computing device, such as an integrated personal digital assistant (PDA) and a wireless phone. System 802 may include a display 805 (similar to display 705), such as a touch screen display or other suitable user interface. System 802 may also include an optional keyboard 835 (similar to keyboard 735) and one or more peripheral device ports 830, such as input and / or output ports for audio, video, control signals, or other suitable signals.

[0086] In some examples, the system 802 may include a processor 860 coupled to a memory 862. The system 802 may also include a special purpose processor 861 such as a neural network processor. One or more application programs 866 may be loaded into the memory 862 and run on or in association with the operating system 864. Examples of application programs include a phone dialer, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. The system 802 also includes a non-volatile storage area 868 within the memory 862. The non-volatile storage area 868 may be used to store persistent information that is not lost when the system 802 is powered off. The application 866 may use and store information in the non-volatile storage area 868, such as email or other messages used by the email application. A synchronization application (not shown) also resides on the system 802 and is programmed to interact with a corresponding synchronization application residing on a host computer to synchronize the information stored in the non-volatile storage area 868 with the corresponding information stored on the host computer.

[0087] System 802 has a power supply 870 that can be implemented as one or more batteries. Power supply 870 can also include an external power source, such as an AC adapter or a powered docking station to replenish or recharge the batteries.

[0088] The system 802 may also include a radio interface layer 872 that performs the functions of transmitting and receiving radio frequency communications. The radio interface layer 872 facilitates wireless connections between the system 802 and the "outside world" via a communications carrier or service provider. Transmissions to and from the radio interface layer 872 are performed under the control of the operating system 864. In other words, communications received by the radio interface layer 872 may be propagated to the application 866 via the operating system 864, and vice versa.

[0089] Visual indicator 820 can be used to provide visual notification, and / or audio interface 874 can be used to communicate via audio transducer 725 (eg, Figure 7825) produces an audible notification. In the illustrated example, the visual indicator 820 is a light emitting diode (LED), and the audio transducer 725 can be a speaker. These devices can be directly coupled to the power supply 870 so that when activated, even if the processor 860 and other components may be turned off to save battery power, they will remain turned on for the duration specified by the notification mechanism. The LED can be programmed to remain turned on indefinitely until the user takes action to indicate the power-on state of the device. The audio interface 874 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 725, the audio interface 874 can also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. According to an example of the present disclosure, the microphone can also be used as an audio sensor to facilitate the control of the notification, as described below. The system 802 can also include a video interface 876, which enables the peripheral device 830 (e.g., an onboard camera) to operate to record static images, video streams, etc.

[0090] The mobile computing device 700 implementing the system 802 may have additional features or functionalities. For example, the mobile computing device 700 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage devices may be stored in the form of a computer program product. Figure 8 , illustrated by non-volatile storage area 868.

[0091] Data / information generated or captured by the mobile computing device 700 and stored via the system 802 may be stored locally on the mobile computing device 700, as described above, or the data may be stored on any number of storage media that may be accessed via the radio interface layer 872 or via a wired connection between the mobile computing device 700 and a separate computing device associated with the mobile computing device 700 (e.g., a server computer in a distributed computing network such as the Internet). It should be appreciated that such data / information may be accessed via the mobile computing device 700 via the radio interface layer 872 or the distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use according to well-known data / information transfer and storage means, including electronic mail and collaborative data / information sharing systems.

[0092] It should be understood that Figure 7 and Figure 8 The methods and systems are described for purposes of illustration and are not intended to limit the present disclosure to a specific order of steps or specific combination of hardware or software components.

[0093] Fig. 9A block diagram 900 is shown, according to an example embodiment, illustrating an example process for fine-tuning a neural network model 910. Based on a pre-trained multi-view diffusion model 920, the neural network model 910 can be fine-tuned according to the identity of the subject ("identity fine-tuning"). In this way, the neural network model 910 is trained as a multi-view identity model, or more specifically fine-tuned after some prior training to generate an image from a text prompt, wherein the image contains the subject.

[0094] The neural network model 910 may generally correspond to the image processor 300. For example, the neural network model 910 may include an encoder 912 and a decoder 914, which correspond to the encoder 302 and the decoder 322 of the image processor 300. Similarly, the pre-trained multi-view diffusion model 920 may include an encoder 922 and a decoder 924, which correspond to the encoder 302 and the decoder 322.

[0095] Identity fine-tuning can be based on a “frozen” instance of a multi-view diffusion model, for example, the image processor 300 (where the parameters (or weights) of the model are frozen after training) is shown as a pre-trained multi-view diffusion model 920, where further training of the model 910 results in updated parameters according to a parameter-preserving loss function 950 and a latent diffusion training loss 940. The parameter-preserving loss function 950 can be used to avoid overfitting and language drift of the neural network model 910.

[0096] exist Fig. 9 In the example shown in , the text / image pairs 930 include images of the subject and sample text for these images. The text / image pairs 930 may include 3 to 5 or more images of the subject. During fine-tuning of the neural network model 910, the neural network model 910 receives noise 960 (e.g., the noisy image 312) and learns appropriate parameters to generate images conditioned on the corresponding text from the text / image pairs 930 according to the latent diffusion training loss 940. However, these parameters are affected by the parameter training loss. Fig. 9 In the example shown in , the parameter training loss is defined as:

[0097]

[0098] where θ is a parameter, θ f is the corresponding frozen parameter of the pre-trained multi-view diffusion model 920, θ db are the corresponding parameters of the multi-view identity model 910 and are regularized using L1 loss. This approach improves the retention of previously trained generative capabilities. In some examples, the neural network model 910 is trained using a batch size of four images for approximately 400 epochs.

[0099] The phrases "at least one", "one or more", "or", and "and / or" are open-ended expressions that are both conjunctions and disjunctions in operation. For example, each of the expressions "at least one of A, B, and C", "at least one of A, B, or C", "one or more of A, B, and C", "one or more of A, B, or C", "A, B, and / or C" means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0100] The term "a" or "an" entity refers to one or more of that entity. Therefore, the terms "a" (or "an"), "one or more" and "at least one" can be used interchangeably herein. It should also be noted that the terms "including", "comprising" and "having" can be used interchangeably.

[0101] As used herein, the term "automatic" and variations thereof refer to any process or operation, which is generally continuous or semi-continuous, that is performed without substantial human input in the performance of the process or operation. However, a process or operation may be automatic even if the performance of the process or operation uses substantial or immaterial human input if input is received prior to the performance of the process or operation. Human input is considered substantial if it affects the manner in which the process or operation is performed. Human input that consents to the performance of the process or operation is not considered "substantial."

[0102] Any of the steps, functions, and operations discussed herein may be performed continuously and automatically.

[0103] The exemplary systems and methods of the present disclosure have been described with respect to computing devices. However, in order to avoid unnecessarily obscuring the present disclosure, the foregoing description omits several known structures and devices. Such omissions should not be considered limiting. Specific details are set forth to provide an understanding of the present disclosure. However, it should be understood that the present disclosure can be practiced in a variety of ways beyond the specific details set forth herein.

[0104] Furthermore, while the exemplary aspects illustrated herein show various components of the system being collocated, some components of the system may be remotely located, located in a distant portion of a distributed network (such as a LAN and / or the Internet), or located within a dedicated system. Thus, it should be appreciated that the components of the system may be combined into one or more devices, such as a server, a communication device, or collocated on a particular node of a distributed network (such as an analog and / or digital telecommunications network, a packet-switched network, or a circuit-switched network). It will be appreciated from the foregoing description that for reasons of computational efficiency, the components of the system may be arranged anywhere in a distributed component network without affecting the operation of the system.

[0105] In addition, it should be understood that the various links connecting the elements can be wired or wireless links or any combination thereof, or any other known or later developed (multiple) elements capable of providing data to the connecting elements and / or transmitting data from the connecting elements. These wired or wireless links can also be secure links and can be capable of conveying encrypted information. For example, the transmission medium used as the link can be any suitable electrical signal carrier (including coaxial cable, copper wire, and optical fiber), and can take the form of sound waves or light waves (such as sound waves or light waves generated during radio wave and infrared data communication).

[0106] While the flow charts have been discussed and illustrated with respect to a particular sequence of events, it should be appreciated that changes, additions, and omissions may be made to the sequence without materially affecting the operation of the disclosed configurations and aspects.

[0107] Several variations and modifications of the disclosure may be used.Some features of the disclosure may be provided without providing other features.

[0108] In other configurations, the systems and methods of the present disclosure may be implemented in combination with a special purpose computer, a programmed microprocessor or microcontroller and (multiple) peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital signal processor, a hard-wired electronic or logic circuit (such as a discrete component circuit), a programmable logic device or a gate array (such as a PLD, PLA, FPGA, PAL), a special purpose computer, any similar component, etc. In general, any (multiple) devices or components that can implement the methodology described herein can be used to implement various aspects of the present disclosure. Exemplary hardware that can be used for the present disclosure includes computers, handheld devices, phones (e.g., cellular phones, Internet-enabled phones, digital phones, analog phones, hybrid phones, etc.) and other hardware known in the art. Some of these devices include processors (e.g., single or multiple microprocessors), memory, non-volatile storage devices, input devices, and output devices. In addition, alternative software implementations can also be constructed to implement the methods described herein, including but not limited to distributed processing or component / object distributed processing, parallel processing, or virtual machine processing.

[0109] In yet another configuration, the disclosed method can be easily implemented in conjunction with software using an object or object-oriented software development environment that provides portable source code that can be used on a variety of computer or workstation platforms. Alternatively, the disclosed system can be implemented in part or in whole in hardware using standard logic circuits or VLSI designs. Whether software or hardware is used to implement a system according to the present disclosure depends on the speed and / or efficiency requirements of the system, the specific functionality, and the specific software or hardware system or microprocessor or microcomputer system used.

[0110] In yet another configuration, the disclosed method may be implemented in part in software, which may be stored on a storage medium and executed on a programmed general-purpose computer in cooperation with a controller and memory, a dedicated computer, a microprocessor, etc. In these cases, the disclosed system and method may be implemented as a program (such as an applet, or CGI scripts), implemented as a resource resident on a server or computer workstation, implemented as a routine embedded in a dedicated measurement system, system component, etc. The system may also be implemented by physically incorporating the system and / or method into a software and / or hardware system.

[0111] The present disclosure is not limited to the standards and protocols as described. Other similar standards and protocols not mentioned herein exist and are included in the present disclosure. In addition, the standards and protocols mentioned herein and other similar standards and protocols not mentioned herein are periodically replaced by faster or more efficient equivalents having substantially the same functions. Such alternative standards and protocols having the same functions are considered to be equivalents included in the present disclosure.

[0112] In various configurations and aspects, the present disclosure includes components, methods, processes, systems, and / or devices substantially as depicted and described herein, including various combinations, subcombinations, and subsets thereof. After understanding the present disclosure, those skilled in the art will understand how to make and use the systems and methods disclosed herein. In various configurations and aspects, the present disclosure includes providing devices and processes in the absence of items not depicted and / or described herein or in various configurations or aspects herein (including in the absence of those items that may have been used in previous devices or processes), for example, to improve performance, achieve simplicity, and / or reduce implementation costs.

Claims

1. An image generation system, comprising: a neural network model configured to perform a diffusion process to generate a set of multi-view images from the same input cue, the set of multi-view images having the same subject from different view orientations; The neural network model includes a self-attention layer configured to associate pixels across the set of multi-view images.

2. The image generation system of claim 1 , wherein the neural network model comprises a view encoder configured to generate, as input, corresponding view embeddings representing the different view orientations.

3. The image generation system of claim 2, wherein each view embedding is combined with the corresponding diffusion time step as a residual for a neural network model.

4. The image generation system of claim 2, wherein the neural network model further comprises a criss-cross attention layer configured to receive a text embedding representing the same input prompt, wherein the text embedding is combined with the view embedding.

5. The image generation system of claim 2, wherein the view encoder is a multi-layer perceptron.

6. The image generation system of claim 1, wherein the neural network model is trained using a plurality of 3D image sets, each 3D image within the 3D image set having the same subject matter and a different view orientation.

7. The image generation system of claim 6, wherein a diffusion time step is shared between each 3D image within the set of 3D images.

8. The image generation system of claim 6, wherein the neural network model is further trained using a plurality of separate 2D images having different subjects from each other. 9 . The image generation system of claim 8 , wherein the view embedding is omitted for the plurality of separate 2D images.

10. The image generation system of claim 8, wherein the neural network model is based on a pre-trained 2D diffusion model for transfer learning, and the neural network model is fine-tuned using the plurality of individual 2D images and the plurality of 3D image sets.

11. A method for training an image processor having a neural network model, the method comprising: generating a first training batch including a plurality of 3D image sets, each 3D image in the 3D image set having a same subject and a different view orientation; generating corresponding view embeddings representing the different view orientations as input to the neural network model; as well as The neural network model of the image processor is trained using the first training batch and the view embeddings for multi-view image diffusion.

12. The method of claim 11, wherein training the neural network model of the image processor comprises: A self-attention layer of the neural network model is used to associate pixels across images within a collection of 3D images.

13. The method of claim 12, wherein training the neural network model of the image processor comprises: The respective view embeddings are combined with the corresponding diffusion time steps as residuals for the neural network model.

14. The method of claim 12, wherein each 3D image in the set of 3D images corresponds to the same input prompt; Wherein training the neural network model comprises: generating a text embedding representing the same input prompt, wherein the text embedding is combined with the view embedding; as well as The text embedding is provided to the criss-cross attention layer of the neural network model.

15. The method of claim 11, wherein generating the corresponding view embedding comprises: A multi-layer perceptron is used to generate the corresponding view embeddings.

16. The method of claim 11, wherein generating the first training batch including the plurality of 3D image sets comprises: A 3D image is generated for each 3D image set so that its view orientation has the same elevation angle at evenly distributed azimuth angles.

17. The method of claim 16, wherein generating the first training batch further comprises: A plurality of separate 2D images having subjects that are different from each other are generated.

18. The method according to claim 17, further comprising: Pre-training the neural network model as a 2D diffusion model for transfer learning; as well as The neural network model is fine-tuned using the plurality of 3D image sets and the plurality of individual 2D images.

19. The method of claim 18, wherein fine-tuning the neural network model comprises: receiving a plurality of identity text / image pairs of a subject; as well as Parameters of the neural network model are fine-tuned using parameter-preserving loss.

20. The method of claim 16, wherein training the neural network model comprises: The diffusion time step is shared between each 3D image within the set of 3D images.

Images