View synthesis using a multiview image generative machine learning model

Abstract

Disclosed are systems, apparatuses, processes, and computer-readable media for generating images based on sparse images using a multiview image generative machine learning model. A disclosed method includes determining, using a machine learning (ML) model, a plurality of 3D features from one or more two-dimensional (2D) images based on respective pose information associated with each 2D image of the one or more 2D images, wherein the respective pose information is relative to a target pose of a target 2D image; combining, using the ML model, the plurality of 3D features into a plurality of multiview features; and generating the target 2D image including the target pose based on the plurality of 3D features and the plurality of multiview features.

Description

Qualcomm Ref. No.2400654WO VIEW SYNTHESIS USING A MULTIVIEW IMAGE GENERATIVE MACHINE LEARNING MODEL FIELD

[0001] The present disclosure generally relates to data processing using machine learning models. For example, aspects of the present disclosure are related to systems and techniques for performing view synthesis using a multiview image generative machine learning model. BACKGROUND

[0002] Many devices and systems allow a scene to be captured by generating images (or frames) and / or video data (including multiple frames) of the scene. For example, a camera or a device including a camera can capture a sequence of frames of a scene (e.g., a video of a scene). In some cases, the sequence of frames can be processed for performing one or more functions, can be output for display, can be output for processing and / or consumption by other devices, among other uses.

[0003] An artificial neural network can be implemented using computer technology inspired by logical reasoning performed by the biological neural networks in mammals. Deep neural networks, such as convolutional neural networks, are widely used for numerous applications, such as object detection, object classification, object tracking, big data analysis, among others. For example, convolutional neural networks are able to extract high-level features, such as facial shapes, from an input image, and use these high-level features to output a probability that, for example, an input image includes a particular object. SUMMARY

[0004] In some examples, systems and techniques are described for performing view synthesis using a multiview image generative machine learning model. According to at least one example, a method includes: determining, using a machine learning (ML) model, a plurality of three- dimensional (3D) features from one or more two-dimensional (2D) images based on respective pose information associated with each 2D image of the one or more 2D images, wherein the respective pose information is relative to a target pose of a target 2D image; combining, using the ML model, the plurality of 3D features into a plurality of multiview features; and generating theQualcomm Ref. No.2400654WO target 2D image including the target pose based on the plurality of 3D features and the plurality of multiview features.

[0005] In another example, an apparatus for generating a 3D model is provided that includes one or more memories configured to store one or more 2D images and one or more processors (e.g., implemented in circuitry) coupled to the one or more memories and configured to: determine, using an ML model, a plurality of 3D features from the one or more 2D images based on respective pose information associated with each 2D image of the one or more 2D images, wherein the respective pose information is relative to a target pose of a target 2D image; combine, using the ML model, the plurality of 3D features into a plurality of multiview features; and generate the target 2D image including the target pose based on the plurality of 3D features and the plurality of multiview features.

[0006] In another example, a non-transitory computer-readable medium is provided that has stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: determine, using an ML model, a plurality of 3D features from the one or more 2D images based on respective pose information associated with each 2D image of the one or more 2D images, wherein the respective pose information is relative to a target pose of a target 2D image; combine, using the ML model, the plurality of 3D features into a plurality of multiview features; and generate the target 2D image including the target pose based on the plurality of 3D features and the plurality of multiview features.

[0007] In another example, an apparatus is provided that includes: means for determining a plurality of 3D features from one or more 2D images based on respective pose information associated with each 2D image of the one or more 2D images, wherein the respective pose information is relative to a target pose of a target 2D image; means for combining the plurality of 3D features into a plurality of multiview features; and means for generating the target 2D image including the target pose based on the plurality of 3D features and the plurality of multiview features.

[0008] In some aspects, one or more the apparatuses or devices described herein is, is part of, and / or includes a wearable device, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device, such as an XR head-Qualcomm Ref. No.2400654WO mounted device (HMD) device or XR glasses), a wireless communication device such as a mobile device (e.g., a mobile telephone and / or mobile handset and / or so-called “smartphone” or another mobile device), a vehicle or a computing device or system of a vehicle, a camera, a personal computer, a laptop computer, a server computer, another device, or a combination thereof. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus further includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatuses described above can include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers or gyroscopes, one or more accelerometers, any combination thereof, and / or other sensors).

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0010] Other objects and advantages associated with the aspects disclosed herein will be apparent to those skilled in the art based on the accompanying drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are presented to aid in the description of various aspects of the disclosure and are provided solely for illustration of the aspects and not limitation thereof. So that the above-recited features of the present disclosure can be understood in detail, a more particular description, briefly summarized above, may be had by reference to aspects, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only certain typical aspects of this disclosure and are therefore not to be considered limiting of its scope, for the description may admit to other equally effective aspects. The same reference numbers in different drawings may identify the same or similar elements.

[0012] FIG.1 illustrates an implementation of a system-on-a-chip (SoC), in accordance with some aspects;

[0013] FIG.2A illustrates a fully connected neural network, in accordance with some aspects;Qualcomm Ref. No.2400654WO

[0014] FIG. 2B illustrates a locally connected neural network, in accordance with some aspects;

[0015] FIG. 3 is a diagram illustrating a forward diffusion process and a reverse diffusion process of a diffusion model, in accordance with some aspects;

[0016] FIG.4 is a diagram illustrating how diffusion data distributes from initial data to noise using a diffusion model, in accordance with some aspects;

[0017] FIG. 5 is a diagram illustrating a U-Net architecture for a diffusion model, in accordance with some aspects;

[0018] FIG.6 is a diagram illustrating a multiview image generative machine learning model based on sparse images in accordance with some aspects;

[0019] FIG.7 is a diagram a block diagram of a multiview image generative machine learning model in accordance with some aspects;

[0020] FIG. 8 is a diagram illustrating a single view engine for identification of features for the diffusion model in accordance with some aspects of the disclosure;

[0021] FIG.9 is a diagram illustrating a multiview engine for identification of features for the diffusion model in accordance with some aspects of the disclosure;

[0022] FIG. 10 is a block diagram of another multiview image generative machine learning model in accordance with some aspects of the disclosure;

[0023] FIG.11 is a flow diagram illustrating a method or process for generating images based on sparse images using a multiview image generative machine learning model in accordance with some aspects; and

[0024] FIG. 12 is a block diagram illustrating a computing system for implementing certain aspects described herein.Qualcomm Ref. No.2400654WO DETAILED DESCRIPTION

[0025] Certain aspects of this disclosure are provided below for illustration purposes. Alternate aspects may be devised without departing from the scope of the disclosure. Additionally, well- known elements of the disclosure will not be described in detail or will be omitted so as not to obscure the relevant details of the disclosure. Some of the aspects described herein may be applied independently and some of them may be applied in combination as would be apparent to those of skill in the art. In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of aspects of the application. However, it will be apparent that various aspects may be practiced without these specific details. The figures and descriptions are not intended to be restrictive.

[0026] The ensuing description provides example aspects, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the example aspects will provide those skilled in the art with an enabling description for implementing some aspects of the disclosure. It should be understood that various changes may be made in the function and arrangement of elements without departing from the scope of the application as set forth in the appended claims.

[0027] Demand for and consumption of image and video data has significantly increased in consumer and professional settings. Devices and systems are commonly equipped with capabilities for capturing and processing image and video data. For example, a camera or a computing device including a camera (e.g., a mobile telephone or smartphone including one or more cameras) can capture a video and / or image of a scene, a person, an object, etc. The image and / or video can be captured, processed, and then output (and / or stored) for consumption. As used herein, the terms “image processing” and “video processing” may be used interchangeably, such as in describing an image processing neural network and a video processing neural network (e.g., based on video data comprising a series of frames (e.g., images) that may be processed consecutively).

[0028] In some examples, images and / or video can be processed using one or more machine learning models. In some aspects, a machine learning model can be a deep learning model (e.g., (e.g., deep neural network models and / or other types of deep learning models). Deep learning models can be trained to achieve or assist in the tasks and applications outlined above. ExamplesQualcomm Ref. No.2400654WO of deep neural network models include convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), and diffusion-based neural networks (also referred to as diffusion models), among others.

[0029] A diffusion model provides a general-purpose, high-quality model for performing a task (e.g., depth estimation, optical flow estimation, stereo estimation, etc.) and enables more general applications as well. Diffusion models are latent-variable generative models trained to transform a sample of a noise into a sample from a data distribution. For example, a diffusion model can define a Markov chain of diffusion steps to slowly add random noise (e.g., Gaussian noise) to data and then learn to reverse the diffusion process to construct desired data samples from the noise. Once trained, the diffusion model can successfully perform a particular task (e.g., object classification, depth estimation, etc.) when provided a conditioning image or other conditional input and random noise to perform reverse diffusion for task-specific prediction.

[0030] A stable diffusion model leverages the principles of stable diffusion to generate high- quality and diverse images. Conventional generative models such as GANs or variational encoders (VAEs) experience mode collapse and produce visually unrealistic results. A stable diffusion model, by contrast, focuses on generating stable and coherent image samples by sequentially refining a noise input. By incorporating the stable diffusion process, the stable diffusion model can handle a wide range of data distributions, including heavy-tailed and complex image features, leading to more realistic and robust image synthesis.

[0031] Zero-shot novel view synthesis (NVS) attempts to synthesize high-quality and visually consistent views given a source view image and a relative pose. NVS usually involves three- dimensional (3D) content recovery from source view image and target view image generation. Current solutions may use a 3D reconstruction pipeline based on optimization methods such as neural radiance fields (NeRF) to generate a 3D model and then render an arbitrary pose based on the 3D model. The 3D reconstruction pipeline requires time-consuming iterative optimization for each object. Other current solutions use optimization-free methods that use a neural network to implicitly learn 3D content from a large-scale dataset and require a generative model and uses single view inputs. Optimization-free methods may implement 3D content recovery and targetQualcomm Ref. No.2400654WO view image generation into a single inference and prevent the introduction of multiview information.

[0032] Systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively referred to as “systems and techniques”) are described herein performing view synthesis using a multiview image generative machine learning model. For instance, the systems and techniques can receive sparse multiview images (e.g., at least one to three images) of a subject having different poses (e.g., a respective pose of a camera used to capture each respective image) and generate an image having a target pose (e.g., an arbitrary pose) based on the multiview images. The images can include two-dimensional (2D) images.

[0033] The generative machine learning model is trained to learn features of one or more subjects from one or more training images at different poses, generate multiview features from the features, and identify features using attention mechanisms to generate images of one or more subjects (e.g., the same as or different from the subject(s) in the training image(s)) having target poses. For example, the machine learning system can include one or more transformer networks (e.g., queryformers) including attention-based layers configured to (e.g., trained to) synthesize pose information and image features into multiview features for input into a diffusion model. In some cases, the attention-based layers can include cross-attention layers. In some aspects, the one or more transformer networks can include other layers or another network, such as a feed-forward (FFW) network (e.g., a CNN). The diffusion model is configured to (e.g., trained to) use the image features and the multiview features to synthesize an image of the subject at a pose corresponding to an arbitrary pose.

[0034] In some aspects, the generative machine learning model includes a two-stage pipeline, with a first stage configured to (e.g., trained to) extract single-view features of 2D images, and a second stage configured to synthesize multiview features from the single-view features. In some cases, the first stage and the second stage may each include a transformer neural network (e.g., a first transformer neural network in the first stage and a second transformer neural network in the second stage). In some examples, the first transformer neural network can include a first queryformer and the second transformer neural network can include a second queryformer. Each transformer neural network (e.g., each queryformer) can be separately trained.Qualcomm Ref. No.2400654WO

[0035] The first stage may also include an encoder-decoder neural network configured to determine or generate features from one or more input images (e.g., from one or more tokens of each of the one or more input images, where each token can represent a portion of an input image). In some aspects, the encoder neural network can add positional embeddings (e.g., sine-cosine positional embeddings) to the encoder and decoder inputs. In some cases, the features can be combined (e.g., concatenated, summed, averaged, or otherwise combined) with pose information indicating a respective pose of each image of the one or more images. The combined features and pose information can be input to the first transformer neural network of the first stage. The output of the first transformer neural network can be provided as input to the diffusion model. For example, the output can be used to train parameters (e.g., weights and / or other parameters) of the diffusion model during the first stage.

[0036] The second stage may also include the diffusion model for performing image generation. As noted previously, the parameters (e.g., weights and / or other parameters) of the diffusion model can be trained during the first stage. In some cases, the parameters (e.g., weights and / or other parameters) of the diffusion model are frozen (e.g., static or fixed) during training of the second stage, in which case the parameters of the diffusion model are not updated during training of the second stage (e.g., during training of parameters, such as weights, of the transformer neural network of the second stage). For example, the diffusion model may include one or more attention modules, and the parameters (e.g., weights) of the one or more attention modules may be frozen during training of the second stage. The training of the second stage tunes the parameters associated with the multiview fusion (e.g., the queryformer of the second stage).

[0037] The architecture of the generative machine learning model can be the same during inference as the architecture during training. For instance, during inference of the generative machine learning model, the output of the first transformer neural network of the first stage can also be provided as input to the diffusion model. During the second stage, the output of the first transformer neural network of the first stage can be provided as input to the second transformer neural network in the second stage and not to the diffusion model.

[0038] In some aspects, the generative machine learning model includes a single-stage pipeline. In such aspects, the second transformer neural network (e.g., the second queryformer) isQualcomm Ref. No.2400654WO optional. For instance, in some cases, tokens from multiple views can be combined (e.g., concatenated, summed, averaged, or otherwise combined) and input to the first transformer neural network (e.g., the first queryformer). The first transformer neural network can process the combined tokens to generate a multiview representation. In such cases, the second transformer neural network described above can be skipped or not used. For example, the multiview representation can be provided to a diffusion model for generating an image of the subject of the 2D images at a target pose (e.g., a camera position relative to the subject). In a machine learning model with a single-stage pipeline, a positional encoding may be applied to features identified from the 2D images, and a perceiver may be configured to (e.g., trained to) fuse the features from the 2D images into a multiview representation.

[0039] Various aspects of the present disclosure will be described with respect to the figures.

[0040] FIG.1 illustrates, according to some aspects, an implementation of a system-on-a-chip (SOC) 100, which may include a central processing unit (CPU) 102 or a multi-core CPU, configured to perform one or more of the functions described herein. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computational device (e.g., neural network with weights), and task information, among other information may be stored in a memory block associated with a neural processing unit (NPU) 108, in a memory block associated with a CPU 102, in a memory block associated with a graphics processing unit (GPU) 104, in a memory block associated with a digital signal processor (DSP) 106, in a memory block 118, and / or may be distributed across multiple blocks. Instructions executed at the CPU 102 may be loaded from a program memory associated with the CPU 102 or may be loaded from a memory block 118.

[0041] The SOC 100 may also include additional processing blocks tailored to specific functions, such as a GPU 104, a DSP 106, a connectivity block 110, which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi- Fi connectivity, USB connectivity, Bluetooth connectivity, and the like, and a multimedia processor 112 that may, for example, detect and recognize gestures. In one implementation, the NPU is implemented in the CPU 102, DSP 106, and / or GPU 104. The SOC 100 may also include a sensor processor 114, image signal processors (ISPs) 116, and / or navigation module 120, whichQualcomm Ref. No.2400654WO may include a global positioning system. The SOC 100 may include other sensors not shown in FIG.1, such as one or more NPUs, one or more neural signal processors (NSPs), any combination thereof, and / or other processor(s). In some examples, the sensor processor 114 can be associated with or connected to one or more sensors for providing sensor input(s) to sensor processor 114. For example, the one or more sensors and the sensor processor 114 can be provided in, coupled to, or otherwise associated with a same computing device.

[0042] The SOC 100 may be based on an ARM, RISC-V, or any reduced instruction set. In an aspect of the present disclosure, the instructions loaded into the CPU 102 may comprise code to search for a stored multiplication result in a lookup table (LUT) corresponding to a multiplication product of an input value and a filter weight. The instructions loaded into the CPU 102 may also include code to disable a multiplier during a multiplication operation of the multiplication product when a lookup table hit of the multiplication product is detected. In addition, the instructions loaded into the CPU 102 may comprise code to store a computed multiplication product of the input value and the filter weight when a lookup table miss of the multiplication product is detected. The SOC 100 and / or components thereof may be configured to perform image processing using machine learning techniques according to aspects of the present disclosure discussed herein. For example, the SOC 100 and / or components thereof may be configured to implement a diffusion model as described herein and / or object detection according to aspects of the present disclosure.

[0043] Machine learning (ML) can be considered a subset of artificial intelligence (AI). ML systems can include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and inference, without the use of explicit instructions. In some aspects, an ML system can be a neural network (also referred to as an artificial neural network), which may include an interconnected group of artificial neurons (e.g., neuron models). Neural networks may be used for various applications and / or devices, such as image and / or video coding, image analysis and / or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, among others.

[0044] Individual nodes in a neural network may emulate biological neurons by taking input data and performing simple operations on the data. The results of the simple operations performed on the input data are selectively passed on to other neurons. Weight values are associated withQualcomm Ref. No.2400654WO each vector and node in the network, and these values constrain how input data is related to output data. For example, the input data of each node may be multiplied by a corresponding weight value, and the products may be summed. The sum of the products may be adjusted by an optional bias, and an activation function may be applied to the result, yielding the node’s output signal or “output activation” (sometimes referred to as a feature map or an activation map). The weight values may initially be determined by an iterative flow of training data through the network (e.g., weight values are established during a training phase in which the network learns how to identify particular classes by their typical input data characteristics).

[0045] Different types of neural networks exist, such as CNNs, RNNs, GANs, multilayer perceptron (MLP) neural networks, transformer neural networks, diffusion-based neural networks, among others. For instance, CNNs are a type of feed-forward artificial neural network. Convolutional neural networks may include collections of artificial neurons that each have a receptive field (e.g., a spatially localized region of an input space) and that collectively tile an input space. RNNs work on the principle of saving the output of a layer and feeding the output back to the input to help in predicting an outcome of the layer. A GAN is a form of generative neural network that can learn patterns in input data so that the neural network model can generate new synthetic outputs that reasonably could have been from the original dataset. A GAN can include two neural networks that operate together, including a generative neural network that generates a synthesized output and a discriminative neural network that evaluates the output for authenticity. In MLP neural networks, data may be fed into an input layer, and one or more hidden layers provide levels of abstraction to the data. Predictions may then be made on an output layer based on the abstracted data.

[0046] According to some aspects, Deep learning (DL) can be a machine learning technique and can be considered a subset of ML. Many DL approaches are based on a neural network, such as an RNN or a CNN, and utilize multiple layers. The use of multiple layers in deep neural networks can permit progressively higher-level features to be extracted from a given input of raw data. For example, the output of a first layer of artificial neurons becomes an input to a second layer of artificial neurons, the output of a second layer of artificial neurons becomes an input to a third layer of artificial neurons, and so on. Layers that are located between the input and output of the overall deep neural network are often referred to as hidden layers. The hidden layers learn (e.g.,Qualcomm Ref. No.2400654WO are trained) to transform an intermediate input from a preceding layer into a slightly more abstract and composite representation that can be provided to a subsequent layer, until a final or desired representation is obtained as the final output of the deep neural network.

[0047] As noted above, a neural network is an example of a machine learning system, and can include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes of the input layer, processing is performed by hidden nodes of the one or more hidden layers, and an output is produced through output nodes of the output layer. Deep learning networks typically include multiple hidden layers. Each layer of the neural network can include feature maps or activation maps that can include artificial neurons (or nodes). A feature map can include a filter, a kernel, or the like. The nodes can include one or more weights used to indicate an importance of the nodes of one or more of the layers. In some cases, a deep learning network can have a series of many hidden layers, with early layers being used to determine simple and low-level characteristics of an input, and later layers building up a hierarchy of more complex and abstract characteristics.

[0048] A deep learning architecture may learn a hierarchy of features. If presented with visual data, for example, the first layer may learn to recognize relatively simple features, such as edges, in the input stream. In another example, if presented with auditory data, the first layer may learn to recognize spectral power in specific frequencies. The second layer, taking the output of the first layer as input, may learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For instance, higher layers may learn to represent complex shapes in visual data or words in auditory data. Still higher layers may learn to recognize common visual objects or spoken phrases.

[0049] Deep learning architectures may perform especially well when applied to problems that have a natural hierarchical structure. For example, the classification of motorized vehicles may benefit from first learning to recognize wheels, windshields, and other features. These features may be combined at higher layers in different ways to recognize cars, trucks, and airplanes.

[0050] Neural networks may be designed with a variety of connectivity patterns. In feed- forward networks, information is passed from lower to higher layers, with each neuron in a given layer communicating to neurons in higher layers. A hierarchical representation may be built up inQualcomm Ref. No.2400654WO successive layers of a feed-forward network, as described above. Neural networks may also have recurrent or feedback (also called top-down) connections. In a recurrent connection, the output from a neuron in a given layer may be communicated to another neuron in the same layer. A recurrent architecture may be helpful in recognizing patterns that span more than one of the input data chunks that are delivered to the neural network in a sequence. A connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. A network with many feedback connections may be helpful when the recognition of a high-level concept may aid in discriminating the particular low-level features of an input.

[0051] The connections between layers of a neural network may be fully connected or locally connected. FIG. 2A illustrates some aspects of a fully connected neural network 202. In a fully connected neural network 202, a neuron in a first layer may communicate its output to every neuron in a second layer, so that each neuron in the second layer will receive input from every neuron in the first layer. FIG. 2B illustrates some aspects of a locally connected neural network 204. In the locally connected neural network 204, a neuron in a first layer may be connected to a limited number of neurons in the second layer. More generally, a locally connected layer of the locally connected neural network 204 may be configured so that each neuron in a layer will have the same or a similar connectivity pattern, but with connections strengths that may have different values (e.g., connections 210, 212, 214, and 216). The locally connected connectivity pattern may give rise to spatially distinct receptive fields in a higher layer, as the higher layer neurons in a given region may receive inputs that are tuned through training to the properties of a restricted portion of the total input to the network.

[0052] As noted above, one class of machine learning models includes diffusion models (e.g., diffusion-based neural networks), which can also be referred to as diffusion probabilistic models. Diffusion models are latent-variable models. For example, a diffusion model defines a Markov chain of diffusion steps to slowly add random noise (e.g., Gaussian noise) to data and then learn to reverse the diffusion process to construct desired data samples from the noise. For instance, a diffusion model can be trained using a forward diffusion process (which is fixed) and a reverse diffusion process (which is learned). A diffusion model can be trained to be able to perform a generative process (e.g., a denoising process). A goal of a diffusion model is to be able to denoise any arbitrary noise added to input data (e.g., a video).Qualcomm Ref. No.2400654WO

[0053] FIG.3 provides two sets of images 300 that show the forward diffusion process (which is fixed) and the reverse diffusion process (which is learned) of a diffusion model. As shown in the forward diffusion process of FIG.3, noise 303 is gradually added to a first set of images 302 at different time steps for a total of T time steps (e.g., making up a Markov chain), producing a sequence of noisy samples X1through XT.

[0054] Diffusion models from a training perspective will take an image and will slowly add noise to the image to destroy the information in the image. In some aspects, the noise 303 is Gaussian noise, although the noise is not limited to any specific type of noise. Each time step can correspond to each consecutive image of the first set of images 302 shown in FIG.3. The initial image X0 of FIG.3 is of a vase. Addition of the noise 303 to each image (corresponding to noisy samples X1to XT) results in gradual diffusion of the pixels in each image until the final image (corresponding to sample XT) essentially matches the noise distribution. For example, by adding the noise, each data sample X1 through XT gradually loses its distinguishable features as the time step becomes larger, eventually resulting in the final sample XTbeing equivalent to the target noisedistribution, for instance a unit variance zero-centered Gaussian ^^^0, 1^.

[0055] The second set of images 304 shows the reverse diffusion process in which XTis the starting point with a noisy image (e.g., one that has Gaussian noise or some other type of noise). The diffusion model can be trained to reverse the diffusion process (e.g., by training a model pθ(xt-1| xt)) to generate new data. In some aspects, a diffusion model can be trained by finding the reverse Markov transitions that maximize the likelihood of the training data. By traversing backwards along the chain of time steps, the diffusion model can generate the new data. For example, as shown in FIG.3, the reverse diffusion process proceeds to generate X0as the image of the vase. In other cases, the input data and output data can vary based on the task for which the diffusion model is trained.

[0056] As noted above, the diffusion model is trained to be able to denoise or recover the original image X0in an incremental process as shown in the second set of images 304. In some aspects, the neural network of the diffusion model can be trained to recover Xt given Xt-1, such as provided in the below example equation:Qualcomm Ref. No.2400654WO ^^^^^௧|^^௧ି^^ ൌ ^^൫^^௧; ^1 െ ^^௧^^௧ି^,^^௧^^൯

[0057] A diffusionDefine ∝^௧ ൌ ∏௧^ୀ^ ^1 െ ^^^^ → ^^^^^௧|^^^^ ൌ ^^^^^௧; ^∝^௧ ^^^ , ^1 െ ∝^௧ ^^^^

[0058] Sampling can be defined as^^௧ ൌ ^∝^௧ ^^^ ^ ^1 െ ∝^௧ ^^ where ε ∼ ^^^^^, ^^^.

[0059] In someschedule) is designedsuch that ∝^ఁ ^ 0 and ^^^^^ఁ|^^^^ ^ ^^^^^ఁ ; ^^, ^^^.

[0060] The diffusion model runs in an iterative manner to incrementally generate the input image X0. In one example, the model may have twenty steps. However, in other examples, the number of steps can vary.

[0061] FIG.4 is a diagram 400 illustrating how diffusion data is distributed from initial data to noise using a diffusion model in the forward diffusion direction, in accordance with some aspects. Note that the initial data q(X0) is detailed in the initial stage of the diffusion process. An illustrative example of the data q(X0) is the initial image of the vase shown in FIG. 3. As the diffusion model iterates and iteratively adds sampled noise to the data from t = 0 to t = T, as shown in FIG. 4, the data becomes noisier and may ultimately result in pure noise (e.g., at q(XT)). The example of FIG.4 illustrates the progression of the data and how the data becomes diffused with noise in the forward diffusion process.

[0062] In some aspects, the diffused data distribution as shown in FIG.4 can be as follows: ^^^^^௧^ ൌ ^ ^^^^^^, ^^௧^ ^^^^^ ൌ ^ ^^^^^^^ ^^^^^௧|^^^^ ^^^^^.

[0063] In the represents distribution, ^^^^^^, ^^௧^represents the joint distribution, ^^^^^^^ represents the input data distribution, and ^^^^^௧|^^^^ is thediffusion kernel. In some aspects, the model can sample ^^௧ ∼ ^^^^^௧^ by first sampling ^^^ ∽ ^^^^^^^Qualcomm Ref. No.2400654WOand then sampling ^^௧ ∼ ^^^^^௧|^^^^ (which may be referred to as ancestral sampling). The diffusionkernel takes the input and returns a vector or other data structure as output.

[0064] The following is a summary of a training algorithm and a sampling algorithm for a diffusion model. A training algorithm can include the following steps: 1: repeat 2: ^^^ ∼ ^^^^^^^3: ^^ ∼ Uniform ^^1, ... , ^^ ^^4:∈ ~ ^^^^^, ^^^5: Takeon ∇∥ ∈ െ ∈ ^^∝^ ^^ ^ ^1 െ ^ ଶ∅ ∅ ௧ ^ ∝௧∈, ^^^ ∥

[0065] can steps: 1: ^^ఁ ∼ ^^^^^, ^^^2: for ^^ ൌ ^^, ... , 1 do

[0066] FIG. 5 is a diagram illustrating a U-Net architecture 500 for a diffusion model, in accordance with some aspects. The initial image 502 is provided to the U-Net architecture 500 which includes a series of residual networks (ResNet) blocks and self-attention layers to represent the network ^^(xt, t). The U-Net architecture also includes fully-connected layers 508. In some cases, the time representation 510 can be sinusoidal positional embeddings or random Fourier features. The noisy output 506 from the forward diffusion process is also shown.

[0067] The U-Net architecture 500 includes a contracting path 504 and an expansive path 505 as shown in FIG. 5, which shows the U-shaped architecture. The contracting path 504 can be a convolutional network that includes repeated convolutional layers (that apply convolutional operations), each followed by a rectified linear unit (ReLU) and a max pooling operation. When images are being processed (e.g., the image 502) during the contracting path 504, the spatial information of the image 502 is reduced as features are generated. The expansive path 505Qualcomm Ref. No.2400654WO combines the features and spatial information through a sequence of up-convolutions and concatenations with high-resolution features from the contracting path. Some of the layers can be self-attention layers which leverage global interactions between semantic features at the end of the encoder to explicitly model full contextual information.

[0068] Existing conditioning strategies of U-Net includes sequential conditioning and attention conditioning. For example, in sequential conditioning, a source-view image xs is input into an auto-encoder and an output latent map from the auto-encoder is concatenated with xt, which is a source-view image from a previous time, as input into the U-Net. According to an attention conditioning strategy, the source-view image is input into a Contrastive Language-Image Pre- Training (CLIP) image encoder and the output is fed into the attention modules in U-Net. The network design can be summarized as: ^̅^௧ି^ ൌ∈^ ^^̅^^ ⊕ ^̅^௧ ,^^^^^^^^^^^^^ ⊕ π, t^

[0069] In these examples, ∈^denotes U-Net and t is the timestamp. The notation ∗ത can be used refer to features in latent space. In some aspect, attention decoding only injects pose information because the CLIP image encoder will generate a high level single token which is too coarse, and will degenerate to: ^̅^௧ି^ ≊∈^ ^^̅^^ ⊕ ^̅^௧ ,π, t^

[0070] The source-view image xsand the relative pose π are injected into U-Net at different places, and this works well when only a single source view is available. When multiple source views are available, U-Net can be summarized by: ே ே ^^̅^ ^^ ⊕ ,^ ^^^^^^^^^^^^^^ ⊕ π^^, ^^^

[0071] U-Net may corresponding pose- image data pair because the pose-images are injected at different places. Another issue is that convolutional layers cannot handle variable length input, and the number of input source-view images usually varies. In some aspects, an appearance-entangled strategy may be used. In someQualcomm Ref. No.2400654WO aspects, when U-Net conditioning takes effect, a solution is to feed both source-view images and corresponding poses into U-Net. The appearance-entangled strategy can be summarized as: ே ^̅^௧ି^ ൌ∈^ ൬^ ^^̅^^ ⊕ π^^ ⊕ ^̅^௧ , , t^^

[0072] The appearance-the difficult in directly fusing image data and pose data. For example, directly fusing image data and pose data solution also cannot accommodate variable input length.

[0073] In some aspects, an attention mechanism can be used to combine source-view images and corresponding poses in attention modules in an ML model such as U-Net. In some cases, the attention module can replace CLIP image encoder and learn 3D representation from source-view images. In some aspects, a visual transformer architecture (ViT) or a multiview image generative machine learning model is disclosed and can be represented by: ே ^̅^௧ି^ ൌ∈^ ൬^̅^௧ ,^ ^ViT^x^^ ⊕ π^^, t^^

[0074] FIG.6 is amachine learning model 600 based on sparse images in accordance with some aspects. In some aspects, the multiview image generative machine learning model 600 is configured to receive sparse images having different views, and may be include the visual transformer architecture described above. For example, as shown in FIG.6, the multiview image generative machine learning model 600 receives a first image 602, a second image 604, and a third image 606. Each of the images has a different view and show the features of a subject from different poses. For example, the pose of subject may be represented in spherical coordinates using a radial distance r, a polar angle θ, and an azimuth angle ϕ, with the subject of the image being centered at the origin.

[0075] The multiview image generative machine learning model 600 is configured to identify multiview features based on the input images. The multiview image generative machine learning model 600 is also configured to receive pose information 608 and, based on the multiview features, generate an image 610 corresponding to the pose information 608.Qualcomm Ref. No.2400654WO

[0076] In some aspects, each source-view image may be converted into image patches. In some cases, a positional encoding is added to each small image patch. The patches are fed into multiview image generative machine learning model 600 and are converted into image tokens representing 3D structures. In one aspect, a Masked Auto-Encoder (MAE) and pretrained weights of the MAE can be used for the multiview image generative machine learning model 600. In some cases, a camera pose (corresponding to a pose of a camera when the image is captured and represented by M) is combined with (e.g., concatenated) the image tokens to inform each image token of their corresponding pose. In some cases, 196 patches × N views can be computationally intensive. A token reduction strategy can be used to reduce the computational load. For example, an iterative module may reduce the tokens to a fixed size, such as 64 tokens. The information from the pose- encoded tokens can be aggregated into learnable tokens and can be denoted as: ^^^^^^^^ ^^^, ^^, ^^^ ^ ^^

[0077] For example, a source view tokenization module can extract pose- image tokens Mi for each input source-view image i. Providing all of the source view image tokens will increase the computation proportionally to the number of images and causes out-of-memory issues. In some aspects, a fixed number of tokens can perform favorably and a fixed number (e.g., 64) of learnable seed tokens S0 are initialized at the beginning of training. The seek tokens are used as queries in a multi-view cross former. The initial tokens S0 are shared across training examples and learn to be query token biases to extract relevant information from pose-image tokens Mi. The multi-view cross former block, in particular, is first computed with an attention operation followed by a residual: S'l= Attnl(Q,K,V) + Sl-1

[0078] The output tokens will pass through a feed-forward layer FFWlwhich consists of layer normalization, a linear layer, GeLU activation, another linear layer sequentially, followed by a final residual connection. Sl= FFWl(S’l) + S’lQualcomm Ref. No.2400654WO

[0079] The learnable tokens can be input into attention modules as a key (k) and value (v) to generate different views of the source based on learned features from the 2D images. For example,the query, key, and value in the attention block are calculated with Q = Sl−1and K = V = M1⊕...⊕MN. The output tokens Slmay be used as new target-view seeds for the following layerand the block will then be repeated L times. The target-view seeds SLwill be fed into attention each timestep of the diffusion process and produce realistic images at the target view. The final target view seeds SLare configured to learn to warp the source view tokens into the target-view tokens, which are used for conditioning the diffusion process. The final target view seeds enable the model to always feed a fixed number of query tokens into the attention module and improve efficiency.

[0080] FIG.7 is a diagram a block diagram of a multiview image generative machine learning model 700 in accordance with some aspects. In some aspects, the multiview image generative machine learning model 700 includes a two-stage pipeline for extracting features for a diffusion model. For example, the multiview image generative machine learning model 700 includes a single view engine 710 and a multiview engine 720.

[0081] One or more images 702 are provided into the single view engine 710 for identification of key features from each image, and each image can include pose information (e.g., r, θ, and ϕ) related to a subject of the image. In some cases, the pose information can be provided can also be provided separately in the single view engine 710. In some aspects, the single view engine 710 identifies features from the view and combines pose information with each feature of the subject. For example, the pose information is combined (e.g., concatenated) with each feature. The single view engine 710 uses an attention mechanism to identify features (e.g., single view features) from each image based on weights. For example, the single view engine 710 comprises learnable tokens that are randomly initialized and tuned during training into weights that are configured to identify features from each image.

[0082] The single view engine 710 then selects the features for the multiview engine 720. For example, the single view engine 710 may include a queryformer, which is a machine learning model or natural language processing model designed to transform and generate queries or search strings in a context-aware manner. In some aspects, a queryformer uses user input, specific dataQualcomm Ref. No.2400654WO sources, or search contexts to produce well-structured and relevant search queries. A queryformer may use attention mechanisms to focus on specific parts of the input data or context when generating queries or search strings. For example, a queryformer assigns different levels of importance or attention to various elements in the input (e.g., an input image), allowing the queryformer to weigh and consider them differently when generating queries. The attention mechanism of a queryformer is based on the principles of self-attention, cross-attention, and other types of attention to emphasize relevant aspects of the input, adapt to the specifics of the input, and generate queries that are better suited for the learning and inference tasks.

[0083] In some aspects, the multiview engine 720 is configured to (e.g., trained to) synthesize the features from each image into multiview features using an attention mechanism. The multiview engine 720 receives the features from the single view engine 710 and combines (e.g., concatenates) the features of different images into multiview features. For example, the multiview engine 720 may combine the features at different poses into multiview features that represent the object of the images at different orientations. In some aspects, the multiview engine 720 may include a queryformer that is trained to identify and represent the features in 3D space based on the images 702.

[0084] The multiview engine 720 is configured to (e.g., trained to) use the multiview features, the features from the single view engine 710, and the pose information 704 to generate a view of the subject as it would appear from the coordinates of the pose information 704. In some aspects, the multiview engine 720 may include a queryformer configured to receive the multiview features and generate a three-dimensional feature representation that corresponds to the feature based on the image 702.

[0085] The features identified by the single view engine 710, the multiview features identified by the multiview engine 720, and the pose information 704 may be provided to a diffusion model that is trained to generate an image. A diffusion model may be configured to (e.g., trained to) take a variety of inputs, such as text input. In this case, the diffusion model receives the features from the single view engine 710, the multiview features from the multiview engine 720, and the pose information 704 and generates an image 706. The image 706 represents the subject of the images 702 based on the pose information 704. For example, the multiview engine 720 generates aQualcomm Ref. No.2400654WO representation of the subject of the images 702 at any arbitrary position in 3D space based on the multiview image generative machine learning model 700 learning the single view features and the multiview features.

[0086] FIG.8 is a diagram illustrating a single view engine 800 for identification of features for the diffusion model in accordance with some aspects of the disclosure. In some aspects, the single view engine 800 may be an example of the single view engine 710 of FIG. 7. The single view engine 800 may also be referred to as a visual transformer. Visual transformers adopt a transformer architecture for image data to capture complex spatial relationships and context in images. A visual transformer may include self-attention mechanisms and feed-forward neural networks. The self-attention mechanism allows the model to focus on different parts of the image when processing each pixel or region and enables the visual transformer to capture long-range dependencies and relationships within the image.

[0087] The single view engine 800 is configured to (trained to) receive one or more images 802 at an encoder 804 for identifying a plurality of image features 806 of the subject of the images 802. In some aspects, the encoder 804 transforms the input data (e.g., an image) into a lower- dimensional feature representation, which can be referred to as a feature or a token. For example, the transformation captures the most relevant and discriminative characteristics of the images 802. The encoder 804 typically consists of one or more layers of neural networks. In some aspects, the encoder 804 is configured to generate a plurality of patches from an input image (e.g., 32x32 patches for a 224x224 image) and identify features from each patch.

[0088] In one example, the encoder 804 may be a masked autoencoder (MAE) configured to learn a more sparse and structured representation of the input data. An MAE includes a masking mechanism that selectively retains or discards certain parts of the encoded representation to assist in capturing the most essential and informative features of the input data while ignoring less relevant information, resulting in a more efficient and compact representation.

[0089] The plurality of image features 806 are provided into a combiner 808 (e.g., using a concatenation) to create a plurality of pose-image features 812 that represents the various features identified in the encoder 804 and positions of those features within a single view (e.g., a 2D representation).Qualcomm Ref. No.2400654WO

[0090] The pose-image features 812 are provided to a queryformer 814 to identify relevant pose-image features having high relevance. For example, the queryformer 814 can include a transformer neural network with one or more attention-based layers (e.g., cross-attention layers) and in some cases one or more other layers or another network (e.g., a FFW network, such as a CNN. In some cases, the queryformer 814 is trained based on learnable weights 816 to identify features that high the highest attention (e.g., cross-attention) and provide the most salient information about the subject of the images 802. As an example, if the subject of an image 802 is a person’s face, the most pertinent features may include eyes, lips, mouth, jaw, ears, and hair details. The queryformer 814 identifies the most salient features of the pose-image features 812 of the create the pose-image features 812 based on the weights 816 and generates the key features 818 for each image. The generates key features 818 comprises a plurality of different features that are represented by a different fill pattern in FIG. 8. In some aspects, the queryformer 814 is configured to (e.g., trained to) reduce the number of image features to improve the efficiency of an image generation system using the single view engine 800 (e.g., the multiview image generative machine learning model 700 of FIG.7).

[0091] In one illustrative aspect, the encoder 804 may be a pretrained encoder or MAE and the queryformer 814 can be trained on a dataset. In this case, the learnable weights 816 are randomly initialized and the queryformer 814 is tuned based on backpropagation that updates the learnable weights 816 to learn features from a dataset.

[0092] FIG.9 is a diagram illustrating an example of a multiview engine 900 for identification of features for the diffusion model in accordance with some aspects of the disclosure. The multiview engine 900 is configured to (e.g., trained to) receive the observed image features 902 (e.g., the key features 818 of FIG.8) associated with each image (e.g., from the images 702 in FIG. 7, or the images 802 of FIG.8). In some aspects, the multiview engine 900 may implement at least the features of the multiview engine 720 of FIG.7 and may be configured to receive image features from a single view engine 800 of FIG.8.

[0093] The multiview engine 900 is configured to generate an image 920 corresponding to the various features at an arbitrary pose. In one aspect to generate the image, the multiview engine 900 is configured to synthesize the observed image features 902 into multiview features 904, and inQualcomm Ref. No.2400654WO this case, aligns each feature. For example, a specific feature in the observed image features 902 may be identified and aligned into a data structure that groups similar features. As an example, the multiview features 904 may be a multidimensional array of the features, with a first dimension corresponding to the pose and a second dimension corresponding to the feature.

[0094] In some aspects, the multiview features 904 are input into a combiner 906 to join corresponding features into at least one multidimensional representation 908 of the observed image features. For example, the multidimensional representation 908 can be an array of various observed image features.

[0095] The multidimensional representation 908 are provided to a queryformer 910 that is trained to identify the salient features based on a plurality of weights 912. For example, the queryformer 910 may reduce the number of features based on duplication or identify features that do not significantly vary with respect to a change in position. As noted above, the queryformer 910 uses attention mechanisms (e.g., self-attention, cross-attention) to identify the salient features and generates the key multiview features 914 that represent the important features from the sparse images at different views.

[0096] The key multiview features 914 and the observed image features 902 are provided to a diffusion model 916. The diffusion model 916 is configured to generate a latent representation of the various features (e.g., the receive observed image features 902 and the key multiview features 914) that can be decoded by a decoder 918 into an image 920.

[0097] In one illustrative aspect, the diffusion model 916 may include a U-Net machine learning model. In some aspects, a U-Net model comprises a contacting path and an expanding path. The contracting path (e.g., contracting path 504) is configured to extract meaningful information (e.g., features) from each stage (e.g., convolution layers, pooling layers, and, activation layers). The expanding path (e.g., expansive path 505) expands and combines the features and spatial information through a sequence of up-convolutions and concatenations with features from the contracting path. For example, a U-Net module includes a transposed convolution in addition to the pooling and activation layer to expand and generate the feature maps. The transposed convolution receives information from the extraction to assist in inferringQualcomm Ref. No.2400654WO information. In some aspects, the U-net can be combined with diffusion principles to extract and recreate features.

[0098] For example, the diffusion model 916 can use the key multiview features 914 and the observed image features 902 to infer the features of the subject (e.g., of the images) at an arbitrary pose.

[0099] In some aspects, the diffusion model 916 generates tokens within a latent space corresponding to the subject at the arbitrary pose. The decoder 918 is configured to reconstruct the image 920 from the learned feature representations obtained by the encoder. The decoder 918 is configured to take learned features and generate the image 920 that should ideally closely resemble the original input data based on the arbitrary pose. By doing so, the decoder 918 preserves essential information and outputs a high-fidelity representation of the subject of the images (e.g., the subject of the images 802) in the image 920.

[0100] In some aspects, the multiview engine 900 may be trained to learn weights to learn multiview fusion features. For example, the queryformer 910 is configured to learn weights 912 that emphasize features across multiple views and allow the projection of those features into the image 920.

[0101] In some cases, the diffusion model 916 may comprise frozen attention modules (e.g., static attention modules) that are not trained while training the queryformer 910. For example, the diffusion model 916 may be a tuned machine learning model that is configured to generate images based on the input. By freezing the attention modules, the training emphasizes forcing the queryformer 910 and the diffusion model 916 to learn to fuse multiview features. For example, the multiview engine 900 may uniformly sample pose-image tokens from all images (e.g., the images 802).

[0102] The multiview engine 900 may be trained based on a 3D dataset including different models generated by different artists of varying subjects. In this case, each 3D model, such as a GL binary transmission format file, can be rendered by a batch image generation system. In some aspects, each 3D model from the dataset can be rendered into multiple 2D images at different camera positions to reinforce learning of arbitrary features. In one aspect, a raytracing engine isQualcomm Ref. No.2400654WO configured to randomly render a plurality of views (e.g., twelve (12) views, fifteen (15) views, or other number of views) of an object to create a training set. At training time, two images from the rendered views may be randomly selected for each object and provided to a training system for learning single-view features and multiview features.

[0103] FIG. 10 is a block diagram illustrating another example of a multiview image generative machine learning model in accordance with some aspects. In some aspects, a multiview engine 1000 may include a single stage that uses positional encoding to learn multiview features. For example, a plurality of images 1002 may be input into an MAE 1004 to extract relevant image features from the images 1002. The image feature and provided to a positional encoder 1006 for injecting information about the positions of various image features identified by the MAE 1004. For example, the positional encoder 1006 assigns unique, learnable embeddings to each position in the image features to enable a machine learning model to utilize the order and relative positions of image features.

[0104] The tokens output from the positional encoder 1006 are combined (e.g., concatenated) with pose information 1008 associated with each corresponding image at a combiner 1010, and the image-pose features from the combiner 1010 are then provided to a perceiver 1012. In some aspects, the perceiver 1012 combines a versatile and scalable attention mechanism with learnable permutation equivariant functions to handle multiple domains of data in a unified framework. For example, the perceiver 1012 can use the positional encoding and the pose information 1008 to fuse the image features into multiview features. The perceiver 1012 comprises a multi-head self- attention mechanism for efficiently capturing complex relationships and patterns across different modalities.

[0105] In some aspects, the perceiver 1012 uses weights 1014 that are learned (e.g., during backpropagation) during training to generate the multiview features at runtime (e.g., during inference). In some aspects, the multiview features are normalized in a normalization layer 1016 and then provided into a diffusion model 1018. The diffusion model 1018 comprises image generation functions based on the multiview features and is trained by freezing attention models during training to cause the perceiver 1012 to learn the multiview features. During inference, the diffusion model 1018 receives an arbitrary pose and the diffusion model 1018 generates featuresQualcomm Ref. No.2400654WO in latent space associated with the subject of the images 1002 at the arbitrary pose. A decoder 1022 is configured to decode the latent space features into an image 1024.

[0106] In some aspects, the multiview engine 1000 is configured to generate high-fidelity images of the subject of the images 1002 at arbitrary poses. In particular, the multiview engine 1000 (and the single view engine 800) uses a sparse number of images to learn multiview features that are decoded into an accurate representation of the subject at an arbitrary position.

[0107] In some cases, training of one or more of the machine learning networks described herein (e.g., such as the generative machine learning model 600 of FIG.6, the multiview image generative machine learning model 700 of FIG. 7, among various other machine learning networks) can be performed using online training, offline training, and / or various combinations of online and offline training. In some cases, online may refer to time periods during which the input data (e.g., such as the images 602, 604, 608 of FIG.6, the pose information 608 of FIG.6, etc.) is processed, for instance for the view synthesis processing implemented by the systems and techniques described herein. In some examples, offline may refer to idle time periods or time periods during which input data is not being processed. Additionally, offline may be based on one or more time conditions (e.g., after a particular amount of time has expired, such as a day, a week, a month, etc.) and / or may be based on various other conditions such as network and / or server availability, etc., among various others. In some aspects, offline training of a machine learning model (e.g., a neural network model) can be performed by a first device (e.g., a server device) to generate a pre-trained model, and a second device can receive the trained model from the second device. In some cases, the second device (e.g., a mobile device, an XR device, a vehicle or system / component of the vehicle, or other device) can perform online (or on-device) training of the pre-trained model to further adapt or tune the parameters of the model.

[0108] As described herein, according to some aspects, instead of training all parameters, an efficient training strategy can be utilized by splitting the entire training procedure of any of the machine learning models described herein (e.g., the generative machine learning model 600 of FIG.6, the multiview image generative machine learning model 700 of FIG.7, etc.) into two stages with the same architecture. For example, in a first stage, the training can focus on learning a quality 3D representation, where only a single view is fed into the model (e.g., at the single view engineQualcomm Ref. No.2400654WO 710 of FIG.7). In a second stage, the training can focus on multi-view information fusion, in which case multi-view images are available (e.g., at the multiview engine 720 of FIG.7). At the second stage, the model has capacity for learning a 3D representation from source-view images, and related parameters can be frozen. By only finetuning necessary parameters in the second stage, the model can be trained efficiently.

[0109] In some cases, at the first stage (e.g., for single-view finetuning), the machine learning model can be built on stable-diffusion model. In such cases, it may be assumed that U-Net has an ability to produce realistic images, in which case only the attention modules or engines in the U- Net can be finetuned. In some examples, in the first stage, the entire 2D lifting and attention modules or engines in U-Net can be finetuned with a single source-view image as input. In some cases, at the second stage (e.g., for multi-view finetuning) after the first stage training is performed, multiple source-view images can be fed as input (e.g., in the multiview engine 720 of FIG. 7) to enable multi-view fusion, as described herein. After the first stage and once the second stage is performed, the model architecture already has the ability of producing realistic images, in which case the training (e.g., finetuning) can be performed on the modules or engines related to multi- view fusion (e.g., the parameters, such as weights, of the attention modules in the U-Net can be frozen as they are responsible for image generation, the parameters of the visual transformer architecture in the 2D lifting module or engine can be frozen as they are used to extract 3D representation from single-view image, etc.). Using such a strategy, the training can be focused on multi-view fusion while greatly reducing trainable parameters. In some aspects, to constrain the training burden, instead of feeding image tokens from all source-view images, image tokens can be constrained with a fixed number (e.g., by uniformly sampling posed image tokens from all source-view images (e.g., as shown in FIG.7, FIG.8, etc.). Using such a design can improve the robustness of the machine learning architectures described herein when only partial information is available.

[0110] FIG.11 is a flow diagram illustrating a method or process for generating images based on sparse images using a multiview image generative machine learning model in accordance with some aspects. The method 1100 can be performed by a computing device (or system) or computing component or system (e.g., a chipset, one or more processors (e.g., CPU(s), GPU(s), NPU(s), DSP(s), etc.), one more machine learning systems such as a neural network model, etc.) of theQualcomm Ref. No.2400654WO computing device. The computing device can include a mobile wireless communication device, a vehicle (e.g., an autonomous or semi-autonomous vehicle, a wireless-enabled vehicle, and / or other type of vehicle), a robot device or system (e.g., for residential or manufacturing purposes), a camera, an extended reality (XR) device, or another computing device or computing component or system of the device. In one illustrative example, a computing system (e.g., computing system 1200) can be configured to perform all or part of the method 1100.

[0111] At block 1102, the computing device (e.g., the computing system 1200) may determine, using an ML model, a plurality of 3D features from one or more 2D images based on respective pose information associated with each 2D image of the one or more 2D images. In some aspects, the respective pose information is relative to a target pose of a target 2D image. The ML model is configured to extract the plurality of 3D features. For example, 3D features of a subject of the one or more 2D images can be extracted from each image.

[0112] In some aspects, the computing device, to determine the 3D features at block 1102, may determine a first plurality of 2D features for a first 2D image of the one or more 2D images based on a plurality of patches of the first 2D image. For example, the computing device can separate each 2D image into a patch (e.g., 32x32 pixels) and identify features in each patch. The computing device may combine pose information associated with the first 2D image and the first plurality of 2D features to generate combined features for the first 2D image. In one example, the pose information associated with the 2D image is combined with (e.g., concatenated) the features identified in the patches. The computing device may process the combined features using the ML model to generate a first portion of the plurality of 3D features for the first 2D image (e.g., any image of the one or more 2D images).

[0113] At block 1104, the computing device may combine, using the ML model, the plurality of 3D features into a plurality of multiview features.

[0114] In some aspects, the computing device may further apply, using the ML model, an attention function on the first portion of the plurality of 3D features to identify key features from the first portion of the plurality of 3D features. For example, the key features are identified based on the combined features and tokens that are learned during training of the ML model. In some aspects, the tokens may also be referred to as hyperparameters, parameters, or weights. The MLQualcomm Ref. No.2400654WO model may comprise one or more attention layers configured to apply the attention function. For example, the ML model can include a queryformer including attention layers.

[0115] In some aspects, the ML model is configured to combine key features from each 2D image of the one or more 2D images into the plurality of multiview features. For example, a related features identified in the different 2D images can be provided in a data structure format that aligns the related features. The computing device may apply, using the ML model, an attention function on the plurality of multiview features to identify key multiview features from the plurality of multiview features.In some aspects, the ML model comprises one or more attention layers configured to apply the attention function. For example, the ML model can include a queryformer that include the one or more attention layers.

[0116] In some aspects, the ML model may include a perceiver including one or more attention layers for generating multiview features from the extracted image features. For example, the computing system may add, using the ML model, positional embeddings to the plurality of 3D features. The computing system may identify the key multiview features from the plurality of multiview features based on the positional embeddings.

[0117] At block 1106, the computing device may generate the target 2D image including the target pose based on the plurality of 3D features and the plurality of multiview features. The target 2D image is generated using a second ML model configured to generate one or more target images based on key features identified using the ML model from the one or more 2D images and key multiview features identified using the ML model from the plurality of multiview features. For example, the second ML model is a diffusion model.

[0118] In some examples, the methods described herein (e.g., method 1100, and / or other method described herein) may be performed by a computing device or apparatus. In one example, the method 1100 can be performed by a computing device having a computing architecture of the computing system 1200 shown in FIG.12. The devices or apparatuses configured to perform the operations of the method 1100 and / or other processes described herein may include a processor, microprocessor, microcomputer, or other component of a device that is configured to carry out the steps of the method 1100 and / or other process. In some examples, such devices or apparatuses may include one or more sensors configured to capture image data and / or other sensor measurements.Qualcomm Ref. No.2400654WO In some examples, such computing device or apparatus may include one or more sensors and / or a camera configured to capture one or more images or videos. In some cases, such device or apparatus may include a display for displaying images. In some examples, the one or more sensors and / or camera are separate from the device or apparatus, in which case the device or apparatus receives the sensed data. Such device or apparatus may further include a network interface configured to communicate data.

[0119] The method 1100 is illustrated as a logical flow diagram, the operation of which represents a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer- executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the methods.

[0120] The method 1100, and / or other method or process described herein may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware, or combinations thereof. As noted above, the code may be stored on a computer-readable or machine- readable storage medium, for example, in the form of a computer program including a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0121] FIG. 12 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. In particular, FIG. 12 illustrates an example of computing system 1200, which can be for example any computing device making up internal computing system, a remote computing system, a camera, or any component thereof in which the components of the system are in communication with each other using connection 1205. Connection 1205 can be a physical connection using a bus, or a direct connection into processor 1210, such as in aQualcomm Ref. No.2400654WO chipset architecture. Connection 1205 can also be a virtual connection, networked connection, or logical connection.

[0122] In some aspects, computing system 1200 is a distributed system in which the functions described in this disclosure can be distributed within a datacenter, multiple data centers, a peer network, etc. In some aspects, one or more of the described system components represents many such components each performing some or all of the function for which the component is described. In some aspects, the components can be physical or virtual devices.

[0123] Example computing system 1200 includes at least one processing unit (CPU or processor) 1210 and connection 1205 that couples various system components including system memory 1215, such as ROM 1220 and RAM 1225 to processor 1210. Computing system 1200 can include a cache 1212 of high-speed memory connected directly with, in close proximity to, or integrated as part of processor 1210.

[0124] Processor 1210 can include any general purpose processor and a hardware service or software service, such as services 1232, 1234, and 1236 stored in storage device 1230, configured to control processor 1210 as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor 1210 may essentially be a completely self- contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

[0125] To enable user interaction, computing system 1200 includes an input device 1245, which can represent any number of input mechanisms, such as a microphone for speech, a touch- sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech, etc. Computing system 1200 can also include output device 1235, which can be one or more of a number of output mechanisms. In some instances, multimodal systems can enable a user to provide multiple types of input / output to communicate with computing system 1200. Computing system 1200 can include communications interface 1240, which can generally govern and manage the user input and system output. The communication interface may perform or facilitate receipt and / or transmission wired or wireless communications using wired and / or wireless transceivers, including those making use of an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, a fiber optic port / plug,Qualcomm Ref. No.2400654WO a proprietary wired port / plug, a Bluetooth® wireless signal transfer, a BLE wireless signal transfer, an IBEACON® wireless signal transfer, an RFID wireless signal transfer, near-field communications (NFC) wireless signal transfer, dedicated short range communication (DSRC) wireless signal transfer, 802.11 WiFi wireless signal transfer, WLAN signal transfer, Visible Light Communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), IR communication wireless signal transfer, Public Switched Telephone Network (PSTN) signal transfer, Integrated Services Digital Network (ISDN) signal transfer, 3G / 4G / 5G / LTE cellular data network wireless signal transfer, ad-hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or some combination thereof. The communications interface 1240 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers that are used to determine a location of the computing system 1200 based on receipt of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US- based GPS, the Russia-based Global Navigation Satellite System (GLONASS), the China-based BeiDou Navigation Satellite System (BDS), and the Europe-based Galileo GNSS. There is no restriction on operating on any particular hardware arrangement, and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

[0126] Storage device 1230 can be a non-volatile and / or non-transitory and / or computer- readable memory device and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, a floppy disk, a flexible disk, a hard disk, magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, flash memory, memristor memory, any other solid-state memory, a compact disc read only memory (CD-ROM) optical disc, a rewritable compact disc (CD) optical disc, digital video disk (DVD) optical disc, a blu-ray disc (BDD) optical disc, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Stick® card, a smartcard chip, a EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamicQualcomm Ref. No.2400654WO RAM (DRAM), ROM, programmable read-only memory (PROM), erasable programmable read- only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random-access memory (RRAM / ReRAM), phase change memory (PCM), spin transfer torque RAM (STT- RAM), another memory chip or cartridge, and / or a combination thereof.

[0127] The storage device 1230 can include software services, servers, services, etc., that when the code that defines such software is executed by the processor 1210, it causes the system to perform a function. In some aspects, a hardware service that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor 1210, connection 1205, output device 1235, etc., to carry out the function. The term “computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing, containing, or carrying instruction(s) and / or data. A computer-readable medium may include a non-transitory medium in which data can be stored and that does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as CD or DVD, flash memory, memory or memory devices. A computer-readable medium may have stored thereon code and / or machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, or the like.

[0128] In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other component(s) that are configured to carry out the steps of methods described herein. In some examples, the computing device may include a display, one or more network interfacesQualcomm Ref. No.2400654WO configured to communicate and / or receive the data, any combination thereof, and / or other component(s). The one or more network interfaces can be configured to communicate and / or receive wired and / or wireless data, including data according to the 3G, 4G, 5G, and / or other cellular standard, data according to the Wi-Fi (802.11x) standards, data according to the BluetoothTMstandard, data according to the IP standard, and / or other types of data.

[0129] The components of the computing device can be implemented in circuitry. For example, the components can include and / or can be implemented using electronic circuits or other electronic hardware, which can include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or can include and / or be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein.

[0130] In some aspects the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.

[0131] Specific details are provided in the description above to provide a thorough understanding of the aspects and examples provided herein. However, it will be understood by one of ordinary skill in the art that the aspects may be practiced without these specific details. For clarity of explanation, in some instances the present technology may be presented as including individual functional blocks including functional blocks including devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Additional components may be used other than those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the aspects in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the aspects.

[0132] Individual aspects may be described above as a process or method which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operationsQualcomm Ref. No.2400654WO can be performed in parallel or concurrently. In addition, the order of the operations may be re- arranged. A process is terminated when its operations are completed but may have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination can correspond to a return of the function to the calling function or the main function.

[0133] Processes and methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer- readable media. Such instructions can include, for example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or a processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to described examples include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on.

[0134] Devices implementing processes and methods according to these disclosures can include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and can take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks (e.g., a computer-program product) may be stored in a computer-readable or machine-readable medium. A processor(s) may perform the necessary tasks. Typical examples of form factors include laptops, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rackmount devices, standalone devices, and so on. Functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example.Qualcomm Ref. No.2400654WO

[0135] The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functions described in the disclosure.

[0136] In the foregoing description, aspects of the application are described with reference to specific aspects thereof, but those skilled in the art will recognize that the application is not limited thereto. Thus, while illustrative aspects of the application have been described in detail herein, it is to be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. Various features and aspects of the above-described application may be used individually or jointly. Further, aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and drawings are, accordingly, to be regarded as illustrative rather than restrictive. For the purposes of illustration, methods were described in a particular order. It should be appreciated that in alternate aspects, the methods may be performed in a different order than that described.

[0137] One of ordinary skill will appreciate that the less than (“<”) and greater than (“>”) symbols or terminology used herein can be replaced with less than or equal to (“^”) and greater than or equal to (“^”) symbols, respectively, without departing from the scope of this description.

[0138] Where components are described as being “configured to” perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., microprocessors, or other suitable electronic circuits) to perform the operation, or any combination thereof.

[0139] The phrase “coupled to” refers to any component that is physically connected to another component either directly or indirectly, and / or any component that is in communication with another component (e.g., connected to the other component over a wired or wireless connection, and / or other suitable communication interface) either directly or indirectly.Qualcomm Ref. No. 2400654WO

[0140] Claim language or other language reciting “at least one of” a set and / or “one or more” of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting “at least one of A and B” or “at least one of A or B” means A, B, or A and B. In another example, claim language reciting “at least one of A, B, and C” or “at least one of A, B, or C” means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any duplicate information or data (e.g., A and A, B and B, C and C, A and A and B, and so on), or any other ordering, duplication, or combination of A, B, and C. The language “at least one of” a set and / or “one or more” of a set does not limit the set to the items listed in the set. For example, claim language reciting “at least one of A and B” or “at least one of A or B” may mean A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases “at least one” and “one or more” are used interchangeably herein.

[0141] Claim language or other language reciting “at least one processor configured to,” “at least one processor being configured to,” “one or more processors configured to,” “one or more processors being configured to,” or the like indicates that one processor or multiple processors (in any combination) can perform the associated operation(s). For example, claim language reciting “at least one processor configured to: X, Y, and Z” means a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each tasked with a certain subset of operations X, Y, and Z such that together the multiple processors perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, claim language reciting “at least one processor configured to: X, Y, and Z” can mean that any single processor may only perform at least a subset of operations X, Y, and Z.

[0142] Where reference is made to one or more elements performing functions (e.g., steps of a method), one element may perform all functions, or more than one element may collectively perform the functions. When more than one element collectively performs the functions, each function need not be performed by each of those elements (e.g., different functions may be performed by different elements) and / or each function need not be performed in whole by only one element (e.g., different elements may perform different sub-functions of a function). Similarly, where reference is made to one or more elements configured to cause another element (e.g., an apparatus) to perform functions, one element may be configured to cause the other element toQualcomm Ref. No. 2400654WO perform all functions, or more than one element may collectively be configured to cause the other element to perform the functions.

[0143] Where reference is made to an entity (e.g., any entity or device described herein) performing functions or being configured to perform functions (e.g., steps of a method), the entity may be configured to cause one or more elements (individually or collectively) to perform the functions. The one or more components of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more (or all) of the functions, and / or any combination thereof. Where reference to the entity performing functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to collectively perform the functions. When the entity is configured to cause more than one component to collectively perform the functions, each function need not be performed by each of those components (e.g., different functions may be performed by different components) and / or each function need not be performed in whole by only one component (e.g., different components may perform different sub-functions of a function).

[0144] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0145] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices such as general purposes computers, wireless communication device handsets, or integrated circuit devices having multiple uses including application in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discreteQualcomm Ref. No.2400654WO but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium including program code including instructions that, when executed, performs one or more of the methods described above. The computer- readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as RAM such as synchronous dynamic random access memory (SDRAM), ROM, non- volatile random access memory (NVRAM), EEPROM, flash memory, magnetic or optical data storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as propagated signals or waves.

[0146] The program code may be executed by a processor, which may include one or more processors, such as one or more DSPs, general purpose microprocessors, an application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general purpose processor may be a microprocessor; but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure, any combination of the foregoing structure, or any other structure or apparatus suitable for implementation of the techniques described herein.

[0147] Illustrative Aspects of the present disclosure include:

[0148] Aspect 1. An apparatus for generating a 3D model. The apparatus includes one or more memories configured to store one or more 2D images and one or more processors coupled to the one or more memories. The one or more processors are configured to: determine, using an ML model, a plurality of 3D features from the one or more 2D images based on respective pose information associated with each 2D image of the one or more 2D images, wherein the respectiveQualcomm Ref. No.2400654WO pose information is relative to a target pose of a target 2D image; combine, using the ML model, the plurality of 3D features into a plurality of multiview features; and generate the target 2D image including the target pose based on the plurality of 3D features and the plurality of multiview features.

[0149] Aspect 2. The apparatus of Aspect 1, wherein the ML model is configured to extract the plurality of 3D features.

[0150] Aspect 3. The apparatus of any of Aspects 1 or 2, wherein the one or more processors are configured to: determine a first plurality of 2D features for a first 2D image of the one or more 2D images based on a plurality of patches of the first 2D image; combine pose information associated with the first 2D image and the first plurality of 2D features to generate combined features for the first 2D image; and process the combined features using the ML model to generate a first portion of the plurality of 3D features for the first 2D image.

[0151] Aspect 4. The apparatus of any of Aspects 1 to 3, wherein the one or more processors are configured to: apply, using the ML model, an attention function on the first portion of the plurality of 3D features to identify key features from the first portion of the plurality of 3D features.

[0152] Aspect 5. The apparatus of any of Aspects 1 to 4, wherein the key features are identified based on the combined features and tokens that are learned during training of the ML model.

[0153] Aspect 6. The apparatus of any of Aspects 1 to 5, wherein the ML model comprises one or more attention layers configured to apply the attention function.

[0154] Aspect 7. The apparatus of any of Aspects 1 to 6, wherein the ML model is configured to combine key features from each 2D image of the one or more 2D images into the plurality of multiview features.

[0155] Aspect 8. The apparatus of any of Aspects 1 to 7, wherein the one or more processors are configured to: apply, using the ML model, an attention function on the plurality of multiview features to identify key multiview features from the plurality of multiview features.Qualcomm Ref. No.2400654WO

[0156] Aspect 9. The apparatus of any of Aspects 1 to 8, wherein the ML model comprises one or more attention layers configured to apply the attention function.

[0157] Aspect 10. The apparatus of any of Aspects 1 to 9, wherein the one or more processors are configured to: add, using the ML model, positional embeddings to the plurality of 3D features.

[0158] Aspect 11. The apparatus of any of Aspects 1 to 10, wherein the one or more processors are configured to: identify the key multiview features from the plurality of multiview features based on the positional embeddings.

[0159] Aspect 12. The apparatus of any of Aspects 1 to 11, wherein the one or more processors are configured to generate the target 2D image using a second ML model, the second ML model configured to generate one or more target images based on key features identified using the ML model from the one or more 2D images and key multiview features identified using the ML model from the plurality of multiview features.

[0160] Aspect 13. The apparatus of any of Aspects 1 to 12, wherein the second ML model is a diffusion model.

[0161] Aspect 14. The apparatus of any of Aspects 1 to 13, further comprising at least one camera configured to capture the one or more 2D images.

[0162] Aspect 15. A method of generating a 3D model, comprising: determining, using an ML model, a plurality of 3D features from one or more 2D images based on respective pose information associated with each 2D image of the one or more 2D images, wherein the respective pose information is relative to a target pose of a target 2D image; combining, using the ML model, the plurality of 3D features into a plurality of multiview features; and generating the target 2D image including the target pose based on the plurality of 3D features and the plurality of multiview features.

[0163] Aspect 16. The method of Aspect 15, wherein the ML model is configured to extract the plurality of 3D features.Qualcomm Ref. No.2400654WO

[0164] Aspect 17. The method of any of Aspects 15 or 16, wherein determining the plurality of 3D features comprises: determining a first plurality of 2D features for a first 2D image of the one or more 2D images based on a plurality of patches of the first 2D image; combining pose information associated with the first 2D image and the first plurality of 2D features to generate combined features for the first 2D image; and process the combined features using the ML model to generate a first portion of the plurality of 3D features for the first 2D image.

[0165] Aspect 18. The method of any of Aspects 15 to 17, wherein processing the combined features using the ML model comprises: applying, using the ML model, an attention function on the first portion of the plurality of 3D features to identify key features from the first portion of the plurality of 3D features.

[0166] Aspect 19. The method of any of Aspects 15 to 18, wherein the key features are identified based on the combined features and tokens that are learned during training of the ML model.

[0167] Aspect 20. The method of any of Aspects 15 to 19, wherein the ML model comprises one or more attention layers configured to apply the attention function.

[0168] Aspect 21. The method of any of Aspects 15 to 20, wherein the ML model is configured to combine key features from each 2D image of the one or more 2D images into the plurality of multiview features.

[0169] Aspect 22. The method of any of Aspects 15 to 21, further comprising: applying, using the ML model, an attention function on the plurality of multiview features to identify key multiview features from the plurality of multiview features.

[0170] Aspect 23. The method of any of Aspects 15 to 22, wherein the ML model comprises one or more attention layers configured to apply the attention function.

[0171] Aspect 24. The method of any of Aspects 15 to 23, further comprising: adding, using the ML model, positional embeddings to the plurality of 3D features.Qualcomm Ref. No.2400654WO

[0172] Aspect 25. The method of any of Aspects 15 to 24, further comprising: identifying the key multiview features from the plurality of multiview features based on the positional embeddings.

[0173] Aspect 26. The method of any of Aspects 15 to 25, wherein the target 2D image is generated using a second ML model configured to generate one or more target images based on key features identified using the ML model from the one or more 2D images and key multiview features identified using the ML model from the plurality of multiview features.

[0174] Aspect 27. The method of any of Aspects 15 to 26, wherein the second ML model is a diffusion model.

[0175] Aspect 28. A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to perform operations according to any of Aspects 15 to 27.

[0176] Aspect 29. An apparatus for processing one or more images, comprising one or more means for performing operations according to any of Aspects 15 to 27.

Claims

Qualcomm Ref. No.2400654WO CLAIMS WHAT IS CLAIMED IS:

1. An apparatus for generating a three-dimensional (3D) model, comprising: one or more memories configured to store one or more two-dimensional (2D) images; and one or more processors coupled to the one or more memories and configured to: determine, using a machine learning (ML) model, a plurality of 3D features from the one or more 2D images based on respective pose information associated with each 2D image of the one or more 2D images, wherein the respective pose information is relative to a target pose of a target 2D image; combine, using the ML model, the plurality of 3D features into a plurality of multiview features; and generate the target 2D image including the target pose based on the plurality of 3D features and the plurality of multiview features.

2. The apparatus of claim 1, wherein the ML model is configured to extract the plurality of 3D features.

3. The apparatus of claim 1, wherein the one or more processors are configured to: determine a first plurality of 2D features for a first 2D image of the one or more 2D images based on a plurality of patches of the first 2D image; combine pose information associated with the first 2D image and the first plurality of 2D features to generate combined features for the first 2D image; and process the combined features using the ML model to generate a first portion of the plurality of 3D features for the first 2D image.

4. The apparatus of claim 3, wherein the one or more processors are configured to: apply, using the ML model, an attention function on the first portion of the plurality of 3D features to identify key features from the first portion of the plurality of 3D features.Qualcomm Ref. No.2400654WO 5. The apparatus of claim 4, wherein the key features are identified based on the combined features and tokens that are learned during training of the ML model.

6. The apparatus of claim 4, wherein the ML model is configured to combine key features from each 2D image of the one or more 2D images into the plurality of multiview features.

7. The apparatus of claim 1, wherein the one or more processors are configured to: apply, using the ML model, an attention function on the plurality of multiview features to identify key multiview features from the plurality of multiview features.

8. The apparatus of claim 7, wherein the one or more processors are configured to: add, using the ML model, positional embeddings to the plurality of 3D features.

9. The apparatus of claim 8, wherein the one or more processors are configured to: identify the key multiview features from the plurality of multiview features based on the positional embeddings.

10. The apparatus of claim 1, wherein the one or more processors are configured to use a second ML model to generate the target 2D image, the second ML model configured to generate one or more target images based on key features identified using the ML model from the one or more 2D images and key multiview features identified using the ML model from the plurality of multiview features.

11. The apparatus of claim 1, further comprising at least one camera configured to capture the one or more 2D images.

12. A method of generating a three-dimensional (3D) model, comprising: determining, using a machine learning (ML) model, a plurality of 3D features from one or more two-dimensional (2D) images based on respective pose information associated with eachQualcomm Ref. No.2400654WO 2D image of the one or more 2D images, wherein the respective pose information is relative to a target pose of a target 2D image; combining, using the ML model, the plurality of 3D features into a plurality of multiview features; and generating the target 2D image including the target pose based on the plurality of 3D features and the plurality of multiview features.

13. The method of claim 12, wherein the ML model is configured to extract the plurality of 3D features.

14. The method of claim 12, wherein determining the plurality of 3D features comprises: determining a first plurality of 2D features for a first 2D image of the one or more 2D images based on a plurality of patches of the first 2D image; combining pose information associated with the first 2D image and the first plurality of 2D features to generate combined features for the first 2D image; and process the combined features using the ML model to generate a first portion of the plurality of 3D features for the first 2D image.

15. The method of claim 14, wherein processing the combined features using the ML model comprises: applying, using the ML model, an attention function on the first portion of the plurality of 3D features to identify key features from the first portion of the plurality of 3D features.

16. The method of claim 15, wherein the ML model is configured to combine key features from each 2D image of the one or more 2D images into the plurality of multiview features.

17. The method of claim 12, further comprising: applying, using the ML model, an attention function on the plurality of multiview features to identify key multiview features from the plurality of multiview features.Qualcomm Ref. No.2400654WO 18. The method of claim 17, further comprising: adding, using the ML model, positional embeddings to the plurality of 3D features.

19. The method of claim 18, further comprising: identifying the key multiview features from the plurality of multiview features based on the positional embeddings.

20. The method of claim 12, wherein the target 2D image is generated using a second ML model configured to generate one or more target images based on key features identified using the ML model from the one or more 2D images and key multiview features identified using the ML model from the plurality of multiview features.

Images