Generation of three-dimensional scenes from text prompts

Abstract

According to at least one implementation, a method includes identifying a prompt and generating an image from the prompt based on a first model. The method further includes processing the image to generate a three-dimensional scene of the image. The method also provides updating the three-dimensional scene from a first version to a second version based on a second model and the image.

Description

GENERATION OF THREE-DIMENSIONAL SCENESFROM TEXT PROMPTSBACKGROUND

[0001] Text-to-image synthesis is a process where a computer model creates images from textual descriptions. By using deep learning techniques, such as neural networks, the model interprets written (or voice-to-text) input and generates a visual representation of that scene. This technology blends language understanding with image generation to produce pictures that match the described content. The technology’ can be used for creative design, virtual reality, and automated media generation.SUMMARY

[0002] This disclosure relates to systems and methods that generate three-dimensional scenery using images generated from models. In at least one implementation, a system is configured to identify a text prompt. In some examples, the text prompt is provided via a user’s text input device or software. In some examples, the text prompt corresponds to a voice-to-text prompt from a microphone. The system can further be configured to generate an image from the text prompt based on a first model. In some examples, the first model includes a diffusion model, which is a generative model that creates images by simulating the gradual diffusion process. In some implementations, the diffusion model can start from noise and iteratively refine the noise to match the target image distribution. The system can then be configured to process the image to generate a three-dimensional scene (e.g., performing Gaussian splatting). The system is then configured to update the three-dimensional scene from a first version to a second version based on a second model and the image. The update can provide additional information for the 3D scene absent in the initial image.

[0003] In some aspects, the techniques described herein relate to a method including: identifying a text prompt; generating an image from the text prompt based on a first model; processing the image to generate a three-dimensional scene of the image; and updating the three-dimensional scene from a first version to a second version based on a second model and the image.

[0004] In some aspects, the techniques described herein relate to a computing apparatus including: at least one processor; a computer-readable storage medium operativelycoupled to the at least one processor; and program instructions stored on the computer- readable storage medium that, when executed by the at least one processor, cause the at least one processor to execute operations, the operations including: identifying a text prompt; generating an image from the text prompt based on a first model; processing the image to generate a three-dimensional scene of the image; and updating the three-dimensional scene from a first version to a second version based on a second model and the image.

[0005] In some aspects, the techniques described herein relate to a computer-readable storage medium having program instructions stored thereon that, when executed by at least one processor, cause the at least one processor to execute operations, the operations including: identifying a text prompt; generating an image from the text prompt based on a first model; processing the image to generate a three-dimensional scene of the image; and updating the three-dimensional scene from a first version to a second version based on a second model and the image, the second model including at least a diffusion model.

[0006] The accompanying drawings and the description below outline the details of one or more implementations. Other features will be apparent from the description, drawings, and claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] FIG. 1 illustrates a system for generating three-dimensional (3D) scenery from a text prompt according to an implementation.

[0008] FIG. 2 illustrates a method of operating a system to generate 3D scenery from a text prompt according to an implementation.

[0009] FIG. 3 illustrates an operational scenario of using a diffusion model to generate an image according to an implementation.

[0010] FIG. 4 illustrates an operational scenario of processing an image to generate a 3D scene according to an implementation.

[0011] FIG. 5 illustrates an operational scenario of updating a 3D scene for different perspectives according to an implementation.

[0012] FIG. 6 illustrates a computing system that generates 3D scenes from a text prompt according to an implementation.DETAILED DESCRIPTION

[0013] Computing devices can be configured to process text prompts and generate two-dimensional images associated with the text prompt. Image generation from textprompts, or text-to-image synthesis, is a process in which a machine learning model can create visual representations based on descriptive textual inputs. This can involve using advanced neural networks, such as Generative Adversarial Networks (GANs) or diffusion models, trained on large datasets of images and corresponding text descriptions. By understanding the semantic content of the text, the model generates images that accurately reflect the given descriptions, enabling the creation of customized visuals based on natural language inputs. This technology has applications in various fields, including art. design, advertising, and virtual reality, providing a powerful tool for creative and practical purposes. However, while generating two-dimensional (2D) images can be beneficial, technical problems exist in creating three-dimensional (3D) scenes that permit users to explore the scene from different angles and perspectives.

[0014] In at least one technical solution, a system comprising one or more computing devices or computers is configured to generate an image from a text prompt using a first model. The text prompt can be provided via a keyboard, a touch screen, a speech-to-text service, or other means. In at least one implementation, the first model that generates the image comprises a diffusion model. A diffusion model is a generative model used in machine learning and statistics to describe how data points evolve over time by adding random noise, simulating a diffusion process. In the context of image generation, diffusion models start with a noisy image and iteratively denoise it, gradually refining the image to match a desired data distribution. These models are typically trained to reverse the diffusion process, learning to predict the noise added at each step and progressively reconstructing the original data.

[0015] For example, a user can input text to generate a dining room with six chairs. In response to the request, the diffusion model provided by the system can generate an image corresponding to the text input. Thus, the system can start with a noisy image and gradually refine it, guided by the encoded features of the text. This is a process called denoising. Over multiple iterations, the model can produce an image depicting a dining room with six chairs, accurately reflecting the user’s description. Although a diffusion model is demonstrated as the first model for generating the image, the system can be configured to use GANs or some other text-to-image synthesis model.

[0016] After the image is generated, the system is further configured to process the image to generate a 3D scene. In some implementations, the system can use Gaussian splatting to create the 3D scene of the image. The system can be configured to generate a point cloud from the image using modeled information about depth, size, orientation, and thelike associated with the objects in the image. A point cloud is a collection of data points in a 3D space, representing the external surface of an object or scene. Each point in the cloud has coordinates (x, y, z) and may contain additional information such as color, intensity, or normal vectors. For example, points can be used to define the locations, in space, of the table and chairs within the dining room. The system is then configured to perform three- dimensional (3D) Gaussian splatting on the point cloud. 3D Gaussian splatting is a technique used in computer graphics and visualization to represent and render point clouds by approximating each point as a Gaussian distribution rather than a single discrete point. This approach smooths the point cloud data by spreading each point's influence over a small region, defined by its Gaussian parameters (mean, variance, etc.), resulting in a continuous representation of the surface. Gaussian splatting can help to process noise, fill gaps, and provide a more visually appealing rendering of the 3D data. The 3D Gaussians are generated to reflect the information from the original image, including color and depth, from the generated image. However, at least one technical problem exists in refining the 3D scene and filling in portions of the 3D scene that are not included as part of the original image. For example, returning to the dining room example, the 3D scene may not include information for different perspectives or portions of the image that are not visible in the image generated by the first model.

[0017] As at least one technical solution, the system can be configured to update the 3D scene from a first version to a second version based on a second model and the image generated from the first model. In some implementations, the second model can include diffusion techniques or a diffusion model. The diffusion techniques are used to identify and fill in portions of the scene that could not be identified from the first image. For example, the system may not possess information about the backside of an object depicted in an image (e.g., the back of a chair facing the camera). The system can be configured to identify a new perspective for the scene and use diffusion techniques to generate imaging information for the new perspective based on the original image. The second model can provide imaging information for the Gaussian splatting to fill in portions missing from the first perspective. The diffusion model can generate portions of the 3D scene that were not visible from the first perspective by iteratively refining random noise through a learned process that progressively adds and removes noise to match the target distribution. Gaussian splatting operations then use the imaging information to update the 3D scene. Accordingly, diffusion techniques can create additional imaging information for scenes from different perspectives, while Gaussiansplatting operations add imaging information to 3D scenes. As a technical effect, as more information is generated from different perspectives, the 3D scene can be more realistic.

[0018] Various embodiments of the present technology provide various technical effects, advantages, and / or technical solutions for computing systems and components. For example, various implementations may include one or more of the following technical effects, advantages, and / or improvements: 1) non-routine and unconventional generation of a 3D scene from an image derived from text-to-image synthesis; and 2) non-routine and unconventional operations to use diffusion techniques and 3D Gaussian splatting to generate a 3D model; and 3) non-routine and unconventional operations to update a 3D scene using diffusion techniques.

[0019] FIG. 1 illustrates a system 100 for generating three-dimensional (3D) scenery from a text prompt according to an implementation. System 100 includes computing system 110, prompt 135, and scene 140. Computing system 110 includes prompt identification 120, text-to-image synthesis 121, 3D scene generation, and 3D scene update 123. Computing system 110 represents one or more computing devices, such as desktop computers, server computers, tablets, laptops, or other computing devices.

[0020] In system 100, computing system 110 identifies prompt 135 using prompt identification 120. Prompt 135 is provided to computing system 110 by inputting the desired text through a user interface, such as a command line, text box, or voice input system. Computing system 110 can capture the input using a keyboard, touchscreen, microphone, or other input device. The computer processes the input using natural language processing techniques to understand and respond to the prompt accordingly.

[0021] In at least one technical solution, computing system 110 provides text-to- image synthesis 121. Text-to-image synthesis 121 is a process in which deep learning models convert a textual description into a corresponding image. This can involve training generative models, such as Generative Adversarial Networks (GANs) or diffusion models, on large datasets that pair textual descriptions with images. The model leams to understand the semantic content of the text and generates visual representations that match the description. For example, a user can provide a prompt that requests a scene from a room with a large television. In response to the request, computing system 110 can be configured to process the prompt using NLP and generate an image 131 based on the learned data set. Specifically, generate a scene with at least the room and the large television. The scene can include other objects, such as seating, lighting, or other objects that fill the room based on the learned environments (e.g., pictures associated with word descriptors).

[0022] In some implementations, text-to-image synthesis 121 can provide a diffusion model to produce an image from prompt 135. A diffusion model is a type of deep learning model that synthesizes images from textual descriptions through a gradual denoising process. The model starts with a random noise image and iteratively refines it by removing noise (i.e., a denoising process), guided by the textual input. This process involves a sequence of learned denoising steps, where each step incrementally reduces the noise and enhances the image's features to align with the provided text. Some model components include an encoder to understand and embed the text and a denoiser network that uses these embeddings to steer the denoising process. These diffusion models can be configured to produce high-quality and detailed images that closely match the given textual descriptions. In some examples, the image can comprise a panorama or other image that captures a wide view of the corresponding environment.

[0023] Although demonstrated in the previous example using a diffusion model, computing system 110 can use other types of text-to-image synthesis. Examples of text-to- image synthesis models include GANs and the corresponding variants. A GAN is a class of machine learning frameworks designed to generate new data samples that resemble a given dataset. A GAN can consist of two neural networks: the generator and the discriminator. The generator creates synthetic data samples, while the discriminator evaluates them against real data samples, determining whether they are real or fake. These two networks are trained simultaneously in a zero-sum game, where the generator aims to produce increasingly realistic samples to fool the discriminator, and the discriminator aims to better distinguish real from fake samples. GANs can produce highly realistic images, videos, and other data types through this adversarial process.

[0024] Once image 131 is generated from prompt 135, computing system 110 is configured to provide 3D scene generation 122. 3D scene generation 122 is configured to use algorithms and machine learning techniques to construct a 3D model from a 2D image. This process can extract depth information, textures, and geometric details to create a realistic 3D representation of the scene depicted in the image. In some implementations, 3D scene generation 122 generates a point cloud from image 131. Generating a point cloud from a single image can involve using computer vision techniques and deep learning models to infer depth information from the 2D image. First, the image can be processed through a neural network, such as a convolutional neural network (CNN) or a generative adversarial network (GAN), trained on large datasets of images with known 3D structures. This network estimates the depth at each pixel, creating a depth map. The depth map (which can be combined withother parameters) is then used to project each pixel into 3D space, resulting in a point cloud. This point cloud represents the 3D structure of the scene by mapping the estimated depth values to coordinates in a 3D space. The point cloud can be derived from machine learning models that understand the depth associated with the different objects in the image.

[0025] Once the point cloud is generated, 3D scene generation 122 generates scene 140, representing a 3D version of the original image 131. Scene 140 can represent a 3D Gaussian splatting representation of the scene in some examples. 3D Gaussian splatting generates a scene from the point cloud by transforming each point into a Gaussian distribution, turning discrete points into continuous, volumetric blobs. Each point in the point cloud is assigned a Gaussian function characterized by parameters like position, variance, and orientation, which define its spread and influence in the 3D space. These Gaussians are then collectively rendered, with their overlapping contributions creating a smooth and continuous representation of the scene. This process involves summing the effects of all Gaussians at each point in the space, allowing for a detailed and coherent 3D reconstruction from the initially sparse point cloud data. The Gaussians reflect the originally generated image 131 as scene 140.

[0026] After scene 140 is generated, computing system 110 provides 3D scene update 123, which updates the scene from a first version to a second version based on a second model and image 131. In some implementations, computing system 110 encounters a technical problem of missing information from aspects of the environment that were not captured in the image. For example, while an image can include a first angle associated with a flower, different perspectives associated with the flower cannot be reflected in the 3D scene (e.g., color, shape, etc.). To support the update from the first version to the second version, the system can employ a diffusion model or diffusion operations that update the scene based on the original image 131 and the diffusion model for a different perspective or pose (e.g., a side view or at a different height). The different perspectives can be random, selected angles and locations in the scene, or from another perspective. As an example, a first perspective or pose can be used for the image to generate the original 3D scene, and a different perspective or pose can be selected for the 3D model in association with the image (e.g., a side view pose associated with the scene) to update the scene. The technical effect permits the diffusion model to update the 3D scene (i.e., Gaussian splatting) by predicting different portions of the scene from the context of the original image or prompt. The diffusion model updates the 3D Gaussian splat scene by leveraging the model’s capability to iteratively refine and enhance the representation of the scene. This iterative process allows the diffusion model to reducenoise, improve detail, and achieve a more accurate and visually appealing representation of the 3D scene by factoring information about the original image to identify information about additional perspectives in the 3D scene. Thus, using a combination of a diffusion model and 3D Gaussian splatting, computing system 110 can update the scene for different perspectives not captured in the original image. In some implementations, computing system 110 can be configured to identify at least one additional perspective different from the perspective from the original image and update the 3D Gaussian splats representing the scene based on the second model (e.g., diffusion model) and the image. For example, while the original image corresponds to the first perspective of the scene, additional perspectives can be selected for the scene, and the scene can be updated in view of the original image, the additional perspectives, and the second model. The second model can include diffusion techniques that use the existing context of the scene (e.g., image) and generate updates to provide a coherent scene from different angles and poses.

[0027] In at least one implementation, the diffusion model can iteratively refine the Gaussian splatting to incorporate multiple perspectives. The process begins with the initial 3D representation derived from the original image. The diffusion model then applies a series of probabilistic updates, gradually improving the splatting by predicting and correcting errors while considering the geometric consistency across various viewpoints. By incorporating information from the learned distribution of 3D structures and leveraging multi-view constraints, the model adjusts the Gaussian splats to ensure they align accurately with the visual information from different perspectives. This iterative refinement enhances the 3D reconstruction, making it more realistic when viewed from various angles. In some examples, the diffusion model may incorporate information from the prompt, and the originally generated image to determine how to denoise or smooth the Gaussians (i.e., Gaussian splats) included as part of the scene.

[0028] FIG. 2 illustrates a method 200 of operating a system to generate 3D scenery from a text prompt according to an implementation. Method 200 can be performed by one or more computing devices, such as computing system 110 of FIG. 1.

[0029] Method 200 includes identifying a text prompt at step 201. A computing device can receive the text prompt through various input methods, such as a keyboard, a touch screen, voice recognition, or text input from an application interface. For example, a text prompt can request that a scene be generated that includes a room with a large television. Method 200 further includes generating an image from the text prompt based on a first model at step 202. In some implementations, the first model represents a text-to-image synthesismodel. Text-to-image synthesis is a process in which a computer system generates images based on textual descriptions provided by a user. Utilizing advanced machine learning techniques, such as GANs or diffusion models, the system interprets the semantics of the input text and creates a visual representation that matches the description. This technology combines natural language processing to understand the text with image generation capabilities, enabling the creation of detailed and contextually appropriate images from simple textual prompts.

[0030] In some technical solutions, the first model comprises a diffusion model. The diffusion model generates an image by starting with a noisy image and iteratively refining it through a series of denoising steps. Random noise is initially added to a blank space or a rough image. The model, trained to reverse this noising process, progressively reduces the noise and refines the image at each step, guided by learned probability distributions that capture the underlying data structure. By iteratively applying these denoising transformations, the diffusion model transitions the noisy image towards a coherent, high-quality7image that aligns with the desired output, such as a text prompt. This iterative refinement leverages the model’s ability to predict and remove noise, resulting in a detailed and accurate image. Using the previous example of the room with the television, an image that includes a room and a television can be generated.

[0031] Method 200 further includes processing the image to generate the 3D scene of the image at step 203. In some implementations, the 3D scene comprises a 3D Gaussian Splatting representation of the scene. 3D Gaussian splatting can generate a 3D scene from the original image by first interpreting the depth and structure information from the 2D image. This involves using depth estimation techniques, sometimes including machine learning models trained on large datasets of 2D images and their corresponding 3D scenes, to infer the spatial arrangement of objects in the image. In some examples, the depth information is provided as a point cloud. A point cloud is a collection of data points defined in a 3D coordinate system, representing the external surface of an object or scene. Each point in the cloud can have X, Y, and Z coordinates and additional attributes like color or intensity, capturing the spatial geometry of the environment.

[0032] Once the depth information is obtained, Gaussian splats (points with associated Gaussian kernels representing volumetric information) are distributed throughout the inferred 3D space. These splats are then adjusted to accurately represent the surfaces and textures of objects, effectively transforming the 2D image into a 3D scene. This approach allows for a continuous and smooth scene representation.

[0033] Method 200 also includes updating the 3D scene from a first version to a second version based on a second model and the image at step 204. In some implementations, the second model comprises a diffusion model. The diffusion model can update the scene by iteratively refining the parameters of the Gaussian distributions (i.e., Gaussians) that represent the scene. Starting from an initial scene version, the diffusion model applies a series of transformations that progressively reduce noise and adjust the Gaussians’ positions, sizes, and orientations to better match the desired 3D structure. This can be achieved through a learned denoising process, where the model is trained on a large dataset of 3D scenes to understand how to refine the Gaussians effectively. The diffusion model can also incorporate information from the original image to capture information about different perspectives associated with the first image. The diffusion process can be configured to update information for the scene not necessarily captured in the original image (e.g., portions that were shielded or out of view in the first image). The diffusion model can provide additional imaging information for elements not viewable from the first image (e.g., the backside of an object in the original image). The system can be configured to generate new Gaussians or update previously existing Gaussians using a denoising process that predicts the imaging information at different perspectives to generate a smooth scene from different angles. The different perspectives can include perspectives of the scene that do not correspond to the perspective for the original image.

[0034] FIG. 3 illustrates an operational scenario 300 of using a diffusion model to generate an image according to an implementation. Operational scenario 300 includes prompt 335, prompt identification 320, noise 350, text-to-image synthesis 321, and image 351. Text-to-image synthesis further includes text processing 340 and denoise processing 341. Prompt identification 320 and text-to-image synthesis 321 can be performed or executed by one or more computers, such as computing system 1 10 of FIG. 1.

[0035] Operational scenario 300 includes executing prompt identification 320. Prompt 335 can represent a text prompt from a user or a voice prompt from a user. The prompt can be received via a physical keyboard, touchscreen, microphone, or through some other input device. Once the prompt is received, operational scenario 300 performs text-to- image synthesis 321. Text-to-image synthesis 321 includes text processing 340 that processes and parses the prompt 335 to understand the requested action. This involves analyzing the syntax and semantics of the command to determine information about the requested image. For example, the system can identify requested objects (e.g.. a television).can identify the orientation of the objects, can identify the size of the objects, or can identify some other information in association with the scene.

[0036] In some implementations, text-to-image synthesis provides denoise processing 341 as part of a diffusion model to generate image 351 from noise 350. Image 351 is generated based on based on the information from text processing 340. The diffusion model creates image 351 from text prompt 335 by iteratively refining a noisy initial image (noise 350) until it resembles the desired output. The process starts with a random noise image, then progressively denoised in steps. At each step, the model uses the text prompt to guide the transformation, applying learned patterns and features to gradually shape the image closer to what the prompt describes. This guidance involves translating textual information into visual elements, ensuring that the semantic content of the prompt is reflected in the evolving image. For example, if the user requests that a television be included in the image of a room, the diffusion model can denoise the image with the inclusion of the television.

[0037] The diffusion model is trained on a large dataset of images paired with textual descriptions, learning to associate specific words and phrases with visual patterns. During training, it leams to predict the next step in the denoising process, conditioned on the current noisy image and the text prompt. By iteratively applying these predictions, the model refines the image through successive stages of noise reduction, integrating the textual context at each step. The result is a coherent image that visually represents the input prompt 335, achieved through the sophisticated interplay of denoising and textual guidance.

[0038] Although demonstrated in the example of operational scenario 300 as generating an image using a diffusion process, other models can generate an image from a prompt. For example, a computer or computing system can generate an image from a prompt using a GAN. A GAN generates an image from a prompt by combining a generator model, which creates images, with a discriminator model, which evaluates their realism, in an adversarial training process. The generator takes a random noise vector and the textual prompt as inputs, using learned patterns to produce an image that aligns with the prompt. The discriminator assesses the generated image against real images, providing feedback to improve the generator’s output. This iterative process continues until the generator produces high-quality images that accurately reflect the prompt.

[0039] FIG. 4 illustrates an operational scenario 400 of processing an image to generate a 3D scene according to an implementation. Operational scenario 400 includes image 410. point cloud 412, 3D Gaussian, image to point cloud 420, and Gaussian splatting421. Image to point cloud 420 and Gaussian splatting 421 may be performed by one or more computers, such as computing systems 110 from FIG. 1.

[0040] In operational scenario 400, the application identifies image 410. In some examples, image 410 may be generated using text-to-image synthesis. Text-to-image synthesis is the process of generating images based on textual descriptions using machine learning models. These models, such as GANs or diffusion models, interpret the semantic content of the text and create corresponding visual representations. The models aim to produce realistic and coherent images that accurately reflect the details and context described in the text prompt. Once image 410 is identified, operational scenario 400 performs image to point cloud 420, which processes the image to generate point cloud 412. A point cloud is a collection of data points defined in a three-dimensional coordinate system, representing the external surface of an object or a scene. Each point in the cloud has coordinates (e.g., X, Y, and Z coordinates) and may include additional information such as color, intensity, and normal vectors, providing a detailed representation of the object’s shape and appearance.

[0041] Point cloud 412 can be generated in some implementations using techniques like depth estimation and 3D reconstruction. The process involves analyzing image 410 to estimate the depth of the pixels in the image. This can be accomplished using neural networks trained on large datasets of images with known depth information. By combining the estimated depths with the 2D coordinates of the image pixels, a 3D point cloud is created, representing the spatial structure of the scene. This point cloud can then be refined and processed to better approximate the 3D geometry depicted in the original image.

[0042] Once point cloud 412 is generated, operational scenario 400 further provides Gaussian splatting 421 to generate 3D Gaussian 414, representing a 3D scene. Gaussian splatting 421 is performed with point cloud 412 and image 410 through a process that blends the point cloud’s 3D data with the 2D image to generate a smooth, detailed representation of the scene. Each point in the point cloud is initially associated with a Gaussian distribution in 3D space, where the mean represents the point’s location, and the covariance defines its spread. In the splatting process, these Gaussians are projected onto the 2D image plane, contributing to the image by spreading their influence based on spatial and color properties. This projection combines the point cloud data with the image, creating a visually coherent representation that captures the details of the 3D structure while leveraging the 2D image’s textures and features. The result is 3D Gaussian 414, which can reflect objects and content described by a user in an original prompt.

[0043] FIG. 5 illustrates an operational scenario 500 of updating a 3D scene for different perspectives according to an implementation. Operational scenario 500 includes image 510, updated scene versions 512, and refine scene 520. Refine scene 520 represents an application or service that refines a 3D scene on one or more computing devices. Refine scene 520 is an example operation that can be performed by computing system 110 of FIG. 1 in some examples. Refine scene 520 includes diffusion processing 540 and Gaussian processing 541.

[0044] In operational scenario 500, the first version of the scene is generated from image 510 using Gaussian processing 541. Gaussian processing 541 performs 3D Gaussian splatting, which is used to project Gaussian distributions onto a 2D plane to create a smooth and detailed representation of a scene by blending depth and visual information. In some implementations, Gaussian splatting begins with depth estimation, which involves predicting the distance of each pixel from the camera to create a depth map. This depth map allows for reconstructing a 3D point cloud, where each point corresponds to a specific location in the scene. Each point in this cloud is associated with a Gaussian distribution, characterized by its mean (the point's 3D coordinates) and covariance (describing the spatial spread or uncertainty around that point).

[0045] In the Gaussian splatting process, these 3D Gaussians are projected onto a 2D image plane, where each Gaussian spreads its influence based on its parameters, effectively 'splatting' onto the image. This projection blends the point cloud's depth information with the original image's visual details, creating a 3D representation. The resulting scene can provide a smooth and detailed 3D reconstruction that includes the original image's texture and color fidelity while capturing the scene's spatial geometry. Specifically, using the example in operational scenario 500, image 510 includes a room with a television. Refine scene 520 and Gaussian processing 541 can identify depth information for the elements (e.g., the television and walls in image 510) and generate a point cloud corresponding to the various objects’ depth in the scene. Once the point cloud is established, Gaussian processing 541 is configured to generate Gaussian distributions that reflect the colors, shading, and other elements associated with the onginal image.

[0046] After the first version of the scene is generated using 3D Gaussian splatting, operational scenario 500 further executes diffusion processing 540 to generate second or updated scene versions 512. Diffusion processing 540 uses a diffusion model to update the Gaussians of the scene. In at least one implementation, the diffusion model is used to update portions of the scene that were not visible for the initial image or were in an area that was notvisible (e.g., a second surface hidden behind a first surface). In at least one example, starting with the first version of the 3D Gaussian distribution representing the scene, the model simulates noise and denoising steps to progressively enhance the splats’ accuracy. At each step, the mean and covariance of the Gaussian distributions are updated based on the diffusion process, which incorporates perspective information from various viewpoints. This ensures that the 3D Gaussian splats adapt to changes in perspective, creating a coherent and detailed multi-view representation of the scene. This iterative refinement enables the model to account for parallax and occlusions, enhancing the scene’s representation from different angles. Additionally, the diffusion model can use information from the original image and the prompt to provide context for the new information added to the various splats representing the scene. The prompt and the original image can guide the denoising process to ensure that the updates provide a cohesive representation of the scene.

[0047] FIG. 6 illustrates a computing system 600 that generates 3D scenes from a text prompt according to an implementation. Computing system 600 represents any computing system or systems with which the various operational architectures, processes, scenarios, and sequences disclosed herein for generating a 3D scene may be implemented. Computing system 600 is an example of computing system 110 of FIG. 1, although other examples can exist. Computing system 600 includes storage system 645, processing system 650, communication interface 660, and input / output (I / O) device(s) 670. Processing system 650 is operatively linked to communication interface 660, I / O device(s) 670, and storage system 645. In some implementations, communication interface 660 and / or I / O device(s) 670 may be communicatively linked to storage system 645. Computing system 600 may further include other components such as a batten- and enclosure that are not shown for clarity.

[0048] Communication interface 660 comprises components that communicate over communication links, such as network cards, ports, radio frequency, processing circuitry (and corresponding software), or other communication devices. Communication interface 660 may be configured to communicate over metallic, wireless, or optical links. Communication interface 660 may be configured to use Time Division Multiplex (TDM). Internet Protocol (IP), Ethernet, optical networking, wireless protocols, communication signaling, or some other communication format - including combinations thereof. Communication interface 660 may be configured to communicate with external devices, such as servers, user devices, or other computing devices.

[0049] I / O device(s) 670 may include computer peripherals that facilitate the interaction between the user and computing system 600. Examples of I / O device(s) 670 mayinclude keyboards, mice, trackpads, monitors, displays, printers, cameras, microphones, external storage devices, sensors, and the like.

[0050] Processing system 650 comprises microprocessor circuitry' (e.g., at least one processor) and other circuitry that retrieves and executes operating software (i.e., program instructions) from storage system 645. Storage system 645 may include volatile and nonvolatile, removable, and non-removable media implemented in any method or technology for information storage, such as computer-readable instructions, data structures, program modules, or other data. Storage system 645 may be implemented as a single storage device but may also be implemented across multiple storage devices or sub-systems. Storage system 645 may comprise additional elements, such as a controller to read operating software from the storage systems. Examples of storage media (also referred to as computer-readable storage media) include random access memory, read-only memory, magnetic disks, optical disks, and flash memory, as well as any combination or variation thereof or any other type of storage media. In some implementations, the storage media may be non-transitory. In some instances, at least a portion of the storage media may be transitory’. In no case is the storage media a propagated signal.

[0051] Processing system 650 is ty pically mounted on a circuit board that may also hold the storage system. The operating software of storage system 645 comprises computer programs, firmware, or some other form of machine-readable program instructions. The operating software of storage system 645 comprises scene application 624. The operating software on storage system 645 may include an operating system, utilities, drivers, network interfaces, applications, or some other ty pe of software. When read and executed by processing system 650, the operating software on storage system 645 directs computing system 600 to operate as a computing system as described herein. In at least one implementation, the operating software can provide method 200, described in FIG. 2, to generate a 3D scene from a prompt.

[0052] In at least one implementation, scene application 624 is configured to direct processing system 650 to identify a text prompt and generate an image from the text prompt based on a first model. In some implementations, computing system 600 is configured to receive the text prompt using a keyboard, a touchscreen, a microphone, or some other input device. For example, a user can provide voice input requesting a scene of a room with a table and six chairs. In response to identifying the text prompt (e.g., expressly input or derived via voice-to-text), the computing system is configured to generate an image for the prompt using a first model. In some implementations, the first model comprises a text-to-image synthesismodel. A text-to-image synthesis model can be configured as an advanced artificial intelligence system designed to generate images based on textual descriptions. Utilizing deep learning architectures like GANs or diffusion models, these models are trained on extensive datasets containing images and their corresponding text annotations. The training process enables the model to understand and map textual semantics to visual features (e.g., the table and chairs in the example above). When given a textual prompt, the model processes the description and generates an image that captures the specified details and context, translating words into visually coherent and relevant imagery.

[0053] In some examples, the first model is a diffusion model that can generate an image from a prompt through a multi-step process that involves adding and then progressively removing noise from an image. First, the model starts with a random noise image. During training, the model learns to reverse a gradual noising process where it takes a clean image, progressively adds noise over a series of steps, and leams to predict the intermediate noisy images. When generating an image from a text prompt, the model uses this learned reverse process, starting with a noise image and iteratively denoising it to generate a coherent image that matches the given prompt.

[0054] In some implementations, the diffusion model first encodes the text prompt into a vector representation using a text encoder, such as a transformer. A transformer architecture uses self-attention mechanisms to capture complex dependencies and relationships between words in a sequence, enabling efficient and effective natural language processing. The encoded prompt can guide the denoising process at each step. The denoising process can be implemented through a neural network that is conditioned on the text representation, gradually refining the noisy image by predicting and subtracting the noise at each step. The process continues iteratively, with each step producing an image closer to the final desired output. The result is an image that accurately reflects the content and style described by the text prompt, effectively translating the semantic information from the text into visual elements through the learned diffusion process.

[0055] Although a diffusion model was demonstrated to generate the image, other implementations can use GAN or some other text-to-image synthesis model. For example, the text prompt from the user can be processed using GAN to generate an image of the scene, generates an image by having a generator network create images from random noise and a discriminator network evaluate them, iteratively improving both through adversarial training until the generator produces realistic images. The GAN can be used to generate images that reflect the text provided in the request.

[0056] After the image is generated, scene application 624 is further configured to direct processing system 650 to process the image to generate a 3D scene of the image. In some examples, the 3D scene can be generated using 3D Gaussian splatting. In implementing the 3D Gaussian splatting, scene application 624 can be configured to extract 2D features or depth information from the image. Scene application 624 can be configured to provide the extraction using stereo vision, structure-from-motion. or deep learning models for depth estimation, which provide you with 3D coordinates and features for each point in the image (e.g., a point cloud). Each feature point is then represented as a 3D Gaussian splat, characterized by a 3D Gaussian distribution with a mean position in 3D space and a covariance matrix that defines its spread and shape. These splats are placed in the 3D space based on the extracted coordinates, and their properties are adjusted according to the depth information and the importance of each feature.

[0057] Once the 3D Gaussian splats are created, the next step is to render the 3D scene by combining these splats into a unified visual representation. This can involve techniques like alpha blending or volumetric rendering to visualize the collective effect of the splats, creating a coherent 3D scene from the individual Gaussian distributions. To improve the scene’s accuracy and quality, scene application 624 can also perform optimization tasks such as adjusting splat sizes, merging overlapping splats, or refining depth information. This process results in a detailed 3D scene that captures the visual and spatial information from the original 2D image.

[0058] After the 3D scene is generated, the scene application 624 is further configured to update the 3D scene from a first version to a second version based on a second model and the image. In some implementations, the second model comprises a diffusion model that updates the scene for different perspectives by using its iterative refinement process to enhance the quality of the 3D scene. Starting with an initial 3D Gaussian splat generated from the original image, the diffusion model adds noise to this representation and then progressively refines it through a series of denoising steps. During these steps, the model uses learned patterns from a large dataset to adjust the Gaussian splats' parameters, such as mean positions and covariance matrices, to capture details and correct inaccuracies better. This iterative denoising process enhances the 3D representation by improving detail, consistency, and visual coherence, ultimately creating a more accurate and refined 3D scene from the original image. For example, from an original dining room image, the diffusion model can be used to add noise to the representation from a different perspective and use denoising processes to improve detail and information for objects not included in the originalimage. The denoising is then included in the Gaussian splatted representation of the scene. This information can include the reverse side of objects that were included in the original image. In some implementations, the process is repeated for a threshold quantity of perspectives to refine the Gaussian splatted representation of the scene.

[0059] Clause 1. A method comprising: identifying a text prompt; generating an image from the text prompt based on a first model; processing the image to generate a three- dimensional scene of the image; and updating the three-dimensional scene from a first version to a second version based on a second model and the image.

[0060] Clause 2. The method of clause 1, wherein processing the image to generate the three-dimensional scene of the image comprises: generating a point cloud based on the image; and performing three-dimensional Gaussian splatting on the point cloud to generate the three-dimensional scene of the image.

[0061] Clause 3. The method of clause 1 or 2, wherein the image corresponds to a first perspective of the three-dimensional scene, and wherein updating the three-dimensional scene from the first version to the second version based on the second model and the image comprises: identifying a second perspective of the three-dimensional scene, the second perspective different than the first perspective; and updating the three-dimensional scene from the first version to the second version based on the second model, the image, and the second perspective.

[0062] Clause 4. The method of any one of the preceding clauses, wherein the first model comprises a diffusion model.

[0063] Clause 5. The method of any one of the preceding clauses, wherein the second model comprises a diffusion model.

[0064] Clause 6. The method of any one of the preceding clauses, wherein identifying the text prompt comprises identifying a text request for the three-dimensional scene from a user.

[0065] Clause 7. The method of any one of the preceding clauses, wherein the three- dimensional scene comprises three-dimensional Gaussian splats, wherein the image corresponds to a first perspective of the three-dimensional scene, and wherein updating the three-dimensional scene from the first version to the second version based on the second model and the image comprises: identify ing at least one additional perspective associated with the three-dimensional scene, the at least one additional perspective different than the first perspective; and updating the three-dimensional Gaussian splats based on an application of the second model to the image and the at least one additional perspective.

[0066] Clause 8. The method of any one of the preceding clauses, wherein processing the image to generate the three-dimensional scene of the image comprises: generating a point cloud based on the image; and generating at least one Gaussian splat based on the point cloud and the image, the at least one Gaussian splat comprising a Gaussian distribution.

[0067] Clause 9. The method of any one of the preceding clauses, wherein generating the image from the text prompt based on the first model comprises: identifying a noise image; and performing a denoising process on the noise image to generate the image based on the text prompt.

[0068] Clause 10. A computing apparatus comprising: at least one processor; a computer-readable storage medium operatively coupled to the at least one processor; and program instructions stored on the computer-readable storage medium that, when executed by the at least one processor, cause the at least one processor to execute operations, the operations comprising: identifying a text prompt; generating an image from the text prompt based on a first model; processing the image to generate a three-dimensional scene of the image; and updating the three-dimensional scene from a first version to a second version based on a second model and the image.

[0069] Clause 11. The computing apparatus of clause 10, wherein processing the image to generate the three-dimensional scene of the image comprises: generating a point cloud based on the image; and performing three-dimensional Gaussian splatting on the point cloud to generate the three-dimensional scene of the image.

[0070] Clause 12. The computing apparatus of clause 10 or 11, wherein the image corresponds to a first perspective of the three-dimensional scene, and wherein updating the three-dimensional scene from the first version to the second version based on the second model and the image comprises: identifying a second perspective of the three-dimensional scene, the second perspective different than the first perspective; and updating the three- dimensional scene from the first version to the second version based on the second model, the image, and the second perspective.

[0071] Clause 13. The computing apparatus of any one of clauses 10 to 12, wherein the first model comprises a diffusion model.

[0072] Clause 14. The computing apparatus of any one of clauses 10 to 13, wherein the first model comprises a text-to-image synthesis model.

[0073] Clause 15. The computing apparatus of any one of clauses 10 to 14, wherein the second model comprises a diffusion model.

[0074] Clause 16. The computing apparatus of any one of clauses 10 to 15, wherein identifying the text prompt comprises identifying a text request for the three-dimensional scene from a user.

[0075] Clause 17. The computing apparatus of any one of clauses 10 to 16, wherein the three-dimensional scene comprises three-dimensional Gaussian splats, wherein the image corresponds to a first perspective of the three-dimensional scene, and wherein updating the three-dimensional scene from the first version to the second version based on the second model and the image comprises: identifying at least one additional perspective associated with the three-dimensional scene, the at least one additional perspective different than the first perspective; and updating the three-dimensional Gaussian splats based on an application of the second model to the image and the at least one additional perspective.

[0076] Clause 18. A computer-readable storage medium having program instructions stored thereon that, when executed by at least one processor, cause the at least one processor to execute operations, the operations comprising: identifying a text prompt; generating an image from the text prompt based on a first model; processing the image to generate a three- dimensional scene of the image; and updating the three-dimensional scene from a first version to a second version based on a second model and the image.

[0077] Clause 19. The computer-readable storage medium of clause 18, wherein processing the image to generate the three-dimensional scene of the image comprises: generating a point cloud based on the image; and performing three-dimensional Gaussian splatting on the point cloud to generate the three-dimensional scene of the image.

[0078] Clause 20. The computer-readable storage medium of clause 18 or 19, wherein the first model comprises a diffusion model.

[0079] Clause 21. The computer-readable storage medium of any one of clauses 18 to20, wherein the second model comprises a diffusion model.

[0080] Clause 22. The computer-readable storage medium of any one of clauses 18 to21, wherein the image corresponds to a first perspective of the three-dimensional scene, and wherein updating the three-dimensional scene from the first version to the second version based on the second model and the image comprises: identifying a second perspective of the three-dimensional scene, the second perspective different than the first perspective; and updating the three-dimensional scene from the first version to the second version based on the second model, the image, and the second perspective.

[0081] In this specification and the appended claims, the singular forms “a.” “an” and “the” do not exclude the plural reference unless the context dictates otherwise. Further,conjunctions such as “and,” “or,” and “and / or” are inclusive unless the context dictates otherwise. For example, “A and / or B” includes A alone, B alone, and A with B. Further, connecting lines or connectors shown in the various figures presented are intended to represent example functional relationships and / or physical or logical couplings between the various elements. Many alternative or additional functional relationships, physical connections, or logical connections may be present in a practical device. Moreover, no item or component is essential to the practice of the implementations disclosed herein unless the element is specifically described as “essential” or “critical.”

[0082] Terms such as, but not limited to, approximately, substantially, generally, etc. are used herein to indicate that a precise value or range thereof is not required and need not be specified. As used herein, the terms discussed above will have ready and instant meaning to one of ordinary skill in the art.

[0083] Moreover, the use of terms such as up, down, top, bottom, side, end, front, back, etc. herein are used concerning a currently considered or illustrated orientation. If they are considered concerning another orientation, such terms must be correspondingly modified.

[0084] Further, in this specification and the appended claims, the singular forms “a,” “an” and “the” do not exclude the plural reference unless the context dictates otherw ise. Moreover, conjunctions such as “and,” “or,” and “and / or” are inclusive unless the context dictates otherwise. For example, “A and / or B” includes A alone, B alone, and A with B.

[0085] Although certain example methods, apparatuses, and articles of manufacture have been described herein, the scope of coverage of this patent is not limited thereto. It is to be understood that the terminology employed herein is to describe aspects and is not intended to be limiting. On the contrary, this patent covers all methods, apparatus, and articles of manufacture, fairly falling within the scope of the claims of this patent.

Claims

WHAT IS CLAIMED IS:

1. A method comprising: identifying a text prompt; generating an image from the text prompt based on a first model; processing the image to generate a three-dimensional scene of the image; and updating the three-dimensional scene from a first version to a second version based on a second model and the image.

2. The method of claim 1 , wherein processing the image to generate the three- dimensional scene of the image comprises: generating a point cloud based on the image; and performing three-dimensional Gaussian splatting on the point cloud to generate the three-dimensional scene of the image.

3. The method of claim 1 or 2, wherein the image corresponds to a first perspective of the three-dimensional scene, and wherein updating the three-dimensional scene from the first version to the second version based on the second model and the image comprises: identifying a second perspective of the three-dimensional scene, the second perspective different than the first perspective; and updating the three-dimensional scene from the first version to the second version based on the second model, the image, and the second perspective.

4. The method of any one of the preceding claims, wherein the first model comprises a diffusion model.

5. The method of any one of the preceding claims, wherein the second model comprises a diffusion model.

6. The method of any one of the preceding claims, wherein identifying the text prompt comprises identifying a text request for the three-dimensional scene from a user.

7. The method of any one of the preceding claims, wherein the three-dimensional scene comprises three-dimensional Gaussian splats, wherein the image corresponds to a firstperspective of the three-dimensional scene, and wherein updating the three-dimensional scene from the first version to the second version based on the second model and the image comprises: identifying at least one additional perspective associated with the three-dimensional scene, the at least one additional perspective different than the first perspective; and updating the three-dimensional Gaussian splats based on an application of the second model to the image and the at least one additional perspective.

8. The method of any one of the preceding claims, wherein processing the image to generate the three-dimensional scene of the image comprises: generating a point cloud based on the image; and generating at least one Gaussian splat based on the point cloud and the image, the at least one Gaussian splat comprising a Gaussian distribution.

9. The method of any one of the preceding claims, wherein generating the image from the text prompt based on the first model comprises: identify ing a noise image; and performing a denoising process on the noise image to generate the image based on the text prompt.

10. A computing apparatus comprising: at least one processor; a computer-readable storage medium operatively coupled to the at least one processor; and program instructions stored on the computer-readable storage medium that, when executed by the at least one processor, cause the at least one processor to execute operations, the operations comprising: identifying a text prompt; generating an image from the text prompt based on a first model; processing the image to generate a three-dimensional scene of the image; and updating the three-dimensional scene from a first version to a second version based on a second model and the image.

11. The computing apparatus of claim 10, wherein processing the image to generate the three-dimensional scene of the image comprises: generating a point cloud based on the image; and performing three-dimensional Gaussian splatting on the point cloud to generate the three-dimensional scene of the image.

12. The computing apparatus of claim 10 or 11, wherein the image corresponds to a first perspective of the three-dimensional scene, and wherein updating the three-dimensional scene from the first version to the second version based on the second model and the image comprises: identifying a second perspective of the three-dimensional scene, the second perspective different than the first perspective; and updating the three-dimensional scene from the first version to the second version based on the second model, the image, and the second perspective.

13. The computing apparatus of any one of claims 10 to 12. wherein the first model comprises a diffusion model.

14. The computing apparatus of any one of claims 10 to 13, wherein the first model comprises a text-to-image synthesis model.

15. The computing apparatus of any one of claims 10 to 14, w herein the second model comprises a diffusion model.

16. The computing apparatus of any one of claims 10 to 15, w herein identifying the text prompt comprises identifying a text request for the three-dimensional scene from a user.

17. The computing apparatus of any one of claims 10 to 16, wherein the three- dimensional scene comprises three-dimensional Gaussian splats, wherein the image corresponds to a first perspective of the three-dimensional scene, and w herein updating the three-dimensional scene from the first version to the second version based on the second model and the image comprises: identifying at least one additional perspective associated with the three-dimensional scene, the at least one additional perspective different than the first perspective; andupdating the three-dimensional Gaussian splats based on an application of the second model to the image and the at least one additional perspective.

18. A computer-readable storage medium having program instructions stored thereon that, when executed by at least one processor, cause the at least one processor to execute operations, the operations comprising: identifying a text prompt; generating an image from the text prompt based on a first model; processing the image to generate a three-dimensional scene of the image; and updating the three-dimensional scene from a first version to a second version based on a second model and the image.

19. The computer-readable storage medium of claim 18, wherein processing the image to generate the three-dimensional scene of the image comprises: generating a point cloud based on the image; and performing three-dimensional Gaussian splatting on the point cloud to generate the three-dimensional scene of the image.

20. The computer-readable storage medium of claim 18 or 19, wherein the first model comprises a diffusion model.

21. The computer-readable storage medium of any one of claims 18 to 20, wherein the second model comprises a diffusion model.

22. The computer-readable storage medium of any one of claims 18 to 21, wherein the image corresponds to a first perspective of the three-dimensional scene, and wherein updating the three-dimensional scene from the first version to the second version based on the second model and the image comprises: identifying a second perspective of the three-dimensional scene, the second perspective different than the first perspective; and updating the three-dimensional scene from the first version to the second version based on the second model, the image, and the second perspective.

Images