Immersive environment generation with spatial audio

Abstract

Described techniques enable generation of immersive environments with semantically consistent audio. For example, a textual request for a three-dimensional (3D) environment that includes at least one audio source associated with a location within the 3D environment may be received. The textual request may be processed using a text-to-image machine learning (ML) model to generate a 3D visual representation of the 3D environment. The textual request may also be processed using a text-to-audio ML model to generate an audio file corresponding to the at least one audio source. The 3D visual representation and the audio file may be combined to generate the 3D environment with the at least one audio source at the location within the 3D environment.

Description

IMMERSIVE ENVIRONMENT GENERATION WITHSPATIAL AUDIOTECHNICAL FIELD

[0001] This description relates to immersive environments.BACKGROUND

[0002] Immersive environments generally refer to artificial, interactive visualizations, which a user may experience using a suitable device(s). For example, goggles or other ty pes of head-mounted devices (HMDs) may be used to display a three-dimensional (3D) visual rendering of an environment, and to interact with objects rendered therein.

[0003] Such immersive environments may be provided in many different contexts. For example, immersive environments may be provided for virtual reality (VR) games, interactive education experiences, simulations, or in professional scenarios (e.g., 3D design).SUMMARY

[0004] In a general aspect, a computer program product is tangibly embodied on a non-transitory computer-readable storage medium and comprises instructions. When executed by at least one computing device (e.g., by at least one processor of the computing device), the instructions are configured to cause the at least one computing device to receive a textual request for a three-dimensional (3D) environment that includes at least one audio source associated with a location within the 3D environment. The instructions, when executed by the at least one computing device, may further cause the at least one computing device to process the textual request using a text-to-image machine learning (ML) model to generate a 3D visual representation of the 3D environment, and process the textual request using a text-to-audio ML model to generate an audio file corresponding to the at least one audio source. The instructions, when executed by the at least one computing device, may further cause the at least one computing device to combine the 3D visual representation and the audio file to generate the 3D environment with the at least one audio source at the location within the 3D environment.

[0005] In another general aspect, a wearable device includes at least one frame for positioning the wearable device on a body of a user, at least one display, binaural speakers, atleast one processor, and at least one memoty. The at least one memory' may store a set of instructions, which, when executed, cause the at least one processor to receive a textual request for a three-dimensional (3D) environment that includes at least one audio source associated with a location within the 3D environment. When executed, the instructions cause the at least one processor to process the textual request using a text-to-image machine learning (ML) model to generate a 3D visual representation of the 3D environment. When executed, the instructions cause the at least one processor to process the textual request using atext-to-audio ML model to generate an audio file corresponding to the at least one audio source. When executed, the instructions cause the at least one processor to combine the 3D visual representation and the audio file to generate the 3D environment with the at least one display and with the binaural speakers, and with the at least one audio source at the location within the 3D environment.

[0006] In another general aspect, a method includes receiving a textual request for a three-dimensional (3D) environment that includes at least one audio source associated with a location within the 3D environment. The method further includes processing the textual request using a text-to-image machine learning (ML) model to generate a 3D visual representation of the 3D environment, and processing the textual request using a text-to-audio ML model to generate an audio file corresponding to the at least one audio source. The method further includes combining the 3D visual representation and the audio file to generate the 3D environment with the at least one audio source at the location within the 3D environment.

[0007] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG. 1 is a block diagram of a system for immersive environment generation with spatial audio.

[0009] FIG. 2 is a flowchart illustrating example operations of the system of FIG. 1.

[0010] FIG. 3 A illustrates example training techniques for the system of FIG. 1.

[0011] FIG. 3B illustrates example use case scenarios for the system of FIG. 1.

[0012] FIG. 3C illustrates screenshots of an additional example implementation of the system of FIG. 1.

[0013] FIG. 4 is a block diagram of a more detailed example of a sound generator ofthe system of FIG. 1.

[0014] FIG. 5 is a block diagram of an example audiovisual combination pipeline illustrates a second example use case scenario for the system of FIG. 1.

[0015] FIG. 6 is a block diagram of an example software architecture that may be used to implement the sy stem of FIG. 1.

[0016] FIG. 7 is a third person view of a user in an ambient computing environment.DETAILED DESCRIPTION

[0017] Described systems and techniques enable use of a text prompt to provide low- latency generation of an immersive environment with sparial audio. As a result, for example, users may experience on-demand availability of a desired immersive environment. In other examples, developers may be aided in creating new games, educational classes, or 3D design projects.

[0018] Conventional techniques for providing 3D, immersive environments typically require specialized coding knowledge and / or use of a corresponding development platform (such as, e.g., the Unity development platform), and may be time-consuming to create. It may be possible to attempt to train a machine learning (ML) model to generate a 3D environment, but constructing realistic, immersive environments using conventional ML techniques may be impractical, due to, e.g., lack of available / sufficient training data. Moreover, even if a sufficiently and correctly trained ML model were available, required processing times needed to generate a desired immersive environment may be unacceptably long for many use case scenarios.

[0019] In contrast, example implementations provide immersive environments with spatial audio using a custom 2D text-to-image generative model that is trained to generate realistic 3D scenes through the use of a boundary condition(s) that enables generation of seamlessly combined 2D images to simulate or provide the desired immersive 3D environment. Consequently, training may be provided in an efficient manner, using available training data. Additionally, desired immersive environments may be generated in real time or near real time, in response to a simple verbal or textual request from a user.

[0020] Moreover, described techniques enable inclusion of spatial audio within the context of the generated immersive environment. For example, a separate text-to-audio generative model may be trained or available for generating a corresponding audio file(s). Described techniques ensure a consistency between a resulting audio file and an image file generated in parallel by the type of text-to-image generative model referenced above.

[0021] For example, a received verbal / textual request may include an implicit or explicit specification of one or more audio sources and / or associated audio characteristics to be included within a generated immersive environment. Described techniques ensure both a spatial and semantic matching between generated image and audio files, to thereby enable generation of an immersive environment in which audio characteristics correspond to the simulated acoustic environment,

[0022] For example, such audio characteristics may include representation of sounds from expected directions and / or distances. An acoustic environment may dictate certain audio characteristics, such as audio reflection (e.g., echo or reverberation), amplification, or absorption.

[0023] Implementations of the described techniques are designed to maintain consistency between generated image and audio files, e.g., by ensuring semantic consistency between image and audio prompts and between corresponding model outputs. For example, as an image model and audio model may be implemented independently of one another, each model may output semantically different responses to a single input. For example, a user submitting an environment request that includes a reference to water could receive a visual of a river but audio of an ocean. To avoid such divergent semantics, described techniques may use iterative processing of model outputs to converge initially divergent output semantics into a single and consistent semantic description of a specified immersive environment, so that the specified immersive environment may be generated in an expected manner and within required time constraints.

[0024] FIG. 1 is a block diagram of a system for immersive environment generation with spatial audio. In the example of FIG. 1. a head-mounted device (HMD) 102 is illustrated as being worn by a user 104 and providing an immersive environment 106 for the user, including projecting an audio source 108 within the immersive environment 106.

[0025] For example, as referenced above, and illustrated and described in various example implementations, below, the HMD 102 may represent, e.g., smart goggles or smartglasses, or other suitable HMD. The immersive environment 106 may include a visual image(s) of virtually any desired location or other context requested by the user 104, provided, for example, as a photosphere, skybox, panoramic view, or any 360° visual representation.

[0026] The immersive environment 106 may be provided using an image file and corresponding audio file (not shown explicitly in FIG. 1, but illustrated and described in detail, below, e.g., with respect to FIGS. 3A-3C and FIG. 4. The audio source 108 thusrepresents an audio rendering of the audio file within the context of the immersive environment 106.

[0027] For example, the audio source 108 may represent an image object that is visible within the image file of the immersive environment 106. Such an image object may include, e.g., an image of a person or animal, a musical instrument or other source of music, a natural object such as a river or the ocean, or any potential audio source that might be generated in response to a request from the user 104.

[0028] In other example implementations, the audio source 108 need not be explicitly visible or associated with a visible image object. For example, the audio source 108 may be included in the immersive environment 106 at a distance from a point of view of the user 104 within the immersive environment 106, or may be otherwise obscured from view within the immersive environment 106.

[0029] Regardless of whether the audio source 108 is visible or not visible to the user 104 within the immersive environment 106, the audio source 108 may be associated with a location within the immersive environment 106, and audio from the audio source 108 may thus be provided as originating from the location. For example, the user 104 may have a 360° view within the immersive environment 106, so that, depending on a direction faced by the user 104, the audio source 108 may be experienced by the user 104 as being, e.g., in front of, behind, or to a side of the user 104.

[0030] In addition, the immersive environment 106 may be associated with a set of audio characteristics. The audio characteristics may be defined by, or associated with, virtual physical characteristics of the generated immersive environment 106. As referenced above, the audio characteristics may include reflection or absorption characteristics, as well as any other audio characteristic, including, e.g., volume, pitch, timbre, or tone.

[0031] Many various virtual physical characteristics of the generated immersive environment 106, including virtual physical characteristics of the audio source 108, may be associated with such audio characteristics. By way of non-limiting example, such virtual physical characteristics may include a size, type, or distance of any object included within the immersive environment 106. For example, audio characteristics may conform to the immersive environment 106 as a small room, or to the immersive environment 106 as an outdoor scene, such as a forest.

[0032] In some example implementations, audio characteristics may be explicit in, or inferred from, a request of the user 104 for the immersive environment 106. For example, the user 104 may request relaxing background music, which may be rendered at a relatively lowvolume.

[0033] Advantageously, the user 104 may obtain the immersive environment 106, including the audio source 108, in response to a spoken request, and with very low latency (e.g., on the order of a few seconds). As a result, the user 104 may easily and quickly experience many different instances of the immersive environment 106, including cohesive inclusion of the audio source 108.

[0034] As shown in the exploded view of FIG. 1, the HMD 102 may include a processor 110 (which may represent one or more processors), as well as a memory 112 (which may represent one or more memories (e.g., non-transitory computer readable storage media)). Although not shown separately in FIG. 1, the HMD 102 may also include a battery, which may be used to power operations of the processor 110, the memory 112. and various other resources of the HMD 102. More detailed examples of the HMD 102 and various associated hardware / software resources are provided below, e.g., with respect to FIGS. 7 and 8.

[0035] For purposes of the simplified example of FIG. 1, the HMD 102 should be further understood to include a microphone 116 and speakers 118. For example, the speakers 118 may represent left ear / right ear speakers provided to the user 104 using, e.g., a headset band of smart googles or other VR headset, arms of a pair of smartglasses, a pair of earbuds, or any pair of left / right speakers that may be positioned at left / right ears of the user 104. The speakers 118 may be referred to as binaural speakers 118.

[0036] Displays 120 may represent, e.g., two displays of the HMD 102 that may be used to display offset images to a left eye and right eye of the user 104, and thereby provide an illusion of depth to the user 104. Other types of displays capable of rendering a 3D representation may also be used, such as, e.g.. holographic displays or light field displays.

[0037] A transcription engine 122 represents software configured to perform speech- to-text processing. That is, the transcription engine 122 may be configured to convert spoken words (audio data) to transcribed text. For example, the transcription engine 122 may include an automatic speech recognition (ASR) engine or a speech-to-text (STT) engine. For example, the transcription engine 122 may also utilize natural language understanding (NLU) and / or natural language processing (NLP) models, or related techniques. The transcription engine 122 may also utilize a hot word or wake word techniques to initiate operations for generating the immersive environment 106. In other example implementations, text may be input using a text editor and associated input hardware, such as a keyboard.

[0038] An audio spatializer 124 may be configured to input a raw audio file andgenerate 3D audio effects using the binaural speakers 118. As referenced above, and described in more detail, below, 3D audio techniques for wearable devices may be used to generate spatially accurate sound experiences by algorithmically modulating binaural speaker sounds produced by the speakers 118. For example, channel-based rendering techniques generate audio events at fixed, pre-defined speaker locations of real or virtual speakers. In other examples, object-based rendering techniques generate individual audio events with corresponding coordinates of a spatial domain. Another example technique, known as ambisonics, utilizes a two-step process in which received sound signals are mapped (encoded) to intermediate spherical harmonic representations, which are then spatially mapped (decoded) to a desired or available arrangement of output speakers. Ambisonics has been used for immersive applications, such as virtual reality’ (VR), because the decoupling of encoding and decoding enables an experience of 360° sound for a user wearing a VR headset or other suitable HMD represented by the HMD 102.

[0039] A stereoscopic mapping engine 126 may be configured to input a raw image file and generate the type of 3D-simulated view described above with respect to the displays 120. That is, the stereoscopic mapping engine 126 may be configured to project a raw image file to two displays represented by the displays 120, to provide an illusion of depth to the user 104 when using the HMD 102. For example, known techniques such as stereoscopic reprojection, or other suitable techniques, may be used.

[0040] An environment generator 128 represents software and related hardware configured to use resources of the HMD 102, as well as separate and / or remote resources represented in FIG. 1 by a cloud server 138, to generate the immersive environment 106 as described herein. In the example of FIG. 1, the environment generator 128 is illustrated as being implemented at the HMD 102. In other implementations, some of which are described below, some or all of the environment generator 128 may be implemented at the cloud server 138 or other suitable remote resource, and / or at a separate, local device, such as a laptop or desktop computer in communication w ith the HMD 102.

[0041] The environment generator 128 is illustrated as including a visual model 130. which may be trained and otherwise configured to receive text requesting the immersive environment 106, e g., from the transcription engine 122, and generate corresponding image files that facilitate rendering of the immersive environment 106. For example, the visual model 130 may represent a large language model (LLM), such as a generative model. In more specific example implementations, the visual model 130 may be implemented as a 2D diffusion model. Other suitable models may be used, as well.

[0042] An audio model 132 may be trained and otherwise configured to receive text requesting the immersive environment 106, e.g., from the transcription engine 122, and generate corresponding audio files that facilitate rendering of the immersive environment 106. That is, the audio model 132 may be configured to receive the same input text as the visual model 130. Similar to the visual model 130, the audio model 132 may represent a LLM, such as a generative model. In more specific example implementations, the audio model 132 may be implemented using the MusicLM model from Google LLC.

[0043] Other suitable models may be used, as well. Additionally, with respect to both the visual model 130 and the audio model 132, more than a single model may be used. For example, the audio model 132 may represent the MusicLM model just referenced for purposes of generating music, but may include a separate model(s) for generating other types of audio, such as voices, sounds associated with a city (e.g., traffic), sounds associated with nature (e.g., a forest or ocean), or any suitable audio category; In some example implementations, when multiple models are available, the environment generator 128 and / or the transcription engine 122 may include a classifier model that classifies input text for routing to a correct type of model.

[0044] A combination pipeline 134 may' be configured to process image files from the visual model 130 and audio files from the audio model 132 to obtain semantically consistent image and audio files, which may then be rendered by a rendering engine 136 to generate the immersive environment 106. For example, to the extent that the visual model 130 and the audio model 132 operate independently of one another and in parallel with one another, it is possible that generated image files and audio files will have inconsistent semantics.

[0045] For example, a request for the immersive environment 106 to include “trees by the water’ might result in an image of a forest by a river but audio of the ocean. More generally, as the visual model 130 and the audio model 132 may both be implemented using generative models, the two models may independently generate divergent outputs. For example, if received text prompt includes reference to an office environment, the visual model 130 might generate an image of a large office with cubicles, while the audio model 132 might generate audio characteristics consistent with a single, small room (or vice versa).

[0046] To avoid image / audio inconsistencies such as the above and similar, the combination pipeline 134 may be configured to oversee iterative operations of the visual model 130 and / or the audio model 132, including performing a comparison of the outputs of the two models, and then, based on the comparison, feeding some aspect of an output(s) of one or both models into an input(s) of one or both models until a current comparisonindicates a suitable match between a current image output and a current audio output.

[0047] For example, as described in detail, below, with respect to FIG. 5. the combination pipeline 134 may include, or access, an image-to-text summarization model and an audio-to-text summarization model. Consequently, the combination pipeline 134 may obtain an image summary7of an image file of the visual model, and an audio summary7of an audio file of the audio model 132. The image summary and the audio summary may then be compared to one another relative to a similarity threshold, and if the similarity threshold is not met, one or both of the summaries may be fed back to one or both of the visual model 130 and / or the audio model 132, perhaps in conjunction with (e.g., concatenated with) the original textual request from the user 104.

[0048] Iterations may thus continue until convergence is reached. Then, the rendering engine 136 may be configured to proceed with rendering the immersive environment 106. For example, the rendering engine 136 may communicate with the audio spatializer 124 to perform an ambisonics encoding / decoding for spatialized audio provided using the speakers 118. Similarly, the stereoscopic mapping engine 126 may communicate with the displays 120 to provide a stereoscopic rendering of the image file(s) obtained from the rendering engine 136.

[0049] As referenced above, FIG. 1 further illustrates the use of a remote cloud server 138, which may also be in direct or indirect communication with the HMD 102. In example implementations, some or all of the visual model 130. the audio model 132, and / or the combination pipeline 134 may be hosted on the cloud server 138.

[0050] In the example of FIG. 1, the cloud server 138 is illustrated as providing resources for executing a training engine 140 using suitable training data 142. For example, the training data 142 may include labeled image files suitable for training the visual model 130, and labeled audio files for training the audio model 132. In example implementations, training may involve defining a loss function and then iteratively minimizing an error associated with such a loss function by updating, e.g., weights and biases of the model being trained at each iteration, until an acceptable level of error is reached.

[0051] In general, multiple loss functions and associated training techniques / parameters may be considered and used when implementing the training engine 140. Additionally, in some example implementations, specialized training may be performed with respect to the visual model 130, in order to obtain the types of low-latency generation of the immersive environment 106 of FIG. 1.

[0052] For example, the training data 142 may include 2D images, and the visualmodel 130, as already mentioned, may represent a 2D diffusion model or other model with similar features and capabilities. As described in detail below, e.g., with respect to FIG. 3 A, the training engine 140 may be parameterized with a loss function that ensures a seamless wrapping of the types of 2D images generated by the visual model 130, so as to train the visual model 130 to generate the type of stereoscopic 3D environment described herein. Put another way, the training engine 140 may maintain a boundary condition that minimizes differences between edges of 2D images being generated, so that multiple images may be simultaneously generated in a manner that does not include discontinuities at edges at which the multiple images are joined.

[0053] As a result, the visual model 130 and the audio model 132 may operate in parallel, and the visual model 130 may be trained and operated using 2D images used to present a 3D stereoscopic output. Consequently, the environment generator 128 may be configured to generate the immersive environment 106 in a low-latency fashion, as described herein, and using minimal computing resources, as compared to the use of models trained using, and / or trained to generate, individual three-dimensional visual environments. Moreover, the resulting immersive environment 106 may be generated with the audio source(s) 108, and associated audio characteristics, that are consistent and cohesive with the visual characteristics of the immersive environment 106. Therefore, the user 104, including developers, designers, remote conferencing participants, game players, and various other users, may be provided with unique, desired immersive environments.

[0054] FIG. 2 is a flow chart illustrating example operations of the system of FIG. 1 . In the example of FIG. 2, operations 202-208 are illustrated as separate, sequential operations. However, in various example implementations, the operations 202-208 may be implemented in a different order than illustrated, in an overlapping or parallel manner, and / or in a nested, iterative, looped, or branched fashion. Further, various operations or suboperations may be included, omitted, or substituted.

[0055] In FIG. 2, a textual request for a three-dimensional (3D) environment may be received (202). For example, the transcription engine 122 may receive spoken text from the user 104 via the microphone 116 of FIG. 1, and may output corresponding transcribed text to the environment generator 128. In other examples, the user 104 may directly enter text specifying a desired immersive, 3D environment, using a suitable input device such as a keyboard.

[0056] The textual request may be processed using a text-to-image machine learning (ML) model to generate a 3D visual representation of the 3D environment (204). Forexample, the visual model 130 of FIG. 1 may be configured to receive the text input and generate image files that provide visual aspects of the immersive environment 106.

[0057] The textual request may be processed using a text-to-audio ML model to generate an audio file corresponding to at least one audio source associated with a location within the 3D environment (206). For example, the audio model 132 of FIG. 1 may be configured to receive the text input and generate one or more audio files that provide audible aspects of the immersive environment 106.

[0058] The 3D visual representation and the audio file may be combined to generate the 3D environment with the at least one audio source at the location within the 3D environment (208). For example, the combination pipeline 134 may be configured to ensure a semantic consistency between the generated image file(s) and audio file(s). For example, as in FIG. 1, if the audio source 108 is positioned ahead of the user 104 and to a left of the user 104 when the user is facing in a given direction, the combination pipeline 134 may ensure that both the visual representation and the audio representation of the audio source consistently reflect this position relative to the user 104. In addition, the combination pipeline 134 may be configured to ensure semantic consistency between the audio source 108 and audio characteristics of the immersive environment, including, e.g., reflective properties, tonal qualities, or other audio characteristics that might be dictated by, or relevant to, e.g., a size, material, or configuration of the visually presented environment.

[0059] FIG. 3 A illustrates example training techniques for the system of FIG. 1. FIG. 3A illustrates an image XI 302, an image X2 304, an image X3 306, an image X4 308, an image X5 310, and an image X6 312. The images 302-312 may be generated in the context of training the visual model 130 to generate a skybox view' 314 that provides an example of a 360° visual representation of an ocean skyline with clouds above a waterline, as shown.

[0060] Put another way, the images 302-312 may be considered to be front, back, up, down, left, right views of the skybox view- 314, w hich are unwrapped in a 2D plane for modelling purposes. The corresponding skybox view 314 may thus be used as an example of the immersive environment 106 of FIG. 1. e.g., in the context of a 360 equirectangular format for 360° images. When trained using the techniques described herein, for example, the visual model 130 may output a 360° image converted into a skybox form for rendering within the immersive environment 106.

[0061] Conventional diffusion models are capable of generating individual 2D images, but such conventional 2D images do not have boundary conditions with respect to one another. In other w ords, such conventionally generated 2D images would suffer frommismatch(es) in boundary conditions that would prevent the type of wrapping illustrated inFIG. 3.

[0062] To provide the type of panoramic output needed for the skybox view 314, described techniques implement the training engine 140 with a boundary condition that effectively defines an edge-to-edge seam through the use of an additional loss function used during training.

[0063] Thus, during training, as the visual model being trained generates the images 302-312, a loss can be defined with respect to boundary pixels of ajoined boundary of adjacent figures. Such boundary pixels are illustrated in the example of FIG. 3A by an edge 316 of the image XI 302 and an edge 318 of the image X5 310. which may be joined to form an edge 320 in the skybox view 314.

[0064] For example, a total variation function |(X_edge_l) - (X_edge_2)| may be defined for minimization. In similar example implementations, boundary pixels may be indexed by i and a norm of each pixel pair of a pair of boundaries may be summed over i, e.g., Sum_i = || Xl_i - X5_i ||. One of the preceding (or similar) loss functions may be defined as a new loss function, referred to herein as Lnew, which may be added to an existing loss function L(w) for the visual model being trained that is chosen for purposes of mapping a text token label to an image being generated, where w represents a set of updateable weights that are changed during training to optimize image generation properties of the visual model 130.

[0065] More specifically, for example, the new loss Lnew and a corresponding weight X may be added as a regularizer to the existing loss function L(w). Put another way, training of the visual model 130 may be implemented as: minimize w for L(w) = (X)(Lnew). Thus, training may be implemented including specifying a boundary condition to minimize differences between adjoining pixels of each pair of generated 2D images, including differences in, e.g., brightness, color, contrast, and other pixel characteristics.

[0066] FIG. 3B illustrates example use case scenarios for the system of FIG. 1. As shown in FIG. 3B, the user 104 may provide spoken input 322 for a desired immersive environment 324, such as “Put me in a modem office space. In the style of Van Gogh. Low- res cartoon.” The user 104 may provide further spoken input 326 for a second immersive environment 328, such as “Change style to high-res. Also put elevator music in the background. Jazz influenced.”

[0067] As shown in FIG. 3B, the immersive environment 328 may include the requested audio (music) 330. As described herein, the audio 330 may be associated with anaudio source and corresponding audio location within the immersive environment 328. For example, one or more virtual audio speakers, which may or may not be visually explicit within the immersive environment 328, may be generated and positioned within the immersive environment 328.

[0068] With respect to both the immersive environments 324, 328, audio characteristics of the scenes being simulated may be generated consistently with the visual elements and other characteristics of the generated scenes. For example, as referenced above, audio characteristics such as echo and reverberation may correspond to a simulated size and composition of a corresponding scene, and to elements thereof. Consequently, the audio 330 of FIG. 3B would sound different in the simulated office of the immersive environment 328 than would the same audio requested in the context of, e.g.. an assembly hall, or in an outdoor plaza.

[0069] For example, the audio model 132 may be trained using various such contexts or environments, so as to suitably match audio characteristics to ty pes of generated scenes. As described herein, and illustrated in more detail with respect to FIG. 5, each of the visual model 130 and the audio model 132 have the freedom to generate any visual and audio environments that correspond to the ty pe of spoken inputs 322, 326 shown in FIG. 3, as long as the visual and audio environments correspond with one another to provide a cohesive audiovisual experience for the user 104.

[0070] Put another w ay, the visual model 130 may’ be configured to generate multiple possible visualizations of, e.g., the spoken input 326, while the audio model 132 may be configured to generate multiple possible audio source(s) / enviromnents of the spoken input 326. Therefore, there is not a ‘correct’ or particular visual or audio output required to be determined for a given spoken input requesting an immersive environment. Rather, the combination pipeline 134 may be configured to ensure that any suitable combination or pair of a visual and an audio environment, w hich both correspond to the spoken input, are matched to one another to ensure the cohesive audiovisual experience for the user 104 described herein.

[0071] FIG. 3C illustrates screenshots of an additional example implementation of the system of FIG. 1. In more detail, a screenshot 332, a screenshot 334, and a screenshot 336 represent a view of the user 104 when viewing the t pe of immersive environment described herein.

[0072] For example, the user 104 may verbally request an immersive environment such as, ‘'a spaceship.” The resulting screenshots 332, 334, 336 of FIG. 3C may thus reflect,for example, the user 104 wearing goggles and viewing the environment from left to right in a direction of the illustrated arrows. Alternatively, the screenshots 332, 334, 336 may be understood to represent a screen of a laptop, monitor, or other 2D display, on which a 3D representation of the requested immersive environment is displayed. For example, the user 104 may navigate through the generated environment when using a laptop to play a game in the generated environment, with navigation enabled by the laptop keyboard, mouse, or other input device.

[0073] FIG. 4 is a block diagram of a more detailed example of a sound generator of the system of FIG. 1. In FIG. 4, the user 104 provides a verbal environment request 402, which may be captured as a multi-microphone recording and subjected to neural enhancement 404. For example, the environment request 402 may be fdtered and otherwise processed to remove background noise or otherwise clarify the content of the environment request 402.

[0074] Resulting speech may be passed to a transcription engine 406, which may then pass the resulting text to an environment generator 408. As described above, the environment generator 408 includes an audio model 410, a combination pipeline 412, and an image model 414. As shown in FIG. 4, the audio model 410 inputs text from the transcription engine 406 and generates an audio fde(s) 416. Similarly, the image model 414 inputs text from the transcription engine 406 and generates an image file(s) 418.

[0075] The combination pipeline 412 may be configured to oversee and moderate operations of the audio model 410 and the image model 414, to ensure consistency between the audio file 416 and the image file 418. For example, the combination pipeline 412 enables and facilitates a semantic understanding of a location(s) of one or more audio sources of the audio file 416 within the image file 418.

[0076] Then, for a six degree of freedom (6DOF) display context 420, a 3D audio spatializer 422 may be configured to process the audio file 416 to generate binaural audio, which may then be passed to a binaural rendering driver 424. For example, ambisonics techniques, referred to above, may be used to spatially represent sound sources within virtual immersive environments described herein so that the user 104 may experience virtual sound sources from directions above, below, in front of, or behind the user 104.

[0077] Similarly, a stereoscopic mapping engine 426 may be configured to map the image file 418 to a stereo image for further processing by a binocular rendering driver 428. For example, the image file 418 may be projected to each of two physical displays or two virtual displays of the HMD 102.

[0078] FIG. 5 is a block diagram of an example audiovisual combination pipeline illustrates a second example use case scenario for the system of FIG. 1. and which may be used in FIGS. 4 and 6, as well. As described above, the combination pipeline of FIG. 5 may be configured to bridge and relate the generative audio and visual semantics of corresponding audio and visual generative models.

[0079] In FIG. 5. a text query 502 is input in parallel to a first instance 504a of a generative audio model, and to a first instance 508a of a generative visual model. A semantic audio-to-text summarization model 506 thus receives a first audio file and outputs a first textual audio summary', i.e., text summarizing content of the received audio file. Similarly, a semantic visual -to-text summarization model 510 receives a first image file and outputs a first textual visual summary file, i.e., text summarizing content of the received image.

[0080] Then, text of the audio summary is compared against text of the visual summary', to determine whether a similarity threshold has been reached (512). For example, a cosine similarity analy sis may be performed between the audio summary text and the visual summary text.

[0081] If the similarity threshold is not reached, then the original text query' 502 may be input to a second instance 504b of a generative audio model in combination with the audio summary'. For example, text of the original text query' 502 may be concatenated with text of the audio summary and submitted in the aggregate to the second instance 504b of the generative audio model.

[0082] Similarly, the original text query 502 may be input to a second instance 508b of a generative visual model in combination with the visual summary'. For example, text of the original text query 502 may be concatenated with text of the visual summary and submitted in the aggregate to the second instance 508b of the generative visual model.

[0083] Iterations may continue as shown by re-submitting a second audio file generated from the combined original text and first audio summary' to the semantic summarization model 506 to obtain a second audio summary, and by re-submitting a second visual file generated from the combined original text and first visual summary to the semantic summarization model 510 to obtain a second visual summary. The second audio summary and the second visual summary may' be subjected to a similarity analysis to determine whether the similarity' threshold has been reached (512). If not, iterations may continue until the similarity threshold is reached.

[0084] FIG. 6 is a block diagram of an example software architecture that may be used to implement the system of FIG. 1. In FIG. 6, local resources 602 include a device 604,corresponding, e g., to the HMD 102 of FIG. 1. A local speech-to-text handler 606 may include a hot word handler 608 configured to recognize a wake word for the device 604.

[0085] A server interface 610 represents a local software package configured to enable and facilitate communications between the local resources 602 and a cloud server 616, similar to the cloud server 138 of FIG. 1. As shown, the server interface 610 may include a prompt generator 614 that is configured to format and otherwise modify text received from the speech to text handler 606 in a format suitable for submission to the cloud server 616.

[0086] A query handler 612 may be configured to open a channel and otherwise communicate with the cloud server 616, including sending the prompt received from the prompt generator 614 and receiving data needed for generating an immersive environment as received from the cloud server 616. For example, the query handler 612 may generate a remote procedure call (RPC) request to communicate with the cloud server 616. In more specific examples, the query handler 612 may open a channel for the RPC request as soon as a wake word is received from the hot word handler 608, so as to reduce a latency experienced by a user of the device 604.

[0087] The cloud server 616 may utilize one or more generative application program interface(s) (APIs) 618 to communicate with an environment generator 620, e.g., to call correct models of the environment generator 620, or to call the correct environment generator 620. Although not illustrated separately in FIG. 6, it will be appreciated that the environment generator 620 may include one or more audio models, visual models, and combination pipeline(s). In general, use of the cloud server 616 and related resources to host and access the environment generator 620 enables the use of large models capable of generating detailed and accurate immersive environments across a wide range of user requests.

[0088] An immersive application 622 may be configured to receive various media files from the environment generator 620 via the cloud server 616, for rendering as a photosphere 624 with corresponding, semantically matched, spatialized audio. For example, the local resources 602 may all be implemented by the device 604 itself, or one or more of the local resources 602 other than the device 604 may be implemented using a separate local device, such as a laptop or desktop computer.

[0089] Additionally, the architecture of FIG. 6 enables other optimizations to enhance and optimize features of the generated photosphere 624. For example, the photosphere 624 may include dy namic elements to enhance a realism of the corresponding immersive environment. For example, the photosphere 624 may include a river, and the rendering of the river may include multiple frames rendered in succession and / or on a loop, so as to simulate aflowing motion of the river.

[0090] In these and similar examples, the cloud server 616 may transmit a batch of generated image frames to the local resources 602. Then, the immersive application 622 may be configured to render the photosphere 624 including iteratively looping through the batch of frames associated with rendering the motion of the river, or any other desired environment motions or renderings.

[0091] Thus, FIGS. 1-6 illustrate example techniques for implementing query -based ways to create compelling multimedia assets using, e.g., diffusion, autoregression, and / or sequence-to-sequence modeling tools. As a result, goggles and other HMDs may be facilitated in providing various types of immersive applications, including, e.g., co-presence environments, infinite screen office space environments, and various gaming or design environments. Described techniques provide 360°, 3D audiovisual asset generation models that enable new, user-friendly extended reality (XR) scenes with low latency, including asset(s) creation interface with a live speech prompt.

[0092] FIG. 7 is a third person view of a user 702 (analogous to the user 104 of FIG. 1) in an ambient environment 7000, with one or more external computing systems shown as additional resources 752 that are accessible to the user 702 via a network 7200. FIG. 7 illustrates numerous different wearable devices that are operable by the user 702 on one or more body parts of the user 702, including a first wearable device 750 in the form of goggles worn on the head of the user, a second wearable device 754 in the form of ear buds worn in one or both ears of the user 702, a third wearable device 756 in the form of a watch worn on the wrist of the user, and a computing device 706 held by the user 702. In FIG. 7, the computing device 706 is illustrated as a handheld computing device but may also be understood to represent any personal computing device, such as a laptop, tablet, or personal computer.

[0093] In some examples, the first wearable device 750 is in the form of smart goggles or other type of HMD, including, for example, a display, one or more images sensors that can capture images of the ambient environment, audio input / output devices, user input capability, computing / processing capability and the like. More generally, the first wearable device 750 may represent virtually any type of virtual reality (VR) device or augmented reality (AR) device. The first wearable device 750 includes a display system configured to present a virtual or augmented reality environment on one or more displays. The display(s) may include one or more optical components configured to direct and focus content. The optical component(s) may include lenses and / or waveguides. In some examples, the displayincludes a near-eye display that uses liquid crystal display (LCD) or organic light-emitting diode (OLED). In some examples, the display includes a projection display, where virtual images are projected onto a semi-transparent screen that is positioned in front of the user’s eyes, which may use micro-projectors or waveguides.

[0094] In some examples, the second wearable device 754 is in the form of an ear worn computing device such as headphones, or earbuds, that can include audio input / output capability, an image sensor that can capture images of the ambient environment 7000, computing / processing capability, user input capability and the like. In some examples, the third wearable device 756 is in the form of a smart watch or smart band that includes, for example, a display, an image sensor that can capture images of the ambient environment, audio input / output capability, computing / processing capability, user input capability and the like. In some examples, the handheld computing device 706 can include a display, one or more image sensors that can capture images of the ambient environment, audio input / output capability, computing / processing capability, user input capability, and the like, such as in a smartphone. In some examples, the example wearable devices 750, 754, 756 and the example handheld computing device 706 can communicate with each other and / or with external computing system(s) 752 to exchange information, to receive and transmit input and / or output, and the like. The principles to be described herein may be applied to other types of wearable devices not specifically shown in FIG. 7 or described herein.

[0095] The user 702 may choose to use any one or more of the devices 706, 750. 754, or 756, perhaps in conjunction with the external resources 752, to implement any of the implementations described above with respect to FIGS. 1-6. For example, the user 702 may use an application executing on the device 706, the goggles 750, and additional resources 752 to execute the environment generator 128 of FIG. 1.

[0096] As referenced above, the device 706 may access the additional resources 752 (which may include the cloud server 138 of FIG. 1) to facilitate the various environment generation techniques described herein, or related techniques. In some examples, the additional resources 752 may be partially or completely available locally on the device 706. In some examples, some of the additional resources 752 may be available locally on the device 706, and some of the additional resources 752 may be available to the device 706 via the network 7200. As shown, the additional resources 752 may include, for example, server computer systems, processors, databases, memory storage, and the like. In some examples, the processor(s) may include training engine(s). transcription engine(s), translation engine(s), rendering engine(s), and other such processors. In some examples, the additional resourcesmay include ML model(s), such as the visual model 130 or the audio model 132 of FIG. 1.

[0097] The device 706 may operate under the control of a control system 760. The device 706 can communicate with one or more external devices, either directly (via wired and / or wireless communication), or via the network 7200. In some examples, the one or more external devices may include various ones of the illustrated wearable computing devices 750, 754, 756. another mobile computing device similar to the device 706, and the like. In some implementations, the device 706 includes a communication module 762 to facilitate external communication. In some implementations, the device 706 includes a sensing system 764 including various sensing system components. The sensing system components may include, for example, one or more image sensors 765, one or more position / orientation sensor(s) 764 (including for example, an inertial measurement unit, an accelerometer, a gyroscope, a magnetometer and other such sensors), one or more audio sensors 766 that can detect audio input, one or more image sensors 767 that can detect visual input, one or more touch input sensors 768 that can detect touch inputs, and other such sensors. The device 706 can include more, or fewer, sensing devices and / or combinations of sensing devices. Various ones of the various sensors may be used individually or together to perform the types of sahency detection described herein.

[0098] Still and / or moving images may be displayed by a display device of an output system 772, and / or transmitted externally via a communication module 762 and the network 7200, and / or stored in a memory 770 of the device 706. The device 706 may include one or more processor(s) 774. The processors 774 may include various modules or engines configured to perform various functions. In some examples, the processor(s) 774 may include, e.g, training engine(s), transcription engine(s), translation engine(s), rendering engine(s), and other such processors. The processor(s) 774 may be formed in a substrate configured to execute one or more machine executable instructions or pieces of software, firmware, or a combination thereof. The processor(s) 774 can be semiconductor-based including semiconductor material that can perform digital logic. The memory 770 may include any type of storage device or non-transitory computer-readable storage medium that stores information in a format that can be read and / or executed by the processor(s) 774. The memory 770 may store applications and modules that, when executed by the processor(s) 774, perform certain operations. In some examples, the applications and modules may be stored in an external storage device and loaded into the memory 770.

[0099] Although not shown separately in FIG. 7, it will be appreciated that the various resources of the computing device 706 may be implemented in whole or in partwithin one or more of various wearable devices, including the illustrated goggles 750, earbuds 754, and smartwatch 756, which may be in communication with one another to provide the various features and functions described herein.

[0100] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry', integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0101] These computer programs (also known as modules, programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms “machine- readable medium” “computer-readable medium” refers to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0102] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or LED (light emitting diode)) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well. For example, feedback provided to the user can be any form of sensory feedback (e.g.. visual feedback, auditory’ feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0103] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which auser can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN’'), a wide area network (“WAN”), and the Internet.

[0104] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0105] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the description and claims.

[0106] In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components maybe added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.

[0107] Further to the descriptions above, a user is provided with controls allowing the user to make an election as to both if and when systems, programs, devices, networks, or features described herein may enable collection of user information (e.g.. information about a user’s social network, social actions, or activities, profession, a user’s preferences, or a user’s current location), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that user information is removed. For example, a user's identity may be treated so that no user information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0108] The computer system (e.g., computing device) may be configured to wirelessly communicate with a network server over a network via a communication link established with the netw ork server using any known wireless communications technologies and protocols including radio frequency (RF). microw ave frequency (MWF). and / or infraredfrequency (IRF) wireless communications technologies and protocols adapted for communication over the network.

[0109] In accordance with aspects of the disclosure, implementations of various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may be implemented as a computer program product (e.g., a computer program tangibly embodied in an information carrier, a machine-readable storage device, a computer-readable medium, a tangible computer-readable medium), for processing by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). In some implementations, a tangible computer-readable storage medium may be configured to store instructions that when executed cause a processor to perform a process. A computer program, such as the computer program(s) described above, may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be processed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

[0110] Specific structural and functional details disclosed herein are merely representative for purposes of describing example implementations. Example implementations, however, may be embodied in many alternate forms and should not be construed as limited to only the implementations set forth herein.

[0111] The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the implementations. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes," and / or "including," when used in this specification, specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0112] It will be understood that when an element is referred to as being "coupled," "connected," or "responsive" to, or "on," another element, it can be directly coupled, connected, or responsive to, or on, the other element, or intervening elements may also be present. In contrast, when an element is referred to as being "directly coupled," "directly connected," or "directly responsive" to, or "directly on," another element, there are nointervening elements present. As used herein the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0113] Spatially relative terms, such as "beneath," "below," "lower," "above," "upper," and the like, may be used herein for ease of description to describe one element or feature in relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be oriented "above" the other elements or features. Thus, the term "below" can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 130 degrees or at other orientations) and the spatially relative descriptors used herein may be interpreted accordingly.

[0114] Example implementations of the concepts are described herein with reference to cross-sectional illustrations that are schematic illustrations of idealized implementations (and intermediate structures) of example implementations. As such, variations from the shapes of the illustrations as a result, for example, of manufacturing techniques and / or tolerances, are to be expected. Thus, example implementations of the described concepts should not be construed as limited to the particular shapes of regions illustrated herein but are to include deviations in shapes that result, for example, from manufacturing. Accordingly, the regions illustrated in the figures are schematic in nature and their shapes are not intended to illustrate the actual shape of a region of a device and are not intended to limit the scope of example implementations.

[0115] It will be understood that although the terms "first." "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Thus, a "first" element could be termed a "second" element without departing from the teachings of the present implementations.

[0116] Unless otherwise defined, the terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which these concepts belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and / or the present specification and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0117] While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes, and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover such modifications and changes as fall within the scope of the implementations. It should be understood that they have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and / or methods described herein may be combined in any combination, except mutually exclusive combinations. The implementations described herein can include various combinations and / or sub-combinations of the functions, components, and / or features of the different implementations described.

Claims

WHAT IS CLAIMED IS:

1. A method comprising: receiving a textual request for a three-dimensional (3D) environment; processing the textual request using a text-to-image machine learning (ML) model to generate a 3D visual representation of the 3D environment; processing the textual request using a text-to-audio ML model to generate an audio file corresponding to at least one audio source associated with a location within the 3D environment; and combining the 3D visual representation and the audio file to generate the 3D environment with the at least one audio source at the location within the 3D environment.

2. The method of claim 1, further comprising: receiving an audio request for the 3D environment; and converting the audio request to text to obtain the textual request.

3. The method of claim 1, wherein the processing the textual request using the text-to-image machine learning (ML) model further comprises: using at least two 2D images generated by the text-to-image ML model.

4. The method of claim 3, further comprising: training the text-to-image ML model including specifying a boundary' condition to minimize differences between adjoining pixels of the at least two 2D images.

5. The method of claim L further comprising: determining visual semantics of the 3D visual representation; determining audio semantics of the audio file; and combining the 3D visual representation and the audio file based on the visual semantics and the audio semantics to generate the 3D environment with the at least one audio source at the location within the 3D environment.

6. The method of claim 1. further comprising: processing the textual request using the text-to-image machine learning (ML) model to generate a first 3D visual representation of the 3D environment;processing the first 3D visual representation using an image-to-text ML model to obtain a first visual summan’; processing the textual request using the text-to-audio ML model to generate a first audio file corresponding to the at least one audio source; processing the first audio file using an audio-to-text ML model to obtain a first audio summary; determining that a first comparison of the first visual summary and the first audio summary is below a similarity threshold; processing the textual request and the first visual summary’ using the text-to-image machine learning (ML) model to generate a second 3D visual representation of the 3D environment; processing the second 3D visual representation using the image-to-text ML model to obtain a second visual summary; processing the textual request using the text-to-audio ML model to generate a second audio file corresponding to the at least one audio source; processing the second audio file using the audio-to-text ML model to obtain a second audio summary; determining that a second comparison of the second visual summary and the second audio summary is above the similarity threshold; and using the second 3D visual representation as the 3D visual representation and the second audio file as the audio file, based on the second comparison.

7. The method of claim 1, further comprising: determining an image-to-text summary’ of the 3D visual representation; determining an audio-to-text summary of the audio file; determining audio characteristics of the 3D visual representation and of the audio file, based on the image-to-text summary and the audio-to-text summary; and combining the 3D visual representation and the audio file to generate the 3D environment with the audio characteristics.

8. The method of claim 1, further comprising: mapping the audio file to binaural speakers using a binaural rendering driver.

9. The method of claim 1, further comprising: mapping the 3D visual representation to a stereoscopic display using a stereoscopic mapping engine.

10. The method of claim 1, further comprising: displaying the 3D environment using a virtual reality (VR) headset.

11. A computer program product, the computer program product being tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to: receive a textual request for a three-dimensional (3D) environment; process the textual request using a text-to-image machine learning (ML) model to generate a 3D visual representation of the 3D environment; process the textual request using a text-to-audio ML model to generate an audio file corresponding to at least one audio source associated with a location within the 3D environment; and combine the 3D visual representation and the audio file to generate the 3D environment with the at least one audio source at the location within the 3D environment.

12. The computer program product of claim 1 1, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: process the textual request using the text-to-image machine learning (ML) model to generate the 3D visual representation of the 3D environment, including using at least two 2D images generated by the text-to-image ML model.

13. The computer program product of claim 11, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: determine visual semantics of the 3D visual representation; determine audio semantics of the audio file; andcombine the 3D visual representation and the audio file based on the visual semantics and the audio semantics to generate the 3D environment with the at least one audio source at the location within the 3D environment.

14. The computer program product of claim 11, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: process the textual request using the text-to-image machine learning (ML) model to generate a first 3D visual representation of the 3D environment; process the first 3D visual representation using an image-to-text ML model to obtain a first visual summary; process the textual request using the text-to-audio ML model to generate a first audio file corresponding to the at least one audio source; process the first audio file using an audio-to-text ML model to obtain a first audio summary; determine that a first comparison of the first visual summary and the first audio summary is below a similarity threshold; process the textual request and the first visual summary using the text-to-image machine learning (ML) model to generate a second 3D visual representation of the 3D environment: process the second 3D visual representation using the image-to-text ML model to obtain a second visual summary; process the textual request using the text-to-audio ML model to generate a second audio file corresponding to the at least one audio source; process the second audio file using the audio-to-text ML model to obtain a second audio summary; determine that a second comparison of the second visual summary' and the second audio summary is above the similarity threshold; and use the second 3D visual representation as the 3D visual representation and the second audio file as the audio file, based on the second comparison.

15. The computer program product of claim 11, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to:determine an image-to-text summary' of the 3D visual representation; determine an audio-to-text summary of the audio fde; determine audio characteristics of the 3D visual representation and of the audio file, based on the image-to-text summary' and the audio-to-text summary; and combine the 3D visual representation and the audio file to generate the 3D environment with the audio characteristics.

16. A wearable device comprising: at least one frame for positioning the wearable device on a body of a user; at least one display; binaural speakers; at least one processor; and at least one memory, the at least one memory storing a set of instructions, which, when executed, cause the at least one processor to: receive a textual request for a three-dimensional (3D) environment; process the textual request using a text-to-image machine learning (ML) model to generate a 3D visual representation of the 3D environment; process the textual request using a text-to-audio ML model to generate an audio file corresponding to at least one audio source associated with a location within the 3D environment: and combine the 3D visual representation and the audio file to generate the 3D environment with the at least one display and with the binaural speakers, and with the at least one audio source at the location within the 3D environment.

17. The wearable device of claim 16, wherein the set of instructions, when executed by the at least one processor, are further configured to cause the wearable device to: process the textual request using the text-to-image machine learning (ML) model to generate the 3D visual representation of the 3D environment, including using at least two 2D images generated by the text-to-image ML model.

18. The wearable device of claim 16, wherein the set of instructions, when executed by the at least one processor, are further configured to cause the wearable device to: determine visual semantics of the 3D visual representation; determine audio semantics of the audio file; andcombine the 3D visual representation and the audio file based on the visual semantics and the audio semantics to generate the 3D environment with the at least one audio source at the location within the 3D environment.

19. The wearable device of claim 16, wherein the set of instructions, when executed by the at least one processor, are further configured to cause the wearable device to: process the textual request using the text-to-image machine learning (ML) model to generate a first 3D visual representation of the 3D environment; process the first 3D visual representation using an image-to-text ML model to obtain a first visual summary; process the textual request using the text-to-audio ML model to generate a first audio file corresponding to the at least one audio source; process the first audio file using an audio-to-text ML model to obtain a first audio summary; determine that a first comparison of the first visual summary and the first audio summary is below a similarity threshold; process the textual request and the first visual summary using the text-to-image machine learning (ML) model to generate a second 3D visual representation of the 3D environment: process the second 3D visual representation using the image-to-text ML model to obtain a second visual summary; process the textual request using the text-to-audio ML model to generate a second audio file corresponding to the at least one audio source; process the second audio file using the audio-to-text ML model to obtain a second audio summary; determine that a second comparison of the second visual summary and the second audio summary is above the similarity threshold; and use the second 3D visual representation as the 3D visual representation and the second audio file as the audio file, based on the second comparison.

20. The wearable device of claim 16, wherein the set of instructions, when executed by the at least one processor, are further configured to cause the wearable device to: determine an image-to-text summary’ of the 3D visual representation; determine an audio-to-text summary of the audio file;determine audio characteristics of the 3D visual representation and of the audio file, based on the image-to-text summary and the audio-to-text summan’; and combine the 3D visual representation and the audio file to generate the 3D environment with the audio characteristics.

Images