Real-time virtual reality scene system based on natural language description using multimodal artificial intelligence

Abstract

A real-time system for the multimodal generation of virtual reality scenes based on artificial intelligence for the creation of immersive three-dimensional environments from natural language narratives, consisting of: a speech capture module configured to continuously record a user's spoken narrative via one or more directional microphones, preprocesses the captured signal by noise reduction and temporal alignment, and outputs a digital speech stream; A speech-to-text processing unit that is operationally coupled to the speech capture module and configured for real-time speech recognition using a continuous neural transformer model. The unit is trained to transcribe natural language utterances into structured text data while maintaining contextual continuity throughout the evolving narrative. a semantic interpretation processing unit that is communicatively linked to the speech recognition unit and configured to perform natural language understanding techniques to extract contextual entities, spatial references, temporal relationships, and object attributes from the transcribed narrative; the engine includes a large language model that is fine-tuned for spatial reasoning tasks; a scene graph generation module configured to transform the interpreted semantic data into a structured, hierarchical representation that defines nodes for identified entities and edges for corresponding relationships, with each node associated with metadata describing geometry, position, orientation, texture, and linking attributes between objects; a multimodal image-language model processor coupled with the scene graph generation module, wherein the processor is configured to retrieve, adapt, or synthesize appropriate three-dimensional elements from a pre-trained visual-lexical embedding space and align these elements with their semantic and spatial definitions derived from the scene graph; a scene assembly and rendering controller configured to create a cohesive virtual scene from the aligned assets, perform real-time rendering using a GPU-accelerated ray tracing pipeline, and produce a stereoscopic visual output that corresponds to the evolving narrative; A head-mounted virtual reality visualization device connected to the rendering engine and configured to display the generated immersive environment to the user in real time. The device features motion sensors and inside-out tracking cameras to detect head and body movements, dynamically updating viewing angles and perspective within the rendered scene; and a bidirectional feedback module integrated into the head-mounted device and connected to the semantic interpretation processing unit; the module is configured to interpret corrective commands, gestures, or supplementary comments from the user to refine or modify specific scene elements without interrupting the real-time visualization; The system continuously updates the virtual scene as the narrative develops, ensuring temporal synchronization between speech input and rendered output below a defined latency threshold, thus enabling a natural, dialogic construction of complex three-dimensional virtual environments.

Description

TECHNICAL FIELD

[0001] The present invention relates to the fields of artificial intelligence, virtual reality (VR), and human-computer interaction. In particular, it relates to a real-time system and a device architecture for creating immersive three-dimensional virtual scenes directly from spoken natural language using integrated multimodal AI models. The invention is particularly applicable in law enforcement and training scenarios, but can also be extended to general scene generation, simulation development, and learning environments. BACKGROUND OF THE INVENTION

[0002] Existing systems for generating virtual reality scenes rely heavily on manual user input via graphical interfaces, predefined 3D object libraries, or script-based workflows. These conventional approaches require specialized design expertise and are incapable of dynamically generating immersive scenes directly from unstructured spoken narratives.

[0003] Current technologies in voice-controlled VR systems allow for limited object manipulation or the execution of predefined commands, but are unable to generate coherent, context-aware 3D environments from spoken language. Text-to-image generation tools and AI-powered visualization systems can also produce static images, but do not support real-time interactive VR displays based on natural speech. Therefore, there is an urgent need for a system that can autonomously interpret spoken descriptions, extract spatial and semantic information, and render a dynamically updated virtual scene for head-mounted displays.

[0004] The development of large language models (LLMs) and image-speech fusion models has made it technically possible to interpret natural language contextually and translate it into structured representations suitable for visual display. However, integrating these capabilities into a closed, real-time system for immersive VR visualization with feedback and correction mechanisms remains a technical challenge, which the present invention successfully overcomes.

[0005] The development of virtual reality (VR) and artificial intelligence (AI) has fundamentally changed how humans interact with digital environments. VR, once limited to entertainment and simulations, is now a powerful medium for immersive visualization, education, training, and real-time analysis. However, creating VR environments remains a technically demanding process requiring extensive manual planning, modeling, and configuration. While the integration of AI, particularly natural language processing and computer vision, has opened up possibilities for automating aspects of scene creation, current solutions are still fragmented and far from achieving a seamless, real-time translation of spoken descriptions into fully interactive 3D environments.The present invention addresses these long-standing challenges by introducing an AI-supported VR scene generator in real time, capable of interpreting spoken language and converting it into dynamically rendered visual environments.

[0006] Traditionally, the creation of virtual scenes begins with 3D modeling using specialized software such as Blender, Autodesk Maya, or Unity. Designers manually import meshes, adjust textures, define lighting, and place objects according to the desired layout. Even minor changes to the environment require human intervention. This manual reliance not only slows down development cycles but also makes the technology inaccessible to users without expertise in 3D design or spatial computing.

[0007] To address these issues, researchers have explored semi-automated scene generation using predefined templates or parameterized design models. For example, architectural visualization platforms allow the generation of rooms or city maps by adjusting overarching parameters such as dimensions or material properties. While these systems increase efficiency, they still rely on structured input rather than free speech or natural language. Similarly, game engines have adopted procedural generation techniques that create landscapes, terrains, or cities using random rules, noise functions, or grammars. Although these techniques produce complex and diverse environments, they lack context and cannot interpret human input to generate semantically meaningful scenes.Despite advances in procedural and template-based modeling, the process therefore remains decoupled from natural language-driven workflows.

[0008] In the area of ​​voice interaction, existing VR systems primarily use voice commands for control, but not for content creation. Platforms like NVIDIA Omniverse or Autodesk VRED integrate basic speech recognition systems to perform limited functions, such as rotating an object, changing a color, or loading a predefined element. However, these systems are limited to a closed vocabulary and require users to input specific, pre-programmed commands. They cannot interpret open, narrative language that describes a scene with diverse spatial and contextual dependencies. For example, a user might say, "Place a wooden table in the middle of the room and a chair next to it facing the window." Existing systems cannot infer the spatial relationships or material properties implied in this statement.Therefore, current VR systems do not achieve natural, dialogic interaction in the creation and editing of scenes.

[0009] Another emerging area in this field is text-to-image and text-to-video generation using deep learning. These systems are based on extensive image-speech pre-training, enabling them to link words and phrases with visual features. However, the results of such systems are static and two-dimensional, and therefore unsuitable for real-time 3D or VR applications. They do not offer spatially consistent environments that can be navigated or interacted with in virtual space. Furthermore, their rendering processes are computationally intensive and typically operate in batch mode, preventing the low-latency feedback necessary for real-time visualization.

[0010] Some academic research explores text-to-3D generation, where a neural model creates a volumetric representation or mesh based on text input. Tools like DreamFusion and Point-E represent early attempts at this paradigm. While these systems demonstrate the ability to synthesize 3D content from descriptive language, they still require extensive post-processing and optimization before the generated objects can be used in a VR scene. Furthermore, these models operate offline and generate static 3D assets rather than continuously evolving environments. They lack the ability to dynamically update scenes in response to an ongoing narrative, a crucial capability for real-world applications such as law enforcement and training simulations.

[0011] In the field of training and simulation, most virtual environments for police and emergency services training are predefined and static. Trainees are exposed to only a limited number of predefined scenarios, which lack variation and adaptability. Integrating an instructor-driven narrative-based scene generation mechanism could significantly improve the realism and variety of exercises and enable the creation of unpredictable, context-rich situations on demand. Current systems do not achieve this dynamic adaptability because they are based on pre-built elements and scenarios, which limits the educational value and scalability of training programs.

[0012] Several commercial VR environment building blocks, including ReadySet VR and MindPort, have attempted to simplify environment creation through intuitive graphical interfaces and drag-and-drop functionality. While these tools reduce the complexity of 3D modeling, they still require manual interaction, keyboard or controller input, and visual design skills. None of these systems are capable of autonomously understanding spoken language, extracting spatial information, and generating a coherent 3D environment in real time. Therefore, these systems are unsuitable for time-critical applications where fast and accurate speech-based visualization is essential.

[0013] Recent developments in multimodal AI enable systems to process text and image input simultaneously. This has led to the emergence of image-speech models that can analyze scenes and generate relevant images. Models like CLIP and Flamingo combine the representational capabilities of rich language models with image processing capabilities. However, their integration into a fully interactive VR rendering pipeline remains largely unexplored. Existing implementations typically focus on classification or image description rather than real-time, scene-level synthesis. To bridge this gap, a tightly coupled architecture is needed that can interpret complex language semantics, ensure spatial coherence, and render 3D environments with low latency—all in response to continuous language input.

[0014] In addition to the technical challenges, current solutions exhibit operational inefficiencies in feedback and validation. Once a VR scene is generated—whether manually or automatically—correcting or optimizing it becomes cumbersome. Users must leave the immersive environment, change parameters via a conventional user interface, and re-render the scene. This interrupts the immersion and slows down the workflow.

[0015] Furthermore, real-time synchronization of narrative and scene rendering presents a computational challenge that existing systems have been unable to overcome. Rendering engines optimized for static or predefined scenes struggle to dynamically update geometries and lighting as new information is added. Maintaining a stable frame rate and spatial consistency under these conditions requires advanced optimization strategies, memory management, and adaptive rendering techniques that are lacking in conventional systems. Consequently, attempts to extend existing text-to-image pipelines to interactive VR contexts result in significant latency, reduced realism, and an unsatisfactory user experience.

[0016] In summary, existing scene generation technologies—whether through manual design, template-based modeling, procedural generation, or AI-powered visualization—do not provide an integrated, real-time system that autonomously creates, updates, and refines virtual environments based on evolving natural language output. They lack either contextual understanding, interactivity, scalability, or real-time capability. A significant technological gap remains in combining multimodal AI interpretation, dynamic 3D scene synthesis, and immersive VR rendering into a seamless, user-driven system.The present invention overcomes these shortcomings by introducing a robust, end-to-end pipeline that continuously interprets language, extracts semantic and spatial meaning, creates coherent virtual environments, and enables immersive real-time visualization with integrated feedback and correction mechanisms. This innovation fundamentally redefines the interface between human language and digital spatial design, bridging the long-standing gap between verbal imagination and virtual manifestation. OBJECTS OF INVENTION

[0017] The main objective of the invention is to provide a multimodal AI-based real-time system that creates and updates immersive virtual environments from spoken narratives.

[0018] Another goal is the integration of advanced language and image processing models for the interpretation of semantic, spatial, and contextual features contained in natural language input.

[0019] Another goal is to display the generated virtual scene in real time in a head-mounted VR device, allowing users to visualize, explore, and refine the environment as the narrative unfolds.

[0020] Another goal is the integration of user feedback mechanisms that enable scene validation and correction directly within the immersive environment.

[0021] Another goal is to improve situational understanding, reduce ambiguities, and increase efficiency in investigations, training, and operational activities. SUMMARY OF THE INVENTION

[0022] The invention describes a real-time system for generating virtual reality scenes based on spoken descriptions and utilizing an integrated pipeline of multimodal AI models. The system comprises a speech recognition module, a semantic parser with large language models, an image-to-speech sequence synthesis engine, and a real-time rendering engine connected to a head-mounted display. The system interprets the user's description to identify objects, relationships, and environmental cues, which are then translated into 3D objects and spatial configurations within a virtual environment.

[0023] As the narrative progresses, the system continuously updates the scene to reflect new or changed information. Integrated user feedback via gestures or voice commands enables corrections and optimizations. The invention's modular AI framework ensures adaptability, scalability, and precise visual representation, making it ideal for applications such as crime scene reconstruction, emergency situation visualization, and immersive learning environments.

[0024] The present invention aims to provide an AI-powered system for the real-time generation of virtual reality scenes, capable of creating immersive three-dimensional environments directly from spoken language. The invention seeks to eliminate reliance on manual modeling, graphical interfaces, or predefined templates by enabling a seamless and automated translation of spoken descriptions into dynamic visual environments that accurately reflect the speaker's intention. A further objective of the invention is the integration of large language models and multimodal image processing systems to understand the semantics, spatial relationships, and contextual features contained in the spoken language, thereby generating contextually coherent and visually realistic scenes that evolve as the narrative progresses.

[0025] Another important objective of the invention is the real-time display and modification of scenes using an integrated virtual reality interface. This allows users to view, interact with, and customize generated environments during speech output. The system continuously updates the virtual scene synchronously with the speech input, ensuring that every verbally described change or addition is immediately visible in the visual output without requiring manual reconfiguration. This feature not only accelerates scene creation but also increases accuracy by ensuring the correspondence between the user's speech output and the displayed virtual environment.

[0026] Another objective of the invention is the integration of an intelligent feedback and correction mechanism into the VR interface. This mechanism allows users such as investigators, trainees, or instructors to provide direct verbal or gestural feedback to modify, correct, or refine specific elements of the scene. For example, if an object is incorrectly placed or inaccurately described, the user can immediately issue corrective commands—such as repositioning, scaling, or replacing elements—without leaving the immersive environment. This feature promotes precision and adaptability and enables the continuous improvement of scene fidelity during interactive use.

[0027] The invention also includes the provision of a modular architecture that enables scalable deployment on various hardware platforms, including desktop computers, cloud-based AI processors, and standalone VR headsets. The system design allows for flexible distribution of the computational load: performance-intensive tasks such as semantic parsing and multimodal scene synthesis are performed on dedicated AI accelerators, while resource-efficient rendering and visualization operations are handled locally by the VR device.

[0028] By transforming verbal crime scene descriptions into precise visual reconstructions, the invention minimizes ambiguity, reduces the risk of misunderstandings, and provides investigators and decision-makers with a shared, immersive perspective of the scenario. This enables more effective collaboration between officers, analysts, prosecutors, and jurors, who can jointly examine and validate the reconstructed crime scene in the same virtual environment. The system's ability to visually represent complex spatial relationships also supports evidence interpretation and hypothesis testing, thereby improving the accuracy and reliability of investigations.

[0029] Another objective of the invention is the further development of immersive learning and training methods through the spontaneous creation of scenarios. Conventional training environments in police work, firefighting, or medical simulation are based on static or pre-defined scenarios that limit realism and adaptability. The proposed invention enables instructors to verbally describe dynamic training situations—such as hostage rescues, accident responses, or disaster scenarios—which are immediately visualized as interactive VR environments. Participants can dynamically experience and react to these scenarios, thereby improving learning outcomes and decision-making skills. This adaptive content generation provides a versatile platform for continuous training without requiring manual scene design or programming skills.

[0030] The invention aims to solve the long-standing problem of accessibility in VR content creation by enabling users without technical knowledge of 3D modeling or programming to create high-quality virtual scenes using only natural language. By harnessing the expressive power of spoken communication, the invention democratizes the process of creating immersive content and allows professionals from various fields—such as law enforcement, education, healthcare, urban planning, or entertainment—to visualize and interact with their ideas in three dimensions. This invention aligns with the overarching vision of making immersive visualization technology universally accessible, thereby lowering barriers to entry for individuals and institutions.

[0031] The invention aims to ensure computational efficiency and synchronization between speech interpretation and scene rendering. One objective is to maintain sub-second latency between spoken input and corresponding visual output, thereby preserving the natural rhythm of interaction. This involves developing optimized data flow architectures and adaptive caching mechanisms that prioritize the rendering of important scene components while loading background elements gradually. This performance is crucial for maintaining immersion in the virtual world and ensures that the virtual environment evolves smoothly and harmoniously with the user's narrative.

[0032] The invention also includes the integration of validation mechanisms that evaluate the consistency, completeness, and plausibility of the generated scenes in relation to the spoken description. The system uses AI-based reasoning to detect inconsistencies—such as contradictory spatial relationships or physically implausible configurations—and request clarification from the user. This intelligent validation not only improves the accuracy of the scene representation but also strengthens the user's confidence in the system's interpretive capabilities, thus establishing it as a reliable tool for professional and investigative applications where factual accuracy is essential.

[0033] Another objective of the invention is to support collaborative and multi-user functions that allow multiple participants to interact simultaneously in a shared virtual environment. For example, during a crime scene reconstruction, several officers can view, comment on, or edit the same crime scene in real time from different locations, using synchronized comments and gesture input. This collaborative mode improves communication efficiency, facilitates distributed decision-making, and promotes a shared understanding of spatial and contextual information among the participants.

[0034] Furthermore, the invention aims to improve scalability and data interoperability by supporting standardized scene representation formats and integrating with existing VR platforms. This includes the use of scene graphs, semantic ontologies, and 3D asset libraries that comply with open standards such as USD (Universal Scene Description) or g1TF. This compatibility ensures that generated scenes can be exported, archived, or integrated into third-party systems for further analysis or presentation. The goal is to make the invention future-proof and adaptable to evolving VR ecosystems.

[0035] A key objective of the invention is to demonstrate the technological advancement of merging multimodal artificial intelligence with immersive, real-time rendering to create a new class of intelligent, narrative-driven virtual environments. By unifying natural language processing, spatial reasoning, and visual synthesis within a single adaptive framework, the invention introduces a transformative approach to human-computer interaction—one in which spoken imagination becomes an immediately explorable visual reality. This advancement not only addresses long-standing challenges in VR content creation but also lays the foundation for new applications in simulation, digital storytelling, evidence visualization, and cognitive computing.The invention thus represents a significant advance in the convergence of AI, VR and human expression, transforming verbal communication into a direct medium for the construction of digital worlds. BRIEF DESCRIPTION OF THE IMAGE

[0036] These and other features, aspects and advantages of the present invention will be better understood if the following detailed description is read with reference to the accompanying drawing, in which the same symbols represent the same parts: Fig. Figure 1 shows a block diagram of a real-time virtual reality scene generator from natural language narration using multimodal artificial intelligence.

[0037] Furthermore, those skilled in the art will recognize that the elements in the drawing are simplified and not necessarily drawn to scale. For example, the flowcharts illustrate the process by highlighting the main steps to facilitate understanding of the present disclosure. With regard to the construction of the device, one or more components may be represented in the drawing by conventional symbols. The drawing may show only those specific details relevant to understanding the embodiments of the present disclosure, so as not to clutter the drawing with details that are already apparent to those skilled in the art from the description contained herein. Detailed description of the invention

[0038] To facilitate understanding of the principles of the invention, reference is made below to the embodiment shown in the drawing, which is described using specific terms. It is understood, however, that this does not limit the scope of protection of the invention. Rather, modifications and further developments of the depicted system, as well as further applications of the inventive principles shown therein, are conceivable, insofar as they would normally occur to a person skilled in the art in the field of the invention.

[0039] It will be clear to those skilled in the art that the foregoing general description and the following detailed description are exemplary and explanatory of the invention and are not to be understood as a limitation of it.

[0040] References to “an aspect”, “another aspect”, or similar phrases in this description mean that a particular feature, structure, or property described in connection with the embodiment is included in at least one embodiment of the present disclosure. Therefore, phrases such as “in one embodiment”, “in another embodiment”, and similar expressions in this description may, but do not necessarily, all refer to the same embodiment.

[0041] The terms "includes," "comprehensive," or similar expressions denote non-exclusive inclusion. Thus, a procedure or method containing a list of steps does not only include those steps but may also include further steps not explicitly listed or inherent in the procedure or method. Likewise, the statement "includes..." for one or more devices, subsystems, elements, structures, or components, without further limitations, does not preclude the existence of other devices, subsystems, elements, structures, or components.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meanings generally known to those skilled in the art in the field to which this invention belongs. The systems, methods, and examples described herein serve only for illustration and are not to be understood as limiting.

[0043] Embodiments of the present disclosure are described in detail below with reference to the attached drawing.

[0044] In Fig.Figure 1 is a block diagram of a real-time virtual reality scene generator from natural language narration using multimodal artificial intelligence. The system 100 comprises: a speech capture module (102) configured to continuously record a user's spoken language via one or more directional microphones, preprocess the captured signal by noise reduction and temporal alignment, and output a digital speech stream; a speech-to-text processing unit (104) operationally coupled to the speech capture module and configured to perform real-time speech recognition using a continuous neural transformer model. The unit is trained to transcribe natural language utterances into structured text data while maintaining context continuity throughout the evolving narrative; and a semantic interpretation processing unit (106).which is communicatively connected to the speech-to-text unit and configured to apply natural language understanding techniques to extract contextual information, spatial references, temporal relationships, and object attributes from the transcribed narrative. The engine includes a large language model fine-tuned for spatial reasoning tasks; a scene graph generation module (108) configured to transform the interpreted semantic data into a structured, hierarchical representation that defines nodes for identified entities and edges for corresponding relationships, with each node associated with metadata describing geometry, position, orientation, texture, and linking attributes between objects; and a multimodal image processing and speech modeling processor (110) coupled to the scene graph generation module and configured tothat it retrieves, adapts, or synthesizes appropriate three-dimensional elements from a pre-trained visual-lexical embedding space and aligns these elements with their semantic and spatial definitions derived from the scene graph; a scene assembly and rendering controller (112) configured to create a coherent virtual scene from the aligned elements, performs real-time rendering using a GPU-accelerated ray tracing pipeline, and generates a stereoscopic visual output that corresponds to the evolving narrative; a head-mounted virtual reality visualization device (114) connected to the rendering engine and displaying the generated immersive environment in real time, incorporating motion sensors and inside-out tracking cameras for capturing head and body movements,to dynamically update the viewpoint and perspective in the rendered scene. A bidirectional feedback module (116) is integrated into the device and connected to the semantic interpretation unit. It interprets corrective commands, gestures, or supplementary comments from the user to refine or modify specific scene elements without interrupting the real-time visualization. The system continuously updates the virtual scene with the flow of comments, thus ensuring temporal synchronization between speech input and rendered output within a defined latency threshold. This enables the natural, dialogic creation of complex three-dimensional virtual environments.

[0045] In one embodiment, the scene graph generation module (108) includes a context coherence verification subunit configured to perform referential disambiguation between multiple entities described in successive narrative segments by evaluating spatial continuity, temporal proximity, and pronoun resolution. This ensures that each node within the scene graph remains semantically consistent with previous context descriptions, thereby preventing redundant or contradictory object instantiations during continuous scene development.

[0046] In one embodiment, the multimodal image-speech model processor (110) uses a dual-encoder architecture consisting of a speech encoder and an image embedding encoder, which were jointly trained on paired datasets of text descriptions and corresponding 3D object representations. The system identifies the most semantically relevant asset vectors by performing a cosine similarity mapping between the encoded narrative features and pre-indexed object embeddings. Subsequently, an adaptive geometric adjustment routine scales, rotates, and positions the selected asset according to the spatial instructions extracted from the narrative.

[0047] In one embodiment, the scene assembly and rendering controller (112) uses a progressive path-tracking renderer that dynamically adjusts the sampling rates to the importance of visual areas within the user's field of view, as detected by the integrated eye-tracking sensors of the head-mounted device, so that computing resources are concentrated on areas of high prominence in the scene, while peripheral areas are rendered with reduced accuracy, thereby achieving real-time frame rate stability under conditions of high visual complexity.

[0048] In one embodiment, a latency optimization controller is configured to maintain sub-250-millisecond synchronization between spoken text and corresponding rendered visual changes. This is achieved through an asynchronous, event-driven architecture that includes a scheduler for prioritizing linguistic tokens, which suspends rendering operations for objects last mentioned in speech, and a parallel GPU inference pipeline that concurrently processes multimodal embeddings to prevent input queue buildup during fast or overlapping narrative sequences.

[0049] In one embodiment, the bidirectional feedback module (116) includes a semantic feedback interpreter that categorizes user correction inputs—verbal or gestural—into one or more categories, such as spatial adjustment, modification of visual attributes, object deletion, or insertion. The interpreter updates the underlying scene graph according to the detected correction intent and triggers a local scene redraw instead of a complete frame regeneration. This preserves temporal continuity and computational efficiency during user-driven refinement.

[0050] In one embodiment, a consistency validation submodule is integrated into the scene graph generation module to automatically detect physical or logical inconsistencies such as object penetrations, violations of spatial overlaps, or physically implausible placements (e.g., floating objects, incorrect gravity orientation) by evaluating geometric relationships between node attributes using a spatial physics model. Upon detection, the system generates a correction suggestion or prompts the user for clarification via audio to restore the realism of the environment.

[0051] In one embodiment, the semantic interpretation processing unit (106) and the multimodal image-language model are pre-trained on a composite dataset comprising general object-label pairs and domain-specific datasets such as urban structures and indoor crime scenes. The system dynamically adapts its generative behavior by applying a context domain weighting matrix that prioritizes vocabulary, object types, and relationship structures relevant to the user's operational domain, thus ensuring scenario-specific realism and interpretive precision.

[0052] In an embodiment further comprising a persistent scene archiving system configured to store each generated scene instance in a hierarchical data structure containing a combination of textual narrative logs, intermediate scene graphs, and final 3D geometry metadata, wherein the stored data is versioned to enable the reconstruction of scene evolution over time and to facilitate the review, audit, or reproduction of narrative visualization sequences for legal or educational documentation purposes.

[0053] In one embodiment, the head-mounted virtual reality visualization device (114) is network-synchronized with multiple user nodes via a distributed server infrastructure, enabling shared scene viewing, annotation, and modification. Each participant's verbal input is processed via an independent speech-to-text pipeline but aggregated by a multi-agent context synchronization manager that harmonizes concurrent updates by resolving timing conflicts and maintaining a globally consistent, shared scene state.

[0054] The detailed description of the invention explains the underlying architecture, functional workflow, and technical processes of the real-time virtual reality scene generator, which creates immersive three-dimensional environments using multimodal artificial intelligence and natural speech output (as defined in the claims above). The invention presents a fully integrated real-time AI pipeline that interprets natural speech output and dynamically constructs immersive three-dimensional environments within a virtual reality framework. The invention operates with a sequence of interconnected modules that include audio recording, speech processing, multimodal closure, 3D asset generation, spatial arrangement, and rendering. The entire process occurs in near real time, thus enabling smooth and dialogue-oriented scene creation that directly maps human speech onto visual environments.

[0055] The system is based on a speech capture module with an array of directional microphones optimized for spatial audio recording and noise reduction. The input signal is digitized using a high-resolution analog-to-digital converter and then spectrally filtered and time-normalized to ensure consistent signal quality. After preprocessing, the speech input is transmitted to a speech recognition module (STL), which executes a continuous, neural transformer-based automatic speech recognition model. This model is trained on extensive datasets of speech with varying accents and contexts to achieve high transcription accuracy. The architecture incorporates attention mechanisms that preserve long-term dependencies within the narrative.This allows the system to maintain context even when the user provides continuous and changing scene descriptions. The transcribed text is time-stamped and forwarded to the next processing stage without any perceptible delay.

[0056] The transcribed narrative is processed by a semantic interpretation unit, which forms the linguistic core of the invention. This unit utilizes a finely tuned, comprehensive language model developed to extract structural meaning, spatial references, and object relationships from natural language sentences.

[0057] The unit decomposes the narrative into syntactic and semantic components, identifying nouns as entities, verbs as actions, and prepositions as indicators of spatial relationships. Each linguistic unit is represented as a token embedding in a high-dimensional semantic vector space. Using dependency analysis and coreference resolution, the unit links pronouns or repeated mentions to previously identified entities, thus ensuring the preservation of continuity and context. The model employs specialized spatial inference layers that derive relative positions and orientations such as "to the left of," "behind," or "near the door," thereby translating linguistic cues into quantifiable spatial data.

[0058] The interpreted data is transferred to the scene graph generation module, which acts as an interface between language and image. A scene graph is a structured, hierarchical data model consisting of nodes representing objects and edges representing their spatial or relational attributes. The graph is initialized with the global coordinate system as the root node. Subsequent entities are positioned relative to each other based on linguistic constraints arising from the narrative. For example, in a narrative such as "Place a table in the middle of the room and a chair next to it," the table node would be anchored at the origin of the coordinate system, while the chair node would be positioned at a calculated offset based on the semantic interpretation of "next to."The scene graph is continuously updated as the narrative develops to ensure that new entities are added or changed in accordance with the progression of time.

[0059] The structured graph is then fed to a multimodal image processing and speech modeling processor, which represents one of the most technically advanced components of the invention. This module uses a dual-encoder neural network architecture: one encoder processes the linguistic embeddings from the scene graph, the other the embeddings from a large corpus of 3D object representations. The two encoders operate in a shared latent space, thus enabling the direct comparison of semantic and visual features. When a new object node is defined in the scene graph, the linguistic embedding of this object is compared to the database of visual embeddings using cosine similarity. The object with the highest similarity value is retrieved as the most contextually appropriate 3D model.The processor then applies a geometric adjustment routine to scale, rotate, and orient the retrieved object according to the spatial specifications of the description. If the described object does not exist in the library, the system synthesizes a new object approximation using a generative diffusion model trained to generate mesh structures from latent visual embeddings.

[0060] The scene builder and renderer receives the aligned object data from the multimodal processor and uses a physically based rendering pipeline to create a coherent 3D scene. The rendering engine is GPU-accelerated and uses ray tracing or path tracing to calculate realistic lighting, shadows, and reflections. To ensure real-time capability, a progressive rendering method is employed that dynamically adjusts the sampling rate. The headset integrated into the system has gaze sensors that identify visually relevant areas. These areas are rendered at a higher resolution, while peripheral areas are rendered at a lower resolution to ensure efficient use of computing resources. The rendering engine is designed to operate within less than 250 milliseconds between receiving voice input and updating the scene accordingly.This creates the illusion of an immediate visual response to speech.

[0061] The feedback and correction mechanism integrated into the VR headset ensures continuous coupling between user perception and scene generation. This subsystem includes a semantic feedback interpreter that recognizes correction commands via speech or gestures. For example, if the user gives a verbal correction like "Move the chair closer to the table" or a gesture to reposition it, the system identifies the corresponding object in the scene graph and applies incremental transformations instead of re-rendering the entire scene. This local update mechanism guarantees an uninterrupted immersive experience while simultaneously integrating user-driven optimizations in real time. Additionally, the system implements a reinforcement learning agent that monitors user corrections across multiple sessions to optimize its interpretation and rendering behavior.Over time, the model learns user preferences for object placement, spatial scaling, and lighting conditions, leading to steadily improved prediction accuracy.

[0062] A consistency validation submodule checks the logical and geometric integrity of each generated scene. This module applies physics-based constraints such as collision detection, gravity consistency, and object boundary intersection checks to prevent physically impossible configurations like floating or intersecting objects. If an inconsistency is detected, the system automatically generates a suggested correction or prompts the user for clarification via audio or haptic feedback. The validation system also verifies spatial relationships to ensure that the visual representation matches the intended narrative.

[0063] The technical pipeline operates within a distributed computing framework to optimize latency and performance. The hybrid cloud-edge architecture enables computationally intensive tasks such as multimodal embedding inference and generative model synthesis to be executed on cloud-based AI accelerators, while resource-efficient rendering and event processing occur locally on the edge device integrated into the VR headset. The system continuously monitors network bandwidth and dynamically redistributes the workload to ensure optimal responsiveness.

[0064] An integral part of the system is the temporal scene development tracker, which logs speaker segments and associated scene updates as time-stamped events. This allows for the reconstruction of the entire scene creation session in playback mode and enables analysts or instructors to trace the chronological evolution of the environment. The stored data, including speaker transcripts, scene cutscenes, and metadata of the final 3D geometry, are versioned using blockchain anchoring techniques to ensure the traceability and immutability of the generated content.

[0065] For multi-user collaboration, the invention utilizes a context synchronization manager that harmonizes the simultaneous input of multiple users interacting with the same scene from different locations. Each participant's language is processed independently by their local ASR and semantic modules, and updates are merged into a common global scene state using temporal conflict resolution methods. This architecture enables multiple users—for example, investigators or trainees—to jointly view, annotate, or edit the same virtual scene in real time.

[0066] From a technical perspective, the entire system functions as an adaptive, continuous learning pipeline. The AI ​​models within the system are updated through reinforcement learning and supervised fine-tuning based on user interactions and corrections. Each correction, adjustment, or validation feedback contributes to a cumulative dataset that refines future model behavior. This self-optimization ensures that the invention not only enables scene-based, real-time narrative generation but also improves its precision and contextual intelligence with increasing use.

[0067] The invention essentially provides a comprehensive computing framework that integrates linguistic interpretation, multimodal reasoning, and real-time visual synthesis within an interactive virtual reality. The synergy of speech recognition, semantic understanding, scene graph construction, multimodal information retrieval, and GPU-accelerated rendering results in a system that transforms narratives into immersive visualizations with unprecedented accuracy and responsiveness. This detailed technical integration constitutes the core of the invention's technical novelty and establishes it as a significant advancement in the field of AI-powered virtual reality systems.

[0068] The proposed system comprises several hardware and software subsystems that work together seamlessly to enable real-time, narrative-driven VR scene construction. System architecture

[0069] The system comprises a central processing unit (CPU) and a graphics processing unit (GPU) configured for parallel data processing and rendering. A microphone array serves as the primary input device for capturing spoken language. The audio signals are processed by a speech recognition module, which transcribes the speech into structured text using transformer-based automatic speech recognition techniques. The transcribed text is then fed to a semantic analysis engine that uses large language models fine-tuned for extracting spatial relationships and interpreting object context.

[0070] The output of this engine is a structured scene graph, consisting of nodes representing entities (objects, people, structures) and edges representing relationships (positions, orientations, interactions). This graph serves as input for an image processing speech synthesis module, which maps semantic descriptions to corresponding 3D objects from a predefined or dynamically generated library. The scene assembly unit creates the spatial arrangement of these objects in a virtual coordinate system.

[0071] A VR rendering module, powered by a high-performance GPU, generates immersive images. This module communicates with a head-mounted display (HMD) or VR headset, enabling real-time visualization of the evolving scene. Users can interact with the scene via voice output or gesture control. Movements are captured either by sensors integrated into the VR headset or by external motion-tracking cameras. Device structure

[0072] The physical embodiment of the invention comprises a portable, immersive visualization device with the following structural design: • A head-mounted display unit with two high-resolution micro-OLED panels and inward-facing motion-sensing cameras for detecting head position and gestures. • A wireless communication interface that enables synchronization with a computing base station on which the AI ​​and rendering modules are run. • A microphone array near the visor for real-time recording of voices. • A tactile feedback interface for user confirmation or error correction. • An embedded processing module for scene caching on the device and minimizing latency.

[0073] The device can optionally be connected to an external AI processing unit that has specialized hardware accelerators (e.g., tensor cores) for performing transformer-based multimodal calculations, thus ensuring low-latency rendering performance. Technical workflow 1. Voice recording and transcription: The microphone captures the spoken language, which is then processed using end-to-end speech recognition models to generate a text transcription. 2. Semantic analysis and object extraction: The language model identifies entities, actions, and spatial relationships, thus generating a structured scene representation. 3. Scene graph generation: The extracted information is encoded as a directed graph that represents hierarchical relationships between entities. 4. 3D scene synthesis: The image processing model generates a 3D layout and selects suitable meshes, textures, and lighting conditions that correspond to the described scene. 5. Rendering and updating: The VR engine renders the synthesized scene, and any change in the narrative triggers incremental updates without reinitialization. 6. User feedback loop: The observer provides corrective input ("Drive the car to the left", "Add a street lamp"), which is interpreted by the system and applied immediately.

[0074] The entire system operates with a latency in the sub-second range, thus ensuring smooth visualization that is synchronized with the narrative pace. Technical qualification

[0075] The invention can be implemented using hardware such as NVIDIA RTX GPUs, neural network accelerators, and compatible VR headsets like the MetaQuest or HTC Vive Pro. The software stack can utilize frameworks such as PyTorch for AI inference, Unity or Unreal Engine for rendering, and gRPC middleware for module communication. Power management and memory management are optimized for continuous operation in real-time research or educational contexts. TECHNICAL PROGRESS AND IMPACT

[0076] The proposed invention achieves several technical advances compared to existing technologies. It eliminates the dependence on manual modeling interfaces and enables autonomous scene generation based solely on voice input. It integrates extensive speech and image-speech models into a continuous feedback loop with VR rendering, thus ensuring context consistency and dynamic adaptability. The invention significantly reduces the latency between description and visualization, thereby enabling real-time interactivity.

[0077] The drawing and the preceding description illustrate embodiments. Those skilled in the art will recognize that one or more of the described elements can be combined to form a single functional element. Alternatively, certain elements can be divided into several functional elements. Elements of one embodiment can be added to another. For example, the process flows described here can be modified and are not limited to the manner described herein. Furthermore, the actions of a flowchart need not be performed in the sequence shown; nor do all actions necessarily need to be carried out. Actions that do not depend on other actions can be performed in parallel with the other actions. The scope of protection of the embodiments is in no way limited by these specific examples. Numerous variations, whether explicitly stated in the description or not, such as...Differences in structure, dimensions, and materials are possible. The scope of protection of the embodiments is at least as comprehensive as described by the following claims.

[0078] The advantages, other benefits, and problem solutions have been described above with reference to specific embodiments. However, the advantages, benefits, problem solutions, and any components that can effect or enhance an advantage, benefit, or solution are not to be construed as critical, necessary, or essential features or components of the claims. REFERENCES 100 A real-time virtual reality scene generator from natural language description using multimodal artificial intelligence. 102 Voice recording module 104 Speech-to-text processing unit 106 Semantic Interpretation Processing Unit 108 Scene Graph Generation Module 110 Multimodal Image Speech Model Processor 112 Scene Assembly and Rendering Controllers 114 Head-mounted virtual reality visualization device 116 Bidirectional feedback module

Claims

[1] A real-time system for the multimodal generation of virtual reality scenes based on artificial intelligence for the creation of immersive three-dimensional environments from natural language narratives, consisting of: a speech capture module configured to continuously record a user's spoken narrative via one or more directional microphones, preprocesses the captured signal by noise reduction and temporal alignment, and outputs a digital speech stream; A speech-to-text processing unit that is operationally coupled to the speech capture module and configured for real-time speech recognition using a continuous neural transformer model. The unit is trained to transcribe natural language utterances into structured text data while maintaining contextual continuity throughout the evolving narrative. a semantic interpretation processing unit that is communicatively linked to the speech recognition unit and configured to perform natural language understanding techniques to extract contextual entities, spatial references, temporal relationships, and object attributes from the transcribed narrative; the engine includes a large language model that is fine-tuned for spatial reasoning tasks; a scene graph generation module configured to transform the interpreted semantic data into a structured, hierarchical representation that defines nodes for identified entities and edges for corresponding relationships, with each node associated with metadata describing geometry, position, orientation, texture, and linking attributes between objects; a multimodal image-language model processor coupled with the scene graph generation module, wherein the processor is configured to retrieve, adapt, or synthesize appropriate three-dimensional elements from a pre-trained visual-lexical embedding space and align these elements with their semantic and spatial definitions derived from the scene graph; a scene assembly and rendering controller configured to create a cohesive virtual scene from the aligned assets, perform real-time rendering using a GPU-accelerated ray tracing pipeline, and produce a stereoscopic visual output that corresponds to the evolving narrative; A head-mounted virtual reality visualization device connected to the rendering engine and configured to display the generated immersive environment to the user in real time. The device features motion sensors and inside-out tracking cameras to detect head and body movements, dynamically updating viewing angles and perspective within the rendered scene; and a bidirectional feedback module integrated into the head-mounted device and connected to the semantic interpretation processing unit; the module is configured to interpret corrective commands, gestures, or supplementary comments from the user to refine or modify specific scene elements without interrupting the real-time visualization; The system continuously updates the virtual scene as the narrative develops, ensuring temporal synchronization between speech input and rendered output below a defined latency threshold, thus enabling a natural, dialogic construction of complex three-dimensional virtual environments. [2] System according to claim 1, wherein the scene graph generation module comprises a subunit for checking context coherence, configured to perform referential disambiguation between multiple entities described in successive narrative segments by evaluating spatial continuity, temporal proximity and pronoun resolution, and ensuring that each node within the scene graph remains semantically consistent with previous context descriptions, thereby preventing redundant or contradictory object instantiations during continuous scene development. [3] System according to claim 1, wherein the multimodal image-speech model processor uses a dual-encoder architecture comprising a speech encoder and an image embedding encoder jointly trained on paired datasets of text descriptions and corresponding 3D object representations, wherein the system retrieves the most semantically relevant asset vectors by means of a cosine similarity mapping between the encoded narrative features and pre-indexed object embeddings, followed by an adaptive geometric adjustment routine that scales, rotates, and positions the selected asset according to the spatial instructions extracted from the narrative. [4] System according to claim 1, wherein the scene assembly and rendering controller uses a progressive path-tracking renderer that dynamically adjusts the sampling rates to the importance of visual areas within the user's field of view, as detected by the integrated eye-tracking sensors of the head-mounted device, so that computing resources are concentrated on areas of high relevance to the scene, while peripheral areas are rendered with reduced accuracy. [5] System according to claim 1, wherein a latency optimization controller is configured to maintain a synchronization of less than 250 milliseconds between spoken text and corresponding rendered visual changes, achieved by an asynchronous event-driven architecture comprising a linguistic token prioritization scheduler that suspends rendering operations for objects last mentioned in the speech, and a parallel GPU inference pipeline that processes multimodal embeddings concurrently to prevent backlogs in the input queue during fast or overlapping narrative sequences. [6] System according to claim 1, wherein the bidirectional feedback module comprises a semantic feedback interpreter that categorizes user correction inputs - verbal or gestural - into one or more categories such as spatial adjustment, change of visual attributes, object deletion or insertion, and wherein the interpreter updates the underlying scene graph according to the detected correction intent and triggers a localized scene redraw instead of a complete frame regeneration. [7] System according to claim 1, wherein a consistency validation submodule is integrated into the scene graph generation module to automatically detect physical or logical inconsistencies such as object penetrations, violations of spatial overlaps or physically implausible placements (e.g. floating objects, incorrect gravity orientation) by evaluating geometric relationships between node attributes using a spatial physics model, and the system, after detection, generates a correction suggestion or prompts the user for clarification via audio query in order to restore the realism of the environment. [8] System according to claim 1, wherein the semantic interpretation processing unit and the multimodal image-language model are pre-trained on a composite dataset comprising general object-label pairs and domain-specific datasets such as urban structures and crime scenes in interiors, and wherein the system dynamically adapts its generative behavior by applying a context domain weighting matrix that prioritizes vocabulary, object types and relationship structures relevant to the user's operative domain, thereby ensuring scenario-specific realism and interpretative precision. [9] The system according to claim 1 further comprises a persistent scene archiving subsystem configured to store each generated scene instance in a hierarchical data structure containing a combination of textual narrative logs, intermediate scene graphs and final 3D geometry metadata, wherein the stored data are versioned to enable the reconstruction of scene evolution over time and to facilitate the review, audit or reproduction of narrative visualization sequences for legal or educational documentation purposes. [10] System according to claim 1, wherein the head-mounted virtual reality visualization device is network-synchronized via a distributed server infrastructure with multiple user nodes, enabling shared scene viewing, annotation and modification, and wherein the verbal input of each participant is processed by an independent speech-to-text pipeline but is aggregated by a multi-agent context synchronization manager that harmonizes concurrent updates by resolving temporal conflicts and maintaining a globally consistent shared scene state.

Images