Scalable systems and methods for creating interactive, dynamic, and clinically realistic medical simulations using artificial intelligence-based assistance

Abstract

Systems and methods are described for creating and deploying immersive, interactive, and dynamic medical simulation training experiences within augmented reality (AR) and virtual reality (VR) environments. Embodiments include a creation framework that decomposes complex clinical scenarios into modular components, allowing users to construct simulations using an intuitive, block-based interface without requiring programming expertise. These components capture the medical, physiological, and psychological elements of the scenario, enabling the creation of detailed, multidimensional simulations. An integrated AI engine further enhances the system by leveraging a large language model (LLM) to automate and assist in the scenario creation process. The AI engine interprets high-level natural language inputs, predicting and generating detailed simulation instructions, thereby accelerating the development and deployment of custom training scenarios. Once created, the scenarios are processed by a scenario engine that controls various elements of the simulation, including virtual characters, environments, tools, physiological responses, and assessable actions within AR / VR settings.

Description

SCALABLE SYSTEMS AND METHODS FOR CREATING INTERACTIVE, DYNAMIC, AND CLINICALLY REALISTIC MEDICAL SIMULATIONS USING ARTIFICIAL INTELLIGENCE-BASED ASSISTANCETECHNICAL FIELD[1] The present disclosure relates to artificial intelligence (Al) in augmented and virtual reality applications. In particular, the present disclosure relates to methods and systems for fusing interaction tracking in virtual and augmented reality environments with an Al-based simulation engine.BACKGROUND[2] Augmented reality (AR) and virtual reality (VR) both encompass interactive technologies that create immersive environments but differ in how they blend virtual and real- world elements. Augmented reality overlays virtual content onto the real world, enhancing the user's perception of their surroundings. It typically involves using a device, such as a smartphoneor AR glasses, to display digital information, images, or three-dimensional models that are superimposed onto the user's view of the physical environment.[3] Virtual reality, on the other hand, transports the user to a completely simulated digital environment. By wearing a VR headset, users are immersed in a computer-generated world that may be highly realistic or fantastical. VR enables users to interact with and explore a virtual environment through specialized input devices such as controllers or hand-tracking systems. Both AR and VR are used in various applications including gaming, education, navigation, architecture, healthcare, and industrial training.[4] One area where AR and VR applications are gaining traction is in medical training applications. Creating realistic and useful medical simulation training sessions in AR and VR environments is challenging due to a high degree of complexity in creating clinically realistic simulations. Many different factors are involved in the design of a simulation of a medical scenario. For example, various characteristics of a patient, including demographic information and medical history, may impact the simulation in a variety of ways depending on the specific medical situation involved. Other factors that may impact the simulation may include the environment where the simulation occurs, actions participants take, and vocalizations uttered during the simulation. If such details are neglected or unable to be determined, then the perceived realism and immersion of a simulation may be negatively impacted.[5] The complexity of issues in training simulations serves as a barrier to potential case creators. Medical professionals and educators may have interesting medical scenarios that they would like to test and simulate. However, such individuals may lack the programming skills or knowledge to create immersive experiences in AR and VR environments. As a result, medical experts often purchase pre-prepared scenarios that are created by dedicated simulation programming staff. The use of these “canned” scenarios created by experts enables busy educators to save time but does not allow the educators to create scenarios tuned to a specific need. Rather the educator is forced to pick the best preset option available for a given context.[6] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.BRIEF DESCRIPTION OF THE DRAWINGS[7] The embodiments are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings. It should be noted that references to “an” or “one” embodiment in this disclosure are not necessarily to the same embodiment, and they mean at least one. In the drawings:[8] Figure 1 illustrates a system in accordance with one or more embodiments;[9] Figure 2 illustrates an example dataflow diagram in accordance with some embodiments;

[0010] Figure 3 shows a block diagram that illustrates a computer system in accordance with one or more embodiments.DETAILED DESCRIPTION

[0011] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in a different embodiment. In some examples, well-known structures and devices are described with reference to a block diagram form in order to avoid unnecessarily obscuring the present invention.

[0012] 1. GENERAL OVERVIEW

[0013] Systems and methods are described herein for creating and deploying fully immersive, interactive, and dynamic medical simulation training experiences within AR and VR environments. The systems and methods include interactive tools and interfaces for composing clinical elements, physiological models, digital assets, educational assets, and other components for rendering a simulation. Various embodiments described herein specifically relate to techniques for composing medical training simulations. However, in other embodiments, techniques described herein may be used to compose training simulations in other contexts, such as training for other types of professional trades.

[0014] In some embodiments, the system includes a creation framework that decomposes complex clinical scenarios into interlocking, modular components. The creation framework may provide an interface and set of tools through which users may independently specify and connect components to create complex scenarios. For example, the interface may allow a scenario to be built from a set of blocks in a user-friendly case creator application page (or set of pages). Theblocks may represent various interlocking components that capture the medical, physiological, and psychological spectrum of the scenario, allowing multidimensional VR / AR medical simulations to be specified without requiring the scenario to be programmed in and compiled by a base VR / AR engine.

[0015] In some embodiments, the creation framework includes an artificial intelligence (Al) engine that facilitates and / or automates various aspects of case creation. The Al engine may incorporate a large language model (LLM) that has learned the context and structure of a clinical simulation. For example, the Al engine may pull context and examples from other, previously created scenarios. Based on the learned patterns, the Al engine may impute, predict, and / or generate details of the scenario. The system may use the Al engine to process high-level requests submitted by scenario creators. For example, a scenario creator may submit a high-level request in a natural language to create a specific scenario even though several of the details of the scenario needed to create the AR / VR simulation file(s) are omitted. The Al engine may transform the high-level request into a detailed description by inferring details based on learned patterns from the clinical scenario training data. Thus, the system may generate clinical simulations for simulation creators using natural language prompts without requiring input of every detail of the scenario or complex programming, thereby reducing the amount of required system input and increasing the speed / efficiency with which new, custom AR / VR scenarios may be deployed within an AR / VR simulation environment.

[0016] In some embodiments, the system includes a scenario engine that runs clinical simulation scenarios generated through the creation framework. The creation framework may convert the set of interlocking components into a file that is interpretable by the scenario engine. The scenario engine may process and execute the interpretable instructions in the file to control various aspects of a clinical simulation within an AR / VR environment. For example, the scenario engine may determine how to generate and project virtual characters, environments, tools / equipment / loadouts, and / or other virtual objects. The instructions may further drive other aspects of the simulation including states / vitals / transitions, instrument findings, dialogue, animations, events, critical actions, and / or assessables.

[0017] One or more embodiments described in this Specification and / or recited in the claims may not be included in this General Overview section.

[0018] 2. AI-DRIVEN AR / VR CASE CREATION AND SIMULATION SYSTEM

[0019] Figure 1 illustrates a system 100 in accordance with one or more embodiments. As illustrated in Figure 1, system 100 includes AR / VR headset 102, input devices 104, AR / VR peripherals 106, computer system 108, creation framework 110, scenario simulation engine 122, sensors / trackers 130, and Al engine 148. In one or more embodiments, system 100 may include more or fewer components than the components illustrated in Figure 1.

[0020] AR / VR headset 102 is a device that is worn by a participant in a simulation training to immerse the participant in an augmented or virtual reality scene. An AR headset combines real-world views with virtual overlays and digital information. An AR headset allows users to see and interact with virtual content while maintaining awareness of their physical surroundings. The AR headset may further use the physical surroundings to render virtual overlays (e.g., such as a virtual avatar projection over a medical training manikin that is present in the physical space) and generate reactions based on interactions with real world objects and / or the virtual objects projected by the AR headset into the physical space. AR headsets may include a transparent or semi-transparent display, which may be a see-through lens or a screen that displays an image of the physical space (e.g., using a camera on the front of the headset). A VR headset provides a display that immerses the participant in a virtual environment, which may limit the participant’s vision / awareness of their physical surroundings. AR / VR headset 102 may be a dedicated device built specifically for AR / VR applications or may be a multi-function device such as a smartphone, which may optionally be strapped to a participant’s head using an accessory device.

[0021] Input devices 104 include devices that a participant may use to provide input to AR / VR headset. For example, AR / VR headset 102 may include handheld controllers or other input devices that allow a participant to start a new simulation, manipulate objects in the environment, etc. As another example, AR / VR headset 102 may include an input device representing a training tool, such as a real-world medical device or object representing a medical device (e.g., bag and mask, stethoscope, scalpel, syringe, IV bag, x-ray machine, MRI machine, etc.). Controllers may include buttons, triggers, touchpads, embedded markers, accelerometers, and / or other motion-sensing / position tracking capabilities.

[0022] Sensors / trackers 130 track a participant’s movements and translate them into the augmented / virtual reality environment. Sensors / trackers 130 may be embedded in the headset, placed on the participant, and / or positioned within the surrounding physical space. Sensors maycapture a participant’s position, orientation, and movements, among other data. In some embodiments, sensors / trackers include sensors / trackers to perform:• Head tracking 132: AR / VR headset 102 may incorporate sensors, such as accelerometers, gyroscopes, and magnetometers, to track the user’s head movements and orientation, which allows the virtual environment to adjust accordingly to provide an immersive experience.• Eye tracking 134: AR / VR headset 102 may track where the user is looking. Eye tracking may be computed / inferred from the head position captured via the head tracking sensors and / or via additional sensors, such as a pupillometry sensor or an infrared light that creates a reflection used to measure the distance between the pupil center and the reflection of the cornea.• Positional tracking 136: System 100 may include external sensors and / or cameras to track the user’s position in a physical space. The sensors may capture the movement of markers or built-in sensors on the headset to provide precise tracking of the user’s location as the user moves within the augmented or virtual reality environment.• Hand tracking 138: System 100 may include cameras, depth sensors, and / or infrared sensors to track hand movements and gestures without the need for handheld controllers. The sensors may capture hand movements and / or finger articulations.• Object tracking 140: System 100 may include sensors and cameras to track objects other than the user in a physical space. Similar to hand tracking 138, system 100 may use cameras, depth sensors and / or infrared sensors to track object movements. Additionally or alternatively, electromagnetic markers and / or built in sensors may be affixed / included in the object to provide data about the object’s position and orientation within the augmented or virtual reality environment.• Controller tracking 142: Handheld controllers may include built-in sensors, such as accelerometers and gyroscopes, which capture the controller’s orientation and movements. The controllers may include buttonsjoysticks, and / or other input devices that allow the user to navigate through menus and interfaces projected / displayed on VR headset.• Body tracking 144: Advanced AR and VR training systems may incorporate full body tracking, which involves capturing movements of the participant’s entire body. Tocapture full-body movements, system 100 may include motion-capture cameras, depth sensors, and / or wearable devices (e.g., full body suits with embedded sensors), and• Environmental tracking 146: VR training systems may utilize sensors and / or cameras placed in the physical environment to track the user’s movements and enforce safety protocols. For example, sensors may help prevent collisions with real-world objects and create a boundary within which the user may freely move.

[0023] AR / VR peripherals 106 includes accessories and / or other devices that are separate / external from the headset that may enhance the training experience. For example, AR / VR peripherals 106 may include haptic feedback devices, which provide tactile sensations to simulate a real-world environment, or motion platforms that simulate physical movement to enhance immersion. AR / VR peripherals 106 may be paired wirelessly or connected via a physical wire with the AR / VR headset 102 and / or computer system 108. Additionally or alternatively, AR / VR peripherals 106 may include electromagnetic markers, which may include a coil of wire that generates an electromagnetic field that may be tracked by sensors / trackers 130 (e.g., by triangulating the signals received from multiple sensors to determine a marker’s position and orientation in real-time).[24J Computer system 108 includes one or more hardware processors for running AR / VR software and rendering overlays / complex graphics in the augmented or virtual reality environment. Computer system 108 may process a participant’s inputs and generate the virtual world, including the reactions within the environment to the participant’s inputs. The computer may be embedded directly within AR / VR headset 102 or may be an external device that communicates wirelessly or via a tethered connection with AR / VR headset 102. Example computer architectures are discussed further below.[25J Creation framework 110 includes tools and interfaces through which creators may build clinical scenarios for simulation in AR / VR environments. In some embodiments, creation framework 110 includes interlocking components 112, block interface 114, dependency checks 116, prompt generator 118, and configuration file generator 120. Creation framework 110 may be implemented as a service (e.g., a network / cloud / web service), standalone application, or suite of applications. Case creators (e.g., medical professionals, educators, etc.) may access the tools and interfaces of creation framework 110 (e.g., via a browser, mobile app, desktop application,or other client) to build new scenarios without requiring the case creator to program a game engine, write code in a programming language, or compile the code. Thus, users with little to no knowledge in the area of software development may quickly create complex, dynamic AR / VR simulations of medical operations and / or other clinical scenarios.

[0026] Modular components 112 represent different constructs for creating clinical scenarios. In some embodiments, modular components 112 include multiple categories of interlocking building blocks, which may be individually specified and combined with other blocks. Users may composite the building blocks into a fully or partially complete scenario.

[0027] Block interface 114 is a graphical user interface that allows users to visually compose scenarios using modular components 112 as building blocks. Block interface 114 may allow a user to add new blocks, connect / interlock blocks, select individual blocks to defme / modify configure, and / or otherwise visually create a new scenario. Block interface 114 may include a drag-and-drop interface through which users may drag individual blocks and drop the blocks into a clinical scenario. Additionally or alternatively, block interface 114 may allow the user to assemble the blocks to create sequences of events or actions within the simulated scenario.

[0028] Consistency engine 116 may ensure and / or verify the consistency of a clinical scenario. For example, consistency engine 116 may ensure that the details captured by modular components 112 are consistent with the general structure of a scenario and / or that the details are clinically consistent internally. Consistency engine 116 may update block interface 114 to prevent inconsistencies, such as disabling certain blocks or available block configurations that are inconsistent with the current state of a scenario as it is being built by the creator.

[0029] Prompt generator 118 interfaces with Al engine 148 to assist with the creation of clinical scenarios. Prompt generator 118 may generate input prompts based on the current state of a scenario that is being built and / or high-level queries submitted by the user. The prompt may include the details of the scenario that have been specified, which may be used by Al engine 148 to infer missing details at varying levels of abstraction. The interface Al engine 148 may utilize data internal or external sources to interpolate the missing details to complete the scenario for the user. The auto-completed scenario may be presented to the user through block interface 114 as a collection of interlocking blocks, which may be editable by the user to fine-tune the custom scenario in the event that the user is not satisfied with an Al-inf erred detail.

[0030] Configuration file generator 120 receives a set of interlocking blocks as input for a completed scenario and outputs a configuration file that is interpretable by scenario simulation engine 122.

[0031] Scenario simulation engine 122 is an AR or VR application, which may be executed by computer system 108 to simulate the custom-defined medical scenarios. In some embodiments, scenario simulation engine 122 receives configuration file 124 for a scenario built by a case creator through creation framework 110. Configuration file 124 may capture various customizable scenario details that control various aspects associated with simulating a medical operation or other clinical scenario.

[0032] In some embodiments, scenario simulation engine 122 includes interpreter 126, which processes the configuration file 124 and creates machine code to render the scenario within the AR or VR environment. For example, interpreter 126 may generate machine code that cause AR / VR headset 102 to project virtual avatars and objects into the AR or VR environment with which participants in the simulation (e.g., one or more users wearing AR / VR headsets) may interact. Configuration file 124 may control aspects of a simulation including the appearance, behavior, dialogue, sounds, location, and positioning of virtual avatars and objects. Configuration file 124 may further control what events transpire during the scenario (e.g., a patient going into shock or exhibiting a symptom of an illness), the timing of the event, responses to participant interactions (e.g., how a participant action affects the virtual patient), assessments (e.g., how well did the participant perform in the given scenario based on the interactions with the virtual patient), environment, and other conditions to simulate a realistic clinical situation.

[0033] Simulation / application data 128 includes software code and other data, which may be executed by computer system 108 to provide the augment or virtual reality environment and interactive experiences. In some embodiments, interpreter 126 combines details about the scenario captured by configuration file 124 with simulation / application data 128. For example, details about a patient’s physiological state (e.g., weight, height, blood type, blood pressure, etc.) may be integrated into code that controls how a virtual avatar looks and acts within the simulation.

[0034] Artificial intelligence (Al) engine 148 is a software component or system that powers machine understanding and learning from data, which is used to generate and / or complete clinical scenarios to simulate within the augmented or virtual environment. Al engine 148 mayserve as the underlying infrastructure that is responsible for processing and analyzing user queries (e.g., high-level requests to create a scenario and / or requests to complete a partially built scenario), making decisions about how to create / complete the scenario, and autonomously creating / completing a file (or set of files) that is interpretable by scenario simulation engine 122 to render the scenario within an AR or VR environment. Al engine 148 may incorporate machine learning algorithms (e.g., contextual learning algorithm 150) and models (e.g., Al model 152) to make decisions and generate intelligent responses that are not hard coded into the system, which allows the system to adapt and evolve over time without any programming changes. Al engine 112 may include one or more of the following components:• Data Processing: Al engine 148 may process and ingest large volumes of data, which may include contextual data extracted from previously built clinical scenarios and / or external resources relevant to creating realistic clinical simulations (e.g., online medical journals, books, a sources of trusted information). Al engine 112 may apply data pre-processing techniques to clean, transform, and organize the data for analysis.• Machine Learning: Al engine 148 may use one or more machine learning algorithms (e.g., contextual learning algorithm 150) and models (e.g., Al model 152) to learn from data, identify patterns, and make predictions or decisions based on the learned information. A machine learning algorithm that may be iterated to learn a target model f that best maps a set of input variables to an output variable. The machine learning algorithm may iterate over a training dataset comprising multiple examples of simulation components that may or may not be represented in a domain-specific language or as free text, which may be labelled according to the corresponding target output variables for the associated task. The machine learning algorithm may use supervised learning, unsupervised learning, and / or reinforcement learning to train a ML model based on the training dataset. Various types of algorithms may be used, such as linear regression, logistic regression, linear discriminant analysis, classification and regression trees, naive Bayes, k-nearest neighbors, learning vector quantization, support vector machine, bagging and random forest, boosting, backpropagation, and / or clustering.• Deep Learning: Deep learning is a subset of machine learning that involves learning using neural networks with multiple layers. Deep learning algorithms may be used fortasks like image and speech recognition, natural language processing, and complex pattern recognition.• Reasoning and Inference: Al engine 148 may employ logical reasoning and inference mechanisms to derive insights, draw conclusions, and make logical deductions based on the available data and knowledge.• Knowledge Representation: Al engine 148 may include methods for representing learned knowledge in a structured format. The structured format may optimize the system’s capacity to organize and utilize domain-specific knowledge to perform decision-making and problem-solving tasks.• Natural Language Processing (NLP): NLP techniques may be used by Al engine 148 to understand and process human language, including speech recognition, language understanding, and language generation capabilities.• Translation layer: Al engine 148 may include a translation layer to integrate the Al functions with a simulation application. For example, the translation layer may translate the output of an ML model (e.g., a deep learning / neural network model prediction) to code / instructions that cause changes in the augmented / virtual reality environment.

[0035] In some embodiments, Al model 152 may include a large language model (LLM) that is configured to learn patterns from contextual data and apply the learned patterns to infer details for building the AR / VR scenario simulation file. One or many LLMs may provide natural language responses to inputs, which may include the details of the scenario. The predicted details may be mapped to component blocks and configurations thereof according to domainspecific languages or structured formats to interface with scenario simulation engine 122.

[0036] During training, LLMs may be exposed to large amounts of text data and learn to recognize patterns and relationships within language. A large language model may use deep learning and / or other machine learning techniques to process and generate text. The underlying architecture for a generative language model may vary depending on the particular implementation. Example architectures include for a large language model may include: a. Recurrent neural networks (RNNs): RNNs are a class of neural networks designed to handle sequential data, making them suitable for processing sequences of words in a sentence. RNNs maintain an internal state (hidden state) that is updated with each newinput word. The hidden state carries information from previous words and influences the predictions for the next word. This process is referred to as “unrolling” the RNN through time. During training, the RNN is fed sequences of words, and it learns to predict the next word in a sequence given the context of the previous words. One technique, referred to as "teacher forcing," occurs where the true next word is provided during training to help guide the learning process. The model is optimized to minimize the difference between its predictions and the actual next words in the training data. Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) RNNs may be used to capture long-range dependencies in text. b. Transformers-based models: Unlike RNNs, transformers process the entire sequence of words simultaneously, allowing these models to better capture long-range dependencies in the text. In the transformer model, the input text is split into word embeddings, which represent each word as a high-dimensional vector. These embeddings are then passed through multiple layers of self-attention and feedforward neural networks. The selfattention mechanism allows the model to weigh the importance of each word in the context of the entire sequence, enabling it to focus on relevant words for generating the next word. During training, the model is exposed to vast amounts of text data and learns to predict the likelihood of each word occurring given its context. This process involves adjusting the model's parameters (weights) to minimize the difference between its predictions and the actual words in the training data. The transformer model can be fine-tuned on specific tasks or used as a pre-trained language model for various downstream tasks.

[0037] In some embodiments, system 100 may further include components for synchronizing a simulation when there are multiple participants involved. For example, system 100 may include components that concurrently broadcast a medical training session such that different participant headsets concurrently project the same scene though from different physical locations as described in U.S. Patent No. 10,672,288, which is incorporated by reference herein.

[0038] 3. CREATING MEDICAL SIMULATIONS USING ARTIFICIAL INTELLIGENCEBASED ASSISTANCE

[0039] 3.1 COMPONENT BUILDING BLOCKS

[0040] In some embodiments, a case creation application presents users with various component blocks, which may be configured and assembled to build new clinical scenarios. Case creation framework 110 may include multiple categories of building blocks. A user may composite the building blocks into a complete scenario. Al engine 148 may assist the user to complete the details to enhance and speed up the build process, or the user may complete the process without any Al assistance.

[0041] In some embodiments, case creation framework 110 includes the following categories of building blocks:

[0042] 1. Characters

[0043] The virtual characters present in the simulation may be specified through one or more character components. Character building blocks may be presented in the form of virtual avatars comprising multiple subcomponents. Characters may represent patients or other non-player characters (such as family, bystanders, other medical personnel, etc.).

[0044] a. Character Sub-Specification

[0045] Character building blocks may be available either as fully specified building blocks for selection (i.e., select character “Rebecca Jackson” who comes with a complete backstory, physical features, body avatar, injury / scar pattern, clothes, etc.) or characters may be subspecified from character sub-blocks (i.e. specific body pattern, skin color, facial structure, clothing, backstory, injuries and dysmorphias, and other visual features, etc.). These sub-blocks may then composed into a full character block.

[0046] 2. Environments

[0047] The environment s) that simulations take place within can be specified. There may be one or more environments associated with any particular scenario, and they may be arranged temporally in any way. For example, a scenario may take place entirely within one environment, e.g., an emergency department room. Or, the scenario may take place over multiple environments, e.g., a patient first treated on the battlefield at the point of injury, then in a helicopter during evacuation, then in a hospital after arriving. The scenario may also take place in multiple environments simultaneously (e.g., patient A may be on the battlefield while patient B is in a helicopter flying away). Environments may be composed of many components, such as structures, walls, etc., objects (i.e. a chest of drawers, a burning Humvee), medical equipment (i.e. a vitals monitor, a ventilator, etc.), and other components. Environments may be fullypreplanned or may have configurable components that can be sub-specified. Environments may also have optional environmental effects (i.e. rain, sandstorm, etc.) or temporal effects (i.e. nighttime, daytime, etc.).

[0048] a. Environments Sub-Specification

[0049] Environments may have different types of blocks specified. For example, environments may have configurable sub-locations (i.e. the inside of a drawer, free space on the ground or wall, etc.) with blocks that specify additional contents at that location (i.e. an intubation kit inside a drawer, a vacuum pump attached to the wall, walls or a room, etc.). Environments may also have location coordinate block specified, which can then be referenced in other specifications (i.e., the door coordinates, bed coordinates, etc.), as well as floor coordinates that may be used for animations (i.e. an animation may reference a portable CT scanning team moving between multiple locations), or for player start locations.

[0050] b. Environmental and Temporal Effect Specifications

[0051] Environments may also have associated effects specified. These effects may include weather impacts (i.e. rain, snow, sand, etc.) or time-based impacts (i.e. night / darkness, daylight, etc.). These impacts may be constant during the use of the environment in a scenario or may be associated with timers or other events (specified below) or triggers.

[0052] 3. Tools, Equipment and Loadouts

[0053] Tools, medical equipment, and equipment loadouts available within the scenario may be specified. This may be a specification of exactly what is available in the scenario, or a specification of options that can be made available, with the final choice of which options to select made at runtime. Tools could include items such as a stethoscope, syringe, medications, chest tubes, etc. Equipment may include bulkier / more complex items such as a field ventilator, CO2 monitor, etc. Loadouts may be sets of equipment and tools made available to players. For example, a backpack full of field medical equipment, with sub-compartments containing tools of different types. Different loadouts may involve different tools being available, or different subcompartments, or different bags or other containers, or simply equipment and tools being available on the ground or in the environment.

[0054] a. Equipment Sub-Specification

[0055] Individual tools, pieces of equipment, and containers may be sub-specified or even newly fully specified. Such specifications may include aspects of the tools themselves (i.e. thediameter of the needle on a syringe, or the layout of compartments in a container, or the label on a medication bottle), and may include or impact the functionality of a tool (i.e., a block could specify that when a point or region (AKA collider) on the tool impacts a certain point or region on the patient’s body, an event (specified below) occurs, etc. It may specify that tools attach together, or that users may interact with a tool or piece of equipment (i.e. when the user touches the equipment in a certain location, such as on a virtual button, a certain change occurs, etc ).

[0056] 4. States, Vitals and Transitions

[0057] Scenario flow elements may be specified for a specific character or for the entire scenario. For each character, state, vital, and transition building blocks are composed. Each character may have multiple states created for a single scenario. State blocks can be subspecified with vitals and other physiological parameters (i.e. heart rate, blood pressure, oxygenation level, heart rhythms, etc.). At the scenario level, states may specify global elements including the environment and surrounding conditions, presence and / or location of specific tools and equipment sets, and learner objectives. Transition blocks may be used to connect states, to create paths through the dynamic scenario on a per-character or per-scenario basis. These transitions can then be executed based on event specifications related to those transitions.

[0058] 5. Instrument Findings

[0059] For each character, instrument finding blocks can be specified. Sub-blocks can be selected for each instrument desired. For example, if a stethoscope instrument is to be included, stethoscope sub-blocks could be selected for each stethoscope listening area (i.e. the “stridor” block could be selected for certain lung field areas for a stethoscope, a “pneumothorax” block could be selected for certain ultrasound areas, etc.). Findings may also be specified on a per- character basis not linked to a specific instrument and included as details in the patient record that may be accessible by the learner. For example, if a learner is required to interpret CT findings and blood test results during the course of the scenario, sub-blocks can specify the associated imaging and labs to be included in the patient record. Such finding blocks can also be mapped to states, enabling the findings to change as states change (i.e. a character may have pneumothorax areas in certain states, but not in others, i.e. if certain states become active when a decompression needle is used).

[0060] 6. Dialogue and Animations

[0061] For each character, dialogue can be specified. Dialogue can be specified in text format and / or in audio recording format, and could consist of words or other sounds (i.e. wheezing, screaming, etc.). Similarly, animations can be specified based on pre-set animation building blocks (i.e. a contraction building block can be employed for a patient in labor). Such blocks can also be linked to states to enable context-aware dialogue and animations.

[0062] 7. Events

[0063] Events can be specified for a scenario. Events describe some of the dynamic things that may occur during the execution of a scenario. This can include medically relevant events within the scenario or other runtime elements used by scenario engine 122 or consistency engine 116. Events include “triggers,” the precipitating condition that, when met, cause the event to run. Events also include “effects,” which describe what happens when the triggers are met. Triggers can be specified using Boolean logic, enabling the simultaneous presence of multiple triggers that can interact in any way. Triggers may refer to any occurrences within the course of a simulation, which could include the passage of a certain amount of time, actions by participants (i.e. using certain tools, giving certain medications, touching or moving to certain areas, etc.), inaction by participants (i.e. delaying certain actions, or making one choice instead of another), and other scenario occurrences including changes in characteristics associated with any virtual asset. Triggers can also refer to manual selection performed by a moderator or facilitator, (i.e., clicking a button within the moderator interface). Effects can involve anything within the scenario, such as actions by a non-player character or patient, changes to physiology, changes to objects within the scene, dialogue or other sounds, movement within the scene, changes in the environment or changes of environment, etc.

[0064] 8. Critical Actions and AssessablesCritical actions and assessables can be specified for any given scenario. These factors measure the performance of individual trainees and compare that performance to expectations. Critical actions can be associated with events, with the execution of a given event leading to the critical action being marked completed and may be given an arbitrary name. For example, a critical action marked “provided pain control” may be associated with an event with triggers related to a variety of appropriate pain control medication options, and with the result that the critical action is marked as completed. Critical actions may also be recorded manually by the facilitator; for example, an instructor may mark the critical action “speaks to parent” after evaluating the learnerhaving a conversation with the non-patient character within the scenario. Assessables are configurable point-based scores that can be assigned to an individual learner’s performance. Assessables can be configured to be Boolean, numerical, or continuous values, with an arbitrary name. Assessables can be configured to automatically score via events (similarly to critical actions) or can be configured to be manually scored by moderators during scenario execution.

[0065] 3.2 Al Assistance System

[0066] In some embodiments, an Al assistance system may automatically build and / or complete scenarios based on natural language prompts from a user. The Al assistance system may include multiple subcomponents. Example subcomponents may include a scenario reference and information retrieval system, a clinical reference and information retrieval system, and an Al generation and completion system.

[0067] In the Al assistance system, users who wish to create a new scenario provide information about that scenario. The system then finds appropriate references related to the scenario to be created, and then makes part (completion) or all (generation) of a scenario for the user. User input to the system may be as broad or as detailed as preferred by the user (within technical input size limits). For example, a user may simply tell the system that they wish to create a “myocardial infraction scenario for the resident level” or they may provide a very detailed specification (including components and block specifications as described above) of what the wish to complete. The Al can generate details that are consistent with the general structure of educational scenarios and clinically consistent internally. This allows for educators to generate their own content as needed, tailored to specific educational and institutional goals, rather than having to use pre-canned content.

[0068] Scenario References and Information Retrieval

[0069] The Al system may process the user input and turn it into a scenario search query, which may be used to query a database of scenarios or scenario components and retrieve entries that may be relevant to the desired scenario. To do this, the Al system may predict an embedding of the user input after optionally preprocessing it (i.e. by filtering terms, mapping to a lexicon or terminology set, etc.). The embedding may then be used to search the database of scenarios, optionally after preprocessing each scenario (i.e. via truncation, data extraction, terminology mapping, subsetting, etc.). The resulting scenarios or scenario components that are found maythen be included in the context for the request. These retrieved results may be used by the Al system in order to generate blocks or sub blocks that can be used in the case creation framework.

[0070] Clinical References and Information Retrieval

[0071] Similarly, the Al system may process the user input and turn it into a clinical reference search query, which it may use to query available clinical reference databases (i.e. potentially including encyclopedias, UpToDate-style reference libraries, textbooks, protocol and standard libraries, etc.). This may be done by embedding the user input after preprocessing and using the embedded input as a search query against the available databases. This query may optionally be further enhanced through processing and embedding of the discovered scenario references.

[0072] Al Generation and Completion

[0073] Finally, the Al model may predict and / or impute a full scenario specification based on the user input as well as the discovered references. The model may create the specification using a GPT or LLM style model, or other Al-based models. The model may be pre-trained or primed using previous scenario templates and the scenario specification language (which may be a domain-specific language, JSON, HTML, or other form of data). The model may make use of a partial specification already created by the user and complete it with additional information or may create the specification from scratch using the input and references. The model may perform generation or completion using a token prediction approach, a synthetic approach, or other Al approach.

[0074] 3.3 Case Creation Flow

[0075] Figure 2 illustrates an example dataflow diagram 200 for building and deploying clinical scenarios within AR / VR environments in accordance with some embodiments. One or more operations illustrated in Figure 2 may be modified, rearranged, or omitted all together. Accordingly, the particular sequence of operations illustrated in Figure 2 should not be construed as limiting the scope of one or more embodiments.

[0076] Al model 206 receives, as input, high-level request 202 from the case building application. A high-level request may include a natural language description of a scenario for which a user would like to create a new AR / VR simulation. For example, a high-level request may specify information about virtual patients / characters, tools, goal s / criti cal actions,assessables, environments, and / or other of the aforementioned components in a descriptive manner.

[0077] Additionally or alternatively, Al model 206 may receive modular components 204 from the case building application as input. For example, a set of component blocks (or a config file) for a partially built scenario may be fed as input to Al model 206. The blocks may be assembled and specified by a user creating a new scenario. Al model 206 may provide real-time suggestions to complete the scenario or auto-complete the details of the scenario on behalf of the case creator.

[0078] In some embodiments, Al model 206 may fetch scenario context data 208 to extrapolate patterns based on similar clinical scenarios. The context data may include the scenario and / or clinical reference information previously described, which may be used to train or tune the model. Al model 206 may be pre-trained or rely on generalized training to extrapolate from the example context data on the fly. As previously noted, Al model 206 may be an LLM, where a set of one or more prompts may be input based on the high-level request, scenario context data, and / or modular components. For example, a set of prompts may request the Al model 206 to complete one or more omitted details of the scenario based on which details the user building the scenario has not specified.

[0079] In some embodiments, Al model 206 preprocesses the inputs (operation 210). For example, the model may create word embeddings for a high-level request. Additionally or alternatively, a prompt engineering process may create a series of input prompts to an LLM model to be used by a prompt generator to extract the missing component blocks and configurations.

[0080] In some embodiments, Al model 206 generates one or more inferences based on the inputs to complete the scenario details (operation 212). For example, Al understanding model 206 may infer virtual avatar characteristics (operation 214), environmental attributes (operation 216), tool and equipment loadouts (operation 218), states, vitals and transitions (operation 220), instrument findings (operation 222), dialogue and animations (operation 224), events (operation 226), and / or critical actions / assessments (operation 228). The inferences may be generated by creating a feature vector using the inputs and / or fused data, which may be used to perform a forward pass through one or more trained neural networks to generate the predictions based on patterns learned from a training dataset. For instance, an LLM may feed the vectors through atransformer-based architecture that predicts a sequence of tokens representing the missing scenario details.

[0081] In some embodiments, the case creation framework generates or updates a set of modular components based on the output of Al model 206 (operation 230). For example, the case creation framework may map the missing details to specific component blocks, block configurations, and block assemblies (how the blocks connect together). The missing details may be merged with details, if any, of a scenario submitted by the user through the case creation framework (e.g., via high-level queries or block interface).

[0082] The case creation framework could then generate an interpretable file based on the autocompleted set of modular components (operation 232). For example, the case creation framework may encode the details in a domain-specific language that the interpreter is configured to process, such as JSON, XML, HTML, etc. Additionally or alternatively, the set of modular components may be presented to a user (e.g., through the block interface) for the user to review the details filled in by the Al assist system. The user may approve, reject, and / or edit any of the details through the interface. Changes may be fed back to Al model 206 and used to retrain and / or fine-tune the model (e.g., by updating weights and / or other parameters / hyperparameters through learning processes such as backpropagation) to improve the model for future predictions / auto-completion tasks.

[0083] Once the interpretable file(s) for the case scenario have been created, they may be downloaded by or otherwise deployed to the case simulation AR / VR application. The simulation engine may load the interpretable files and execute the instructions to simulate the scenario in the AR / VR environment (operation 234). In some embodiments, the details included in the interpretable file are merged with or otherwise injected into application code to run the simulation. Character attributes, tool buildouts, animations, dialogue, simulated clinical events, clinical objectives, assessment criteria, environments, and / or other details may be customized by the case creator through natural language prompts and / or visual interfaces (with optional Al assistance) without any complex coding, compiling, and debugging. Thus, the ability to create highly complex VR / AR simulations does not require highly specialized developers, lowering the cost and barrier to entry. Additionally, the speed of creating and deploying new scenarios may be greatly reduced, even in the case that the creator has extensive programming experience, as thedevelopment process is streamlined, and the Al assistance system minimizes the amount of user input required to build a newly conceived scenario.

[0084] 4. COMPUTER NETWORKS AND CLOUD NETWORKS

[0085] In one or more embodiments, a computer network provides connectivity among a set of nodes. The nodes may be local to and / or remote from each other. The nodes are connected by a set of links. Examples of links include a coaxial cable, an unshielded twisted cable, a copper cable, an optical fiber, and a virtual link.

[0086] A subset of nodes implements the computer network. Examples of such nodes include a switch, a router, a firewall, and a network address translator (NAT). Another subset of nodes uses the computer network. Such nodes (also referred to as “hosts”) may execute a client process and / or a server process. A client process makes a request for a computing service (such as, execution of a particular application, and / or storage of a particular amount of data). A server process responds by executing the requested service and / or returning corresponding data.

[0087] A computer network may be a physical network, including physical nodes connected by physical links. A physical node is any digital device. A physical node may be a functionspecific hardware device, such as a hardware switch, a hardware router, a hardware firewall, and a hardware NAT. Additionally or alternatively, a physical node may be a generic machine that is configured to execute various virtual machines and / or applications performing respective functions. A physical link is a physical medium connecting two or more physical nodes. Examples of links include a coaxial cable, an unshielded twisted cable, a copper cable, and an optical fiber.

[0088] A computer network may be an overlay network. An overlay network is a logical network implemented on top of another network (such as a physical network). Each node in an overlay network corresponds to a respective node in the underlying network. Hence, each node in an overlay network is associated with both an overlay address (to address to the overlay node) and an underlay address (to address the underlay node that implements the overlay node). An overlay node may be a digital device and / or a software process (such as, a virtual machine, an application instance, or a thread) A link that connects overlay nodes is implemented as a tunnel through the underlying network. The overlay nodes at either end of the tunnel treat theunderlying multi-hop path between them as a single logical link. Tunneling is performed through encapsulation and decapsulation.

[0089] In an embodiment, a client may be local to and / or remote from a computer network. The client may access the computer network over other computer networks, such as a private network or the Internet. The client may communicate requests to the computer network using a communications protocol, such as Hypertext Transfer Protocol (HTTP). The requests are communicated through an interface, such as a client interface (such as a web browser), a program interface, or an application programming interface (API).

[0090] In an embodiment, a computer network provides connectivity between clients and network resources. Network resources include hardware and / or software configured to execute server processes. Examples of network resources include a processor, a data storage device, a virtual machine, a container, and / or a software application. Network resources are shared amongst multiple clients. Clients request computing services from a computer network independently of each other. Network resources are dynamically assigned to the requests and / or clients on an on-demand basis. Network resources assigned to each request and / or client may be scaled up or down based on, for example, (a) the computing services requested by a particular client, (b) the aggregated computing services requested by a particular tenant, and / or (c) the aggregated computing services requested of the computer network. Such a computer network may be referred to as a “cloud network.”

[0091] In an embodiment, a service provider provides a cloud network to one or more end users. Various service models may be implemented by the cloud network, including but not limited to Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), and Infrastructure-as-a- Service (laaS). In SaaS, a service provider provides end users the capability to use the service provider’s applications, which are executing on the network resources. In PaaS, the service provider provides end users the capability to deploy custom applications onto the network resources. The custom applications may be created using programming languages, libraries, services, and tools supported by the service provider. In laaS, the service provider provides end users the capability to provision processing, storage, networks, and other fundamental computing resources provided by the network resources. Any arbitrary applications, including an operating system, may be deployed on the network resources.

[0092] In an embodiment, various deployment models may be implemented by a computer network, including but not limited to a private cloud, a public cloud, and a hybrid cloud. In a private cloud, network resources are provisioned for exclusive use by a particular group of one or more entities (the term “entity” as used herein refers to a corporation, organization, person, or other entity). The network resources may be local to and / or remote from the premises of the particular group of entities. In a public cloud, cloud resources are provisioned for multiple entities that are independent from each other (also referred to as “tenants” or “customers”). The computer network and the network resources thereof are accessed by clients corresponding to different tenants. Such a computer network may be referred to as a “multi-tenant computer network.” Several tenants may use a same particular network resource at different times and / or at the same time. The network resources may be local to and / or remote from the premises of the tenants. In a hybrid cloud, a computer network comprises a private cloud and a public cloud. An interface between the private cloud and the public cloud allows for data and application portability. Data stored at the private cloud and data stored at the public cloud may be exchanged through the interface. Applications implemented at the private cloud and applications implemented at the public cloud may have dependencies on each other. A call from an application at the private cloud to an application at the public cloud (and vice versa) may be executed through the interface.

[0093] In an embodiment, tenants of a multi-tenant computer network are independent of each other. For example, a business or operation of one tenant may be separate from a business or operation of another tenant. Different tenants may demand different network requirements for the computer network. Examples of network requirements include processing speed, amount of data storage, security requirements, performance requirements, throughput requirements, latency requirements, resiliency requirements, Quality of Service (QoS) requirements, tenant isolation, and / or consistency. The same computer network may need to implement different network requirements demanded by different tenants.

[0094] In one or more embodiments, in a multi-tenant computer network, tenant isolation is implemented to ensure that the applications and / or data of different tenants are not shared with each other. Various tenant isolation approaches may be used.

[0095] In an embodiment, each tenant is associated with a tenant ID. Each network resource of the multi-tenant computer network is tagged with a tenant ID. A tenant is permitted access to aparticular network resource only if the tenant and the particular network resources are associated with a same tenant ID.

[0096] In an embodiment, each tenant is associated with a tenant ID. Each application, implemented by the computer network, is tagged with a tenant ID. Additionally or alternatively, each data structure and / or dataset, stored by the computer network, is tagged with a tenant ID. A tenant is permitted access to a particular application, data structure, and / or dataset only if the tenant and the particular application, data structure, and / or dataset are associated with a same tenant ID.

[0097] As an example, each database implemented by a multi-tenant computer network may be tagged with a tenant ID. Only a tenant associated with the corresponding tenant ID may access data of a particular database. As another example, each entry in a database implemented by a multi-tenant computer network may be tagged with a tenant ID. Only a tenant associated with the corresponding tenant ID may access data of a particular entry. However, the database may be shared by multiple tenants.

[0098] In an embodiment, a subscription list indicates which tenants have authorization to access which applications. For each application, a list of tenant IDs of tenants authorized to access the application is stored. A tenant is permitted access to a particular application only if the tenant ID of the tenant is included in the subscription list corresponding to the particular application.

[0099] In an embodiment, network resources (such as digital devices, virtual machines, application instances, and threads) corresponding to different tenants are isolated to tenantspecific overlay networks maintained by the multi-tenant computer network. As an example, packets from any source device in a tenant overlay network may only be transmitted to other devices within the same tenant overlay network. Encapsulation tunnels are used to prohibit any transmissions from a source device on a tenant overlay network to devices in other tenant overlay networks. Specifically, the packets received from the source device are encapsulated within an outer packet. The outer packet is transmitted from a first encapsulation tunnel endpoint (in communication with the source device in the tenant overlay network) to a second encapsulation tunnel endpoint (in communication with the destination device in the tenant overlay network). The second encapsulation tunnel endpoint decapsulates the outer packet to obtain the original packet transmitted by the source device. The original packet is transmittedfrom the second encapsulation tunnel endpoint to the destination device in the same particular overlay network.

[0100] 5. HARDWARE OVERVIEW

[0101] According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or network processing units (NPUs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, FPGAs, or NPUs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard-wired and / or program logic to implement the techniques.

[0102] For example, Figure 3 is a block diagram that illustrates a computer system 300 upon which an embodiment of the invention may be implemented. Computer system 300 includes a bus 302 or other communication mechanism for communicating information, and a hardware processor 304 coupled with bus 302 for processing information. Hardware processor 304 may be, for example, a general-purpose microprocessor.

[0103] Computer system 300 also includes a main memory 306, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 302 for storing information and instructions to be executed by processor 304. Main memory 306 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 304. Such instructions, when stored in non-transitory storage media accessible to processor 304, render computer system 300 into a special-purpose machine that is customized to perform the operations specified in the instructions.

[0104] Computer system 300 further includes a read only memory (ROM) 308 or other static storage device coupled to bus 302 for storing static information and instructions for processor304. A storage device 310, such as a magnetic disk or optical disk, is provided and coupled to bus 302 for storing information and instructions.

[0105] Computer system 300 may be coupled via bus 302 to a display 312, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 314, including alphanumeric and other keys, is coupled to bus 302 for communicating information and command selections to processor 304. Another type of user input device is cursor control 316, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 304 and for controlling cursor movement on display 312. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.

[0106] Computer system 300 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and / or program logic which in combination with the computer system causes or programs computer system 300 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 300 in response to processor 304 executing one or more sequences of one or more instructions contained in main memory 306. Such instructions may be read into main memory 306 from another storage medium, such as storage device 310. Execution of the sequences of instructions contained in main memory 306 causes processor 304 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

[0107] The term “storage media” as used herein refers to any non-transitory media that store data and / or instructions that cause a machine to operate in a specific fashion. Such storage media may comprise non-volatile media and / or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 310. Volatile media includes dynamic memory, such as main memory 306. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge, content-addressable memory (CAM), and ternary content addressable memory (TCAM).

[0108] Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 302. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

[0109] Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 304 for execution. For example, the instructions may initially be carried on a magnetic disk or solid-state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 300 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 302. Bus 302 carries the data to main memory 306, from which processor 304 retrieves and executes the instructions. The instructions received by main memory 306 may optionally be stored on storage device 310 either before or after execution by processor 304.

[0110] Computer system 300 also includes a communication interface 318 coupled to bus 302. Communication interface 318 provides a two-way data communication coupling to a network link 320 that is connected to a local network 322. For example, communication interface 318 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 318 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 318 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0111] Network link 320 typically provides data communication through one or more networks to other data devices. For example, network link 320 may provide a connection through local network 322 to a host computer 324 or to data equipment operated by an Internet Service Provider (ISP) 326. ISP 326 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the “Internet” 328. Local network 322 and Internet 328 both use electrical, electromagnetic or optical signals thatcarry digital data streams. The signals through the various networks and the signals on network link 320 and through communication interface 318, which carry the digital data to and from computer system 300, are example forms of transmission media.

[0112] Computer system 300 can send messages and receive data, including program code, through the network(s), network link 320 and communication interface 318. In the Internet example, a server 330 might transmit a requested code for an application program through Internet 328, ISP 326, local network 322 and communication interface 318.

[0113] The received code may be executed by processor 304 as it is received, and / or stored in storage device 310, or other non-volatile storage for later execution.

[0114] 6. MISCELLANEOUS; EXTENSIONS

[0115] Embodiments are directed to a system with one or more devices that include a hardware processor and that are configured to perform any of the operations described herein and / or recited in any of the claims below.

[0116] In an embodiment, a non-transitory computer readable storage medium comprises instructions which, when executed by one or more hardware processors, causes performance of any of the operations described herein and / or recited in any of the claims.

[0117] Any combination of the features and functionalities described herein may be used in accordance with one or more embodiments. In the foregoing specification, embodiments have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent scope of the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction.

Claims

CLAIMSWhat is claimed is:

1. A method compri sin : receiving, by an application for creating simulations within augmented or virtual reality environments, a set of modular component blocks and / or a natural language request that specifies a first set of one or more details regarding a custom medical scenario to simulate within an augmented and / or virtual reality environment; generating, by the application based on the first set of one or more details, at least one input prompt for an artificial intelligence model that causes the artificial intelligence model to infer a second set of one or more details regarding the medical scenario that were not included in the first set of one or more details; generating, by the application based on the first set of one or more details and the second set of one or more details, at least one interpretable file for simulating the custom medical scenario within the augmented and / or virtual reality environment.

2. The method of Claim 1, wherein the set of modular component blocks includes a user- defined character specification for a virtual patient in the medical scenario.

3. The method of Claim 1, wherein the set of modular component blocks includes a user- defined tool specification for a medical tool in the medical scenario.

4. The method of Claim 1, wherein the set of modular component blocks include user- defined animations, dialogue, environments, critical actions, and / or assessments for the medical scenario.

5. The method of Claim 1, wherein the set of modular component blocks includes two or more interlocking blocks.

6. The method of Claim 1, wherein the set of modular component blocks are received by a case creator through a graphical user interface that presents building blocks for building new medical scenarios to simulate.

7. The method of Claim 1, further comprising: loading, by a simulation application executing on an AR / VR headset, the interpretable file; and executing, by the simulation application, instructions that cause the AR / VR headset to project virtual avatars and / or objects that exhibit behaviors based on the data contained in the interpretable file.

8. The method of Claim 1, further comprising: training the Al model based on example clinical scenarios.

9. The method of Claim 7, wherein the Al model learns patterns that are clinically consistent; wherein the second set of details are output based on clinical consistency with the first set of details.

10. The method of Claim 1, wherein the Al model is a large language model; wherein the at least one input prompt is a natural language prompt that includes the first set of one or more details and requests the Al model to output the second set of one or more details.

11. One or more non-transitory computer-readable media storing instructions which, when executed by one or more hardware processors cause execution of the method of any of claims 1- 10.

12. A virtual or augmented reality system comprising: one or more hardware processors; a virtual or augmented reality display; and one or more non-transitory computer-readable media storing instructions which, when executed by one or more hardware processors cause execution of the method of any of claims 1-10.

Images