Virtual reality training system and method

Abstract

A virtual reality stress inoculation training system analyzes the biological response of a subject exposed to a simulated high stress environment. The system includes processor connected with a haptic device, a display device, and a biometric sensor. A memory storing a high stress training content application is connected with the processor. The content application includes a scenario with one or more high stress events. The processor executes the content application to display images on the display device and to send and receive signals with the haptic device to actuate the haptic device to simulate the high stress environment and induce sensations in the subject to simulate the high stress environment according to a scenario. The processor receives biometric parameters of the subject simultaneously with the scenario and analyzes the parameters in relation to the one or more high stress events. The analysis is used to reinforce psychological skills and mitigate long-term negative behavioral health outcomes associated with traumatic stress (e.g., PTSD).

Description

VIRTUAL REALITY TRAINING SYSTEM AND METHODBACKGROUNDField

[0001] The present disclosure generally relates to systems and methods for training, evaluating, and improving a trainee’s performance and for identifying and / or mitigating mental health injuries in highly stressful situations. More particularly, the present disclosure relates to systems and methods for exposing a trainee to a stressful scenario using a virtual reality (VR), augmented reality (AR), and / or mixed reality (MR) system, monitoring biometric parameters of the trainee, and for identifying and / or mitigating mental health vulnerabilities of the trainee.Description of the Related Art

[0002] In certain career fields, such as first responders (e.g., law enforcement, emergency medical technicians, etc ), military, medical, etc., professionals are at an increased risk of developing PTSD, moral injury, or experiencing other deleterious mental health issues. Much of the training these professionals receive neglect mental health resilience training and instead focus on task performance only. However, mental health problems will often have a negative impact on a professional’s cognitive processing ability and impedes their job performance.Furthermore, any mental health issue will affect the professional’s home life which may exacerbate any negative job performance the professional might have.

[0003] Being able to perform under highly stressful conditions requires first responders, military members, medical providers, and warfighters to regulate their psychophysiological state in relation to their emergency task-specific mission. It has been reported that combat stress increases sympathetic nervous system responses, which have a direct effect on cortical arousal (Aguilera et al., 2021), leading to deficiencies in cognitive and behavioral health processes, such as decision-making, memory, and psychological stress (Clemente-Suarez et al., 2018; Clemente- Suarez, & Perez, 2013; Taverniers et al., 2010). In addition to acute psychophysiological impact, chronic stress can have deleterious biopsychosocial consequences for warfighters’ health.SUMMARY

[0004] Virtual Reality (VR) is the term used to describe a three-dimensional computergenerated environment with which a user can interact. VR environments may resemble the real world by providing users with sensory experience including seeing, feeling, touching, smelling, and hearing experiences. A person using VR may interact with objects displayed in VR or perform a series of actions, and may be assisted by VR devices, wearable haptic devices, a headmounted display, omnidirectional treadmills, etc.

[0005] The use of a virtual reality, augmented reality, and / or mixed reality application according to the present disclosure assists in training and evaluating j ob specific, high stress performance skills, and identifying and / or mitigating mental health vulnerabilities.

[0006] Modern warfare operations occur in volatile, highly complex environments, placing immense physiological, psychological, and cognitive demands on the warfighter. To maximize cognitive performance, warfighter resilience and readiness, training must integrate psychological skills which, to the greatest extent possible, enhance health and performance. Resiliency is the capacity to face and adapt to adversity and is fundamentally rooted in an individuals’ psychophysiological stress response. It is optimized through decreased susceptibility to the negative impact of trauma exposure and combat. Resilience has repeatedly been demonstrated as a protective factor in mitigating military mental health concerns, including posttraumatic stress disorder (PTSD), suicidal ideation, substance and mood disorders, and moral injury.

[0007] Recreating real-world training scenarios is expensive and time consuming. Multiple actors may be required to simulate a real-world situation (battlefield, natural disaster site, etc.). Recreations may be dangerous, for example, where live ammunition, smoke, low visibility, etc. are used to enhance realism. Certain actions are difficult to recreate without causing harm to human actors. For example, training medical personnel to perform tactical casualty care procedures (e.g., tracheotomy, applying sutures, etc.) may require a “cut suit” simulation mannequin, or other specialized equipment.

[0008] Thus, there is a need for a safe and scalable controlled stress exposure / inoculation training environment.

[0009] Stress inoculation training (SIT) in FDVR is the exposure to highly realistic and traumatic mass casualty events. It provides the opportunity to exercise psychological skills and build resilience while advancing technical expertise to increase warfighter medical capabilities in real, life-or-death settings. In addition to resilience building, the FDVR platform with SIT and embedded psychological skills and evaluation may increase hardiness, improve emotion recognition and regulation (i.e., emotional intelligence), and serve as a protective factor in mitigating long-term behavioral health concerns associated with traumatic stress exposure (e g., PTSD, suicidal ideation, moral injury)

[0010] As trainees are exposed to high levels of stress during an exercise, there is a risk that the training itself could lead to mental health injury, for example moral injury or PTSD. There is a need for a training environment that allows a trainee’s cognitive, behavioral, and biometric parameters to be monitored to assure that the training to the highest extent possible, will not cause mental or physical injury.

[0011] The present disclosure provides systems and methods for immersing a trainee in a highly realistic VR training and evaluation simulation, providing elements of the simulation to expose the trainee to mental and / or physical stress and evaluating performance skills while also identifying and / or mitigating mental health vulnerabilities experienced by the trainee. The VR training environment improves trainee expertise, resilience, adaptability, self-awareness, and performance by creating a high-fidelity, safe, and scalable controlled stress exposure training environment with embedded cognitive, behavioral, and psychophysiological skills development and evaluation. Embodiments of the present disclosure may include methods for identifying a professional at an increased risk of developing a negative mental health disorder, e.g., post- traumatic stress disorder (PTSD). In one embodiment, this is achieved by monitoring a professional’s psychophysiological responses when placed in a virtual reality system with an application running a high-stress scenario.

[0012] In another embodiment, a professional may be placed in a virtual reality system with an application running a high-stress scenario and the professional’s physiological responses recorded and stored to gather a baseline. The professional may then be retested at various pointsin time during their career to see if he / she has developed or is at a greater risk of developing PTSD or the like.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] A more complete appreciation of the present disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:

[0014] Fig. 1 illustrates a cloud-based training simulation system according to an embodiment of the disclosure;

[0015] Fig. 2 is a schematic of a cloud-based training simulation system according to another embodiment of the disclosure;

[0016] Fig. 3 is a flow chart illustrating a method of tracking eye movements of a user according to an embodiment of the disclosure;

[0017] Fig. 4 illustrates a user immersed in a cloud-based training simulation system according to an embodiment of the disclosure ;

[0018] Fig. 5 illustrates multiple users in multiple locations immersed in a cloud-based training simulation system according to an embodiment of the disclosure wherein the users are able to interact with each other;

[0019] Fig. 6 illustrates a cloud-based training simulation system according to a further embodiment of the disclosure;

[0020] Fig. 7 is a flow chart illustrating operations of a cloud-based training simulation system according to embodiments of this disclosure;

[0021] Figs 8a and 8b are graphs of biometric data collected during a simulation executed on a cloud-based training simulation system according to an embodiment of the disclosure from a first and a second operator, respectively, mapped against significant events during the simulation scenario; and

[0022] Fig. 9 is a flowchart showing a training implementation executed on a cloud-based training simulation system according to an embodiment of the disclosure.DETAILED DESCRIPTION

[0023] Exemplary embodiments of the disclosure will now be described below by reference to the attached Figures. The described exemplary embodiments are intended to assist the understanding of the invention and are not intended to limit the scope of the invention in any way. Like reference numerals refer to the elements throughout.

[0024] The present disclosure provides exemplary embodiments of a method of using extended reality (XR) systems to improve a user’ s technical and operational skills, and emotional intelligence, hardiness, resilience, and to help mitigate the development of post-traumatic stress disorder (PTSD) or other mental health disorders. Extended reality (XR) includes augmented reality (AR), virtual reality (VR), mixed reality (MR), and the like. For ease of description, the XR system may also be referred to herein as the “system” in the singular and the “systems” in the plural. One skilled in the art will recognize that the present invention may be practiced without some or all of these specific details. In other instances, well-known process operations have not been described in detail in order not to unnecessarily obscure the present disclosure.

[0025] Turning to the figures and in particular Figs. 1-2, an exemplary embodiment of an XR system according to the present disclosure is shown. In the embodiment shown, system 100 includes a variety of input devices such as a mobility platform 104, haptic devices 106a, 106b, a display device 108, an olfactory virtual reality device 110 and a biometric unit 112 connected with a computing element 102. According to one embodiment, the system’s input devices communicate wirelessly with one another and with the computing element. According to another embodiment, the system’s input devices communicate via wired connections. It is envisioned that a combination of wired and / or wireless connections may be used so long as the system 100 can communicate with a network 202 and enable input from a user to drive interactivity within a virtual world, as will be described below.

[0026] The computing element 102 may be any of a variety of types of computing devices, such as a personal computer, set top box, gaming console, or any other type of device having atleast one processor and memory for processing and storing data. In an exemplary embodiment, computing element 102 is networked, and communicates over a network, such as the Internet 202, with one or more cloud servers 204. In the exemplary embodiment shown in Fig. 1 the computing element 102 is a standalone device that is connected to a display device 108 and other input devices such as haptic devices 106, mobility platform 104, and / or biometric unit 112, and provides video data for rendering on the display device 108. In other embodiments, the computing element 102 may be integrated into a display device 108, a haptic device 106, and / or a mobility platform 104. For example, the mobility platform 104 is a networked mobility platform providing a platform operating system for applications or “apps” utilizing the network connectivity of the mobility platform 104. Computer element 102 is connected with a data storage and retrieval device 102a that stores one or more XR environment scenarios 102b.

[0027] An exemplary embodiment of the mobility platform 104 allows for unrestricted motion, for example in the form of an omnidirectional treadmill as shown in Fig. 1. This embodiment allows the user to interact with the virtual world in all manners of positions such as prone, kneeling, and / or standing. In an embodiment which includes an omnidirectional treadmill, the velocity and / or acceleration of the user’s locomotive movement is measured by suitable sensors included on, in, and / or about the omnidirectional treadmill. The measured velocity and / or acceleration can be transmitted to the computing element 102 and processed in parallel with data from the other input devices, allowing the avatar of the user to be plotted as moving within a virtual world in accordance with the user’s own physical movement. The 360 omnidirectional treadmill is controlled by the movements of the operator within the XR environment, and automatically moves the user back to the center of the treadmill regardless of the direction of movement within the virtual scenario. This offers a high level of safety to the user. According to one embodiment, platform 104 is an Infinadeck Omnidirectional Treadmill.

[0028] According to one embodiment, mobility platform 104 uses a tether or a restraining device to keep the user from accidentally moving off the mobility platform while moving / interacting in the virtual world. According to another embodiment, the mobility platform doesn’t use a tether or a restraining device. This may allow a subject using the system a greater freedom of movement.

[0029] According to one embodiment, the display device 108 is a wearable device, such as a head-mounted device (HMD), for example, smart glasses, smart contact lenses, headsets or any other device worn by a human for holographic projector or any other device or system capable of providing visual and / or auditory elements of an augmented reality (AR), virtual reality (VR), mixed reality (MR), or any immersive experience. Typical components of wearable XR devices may include at least one of: a stereoscopic HMD, a stereoscopic head-mounted sound system, head-motion tracking sensors (such as gyroscopes, accelerometers, magnetometers, image sensors, structured light sensors, etc.), head mounted projectors, eye-tracking and hand tracking sensors, and additional components. In another embodiment, the display device 108 is a nonwearable device capable of immersing a user in an extended reality. The non-wearable display device includes multi-projected environment devices. In an exemplary embodiment, the display device 108 is configured to change the viewing perspective of the XR environment in response to movements of the user.

[0030] In an exemplary embodiment, the HMD 108 includes Hand Tracking allowing the user to interact with the virtual environment without the need for the user to wield a VR controller. According to one embodiment, the HMD 108 uses interface Hand Tracking in order to allow the user to visualize “virtual hands” that emulate the user’s actual hands in real time and interact with objects and / or people within the virtual world. Non-limiting examples of interface Hand Tracking include inertial motion capture gloves and optical motion capture systems. Inertial motion capture gloves use Inertial Measurement Units (IMUs) with built-in sensors to track the movement of a user’s hands. The IMUs include gyroscopes, accelerometers, magnetometers, or a combination thereof. The HMD 108 shown in Fig. 1 utilizes interface Hand Tracking via an optical motion capture system that uses one or more cameras and one or more reflective sensors or markers to track movement in three-dimensional space. The camera emits infrared light and the reflective sensors / markers reflect the light which is then captured by the camera. The captured images of the markers are used to reflect the movement of the markers and create a three-dimensional model of the user’s hand. One example of a suitable device for hand tracking includes the Leap Motion Controller as marketed by Ultra Leap of Mountain View, California. According to another embodiment, Hand Tracking uses markerless Hand Tracking, i.e., tracking without an interface. It will be readily appreciated after a thorough review of the presentdisclosure, the accuracy reflected in the created three-dimensional model of the user’s hand may be readily optimized or understood with the selection of the desired number of cameras. It is also envisioned that any type of hand tracking technology that is known may be used within the scope of the disclosure.

[0031] In the depicted system 100 of Fig. 1, the HMD 108 includes an eye tracking system. According to one embodiment, the eye tracking system uses Pupil Center Corneal Reflection (PCCR). Fig. 3 shows a simplified flow chart 300 illustrating an example method for tracking the eye of a user of an HMD 108 using the PCCR method. At block 310, one or more light sources direct near-infrared light towards the center of the user’s eyes (pupil), causing detectable reflections in both the pupil and the cornea. According to one embodiment, the light sources may be located in the field of view of the user’s eye or at a periphery of the field of view of the user’s eye. According to another embodiment, a light source is located at the periphery of the field of view of the user’s eye, and the light from the light source may be guided and directed to the user’s eye from locations in the field of view of the user’s eye. At block 320 and 330, one or more cameras (or other type of optical sensor) detects the reflections so that a vector is calculated based on the angle between the cornea and pupil reflections. At block 340, the direction of the vector combined with other geometrical features on the reflections are used to determine eye position. According to another embodiment, the eye tracking system uses electro-oculography technology. Eye, gaze, and pupil tracking allows assessment of the operator’s engagement physiological stress to tasks within the virtual training / evaluation environment. However, it is envisioned that any type of eye tracking technology that is known may be used. According to one embodiment the HMD 108 is a Varjo-XR3 headset manufactured by Varjo. According to one embodiment, HMD 108 is integrated with a system for eye and hand tracking, for example, the Leap 2 Motion Controller or an Ultraleap hand tracking camera, manufactured by Ultraleap. According to one embodiment, HMD 108 provides biometric eye tracking to measure cognitive, emotional, and attentional processes and is an indicator of neural impairment.

[0032] Visuomotor control assessment represents a window into neuro-functioning and is a noninvasive, rapid, and reliable method to examine how the subject interacts with the visual world. Knowing where and when to look is crucial for successful warfighter performance and a reliance on visual attentional control allows increased emotional regulations and improvedwarfighter response. According to one embodiment, the system enables a feedback loop wherein biometric data coupled together in real-time with personality and cognitive behavioral assessments of a warfighter’s psychophysiological state increases individual self-awareness, self- efficacy, task performance, and overall well-being.

[0033] Data related to the psychophysiological states of subjects are tracked longitudinally for individuals and teams. According to one embodiment, training data is correlated in time and space, per participant with reference frames, and then processed by automated algorithms that produce analytical results based on reproducible scientific principles. Displays of analytic results allow subjects and superior officers to readily assess if the event or effort is producing the desired outcomes and provide associated data-driven insights, conclusions, and predictions, as desired and to evaluate how the subjects’ training is progressing.

[0034] The integration of two distinct technological discipline areas is required to adequately address this identified need. The first technology discipline required is Data Engineering and Management, which focuses on collecting, extracting, transforming, and loading data into a data repository. This is known as an extract, transform, load (ETL) Data Ingestion process.Associated data engineering processes include data curation, cataloging, provenance, and data modeling along with providing stored data to authorized entities via Application Program Interfaces (API). The second technology discipline area is Data Science and Analytics, which includes machine learning (ML) and artificial intelligence (Al) combined with mathematics and statistics that rapidly analyzes large amounts of data to establish ‘meaning’ of data. According to one embodiment, the system uses mathematical and statistical algorithms and methods for regression, instance-based learning models, regularization, decision trees, Bayesian modeling, clustering, association rule learning, neural networks, deep learning, dimensionality reduction, and ensemble methods.

[0035] In the embodiment depicted in Fig. 1, the HMD 108 worn by the user tracks the gaze of the user and has a display to display the virtual world of the XR application to the user. The image displayed to the user of the virtual world depends on the gaze of the user tracked by the HMD 108. For example, if the user turns their head to the left, the HMD 108 displays an image to the left of the user’s avatar in the virtual world. If the user looks down, the HMD 108 displaysthe ground in front of the character in the virtual world. In some examples, the HMD 108 tracks the gaze of the user and an image of the virtual world of the gaming system is displayed on an external display (e.g., computer monitor or television). In addition, as explained above, in some examples, the HMD 108 tracks the motion of the hands of the user.

[0036] Example of HMDs include but are not limited to Varjo XR-3, Varjo XR-4, Varjo VR- 3, and Varjo VR-4 headsets manufactured by Varjo; Oculus Rift, Oculus Go, and Oculus Quest headsets manufactured by Oculus VR; Meta Quest 3 and MetaQuest 3S headsets manufactured by Meta; Focus 3, Focus Plus, VIVE XR Elite, VIVE Flow, and Pro 2 headsets manufactured by manufactured by Vive; Valve Index headset manufactured by Valve Corporation; PlayStation VR headset manufactured by Sony Corporation; Razer OSVR headset manufactured by Razer Inc.; Fove VR headset manufactured by Fove, Inc.; Pimax Crystal Light headset manufactured by Pimax; VRHero and XTAL headsets manufactured by VRgnineers; Deepoon VR headset manufactured by DPVR; Dell Visor headset manufactured by Dell Inc.; Asus HC headset manufactured by ASUSTeK Computer Inc.; Acer WMR headset manufactured by Acer Inc.; HP WMR and HP Reverb headsets manufactured by Hewlett-Packard; Lenovo Explorer headset manufactured by Lenovo Group Limited; Samsung Odyssey and Samsung Gear VR headsets manufactured by Samsung Electronics; LG Steam VR headset manufactured by LG Electronics, and the like.

[0037] In an exemplary embodiment, the system 100 provides non-visual feedback to the user using haptic technology. Haptic technology is one class of solutions for providing user(s) with physical feedback, which creates an experience of touch by applying forces, vibrations, or motions to the user. According to one embodiment, a haptic device 106 is configured to provide haptic feedback to the user in response to receiving input corresponding to the user-initiated action and / or in response to the initiation of the responsive action(s). Haptic devices 106 are well known in the art, and can include, for example, gloves, watches, headsets, vests, or other wearable apparel, fitted with haptic feedback technology that produce vibrations, electrical stimulation or pressure (or other haptic feedback) that is sensed by the user by touch or feel. Haptic feedback may be implemented using motors, piezoelectric actuators, fluidic (e.g., hydraulic or pneumatic) systems, and / or a variety of other types of feedback mechanisms. According to one embodiment, haptic feedback devices are implemented independent of otherXR devices. According to another embodiment, haptic feedback devices are implemented within other XR devices. According to yet another embodiment, haptic feedback devices are implemented in conjunction with other XR devices. In a non-limiting example, the haptic device may be implemented as a haptic glove 106a that is configured to be worn on the hand of the user and that provides pressure, resistance, and / or vibration feedback to one or more of the fingers on the hand that wears the haptic glove 106a. In such an example, the user initiates an action on a virtual object by “touching” the virtual object in the display image with his / her finger(s), and the haptic glove 106a vibrates and / or applies pressure to one or more fingers in response to the action initiated by the user. In another embodiment, haptic gloves 106a include Hand Tracking allowing the user to interact with the virtual environment without the need for the user to wield a VR controller as discussed above.

[0038] In another non-limiting example, the haptic device 106 may be implemented in or be a component of a haptic suit 106b worn by the user. According to one embodiment, the haptic suit 106b is a full body suit. According to another embodiment, the haptic suit 106b covers only portions of the user’s body. In an exemplary embodiment, the haptic suit 106b uses electro muscle stimulation (EMS) and / or transcutaneous electrical nerve stimulation (TENS) to simulate a range of real-life feelings and sensations that a user’s avatar might experience while interacting with the virtual world. For example, if the user’s avatar is “shot” in the virtual world, the haptic suit 106b may use EMS and / or TENS to simulate the user being shot in the location where the avatar was “shot”. In yet another embodiment, the haptic suit 106b may be a motion capture suit that allows the movements of the user to translate into the movement of the user’s avatar in the virtual world.

[0039] In another exemplary embodiment, system 100 includes an olfactory virtual reality or “OVR” device 110. According to one embodiment, the OVR device 110 is attached to an HMD 108, such that the outputs of the OVR device 110 are positioned near the user’s nose. According to another embodiment, the OVR device 110 features are fully incorporated within the HMD 108. In one implementation of a fully integrated system, commands used to control OVR functions are integrated within the headset inputs provided by the XR application. In some embodiments, the OVR device 110 is integrated within other inputs, such as mobility equipment 104, haptic devices 106, or other devices. According to yet another embodiment, the OVRdevice 110 is a standalone device. The OVR device 110 is configured to release scents according to the virtual environment. For example, if the user is immersed in a virtual world of a burning building, the OVR device 110 releases scents similar to that of a burning building. According to one embodiment, the one or more OVR devices are controlled to release scent automatically responsive to actions performed within an XR environment. According to one embodiment, the one or more OVR devices are controlled to release scent automatically responsive to a proximity to a scent-generating asset within the XR environment. It is envisioned that the OVR device 110 may include elements of digital scent technology, such as, but not limited to, olfactometers, electronic noses, scent generating devices, synthetic and natural odor cartridges, and aroma mixers, as may be understood by a person having ordinary skill in the art.

[0040] In another exemplary embodiment, system 100 includes biometric sensor(s) 112 for measuring the change in a user’s biometric characteristics as the user is interacting in the virtual world. The biometric sensors may include one or more sensors for detecting various biometric characteristics of a user such as, for example, heart rate, systolic blood pressure, breathing rate, skin temperature, core body temperature, pulse oximetry, stroke volume, and cardiac output , or other biometric data. In an embodiment, the biometric sensor(s) are integrated into a wearable haptic device 106 previously described. In another embodiment, the biometric sensor(s) is a standalone device. According to one embodiment, the biometric sensor 112 includes a telemetry system for communicating biological parameters wirelessly to a server for analysis, as will be described below. Biometric sensor 112 may be a hemodynamic monitor such as a Caretaker 4 or Caretaker Vitalstream monitor manufactured by Caretaker Medical or a LifeLens wearable patch manufactured by LifeLens. According to another embodiment, biometric sensor 112 measures one or more of heart rate (HR), heart rate variability (HRV), skin conductance, core temperature estimation, gait stability analysis, blood pressure estimate, pulse oxygenation, respiration rate (including respiration depth and effort), vocal biomarkers, and sleep analysis including sleep identification, actigraphy, sleep respiration, sleep HR / HRV, sleep posture, snoring identification, snore counting, non-audio snoring analysis (type, source location, obstructive / un obstructive), sleep state assessment (first pass assessment only including slow wave sleep, wake-ups, REM, and light sleep states).

[0041] Fig. 4 illustrates a user interacting with a cloud-based XR application using system 100 as described above, in accordance with embodiments of the invention. As shown, a user 1 A, through the use of system 100, is immersed in extended reality via a cloud-based XR application where the user interacts with his / her surroundings to navigate a virtual world. A cloud-based XR application is an application that immerses a user in a virtual world and is primarily executed on a remote server 204. In an exemplary embodiment, the application is a simulation designed to elicit a physiological stress response from a user. A server, in one embodiment, includes one or more individual servers or servers that are executed in a virtual machine data center, where many servers can be virtualized to provide the required processing capability. According to one embodiment, system 100 is situated at the user’s location to receive and process inputs and communicate these to cloud servers 204, and also to receive video and audio data from the cloud servers 204. System 100 and the cloud servers 204 communicate over a network 202, such as the Internet. In the illustrated embodiment, cloud server(s) 204 execute the application that is rendered on the HMD 108 of system 100. In various embodiments, the degree of processing performed by system 100 may vary with respect to input and output processing. According to one embodiment, the game state is substantially maintained and executed on the cloud servers 204, with system 100 primarily functioning to receive and communicate user inputs, and receive video / audio data for rendering, as discussed in greater detail below with references to Fig. 8. According to another embodiment, instead of, or in addition to executing processing on a server 204 remotely from system 100, processing is executed by a server located near system 100. According to one embodiment, system 100 and the local server communicate via a local area network.

[0042] Fig. 5 illustrates multiple users in multiple locations engaged in a virtual world of a cloud-based XR application. User 1 A is shown at a first location interacting with a virtual world using system 100 and the perception of the virtual world to user 1A is rendered by system 100. Users IB and 1C are shown at a second location interacting with a virtual world using system 100’ and the perception of the virtual world to user IB and 1C are rendered by the respective system 100’ associated with each user. A user ID shown at a third location interacting with a virtual world using system 100” and the perception of the virtual world to user ID is rendered by system 100”. Users IE, IF, and 1G are shown at a fourth location interacting with a virtualworld using system 100”’ and the perception of the virtual world to users IE, IF, and 1G is rendered by the respective system 100’” associated with each user. In an exemplary embodiment, each of the users will be immersed within the same virtual world and able to interact with one another and the surrounding virtual environment to accomplish various tasks and goals associated with the application being executed.

[0043] Turning to Fig. 6, a user operates system 100 to provide input to a cloud-based XR application. As mentioned previously, system 100 may include any of various kinds of input devices, such as a mobility platform, haptic devices, a display device and / or a biometric unit. The input devices may be connected through a wired or wireless connection with system 100 and / or other parts of the system 100. System 100 communicates over a network 202 with a cloud service 204. System 100 processes data from the input devices to generate input data that is communicated to an XR application executed by cloud service 204. Additionally, system 100 receives video data from cloud service 204, for rendering on the display device.

[0044] Cloud service 204 includes resources for providing an environment in which an XR application can be executed. Broadly speaking, resources can include various kinds of computer server hardware, including processors, storage devices, and networking equipment, which can be utilized to facilitate execution of a XR application. In the illustrated embodiment, an XR application library 212 includes various scenario titles. Each scenario title defines executable code as well as associated data and asset libraries which are utilized to instantiate an XR application running the selected title. The host 214 can be a single computing device that defines a platform for instantiating virtual machines 216. In another embodiment, the host 214 can itself be a virtualized resource platform. In other words, the host 214 may operate over one or more server computing devices, handling the allocation and usage of the resources defined by the server computing devices, while presenting a unified platform upon which virtual machines 216 can be instantiated.

[0045] Each virtual machine 216 defines a resource environment which can support an operating system, upon which a XR application 218 can be run. In one embodiment, a virtual machine 216 is configured to emulate the hardware resource environment of a system 100, with an operating system associated with the system 100 being run on the virtual machine 216 tosupport the running of the XR application 218 which were developed for the system 100. In another embodiment, the operating system is configured to emulate a native operating system environment of a system 100, though the underlying virtual machine may or may not be configured to emulate the hardware of the system 100. In another embodiment, an emulator application is run on top of the operating system of a virtual machine, the emulator being configured to emulate the native operating system environment of a system 100 so as to support XR applications designed for that system 100.

[0046] Fig. 7 is a flow chart of a user accessing an XR application 218 and interacting with a virtual world provided by the executed application. At 910, an XR application is executed. For example, the trainer or user(s) selects a particular title scenario from the application library 212. Each title scenario offers a different virtual world with different tasks / missions for the user to complete. For example, selecting a title scenario titled “Burning Building” will immerse the user within a virtual burning building where the user has to make tactical decisions about extinguishing the fire, evacuating others, and / or treating casualties. As another example, selecting a title scenario titled “Medical Emergency” will immerse the user within a healthcare facility to make medical decisions and conduct medical procedures regarding an ongoing medical emergency. In yet another example, selecting a title scenario titled “Battlefield” will immerse the user within an ongoing firefight on a battlefield where the user must make tactical decisions such as troop movement, casualty care, and / or evacuation. It is contemplated in this disclosure, the title scenarios may be titled with any alphanumeric identifier that is descriptive to the scenario defined by the XR application. Once the title scenario is selected, the application 218 associated with the selected title scenario is then executed via the host 214 and / or virtual machine 216.

[0047] At 920, the simulation output may be determined and / or sent. For example, cloud service 204 executes the XR application and defines the scenario’s game state from moment to moment. Cloud service 204 communicates the defined game state over a network 202, e.g., the Internet, to system 100 so that video / audio / haptic data may be rendered on the appropriate input device. At 930, the simulation input is communicated by system 100 and received and / or processed by cloud service 204. For example, system 100 processes data from the input devices (HMD, mobility platform, haptic devices, biometric sensor, etc.) and generates input data that isthen communicated to cloud service 204 such that the user interacts with the virtual world the user is immersed in.

[0048] At 940, a subsequent simulation state may be determined. For example, a subsequent simulation state may be determined from the current simulation state and / or any received inputs from system 100.

[0049] According to one embodiment, the system determines biometric parameters and changes thereto when experiencing simulated environments. According to some embodiments, the simulated environments include real world scenarios. According to one embodiment, an operator useability test of the FDVR platform with two operators was conducted. The operators were placed in a VR simulated industrial disaster scenario and required to conduct the duties of a firefighter and paramedic. The scenario included a number of high stress events that included simulations of a fire blocking the path of the operators, an explosion that results in another fire, and an injured victim that requires first aid treatment and evacuation. The operators are tasked with fighting the fire and assisting the victim while being subject to the dangers posed by the fire and unexpected explosion. Each user was provided with system 100, including mobility platform 104 (omni directional treadmill), haptic suit 106b 9 (Teslasuit), an HMD 108 (Varjo XR-3), and biometric unit 112 (Caretaker 4 ). Following acclimation, operators were placed in an industrial fire metaverse and instructed to respond as a firefighter and paramedic. Biometrics were tracked and recorded using the Caretaker 4 and the data was mapped against significant events during the simulation scenario. The physiological responses from the two operators demonstrated real-time, beat-by-beat blood pressure (BP) and heartrate (HR) changes that correlated with stressors in the virtual scenario.

[0050] Figs. 8a and 8b are graphs illustrating examples of changes in biometric data of Operator 1 (acting as the firefighter) and Operator 2 (acting as the paramedic), respectively, while they were immersed in the firefighting scenario provided by an XR application. The graphs show the change in systolic blood pressure and the change in heartrate over time as the users navigate the scenario, perform tasks, and as the scenario further develops. Events occurring during the scenario are identified on the timeline of each graph. As shown in Figs. 8a and 8b, operator l ’s BP and HR are somewhat elevated as the scenario commences. An explosion andfire in his path occur just before about the 0.5-minute point in the scenario. Operator 1’s BP and HR decrease during this event. This may be a result of good training and acclimation of Operator 1 to the high stress events such as the explosion and fire in the scenario. Operators l’s BP and HR decrease and remain relatively low until, at about minute 2.5, the scenario presents another fire in Operator l’s path. Operator l’s BP and HR increase while fighting the fire.

[0051] Operator 2’s (acting as the paramedic) BP and HR are shown in Figs. 8a and 8b. Operator 2’s BP and HR remain relatively low until about the 1.5 minute point when the operator finds a deceased victim. His BP and HR remain relatively high when Operator 2 is instructed “we gotta go.” Additionally, there was a rise in BP and HR of both operators when they found a significant fire blocking their escape at about the 3.7 minute point.

[0052] The increase in biometric parameters may be due to physical exertion during the simulation, for example, running toward or away from the fire, and from the psychological stress of a disaster scenario. Because system 100 monitors bodily motions, data is available to measure the amount of physical exertion (for example, by steps taken on the treadmill and by bodily motions detected by the Teslasuit) can be determined and can be used to determine whether the increase in a subject’s vital signs is a reaction to stress or to physical exertion. According to one embodiment, parameters related to physical exertion are analyzed in relation to the psychological stresses induced by the XR simulation to determine the operator’s response to stress.

[0053] In this exemplary embodiment, the BP and HR were monitored and recorded. However, it is contemplated that any biometric data may be chosen to be graphed, monitored, and or analyzed including, but not limited to pulse oxygenation, skin temperature, core body temperature, respiration rate, skin conductivity, heart rate variability, and the like.

[0054] According to a further embodiment, training using systems according to embodiments of the disclosure may provide multisession programs, wherein biometric (including eye tracking), psychological, physical, and emotional data are collected. Fig. 9 is a flowchart showing a training implementation according to one embodiment comprising a total of six scenarios: (1 A & B, 2A & B, 3A & B) conducted over multiple days.

[0055] At step 900, subjects are trained for a minimum of one hour in resilience skills.

[0056] At step 902, subjects are fitted with a biometric device (for example, the Caretaker 4 and / or the LifeLens patch).

[0057] At step 904, baseline date is gathered, including, but not limited to heart rate, blood pressure, temperature, and pulse oxygenation. According to one embodiment, resting measurements are gathered for 15 minutes to set baseline results.

[0058] At step 906, subjects are logged into the system data management portal and participate in pre-training psychological and cognitive behavioral assessments. These may include, but are not limited to Hardiness-Resilience Gauge (HRG), Depression Anxiety Stress Scales (DASS), Moral Injury Symptom Scale ~ Healthcare Professionals (MISS-HP), Conner- Davidson Resilience Scale (CD-RISC) and Insomnia Severity Index (ISI).

[0059] At step 908, subjects are then fitted with all FDVR equipment, and equipment is calibrated to the individual. Equipment required is based on specific training scenarios. According to one embodiment, the equipment includes a VR / AR headset 108, Haptic body suit 106b, Haptic gloves 106a, 360 omnidirectional treadmill 104, and biometric monitoring device 112.

[0060] Subjects conduct final equipment calibration and VR metaverse familiarity training at step 910 to ensure individual and team ability to maneuver, react, and communicate within the VR metaverse, and without any physiological symptoms of VR sickness. Brief familiarity training was conducted before each FDVR training session. Prior to each training scenario, each participant conducted two minutes of grounding exercises that involve visual as well as biofeedback on heartrate and breathing regulation skills.

[0061] Subjects begin a TCCC training exercise in the FDVR training platform (1 A) for SO- 45 minutes at step 912. Individual and team performances are video recorded, along with continuous biometric monitoring, and maintained in a data management portal. The system tracks, in real-time, any biometric health measures that are outside of safe parameters. An Observer Controller Trainer (OC / T) will remove any participant from the FDVR training environment demonstrating unsafe biometric parameters, and request medical evaluation as directed based on set Standard Operating Procedures (SOP) for the training.

[0062] After completion of 1A training, each subject completes a TCCC skills self-evaluation checklist of their perceived performance of the TCCC skills used during the training scenario at step 914. The subject then views the video playback of their performance, and re-evaluates themselves using the same TCCC skills checklist. Each subject receives evaluation and feedback from the OC / T observing and monitoring the training. The OC / T may be physically present, or via a virtual presence.

[0063] During the morning session, at step 916 the subjects rest for a minimum of one hour before conducting the next training session in the afternoon.

[0064] Day one PM (IB):

[0065] Subjects are logged into the system data management portal to begin the next training session at step 906.

[0066] Subjects are then re-fitted with all FDVR equipment required for the scenario, and equipment is re-calibrated if needed at step 908. Brief familiarity training and the grounding exercises are conducted prior to initiating the FDVR training session at step 910.

[0067] Subjects begin a TCCC training exercise in the FDVR training platform (IB) for 30- 45 minutes at step 912. Individual and team performances are video recorded, along with continuous biometric monitoring, and maintained in the data management portal. The system tracks, in real-time, any biometric health measures that are outside of safe parameters. An Observer Controller Trainer (OC / T) will remove any participant from the FDVR training environment demonstrating unsafe biometric parameters, and request medical evaluation as directed based on set Standard Operating Procedures (SOP) for the training.

[0068] After completion of IB training, each subject completes a TCCC skills self-evaluation checklist of their perceived performance of the TCCC skills used during the training scenario at step 914. The subject then views the video playback of their performance, and re-evaluates themselves using the same TCCC skills checklist. Each subject also receives evaluation and feedback from the OC / T observing and monitoring the training. The OC / T may be physically present, or via a virtual presence.

[0069] After completion of training and evaluations, subjects are released for the day at step 918. According to one embodiment, biometric monitoring includes providing a portable electrocardiogram (ECG) device that is worn by the subject. According to one embodiment, such a device is an Ascent Platform manufactured by LifeLens Technologies is provided to continuously monitor the subject’s biometric parameters throughout the entire training, including during non-training times.

[0070] Day 2 AM (2A):

[0071] Subjects are logged into the system data management portal to begin the next training session at step 906.

[0072] At steps 908 and 910 subjects are then re-fitted with all FDVR equipment required for the scenario, and equipment is re-calibrated if needed. Brief familiarity training and grounding exercises are conducted prior to initiating the FDVR training session.

[0073] Subjects begin a TCCC training exercise in the FDVR training platform (2A) for 30- 45 minutes at step 912. Individual and team performances are video recorded, along with continuous biometric monitoring, and maintained in the data management portal. The system tracks, in real-time, any biometric health measures that are outside of safe parameters. An Observer Controller Trainer (OC / T) will remove any participant from the FDVR training environment demonstrating unsafe biometric parameters, and request medical evaluation as directed based on set Standard Operating Procedures (SOP) for the training.

[0074] After completion of 2A training at step 914, each subject completes a TCCC skills self-evaluation checklist of their perceived performance of the TCCC skills used during the training scenario. The subject then views the video playback of their performance, and reevaluates themselves using the same TCCC skills checklist. Each subject also receives evaluation and feedback from the OC / T observing and monitoring the training. The OC / T may be physically present, or via a virtual presence.

[0075] Subjects will then rest for a minimum of one hour before conducting the next training session in the afternoon at step 916.

[0076] Day Two PM (2B):

[0077] Subjects are logged into the system data management portal to begin the next training session at step 906.

[0078] Subjects are then re-fitted with all FDVR equipment required for the scenario, and equipment is re-calibrated if needed and a brief familiarity training and grounding exercises are conducted prior to initiating the FDVR training session at steps 908 and 910.

[0079] Subjects begin a TCCC training exercise in the FDVR training platform (2B) for SO- 45 minutes at step 912. Individual and team performances are video recorded, along with continuous biometric monitoring, and maintained in the data management portal. The system will track, in real-time, for any biometric health measures that are outside of safe parameters. An Observer Controller Trainer (OC / T) will remove any participant from the FDVR training environment demonstrating unsafe biometric parameters, and request medical evaluation as directed based on set Standard Operating Procedures (SOP) for the training.

[0080] After completion of 2B training at step 914, each subject completes a TCCC skills self-evaluation checklist of their perceived performance of the TCCC skills used during the training scenario. The subject then views the video playback of their performance, and reevaluates themselves using the same TCCC skills checklist. Each subject also receives evaluation and feedback from the OC / T observing and monitoring the training. The OC / T may be physically present, or via a virtual presence.

[0081] After completion of training and evaluations, subjects are released for the day at step 918. If utilizing the LifeLens system, subjects will continue to wear the waterproof, lite- weight sensor continuously throughout the entire training, including during non-training times.

[0082] Day 3 AM (3 A):

[0083] Subjects are logged into the system data management portal to begin the next training session at step 906.

[0084] Subjects are then re-fitted with all FDVR equipment required for the scenario, and equipment is re-calibrated if needed and brief familiarity training and grounding exercise are conducted prior to initiating the FDVR training session a steps 908 and 910.

[0085] Subjects begin a TCCC training exercise in the FDVR training platform (3 A) for SO- 45 minutes at step 912. Individual and team performances are video recorded, along with continuous biometric monitoring, and maintained in the data management portal. The system tracks, in real-time, for any biometric health measures that are outside of safe parameters. An Observer Controller Trainer (OC / T) will remove any participant from the FDVR training environment demonstrating unsafe biometric parameters, and request medical evaluation as directed based on set Standard Operating Procedures (SOP) for the training.

[0086] After completion of 3 A training, each subject completes a TCCC skills self-evaluation checklist of their perceived performance of the TCCC skills used during the training scenario at step 914. The subject then views the video playback of their performance, and re-evaluates themselves using the same TCCC skills checklist. Each subject also receives evaluation and feedback from the OC / T observing and monitoring the training. The OC / T may be physically present, or via a virtual presence.

[0087] Subjects will then rest for a minimum of one hour before conducting the next training session in the afternoon at step 916.

[0088] Day Three PM (3B):

[0089] Subjects are logged into the system data management portal to begin the next training session at step 906.

[0090] Subjects are then re-fitted with all FDVR equipment required for the scenario, and equipment is re-calibrated if needed and brief familiarity training and grounding exercises are conducted prior to initiating the FDVR training session at steps 908 and 910.

[0091] Subjects begin a TCCC training exercise in the FDVR training platform (2B) for 30- 45 minutes at step 912. Individual and team performances are video recorded, along with continuous biometric monitoring, and maintained in the data management portal. The system tracks, in real-time, any biometric health measures that are outside of safe parameters. An Observer Controller Trainer (OC / T) will remove any participant from the FDVR training environment demonstrating unsafe biometric parameters, and request medical evaluation as directed based on set Standard Operating Procedures (SOP) for the training.

[0092] After completion of 3B training at step 914, each subject completes a TCCC skills self-evaluation checklist of their perceived performance of the TCCC skills used during the training scenario. The subject then views the video playback of their performance, and reevaluates themselves using the same TCCC skills checklist. Each subject also receives evaluation and feedback from the OC / T observing and monitoring the training. The OC / T may be physically present, or via a virtual presence. All FDRV equipment is removed by the subject, except for the biometric monitoring device at step 920.

[0093] After completion of training and evaluations at step 922, subjects rest for 15 minutes to gain final resting biometrics. At the end of the resting period, the biometric monitoring device is removed.

[0094] Subjects participate in post-training psychological and cognitive behavioral assessments (the same as conducted prior to initiating training) at step 924, including, but not limited to Emotional Intelligence Hardiness-Resilience Gauge (HRG), Depression Anxiety Stress Scales (DASS), Moral Injury Symptom Scale - Healthcare Professionals (MISS-HP), Conner- Davidson Resilience Scale (CD-RISC) and Insomnia Severity Index (ISI).

[0095] All subjects participate in an After- Action Review (AAR) with the OC / T at step 926. A standardized list of questions will be asked, and subjects will have the opportunity for open dialog on their experiences and feeling, and give feedback, and recommendations. The AAR will be recorded and transcribed within the data management portal and reviewed by researchers for future platform training improvements.

[0096] Subjects are released from training after conducting the AAR.

[0097] Training scenarios according to embodiments of the disclosure provide real world combat training environments that challenge the medical knowledge and medical care provided by the subjects in real-time. Such scenarios are intended to create a realistic increasing stressinducing environment, with increasing medical complexity and environmental challenges. According to the embodiment described above, six training scenarios are used, however a greater or fewer number of scenarios may be used as part of a training protocol.

[0098] According to some embodiments, subjects will experience scenarios where they will not have enough resources or potentially enough training to manage the complexity of casualty needs. They will potentially experience personal injuries, team member injuries, team working dog injuries, coalition forces and enemy combatant injuries. The subjects will face moral dilemmas in how they manage medical care in a resource limiting environment. Gathered biometric data contemporaneous with such experiences is analyzed and may be used to improve resiliency of the subject in the face of such physically and morally challenging situations. According to some embodiments, the final training scenario 3B, provides a manageable training scenario to allow the subjects to complete a final training “mission” successfully.

[0099] According to some embodiments, along with OC / Ts, Performance Experts (PE), or Mental Performance Specialists (MPS), will be assessing the training environment scenarios, and present for the AAR at the end of Day three to provide any additional needed resources for the subjects, and provide one-on-one feedback if requested confidentially.

[0100] As shown throughout the drawings, like reference numerals designate like or corresponding parts. While illustrative embodiments of the present disclosure have been described and illustrated above, it should be understood that these are exemplary of the disclosure and are not to be considered as limiting. Additions, deletions, substitutions, and other modifications can be made without departing from the spirit or scope of the present disclosure. Accordingly, the present disclosure is not to be considered as limited by the foregoing description.

Claims

ClaimsWhat is claimed is:

1. A method of analyzing a biological response of a subject exposed to a high stress environment, the method comprising: providing a full dive virtual reality system, wherein the full dive virtual reality system comprises a processor connected with a haptic device, a display device, and a biometric sensor; loading a high stress training content application to the virtual reality system, wherein the content application comprises images adapted to be displayed on the display device and signals to and from the haptic device adapted to actuate the haptic device to simulate the high stress environment according to a first scenario, and wherein the content application comprises one or more high stress events; connecting the haptic device, the display device, and the biometric sensor to the subject; running the application on the system by the processor, wherein the display device shows the images to the subject and wherein signals to and from the haptic device induce sensations in the subject to simulate the high stress environment; simultaneous with the running of the application, obtaining biometric data from the biometric sensor; and correlating the biometric data with the one or more events to determine the biological response to the event.

2. The method of claim 1, further comprising: providing a mobility platform connected with the processor, wherein signals to and from the mobility platform facilitate movements of the subject relative to the platform, and wherein the step of running the application further comprises actuating the mobility platform to simulate movement of the subject within the high stress environment.

3. The method of claims 1 or 2, wherein the display device is a head-mounted virtual reality display.

4. The method of claim 3, wherein the head mounted display comprises an eye illumination light source and an eye-tracking camera connected with the processor, wherein, during the step of running the application, the processor monitors a position of the subject’s eye and adjusts the images to correspond with the eye position.

5. The method of any one of the preceding claims, wherein the haptic device comprises one or more of a glove, a watch, a wristband, a headset, a vest, or a full-body suit.

6. The method of claim 5, wherein the haptic device comprises a glove, a watch or a wristband, wherein the system further comprise a locating device associated with the haptic device and connected with the processor, wherein the processor determines an orientation of a hand of the subject and modifies the images to display a simulated image of the hand.

7. The method of claim 6, wherein the locating device comprises one of more of an inertial measurement unit connected with the hand or an optical motion capture system comprising one or more cameras and one or more reflective sensors or markers connected with the hand.

8. The method of any one of the preceding claims, wherein the first scenario comprises one or more tasks, wherein the subject performs the tasks in response to the scenario, the method further comprising: monitoring signals from the haptic device and the biometric sensor during performance of the tasks to gather performance data; and analyzing the biometric data and performance data of the user to determine parameters showing one or more of a cognitive stress response and a psychophysiological stress response.

9. The method of claim 8, further comprising the step of analyzing the biometric parameters and demonstration of skills to determine a stress level experienced by the user during the scenario.

10. The method of claim 9, further comprising terminating the application in response to a stress level above a safety threshold.

11. The method of claim 8, further comprising analyzing the biometric data in relation to the scenario and diagnosing one or more of post-traumatic stress disorder and moral injury.

12. The method of any one of the preceding claims, wherein the content application comprises a plurality of scenarios, the method further comprising the steps of: providing a second scenario to the system, the second scenario including one or more additional high stress events; running the application by the processor, wherein the display device shows the images of the second scenario to the subject and wherein signals to and from the haptic device induce sensations in the subject to simulate the high stress environment of the second scenario; simultaneous with the running of the application to display and simulate the second scenario, obtaining biometric data from the biometric sensor; correlating the biometric data from the second scenario with the one or more additional events to determine the biological response to the additional events of the second scenario; and analyzing the biological response to the event of the first scenario with the biological response to the additional event of the second scenario to assess as psychological condition of the subject relative to the high stress environment.

13. The method of any one of the preceding claims, wherein the biometric parameters include one or more of heart rate, systolic blood pressure, breathing rate, skin temperature, core body temperature, pulse oximetry, stroke volume, and cardiac output.

14. The method of claim 12, wherein the one or more additional events of the second scenario are modifications of the one or more high-stress events of the first scenario, wherein the modifications are based on the assessment.

15. The method of any one of the preceding claims, further comprising administering a psychological or medical evaluation before or after displaying the first scenario to the user.

16. A system for analyzing a biological response of a subject exposed to a high stress environment, comprising: a full dive virtual reality system, wherein the full dive virtual reality system comprises a processor connected with a haptic device, a display device, and a biometric sensor; a memory connected with the processor, the memory comprising a high stress training content application, wherein the content application comprises one or more high stress events; wherein the processor executes the content application to display images on the display device and to send and receive signals with the haptic device to actuate the haptic device to simulate the high stress environment and induce sensations in the subject to simulate the high stress environment according to a first scenario, and wherein the processor receives biometric parameters of the subject simultaneous with the first scenario and analyzes the parameters in relation to the one or more high stress events.

17. The system of claim 16, further comprising a mobility platform connected with the processor, wherein signals to and from the mobility platform facilitate movements of the subject relative to the platform, and wherein the processor actuates the mobility platform to simulate movement of the subject within the high stress environment.

18. The system of claims 16 or 17, wherein the display device is a head-mounted virtual reality display.

19. The system of claim 18, wherein the head mounted display comprises an eye illumination light source and an eye-tracking camera connected with the processor, wherein, during the step of running the application, the processor monitors a position of the subject’s eye and adjusts the images to correspond with the eye position.

20. The system of any one of claims 16 to 19, further comprising a hand locating device associated with the haptic device and connected with the processor, wherein the processordetermines an orientation of a hand of the subject and modifies the images to display a simulated image of the hand.

Images