Hybrid interactive virtual environment system and method of use thereof

Abstract

A hybrid interactive virtual environment (HiVE) system and method of using the same are provided, which are configured for clinical education and training. The system includes: an immersive room, which is a 6-sided cave automatic virtual environment (CAVE) platform and is configured as a fully immersive interactive hybrid classroom; an image generator configured to process input content and render the input content in two-dimensional (2D) mode or three-dimensional (3D) mode to produce interactive content for clinical education and training; a projection warping system for performing projection warping and blending processing to produce a projection warped display image; and a video processing system for processing the projection warped display image to produce a frame-synchronized display output; wherein the immersive room includes: a surround sound system for generating audio output to the interactive content; a motion capture system for producing tracking information of users inside the immersive room and sending the tracking information to the interactive content; and a projector for projecting the frame-synchronized display output from the video processing system to generate a 3D interactive image inside the immersive room for clinical education and training, such that users in the immersive room can view and hear the 3D image and interact with the 3D image and / or with each other for clinical education and training.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 456,323, filed March 31, 2023. The disclosure of this provisional application is incorporated herein by reference in its entirety. Technical Field

[0003] This application generally relates to Cave Automatic Virtual Environment (CAVE) technology, and more particularly to Hybrid Immersive Virtual Environment (HiVE) systems employing fully immersive CAVE and methods of using them for practical and collaborative learning. Background Technology

[0004] A CAVE (Automatic Virtual Environment), also known as a recursive abbreviation CAVE, is an immersive virtual reality environment in which a projector is pointed between three to six walls of a room-sized cube. CAVEs are typically video theaters located within a larger room. The walls of a CAVE are usually composed of rear-projection screens or large LED display screens. The floor of a CAVE can be a downward-projecting screen, a bottom-projecting screen, or a flat panel display. The projection system is at a very high resolution because very small pixel sizes are required for close-up viewing to maintain realistic visuals. Users wear 3D (3D) glasses inside the CAVE to view 3D graphics generated by the CAVE. Users of a CAVE can see objects that appear to float in mid-air and can walk around them to get a sense of what they would look like in reality, allowing them to view and / or study them from various angles. The movement of the CAVE user is tracked by sensors typically attached to the 3D glasses, and the video is constantly adjusted to maintain the viewer's perspective. Computers control this aspect of the CAVE, as well as its audio aspects. Multiple speakers are typically placed at multiple angles within a CAVE to provide 3D sound to complement the 3D video. To create images that are not distorted or misaligned, the displays and sensors must be calibrated. The calibration process depends on the motion capture technology used. Optical or inertial acoustic systems only require configuring the zero point and coordinate axes used by the tracking system. Calibration of electromagnetic sensors (such as those used in the first CAVE) is more complex. In this case, people will wear special glasses required to view 3D images.

[0005] Current practice of clinical education and training mainly involves clinical rotations, which are highly restrictive in terms of logistics and teaching. The main pre-clinical training and placement supplemental material contains video-recorded virtual hospital visits. However, it cannot facilitate the development of students’ spatial sense needed to establish safe and efficient ways of working in complex professional environments involving various clinical equipment and hazards. Video-recorded virtual visits deprive students of hands-on experience due to the lack of interactive components. Video-recorded virtual visits deprive students of interaction with each other since they are not required to interact and communicate with each other during the virtual visits.

[0006] Today, CAVEs are used for research in a range of disciplines, but not for clinical education and training.

[0007] On the other hand, the increasing number of global childhood cancer diagnoses highlights the urgent need for more effective, child-oriented treatment options to address the unique challenges of treating cancer in children.

[0008] Conventional cancer treatments such as chemotherapy and radiotherapy often come with severe side effects that can affect children’s development, quality of life, and long-term health, and thus have a negative impact on cancer treatment. In addition, the emotional and psychological impact of cancer treatment on young patients and their families requires innovative care strategies that support physical, emotional, and mental health simultaneously to avoid adverse psychological effects.

[0009] Children’s different reactions to cancer treatment highlight the importance of personalized medicine and the need for personalized care. By taking into account each pediatric patient’s individual genetic makeup, cancer type, psychological needs, and overall health, a tailored treatment plan can improve outcomes. In radiotherapy, repeated general anesthesia (GA) is required due to the need for complete immobilization of pediatric patients during treatment, which carries the risk of developmental and morbid complications and exacerbates emotional distress for patients and families. There is a clear gap in comprehensive support services for children receiving cancer treatment and their families, requiring innovative solutions. These services should not only meet medical needs but also psychological, educational, and social support needed throughout the treatment process.

[0010] Therefore, there is also an urgent need for innovative care in pediatric radiotherapy, and HiVE is an excellent platform for educational workshops and rehearsals that provide psychological, educational, and social support for children with cancer and their caregivers. SUMMARY

[0011] It is an object of the present invention to provide a hybrid immersive virtual environment (HiVE) system for practicing and collaborative learning with a fully immersive CAVE, in particular for clinical education and training.

[0012] According to an aspect of the present application, there is provided a hybrid interactive virtual environment (HiVE) system configured for clinical education and training, comprising: an immersive room which is a 6-sided cave automatic virtual environment (CAVE) platform and configured as a fully immersive interactive hybrid classroom; an image generator configured to process input content and render the input content in two-dimensional (2D) mode or three-dimensional (3D) mode to produce interactive content for clinical education and training; a projection warping system for performing projection warping and blending processing to produce a projection warped display image; and a video processing system for processing the projection warped display image to produce a frame-synchronized display output; wherein the immersive room comprises: a surround sound system for producing audio output to the interactive content; a motion capture system for producing tracking information of users inside the immersive room and sending the tracking information to the interactive content; and a projector for projecting the frame-synchronized display output from the video processing system to generate 3D interactive images inside the immersive room for clinical education and training, such that users in the immersive room can view and hear the 3D images and interact with the 3D images and / or with each other for clinical education and training.

[0013] Preferably, in the processing of projection warping performed by the projection warping system, the input content on the image generator is rendered and arranged based on the physical layout of the projection surfaces of the projectors, and then warped based on the rendered display layout, and then further processed, rearranged and distributed to each projector; wherein when running in 2D mode with projection warping and blending, all pixels assigned to each projector are displayed directly by the projector, and when running in 3D mode, the image generator renders images for both the left and right eyes of the user and arranges as final output to the video processing system in a side-by-side manner.

[0014] Preferably, in the immersive room, the projectors for the walls are installed inside the space of the ceiling area as front, back, left and right projectors, a set of custom projector cover structures are installed around the projectors to hide the projectors while allowing light from the ceiling projectors to project onto them; the projectors for the ceiling are installed outside the space behind the left and right walls, an opening is carved on the wall for each projector to match the shape of the projected light cone to allow light to pass through while minimizing the impact on immersion; and the projectors for the floor are installed on top of the ceiling wall and project onto the floor through the openings carved on the ceiling in the same way as the ceiling projectors.

[0015] Preferably, a plurality of calibration cameras with fisheye lenses are strategically placed inside the immersive room and connected to the image generator to facilitate projection warping and blending.

[0016] Preferably, to increase the interactivity of the system, the motion capture system comprises a plurality of motion capture or sensing devices using optical or infrared light mechanisms, installed at strategic locations inside the immersive room under or near the ceiling of the immersive room.

[0017] Preferably, dry walls are used to create the immersive room inside the structural room, separating the structural room into the immersive room for fully immersive interactive projection system and additional room or space for storing, hiding the related hardware equipment or equipment or for any other purpose, wherein each side wall of the dry walls at the left and right sides of the immersive room is composed of two parts: a vertical lower segment and an upper segment that is tilted inward, wherein the upper segment of the side wall is tilted inward at an angle that closely matches the light path of the bottom of the projection cone of the projector, so that the immersive room is 6-sided, including the ceiling, the floor, two vertical segments and two upper segments that are tilted inward.

[0018] Preferably, the system further comprises a camera that takes 360-degree photos of the immersive room during clinical education and training, and the projector is capable of displaying 360-degree images obtained in different clinical environments.

[0019] Preferably, the system further comprises goggles and a handheld device to be used by the user to interact with the 3D interactive images and with each other during clinical education and training.

[0020] Preferably, the system can switch between different teaching modes to meet different learning objectives in teaching sessions during clinical education and training, wherein in one of the teaching modes, the immersive room is used as a regular classroom.

[0021] Preferably, real physical objects can be placed in the immersive room, and the system integrates the virtual objects of the 3D image and the real physical objects together to facilitate realistic simulation.

[0022] According to another aspect of the present application, there is provided a method for creating a hybrid interactive virtual environment (HiVE) and configured for clinical education and training, comprising: forming an immersive room through a 6-sided cave automatic virtual environment (CAVE) platform to configure the immersive room as a fully immersive interactive hybrid classroom; inputting or saving content for clinical education and training into an image generator positioned in the immersive room, wherein the image generator is configured to process the inputted or saved content and render the content in two-dimensional (2D) mode or three-dimensional (3D) mode to generate interactive content; performing projection warping and blending processing through a projection warping system to generate a projection warped display image; and processing the projection warped display image through a video processing system to generate a frame-synchronized display output, wherein the immersive room comprises: a surround sound system for producing audio output to the interactive content; a motion capture system for producing tracking information of users inside the immersive room and sending the tracking information to the interactive content; and a projector for projecting the frame-synchronized display output from the video processing system to generate 3D interactive images inside the immersive room for clinical education and training, such that users in the immersive room can view and hear the 3D images and interact with the 3D images and / or with each other for clinical education and training.

[0023] Preferably, during the projection warping by the projection warping system, the input content on the image generator is rendered and arranged based on the physical layout of the projection surface of the projector and then warped based on the rendered display layout and then further processed, rearranged and distributed to each projector; wherein when running in 2D mode with projection warping and blending, the total pixels assigned to each projector are displayed directly by the projector and when running in 3D mode, the image generator renders images for both left and right eyes of the user and arranged in side-by-side manner as the final output to send to the video processing system.

[0024] Preferably, to increase the interactivity of the system, the motion capture system comprises a plurality of motion capture or sensing devices using optical or infrared light mechanism, which are installed at strategic locations inside the immersive room under or near the ceiling of the immersive room.

[0025] Preferably, the method further comprises taking 360-degree photos of the immersive room using a camera during the clinical education and training, and displaying the 360-degree images obtained by the projector in different clinical environments.

[0026] Preferably, the method further comprises integrating real physical objects and virtual objects of the 3D images in the immersive room to facilitate realistic simulation during the clinical education and training.

[0027] Hybrid Interactive Virtual Environment (HiVE) is a large-scale XR hybrid classroom with fully immersive CAVE technology to allow customized clinical simulations, clinical skill execution, collaborative education (student numbers between 20 to 40), human-to-equipment and human-to-human interaction for collaborative learning in clinical education and training.

[0028] The present system of a hybrid interactive virtual environment for clinical education and training is an innovative experiential learning environment. In order to effectively translate knowledge into clinical practice, students must gain a thorough understanding through practical application. The system utilizes Cave Automatic Virtual Environment (CAVE) technology to provide students with a highly immersive and realistic environment that replicates the actual clinical setting combined with realistic representation of equipment and space. Through the use of this innovative technology, students can practice skills and are better prepared for clinical training in the hospital. This technology provides a safe and "fault-tolerant" environment for students to engage in different scenarios such as routine and emergency situations. The CAVE technology not only creates a realistic clinical environment, but also facilitates peer learning by enabling up to 30-40 students to interact with each other and with the virtual environment. Most importantly, the vivid visualization and interactive elements in the CAVE provide a stress-free platform for collaborative learning. For example, students can practice their skills together in groups, which is not possible in a real clinical environment. All of this enables students in the healthcare professions to be better and more thoroughly prepared for future clinical training. BRIEF DESCRIPTION OF DRAWINGS

[0029] Certain examples are illustrated in the above-described drawings and are described in detail below. In describing these examples, like or similar elements are identified with like or similar reference numbers. The drawings are not necessarily to scale and certain features and certain views of the drawings can be shown exaggerated in scale or in schematic for purposes of clarity and / or conciseness.

[0030] Figures 1A-1C Some photos showing students learning in an example hybrid CAVE classroom.

[0031] Figure 2 The main components of a fully immersive interactive projection system are shown.

[0032] Figures 3A-3B The 2D and 3D mode projection warping processes are shown accordingly.

[0033] Figure 4 A 3D model diagram showing the projectors of the immersive projection system and the projection cones inside the immersive room.

[0034] Figure 5 The motion capture cameras of a motion capture system installed near the ceiling area are shown.

[0035] Figure 6 The speaker is shown as part of a surround sound system integrated into the dry wall of an immersive room.

[0036] Figure 7 This illustrates the structural concept of the interior of an immersive room.

[0037] Figures 8A-8B A conceptual diagram of the projector cover structure for an immersive room is shown.

[0038] Figure 9 A projector cover for an immersive projection system on which an image is projected is shown.

[0039] Figure 10 An image is shown demonstrating the sloping side wall of the immersive room.

[0040] Figure 11 The image shows a wall coated with a gray-tinted surface paint, in which an air ventilation panel painted the same color is embedded.

[0041] Figures 12A-12B This is a design drawing of a complementary visual aid system (left) with a hidden control panel and projection area (right) in an immersive room.

[0042] Figure 13A This invention is shown to be used in a regular classroom setting, while Figure 13B This invention is shown to be used as a classroom in a virtual environment.

[0043] Figures 14A-14C Examples of augmented reality (AR), virtual reality (VR), and mixed reality (MR) are shown accordingly.

[0044] Figures 15A-15C Accordingly, examples of actual learning through this system are shown.

[0045] Figure 16 The hybrid learning mode of HiVE is shown.

[0046] Figure 17A (left) and Figure 17B (Right) An immersive projection that corresponds to the actual clinical environment and a 360-degree image taken in the same environment.

[0047] Figure 18 This demonstrates the integration of Mixed Reality (MR) with HiVE teaching.

[0048] Figure 19 This demonstrates a tutorial classroom environment in HiVE that can be conducted within the same teaching session. Detailed Implementation

[0049] The descriptors "first," "second," "third," etc. are used herein to denote multiple elements or components that can be individually referred to. Unless otherwise stated or understood based on its usage context, such descriptors are not intended to impart any meaning of priority or temporal order, but merely serve as labels for referring to multiple elements or components, respectively, for ease of understanding the disclosed examples. In some examples, the descriptor "first" can be used to refer to an element in the detailed description, while the same element can be referred to in the claims with a different descriptor such as "second" or "third." In such cases, it should be understood that the above descriptors are merely for ease of referencing multiple elements or components.

[0050] I. Virtual Environment Virtual Environment (HiVE)

[0051] HiVE (Hybrid Immersive Virtual Environment) is the world's first large-scale X-Reality hybrid classroom developed by the Hong Kong Polytechnic University. The HiVE employs fully immersive Cave Automatic Virtual Environment (CAVE) technology for practical and collaborative learning, and has developed it into a 6-sided CAVE platform. The 6-sided CAVE platform can create immersive 2D or 3D environments to help students visualize abstract concepts and experience the limitless possibilities of the digital world. In addition, this 6-sided CAVE platform provides virtual training environments that enable students to experience and practice skills related to work in realistic scenarios that are difficult to access in the real world, such as fire outbreaks, medical procedures, and airplane control rooms. The hybrid CAVE also allows teachers to seamlessly switch between face-to-face and immersive teaching, combining virtual technology with conventional learning to improve the learning experience of students.

[0052] The concept of the XR hybrid classroom includes, for example, how to combine a teaching classroom and a CAVE system into a hybrid teaching facility. The fully immersive CAVE technology includes, for example, how to design and install a 6-sided projection VR projection in a classroom environment.

[0053] Most prior art CAVEs only have 4 or 5 walls, and therefore cannot give a fully immersive experience. In addition, conventional CAVEs are limited to independent applications that only belong to virtual technology applications, so when using conventional CAVEs, teachers find it difficult to switch between face-to-face and immersive teaching in a classroom environment.

[0054] The present HiVE is a fully immersive CAVE that can give users a fully immersive experience. The HiVE can create a hybrid CAVE classroom and provide X-Reality simulation, enabling seamless conversion of teaching modes, and enabling users to interact with real or digital objects simultaneously in a CAVE environment; in addition, the HiVE can create a multi-CAVE platform and can provide real-time interaction between multiple CAVEs.

[0055] Figure 1A Some photos of students learning in an example hybrid CAVE classroom, i.e., the immersive room 120 of the present application, are shown. Figure 1A A group photo of students in the immersive room 120 is shown; Figure 1B An ongoing class is shown, where students are listening to a class and watching a 3D image of the class in the immersive room 120; and Figure 1C Interaction of a student with a teacher during a class in the immersive room 120 is shown. Understanding a theoretical-based conceptual subject matter can be extremely challenging for many students. The present hybrid CAVE classroom is a 6-sided CAVE-based technology that creates a very real immersive environment to assist students in visualizing abstract concepts and novel ideas that are not possible to display in a real environment. In addition, this technology allows students to participate in experiences such as fire explosions, medical procedures, and airplane control rooms that they can have difficulty accessing in the real world. Theoretical-based knowledge classes can be very difficult for students to understand. With the help of the new 6-sided CAVE technology, students can visualize abstract concepts that cannot be depicted in the real world. Additionally, this technology enables students to experience and participate in realistic scenarios such as fire explosions, medical procedures, and airplane control rooms that are difficult to access in the real world.

[0056] Accordingly, the present HiVE is a fully immersive CAVE, which is a very real immersive environment, and can assist students in visualizing abstract concepts and novel ideas that are not possible to display in a real environment.

[0057] II. Design and system of the HiVE as a fully immersive interactive hybrid classroom

[0058] The HiVE can be configured as a fully immersive interactive hybrid classroom, which is an example of the inventive 6-sided CAVE platform.

[0059] The fully immersive interactive hybrid classroom is a fully immersive interactive projection system. The fully immersive interactive projection system can include an image generator 110, a projection warping system 130, a video processing system 140, and an immersive room 120. An immersive projection system or projector 125, a motion capture system 122, a surround sound system 121, a supplemental visual aid system 126, etc. can be provided in the immersive room 120. The fully immersive interactive projection system can be connected to an external audiovisual system 150 for inputting or outputting audiovisual signals, and / or to an external control interface 160 for external control by a human or a computer.

[0060] Figure 2The main components of the fully immersive interactive projection system 100 including the image generator 110 for projecting the fully immersive interactive image are shown. The details of each component of the system will be elaborated in the following sections.

[0061] The image generator 110 can be implemented by a computing workstation having at least a central processing unit (CPU), a memory, a storage system, a set of graphics processing units (GPU), a power supply unit (PSU), and a set of I / O interfaces. The image generator can run on a compatible operating system (OS) to process input signals, execute commands written in applications saved to the storage system, and render display and sound output for the fully immersive interactive projection system.

[0062] A configurable launcher application with a graphical user interface is available on the image generator to interface with connected hardware devices and launch commands in raw or executable format. The image generator 110 can be configured to render content in 2D or 3D mode. When set to render content in 2D mode 200, the launcher application sends a command to set the projectors 125 to operate in 2D mode and another command to the projection warping system 130 to use the 2D mode mapping profile (which can also be used for blending); when set to render content in 3D mode 300, the launcher application sends a command to the projection warping system 130 to set the projectors to operate in 3D mode and another command (also used for blending) to use the 3D mode mapping profile.

[0063] The projection warping system performs the processing of projection warping and blending. Figures 3A-3B The 2D mode and 3D mode projection warping processes are shown accordingly. During projection warping 220 and blending, a two-step process is employed to produce the desired immersive visual effect. First, in step 210, the interactive content on the image generator is rendered and arranged based on the physical layout of the projection surface from the content rendered by the image generator. Here, the projection surface is defined as a continuous surface facing the same direction in the immersive room. The projection warping system process 220 is then completed based on the rendered display layout, which is then further processed, rearranged, and distributed to each projector. When running in 2D mode 200 with projection warping and blending, the total pixels assigned to each projector will be displayed directly by the projector. When running in 3D mode 200, the image generator will render images for both the left and right eyes and arrange as final output to the video processor in a side-by-side manner.

[0064] Interactive content executing on the image generator 110 can use motion capture information provided by the motion capture system 122 as user input to update the interactive content. As a viewer fitted with a mobile tracker moves around the immersive room 120, the interactive content can be rendered from the viewer's perspective to provide a first-person perspective rendering customized for the viewer. Optionally, other input devices such as controllers and haptic devices can also be connected to the image generator 110 to serve as additional I / O devices to add additional interactive elements to the system 100.

[0065] Figure 4 A 3D model diagram showing the projectors 125 of the immersive projection system 100 and the projection cones inside the immersive room 120 of the immersive projection system 100.

[0066] The immersive projection system 100 features a combination of ultra-short throw (UST) (i.e., projector throw ratio < 0.5: 1) and short throw (ST) (i.e., projector throw ratio between 0.5-1: 1) projectors 125 strategically placed outside the viewable space of the system operating in a front projection manner. Projectors used to provide image projection for the front, back, left (171, 172), right (173, 174), and ceiling 175 of the immersive room 120 of the immersive projection system 100. UST projectors or projectors equipped with UST lenses are used to minimize the required projection distance between the projector and the projection surface and maximize the effective space for the viewer to move inside the system without encountering the projection cone and shadow of the projector. For projectors used to supply image projection for the floor 176, ST projectors or projectors equipped with ST lenses are used. There is at least 1 projector for each projection face (front, back, left, right, ceiling, floor). When more than 1 projector is used for any given side of the system, the projected images output from each of the projectors should have some overlap so that effective projection warping and blending can be achieved to seamlessly merge the projected images of these projectors.

[0067] Projectors 125 for the walls (i.e., front, back, left, and right projectors) are installed inside the space of the ceiling area, a set of custom projector cover structures 500 are installed around the projectors and lens bodies to hide the projectors 125 while allowing light from the ceiling projectors to project onto them. Projectors for the ceiling are installed behind the left and right walls outside the space, an opening matching the shape of the projected projection cone is carved on the wall for each projector to allow light to pass through while minimizing the impact on the immersion. Projectors for the floor are installed on top of the ceiling wall and project onto the floor through openings carved on the ceiling in the same way as the ceiling projectors.

[0068] The projector 125 is plugged into a video processing system 140 that interfaces with the image generator 110 through a layer of projection warping system 130 to generate frame synchronized image signals to the projector. The projector is capable of operating in a normal 2D mode or a 3D stereoscopic mode. When operating in the 2D mode 200, the user is able to see a clear image with naked eyes. When operating in the 3D mode 300, the projector displays images for the left or right eye in an alternating fashion, and the user needs to wear a pair of active stereoscopic 3D glasses that are synchronized with the display frequency of the projector to separate the alternating images in order to see a clear stereoscopic image. During the 3D mode, a 3D synchronization signal is needed to synchronize the active stereoscopic 3D glasses with the display images from the projector. Such 3D synchronization signal can be generated by the video processing system when the video processing system processes and creates the 3D images, or can be generated from any one of the projectors connected to the video processing system when the projectors themselves process and create the 3D images.

[0069] A plurality of calibration cameras 124 with fisheye lenses are strategically placed inside the system and connected to the image generator for projection warping and blending. To facilitate projection warping and blending, a camera-based calibration process by the projection warping and blending system is needed that utilizes a predefined 3D model of the room to compute the UV mapping transformation of the real-world images captured by the calibration cameras. Projection warping and blending can then be implemented by applying the computed UV mapping to the render buffers of the GPU of the image generator. The UV mapping is computed for both 2D and 3D modes, and the user can switch between 2D and 3D projection warping and blending by sending corresponding commands on the image generator.

[0070] Figure 5 Motion capture cameras of a motion capture system 122 are shown mounted near the ceiling 175 of the immersive room 120.

[0071] To increase the interactivity of the system, as an example of the motion capture system 122, a plurality of motion capture / sensing devices using optical or infrared light mechanisms are mounted inside the system in strategic locations near the ceiling 175 of the immersive room 120. Users / objects fitted with retro-reflective markers forming predefined shapes or battery-powered trackers with pre-registered hardware identification numbers can be recognized by the motion capture / sensing devices, allowing the motion capture system to resolve the positions / movements of the tracked bodies (motion capture information). The motion capture information is streamed out using a network or a related universal protocol for use as input data by the fully immersive interactive projection system to render interactive content on the display system.

[0072] Figure 6Speakers of a surround sound system 121 integrated into the dry walls of the immersive room 120 are shown.

[0073] The surround sound system 121 is installed outside the display area to provide immersive spatial sound inside the system 100. The surround sound system 121 includes at least one subwoofer and a set of speakers for the front, left, right and back of the system, which are placed in strategic locations around the system to enhance the produced surround sound effect.

[0074] The walls on which the projected images will be displayed are sized just enough to fit the speakers of the surround sound system, so that the sound produced by the speakers can be projected to the carved rectangular openings inside the immersive room 120. The openings on the walls are covered with perforated panels of the same color as the walls to minimize their impact on the system's immersion, while allowing enough sound to pass through.

[0075] Figure 7 A construction concept of the "room inside room" design of the immersive room 120 is shown, and is a top view of the design of the immersive room 120. Figure 7 The sizes of the above markings are for the example immersive room design, and can vary according to actual requirements and practical applications when implementing the present invention.

[0076] The immersive room 120 is created inside the original structure room 400, for example, by using dry walls. This partitions the original structure room into the immersive room 120 for the fully immersive interactive projection system 100 and additional room(s) 401 or space(s) for storing (e.g. housing the image generators), hiding the relevant hardware equipment / equipment or for any other purpose, and / or leaving a void 402.

[0077] As described in the immersive projection system section, the projections 125 are installed behind the dry walls, under or on the ceiling 175 and on the floor, or installed inside the immersive room surrounding the ceiling area hidden behind custom covers (front, back, left and right wall projectors).

[0078] Figure 8A A conceptual diagram of the projector cover structure 500 of the immersive room 120 is shown, Figure 8B A construction concept of the projector cover structure 500 installed on the ceiling 175 of the immersive room is shown, and Figure 9 A projector cover 500 of the immersive projection system 100 with an image 125 projected onto it is shown. The custom projector cover structure is made of rigid material, such as fiberglass, and has curved edges to minimize the obstruction of the projected light from the ceiling projectors. There are openings on the projector cover to allow the projected light to pass through.

[0079] Figure 10 An image showing the side walls of the immersive room 120 tilted open is shown.

[0080] The side walls (i.e. the dry walls on the left and right sides of the immersive room) are composed of two segments - a lower segment 172 that is vertical and an upper segment 171 that is tilted inwards. The ceiling projectors are placed behind the walls, at a height higher than the normal height of a person, and project towards the ceiling. For each projector, an opening with the shape of the projector’s projection cone is cut on the lower segment 172 of the dry wall to allow the projected light to pass through. As shown in the middle, the upper segment 171 of the dry wall is tilted inwards at an angle that closely matches the light path of the bottom of the projector’s projection cone, at a suitable distance above the cut opening. Figure 4

[0081] The entrance to the immersive room is equipped with a hidden door, where the surface material facing the system’s interior matches the material used inside the immersive room 120 and is painted with the same surface paint.

[0082] The interior of the immersive room 120 is painted with a specially selected grey surface paint to minimize internal reflection of the projector light while maintaining the vividness of the projected colors.

[0083] Figure 11 An image showing the walls of the immersive room 120 painted grey as the immersive room’s colored surface, provided with embedded air ventilation panels (painted the same color) in the upper segment 171, where a set of projectors 125 is installed below the ceiling 175 of the immersive room.

[0084] Other equipment or equipment necessary to support the normal use or related management requirements of the system are installed behind the dry walls, embedded in the dry walls in a similar manner to the speakers of the surround sound system 121, or placed inside the immersive room 120 in locations that are least distracting to the viewer’s attention. Examples of such equipment / equipment include, but are not limited to: air ventilation systems, light panels, carbon dioxide detectors, sprinklers, etc. The appearance of the equipment can be set to match the paint color of the immersive room without sacrificing the functionality of the equipment.

[0085] Figures 12A-12B is a design drawing of a supplemental visual aid system (left) with a hidden control panel and projection area in the immersive room 120, and shows a construction concept of a vertical cross-sectional view of the immersive room 120 inside the original structure room 400.

[0086] ​A further 2D projector adapted to the motorized ceiling projector lift is hidden behind the projector cover 500 under the drywall ceiling 175 and the bottom of the motorized ceiling projector lift forms part of the ceiling drywall structure in the immersive room 120. It is activated by pressing a button on the control panel hidden behind the cabinet door inside the system. External audiovisual equipment can be connected to this supplemental visual aid system to provide the viewer with additional audiovisual content rendered in 2D.

[0087] As described above, important features of the present fully immersive interactive hybrid classroom can include, for example:

[0088] (1) It is a full front projection fully immersive projection system.

[0089] (2) The method can build an immersive room inside a large room to hide the related hardware system and reduce the construction cost.

[0090] (3) The shape of the room includes 6 sides and a slanted upper section side wall, allowing more simultaneous viewers and having more space inside to place equipment to create a mixed reality environment.

[0091] (4) It is a room inside room design.

[0092] (5) Projection warping and display screen transformation can be performed.

[0093] (6) It can meet the air ventilation and regulatory restrictions of a fully immersive space and has better air ventilation to support extended usage sessions.

[0094] (7) Its highly complex system structure of projection warping uses side-by-side passive stereo.

[0095] (8) It achieves a higher ratio of system footprint to available immersive space (requires less space) and has higher scalability,

[0096] Applications of the present invention can include hybrid mode classroom learning with mixed reality elements, multi-user simulation training, and academic institution classrooms, particularly for clinical education and training.

[0097] III. Hybrid Interactive Virtual Environment System for Clinical Education and Training with a Fully Immersive CAVE

[0098] Hybrid Interactive Virtual Environment (HiVE) is a large scale XR hybrid classroom with a fully immersive CAVE technology to allow customized clinical simulations, clinical skill execution, collaborative education (student numbers between 20 to 40), human-to-equipment and human-to-human interaction for collaborative learning in clinical education and training.

[0099] The mixed interactive virtual environment system for clinical education and training of the present invention employs a fully immersive CAVE that can take 360 degree photos and calibrate imaging equipment for media projection on the HiVE. The system integrates virtual and physical objects for realistic simulation and can select available software for scene design for simulation. The immersive room of the system is a fully immersive 6-sided CAVE platform and is capable of displaying 360 degree images obtained in different clinical environments, which can be configured as a hybrid classroom that enables XR (Extended Reality). The system enables discipline-specific training installations of large-scale CAVEs and can save high logistical and administrative costs of in-person and immersive teaching in multiple physical locations, including on-campus and off-campus (e.g., hospitals).

[0100] The present system of mixed interactive virtual environment for clinical education and training is an innovative experiential learning environment. To effectively translate knowledge into clinical practice, students must gain a thorough understanding through practical application. The system utilizes Cave Automatic Virtual Environment (CAVE) technology to provide students with a highly immersive and realistic environment that replicates actual clinical settings combined with realistic equipment and spatial representations. Through the use of this innovative technology, students can practice skills and are better prepared for clinical training in hospitals. This technology provides a safe and "fault-tolerant" environment for students to engage in different scenarios such as daily and emergency situations. The CAVE technology not only creates realistic clinical environments, but also facilitates peer learning by enabling up to 30-40 students to interact with each other and with the virtual environment. Most importantly, the vivid visualizations and interactive elements in the CAVE provide a stress-free platform for collaborative learning. For example, students can practice their skills together in groups, which is not possible in real clinical environments. All of this enables students in health care professions to be better and more fully prepared for future clinical training.

[0101] The present system of mixed interactive virtual environment for clinical education and training includes a hybrid classroom. Space is a valuable resource for universities. Combining a teaching classroom and a CAVE system into a hybrid teaching facility can save installation space. More importantly, this hybrid classroom allows teachers to easily switch between in-person teaching and immersive teaching in a classroom environment to mix virtual technology with traditional learning. This hybrid teaching facility enables teachers to perform seamless transitions between delivery modes between in-person lectures and immersive experiences to achieve the goal of mixing virtual technology with conventional learning.

[0102] Clinical education often requires space to accommodate various equipment and large class teaching and skill demonstration. However, space is a precious resource for a university. A multi-functional classroom that can conduct both traditional teaching and hands-on practice will save a lot of space. In addition, this classroom allows instant switching between different teaching modes to meet different learning objectives in a teaching session.

[0103] For example, Figure 13A The immersive room 120 of the present application is shown to be used as a regular class in which students sit on chairs placed in rows and listen to a teacher teaching through a projection screen. Figure 13B The immersive room 120 of the present application is shown to be used as a class in a virtual environment in which students and teachers stand in a virtual environment generated by the system of the present application, which shows a stereoscopic 3D image including clouds in the sky and buildings on the earth. In Figure 13A And Figure 13B In the immersive room 120, the vertical lower sections 172, 174 and the inwardly inclined upper sections 171, 173 are arranged in the left and right sides of the immersive room, respectively. In Figure 13B In the immersive room 120, the stereoscopic 3D image is created by the image generator 110 of the present application, which can be configured to process input content and render the input content in 2D mode or 3D mode to generate interactive content for clinical education and training. As Figure 2 , Figure 3A And Figure 3B The projection warping system shown in The video processing system 140 processes the projection warped display image to generate a frame-synchronized display output. The immersive room 120 includes a surround sound system 121 for generating audio output to the interactive content, a motion capture system 122 for generating tracking information of users inside the immersive room and sending the tracking information to the interactive content, and a projector 125 for projecting the frame-synchronized display output from the video processing system 140 to generate a 3D interactive image inside the immersive room for clinical education and training, so that users in the immersive room can watch and listen to the 3D image and interact with the 3D image and / or each other for clinical education and training.

[0104] The present system for mixed interactive virtual environments for clinical education and training utilizes extended reality, which enables advanced visualizations. Visualizing concepts of human anatomy, physiology, and pathology is essential in clinical education as it facilitates the translation of theoretical knowledge into clinical application, bridging the gap between theory and practice. The 6-sided CAVE technology suite enables extended reality (XR) demonstrations, combining augmented (AR) reality, virtual (VR) reality, and mixed (MR) reality to provide richer visualization details. For example, students can interact with a MR-based mannequin to solidify their spatial understanding of anatomical structures and pathologies. Students can also manipulate actual equipment in the same suite to practice their skills.

[0105] XR refers to the family that includes AR, VR. VR is only able to allow users to visualize and interact with digital objects in a virtual environment without feeling touch or interaction with real objects. To enhance the feeling and interaction, the present system 100 for mixed interactive virtual environments for clinical education and training successfully provides MR applications to the CAVE using a holographic lens (not shown) in the CAVE environment. Articulated hand tracking input of the MR device enables users to interact with real or digital objects simultaneously in an immersive environment.

[0106] Figures 14A-14C Examples of AR, VR, and MR are accordingly shown. In the AR example shown in Figure 14A the AR example shown in, a virtual patient 601 is lying on a virtual medical bed in the immersive room 120, a 3D image is projected to show the patient and, if desired, a partial anatomy of its body, and students watch the patient and its body and listen to a teacher teaching a course. In the VR example shown in Figure 14B the immersive room 120, its 3D image shows a virtual patient 601 being treated by a medical instrument 602, such as a linear accelerator, while students stand around the 3D image listening to a teacher lecturing, where students and teachers can interact with digital objects in the virtual environment without feeling touch or interaction with real objects. In the MR example shown in Figure 14C the MR example shown in, students and teachers stand within (i.e., immersed in) the 3D image in the immersive room 120, listening to a course and wearing goggles 603 to watch 3D images interacting with the image and / or each other.

[0107] Thus, the mixed CAVE classroom of the present invention achieves the goal of combining virtual technology with conventional learning, facilitates the purpose of hands-on practical learning, and provides a virtual environment for collaborative learning. This enables students from all over the world to demonstrate and even interact with each other.

[0108] The hybrid interactive virtual environment presentation system 100 of the present invention for clinical education and training facilitates both hands-on learning and collaborative learning. "Knowledge without practice is worthless." VR can create interactive hands-on experiences in simulated environments. Figures 15A-15C An example of hands-on learning with the present system is shown. In Figure 15A the student is wearing goggles 603 and holding a handheld device 605 (such as a game controller) in the immersive room 120, the image shows virtual buttons corresponding to real buttons on the handheld device 605, and the teacher is teaching the student to use the handheld device to interact with the 3D image. In Figure 15B a plurality of students are standing around the teacher, listening to the teacher teach a lesson in the immersive room 120, each of them wearing goggles 603 accordingly. For example, in Figure 15C some students are standing inside the 3D image listening to the lesson and watching the 3D image of the street scene.

[0109] This safe, risk-free, and practical environment helps students develop professional skills by doing as they learn. The VR hybrid classroom of the present invention also supports real-time multi-CAVE collaboration between CAVE systems. This enables students from all over the world to present and even interact with each other.

[0110] In addition, Figure 16 A hybrid learning mode of the HiVE system 100 of the present invention for clinical education and learning is shown. Projected on the screen 604 (or wall) is a set of computed tomography (CT) images and 3D volume rendering of a cancer patient 601 represented by a mannequin lying on a couch in a simulated treatment room and applied with appropriate accessories. The software displays the CT images of the patient to allow students to visualize the relevant 3D anatomy via VR. The students stand inside the 3D image listening to the lesson and watching the 3D image. In this embodiment, both 2D mode and 3D mode images are projected by the present system. In Figure 16 the vertical lower sections 172, 174 and the inwardly inclined upper sections 171, 173 are arranged on the left and right sides of the immersive room 120, respectively, and a plurality of projectors 125 are arranged below or above the ceiling 175 of the immersive room 120.

[0111] Figure 17A represents a real clinical environment, while Figure 17B represents an immersive projection of 360-degree images captured in the same clinical environment projected by the present system. In Figure 17A the room 701 has a linear accelerator 702 and a carpet 703. In Figure 17B when the present system 100 projects 360-degree or 3D images, a virtual linear accelerator 602 and a carpet are displayed in the immersive room 120, and correspond to Figure 17AAlmost identical to the one in the picture, in order to imitate Figure 17A The actual clinical environment setup; in addition, in Figure 17B The image shows a virtual patient 601 lying in bed undergoing clinical education and training in an immersive room 120.

[0112] Figure 17B The immersive room 120 is constructed according to the concept of the present invention and can be employed as follows: Figure 7 The "room within a room" design shown herein forms the Hybrid Interactive Virtual Environment (HiVE) system 100 of the present invention. The HiVE system is configured for clinical education and training and includes: an immersive room 120 configured as a fully immersive interactive hybrid classroom, the immersive room being a six-sided cave automated virtual environment (CAVE) platform; an image generator 110 configured to process input content and render the input content in a two-dimensional (2D) or three-dimensional (3D) mode to generate interactive content for clinical education and training; a projection distortion system 130 configured to perform projection distortion and blending processing to generate a projection-distorted display image; and a video processing system 140 for processing the projection-distorted display image to generate… The immersive room includes: a surround sound system 121 for generating audio output to interactive content; a motion capture system 122 for generating tracking information of users within the immersive room and sending the tracking information to the interactive content; and multiple projectors 125 for projecting frame-synchronized display output from a video processing system to generate 3D interactive images within the immersive room for clinical education and training, enabling users in the immersive room to view and listen to the 3D interactive images and interact with and / or each other for clinical education and training.

[0113] Figure 18 It shows that based on, Figure 17B The virtual 3D images shown here combine MR with HiVE teaching. Figure 18 Immersive Room 120 and 3D Images with Figure 17B The system is identical to the one used in the 3D model and also includes a screen 604 displaying the course content in 2D mode. Some students stand within the 3D image and accordingly wear goggles 603 to view the 3D image and listen to the clinical education and training course. Figure 18 In the immersive room 120, vertical lower sections 172 and 174 and inwardly inclined upper sections 171 and 173 are respectively arranged on the left and right sides of the immersive room 120, and multiple projectors are arranged below or above the ceiling 175 of the immersive room 120, including projectors 125 that display 2D images on screen 604.

[0114] Figure 19A tutorial classroom environment in the immersive room 120 of the HiVE system 100 of the present application is shown that can be conducted in the same teaching session. Figure 19 The immersive room 120 and 3D image of Figure 17B are the same, but viewed from a different direction than the vertical viewing direction of Figure 17B . Figure 19 The immersive room 120 shown in FIG. 17 further includes a screen 604 showing course content. The teacher stands near the screen and teaches the course to the students standing within the 3D image. Figure 18 and Figure 19 shows a different side view of the 3D image of Figure 17B .

[0115] Based on FIG. 17 to Figure 19 The present application also provides a method for creating a hybrid interactive virtual environment (HiVE) for clinical education and training, comprising: forming an immersive room by a 6-sided cave automatic virtual environment (CAVE) platform to configure the immersive room as a fully immersive interactive hybrid classroom; inputting or saving content for clinical education and training to an image generator positioned in the immersive room, wherein the image generator is configured to process the inputted or saved content and render the content in a 2D mode or a 3D mode to produce interactive content; performing projection warping and hybrid processing by a projection warping system to produce a projection warped display image; and processing the projection warped display image by a video processing system to produce a frame-synchronized display output, wherein the immersive room comprises: a surround sound system for generating audio output to the interactive content; a motion capture system for producing tracking information of users inside the immersive room and sending the tracking information to the interactive content; and a projector for projecting the frame-synchronized display output from the video processing system to generate a 3D interactive image inside the immersive room for clinical education and training, such that the users in the immersive room can view and hear the 3D image and interact with the 3D image and / or each other for the clinical education and training.

[0116] As described above, the mixed interactive virtual environment system for clinical education and training employs a fully immersive CAVE, and can take 360-degree photos and calibrate imaging equipment for media projection on the HiVE. The system integrates virtual and physical objects to facilitate realistic simulation, and can select available software to facilitate scene design for simulation. The system is a fully immersive 6-sided CAVE, capable of displaying 360-degree images obtained in different clinical environments, which can be configured as an XR-enabled hybrid classroom. The system enables discipline-specific installation of large-scale CAVEs, and can save high logistics and management costs of face-to-face and immersive teaching in multiple physical locations, including on-campus and off-campus (e.g., hospitals).

[0117] The present system of mixed interactive virtual environment for clinical education and training can achieve advantages. For example, it can provide pre-clinical training for medical students with immersive experience. As on-site visits are usually limited, it can provide immersive virtual visits of hospital environments, so it can become an important bridge between theory and practice in healthcare education, allowing interaction between learners and teachers. Therefore, compared with earphone-based learning methods, it is more convenient and reduces technical barriers, can promote human-to-human interaction and dynamic team cooperation training, and narrow the gap between theory and practice in healthcare education through immersive scene simulation, allowing students to practice various clinical skills repeatedly.

[0118] The background environment of the HiVE involves manually scaling 3D images and videos to improve realistic visualization of specified clinical areas. This is essential for combining virtual and actual objects to enable actual training such as connecting an oxygen source displayed on a 3D image during clinical simulation to actual tubing.

[0119] Various software applications are combined to bridge the gap between theory and practice in healthcare education (see FIG. 15), including projecting virtual anatomy or medical images (theory) onto a manikin (AR) during skill training (practice). This allows real-time assessment of capabilities, enhances knowledge acquisition, and improves performance. The present invention can provide fully immersive clinical simulation for a wide range of clinical education and training, interactive VR and XR models for interactive hands-on training involving human-to-equipment and human-to-human interaction, and live streaming of immersive videos for collaborative learning, thereby facilitating large-scale teaching to increase students' exposure to a wider range of clinical practice.

[0120] IV. Implementation example in pediatric cancer care through the mixed interactive virtual environment (HiVE) system of the present invention and methods of using the same:

[0121] In addition to the above in Figures 16-19In addition to the medical courses described above for teaching medical students in the immersive room 120, the HiVE system 100 of the present application and its method of use can be used for patient care and / or rehearsal, for example, for pediatric cancer care, in a project called HEROCARE (Holistic Education and Care Authorization in Radiooncology for Children with Cancer and Caregivers) and an overview of its support strategies is described below. The HEROCARE project provides clinical education and training for patients, caregivers, healthcare personnel, and medical students simultaneously.

[0122] As an example of the HEROCARE project, a child patient with pediatric cancer can lie on an actual bed in the immersive room 120 instead of the virtual patient 601 and virtual bed in the virtual room 600 Figure 16 、 Figure 17B and Figures 18-19 The virtual linear accelerator 602 can be replaced by another virtual medical instrument for treating the patient. The HiVE system 100 can project 3D images of a simulated treatment process to the child patient to rehearse the actual treatment he / she will undergo to familiarize him / her with the treatment process and to reduce his / her stress.

[0123] As introduced in the "BACKGROUND" section of the present specification, the current challenges in pediatric radiotherapy services can include, for example: cancer is the second leading cause of death in children after accidents. Children cancer patients require special treatment protocols that are different from adult cancer. The psychological burden on children and their families from cancer diagnosis and treatment is enormous. Anxiety and depression are common, complicating recovery and quality of life. In addition, the challenge of radiotherapy is that for children receiving radiotherapy, complete immobilization is required during multiple fractionated treatments, which generally requires the use of multiple general anesthesia (GA) or sedation, increasing the risk of anesthesia-related complications and increasing the emotional and physical burden on young patients.

[0124] According to the hybrid interactive virtual environment (HiVE) system of the present application and its method of use, the innovative approach of HEROCARE can be used to improve the health of patients and caregivers, reduce treatment time, and advocate a new service model.

[0125] The HiVE system of the present application and its method of use can provide personalized preparation services, introducing comprehensive and customized preparation sessions for children and their caregivers, reducing anxiety and dependence on GA / sedation using immersive technology and simulation.

[0126] The HiVE system of the present application and its method of use can provide treatment efficiency, significantly reducing treatment time through efficient preparation, eliminating the need for GA / sedation, thereby reducing hospitalization time and improving the throughput of radiotherapy departments.

[0127] The HiVE system of the present invention and its method of use can provide support for service model innovation, actively promote the adoption of the HEROCARE model as a standard practice in pediatric oncology, and highlight its benefits in terms of patient care, cost reduction, and treatment efficacy to the healthcare sector.

[0128] The HiVE system of the present invention and its method of use can provide an overall support network that establishes interdisciplinary collaboration between health care professionals, scholars, and support personnel to provide an overall care environment that addresses not only the medical needs of patients and caregivers but also their psychological needs.

[0129] The HiVE system of the present invention and its method of use can provide new uses for treatment preparation and rehearsal to minimize the need for GA or sedation.

[0130] The HiVE system of the present invention and its method of use can provide immersive preparation workshops that utilize mixed reality and realistic simulations in preparation workshops to familiarize children with the treatment environment, reducing the need for GA / sedation via reduced fear and anxiety.

[0131] The HiVE system of the present invention and its method of use can provide customized treatment rehearsals without the risks associated with GA, providing personalized treatment rehearsals that simulate actual radiotherapy sessions, allowing children to practice quietness and compliance.

[0132] The HiVE system of the present invention and its method of use can provide educational empowerment by engaging children and their caregivers in educational conversations about the treatment process, equipping them with the knowledge and confidence to undergo treatment with minimal or no GA.

[0133] The HiVE system of the present invention and its method of use can provide multidisciplinary team collaboration. Bringing together healthcare professionals from various fields, including oncology, radiology, psychology, and child life experts, to provide a comprehensive and integrated approach to care that ensures all aspects of a child's health are addressed during treatment.

[0134] The HiVE system of the present invention and its method of use can provide economic and efficiency benefits by reducing dependence on GA, resulting in significant time and cost savings, leading to shorter treatment durations, reduced use of hospital resources, and improved patient and caregiver satisfaction, while promoting interdisciplinary collaboration that improves care efficiency and innovation.

[0135] Accordingly, in accordance with the present invention, the objectives of the HEROCARE project of the mixed interactive virtual environment (HiVE) system of the present invention and its method of use include:

[0136] Improving the health of patients and caregivers: By providing a comprehensive and personalized preparation experience that minimizes the reliance on general anesthesia (GA), the adverse effects on the body and emotions are reduced, thus improving the overall health of pediatric cancer patients and their caregivers.

[0137] Reducing treatment time and cost: By eliminating the need for GA / sedation in pediatric radiotherapy, significant time and cost savings are achieved for healthcare institutions, which in turn significantly reduces treatment time and leads to more efficient use of healthcare resources.

[0138] Advocating a new service model: Promoting the innovative HEROCARE service model to the healthcare sector and institutions. By demonstrating the efficiency of the model in improving patient care and treatment efficiency, support from the medical community and stakeholders is obtained.

[0139] Filling the gap in preparation services: Addressing the severe lack of pediatric radiotherapy patient preparation services that typically require the use of GA / sedatives every day, resulting in both physical and mental trauma to children and their caregivers.

[0140] The HEROCARE project leverages the unique capabilities of the Hong Kong Polytechnic University (PolyU) as the only undergraduate radiotherapy education provider in Hong Kong, as well as its history of successful pilot projects, to lead and expand innovative radiotherapy services. As such, it takes full advantage of the unique position of PolyU.

[0141] Promoting interdisciplinary collaboration: Developing a collaboration model that brings together multidisciplinary teams from healthcare, academia, and community organizations. This model aims to provide a holistic approach to care that meets the diverse needs of pediatric cancer patients and their families.

[0142] Ensuring long-term sustainability: Ensuring the future of the project through strong support mechanisms, including dedicated funding and partnerships with healthcare providers and educational institutions. This ensures that the innovations introduced by HEROCARE can continue to benefit patients and the medical community in the long term.

[0143] Enabling expansion of beneficiaries: The project service model utilizing HiVE and immersive installations can be expanded to include a wider range of healthcare target groups, such as adult and elderly populations that require customized psychosocial and educational enablement in the current model framework.

[0144] Although certain example methods, apparatus, and articles of manufacture have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus, and articles of manufacture falling within the scope of the claims.

[0145] In aspects, descriptions are made with reference to the accompanying drawings. However, certain aspects can be practiced without one or more of these specific details, or in combination with other known methods and configurations. In the description, numerous specific details are set forth in order to provide a thorough understanding of the implementations. In other instances, well-known process and manufacturing techniques have not been described in particular detail in order to not unnecessarily obscure aspects of the description. Reference throughout this specification to "an embodiment," "embodiments," "an aspect," "aspects," "one implementation," "implementation," or the like means that a particular feature, structure, configuration, or characteristic described is included in at least one embodiment, aspect, or implementation. Therefore, the appearance of the phrases "in one embodiment," "in an embodiment," "in aspects," "in one aspect," "in one implementation," "in an implementation," or the like in various places throughout this specification are not necessarily referring to the same embodiment, aspect, or implementation. Furthermore, the particular features, structures, configurations, or characteristics can be combined in any suitable manner in one or more implementations.

[0146] While this specification contains many details, these should not be construed as limitations on the scope of what can be claimed or might be claimed, but as descriptions of features specific to particular embodiments. Some features described in the context of separate embodiments can be combined with each other in a single embodiment. Conversely, various features described in the context of a single embodiment can be separated and performed in multiple embodiments. Also, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a sub-combination or a variation of a sub-combination. Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such an order, nor that all illustrated operations be performed, to achieve desirable results. Only a few examples, embodiments, aspects and implementations are disclosed. Numerous modifications and adaptations will be apparent to those skilled in the art based on the disclosure provided and the general principles described.

Claims

1. A hybrid interactive virtual environment (HiVE) system (100) configured for clinical education and training, and comprising: an immersive room (120) configured as a fully immersive interactive hybrid classroom, said immersive room being a 6-sided cave automatic virtual environment (CAVE) platform; an image generator (110) configured to process input content and render said input content in two-dimensional (2D) mode or three-dimensional (3D) mode to generate interactive content for clinical education and training; a projection warping system (130) configured to perform projection warping and blending processing to generate a projection warped display image; and a video processing system (140) for processing said projection warped display image to generate a frame-synchronized display output; wherein said immersive room includes a surround sound system (121) for producing audio output to said interactive content, a motion capture system (122) for producing tracking information of users inside said immersive room and sending said tracking information to said interactive content, and a plurality of projectors (125) for projecting frame-synchronized display output from said video processing system to generate 3D interactive images inside said immersive room for clinical education and training, such that users in said immersive room can view and hear said 3D interactive images and interact with said 3D interactive images and / or with each other for clinical education and training.

2. The hybrid interactive virtual environment system of claim 1, wherein in the processing of projection warping performed by said projection warping system (130), said input content on said image generator (110) is rendered and arranged based on the physical layout of the projection surfaces of said projectors, and then warped based on the rendered display layout, and then further processed, rearranged and distributed to each projector; wherein when running in 2D mode (200) with projection warping and blending, all pixels assigned to each projector are directly displayed by said projector, and when running in said 3D mode (300), said image generator (110) renders images for both left and right eyes of said users and sends as final output to said video processing system in side-by-side manner. Projectors for said ceiling (175) are mounted outside said space behind left and right walls, with an opening carved in the wall matching the projected light cone shape for each projector to allow light to pass through while minimizing the impact on immersion; 3. The mixed interactive virtual environment system of claim 1, wherein in the immersive room, projectors for walls are installed as front, back, left and right projectors inside the space on the ceiling area, a set of custom projector cover structures (500) are installed around the projectors to hide the projectors while allowing light from the ceiling projectors to project onto them; and projectors for the floor are mounted on the ceiling wall and projected onto said floor through an opening carved in said ceiling in the same way as said ceiling projectors.

4. The hybrid interactive virtual environment system of claim 1, wherein a plurality of calibration cameras (122) are placed inside said immersive room (120) and connected to said image generator (110) to facilitate projection warping and blending. ​ 5. The mixed interactive virtual environment system of claim 1, wherein to increase the interactivity of the system, the motion capture system (122) comprises a plurality of motion capture or sensing devices using optical or infrared light mechanisms, which are mounted under the ceiling of the immersive room.

6. The mixed interactive virtual environment system of claim 1, wherein the immersive room (120) is created inside a structural room (400) using dry walls that separate the structural room into an immersive room for a fully immersive interactive projection system and further rooms or spaces (401-402), wherein each side wall of the dry walls on the left and right sides of the immersive room is composed of two parts: a vertical lower segment (172, 174) and an inwardly tilted upper segment (171, 173), wherein the upper segment of the side wall is tilted inwardly at an angle that closely matches the optical path of the bottom of the projection cone of the projector, such that the immersive room is 6-sided, comprising a ceiling (175), a floor (176), two vertical segments (172, 174) and two inwardly tilted upper segments (171, 173).

7. The mixed interactive virtual environment system of claim 1, wherein the system further comprises a camera (122) for taking 360-degree photos of the immersive room during clinical education and training, and the projector (125) is capable of displaying 360-degree images obtained in different clinical environments.

8. The mixed interactive virtual environment system of claim 1, wherein the system further comprises goggles (603) and handheld devices (605) for interacting with the 3D interactive images and with each other during clinical education and training.

9. The mixed interactive virtual environment system of claim 1, wherein the system comprises different teaching modes to meet different learning objectives in a teaching session during clinical education and training, wherein in one of the teaching modes, the immersive room is used as a regular classroom.

10. The mixed interactive virtual environment system of claim 1, wherein real physical objects can be placed in the immersive room, and the system integrates the virtual objects (601, 602) of the 3D images and the real physical objects together for realistic simulation.

11. A method for creating a hybrid interactive virtual environment (HiVE) (100), the method being configured for clinical education and training, and comprising: forming an immersive room through a 6-sided cave automatic virtual environment (CAVE) platform to configure the immersive room as a fully immersive interactive hybrid classroom (120); inputting or saving content for clinical education and training into an image generator (110) positioned in the immersive room, wherein the image generator is configured to process the inputted or saved content and render the content in two-dimensional (2D) mode or three-dimensional (3D) mode to generate interactive content; performing projection warping and blending processing through a projection warping system (130) to generate a projection warped display image; and processing the projection warped display image through a video processing system (140) to generate a frame-synchronized display output; wherein the immersive room (120) comprises a surround sound system (121) for producing audio output to the interactive content; a motion capture system (122) for generating tracking information of a user inside an immersive room and sending the tracking information to interactive content; and a plurality of projectors (125) for projecting frame-synchronized display output from a video processing system to generate 3D interactive images (601, 602) inside the immersive room for clinical education and training, such that the user in the immersive room can view and hear the 3D interactive images, and interact with the 3D interactive images and / or with each other for clinical education and training.

12. The method of claim 11, wherein during the projection warping by the projection warping system (130), the input content on the image generator (110) is rendered and arranged based on the physical layout of the projection surface of the projectors (125), and then warped based on the rendered display layout, and then further processed, rearranged, and distributed to each projector; wherein when running the 2D mode (200) with projection warping and blending, the total pixels assigned to each projector are displayed directly by the projector, and when running in the 3D mode (300), the image generator (110) renders images for both the left and right eyes of the user, and arranges as a final output to send to the video processing system in a side-by-side manner.

13. The method of claim 11, wherein to increase the interactivity of the system, the motion capture system (122) includes a plurality of motion capture or sensing devices using optical or infrared light mechanisms, which are installed under the ceiling of the immersive room.

14. The method of claim 11, further comprising taking 360-degree photos of the immersive room using a camera during clinical education and training, and displaying the 360-degree images obtained in different clinical environments by the projectors.

15. The method of claim 11, further comprising integrating real physical objects and virtual objects of the 3D images (601, 602) in the immersive room (120) to facilitate realistic simulation during clinical education and training.

Images