Systems and methods for immersive mixed reality spaces

Abstract

A system and a method for mixed reality immersive simulation of a virtual space that can be used for simulated dynamic deformation, the virtual space being larger and more spacious than the actual physical source space surrounding the participants. The system and method can also provide multi-space simulation, e.g. of alternate battlefields, in which several different participants can operate in different mixed reality sub-scenarios, while sharing the same physical space as source space, thus enabling maximum close quarters space.

Description

TECHNICAL FIELD

[0001] The present invention relates generally to immersive mixed reality systems and to those systems used for multi-participant multi-scenario teaching or training to improve skills in, for example, narrative situations in combat, triage or complex evolutions. BACKGROUND

[0002] Virtual reality (VR) systems give participants the sensation of being in an environment different from the one they are actually in. Some computer game systems have the technology to place participants in an interactive world. For example, a player can be placed in a variety of locations such as a car seat, an airplane or other man-made environment. In such systems, the user is essentially a spectator watching events occur in a world created by the system. Such experiences have traditionally been obtained through accessories that provide visual and aural sensory input. These accessories can be head-mounted displays (HMDs) such as the HTC Oculus ) which provide a fully computer-generated stereoscopic display along with adaptive sound effects.

[0003] Augmented reality (AR) systems will consist of adding additional information on a real-world physical scene or object in real time. This can be obtained by combining computer-generated graphics with graphics perceived by the user through normal vision. This can be obtained through an HMD with semi-transparent optics, enabling the addition of another layer of computer-generated optics (e.g., Google ). Audio augmentation can also be used to enhance the augmented experience. Thus, the participant's sensory experience includes real-world sensory as well as artificial sensory data.

[0004] AR and VR can lie on a continuum of mixed reality (MR) or mixed reality (as first discussed by P. Milgram and A. F. Kishino (1994). "Taxonomy of Mixed Reality Visual Displays," IEICE Transactions on Information and Systems, pp. 1321-1329). This continuum extends from a completely real environment to a completely virtual environment. The location on the continuum will depend on the rate of immersive technology that implements the experience. In AR, the real world can still be experienced, but with an additional layer, while in VR, the user is isolated from the real world, for which, in most cases, the user is placed in a sterile enclosure during the experience to prevent obstruction of the virtualization immersive experience. In most cases, VR systems isolate the user from the physical world and provide a unique personal experience separate from the experience of people around or elsewhere.

[0005] It is well known that simulation training enjoys many advantages over conventional real life training. Trainees can benefit from reaching the extreme scenarios when they feel relief from life threatening situations that can be simulated during, for example, virtual flight training. As a result, virtual flight simulators have been found to produce more effective training than conventional flight practice. In some cases, simulation can be more cost effective than real life training.

[0006] In addition to video and audio, adding haptic sensory means will enhance the immersion into the virtual experience. Real time interaction between users in the same environment and in alternative environments will further enhance the experience and simulation of real world interaction.

[0007] Conventional AR systems and VR systems are targeted at games and entertainment that enjoy relaxed specification requirements and tolerance for deviation from real life simulation accuracy. In contrast, high end AR and VR require accuracy and responsiveness that involve high cost and implementation complexity.

[0008] AR systems require tailoring to the physical location. For example, simulation of obstacle courses will require building actual physical obstacles. Flight simulators require 6 degrees of freedom moving cabins to simulate the feeling of gravity. Tailored systems require complex and expensive electronic tracking and positioning devices. Although some current systems facilitate personal mobile devices (e.g. smartphones) as personal electronic devices, these systems are too simple and limited and in most cases are related to the gaming industry.

[0009] Another component that provides a realistic experience can be obtained by using accessories similar to real life equipment. The great difference between MR accessories and real life tools can have an adverse effect on the effectiveness of teaching or training activities. The credibility of the tools directly affects the user's relationship with the severity of the exercise - is it a game or a simulation of real life situations and their dangers. System standards for specific operational tasks differ from the lower standards of systems assigned only for games. For example, a laser gun arena uses accessories of simulated weapons, although these accessories do not have the feel of a real rifle weapon. Simulated accessories usually lack the use of dynamics (e.g. ballistic trajectory or recoil effect). Although VR systems are indeed considered a major part of the gaming field, there are also examples of VR applications in other material fields such as the medical field. For example, surgeons evaluate 3D models in a VR environment to prepare for surgery. On the other hand, AR systems are known to be implemented in actual medical procedures (see for example da Vinci® surgical system).

[0010] ​The speed and range of computing and technological capabilities have greatly increased the ability to immerse. The development of computing power (especially real-time complex computing), the proliferation of high-quality graphical display means, and holographic capabilities and haptic input accessories have strengthened the study of the benefits of using MR immersive tools to simulate real-life experiences.

[0011] There are several factors that need to make an individual's brain perceive being in a different immersive reality, and display is a major component. Primarily, 3-dimensional displays are an important means of creating this perception, however, a 3D omnidirectional display by itself does not create a life-like experience. Tracking motion, primarily head and eye motion, can change the 3D headset display accordingly. Real-time reproduction of dynamic images, effective haptic feedback, sensory input interaction according to the individual parameters of the participant (temperature, perspiration, respiration, hyperventilation, brain waves, neural electrical rates, vital signs, etc.), and interactive scene adaptation can produce an effective simulation.

[0012] Most VR experiences cater to the needs of a single participant immersed. The system simulates an environment for one user based on the participant's actions and reactions. Overall, immersive VR experiences are isolated. This isolation prevents interaction with others in the real world, whether or not they are experiencing a corresponding VR experience.

[0013] It has been recognized that using collaborative VR technology can be an effective way to work together with users in disparate, different remote geographic locations (https: / / venturebeat.com / 2017 / 10 / 11 / 3-reasons-why-vrs-killer-app-will-be-collaborative / ). Some systems have extended the experience to team-based VR experiences (e.g. Virtual Room: https: / / virtual-room.com / en / ): several people can perform joint tasks by communicating and working together without sharing the same physical space. Each participant has their own dedicated room, which is specially designed and equipped with virtual reality equipment. The way to participate in the game is to stand up and walk around, look for items, manipulate objects, and collaborate with other team members.

[0014] Collaborative VR / AR is also used for collaboration between colleagues at different locations. An example of a shared virtual workspace connecting multiple sites around the same 3D project can be found at https: / / www.techviz.net / collaborative. Participants from different locations will see each other, visualize and work on the same 3D model in real time from any 3D application (e.g. Catia, NX, Creo, Navisworks) after being immersed in their own 3D system through various VR devices (immersive room, HTC Vive, video wall). This software aims to improve communication between colleagues and decision making and validation processes without actually meeting. TechViz VR collaboration can also be used to provide remote training in real time at any distance. Teamwork becomes easier within a company structure, especially between people working all over the world. This kind of application supports project work sharing, where manipulation of a 3D model is carried out by several users who can interact and make annotations on the 3D model. All users can see the same 3D model and annotations in real time, but their actions and unique perspectives are not necessarily shared.

[0015] Some collaborative VR systems facilitate communication of remote users through avatars in a virtual environment (US 7,168,051). This avatar communication aims at discussion rather than actual joint action.

[0016] Current collaborative immersive environment systems aim to jointly solve problems. Several participants gather in a virtual environment and contribute to perform a task. Such users can be in different places. US 9,779,548, for example, proposes an immersive system that records the location of the users in the physical world to conform to the coordinates of the virtual world and thus create a consistent VR / AR experience. User location data ambiguity can arise when several users are in the same space. This problem is mitigated in US 9,779,548 with respect to AR in mobile devices. By using separate mobile devices to share a mutually consistent AR, each mobile device enjoys a different anchor point in the AR world (also mentioned in US 9,122,391). By having an AR device report its own orientation and distance to one or more other anchors to other AR devices, each participant is presented as an AR representation to other users. US 9,779,548 allows multiple participants to share the same AR session when each user is in a different location. US 9,779,548 does not address the case where users are in the same location and wish to join the same AR session. US 9,779,548 does not address the case where several participants at the same location wish to join / have separate different immersive sessions. This can not be a problem with respect to AR - AR relates to physical terrain augmented by virtual components, each participant at the same location has a different perspective on the current physical components, depending on their slightly different positioning in space.

[0017] Immersive room technology is the case where the immersive room user wears a pair of VR glasses that display 3D images through stereoscopy. The 3D effect is rendered by displaying two images - one for each eye, which allows the brain to interpret the depth of objects. With the help of a tracking system that records the behavior of the person inside the immersive room, the user's point of view is defined to display the images with a normal point of view.

[0018] Current systems and methods struggle to implement several users in the same MR environment. The common registration of more than one user in the same simulated environment can lead to data ambiguity. AR systems, which are based on basic physical data and real-life coordinates, do not suffer from such problems. VR systems solve such problems by placing different users in separate physical areas (e.g. sterile experience cubes). Isolating multiple users for such VR experiences requires a considerable amount of physical space.

[0019] As mentioned before, an important part of an effective simulation is the interactivity between participants (e.g. cooperation between soldiers on a simulated battlefield or medical experts on a surgical table). Interactivity between different environments (an event in one environment can affect an input in another environment) is also desirable.

[0020] None of the systems provide a system that enables several participants to enjoy interaction through different immersive environments in a cost-effective manner. Most current multi-participant solutions are tailored for specific cases (e.g. collaborative CAD) and such systems lack flexibility to adapt to changing simulation requirements in a cost-effective manner.

[0021] The present invention relates generally to immersive mixed reality systems and to those for multi-participant multi-scenario teaching or training to improve skills in narrative situations such as in combat, triage or complex evolutions, which overcome the aforementioned drawbacks.

[0022] The present invention enables several participants to use the same or different virtual environments in a life-like manner while interacting and communicating with each other to achieve the completion of operational tasks. The proposed immersive technology proposes a flexible cost-effective system that is able to simulate truly life-threatening situations. SUMMARY

[0023] The present invention provides a system and a method for MR immersive simulation of virtual spaces that can be used to simulate dynamic deformations, which are larger and more spacious than the actual physical source space around the participants. The system and method can also provide multi-space simulation of e.g. alternate combat arenas in which several different participants can operate in different MR sub-scenarios while sharing the same physical space as source space, thus maximizing the close-proximity space. By way of example, such scenarios can include participants fighting on different floors of the same virtual building or fighting in different parts of a war stage.

[0024] The system and method can also include applying safety mechanisms for possible collision detection and supervision of the movement of the participants with respect to obstructions and obstacles, thus allowing the participants to perform real-world type of movements within a certain possibly limited physical space while improving the MR operation and the sense of immersion.

[0025] The following embodiments and aspects thereof will be described and explained with additional specificity and detail to exemplify and teach the claimed invention. In various embodiments, one or more of the above-identified problems have been reduced or eliminated, and yet other embodiments are directed to other advantages or improvements.

[0026] According to one aspect, there is provided an MR immersive system, the system comprising a controller configured to operate at least one MR scenario and at least one sensory device associated with at least one participant.

[0027] According to some embodiments, the system can be deployed in at least one source space physically occupied by the at least one participant, wherein the source space is configured to dynamically morph into at least one vast and spatially changing object space, wherein the object space is transmitted to the at least one sensory device associated with the at least one participant.

[0028] According to some embodiments, at least one sensor is configured to monitor parameters of the at least one participant.

[0029] According to some embodiments, the parameters of the participant include vital signs of the participant.

[0030] According to some embodiments, the parameters of the participant include position of the participant.

[0031] According to some embodiments, the controller is integrated into the sensory device.

[0032] According to some embodiments, the sensory device includes a head mounted display (HMD) configured to display images to the at least one participant.

[0033] According to some embodiments, the sensory device includes a communication device configured to enable communication between participants.

[0034] According to some embodiments, the communication is configured to provide inter-participant and supervisor-participant communication.

[0035] According to some embodiments, the sensory device includes a haptic device configured to provide haptic stimulation to the participant.

[0036] According to some embodiments, a safety mechanism prevents possible physical collisions between the participants themselves or between the participants and physical objects that can exist within the source space.

[0037] According to some embodiments, the safety mechanism uses pre-mapping of the source space.

[0038] According to some embodiments, the object space is simulated by a plurality of image tiles dynamically morphed to fit the source space.

[0039] According to some embodiments, a plurality of image tiles properties are optimized to reduce latency and save system resources.

[0040] According to some embodiments, the same source space can be used for several MR scenarios conducted simultaneously.

[0041] According to some embodiments, the MR scenarios are subject to constraints generated by measurable important scenario parameters.

[0042] According to some embodiments, the MR restriction fraction is calculated according to said measurable important situational parameter, which in turn affects the selection of the next object image block forming the object space presented to the participant.

[0043] According to some embodiments, the shape of the object space can be warped to fit a given source space while preserving the measurements of the object space.

[0044] According to some embodiments, the MR situation is configured to dynamically form a vast object space suitable for at least one participant.

[0045] According to some embodiments, the MR situation is configured to dynamically form at least two object sub-spaces suitable for at least two participants.

[0046] According to some embodiments, the at least two object sub-spaces are replicas of the same object space type.

[0047] According to some embodiments, the at least two object sub-spaces are different active site object sub-spaces.

[0048] According to some embodiments, at least two participants can simultaneously perform different tasks while immersed in the at least two object sub-spaces.

[0049] According to some embodiments, the at least two participants operating within the at least two object sub-spaces have real-time mutual communication.

[0050] According to some embodiments, the at least one MR situation is configured to dynamically correspond and adapt to each other according to situational evolution and participant actions.

[0051] According to some embodiments, the system further comprises a real-life accessory.

[0052] According to some embodiments, the accessory is equipped with at least one sensing device.

[0053] According to some embodiments, the accessory is equipped with at least one haptic device.

[0054] According to some embodiments, the accessory further comprises a real-life patient operating tool.

[0055] According to some embodiments, the haptic device can simulate the feeling of using and operating a real weapon.

[0056] According to some embodiments, the haptic device can simulate the feeling of being hit by another participant.

[0057] According to a second aspect, there is provided a method for performing a MR scenario, the method comprising the steps of: operating at least one MR scenario using a controller, the at least one MR scenario involving at least one participant present within a source space; dynamically morphing spatial characteristics of the source space to form a vast and spatially varying object space; and delivering at least one object space to at least one sensory device associated with the at least one participant.

[0058] According to some embodiments, dynamically morphing the spatial characteristics of the source space forms at least two object subspaces that are vast and spatially varying, suitable for at least two participants.

[0059] According to some embodiments, the at least two object subspaces are duplicates of the same object space type.

[0060] According to some embodiments, the at least two object subspaces are different active site object subspaces.

[0061] According to some embodiments, at least two participants can perform different tasks simultaneously while immersed in the at least two object subspaces.

[0062] According to some embodiments, the steps can be performed using off-the-shelf components. BRIEF DESCRIPTION OF DRAWINGS

[0063] Some embodiments of the application will be described herein with reference to the accompanying drawings. The description and drawings are illustrative of some embodiments and are not intended to limit the scope of the application. Various Figure One skilled in the art can be made in light of the description. The drawings serve the purpose of illustrative description only and not intended to limit the scope of the embodiments. Some of the objects depicted in the drawings are not drawn to scale.

[0064] In the drawings:

[0065] Figure 1 Schematic perspective view of a VR scenario being conducted during use of a MR immersive system, constituting some embodiments.

[0066] Figure 2 Schematic perspective view of a VR scenario being conducted during use of a MR immersive system, constituting some embodiments.

[0067] Figure 3 Schematic perspective view of a VR scenario being conducted during use of a MR immersive system, constituting some embodiments.

[0068] Figure 4 Schematic perspective view of a VR scenario being conducted during use of a MR immersive system, constituting some embodiments.

[0069] Figure 5 The diagram illustrates a flowchart of a method for using a participant-immersive system based on some implementation schemes.

[0070] Figure 6 A graph that comprises the source space, object space, and blocks.

[0071] Figure 7 The flowchart illustrates a method for selecting object blocks using important scenario property scoring.

[0072] Figures 8a and 8b constitute examples of distortion deformation from 2D object space to constrained source space. Detailed Implementation

[0073] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, those skilled in the art will appreciate that the invention can be practiced without these specific details. In other instances, well-known methods, procedures, components, modules, units, and / or circuits have not been described in detail so as not to obscure the invention. Some features or elements described with respect to one embodiment may be combined with features or elements described with respect to other embodiments. For clarity, discussions of the same or similar features or elements may not be repeated.

[0074] While embodiments of the invention are not limited in this respect, discussions using terms such as “processing,” “computing,” “calculating,” “determining,” “establishing,” “analyzing,” “checking,” “setting,” and “receiving” can refer to the operation and / or process of a controller, computer, computing platform, computing system, or other electronic computing device that manipulates and / or transforms data represented as physical (e.g., electronic) quantities in computer registers and / or memory into other data represented similarly in computer registers and / or memory or other information non-transitory storage media that can store instructions to perform the operation and / or process.

[0075] While embodiments of the invention are not limited in this respect, the terms "multiple" and "a plurality of" as used herein may include, for example, "multiple" or "two or more". The terms "multiple" or "a plurality of" may be used throughout the specification to describe two or more components, devices, elements, units, parameters, etc. Unless explicitly stated otherwise, the method embodiments described herein are not limited to a particular order or sequence. Furthermore, some of the described method embodiments or elements thereof may occur or be performed simultaneously, at the same time, or concurrently.

[0076] As used herein, the term "controller" refers to any type of computing platform that can be equipped with a storage device, a central processing unit (CPU) or microprocessor device, and several input / output (I / O) ports, such as a general purpose computer, for example a personal computer, a laptop computer or a tablet computer or a cloud computing system.

[0077] As used herein, the term "mixed reality scenario" or "MR scenario" refers to a computer-generated scenario that computes an MR space and allows it to be realized by a participant. The scenario can be described in geometric terms and can contain any kind of properties.

[0078] As used herein, the term "sensor" refers to any kind of sensing device capable of tracking the position of a participant or capturing images that detect any other type of parameter during a VR / AR scenario or a MR scenario that can be a mix thereof.

[0079] As used herein, the term "source space" refers to the actual physical space that is occupied and in which a participant operates. For example, a room in which a participant wears an HMD and in which the participant moves, rotates, crouches, jumps, etc.

[0080] As used herein, the term "object space" refers to a virtual modeling space that can be described by a pre-generated model or by an algorithm that dynamically creates the object space. The modeling of the object space can be carried out in various ways using various mathematical methods. Such modeling can be with respect to the entire space or segments therein, which are referred to herein as "object chunks".

[0081] As used herein, the term "object space layout" refers to a set of 3d coordinates that describe a virtual shape in a 3D space, such as a room floor, a table or a staircase.

[0082] As used herein, the term "trackable space" refers to a portion of a physical space, mainly a portion of the source space, in which a sensor is capable of tracking the motion of a participant therein.

[0083] As used herein, the term "occupied trackable space" refers to a portion of a trackable space that is marked for use by a participant executing a specific virtual reality scenario.

[0084] As used herein, the term "shape transformation" refers to a process of changing the coordinates of a shape in a virtual reality space without changing the shape itself. For example: moving or rotating are considered shape transformations.

[0085] As used herein, the term "dynamic deformation" refers to the process by which a 3D spatial shape adaptively transforms into another 3D shape during the generation of an MR scene. Without limitation, dynamic deformation of a 2D shape as part of an MR experience will be considered a subuse case of said 3D shape deformation. Dynamic deformation can include, but is not limited to, deformations achieved through scaling, twisting, rotating, folding, shrinking, enlarging, overlapping, and other mathematical operations on the matrix expression of 3D / 2D shapes.

[0086] As used in this article, the term "important scenario property / parameter" refers to an attribute or property of an MR scenario that is measurable (i.e., it has a score) and of a certain degree of importance to the creator of the virtual reality scenario.

[0087] Such properties may contain an initial measurable value (aka initial score) and a minimum measurable threshold (aka minimum score), below which the property is considered invalid. Each MR scenario may include a list of ranked properties defined by its creator. According to some implementations, this ranking may be obtained through a scoring method.

[0088] refer to Figure 1 This constitutes a schematic perspective view of an MR scenario using an MR immersion system 10 according to some embodiments of the present invention. As shown, the MR immersion system 10 includes three main devices:

[0089] 1. A controller 100, such as a computer configured to execute MR scenarios and control various devices including the MR immersive system 10. According to some embodiments, the controller 100 may be a separate device or may be integrated into a personal sensory device 200 (disclosed below).

[0090] 2. At least one personal sensory device 200, said personal sensory device being configured to generate sensory outputs for use by a participant (e.g.,) associated with each personal sensory device 200. Figure 1 The personal sensory device 200 may be perceived by participant A (as depicted in the previous figure) or participant BG (as depicted in subsequent figures). According to some embodiments, the personal sensory device 200 may include a head-mounted display (HMD) 202 configured to generate images for perception by participant A associated with the HMD 202. According to some embodiments, the HMD 202 may generate 3D images of a simulated object space 20. According to some embodiments, the object space 20 may simulate any type of scenario in which participant A operates as part of an MR simulation. According to some embodiments, the object space 20 includes an object space layout 22, which may be any kind of shape, object, or formation simulated as part of an MR scenario, such as floors, stairs, doors, windows, vehicles, terrain, etc.

[0091] According to some embodiments, the personal sensory device 200 can comprise an auditory device 204, e.g. headphones, microphones or speakers, configured to enable the participant A to communicate with other participants of the immersive system 10.

[0092] According to some embodiments, the personal sensory device 200 is equipped with a haptic device 208 (disclosed below) configured to provide haptic stimulation to the participant A to simulate the actual situation.

[0093] 3. At least one sensor 206 configured to sense the position of the participant A throughout or parts of the source space 24. According to some embodiments, the input from the sensor 206 can be processed by the controller 100 and can be used to determine the position of the participant A during the MR scenario. According to some embodiments, the determined position can play a role in the representation of the participant A’ on the simulated object space 20, which will be explained below. According to some embodiments, the sensor 206 can be, for example, a motion or tracking sensor configured to detect the position of the participant A or alternatively the position of certain parts of the participant A’s body, where these positions are then simulated as part of the MR scenario.

[0094] According to some embodiments, the sensor 206 can sense various parameters associated with the participant A. These parameters can include, for example, body temperature, perspiration level, respiratory rate, hyper-ventilation level, brain wave activity, nervous system electrical rate, etc. According to some embodiments, the sensor 206 can also sense vital signs of the participant A while immersed in the MR scenario.

[0095] According to some embodiments, the controller 100 can calculate and determine the health (e.g. physical and mental) state of the participant A according to the parameters or vital signs detected by the sensor 206. According to some embodiments, the sensor 206 can be a separate sensing device or can be integrated into the personal sensory device 200.

[0096] According to some embodiments, the MR immersive system 10 can comprise a haptic device 208 configured to provide haptics according to the specific MR scenario. According to some embodiments, the haptic device 208 can be implemented as part of the personal sensory device 200 and configured to provide haptic stimulation to the head of the participant A. According to some embodiments, the haptic device 208 can be implemented as part of the clothing of the participant A, for example, the haptic device 208 can be implemented as part of a tactical vest and can, for example, simulate a projectile hitting the upper body of the participant A.

[0097] According to some embodiments, the haptic device 208 can be implemented in a variety of real-life accessories used by the participant A during the MR scenario, such accessories can be adapted weapons, such as assault rifles.

[0098] According to some embodiments, the haptic device 208 implemented in a real-life accessory such as an assault rifle is able to provide the participant A with a realistic haptic, such as the real feeling of shooting live ammunition. This can be achieved by mimicking the recoil force associated with the use of live ammunition or by mimicking the feeling of a magazine change or loading operation.

[0099] According to some embodiments, the haptic stimulation can prevent the participant from injuring himself by colliding with physical objects of the real world (walls, stairs, corners, other participants, etc.). According to some embodiments, the MR immersive system 10 comprises a safety mechanism that uses a pre-mapping of the confined physical space and supervises the motion of the participant relative to obstacles and barriers. According to some embodiments, the proposed safety mechanism can be archived by using room-level VR technology that allows the participant to move in real-world type within the physical source space 24, thus improving the immersion.

[0100] According to some embodiments, the controller 100 can execute MR scenarios that can include dynamic deformations, such as shape deformations of the object space 20. According to some embodiments, the dynamic deformations are performed by intervening in the registration process, meaning intervening in the calculation process that is performed to synchronize the coordinates between the source space 24 and the object space 20 and to allow accurate conversion of coordinates from the source space 24 to the coordinates of the object space 20 and vice versa. Intervening and intentionally changing the registration process can result in dynamic deformations, in which the object space 20 perceived by the participant (the representation of participant A on the object space 20 is marked as participant A’) can change spatially when compared to the source space 24 physically occupied by the participant A.

[0101] According to some embodiments, the source space 24 in which the participant A executes the MR scenario can be confined to relatively small dimensions. For example, the participant A can stand in a standard room that does not allow much freedom of movement, while at the same time the object space 20 of the MR scenario perceived by the participant A undergoes dynamic deformations, in which the dimensions of the source space 24 (e.g., the dimensions of the standard room) are changed so that the modified object space 20 is felt by the participant A as more spacious (i.e., more extensive) than the source space 24 physically occupied by the participant A. According to some embodiments, the participant A can be immersed in a MR scenario that simulates, for example, storming or clearing a large and spacious room, while being in a confined source space 24 that is much smaller than the object space 20 he experiences.

[0102] According to some embodiments, the object space layout 22 can also be susceptible to dynamic morphing, resulting in objects appearing larger or smaller or closer or farther than they actually are. By way of example, a door that is part of the object space layout 22 can appear to be placed at a farther distance than physically possible in the confined dimension of the source space 24 physically occupied by participant A due to dynamic morphing, thereby contributing to the feeling of participant A operating in a spacious object space 20. According to some embodiments, dynamic morphing of the object space layout 22 can be carried out according to important scenario parameters.

[0103] (It should be emphasized that the foregoing disclosure regarding the equipment and operation of the MR immersive system 10 can also apply to each of the participants B-G disclosed hereinafter).

[0104] Reference is made to Figure 2 which constitutes a schematic perspective view of an MR scenario carried out by using the MR immersive system 10 according to some embodiments of the present application. As shown, at least two participants B and C are occupying a limited source space 24. According to some embodiments, the controller 100 can perform dynamic morphing to create an MR scenario that includes a spacious object space 20 having more than one object sub-space, or in other words a modified object space 20. By way of example, the MR scenario can be a tactical operation involving a room clearing mission that is part of a city combat operation. According to some embodiments, the rooms to be cleared can be located on different floors of the same building, the MR scenario can cause participant B to turn around to climb the stairs and clear the room located on the first floor, while participant C can be ordered to clear the room located on the ground floor. According to some embodiments, the MR scenario can include stairs or doors as part of the object space layout 22, wherein the stairs can provide participant B with a realistic feeling of climbing to the first floor, and the doors can provide participants B and C with a realistic feeling of entering the building.

[0105] According to some embodiments, participants B and C are immersed in a VR scenario that enables each of them to perform similar tasks within a modified object space 20, for example on different floors of the same building (the participant representations on the object space 20 are marked as participant B’ and participant C’, respectively). According to some embodiments, participants B and C do not have direct line of sight between each other. In this case, participant B can communicate with participant C through the auditory device 204 (and vice versa). The foregoing MR scenario enables at least two participants to explore and perform similar tasks within a modified object space 20, while being physically located within the same limited source space 24, or alternatively, within two separate limited source spaces 24.

[0106] Reference is made to Figure 3which constitutes a schematic perspective view of an MR scenario according to some embodiments of the application, performed by using the MR immersive system 10. As shown, at least two participants B and C occupy a limited source space 24. According to some embodiments, the controller 100 can perform dynamic morphing to create an MR scenario comprising a vast object space 20 having more than one object sub-space, or in other words, a modified object space 20. For example, the MR modified scenario can be a tactical operation involving a building clearing mission as part of a urban warfare operation. According to some embodiments, the rooms to be cleared can be located on different floors of the same building, the MR scenario can direct participant D to climb the stairs and clear the room located on the first floor, while participant E can turn to scout through the window of the room located on the ground floor. According to some embodiments, the MR scenario can comprise stairs, doors or windows as part of the object space layout 22, wherein the stairs can provide participant D with a realistic feeling of climbing, while the door window can provide participant E with a realistic feeling of entering and scouting the building environment.

[0107] According to some embodiments, participants D and E are immersed in an MR scenario, each participant being able to perform different tasks on the modified object space 20, for example on different floors of the same building (the representation of the participants on the object space 20 are marked as participant D’ and participant E’ respectively). According to some embodiments, participants D and E can not have a direct line of sight towards each other. In this case, participant D can communicate with participant E through the hearing device 204 (and vice versa). The aforementioned MR scenario enables at least two participants to perform similar tasks within the modified object space 20, while being physically located within a limited source space 24, or alternatively, within two separate limited source spaces 24.

[0108] Reference is made to Figure 4which constitutes a schematic perspective view of an MR scenario according to some embodiments of the application, performed by using the MR immersive system 10. As shown, at least two participants F and G are occupying a limited source space 24. According to some embodiments, the controller 100 can perform dynamic morphing to create at least two vast object spaces 20 and 20'. According to some embodiments, the first object space 20 can be a room to be cleared as part of a urban warfare operation by performing a tactical mission. The room to be cleared can be located on the ground, accessible through a walkway. According to some embodiments, the second object space 20' can be the surrounding environment of the room and can require the walkway to be scouted. According to some embodiments, the controller 100 can turn participant F to enter the room through the walkway and clear the room, while, at the same time, the controller 100 can turn participant G to scout the walkway. According to some embodiments, the MR scenario can include a walkway, a door and a rock as part of the object space layout 22, wherein the objects can provide the participants with a realistic feeling of war clearing or outdoor scouting.

[0109] According to some embodiments, participants F and G are immersed in a VR scenario that enables them to perform different tasks on completely different object spaces (the representations of the participants on the object spaces 20 and 20' are marked as participant F' and participant G', respectively). According to some embodiments, participants F and G do not have a direct line of sight between each other. In this case, participant F is able to communicate with participant G through the hearing device 204 (and vice versa). The aforementioned MR scenario enables at least two participants to perform different tasks in completely different environments, while being physically in a limited source space 24, or alternatively, in two separate limited source spaces 24.

[0110] According to some embodiments, the MR immersive system 10 is a cost-effective system that uses off-the-shelf components, such as, for example, head-mounted displays (HMDs) of manufacturers such as HTC or Oculus (e.g. HTC Vive® Oculus Oculus etc.).

[0111] According to some embodiments, the MR immersive system 10 enables real-time supervisory feedback, which can dynamically change the MR scenario according to certain considerations, such as, for example, a supervisor observing participants immersed in and operating within the MR scenario having a real-time influence on the MR scenario narrative and development. After performance evaluation of each participant, the supervisor can decide to change the object space layout 22 and / or the object space 20 to observe the participant's reaction or allow the participant to repeat the task until it is performed satisfactorily.

[0112] According to some embodiments, the MR immersive system 10 enables an on-the-spot debriefing option. Immediately after or while immersed in the MR scenario. According to some embodiments, the debriefing can be conducted using the auditory device 204 or as part of the MR scenario itself, with the supervisor acting as part of the MR scenario or represented within the object space 20.

[0113] According to some embodiments, the current system can benefit from using as many real accessories as possible (e.g. real guns, consoles, tools, etc.), or at least being oblivious to a large number of real-world objects within the source space 24.

[0114] Reference is made to Figure 5 which constitutes a flowchart of a method of conducting a mixed reality scenario using the MR immersive system 10, according to some embodiments of the present application. In operation 302, the method can include using the controller 100 to operate an MR scenario for at least one participant within a confined source space 24, which is an actual physical space that the participant occupies and operates in. In operation 304, the method can include dynamically morphing the spatial characteristics of the confined source space 24 to form a vast (or spacious) and spatially-altered object space 20. In operation 306, the method can include sensing parameters of the at least one participant while immersed within the object space 20. These parameters can include orientation, vital signs or other parameters such as body temperature, perspiration level, respiration rate, hyperventilation, brain waves, nervous system rate, etc. In operation 308, the method can include transmitting the object space 20 to at least one sensory device associated with the at least one participant. The sensory device can be, for example, an HMD configured to display 3D images to the participant.

[0115] According to some embodiments, a plurality of MR scenarios and object spaces 20 can be simulated while the participant is physically in the source space 24. This process can be obtained through mathematical reproduction of stationary and dynamic real and virtual objects in the source space 24 and the object space 20, and can include identifying actual and potential collisions between such objects. This process can also be obtained through Minkowski addition analysis using two-dimensional (2D) approximations of the three-dimensional objects, as exemplified below.

[0116] According to some embodiments, a general mathematical representation can describe some embodiments of the present application, e.g. P represents a source space, n represents the number of participants immersed in an MR scenario X and occupying the source space P. The same source space can be used for several MR scenarios at the same time, but cannot contain more than one participant standing at the exact physical location at the same time - i.e. one participant cannot be at the exact point in the source space currently occupied by another participant. According to some embodiments, real-life accessories / tools are also subject to the same location restrictions.

[0117] According to some embodiments, m represents the number of shapes forming the object space layout of the object space that is part of an MR scenario. According to some embodiments, the shapes can be static or dynamic objects or entities, e.g. static shapes such as walls, cars, etc., or dynamic entities such as animals or enemy soldiers (which can carry weapons or other operating accessories).

[0118] According to some embodiments, k represents the number of MR scenarios S(1...k), where each MR scenario S is modeled by a polygon representation Pi and includes m shapes, several participants and a list of rules. The MR scenario rules correspond to the geometric restrictions of the source space. By way of example, participant 1 must keep a certain distance from the real wall that is part of the source space during the MR scenario. According to some embodiments, all rules relate to the physical world restrictions of the source space and their effect on the corresponding object space during the MR scenario progress, and are defined using geometric terms such as distance, surface, radius, etc.

[0119] According to some embodiments, the MR scenarios can run simultaneously, where the polygon representations Pi(1...k) of the MR scenarios are contained in the source space P and the MR scenarios do not interfere with each other, and according to some embodiments, the shapes m do not collide with each other or with the participants, the participants do not collide with each other, thus maintaining the scenario rules.

[0120] Reference is made to Figure 6 which constitutes a diagram illustrating a source space and an object space and the blocks forming an MR scenario performed by using an MR immersive system 10, according to some embodiments of the present application. Those skilled in the art will appreciate that embodiments of the aforementioned general mathematical representation, which can include space changes, shape transformations and other dynamic formations, can be implemented by various common mathematical means that dynamically deform the object space and / or transform the object space into available source spaces (available because some source spaces can already be occupied by other MR operations / scenarios or physical objects). According to some embodiments, such deformations will be performed taking into account important scenario properties and their predefined scoring (as illustrated below in Figure 7 ).

[0121] According to some embodiments, more important situational properties will be preserved while the object space and / or object chunks are mathematically deformed to fit the source space. For example, the object space 620 includes many potential object chunks 630, whereby the object space 620 and object chunks 630 can be dynamically deformed to fit the source space 610. This adaptation and selection of chunks to be deformed can be operated by a selection method (basically illustrated in Figure 7 the following

[0122] According to some embodiments, some chunks 630 and their required warping are pre-defined and stored using a controller 100 (not shown) to achieve low system latency and to simulate real-time user experience. According to some embodiments, the chunks 630 and their required warping will be continuously calculated according to the decision 720 (basically illustrated in the following Figure 7 ), saving system resources. According to some embodiments, the number, type, characterization required warping, etc. of chunks 630 are optimized while reaching a balance between the desired real-time user experience and system resource limitations or capacity.

[0123] Referring to Figure 7 , which constitutes a flowchart illustrating an example of a method for object chunk selection using important situational property scores during an MR scenario performed by using an MR immersive system 10, according to some embodiments of the present application. According to some embodiments, a feature or a specific situation that is considered indispensable for the completion of an operation can be defined as an “important situational property”. For example, a constant and direct line of sight between participants from within a room towards a recon participant on a walkway can be defined as an important situational property (which for example allows symbolic communication between participants). Another example of an important situational property can be a firing distance from a participant to a target compared to a known range of a certain weapon. Another example of an important situational property can be an upcoming change in the simulated virtual environment, for example an unexpected turn in a tunnel that a trainee is mitigating. Another example of an important situational property can be a priority of a target or action of different participants.

[0124] According to some embodiments, due to participant operation 711 in the source space being initiated by the experience object block 710 and the participant operating in the object block, the participant's said operation is evaluated against source space limits 712, which can provide for the use of hard stop instructions 713 (e.g. preventing collision with a physical wall), and prevent "soft" stops of the user 714 initiated by MR limit scores 715. According to some embodiments, the MR limit scores 715 are computed from important situational parameters 716 (which can incorporate various matters and issues, such as real-time evaluation considerations 717b, dynamic analysis of situational evolution 717a, etc.). The inputs to the hard stop 713 of the participant, the soft stop 714 of the participant, and other important situational parameters 716 are computed in order to rank other situational limits 719.

[0125] According to some embodiments, if the hard stop 713 of the participant has a dominant value (i.e. a clear collision with a physical obstacle), then the override 713a strongly influences the selection 720 of the next object block for user operation in the MR scenario to avoid adverse effects on the personal integrity of the user; otherwise, the ranked MR situational limits 719 are used to select the next object block 720 for user operation in the MR scenario.

[0126] According to some embodiments, dynamic warping can be obtained by a dimensional scaling method, which is characterized by finding the multiplication factors that will convert the dimensions from the object space to the desired dimensions in the source space. For example, if the available source space is a rectangle of dimensions X, Y, Z, and the object space is a rectangle of dimensions A, B, C, then the relevant scaling factors to fit the object space into the available source space would be X / A, Y / B, Z / C.

[0127] Referring to Figures 8a and 8b, which constitute an example of a 2D object space to a limited source space warping during an MR scenario conducted by using MR immersive system 10, according to some embodiments of the present application. The system can receive an object space and choose to apply a certain warping to it to fit it into a given source space. For example, assume that a given object space 802 is described by a cubic volume of size X, however, the source space 804 cubic volume is Y, where Y < X and where source space 804 is the only available. According to some embodiments, the system will check the defined important scenario properties. If no restrictive constraints are found in these properties, the system can choose to apply a warping to the cubic space of object space 802 and map it to a spherical sector bounded by a cubic volume of size Y, thus fitting the available source space. Such warping can be calculated using well known analytical geometry equations of arc length, circular sector length, etc. dimensions and taking into account the preservation that the length of the warped arch after warping is still X. In effect, each point within object space 802 has a corresponding coordinate in source space 804, as exemplified in Figure 8b.

[0128] While the application has been described with reference to specific embodiments thereof, the description is illustrative of the application and not restrictive thereof. Those skilled in the art will upon attaining an understanding of the foregoing description, have the various modifications of the disclosed embodiments and alternative embodiments of the application in mind. It is therefore intended that the appended claims shall cover all such modifications and alternatives as fall within the true spirit and scope of the application.

Claims

1. A mixed reality (MR) immersive system, the system comprising: (i) a controller configured to operate at least one MR scenario; (ii) at least one sensory device associated with at least one participant; the system being deployable in at least one source space physically occupied by the at least one participant, wherein the source space is dynamically morphed into at least one spatially changed object space by mathematically operating on a 3D shape matrix representation of the source space, and wherein the object space is transmitted to the at least one sensory device associated with the at least one participant, such that the at least one participant perceives through the sensory device both the spatially changed object space and the dynamically morphed source space simultaneously; wherein the MR scenario is configured to dynamically form an object space suitable for at least one participant.

2. The system of claim 1, wherein at least one sensor is configured to monitor a parameter of the at least one participant.

3. The system of claim 2, wherein the parameter of the participant comprises a vital sign of the participant.

4. The system of claim 2, wherein the parameter of the participant comprises an orientation of the participant.

5. The system of claim 1, wherein the controller is integrated into the sensory device.

6. The system of claim 1, wherein the sensory device comprises a head mounted display (HMD) configured to display images to the at least one participant.

7. The system of claim 1, wherein the sensory device comprises a communication device configured to enable communication between participants.

8. The system of claim 7, wherein the communication is configured to provide inter-participant and supervisor-participant communication.

9. The system of claim 1, wherein the sensory device comprises a haptic device configured to provide haptic stimulation to the participant.

10. The system of claim 1, wherein a safety mechanism prevents possible physical collisions between the participants themselves or between a participant and a physical object that can exist within the source space.

11. The system of claim 10, wherein the safety mechanism uses a pre-mapping of the source space.

12. The system of claim 1, wherein the object space is simulated by dynamically morphing a plurality of image tiles to fit the source space.

13. The system of claim 12, wherein characteristics of the plurality of image tiles are optimized to reduce latency and preserve system resources.

14. The system of claim 1, wherein the same source space can be used for several MR scenarios that are conducted simultaneously.

15. The system of claim 1, wherein the MR scenario is subject to constraints generated by measurable important scenario parameters.

16. The system of claim 15, wherein an MR restriction score is calculated according to the measurable important scenario parameters, which in turn affects the selection of the next object image tile that forms the object space presented to the participant.

17. The system of claim 1, wherein the shape of the object space can be warped to fit a given source space while preserving the measurements of the object space.

18. The system of claim 1, wherein the MR scenario is configured to dynamically form at least two object subspaces adapted for at least two participants.

19. The system of claim 18, wherein the at least two object subspaces are duplicates of the same object space type.

20. The system of claim 18, wherein the at least two object subspaces are different active site object subspaces.

21. The system of any one of claims 18, 19 or 20, wherein at least two participants can perform different tasks simultaneously while immersed in the at least two object subspaces.

22. The system of claim 18, wherein the at least two participants operating within the at least two object subspaces have real-time mutual communication.

23. The system of claim 1, wherein the at least one MR scenario is configured to dynamically correspond and adapt to each other according to scenario evolution and participant actions.

24. The system of claim 1, wherein the system further comprises real-life accessories.

25. The system of claim 24, wherein the accessories are equipped with at least one sensing device.

26. The system of claim 24, wherein the accessories are equipped with at least one haptic device.

27. The system of claim 24, wherein the accessories further comprise real-life patient handling tools.

28. The system of claim 26, wherein the haptic device is capable of simulating the feeling of handling and operating a real weapon.

29. The system of claim 26, wherein the haptic device is capable of simulating the feeling of being hit by another participant.

30. A method for performing a MR scenario, the method comprising the steps of: (i) operating at least one MR scenario using a controller, the at least one MR scenario involving at least one participant present within a confined source space, (ii) dynamically deforming spatial properties of the source space by mathematically operating on a 3D shape matrix representation of the source space to form a spatially altered object space, (iii) transmitting the at least one object space to at least one sensory device associated with at least one participant, such that the at least one participant perceives, through the sensory device, the spatially altered object space and the dynamically deformed source space simultaneously; wherein the spatial properties of the source space are dynamically deformed to form an object space adapted for at least one participant.

31. The method of claim 30, wherein the spatial properties of the source space are dynamically deformed to form spatially altered at least two object subspaces adapted for at least two participants.

32. The method of claim 31, wherein the at least two object subspaces are duplicates of the same object space type.

33. The method of claim 31, wherein the at least two object subspaces are different active site object subspaces.

34. The method of claim 31, wherein at least two participants can perform different tasks simultaneously while immersed in the at least two object subspaces.

35. The method of claim 30, wherein the steps can be performed using off-the-shelf components.

Images