Method and apparatus for efficiently rendering, managing, recording, and playing back interactive multi-user virtual reality experiences

Abstract

To provide an apparatus that, in some embodiments, comprises: a volatile memory; a non-volatile memory; a first processor coupled to the non-volatile memory and configured to receive user information associated with a dataset associated with an object and a spatial location of a user in a multi-user virtual (IMVR) environment; and a second processor coupled to the volatile memory and configured to render an instance of the portion of the object in the IMVR environment from a user perspective based on the spatial location of the user.SOLUTION: A first processor is configured to generate a look-up table (LUT) based on a set of inputs received from a user. A second processor is configured to render an updated instance of the portion of an object that reveals the effect of an operation in an IMVR environment based on the LUT and information related to the operation by the user.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to and the benefit of U.S. Provisional Application No. 62 / 690,879, filed June 27, 2018, entitled "Method and Apparatus for Efficiently Rendering, Managing, Recording, and Playing Back Interactive Multi-User Virtual Reality Experiences," and U.S. Provisional Application No. 62 / 690,880, filed June 27, 2018, entitled "Method and Apparatus for Efficiently Rendering, Managing, Recording, and Playing Back Interactive Multi-User Virtual Reality Experiences," each of which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates to data processing and distribution, and in particular to the rendering, management, recording, and playback of interactive multi-user virtual reality presentations and experiences. [Background technology]

[0003] A computer may include a keyboard and mouse and a program that responds to user commands from the keyboard or mouse. Such a computer may also provide feedback to the user via a television screen or computer monitor. Summary of the Invention

[0004] In some embodiments, an apparatus includes: a volatile memory; a non-volatile memory; a first processor operatively coupled to the non-volatile memory and configured to receive a dataset associated with an object and user information related to the object and associated with a user's spatial location in a virtual environment (IMVR environment), the dataset including a spatial data sequence of the object; and a second processor operatively coupled to the volatile memory, the second processor configured to render an instance of a portion of the object in the IMVR environment based on the lookup table and the user information, the instance of the portion of the object being from a viewpoint of the user, the viewpoint being based on the user's spatial location. The first processor is configured to receive a set of inputs from the user, the set of inputs being associated with a set of predetermined properties associated with the portion of the object, and the first processor is configured to generate the lookup table based on the set of inputs. The second processor is configured to receive manipulation information related to a manipulation of the instance of the portion of the object by the user, and the second processor is configured to render an updated instance of the portion of the object manifesting an effect of the manipulation in the IMVR environment based on the manipulation information.

[0005] Some embodiments described herein include a non-transitory processor-readable medium storing code representing instructions executed by a processor, the code including code for receiving data, including but not limited to, a medical image associated with an object, and extracting a dataset associated with a structure (e.g., bones, muscles, etc.) of the object and user information associated with a spatial location of the user with respect to the object in a virtual environment. The code may further include code for causing the processor to receive a set of inputs from a user, the set of inputs associated with a set of predetermined properties associated with portions of the object in the virtual environment, render a set of instances of portions of the object in the virtual environment based on the set of inputs and the user information, receive manipulation information related to a manipulation of the rendered instances of the portions of the object by the user based on the virtual environment, and render updated instances of the portions of the object that manifest an effect of the manipulation in the virtual environment based on the manipulation information. The code may also include code for causing the processor to record media associated with the manipulation and rendering of the updated instances of the portions of the object, the medium including audio and spatial information associated with the rendering of the updated instances of the portions of the object from a viewpoint of the user, the viewpoint based on the spatial location of the user.

[0006] Some embodiments described herein include a method that includes receiving, from a first device, at a second device: (1) a data package associated with a first instance of a portion of an object in a virtual environment, the data package including a data set associated with a three-dimensional structure of the object; (2) first user information including a spatial location of a first user associated with the first device with respect to the object in the virtual environment; and (3) a first set of inputs associated with a set of predetermined properties of the object. The method includes the steps of: (1) receiving, at the second device, second user information including a spatial position of a second user associated with the second device with respect to an object in the virtual environment; (2) converting from a first set of inputs to a second set of inputs based on the second user information; (3) generating a lookup table based on the second set of inputs; and (3) rendering, at the second device, a second instance of a portion of the object in the virtual environment based on (i) the dataset, (ii) the second set of inputs, and (iii) the lookup table, wherein the second instance of the portion of the object is from a field of view of the second user, and the field of view is based on the spatial position of the second user.

[0007] Virtual reality systems have a wide range of applications, including entertainment, education, healthcare, business operations, manufacturing, and resource management. Some of these applications can benefit from a collaborative user experience by two or more users. Providing an interactive user experience, where the virtual experience can be manipulated and the results of the manipulation can be experienced by the user, also offers significant benefits.

[0008] The disclosed embodiments therefore address the need for improved apparatus and methods for efficient, managed, recordable, and / or replayable interactive multi-user virtual reality presentations / environments, and in particular, multi-user virtual reality experiences that are efficiently rendered and maintained interactively for multiple users in real-time and / or near real-time in some embodiments.

[0009] In some embodiments, the device includes a memory and a processor. In some embodiments, the system can include multiple devices. The processor is configured to receive information that can be used to generate the interactive multi-user virtual reality environment. The processor is configured to initialize a state machine to define the interactive multi-user virtual reality environment, render the interactive multi-user virtual reality environment based at least in part on the state machine, communicate with one or more user devices connected to the interactive multi-user virtual reality system, and distribute the rendered interactive multi-user virtual reality environment and / or information related to the state machine defining the interactive multi-user virtual reality environment. The processor is further configured to receive information related to the state of the one or more user devices, receive information related to changes induced in the rendered interactive multi-user virtual reality environment by one or more user actions, and update the rendered interactive multi-user virtual reality environment and / or the state machine defining the interactive multi-user virtual reality environment based on the received information. The processor is configured to provide change parameters related to changes in the state of the interactive multi-user virtual reality environment in an efficient, low-bandwidth manner. The processor is further configured to record portions of the interactive multi-user virtual reality environment and enable playback of one or more recorded portions by one or more users. [Brief explanation of the drawings]

[0010] [Figure 1A] FIG. 1 is a schematic diagram of an interactive multi-user virtual reality (IMVR) system according to one embodiment. [Figure 1B] FIG. 1 is a schematic diagram of an interactive multi-user virtual reality (IMVR) system according to one embodiment. [Figure 2] 1 is a schematic diagram of a user device in an IMVR system according to one embodiment. [Figure 3]FIG. 1 is a schematic diagram of an interactive multi-user VR renderer in an IMVR system according to one embodiment. [Figure 4] 1 is a flowchart illustrating a method for initializing, rendering, and updating an IMVR system using an IMVR renderer, according to one embodiment. [Figure 5] 1 is a flowchart illustrating an exemplary method for rendering a volumetric environment portion of an IMVR experience, according to one embodiment. [Figure 6] 1 is a flowchart illustrating an exemplary method of object manipulation by one or more users in an IMVR environment, according to one embodiment. [Figure 7] 1 is a flowchart illustrating an exemplary method for efficiently recording an IMVR experience when initiated by a user of an IMVR system, according to one embodiment. [Figure 8] 1 is a flowchart illustrating an exemplary method for playing back a recording of an IMVR experience initiated by a user of an IMVR system, according to one embodiment. [Figure 9A] FIG. 10 illustrates an example portion of code that, when executed, can cause a processor to generate a portion of an IMVR experience using an IMVR system, according to one embodiment. [Figure 9B] FIG. 10 illustrates an example portion of code that, when executed, can cause a processor to generate a portion of an IMVR experience using an IMVR system, according to one embodiment. [Figure 10] 1 is a flowchart illustrating an exemplary method for synchronizing the IMVR experiences of two or more users of an IMVR system by establishing an integrated IMVR environment, according to one embodiment. [Figure 11] 1 is a flowchart illustrating an exemplary method of using MEG data to generate an IMVR experience using an IMVR system, according to one embodiment. [Figure 12]FIG. 1 is a schematic diagram of a subset of variables with values ​​set based on user input used to render an IMVR environment, according to one embodiment. [Figure 13] 1 is a flowchart illustrating an exemplary method for generating an IMVR environment and recording media associated with the IMVR environment by an IMVR system, according to one embodiment. [Figure 14] 1 is a flowchart illustrating an example method for receiving data associated with a first instance of an object in an IMVR environment and using the data to render a second instance of the object to enable a first user to share the IMVR environment with a second user. [Figure 15] 1 is a flowchart illustrating an exemplary method for rendering a volumetric object in an IMVR environment by an IMVR system, according to one embodiment. [Figure 16] 1 is a flowchart illustrating an exemplary method for rendering a volumetric object in an IMVR environment, according to one embodiment. [Figure 17A] 1 shows a flowchart illustrating an exemplary method for shading a volumetric object rendered in an IMVR environment by an IMVR system, according to one embodiment. [Figure 17B] 1 shows a flowchart illustrating an exemplary method for shading a volumetric object rendered in an IMVR environment by an IMVR system, according to one embodiment. [Figure 18] 1 is a flowchart illustrating an exemplary method for rendering a volumetric object in an IMVR environment using a ray marching approach by an IMVR system, according to one embodiment. [Figure 19A] 1 is a flowchart illustrating an example method for using a depth buffer to render objects in an IMVR environment based on a user's spatial position, according to one embodiment. [Figure 19B]1 is a flowchart illustrating an example method for using a depth buffer to render objects in an IMVR environment based on a user's spatial position, according to one embodiment. [Figure 20] FIG. 1 is a schematic diagram of a sequence of user actions relating to a user's selection to capture a cross-sectional view of an IMVR environment, the user's access to a three-dimensional mesh object associated with a volume object rendered in the IMVR environment, and functions invoked in the IMVR system to implement changes desired by the user actions in the IMVR environment, according to one embodiment. [Figure 21] FIG. 1 is a schematic diagram of a sequence of user actions relating to a user's selection to capture a cross-sectional view of an IMVR environment, the user's access to a three-dimensional mesh object associated with a volume object rendered in the IMVR environment, and functions invoked in the IMVR system to implement changes desired by the user actions in the IMVR environment, according to one embodiment. [Figure 22] 10 is a flowchart illustrating an implementation of capturing a cross-sectional view of an object in an IMVR environment, according to another embodiment. [Figure 23] 10 is a flowchart illustrating an implementation of capturing a cross-sectional view of an object in an IMVR environment, according to another embodiment. [Figure 24] 1 is a flowchart illustrating an exemplary method for rendering a colored view of an object in an IMVR environment by an IMVR system, according to one embodiment. [Figure 25] 1 is a flowchart illustrating an exemplary method for generating a composite view of an object in an IMVR environment obtained from two or more sources by an IMVR system, according to one embodiment. [Figure 26]FIG. 1 is a schematic diagram of an IMVR system configured to manage multiple instances of an object rendered in an IMVR environment, according to one embodiment, to allow an instructor to flexibly join or leave a set of shared IMVR sessions among students implementing the IMVR system. [Figure 27] FIG. 1 is a schematic diagram of a method for sharing data related to an IMVR environment between a host machine and a client machine using an IMVR system, according to one embodiment. [Figure 28] 1 is a flowchart illustrating an exemplary method for generating a scaled volumetric representation of an object in an IMVR environment using an IMVR system, according to one embodiment. [Figure 29A] 1 shows a flowchart illustrating an implementation of an example tool related to rendering volumetric objects in an IMVR environment using an IMVR system, according to various embodiments. [Figure 29B] 1 shows a flowchart illustrating an implementation of an example tool related to rendering volumetric objects in an IMVR environment using an IMVR system, according to various embodiments. [Figure 30] 1 shows a flowchart illustrating an implementation of an example tool related to rendering volumetric objects in an IMVR environment using an IMVR system, according to various embodiments. [Figure 31A] 1 shows a flowchart illustrating an implementation of an example tool related to rendering volumetric objects in an IMVR environment using an IMVR system, according to various embodiments. [Figure 31B] 1 shows a flowchart illustrating an implementation of an example tool related to rendering volumetric objects in an IMVR environment using an IMVR system, according to various embodiments. [Figure 31C] 1 shows a flowchart illustrating an implementation of an example tool related to rendering volumetric objects in an IMVR environment using an IMVR system, according to various embodiments. [Figure 32A] 1 shows a flowchart illustrating an implementation of an example tool related to rendering volumetric objects in an IMVR environment using an IMVR system, according to various embodiments. [Figure 32B] 1 shows a flowchart illustrating an implementation of an example tool related to rendering volumetric objects in an IMVR environment using an IMVR system, according to various embodiments. [Figure 33A] 1 shows a flowchart illustrating an implementation of an example tool related to rendering volumetric objects in an IMVR environment using an IMVR system, according to various embodiments. [Figure 33B] 1 shows a flowchart illustrating an implementation of an example tool related to rendering volumetric objects in an IMVR environment using an IMVR system, according to various embodiments. [Figure 34] 1 is a flowchart illustrating an exemplary method for annotating a representation of an object in an IMVR environment using an annotation tool of an IMVR system, according to one embodiment. [Figure 35] 1 is a flowchart illustrating an exemplary method for generating and using scaled mesh model representation objects in an IMVR environment using an IMVR system, according to one embodiment. [Figure 36A] FIG. 1 is a schematic diagram of an exemplary data package that may be transferred to a device to enable a user associated with the device to participate in an IMVR environment using an IMVR system, according to one embodiment. [Figure 36B] 1 is a flowchart illustrating a sequence of events involving a user joining an ongoing IMVR session, according to one embodiment. [Figure 37] 1 is a flowchart illustrating an example implementation of a method for indicating a user's readiness state when transferring data packages between users in an IMVR session using an IMVR system, according to one embodiment. [Figure 38] 1 is a flowchart illustrating an exemplary method for managing data packages and merging information related to a new user with an ongoing session of an IMVR experience when a user is authorized to join an ongoing IMVR session using an IMVR system, according to one embodiment. [Figure 39] 1 is a flowchart illustrating an exemplary method for managing a user interface associated with an IMVR session involving multiple concurrent users using an IMVR system, according to one embodiment. [Figure 40] 1 is a flowchart illustrating an example implementation of a method for allowing a user to join one of any number of ongoing IMVR sessions, according to one embodiment. [Figure 41A] FIG. 2 is a schematic diagram of an exemplary user interface with a set of object properties represented by a set of keys operable by a user via the user interface, according to one embodiment. [Figure 41B] FIG. 41B is a schematic diagram of exemplary keys associated with the user interface of FIG. 41A used by an IMVR system to render a volume object, according to one embodiment. [Figure 42A] 1 is a flowchart illustrating an exemplary method for a user to select and transmit a new channel of data over a network to a user device. [Figure 42B] 1 is a flowchart illustrating an exemplary method for implementing user interaction with an IMVR environment through key adjustments and propagating the key adjustments to user devices within an IMVR system using a strategy of minimal data transfer over a network, according to one embodiment. [Figure 42C] 1 is a flowchart illustrating an exemplary method for generating a colorized rendering of a volumetric object using a set of keys in an IMVR system, according to one embodiment. [Figure 43]1 is a flowchart illustrating an exemplary method for generating a colored rendering of a volumetric object by interpolating information associated with a set of keys, according to one embodiment. [Figure 44A] FIG. 1 is a schematic diagram of different types of events associated with an IMVR system. [Figure 44B] FIG. 1 is a schematic diagram of an exemplary read and write event-based configuration of an IMVR system according to one embodiment. [Figure 45] 1 is a flowchart illustrating an exemplary method for generating and processing defined events associated with an IMVR session, according to one embodiment. [Figure 46A] FIG. 1 is a diagram of a teardrop-shaped model representation object. [Figure 46B] FIG. 46B is a diagram of a perspective view of an IMVR environment including a representation of data obtained via magnetoencephalography using several instances of the model representation object shown in FIG. 46A, according to one embodiment. [Figure 47A] FIG. 1 is a schematic diagram of an exemplary user interface of an IMVR system configured to allow a user to navigate through datasets associated with MEG data and renderings of objects included in the MEG data, according to one embodiment. [Figure 47B] 47B is a flowchart illustrating an exemplary method for implementing functionality in the user interface of FIG. 47A. [Figure 48] 1 is a flowchart illustrating an exemplary method for implementing procedures associated with reading an MEG dataset, extracting information from the MEG dataset, and rendering instances of objects contained in the MEG dataset, according to one embodiment. [Figure 49] 1 is a flowchart illustrating an exemplary method for implementing frame-by-frame updates to the rendering of an object based on an MEG dataset using an IMVR system, according to one embodiment. [Figure 50]1 is a flowchart illustrating an exemplary method for implementing object rendering based on MEG data using an IMVR system, according to one embodiment. [Figure 51A] FIG. 1 illustrates an exemplary embodiment of an IMVR system used for volumetric and visualization of gemstones and / or metals using spatial data sequences obtained via CT, according to one embodiment. [Figure 51B] FIG. 1 illustrates an exemplary embodiment of an IMVR system used for volumetric and visualization of gemstones and / or metals using spatial data sequences obtained via CT, according to one embodiment. [Figure 51C] FIG. 1 illustrates an exemplary embodiment of an IMVR system used for volumetric and visualization of gemstones and / or metals using spatial data sequences obtained via CT, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiments described herein relate to methods and apparatus for efficiently rendering and updating interactive multi-user virtual reality (VR) environments provided to multiple users via multiple user devices connected within a multi-user virtual reality (VR) system. The present disclosure includes methods, systems, and apparatus for rendering, managing, recording, and playing back interactive multi-user VR presentations and experiences, including, for example, medical data. Such medical data can be converted for VR and can include multiple sources (e.g., computed tomography (CT), magnetic resonance imaging (MRI), electroencephalography (EEG), magnetoencephalography (MEG), ultrasound images, etc.). In some implementations, the medical data can be near real-time or real-time, and interactions between multiple users can be similarly coordinated. Embodiments can be configured to reduce bandwidth used when multiple users are interacting and manipulating data in a shared VR environment. Additionally or alternatively, embodiments can be configured to reduce storage space used when a multi-user session is recorded, so that subsequent playback by one or more users uses reduced processing power and / or bandwidth.

[0012] Described herein are methods and apparatus for efficient rendering and maintenance of interactive multi-user virtual reality environments, which can be used in a variety of applications, including, but not limited to, education, training, entertainment, research, diagnosis, surgery, patient care management, repair technology, the use of prosthetic devices in healthcare, manufacturing and repair in the mechanical and automotive industries, and the like. In some embodiments, the apparatus includes a memory and a processor operably coupled to the memory. The processor can be configured as part of a system including one or more user devices. The processor is configured to render an interactive multi-user virtual reality (IMVR) environment and efficiently distribute information related to the rendered IMVR environment to the user devices, such that users can connect to the user devices and experience and / or interact with the rendered IMVR environment. The processor can be configured to efficiently receive input from the user devices, e.g., input related to user interactions. The system is further configured to enable the processor to efficiently transfer information related to changes in the state of the IMVR environment, including changes caused by user interactions, to the user devices. Furthermore, the user devices can also be configured to efficiently transfer information to the processor and / or to each other, enhancing the user experience of the IMVR environment.

[0013] FIG. 1A is a schematic diagram of an interactive multi-user virtual reality system 100, also referred to herein as an “IMVR system” or “system.” IMVR system 100 is configured to render and maintain an interactive multi-user virtual reality environment accessed by a set of user devices 101-103 by building, initializing, and maintaining a synchronized, interactive, three-dimensional IMVR environment. According to one embodiment, IMVR system 100 can generate the three-dimensional interactive virtual environment from any suitable data provided as the basis for the environment. IMVR system 100 includes user devices 101, 102, and 103 connected to an interactive multi-user virtual reality renderer 105 (also referred to as a “renderer”) via a communications network 106, as shown in FIG. 1A.

[0014] Data suitable for use in rendering an IMVR environment can include any spatial data series containing an object. The spatial data series can, in some cases, include information defining, among other things, the volume of the object, and in some cases, the spatial data series can include information defining the structure (e.g., three-dimensional structure) of the object. The data series can include serial cross-sectional images of the object acquired with any mode of optical microscopy. The data series can include a series of multidimensional data, including image or point cloud data. Examples include data collected using computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography, confocal images, optical microscopy, electron microscopy, electroencephalography (EEG), magnetocardiography (MCG), magnetoencephalography (MEG), and / or ultrasound imaging. In some cases, the spatial data series can include data collected using multiple techniques, technologies, and / or data sources. For example, the spatial data series can include data collected from multiple different data sources and loaded into different channels used by the IMVR renderer 105. For example, CT data can be input into a first channel and MRI data can be input into a second channel. An image can then be rendered in an IMVR environment as described herein using both the CT and MRI data. In other examples, any other combination and / or number of data sources can be used as input.

[0015] Spatial data series can include objects related to human or animal anatomical structures, objects used for research, diagnosis, treatment, veterinary research of mammalian and / or non-mammalian species, botanical research, other living or non-living tissues (e.g., in vivo, ex vivo, in natural situ, etc.), and natural or synthetic organisms. While described herein using medical images as examples, the systems, devices, and methods described herein can be used to visualize any object using spatial data series containing information related to the object's structural characteristics. Other examples of objects are gemstones, samples from geological studies, samples from archaeological studies, samples from construction work, etc. Spatial data series can, in some cases, include information related to one or more cellular or subcellular object structures of molecules (e.g., macromolecules, proteins, etc.). For example, spatial data series can include data related to the structure and / or surface features of an enzyme molecule modeled and / or generated in three dimensions from an appropriate source, e.g., a protein structure prediction server (e.g., an advanced protein secondary structure prediction server). Spatial data series can, in some cases, include information related to atoms or subatomic particles. The spatial data series, in some cases, may include information related to electronically generated objects, such as mesh objects rendered by an artist. In some cases, the objects may be generated in the IMVR environment, as described herein, using tools provided by the IMVR system, as described herein. In some cases, the electronically generated objects may be obtained or imported from an external source (e.g., a suitable commercially available platform for generating two-dimensional and / or three-dimensional models of objects) or other suitable data processing environment, such that the spatial data series may be obtained from an appropriate data format.

[0016] The communications network 106 (also referred to as a "network") may be any suitable data communications path for transferring data, capable of operating over public and / or private networks. The communications network 106 may include a private network, a virtual private network (VPN), a multiprotocol label switching (MPLS) circuit, the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a worldwide interoperability of microwave access networks (WiMAX), an optical fiber (or fiber optic)-based network, a Bluetooth network, a virtual network, and / or any combination thereof. In some cases, the communications network 106 may be a wireless network, such as, for example, a Wi-Fi or wireless local area network ("WLAN"), a wireless wide area network ("WWAN"), and / or a cellular network. In other examples, the communications network 106 may be a wired connection, such as, for example, an Ethernet network, a digital subscriber line ("DSL") network, a broadband network, and / or an optical fiber network. In some cases, the network may use an application programming interface (API) and / or a data exchange format (e.g., Representational State Transfer (REST), JavaScript® Object Notation (JSON), Extensible Markup Language (XML), Simple Object Access Protocol (SOAP), etc.), and / or Java Message Service (JMS). Communications may be encrypted or unencrypted. In some cases, the communications network 106 may include multiple networks or sub-networks operatively coupled to each other by, for example, network bridges, routers, switches, and / or gateways.

[0017] Each of user devices 101, 102, and 103 in IMVR system 100 may be a device or suitable combination of devices interconnected with one another and associated with one or more users or groups of users. That is, user devices 101, 102, and 103 may be structurally and / or functionally similar to one another. While shown as including three user devices, any IMVR system described herein is not limited to two or three user devices and may support a shared IMVR experience via an IMVR session involving any number of users.

[0018] FIG. 1B is a schematic diagram of an interactive multi-user virtual reality system 100′, also referred to herein as an “IMVR system” or “system.” The IMVR system 100′ is configured to render and maintain an interactive, maneuverable multi-user virtual reality environment accessed by a set of user devices 101′-103′, each of which builds, initializes, and maintains a synchronized, interactive, three-dimensional IMVR environment. The IMVR system 100′ can generate the three-dimensional, interactive virtual environment from any suitable data provided as the basis for the environment. The IMVR system 100′ includes user devices 101′, 102′, and 103 connected to each other in a peer-to-peer network via a communication network 106′, as shown in FIG. 1B.

[0019] 1B may be substantially similar to IMVR system 100 of FIG. 1A, except that IMVR system 100′ does not include an IMVR renderer (e.g., IMVR renderer 105) as a separate device as IMVR system 100. Instead, user devices 101′-103′, which are substantially similar to user devices 101-103, may be further configured to create, initialize, and maintain a synchronized, interactive IMVR environment in a peer-to-peer manner without the use of a server (such as IMVR renderer 105 of IMVR system 100). Although devices, apparatuses, and methods described herein may refer to an IMVR renderer (methods or functions performed by an IMVR renderer, e.g., IMVR renderer 105, 305, etc.), in some embodiments, the IMVR system may be configured similarly to IMVR system 100′. The functions, methods, or processes described as being associated with the IMVR renderer may be executed or performed in a distributed manner using one or more components within a user device associated with the IMVR system (e.g., user devices 101′-103′ associated with the IMVR system 100′, etc.).

[0020] FIG. 2 illustrates an exemplary user device 201 that may be substantially similar to user devices 101, 102, and 103 of FIG. 1. User device 201 includes a user computing device 210, a virtual reality controller 219, and a virtual reality display 220. In some embodiments, VR controller 219 and VR display 220 may be separate devices connected to each other using appropriate wired or wireless connections. In other embodiments, VR controller 219 and VR display 220 may be integrated into a single device that includes a display and controller for an IMVR experience. Although not shown in FIG. 2, user device 201 may also include one or more peripheral sensing, receiving, or actuating devices, such as an audio receiver (e.g., a microphone) and / or an audio player (e.g., a speaker), a haptic or tactile sensor or actuator, an accelerometer, or other modalities, that may be interconnected with computing device 210, VR controller 219, and / or VR display 220.

[0021] The VR controller 219 may be any suitable device that can be configured to be used to control one or more components of the user device 201, such as the user computing device 210 and / or the VR display 220. In some embodiments, the VR controller 219 may be a mouse, keyboard, and / or joystick device. In some embodiments, the VR controller 219 may be an inertial measurement unit. In some embodiments, the VR controller 219 may be a motion controller including one or more sensors such as an accelerometer, a magnetometer, etc. The VR controller 219 may include one or more actuators or controls, such as buttons, scroll wheels, or handling sticks, that can be used to manipulate components of the user device 201 and / or the IMVR environment. As described herein, the user device 201 is illustrated as including one VR controller 219, but any suitable number of VR controllers may be used to manipulate the user device or to manipulate instances of objects rendered in association with the user device and / or the IMVR environment. For example, the VR controller 219 may be used to initiate an IMVR experience by initiating a user interface (UI) viewable through the VR display 220. The VR controller 219 can be used to change the UI, for example, by clicking buttons, opening menus, etc. The VR controller can be used to generate control items such as a cursor that can be used to point and manipulate devices in the user interface and / or IMVR environment.

[0022] In some cases, the IMVR environment can include one or more mobile entities generated from within the IMVR environment that can be used to obtain a two-dimensional view of the IMVR environment from within the IMVR environment and participate in an IMVR session. In some cases, a user operating a mobile entity can be an observer of the IMVR session from within the IMVR environment without participating in the IMVR session. For example, a mobile entity (e.g., a flycam) can be defined within the IMVR environment and used to project a perspective two-dimensional view (from the mobile entity's perspective) to one or more users. A user associated with the mobile entity can move, rotate, and / or orient the mobile entity as needed to view different areas or aspects of the IMVR environment.

[0023] In some embodiments, the VR display 220 may be a suitable two-dimensional monitor configured to receive data and generate a two-dimensional window into the IMVR environment for the user. In some embodiments, the VR display also includes one or more sensors and / or actuators that can be used to enhance or augment the user's IMVR experience, such as an audio speaker, a microphone for receiving audio input, a pupil tracker for tracking eye movements or the user's eye movements, etc.

[0024] The VR display 220 may be a suitable virtual reality headset configured to receive data and generate a virtual reality experience for the user. In some embodiments, the VR display may include a graphical display unit that may be configured to project 2D and / or 3D. In some embodiments, the VR display may also include one or more sensors and / or actuators that may be used to enhance or augment the user's VR experience, such as an audio speaker, a microphone for receiving audio input, a pupil tracker for tracking eye movements or the user's eye movements, sensors that recognize changes in the user's body posture and orientation (e.g., head movement, head orientation, gestures, etc.), sensors for capturing body movement and height, etc.

[0025] The user computing device 210 may be a hardware-based computing device and / or multimedia device, such as a server, a desktop computing device, a smartphone, a tablet, a wearable device, a laptop, etc. In some embodiments, the computing device 210 may be integrated with or part of a VR controller 219 and / or a VR display. In some embodiments, the computing device 210 may be housed separately from and connected to the VR controller 219 and the VR display 220. The computing device 210 includes a processor 211, a memory 212, and a communicator 213.

[0026] The processor 211 may be, for example, a hardware-based integrated circuit (IC) or any other suitable processing device configured to execute and / or execute a sequence of instructions or code. For example, the processor 211 may be a general-purpose processor, a central processing unit (CPU), an accelerated processing unit (APU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic array (PLA), a complex programmable logic device (CPLD), a programmable logic controller (PLC), etc. The processor 211 may be operably coupled to the memory 212 via a system bus (e.g., an address bus, a data bus, and / or a control bus).

[0027] 2, the processor 211 may include a data receiver 214, a user state machine 215, a user-specific VR renderer 216, a user state updater 217, a controller handler 218, and a VR recorder 221. Each component, including the data receiver 214, the user state machine 215, the user-specific VR renderer 216, the user state updater 217, the controller handler 218, and the VR recorder 221, may be software stored in the memory 212 and executed by the processor 211 (e.g., code for causing the processor 211 to execute the data receiver 214, the user state machine 215, the user-specific VR renderer 216, the user state updater 217, the controller handler 218, and the VR recorder 221). The code for executing the components may be stored in the memory 212 and / or in a hardware-based device such as, for example, an ASIC, an FPGA, a CPLD, a PLA, a PLC, etc.

[0028] The data receiver 210 is configured to receive data transmitted from a VR controller 219, a VR display 220, an IMVR renderer (e.g., the IMVR renderer 105 of the IMVR system 100 shown in FIG. 1 ) connected to the user device 201 via a communications network, and / or other user devices (e.g., user devices connected to the same IMVR system in which the user device 201 is included). In some implementations, the data receiver 214 can be configured to receive an initialized state indicating the state of an interactive, multi-user VR environment generated by the IMVR renderer. In some cases, the data receiver 214 can receive updates to the state of a VR environment already generated and initialized by the IMVR renderer. For example, the updates can represent changes in the relative positions and / or states of objects or other users in the VR environment, which may be from the interactions of other users connected to the IMVR system.

[0029] The user state machine 215 can be configured to generate and maintain a set of representations that represent the state of a user using the user device 201. In some implementations, the user state machine 215 can determine a user state based on input received from the user, such as input from the VR controllers 219, and / or other sensing, receiving, and / or actuating devices connected to the user (such as a microphone, haptic receiver, accelerometer, etc.), and maintain the user's state during the user's session of the VR experience.

[0030] The user-specific VR renderer 216 can be configured to use data received from a renderer that relays information about the IMVR environment (e.g., the renderer 105 of the IMVR system 100 shown in FIG. 1), and the user-specific VR renderer 216 maintains a user-specific IMVR experience for the user of the user device 201. In some embodiments, in an IMVR system that operates without a server with an IMVR renderer (e.g., the IMVR system 100′ of FIG. 1B), the user-specific VR renderer 216 can be configured to use directly received data relay information about the IMVR environment. The user-specific VR renderer 216 can be configured to generate and maintain a user-specific IMVR experience for the user of the user device 201. For example, the user-specific VR renderer 216 can receive information about the IMVR environment in which the user of the device 201 is participating and the relative actions, position, or viewpoint (e.g., spatial position, posture, interaction with the VR environment, etc.) of the user using the user device 201. Using this information, the user-specific VR renderer 216 can generate a user-specific version of the IMVR environment tailored to apply to the user of the user device 201. The user-specific version of the IMVR environment can include a presentation of the IMVR environment from the perspective of the user of the user device 201, including appropriate modifications to the visual, audio, haptic, and / or other modalities of the IMVR environment as perceived by the user's virtual presentation in the IMVR environment. For example, in some implementations, the user-specific VR renderer 216 can alter the view of the IMVR environment based on the user's spatial position estimated by tracking the positions of the VR controllers 219 and / or VR display 220. In some implementations, the user-specific renderer 216 can be configured to reveal certain aspects of the IMVR environment (e.g., particular objects or views within the IMVR environment) to the user of the user device 201, where the revealed objects would otherwise be unavailable to more other users (e.g., invisible or obstructed from view).

[0031] The processor 211 includes a user state updater 217 configured to receive information from the VR controller 219, the VR display 220, and / or other peripheral sensing, receiving, and / or actuating devices. The user state updater 217 is further configured to use the received information to record changes in the state of the user and / or the user's immediate environment, i.e., the user's actions or manipulations in the IMVR environment. The user state updater 217 uses the information to update the user state machine 215 and communicates the changes to the user state machine 215 and / or the IMVR renderer (e.g., the renderer 105 of the system 100 of FIG. 1A). In some implementations, the user state updater 217 can also be configured to communicate changes to the user state machine 215 to other users who may be connected to the same IMVR system. In some implementations, the user state updater 217 may be configured to communicate changes to the user state machine 215 directly to other user devices that may be connected to the same IMVR system in a peer-to-peer manner, without involving a server such as an IMVR renderer (e.g., in the IMVR system 100′ of FIG. 1B).

[0032] In some implementations, the user state updater 217 can also record changes to the IMVR environment caused by changes to the user state and communicate the changes to the IMVR renderer (e.g., the renderer 105 of the system 100) and / or other user devices (e.g., the user devices 101, 102, and 103 of the IMVR system 100). As an example, if a user of the user device 201 zooms in on a portion of the VR environment, such as a heart portion of the VR environment depicting the internal anatomical structure (e.g., the vasculature) of the subject's circulatory system, the user state updater 217 can record properties of the operation, such as the zoom point, the scale of the zoom, the axis along which the VR environment is zoomed in, etc. The user state updater 217 can update the user state machine 215 associated with the user's state (e.g., the control point used by the user, the user's gesture to zoom in, the position of one or more control objects, such as a cursor used by the user, etc.) and communicate the changes to the IMVR renderer. The user state machine may also record changes induced in the IMVR environment by user actions as perceived by the user and communicate the changes induced in the IMVR environment to the renderer. For example, the user state updater 217 may also communicate resulting changes in the magnification of particular structures or portions of the IMVR environment to the IMVR renderer and / or other user devices in the IMVR system.

[0033] The processor 211 includes a controller handler 218 configured to send commands and receive feedback from the VR controller 219. In some implementations, the controller handler 218 can be configured to generate and guide and / or point one or more control objects associated with the VR controller 219. In some implementations, the controller handler 218 can be configured to perform pointing and tracking procedures using the VR controller 219. In some implementations, the controller handler 218 can estimate focus and actions performed by the user via the VR controller 219 using one or more procedures such as ray casting, direction detection, rotation detection, detecting the position vector of the VR controller in an appropriate coordinate space, etc. In some implementations, the controller handler 218 can operate in coordination with one or more other components or portions of the components of the user device 201. For example, the controller handler 218 can operate in coordination with an eye-tracking device in the VR display 220 or other sensors and actuators (not shown in FIG. 2) that may be included. Although the user device 201 of FIG. 2 shows one VR controller 219, any number of suitable VR controllers may be included in the user device. For example, in some implementations, a user device may include two controllers, each guiding a cursor, that can be used to manipulate two access points in an IMVR environment or two access points in a rendered object instance associated with the user device to perform operations that may use two-point access, such as changing magnification, pivoting, rotating, etc.The controller handler 218 can also be configured to work in coordination with other components of the processor 211, such as the user state machine 215, the user state updater 217, and / or the user-specific VR renderer 216, to present and update the IMVR environment and communicate user-induced changes to the user of the user device 201, as well as to the IMVR renderer and / or other user devices in the IMVR system.

[0034] The processor 211 may include a VR recorder 221 configured to record, upon initiation by the user, at least a portion of the IMVR environment experienced by the user. Such a recording may be stored, for example, in memory 212 of the user computing device 211 for later playback.

[0035] The memory 212 of the user computing device 210 may include non-volatile and volatile memory, such as random access memory (RAM), video RAM (VRAM), memory buffers, a hard drive, read-only memory (ROM), erasable programmable read-only memory (EPROM), etc. In some embodiments, non-volatile memory may be associated with a processor, and volatile memory may be associated with a processor. In some embodiments, different processors (e.g., a first processor (CPU) and a second processor (GPU)) or portions of a processor may be associated with non-volatile memory and volatile memory, respectively. The memory 212 may store one or more software modules and / or code that may include instructions that, for example, cause the processor 211 to execute one or more processes, functions, etc. (e.g., the data receiver 210, the user state machine 215, the user-specific VR renderer 21, the user state updater 217, the controller handler 218, and / or the VR recorder 221). In some implementations, memory 212 can be configured to receive and store data transferred between an IMVR renderer (e.g., IMVR renderer 105 of system 100 of FIG. 1A), a VR controller 219, and / or a VR display 220. In some implementations, memory 212 can be configured to receive and store data transferred between one or more user devices (e.g., user devices 101′-103′ of system 100′ of FIG. 1B) in a peer-to-peer manner, in addition to and / or directly to a VR controller 219 and / or a VR display 220. In some implementations, memory 212 can be portable memory (e.g., a flash drive, a portable hard disk, etc.) that can be operably coupled to processor 211. In some cases, memory can be remotely coupled to user computing device 210. For example, a remote database server can be operably coupled to user computing device 210.

[0036] The communicator 213 may be a hardware device operatively coupled to the processor 211 and the memory 212, and / or software stored in the memory 212 that is executed by the processor 211. The communicator 213 may be, for example, a network interface card (NIC), a Wi-Fi® module, a Bluetooth® module, and / or any other suitable wired and / or wireless communication device. Furthermore, the communicator may include a switch, a router, a hub, and / or any other network device. The communicator 213 may be configured to connect the user computing device 210 to a communication network (such as the communication network 106 shown in FIG. 1). In some cases, communicator 213 may be configured to connect to a communications network such as, for example, the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a worldwide interoperability for microwave access networks (WiMAX®), a fiber optic (or optical fiber) based network, a Bluetooth® network, a virtual network, and / or any combination thereof.

[0037] In some cases, the communicator 213 may facilitate the receipt and / or transmission of files and / or sets of files over a communications network (e.g., communications network 106 in the IMVR system 100 of FIG. 1). In some cases, the received files may be processed by the processor 211 and / or stored in the memory 212, as described in further detail herein.

[0038] Returning to FIG. 1A, user devices 101-103 connected to IMVR system 100 can be configured to communicate with an IMVR renderer. FIG. 3 is a schematic diagram of an IMVR renderer 305 that is part of the IMVR system. IMVR renderer 305 can be structurally and functionally similar to renderer 105 of system 100 shown in FIG. 1. IMVR renderer 305 includes a server processor 351, a server memory 352, and a server communicator 353. As mentioned above, in some embodiments, the IMVR system can be configured to operate in a peer-to-peer manner without an IMVR renderer (e.g., IMVR system 100′ of FIG. 1B). In that case, the components (e.g., server processor 351, server memory 352, and server communicator 353) and the operations associated with the components described below with reference to IMVR renderer 305 may be performed in a distributed manner by one or more components within a user device (e.g., user devices 101'-103' in FIG. 1B).

[0039] In some embodiments, an IMVR environment (a hybrid between the centrally controlled IMVR system 100 of FIG. 1A and the peer-to-peer IMVR system 100′ of FIG. 1B) may be configured such that it includes a server used for one or more functions and / or operations, while other operations are performed in a distributed manner by interconnected user devices. For example, some such embodiments may include a server used to store and provide data packages to user devices during initialization of an IMVR session, but that plays a limited or no role in data transfer or communication during the IMVR session. As another example, in some embodiments, an IMVR system may include a server used to store information related to the IMVR session (e.g., store records of the IMVR session), but that does not perform other functions related to the IMVR session.

[0040] The server processor 351 may be a hardware-based integrated circuit (IC) or any other suitable processing device configured to run and / or execute a sequence of instructions or code. For example, the server processor 351 may be a general-purpose processor, a central processing unit (CPU), an accelerated processing unit (APU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic array (PLA), a complex programmable logic device (CPLD), a programmable logic controller (PLC), etc. The server processor 351 is operably coupled to the server memory 352 via a system bus (e.g., an address bus, a data bus, and / or a control bus). The server processor 351 is operably coupled to the server communicator 353 via appropriate connections or devices, as will be described in further detail.

[0041] Similar to the communicator 213 in the user computing device 210 of the user device 201 of FIG. 2, the server communicator 353 may be a hardware device operatively coupled to the server processor 351 and the server memory 352 and / or software stored in the server memory 352 executed by the server processor 351. The server communicator 353 may be, for example, a network interface card (NIC), a Wi-Fi module, a Bluetooth module, and / or any other suitable wired and / or wireless communication device. Additionally, the communicator 353 may include a switch, a router, a hub, and / or any other network device. The server communicator 353 may be configured to connect the IMVR renderer 305 to a communication network (such as the communication network 106 shown in FIG. 1). In some cases, server communicator 353 may be configured to connect to a communications network such as, for example, the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a Worldwide Interoperability for Microwave Access Networks (WiMAX®), a fiber optic (or optical fiber) based network, a Bluetooth® network, a virtual network, and / or any combination thereof.

[0042] The server memory 352 may be random access memory (RAM), a memory buffer, a hard drive, read-only memory (ROM), erasable programmable read-only memory (EPROM), etc. The server memory 352 may store one or more software modules and / or code that may include, for example, instructions that cause the server processor 351 to perform one or more processes, functions, etc. In some implementations, the server memory 352 may be portable memory (e.g., a flash drive, a portable hard disk, etc.) that may be operably coupled to the server processor 351. In some cases, the server memory may be operably coupled to the server remotely. For example, the server memory may be a remote database server operably coupled to the server and its components.

[0043] Server processor 352 can be configured to perform several functions, including collecting received and stored data from external sources, using the data to generate an IMVR environment, collecting data from user devices connected via an IMVR system (similar to IMVR system 100 of FIG. 1), maintaining and updating the IMVR environment for one or more online users, etc., as described in further detail herein. Server processor 351 can include one or more units configured to perform a predetermined function, where one or more units are hardware-based units (e.g., integrated circuits (ICs) or any other suitable processing devices configured to run and / or execute a sequence of instructions or code) or virtual machine units generated by execution of a sequence of instructions or code, or a combination of the two. For example, as shown in FIG. 3, server processor 351 can include initializer 354, server state machine 355, renderer 356, data transmitter 357, data collector 358, user state mapper 359, and server state updater 360. Each component, including initializer 354, server state machine 355, renderer 356, data transmitter 357, data collector 358, user state mapper 359, and server state updater 360, may be software stored in server memory 352 and executed by server processor 351 (e.g., code for causing server processor 351 to execute initializer 354, server state machine 355, renderer 356, data transmitter 357, data collector 358, user state mapper 359, and / or server state updater 360). Code for executing the components can be stored in server memory 352 and / or in hardware-based devices such as, for example, an ASIC, FPGA, CPLD, PLA, PLC, etc.

[0044] The initializer 354 can be configured to generate and initialize an IMVR environment that can be used by one or more users. In some implementations, the initializer 354 can be configured to generate a user interface (UI) that can be used to communicate with an IMVR system (e.g., the IMVR system 100 of FIG. 1) and / or to generate the IMVR environment. FIGS. 9A and 9B show example portions of code that may be executed by a processor (e.g., the server processor 351) to generate a UI (e.g., a module UI). In some implementations, the initializer can be configured to browse and load data in an appropriate format from an external source (e.g., via the server communicator 353) and use the data to generate the IMVR environment. For example, in some cases, the initializer 351 can receive medical images in a known data format (e.g., Digital Imaging and Communications in Medicine (DICOM) images or RAW images) and initialize the IMVR environment based on the images. As an example, the images can include information about an organ system of interest (e.g., the nervous system, the circulatory system, or the respiratory system), which can be used to generate an IMVR environment in which internal structures can be viewed and / or probed with high magnification, definition, and specificity. The initializer 354 can define various variables and parameters related to the generation of the IMVR environment. In some implementations, the initializer 354 can invoke any number of functions and / or subfunctions that generate various aspects of the initial IMVR environment. In some implementations, the initializer 354 can be configured to build a server state machine 355 that defines the IMVR environment based on a set of properties.

[0045] Server state machine 355 may be a device or unit of server processor 351 that can be in one of a set of predetermined stable states. State machine 355 may be defined by a set of properties, and each property may assume one of a set of parameter values. The state of server state machine 355 may be defined by the parameter values ​​of the set of properties of server state machine 355. The properties and parameter values ​​of server state machine 355 may be set and / or changed by other components or units of server processor 351, such as initializer 354 or server state updater 360. For example, initializer 354 may set initial parameter values ​​for each property of server state machine 355 and initialize server state machine 355.

[0046] Server processor 351 may include a renderer 356 configured to use information such as the state of server state machine 355 to render the IMVR environment for participation and interaction by one or more users. Renderer 356 may then invoke or cause to be executed one or more functions or subfunctions that perform portions of the rendering of the IMVR environment. For example, renderer 356 may invoke other functions, including a volumizer (also referred to herein as a "volume renderer") that fills the volume of objects with three-dimensional shapes in the environment, and / or functions that control various aspects of the surface appearance of objects included in the IMVR environment (e.g., texturizers, shaders, blenders, etc.), example embodiments of which are described in further detail herein.

[0047] In some implementations, the renderer 356 can generate and / or use predetermined values ​​in the form of threshold criteria, preset coefficients, or look-up tables (LUTs) to guide the rendering of objects in the IMVR environment. In some cases, one or more predetermined and parameter values ​​of the server state machine 355 can be accessible for setting or changing by one or more users. In some implementations, the renderer 356 can receive information from a user to set values ​​for a subset of variables used to render the IMVR environment. FIG. 12 illustrates some example variables that can be configured to be set by the renderer 356 based on user input. In some cases, the IMVR renderer 305 can be operated with one or more test implementations so that the renderer 356 can try out various values ​​of one or more predetermined and / or parameter values ​​of the server state machine 355, which are stored and used to initialize the IMVR environment in a later implementation.

[0048] The server processor 351 may include a data transmitter 357 and a data collector 358 configured to send and receive data to and from any number of user devices connected to the IMVR renderer 305. The data transmitter 357 may transmit information about, for example, the rendered IMVR environment, the state of the server state machine, properties of the environment, etc. The data collector 358 may be configured to receive or collect data from the connected user devices, including information about the state of various user state machines (e.g., the user state machine 215 described with respect to the user device 201), movements or gestures by users, communications between users, etc.

[0049] Server processor 351 may include a user state mapper 359 that assimilates information received from user devices and maps the information to potential changes in the IMVR environment and / or server state machine 355. Server state updater 360 may then be configured to update server state machine 355 using the information received from user devices and mapped by user state mapper 359. For example, changes to the IMVR environment from the perspective of a particular user device (i.e., changes caused by the actions of a user using that user device) may be received by data collector 358. The changes caused by the user's actions may be translated by user state mapper 359 from the user's perspective into general coordinates of the IMVR environment. Server state updater 360 may then use this information to update server state machine 355.

[0050] In some cases, the data collector 358 and the data transmitter 357 can help establish a synchronized IMVR experience between two user devices connected to the IMVR renderer 305. In some implementations, the user devices themselves can mediate the establishment of the synchronized IMVR experience, for example, by communicating data related to the server state machine and the user state machine between the user devices.

[0051] Although the description of a user device includes a user computing device including a processor and memory, in some embodiments, a user device can be configured to include multiple memories and multiple processors that can be used to perform the functions described above. The user device can include volatile memory (e.g., memory associated with a graphics processing unit (GPU), video random access memory (VRAM)), non-volatile memory (e.g., memory associated with a central processing unit (CPU), RAM), a first processor (e.g., CPU) operably coupled to the non-volatile memory, and a second processor (e.g., GPU) operably coupled to the volatile memory. Any combination of functions described above as performed by components of a processor can be performed by components of the first processor or the second processor. For example, functions described in connection with user computing device 210 can be divided between the first and second processors, and the data receiver, controller handler, and / or VR recorder can be associated with the first processor, and the user state machine, user-specific VR renderer, and / or user state updater can be associated with the second processor.

[0052] Although the description of the IMVR renderer (IMVR renderer 305) includes a server processor and server memory, in some embodiments, the server memory can include volatile memory (e.g., memory associated with a graphics processing unit (GPU), video random access memory (VRAM)) and non-volatile memory (e.g., memory associated with a central processing unit (CPU), RAM). And, the functionality described as related to the IMVR renderer can be divided between a first processor and a second processor, where the first processor is operably coupled to the non-volatile memory (e.g., the CPU) and the second processor is operably coupled to the volatile memory (e.g., the GPU). For example, in some embodiments, the initializer, data transmitter, and / or data collector can be components associated with the first processor, and the server state machine, renderer, user state mapper, and / or server state updater can be associated with the second processor.

[0053] FIG. 4 is a flowchart illustrating an example method 400 performed by an IMVR renderer (e.g., IMVR renderer 305) to generate and maintain an IMVR environment. As mentioned above, in some embodiments, an IMVR system (e.g., IMVR system 100′ of FIG. 1B) can be configured such that user devices connect to each other in a peer-to-peer manner without an IMVR renderer. In such embodiments, the methods and / or functions described herein as being performed by the IMVR renderer (e.g., method 400 of FIG. 4) can be executed or performed by one or more components of a user device (e.g., user devices 101′-103′ of IMVR system 100′). In use, the IMVR renderer can obtain 471 data related to images or other information that can be used to generate an IMVR environment or experience. In some cases, the data can include a three-dimensional stack of images, such as medical images, that can be used to generate an interactive, multi-user VR environment that can be used for purposes of diagnosis, education, research, training, etc.

[0054] At 472, the acquired data can be loaded into a base appropriate for the IMVR environment. For example, the data can be loaded into an existing or predetermined texture base called a texture asset. The loaded three-dimensional texture asset can be transferred from temporary memory or random access memory (RAM) to graphics processor memory (GPU memory).

[0055] At 473, the IMVR server can render frames of the virtual experience. At 474, the IMVR renderer renders objects in the environment pixel-by-pixel using the object's front and back depth information. At 475, the IMVR renderer performs a sampling and blending procedure on the input object based on the object's distance. For example, the sampling procedure can include querying a set of points in space in the three-dimensional dataset based on the user's calculated spatial position and receiving sampling information related to the object's intensity value at that point in space. Based on the sampling information, the IMVR renderer can present a projection of the three-dimensional object via a projected representation of the set of points in space from the user's viewpoint.

[0056] At 476, the IMVR renderer receives information about the state of the object (e.g., via the state of the server state machine) and evaluates and / or verifies the position of the object based on other relevant data such as the object's relative size, occlusion, etc.

[0057] At 477, the IMVR renderer applies a threshold condition to the samples and discards samples that do not meet the threshold condition. For example, the IMVR renderer can receive input associated with a threshold setting that defines the viewable range of the dataset, with only samples above the threshold being viewable. In some implementations, the IMVR renderer can receive the threshold setting using a threshold slider control. The server state machine can be updated using the threshold criteria and the samples evaluated against the predetermined condition. The evaluation step can be performed repeatedly and iteratively on pixels, on objects listed by distance, and / or on frames of the rendered IMVR environment, as shown by step 479. For example, the renderer can operate with pixels updated every 200 milliseconds. As another example, the renderer can operate with frames updated at 60 frames per second (fps).

[0058] At 478, the IMVR renderer updates the rendered environment based on state changes detected from the user device. The IMVR renderer can receive information related to changes in user state from one or more connected user devices, and based on the received information, the IMVR renderer can update the server state machine. At 479, the IMVR renderer repeats steps 475 through 477, 474 through 477, and 473 through 477 in an iterative cycle for each object at different distances, each pixel, and each frame, respectively. The server state machine can be updated with information from changes in user state in an iterative process after evaluating the rendered environment for each pixel, each object listed by location or distance, and / or each frame of the rendered environment, as shown at 479.

[0059] At 480, the IMVR renderer communicates state changes in the rendered environment to user devices. The IMVR renderer can communicate the updated state of the rendered environment to one or more user devices, for example, via the state of a server state machine. The steps of rendering the IMVR environment or portions of the IMVR environment and communicating state changes to user devices (473-480) can be repeated iteratively as user movements and / or actions are monitored. Once the user experience by one or more connected users is complete, the rendered multi-user environment can be terminated at 481.

[0060] FIG. 5 is a diagram of a flowchart 500 illustrating an exemplary method for rendering a volumetric IMVR environment using the IMVR system described herein, according to one embodiment. In some cases, volume rendering can be performed by an IMVR renderer (e.g., renderer 305) within the IMVR system. As mentioned above, in some embodiments, an IMVR system (e.g., IMVR system 100′ of FIG. 1B) can be configured such that user devices connect to each other in a peer-to-peer manner without an IMVR renderer. In such embodiments, methods and / or functions described herein as being performed by an IMVR renderer (e.g., method 500 of FIG. 5) can be performed or implemented by one or more components of a user device (e.g., user devices 101′-103′ of IMVR system 100′). In some cases, volume rendering can be performed by a renderer associated with a user device (e.g., user-specific renderer 216 of user device 201). For example, the renderer may read data from a series of DICOM files, image files, RAW data files, or other suitable data series and load the data into volatile memory (such as GPU memory) or into a user device.

[0061] The renderer can obtain a set of values ​​associated with properties that can be associated with the volumetric object being rendered. Some example properties include visibility set using a threshold, transparency set using an alpha factor, the state of the object, and the depth of the object from the user's perspective (e.g., a front depth representing the foreground of the object relative to the user, and a back depth representing the furthest depth of the object from the user). In some cases, one or more values ​​associated with a property can be provided by the user.

[0062] The IMVR renderer can sample points in space provided by one or more channels of three-dimensional data. The samples can be, for example, RGBA samples that include intensity values ​​associated with three color-coded data channels (red (R), green (G), and blue (B)) and a transparency channel (A) corresponding to the intensity data. The IMVR renderer can evaluate the relationship between the depth of the sampled point and the object being rendered (e.g., the depth of the sampled point relative to the depths of the front and back surfaces) and evaluate occlusive objects within the user's line of sight. The IMVR renderer can receive input associated with a threshold property configured to set the viewable range of the dataset, such that only samples above the threshold are viewable. In some implementations, the IMVR renderer can receive a threshold setting using a threshold slider control and compare the sampled value to a threshold property value that indicates visibility. If the sample value is greater than a threshold, the renderer can map the sample value to one or more lookup tables (e.g., one LUT per channel) generated based on user-predetermined properties (e.g., color and transparency properties associated with the object) and present the mapped value as an instance of part of the volumetric object being rendered. The IMVR renderer can display the returned mapped values ​​pixel by pixel and render the volumetric object frame by frame. The IMVR renderer can repeat this process of sampling and rendering in a distance-dependent manner depending on the object's distance from the user. In some implementations, the IMVR renderer can blend the value of the RGBA sample with the values ​​of other RGBA samples from distance iterations previously performed for the pixel.

[0063] FIG. 6 is a flowchart illustrating an example method 600 of object manipulation by a user. Method 600 may include recording, evaluating, and / or storing a user's actions, movements, and / or positions via monitoring properties of a VR controller used by the user, a controller handler associated with the user device (e.g., controller handler 218 of user device 201 of FIG. 2 ), a renderer associated with the user device (e.g., user-specific renderer 216), and / or an IMVR renderer of the IMVR system (e.g., IMVR renderer 305), according to one embodiment. For example, a controller handler of a user device (e.g., controller handler 218 of user device 201 shown in FIG. 2 ) may perform one or more functions described in flowchart 600. As mentioned above, in some embodiments, an IMVR system (e.g., IMVR system 100′ of FIG. 1B ) may be configured such that user devices are connected to each other in a peer-to-peer manner without an IMVR renderer. In such embodiments, the methods and / or functions described herein as being performed by the IMVR renderer (e.g., method 600 of FIG. 6) may be performed or implemented by one or more components of a user device (e.g., user devices 101′-103′ of IMVR system 100′). For example, as shown in flowchart 600, the user device may include one or more controllers, each controller including a button with a function that is in a pressed or unpressed state.

[0064] The user device and / or renderer can receive user input via the controllers and determine the button press state of each controller. For a single controller, a button press invokes the renderer, which can determine the user's current pose within the IMVR environment, including the controller's position, rotation, and scale. These values ​​can be stored as the "current pose." The renderer can determine the pose change relative to the center point and calculate the difference between the previous pose and the current pose. Based on the pose change determination, the renderer can change the pose of the manipulated object. This process can be performed repeatedly, with the current pose tagged as the previous pose and used to compare with the current pose for the next iteration.

[0065] In the case of two controllers, the renderer can determine the pose of each controller by referencing the IMVR environment and calculate the center point between the controllers and the distance between the controllers. The renderer can calculate the pose of the center point using the center point and the distance. The pose of the center point can be used to determine a change in pose as described above. The renderer can change the pose of the manipulated object based on the pose change determination. This process can be performed iteratively as described above with reference to the single-controller method.

[0066] In some embodiments, an IMVR system can be configured to record and / or play back IMVR experiences by one or more users. Figure 7 shows a flowchart describing an example method performed by an IMVR system (e.g., a user device (e.g., user device 201) or an IMVR renderer (e.g., renderer 305)) to perform a user-initiated recording of an IMVR experience. In some embodiments, the IMVR system can be configured so that recording can be performed efficiently and the generated recording files can be configured to a reduced size for ease of storage and / or transfer.

[0067] In some implementations, for example, a user can start recording an IMVR experience, and a user device (e.g., user device 201) or an IMVR renderer (e.g., IMVR renderer 305) can generate a list of identified recordable objects available in the IMVR environment associated with that IMVR experience. For each object, the renderer can start a timer. For each point in time in a series of points in time, the renderer can obtain the state of the object and a timestamp associated with the object at that time and write the values ​​to a recording list. At periodic intervals during recording, the renderer can obtain the state of the player or user along with a timestamp associated with the player at that time. The renderer can also be configured to obtain the state of the object at specific events (e.g., when triggered by a trigger event, such as a trigger press or release after a trigger hold). Figure 13 illustrates an exemplary method for generating an IMVR environment and recording media associated with the IMVR environment by an IMVR system, according to one embodiment. In some implementations, the IMVR renderer or user device can perform recording in an event-based manner. Figure 44B is a flowchart illustrating an exemplary method for event-based processing of an IMVR experience.

[0068] In some cases, portions of the IMVR system can be configured to allow playback of recorded IMVR experiences. FIG. 8 illustrates an exemplary method 800 for playing back a recorded IMVR experience using an IMVR system, according to one embodiment. For example, a user device can receive a dataset including a recording of an IMVR experience, and a user initiates playback as shown in method 800. The dataset can include a recording list including objects, actions, and / or events. At the start of playback, the recording list is loaded onto the user device (e.g., user device 201). The user device starts a time when playback begins, and when the timer reaches the recorded timestamp, the user device can perform an action of the type recorded and presented in the recording list. The user device can iteratively execute the execution of the events and / or actions in the recording list. For example, if the recording includes an instructor presenting an instance of the human circulatory system and isolating the heart at a modified magnification, playback can continue by presenting each of the objects, events, and / or actions presented by the instructor, including any manipulation of the object (e.g., the heart) by the instructor.

[0069] In some embodiments, the recording and / or playback or IMVR experience may include information related to the spatial location of the user initiating the recording and / or playback such that one or more objects within the IMVR environment of the IMVR experience can be rendered to the user from a viewpoint based on the user's spatial location. In some embodiments, the recording and / or playback may be managed based on events associated with the IMVR environment or IMVR experience.

[0070] FIG. 44A is a schematic diagram of different types of events. FIG. 44B is a flowchart of an exemplary method for implementing event-based processing in an IMVR system. For example, an IMVR system can include a local machine and a server. The local machine can be a user device (e.g., user device 201), and the server can be an IMVR renderer (e.g., IMVR renderer 305). The user device can be configured to read events, generate events, process events, check event recording capabilities, and record events. The user device can communicate with the IMVR renderer to send events (events read, generated, processed, and / or recorded at the user device) to other users associated with other user devices and receive events from other user devices. FIG. 45 shows an exemplary flowchart illustrating a set of events defined in association with part of an IMVR experience and the management of the defined events, including event generation and event processing. For example, for every period of a predetermined or user-determined value p, the IMVR system can create a user pose and process that information along with user poses from other users to interpolate the position and rotation of the user object within the IMVR environment and over that period. A user device can create an object interpolation event and send it to other users via a server to interpolate the position and rotation about the object's pivot point over the period p. As another example, when a user acquires an object or releases an object within the IMVR environment, the event is processed accordingly and relayed to other user devices. While the events above are described as being relayed to other devices via a server (e.g., via the IMVR renderer 305), in some embodiments, an event from one user device can be relayed to another user device without a server.

[0071] As previously described, in some embodiments, an IMVR system (e.g., IMVR system 100′ of FIG. 1B or a hybrid IMVR system including a server configured to perform only limited functions, such as storing data associated with the IMVR system when not being used in real time) can be configured so that user devices are connected to each other in a peer-to-peer manner without the use of an IMVR renderer. In such embodiments, methods and / or functions described herein as being performed by an IMVR renderer (e.g., methods or functions related to or described in FIGS. 7-50) may be performed or implemented by one or more components of a user device (e.g., user devices 101′-103′ of IMVR system 100′). That is, while some embodiments and implementations described herein include functions shown to be performed by an IMVR renderer, in some other embodiments, these functions and operations may be performed in a distributed manner by one or more components within a user device (e.g., by a user-specific VR renderer 216, or a user state updater 217, or a VR recorder 221 of user device 201). In such an embodiment, events generated and processed at a user device can be sent directly to other user devices to be processed locally via peer-to-peer connections without a centralized server.

[0072] In some implementations, network recording can be used in the IMVR system described herein to provide a low-cost alternative to real-time multi-user experiences, taking advantage of the low data costs of the multi-user platform provided by the IMVR system. For example, the IMVR system can be used to record a "module" at a first time by a first user by saving data that can typically be transmitted in real time to multiple user devices in Internet Protocol (IP) packets. The saved data can be played back at a second user device in the future. During playback, a user of the second user device can have an IMVR experience similar to that of being in the same IMVR environment as the first user or a "ghost" of the first user's IMVR environment (e.g., a virtual representation of the user in the IMVR environment). In addition to video of objects rendered in the IMVR environment, audio can also be recoded and played simultaneously with the first user's ghost. Audio can be associated with the first user's spatial location so that the IMVR system can recreate a three-dimensional audio source during playback, further enhancing the realism of the simulation. A user interface is available for pausing, playing, fast-forwarding, and jumping back.

[0073] In some embodiments, as described above, an IMVR system can be configured to allow two or more users to share a synchronized experience of the IMVR environment. FIG. 10 shows a flowchart 1000 illustrating an example method for establishing an integrated IMVR environment to achieve a synchronized experience among multiple users. The example method 1000 includes a first user identifying and marking two locations within the shared IMVR environment. For example, the first mark can include the center of the shared IMVR environment, and the second mark can include a fixed spatial location relative to the user (e.g., a point at the northeast corner of the shared space). The IMVR system can generate trackable objects and object states associated with the marked locations within the shared IMVR environment. Based on the objects and object states, the IMVR system can calculate the shared IMVR environment and a forward vector and center position associated with each of the multiple users sharing the IMVR environment, allowing each user to recalculate their relative position and orientation with respect to the IMVR environment and reorient the viewpoint of the IMVR environment to match their location.

[0074] During use, an IMVR renderer (e.g., IMVR renderer 305) or a user device (e.g., user device 201) described herein can receive a data package associated with a three-dimensional volumetric object and a set of user inputs including the user's spatial information, and render an instance of the volumetric object from the user's perspective based on the data package and the user input. In some embodiments, the data package can include a spatial data series or three-dimensional volumetric data obtained, for example, via serial sectioning or serial imaging of the three-dimensional object. In some cases, the data package can include three-dimensional point cloud information (e.g., MEG data) associated with the object. The data can include multiple channels of values, each associated with an array of intensity values. For example, in some cases, the data can be obtained from confocal imaging of an object and include multiple channels of data, each corresponding to an intensity value obtained from imaging a fluorophore (e.g., channel 1 including data obtained from imaging green fluorescent protein (GFP) and channel 2 including data obtained from imaging red fluorescent protein (RFP)). The data package can include a lookup table (LUT) for each channel and a sample number associated with the data set used to render the object. The sample number can be the number of samples associated with the data. The renderer (e.g., the user-specific VR renderer 216 of the user device 201 or the renderer 356 of the IMVR renderer 356) can then obtain user information including the user's position within the IMVR environment, e.g., head position and orientation from a head-mounted display coupled to the renderer and part of the user device (UD), hand position from a hand controller coupled to the renderer and part of the user device UD, etc.

[0075] Based on the data package and user information, the renderer casts a ray in the IMVR environment. The ray is simulated to originate from the user's position in the IMVR environment and travel along a path in the direction the user is facing in the IMVR environment. In some cases, the user's position can be based on the position of a head-mounted display coupled to the user device. The renderer then samples a dataset using spatial data of intensity values ​​along the path of the simulated cast ray. In some implementations, the renderer can use predetermined values ​​associated with the front depth and back depth of the object being rendered and determine the sampling depth to be in front of the front depth or further behind the back depth. In this case, the renderer can discard a sample and continue repeatedly sampling until it reaches an area in space that is within the front depth and back depth. In some cases, the renderer can use a near clipping plane to avoid rendering portions of the object that are closer than the near clipping plane. If the samples indicate rays intersecting the surface of the object, the renderer renders an instance of that portion from the user's perspective based on determining the portion of the object intersected by each ray. According to some embodiments, the renderer is configured to sample at increasing distances as depth increases. The sampling distance may increase linearly or non-linearly with depth. In some cases, a predetermined object boundary location may be calculated, and the sample may be rendered using this boundary location only if the sampled location is within the boundary.

[0076] In some implementations, a renderer can implement a skip procedure to increase the frame rate (i.e., the rate at which frames are updated) to improve the quality of the user experience. A renderer implementing a skip procedure can receive user input associated with a skip rate and sample a point in space. If the returned value is a null value, indicated by being outside the region of interest, the renderer is instructed to take the next sample. If it is determined that the next sample is also a null value, based on this determination, the renderer is instructed to skip the next n samples, where n is determined based on the skip rate and samples the n+1th value. If the n+1th value is also a null value, the renderer is instructed to continue sampling along the ray at the 2n+1th value. If the returned value of the n+1th sample is not a null value, the renderer is instructed to return to the nth value and resample. m is a step-back input provided by the user. In some implementations, the user can provide input to set the values ​​of n and m based on the rendering requirements, the nature of the object being rendered, and / or the processor capabilities of the user device.

[0077] In some implementations, the renderer can use a level of detail (LOD) property associated with the rendered IMVR environment. The LOD property can be set by the user. In some implementations, if the renderer is sampling values ​​within a large dataset that may be time- and processor-intensive, the renderer can be used to sample data points in intermediate datasets generated by binning the dataset rather than the raw data points in the settings associated with the LOD property. For example, instead of sampling a 16x16x16 matrix of data points, which may be time- and / or processor-intensive, the renderer can generate a series of intermediate datasets of decreasing size and order. In this example, the renderer can generate intermediate datasets including 8x8x8, 4x4x4, and 2x2x2 datasets. Each lower-level intermediate dataset is generated by averaging data points and binning the values ​​of the dataset immediately above. The renderer uses the lower-level intermediate datasets appropriately to render areas where a high level of detail is not required. For example, in some cases, a lower-order dataset can be used when rendering objects that are far from the user's position in an IMVR environment, when rendering details in the background, or when rendering details of objects that are not in the user's focus (if the user's fovea is tracked). As another example, if a sample is suspected to be returned as a null value, a lower-order dataset can be sampled first, and if the returned value is not null, the renderer can be instructed to repeatedly sample higher-order intermediate data set to greater levels of detail until an appropriate LOD is reached. In some cases, the renderer can use an appropriate LOD on one user device to match the appearance of the rendered object with the appearance of the object rendered on another user device. For example, a user device with a higher display resolution can use a lower LOD to match a user device with a lower display resolution.

[0078] In some implementations, sampling at greater distances using high-order datasets with high LODs can introduce noise into the rendering of IMVR environments. In such cases, the renderer can be instructed to stop sampling at a certain level of detail (and switch to a lower level of detail if sampling exceeds a certain depth). In some implementations, this improves the user experience by allowing the user to increase the sampling rate before the frame rate drops due to processor capacity. Such a procedure (e.g., sampling the first third of the samples at the highest LOD 0 (i.e., no averaging), the second third at LOD 1 (i.e., one level of averaging including sample averaging), and the last third at LOD 2 (i.e., two levels of averaging including pixel averages) allows users to use fast rendering and receive more detail in the front, good detail at intermediate distances, and less detail at the furthest distances. In some implementations, the sampling distance (i.e., the distance between adjacent samples in space) can increase with increasing depth, so if the sampling distance is comparable to the distance associated with pixel values ​​in an intermediate dataset with an intermediate LOD, samples from the intermediate dataset with average pixel values ​​will be better represented and can be sampled more quickly.

[0079] In some implementations, the renderer can be instructed to end sampling at an earlier point based on the opacity or transparency of the object based on alpha values. In some implementations, sampling can be dynamically set to match the desired frame rate of rendering (e.g., a desired frame rate of 60 fps to improve user comfort). In some implementations, the renderer can implement foveated rendering, in which only the user's foveal region is rendered at high quality, reducing processing load while minimizing or not altering the user experience.

[0080] In some implementations, sampling can be offset to reduce the wood graining effect. This is achieved by sampling from a texture composed of "random noise" to obtain the offset value. These randomized ray sample positions between different bands in the volume allow for a more distributed sampling of the data. For example, a renderer can implement a procedure to randomly offset the starting position of the sampling and / or cast rays. When sampling along radially increasing distances, the second ray is slightly offset from the first, and the samples are at a specific radial distance. Implementing the offset in the renderer allows sampling at odd distances rather than a fixed incremental shell of sampling distance relative to the user. Such sampling at random positions relative to a fixed radial shell, also known as splattering, can avoid the so-called "wood graining effect." Calculations can be performed at a position relative to the user's head, with the volume scaled relative to the user. In some implementations, splattering can also be dynamically configured with respect to a variable distance of sampling and / or the scale of the sampled object. The scale of the object can be defined by a scaling factor.

[0081] In some cases, a three-dimensional point in space, when sampled and determined to be within a boundary, can return an intensity value that can be mapped to one or more LUTs associated with properties of the object. For example, a user may have selected a combination of keys (e.g., two keys) to represent that portion of the object. Each key may include a position value, a color value, a transparency value, and a key associated with a LUT for the object. The renderer can use the LUT to return mapped values ​​that can be used to render pixels corresponding to an instance of a portion of the object (e.g., a surface texture associated with the object). Figure 41A is a schematic diagram of a user interface showing an exemplary set of object properties represented by a set of keys, and Figure 41B is a schematic diagram of exemplary keys as used by a renderer to render a volumetric object, according to one embodiment.

[0082] As an example, a user can select and set values ​​associated with keys via the user interface of FIG. 41A. The definition of color keys can be based on their position on the mapping shown in FIG. 41A, based on which a LUT is generated. The axis containing the scaled uv.U values ​​represents the transparency value of the key (high opacity, low transparency), and the axis containing the scaled uv.U values ​​represents the index value of the key. For example, if a key represents a white key, the user can slide the key in space to increase the opacity to a high white value or a low white value. The renderer can generate the LUT by interpolating values ​​between multiple keys. The user can sample a volume and assign values ​​to objects using an appropriate number of keys (e.g., a user can use three keys to select bone representations and ten keys to select muscle representations). The number of keys can be any appropriate integer value.

[0083] A user can adjust keys associated with an object, and the IMVR system can be configured to propagate the adjustments to other user devices, as described herein, so that changes in the appearance of the object instance rendered on the user device of the user who adjusts the key are mimicked in the object instances rendered on the other user devices. For example, in some implementations, when one user moves a key, the muscle data disappears, and the IMVR system transmits the key's location and color, position, and transparency value information to other user devices in the IMVR system, so that the renderer associated with each user device can locally calculate a LUT to mimic the appearance of the object instance on the first user device. In some cases, the dataset can be associated with a preset LUT stored in the dataset. The adjusted keys can be configured to produce an effect in the LUT over the renderer's sampling time (e.g., 200 milliseconds). Each user device can be configured to share the IMVR environment and calculate the LUT in the same way to obtain a unified data representation for users in the IMVR session. The LUT calculation may be independent of the hardware associated with each user device.

[0084] In some implementations, users may be provided with a set of control or data adjustment tools (e.g., sliders) for fine-tuning and updating object instances substantially in real time without modifying the LUT, so that manipulations and / or changes made by one user can be propagated to other users in substantially real time. Manipulations at one user device can be associated with the generation of a manipulated data package that can include manipulation information. The user device can be configured to transmit the manipulated data package to other user devices to propagate manipulations made by the user to the other user devices. By way of example, as described below, a color key slider can be displayed to the user to manipulate key selection and key position to affect the appearance of a rendered object. For example, a user interface (similar to the user interface of FIG. 41A) of an IMVR system in a user device (e.g., user device 201 of FIG. 2) can include a max-min slider configured to stretch the histogram of intensity values ​​associated with the dataset underlying the rendered object. As a result, the complete LUT can be used to visualize portions of an object that include intensity values ​​that map to smaller portions of the intensity histogram associated with the dataset. In some cases, the user interface may include a threshold slider that is used to receive user input related to a cutoff for visualization.

[0085] As described above, a user interface is displayed for a given data set, providing the user with the ability to manipulate ranges of data associated with objects rendered in the IMVR environment and have the manipulations reflected across instances of the user device within the session. In some implementations, the IMVR system can be configured to include a user interface that allows a user to manipulate properties associated with the rendered objects with a range of values ​​present in the data.

[0086] For example, a user can highlight bones in a CT scan to accentuate the cracks in a geode. As an example implementation, at the user interaction level, the IMVR system can provide a simple user interface to the user, allowing the user to adjust the displayed data by moving sliders. One slider can be configured to allow for adjustment of transparency (alpha value). For example, in a user interface similar to that shown in FIG. 41A, a slider can be configured to adjust the keying of the uv and u axes. A series of sliders can be configured to allow the user to select smaller data ranges to display within the full data range. To better visualize a dataset, several preset color gradients can be applied with a single click. Alternatively, an implementation can use advanced menus to create color gradients of the user's own design. For example, a user of an IMVR session can color muscle red and bone white within a given CT scan, omitting other data.

[0087] The IMVR system is configured such that adjustments made to the slider, color key position, and / or key color are propagated to other users in the IMVR session. The IMVR system then invokes user devices associated with the other users to render updated instances of the object, resulting in the same color mapping and enabling comparable visualizations across users.

[0088] 42A is a flowchart illustrating an example method (e.g., performed by a renderer or user device) for implementing user selection of a new channel of data and transmitting the new channel of data to other user devices, including a second user device, using a strategy of transmitting minimal data over a network and preserving the new data channel, which can be loaded into the second user device receiving the data.

[0089] FIG. 42B is a flowchart illustrating an example method (e.g., performed by a renderer or user device) implemented when a user associated with a first user device changes the color of a key to manipulate the rendering of an object. When the user moves a key marker (e.g., a key marker such as that shown in FIG. 41A), a value associated with the key marker's new position is registered by the renderer, and data related to the key's new position is transmitted over the network to other user devices in an IMVR session in which the first user may be participating. The data related to the key position is transmitted using a strategy that involves minimal data being transferred over the network. User devices receiving this data transmitted by the first user can then update the set of keys associated with their respective renderers using a gradient-generating procedure, an example of which is shown in FIG. 42C.

[0090] FIG. 42C is a flowchart illustrating an example method (e.g., performed by a renderer) for creating a color gradient for rendering colored instances of a volumetric object using a set of keys in an IMVR system. When multiple keys are set by a user or received by a user device in an IMVR system, the renderer can be configured to interpolate between the keys (e.g., linearly or nonlinearly) to generate a color gradient between the keys as described in FIG. 42C. For example, the keys can be reordered from left to right, and the color values ​​of each key can be used to generate a list of colors between pairs of keys that generates an updated list of colors. The updated list of colors can be used to invoke a procedure configured to generate an updated color gradient definition, an example of which is shown in FIG. 43.

[0091] FIG. 43 is a flowchart illustrating an exemplary method for generating a new color gradient definition by interpolating information between a pair of keys (key0 and key1) to generate a LUT. For example, a renderer can receive information associated with a pair of keys. The keys include color, position, and transparency values, and each key has red (e.g., key0.color.r), green (e.g., key0.color.g), blue (e.g., key0.color.b), and alpha (e.g., key0.color.a) components. The renderer can generate slope values ​​between the pair of keys (e.g., rSlope associated with the red components of key0 and key1, gSlope associated with the green components of key0 and key1, etc.). The renderer can update the red, green, blue, and alpha (a) color components of key0 with updated values ​​based on rSlope, gSlope, bSlope, and aSlope, and use the updated values ​​to generate a new color.

[0092] In some cases, an IMVR system can be configured to read images generated from magnetoencephalography (MEG) and use that data to generate a rich IMVR environment used to visualize and / or probe the data for some diagnostic, research, educational, and / or training purposes. Figure 11 shows an example method 1100 that can be used to generate a rich, interactive IMVR environment for visualizing MEG data using the IMVR system of some embodiments.

[0093] A user can manipulate the data in 3D space, i.e., traverse through time using a series of features such as rotation and scale, and playback, step, and scrub bars. FIG. 47A shows an exemplary user interface that can be used to navigate an instance of a rendering of MEG data. As illustrated, the user interface can include forward and backward navigation controls for advancing and reversing distances in the rendered IMVR environment. The user interface can include a scrub slider that the user can adjust to set scrub properties (which remove unwanted portions of the object in the IMVR environment by pruning the rendered instance), a frequency slider, and a rendering update frequency associated with the data. FIG. 47B is a flowchart describing an exemplary method for implementing input provided by a user via the user interface of FIG. 47A.

[0094] In some embodiments, an IMVR system can be configured to receive datasets obtained through magnetoencephalography imaging and generate visualizations of the data in three-dimensional space and time. Magnetoencephalography scans (MEG) can be used to visualize the brain's electrical activity by measuring the direction and magnitude (vectors) of dipole moments at the cortical surface. This data is then displayed in the IMVR environment, providing a three-dimensional perspective of this electrical activity. For example, an MEG dataset captured while responding to an auditory stimulus can be displayed in VR, tracking the path of auditory information through the brain. Vector information can be represented by model objects, such as "teardrop" shapes whose length and color change with size; increased brain activity in a particular region appears as a longer teardrop with a hotter color. Point cloud data can be represented by arbitrary shapes and colors.

[0095] FIG. 48 is an exemplary method illustrating reading and rendering spatial data obtained from MEG using the IMVR system described herein. As shown, for example, in some implementations, a user device in an IMVR system can receive data associated with MEG from an external source and extract vector data corresponding to a predetermined set of points. The vector data can include initial positions of the predetermined set of points and a set of time points, as well as changes in position over time. The predetermined set of points (e.g., 7,000 points in space, each point associated with a three-dimensional vector) can be associated with a magnetic dipole moment associated with local neural activity. Each point in the set of points can be associated with properties such as a position (or origin) property, a rotation property, a magnitude property, and a time property. Each point can include a set of values ​​corresponding to each property. For example, each point can be associated with a set of time points and origin, rotation, and magnitude values ​​associated with the time points. The vectors, along with maximum and minimum values ​​associated with the vectors, can be read and stored in volatile memory (e.g., VRAM) associated with the user device, as shown in FIG. 48. The renderer of the user device may include a set of buffers, including a position buffer, a magnitude buffer, and a rotation buffer, as shown in FIG. 11, so that a set of points with associated properties and values ​​can be read from the extracted vector data, added to the corresponding buffers, and associated with an instance of a volume object in the IMVR environment and rendered.

[0096] In some embodiments, points can be rendered by being represented by a model object. In some implementations, the model object can be an artist-rendered mesh object with predetermined attributes or properties, including color, appearance, texture, shape, size, and / or structure. The properties can be mapped to values ​​associated with the points using a lookup table. Figure 46A shows an example model object 4600, which is a mesh object shaped like a teardrop. Figure 46B shows a perspective view of an example visualization 4601 in an IMVR environment of MEG data containing thousands of points, which are represented by thousands of instances of the model object 4600.

[0097] Visualization 4601 represents neural activity at time "t" associated with one instance of a model object, obtained from vector data and associated with a model object instance, where each vector includes position, direction, magnitude, and time, and has corresponding properties including position, orientation, length, and color. The position or origin properties of thousands of instances of model object 4600 can remain fixed over time, while the orientation, length, and color properties can change over time depending on information obtained from the magnitude and vector direction of the corresponding vector at the associated position in the vector data. The color and length properties of each instance of model teardrop-shaped object 4600 are shown varied based on the magnitude of the associated vector in the vector data; the larger the vector magnitude, the larger the length and brighter the color of the teardrop-shaped model object instance, each selected based on an appropriate lookup table.

[0098] An example of visualization 4601 is a single frame in a time-lapse representation of vector data obtained from MEG imaging of a subject's brain over several time points while listening to a predetermined sound. This representation shows increased neural activity in areas associated with the auditory cortex of the subject's brain's temporal lobe. Changes in neural activity at a given time point are represented by changes in properties such as color, length, and orientation of model object instances at that time point. Figure 49 is a flowchart illustrating an exemplary method for updating user input associated with the rendering of MEG data on a user device. Figure 50 is a flowchart illustrating an exemplary method for implementing the rendering of an MEG dataset. The above-described representation of MEG data can help clinicians easily identify changing trends in neural activity. For example, as described above, visualization of MEG data over time can be used to easily and quickly identify the location of seizure activity or the location of desired activity in response to predefined stimuli, which can aid in surgery.

[0099] Other data sets may include EEG, geological data, molecular data, or other point cloud data with or without a time component. In some implementations, the data is collected over a network socket and displayed to the user in substantially real time.

[0100] As described herein, a renderer in an IMVR system can obtain a dataset associated with a three-dimensional object, possibly a dataset including serial two-dimensional images of the object captured using imaging technology, and render the object in an IMVR environment. As a result, multiple users can visualize and manipulate the object simultaneously. The rendering of the object in the IMVR environment can be done using any suitable set of predefined properties that can determine the level of visualization and / or manipulation that may be available to the user. Some of the predefined properties associated with the rendering of the object can be set using input from a user (e.g., a host user or a first user defined to enhance control of the IMVR session; an IMVR session is an unobtrusive IMVR experience including one or more users). In some implementations, properties can be defined to be set by any participating user. FIG. 12 shows a list of user inputs. For example, the IMVR system can receive input associated with “nearClip,” which sets the user's ability to clip a nearby portion of a rendered object from the visualization using a predefined near clipping plane.

[0101] The input "alphaThreshold" can be used to set the transparency threshold property for visualizing an object or part of an object. Inputs associated with samples can be used to set the number of samples used to render an object with reference to the user device (used by the user to visualize the rendering). Some inputs can determine the quality of the rendering and the efficiency of the rendering. For example, enabling "skip" and / or "MIP mapping" can instruct the IMVR system to render objects by taking into account null spaces that do not contain the object. Skipping parts of null spaces upon detection and only returning to sample areas based on the identification of non-null spaces can increase the efficiency of the rendering while maintaining the quality of the visualization. Additional inputs from the user can be set to determine the skip rate (based on the processor speed and capabilities of the user device, as well as the nature of the visualized dataset). Binning and / or foveated rendering can also be used to further increase the efficiency of the rendering.

[0102] As a further example, inputs associated with a section tool enable section functionality in an IMVR environment, and inputs associated with configuration parameters associated with including and excluding objects during rendering enable isolated rendering of an object by including and / or excluding other surrounding objects. The inclusion and exclusion-based definitions may refer to other objects in the immediate vicinity of the object of interest in an IMVR environment. As a further example, inputs associated with a set of annotation tools (e.g., paint tools, voxel painter, measurement tools, annotation tools, etc.) may be used to enable a user to annotate or manipulate a rendered object using the paint tools (e.g., to paint or change appearance properties by adding color or reducing transparency). Examples of such inputs include measurement inputs, annotation inputs, paint inputs, highlight inputs, markup inputs, etc.

[0103] 13 is a flowchart 1300 of an exemplary method for generating an IMVR environment and recording media associated with the IMVR environment by a renderer (e.g., the IMVR renderer 305 or the user-specific VR renderer 216 of the user device 201) in an IMVR system, according to one embodiment. At 1371, the method 1300 includes receiving data including a medical image associated with an object. As previously mentioned, the medical image may include DICOM images, such as images obtained via computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography-computed tomography (PET-CT), and / or magnetoencephalography (MEG) data, as well as one or more other data sources including serial optical sectioning and imaging of three-dimensional objects, such as a confocal microscope, an electron microscope, or any optical microscope with data acquisition across multiple planes.

[0104] At 1372, the renderer extracts from the data a dataset associated with the structure of the object and user information associated with the user's spatial location referencing the object in the virtual environment (also referred to herein as the "IMVR environment"). The data may include additional information associated with the dataset that may also be extracted and used to set one or more parameters of the IMVR environment. For example, the data may include a sample number associated with the object, channel specifications including the number of channels of data, and a lookup table (LUT) associated with each channel of data. The dataset may include one or more channels of information associated with volumetric data associated with the object. Each channel may be defined so that the object can be rendered using one or more channels that the user can optionally visualize individually. For example, each channel may be rendered using a lookup table (LUT), and each LUT may be separable (e.g., spectrally separable) from the other lookup tables so that the LUT can be independently manipulated to change the visualization of the object rendered with that channel of data. The channels may be configured so that objects rendered using each channel can be composited to form a composite object. 24 and 25 are flowcharts illustrating an exemplary method for generating a composite instance of an object rendered using multiple channels of data in an IMVR system according to one embodiment. For example, as shown in FIG. 24, a renderer can implement a procedure involving maximum intensity projection (MIP) to render an object in an IMVR environment. The renderer can obtain information associated with location, color, and MIP value. The renderer can calculate a MIP difference value indicating the difference between the MIP values ​​of samples of a channel of data at a location and compare the value to a threshold set by the user. Based on the comparison, the renderer can generate a LUT that includes a composite of color and transparency information.Figure 25 also shows an example method for returning a composite color given a pair of colors. Figure 25B is a flowchart of an example method for normalizing depth visualization using information related to near clip settings and world depth settings (e.g., by a user) in rendering objects in an IMVR environment.

[0105] In some implementations, the data may include a set of instructions and keys, but may not include a LUT for the channel; the LUT may be generated locally using the keys. Each key may include properties associated with color and transparency, so that a combination of keys may be used to generate the LUT. The user information may include a relative spatial position and a user's relative orientation with respect to the rendered object. In some implementations, one or more user devices in the IMVR system may include a headset, a head-mounted display, and / or one or more manipulation controllers. The renderer may receive head and / or hand positions and poses or pointing directions within the IMVR environment from the headset and hand controllers and use that information to render the object.

[0106] Returning to Figure 13, at 1373, the renderer receives a set of inputs from a user, the set of inputs being associated with a set of predetermined properties associated with a portion of an object in the IMVR environment. The set of inputs may include the inputs defined above with reference to Figure 12. The properties may include a set of visualization properties. In some implementations, the user input may include changes to be applied to a LUT to render the object.

[0107] At 1374, the renderer renders an instance of the portion of the object in the virtual environment based on the set of inputs and the user information. At 1375, the renderer receives manipulation information related to the user and manipulation of the rendered instance of the portion of the object in the virtual environment. In some implementations, the manipulation information can be received through a series of inputs received from the user via a headset or one or more hand controllers. The manipulation can include a user action configured to induce an effect on the IMVR environment that manifests as a change to the visualization of the rendered object. For example, the manipulation can be configured to change an appearance property of one or more portions of the rendered object, such as a change in color or transparency of one or more portions of the rendered object, or a change in the orientation, scale, or view of the rendered object. The manipulation can also be configured to induce a change in the user's position and / or orientation relative to the rendered object.

[0108] At 1376, the renderer renders an updated instance of the portion of the object that manifests the effect of the manipulation in the virtual environment based on the manipulation information.

[0109] At 1377, the user device records media associated with the manipulation and rendering of the updated instance of the portion of the object, the media including a video of the rendering of the updated instance of the portion of the object from a user's perspective within the IMVR environment, the perspective being based on the user's spatial location within the IMVR environment. In some implementations, the recording of the IMVR session or experience can be configured to capture location, host information, timestamps, and audio. In some implementations, the media can include an audio recording of sounds associated with the rendering of the object, including sounds generated by one or more users in the IMVR session. In some cases, the recorded audio can be associated with the recorded spatial location within the IMVR environment, such that playback of the media allows a second user experiencing the video to also experience the audio taking into account the location of the audio source within the IMVR environment. For example, audio from a source located in a spatial location in front and to the right of the user's location can be configured to manifest as a louder audio in front of and to the right of the user. As another example, the recording and / or playback can include a dynamic location of the source of the audio. In other words, audio produced by a user moving through a virtual space and / or interacting with objects within the virtual space can be recorded such that movement and / or changes in the visualization of both the user and the interacting objects are captured, both video and audio with appropriate time stamps, and can be recreated upon playback and experienced by another user. In some implementations, a second user implementing playback can choose between pure audio playback and audio that includes spatial information, such that the audio is dynamically adjusted to the second user's location and the dynamic nature of the source.

[0110] In some implementations, playback can be initiated by multiple users, and audio including spatial information can be presented to each user in a multi-user session according to their spatial location. In some implementations, a host user can automatically record video and / or audio associated with an IMVR session. In some cases, the recorded audio can be associated with the host user. In some implementations, each user in a multi-user IMVR session can be associated with an audio, and audio recordings can be managed to obtain one audio per user in that IMVR session. In some implementations, the IMVR system can sense when a particular user is producing sound (e.g., speaking), and the IMVR system can mute that user's audio to produce a higher quality sound and relay it to other users in the session (e.g., to avoid echo or interference effects). In some implementations, the IMVR system can manipulate the set of audio associated with a user, including starting and stopping, to capture interactions between users (e.g., a discussion between a tutor and a student in a virtual classroom session). In some cases, a subset of the IMVR session can be recorded (e.g., a subgroup of students in a larger classroom). In some implementations, one instance of an IMVR environment from one session (e.g., a subgroup of students in a virtual class) can be exported to another IMVR environment associated with another session (e.g., a different subgroup of students, or a class of students).

[0111] A user device of an IMVR system can be configured to support playback by a second user with a set of control functions. The second user can select a set of properties for playback and / or select values ​​associated with properties of the user device. For example, playback can be started, paused, and / or stopped at the user's discretion. Playback can be rewound, played, or modified in other suitable manners. In some implementations, the second user can select on user device properties such as transparency, color, etc. to change and / or manipulate the instance of portions of objects rendered in the playback.

[0112] As described herein, an IMVR system can be configured to support multiple users interacting with objects within an IMVR environment. The IMVR system can be configured so that a shared IMVR environment is integrated in presentation across multiple user devices, regardless of how each user device may be connected (e.g., via a wired connection, a wireless connection, a communication network of various parameters, etc.) and regardless of the specific hardware configuration of each user device. In some cases, the IMVR system can be configured to perform weighted rendering that takes into account the hardware configuration of a user device so that the appearance of an instance of an object rendered on that user device is similar to or matches the appearance of instances on other devices. For example, if a user device is more powerful than other user devices in a shared IMVR session and is capable of a higher sampling rate or frame refresh rate, the renderer associated with that user device can calibrate rendering properties, such as color and / or transparency, associated with portions of an object to accommodate differences in hardware configuration. As an example, a user device capable of 500 times as many samples may use sample values ​​weighted to a smaller percentage (e.g., 1 / 500 x value) compared to a user device capable of 300 times as many samples (which may use a sample weighting of 1 / 300).

[0113] A new user device can be added to an ongoing IMVR session to seamlessly join the session and experience a unified IMVR presentation. In some implementations, if the new user device does not already have a data package associated with the ongoing IMVR session, the new user device can receive the data package. In some cases, the new user device can receive all of the data in the data package's library. In some cases, the new user can receive data associated with a portion of the ongoing IMVR session (e.g., medical data associated with a single patient or medical data associated with the instruction of a single class) rather than the complete library. Figure 36A is a schematic diagram of an example data package.

[0114] For example, in some implementations, the data package can be raw image data sent to a graphics card of the user device. In some implementations, a palette of images can be selected by the sending user, and the selected data set can be sent to the new user device. As noted above, an IMVR session can include multiple users, and in some implementations, the IMVR system can establish a hierarchy of users with increasing or decreasing privileges. For example, FIG. 26 illustrates an example setup in which an IMVR session can be initiated by a first or host user associated with host device HD0 and having the greatest privileges. The host user can use user devices UD1-4 to invite other users to join the IMVR session and provide data packages to the user devices over a network (e.g., a TCP / IP-connected network). FIG. 26 illustrates an example setup with a host user associated with host device HD0 and a set of user devices UD1-4.

[0115] The IMVR system can be configured so that new user devices can be programmatically instructed to check the integrity of received data packages and synchronize the data packages with respect to other users. Figure 36B is a flowchart illustrating an example method of data package processing to allow users to participate in an IMVR session. In some cases, the data package may be stored locally in non-volatile memory (e.g., memory associated with a central processing unit (CPU)) at the receiving user device in addition to being stored in volatile memory (e.g., memory associated with a graphics processing unit (GPU), VRAM, etc.). In some implementations, the sending user or host user can approve the receiving user to allow local storage in non-volatile memory of the receiving device. In some implementations, the data package can be associated with permission information indicating the available permissions to process, use, and / or store the data set included in the data package.

[0116] In some implementations, data may be bound by privacy protection, and thus the receiving user device may be instructed by the sending or host user device to store the data only in volatile memory (e.g., a graphics processing card) associated with the receiving user device, rather than storing the data in non-volatile memory associated with the receiving user device. Such implementations may be configured to store the data in video RAM used by the renderer, and then delete the data from other volatile memory (e.g., RAM) associated with the device. In some cases, the IMVR system may be configured to transmit data packages one chunk at a time, so that the receiving device cannot access multiple data chunks at a time.

[0117] In some cases, the data package can be modified before being sent to the new user. For example, the data package can be scrubbed to remove patient-identifying metadata and sent to the receiving user device. The receiving user device can be instructed to extract information used to render the object and join an ongoing IMVR session with another user. For example, the received data can extract data associated with the rendering (voxel data, voxel size, voxel width, voxel height, etc.). FIG. 27 is a schematic diagram of a method of sharing data related to an IMVR environment between a host machine and a client machine using an IMVR system, according to one embodiment. In some implementations, the host machine can be a user device associated with a first or host user (e.g., user device 101 of FIG. 1A ), and the client machine can be a user device associated with a subsequent user (e.g., user device 102). In some implementations, the host machine can be an IMVR renderer (e.g., IMVR renderer 105), and the client machine can be a user device (e.g., user device 102).

[0118] In some implementations of the user device portion of the IMVR system described herein, the system can be configured to indicate to a set of user devices of an ongoing IMVR session the upcoming addition of a new user to the session. In some implementations, the IMVR system can indicate readiness for the new user with a level of rendering completion for the new user's device, as shown in the exemplary method illustrated by the flowchart of FIG. 37. The new user's audio (e.g., voice) and location data within the IMVR environment can be broadcast to other user devices before the new user is allowed to join live and operate the IMVR environment. A user interface associated with the IMVR system's user device can visually indicate to other users that the new user is visualizing the same objects as the host user while the new user is participating in the session. The host user (e.g., the first user of a shared IMVR session) can set limits on the new user's control while a data package associated with the new user's user device is loaded or an instance of an object is rendered on the new user's user device. For example, the new user can be limited in access to operations in the IMVR environment while the first instance of one or more objects within the IMVR environment is rendered on the new user's user device.

[0119] In some implementations, a host user may control an intermediate instance of an IMVR environment that is shared by a set of users. Figure 26 (above) is a schematic diagram of an IMVR system configured to manage multiple instances of objects rendered in an IMVR environment, according to one embodiment, to allow an instructor to flexibly join and leave a set of shared IMVR sessions among students implementing the IMVR system.

[0120] FIG. 37 is a flowchart illustrating an example implementation of a method for enabling a user to join an ongoing IMVR session, as described above, using an IMVR system according to one embodiment. FIG. 38 is a flowchart illustrating an example implementation of a method for adding a data package to an existing library of data packages associated with a user device, the user device being associated with a new user joining an ongoing IMVR session. The flowchart of FIG. 38 can be used by a renderer (e.g., renderer 305) or user device (e.g., user device 201) to update data packages associated with the new user and synchronize data packages associated with object rendering across all user devices in an ongoing IMVR session using an IMVR system according to one embodiment. FIG. 39 is a flowchart illustrating an example method for managing a user interface, including providing a visual indication of a new user to a set of existing users associated with an IMVR session, using an IMVR system according to one embodiment. The method described in FIG. 39 can be implemented by a renderer (e.g., renderer 305) or user device (e.g., user device 201) described herein.

[0121] FIG. 40 is a flowchart illustrating an example method for matching students with instructors using a matchmaker server. For example, the matchmaker server can consider the current state of the student's device and the instructor's device to match the student with the instructor for an IMVR session. In some implementations, the instructor's device can be similar to the user device (e.g., user device 201) described herein. The student's device can also be a user device (e.g., user device 201). In some implementations, the student's device can be a user device defined to have limited functionality. For example, one or more student's devices can have limitations related to accessing or accepting data packages, the ability to approve the transfer of data packages, the ability to annotate or modify data packages, the ability to perform certain operations on data or instances of objects rendered on the user device, the ability to invoke, start, or end a shared IMVR session, the ability to record or play back recordings of the IMVR experience, etc.

[0122] In some implementations, the IMVR system can be configured to operate multiple IMVR sessions in parallel using potentially related data packages. Some IMVR sessions are organized and managed by the IMVR system and can have one supervising session with a supervising user and multiple sub-sessions with smaller groups of users (e.g., a class with an instructor and a sub-session including smaller subgroups of students working in groups). Users can be provided with tools that enable them to flag for assistance or other users' attention within an IMVR session or between an IMVR session and another parallel IMVR session (such as a parallel virtual room) and send and receive messages to other users across other sessions. For example, a student in one sub-session can request assistance from an instructor who may be participating in another session. In some cases, the IMVR system can configure a hierarchy of IMVR session settings so that a higher-level session can be accessed by a user in a lower-level sub-session. A sub-session can be configured to use the same data package as the higher-level session. In other examples, a sub-session can be configured to use a copy of the data package associated with the higher-level session, with each copy assigned to each sub-session. In some implementations, the IMVR system can be configured to project or export rendered object instances and / or operations in sub-sessions that move up or down the hierarchy to higher sessions (e.g., from a sub-group to an entire class) or lower sessions (e.g., from a sub-group to an even smaller group).

[0123] FIG. 14 is a flowchart illustrating an example method 1400 of receiving data associated with a first instance of an object in an IMVR environment and using the data to render a second instance of the object so that a first user can share the IMVR environment with a second user.

[0124] At 1471, the renderer receives from the first device and the second device: (1) a data package associated with a first instance of a portion of an object in the virtual environment, the data package including a data set associated with a three-dimensional structure of the object; (2) first user information including a spatial location within the virtual environment of a first user associated with the first device with respect to the object in the virtual environment; and (3) a first set of inputs associated with a set of predetermined properties of the object.

[0125] At 1472, the renderer receives, at the second device, second user information including a spatial location of a second user associated with the second device with respect to an object in the virtual environment.

[0126] At 1473, the renderer transforms the first set of inputs into a second set of inputs based on the second user information. At 1474, the system generates a lookup table based on the second set of inputs.

[0127] At 1475, the renderer, on the second device, renders a second instance of the portion of the object in the virtual environment based on (i) the dataset, (ii) the second set of inputs, and (iii) the lookup table, the second instance of the portion of the object from a viewpoint of a second user, the viewpoint being based on a spatial location of the second user.

[0128] FIG. 15 is a flowchart illustrating an exemplary method for rendering a volume of an object in an IMVR environment by a renderer (e.g., renderer 305) of an IMVR system, according to one embodiment. For example, the renderer may receive information related to a depth texture for the back of a cube and a depth texture for the front of a cube. The renderer may check whether the renderer's section feature is selected (e.g., XSection_On=1 may indicate selection). If selected, the renderer may obtain a depth texture for the section plane. If not selected, the renderer may check whether the DepthTesting feature is selected. If selected, the renderer may obtain a depth texture for all visible opaque objects. The renderer may check the selection of inclusion and exclusion property values ​​and, based on the selection, obtain depth textures for the back and front of included objects or the back and front of excluded objects, respectively. The renderer may obtain a transformation matrix to transform real-world space into cubic space, obtain the position of the main camera associated with the user device, scale the world environment based on the cubic scale, and render the volume data.

[0129] Figure 16 is a flowchart describing an exemplary method for rendering a volumetric object in an IMVR environment, according to one embodiment, involving a sequence of events following the rendering described in Figure 15. Figures 17A and 17B illustrate a portion of a method including shading a texture associated with a volumetric object rendered in an IMVR environment using a set of user inputs using a renderer of an IMVR system, according to one embodiment. Arrows AA and BB in Figure 17A are followed by arrows AA' and BB' in Figure 17B. Figure 18 is a flowchart describing an exemplary method for rendering a volumetric object in an IMVR environment using a ray-marching approach by a renderer of an IMVR system, according to one embodiment.

[0130] In some implementations, an IMVR system can be configured to perform foveated or masked rendering to better present a user with visualization of object instances within the IMVR environment. In some implementations, masked rendering can be implemented with binning. For example, such an approach allows a user to view higher resolution in certain areas of the screen and lower resolution in less important areas, with the overall goal of improving the user experience. An example of this is using eye tracking to render higher resolution where the user is looking (the fovea), which allows for a higher frame rate for updating the rendering compared to attempting to render the entire screen at high resolution. Binning is a method of treating multiple pixels as a single pixel during rendering, achieving lower resolution areas.

[0131] 19A and 19B show an exemplary method for generating the sample distances and weights assigned to each of these samples used by the ray-marching approach of FIG.

[0132] In some implementations, an IMVR system can be configured to use a dataset containing one or more artist-rendered mesh objects. The IMVR system can be used to capture cross sections of the mesh objects using a series of tools, including cross section tools, provided to the user. FIG. 20 illustrates a series of user actions for invoking the capture of a rendered mesh object in an IMVR environment. For example, in some implementations of an IMVR system, a user can select a function to enable cross section capture through a user device's user interface. The user can then hover over a two-dimensional intersection object in the IMVR environment to display the name of the associated three-dimensional mesh object. The user can select the two-dimensional intersection object to create or activate a copy of the associated three-dimensional mesh object at a position relative to a user-defined cross section plane. In some cases, the user can select the three-dimensional mesh object to discard or disable the three-dimensional mesh object.

[0133] FIG. 21 is a flowchart illustrating an exemplary method for capturing a mesh object using a cross-section tool, according to one embodiment. For example, a renderer can be configured to determine the location and orientation of the cross-section. The renderer can generate a list of target meshes and remove meshes from the list whose boundaries do not intersect with the cross-section. For the remaining meshes in the list, the renderer can list the triangles in the mesh and, for each triangle, determine whether the triangle intersects with the cross-section. If so, based on the determination, the renderer determines the edges of the intersection. For each edge of the intersection, if the edge connects to any line segments, the renderer adds them to the line segments (otherwise, it creates a new line segment). If the line segments are closed shapes, the renderer creates a 2D mesh from the closed boundary. The renderer repeats this procedure for all edges and all meshes in the list until it creates a 3D mesh. It creates a single object from the 2D intersection mesh and associates the object with the 3D mesh object, which is then realigned as the captured mesh cross-section.

[0134] FIG. 22 illustrates another exemplary method for capturing a cross-section of a mesh object rendered in an IMVR environment using another cross-section tool, according to another embodiment. According to this method, a resolution is selected and the position and orientation of a cross-section square are determined. For each edge of the cross-section, a square ray is cast on the opposite side at a predetermined interval. For each collision along the ray, the encountered object is determined, the collision point is added to a list, and a cloud of points is returned. The renderer then generates an ordered list, which is calculated for each point, and the nearest neighbor is determined from the unordered list. If the nearest neighbor is closer than the first point in the ordered list, that neighbor is added to the ordered list. For each ordered list, a 2D mesh is generated from the boundary of the closed shape. If the 3D mesh generates at least one 2D intersection, the renderer creates a single object from the 2D intersection mesh and associates the object with the 3D mesh object. In some cases, the renderer can use a triangle generator to generate surfaces by creating triangles to create the surface.

[0135] In some implementations, mesh objects can be used to create ghosts, also known as ghosting. For example, a renderer can create a copy of the object from which a 2D mesh is created. The renderer can restore the mesh associated with a 3D dataset that contains a volume object, but store the mesh in an off state. It can be configured to toggle on and be active and visible when the 3D object is turned off.

[0136] The cross-section tool can be used to take cross-sections at any angle through the mesh model data. This creates a 2D representation of the intersection. These cross-sections are interactive; the user can point at a structure to reveal a label and click on a structure to pop out the complete structure in the cross-section. For example, a cross-section can be taken through the chest, and the user can pop out the aorta and visualize how it progresses up through the chest and then down through the abdomen. This ease of use provides the user with detailed information about the course of a structure above and below a particular plane, which is useful for studying medical images. Figure 23 is a flowchart illustrating an exemplary implementation of the cross-section tool in an IMVR system.

[0137] In some implementations of the IMVR system, one or more users can be provided with additional toolsets for manipulating rendered instances of objects. Some example tools include volume scaling, exclusion, and / or inclusion tools configured to isolate volumes of interest by excluding or including portions of rendered object instances, and cross-section tools configured to cut volumetric objects along any plane that intersects the object. Other examples include voxel painter and voxel eraser tools, as well as counting / labeling tools, mesh creation tools, measurement tools, and the like configured to enable users to annotate rendered object instances.

[0138] As an example, a measurement tool can be configured to give users the ability to make precise measurements within a volumetric dataset. Users can draw lines freehand or point-to-point and label them with precise measurements. For example, they can measure the diameter of an aorta from an angiogram or the distance between two cells from confocal microscopy data. Additionally, some implementations have features that allow users to move individual line vertices from a pre-created line. In some implementations, users can also add or delete vertices.

[0139] FIG. 28 is a flowchart illustrating an exemplary method for scaling the volume of a rendered object instance using a volume scaling tool. A renderer can implement the flowchart of FIG. 28 to render based on a scaling factor calculated based on a set of user inputs and / or sampling information associated with a user device. For example, the renderer can receive information associated with a data file's voxel size and voxel dimensions and scale an object associated with a data set by a predetermined factor based on a generated scale vector. FIGS. 29A, 29B, and 30 are flowcharts illustrating exemplary methods for using the exclude, include, and cross-section tools, respectively. FIGS. 31A-31C, 32A, 32B, 33A, 33B, 34, and 35 are flowcharts illustrating exemplary methods for implementing additional exemplary tools for manipulating rendered object instances in an IMVR environment. For example, as shown in FIG. 34, a measurement or annotation tool can be used to allow a user to draw a line and label the line as an annotation.

[0140] In some cases, a user may measure a portion of a rendered object and label the portion with a measurement annotation (e.g., a measurement annotation indicating the scale of the portion of the object). The IMVR system may be configured to allow the annotation to be propagated to other users of the IMVR session. The IMVR system may also allow a user to associate an annotation with a portion of a rendered instance of an object and save the instance to later send to other users.

[0141] In some implementations of an IMVR system, one or more users can be provided with a set of tools for manipulating objects rendered in the IMVR environment. For example, users can move objects closer, zoom in, zoom out, pivot, rotate, nudge, and perform other appropriate operations on objects. In some cases, users can move otherwise static objects with user actions. For example, users can initiate throws, catches, rolls, and / or pull-ins of objects using one or more hand controllers controlled by each user. The IMVR system can be configured to modify mobility laws of objects and users within the IMVR environment in appropriate ways. For example, a user can select an object and move it closer or further away by pointing the joystick on the controller. The speed of a moving object can depend on its distance from the controller. For example, the IMVR system can be configured so that distant objects move quickly and slow down as they get closer to the user in the IMVR environment. An exemplary fishing rod tool can be configured to allow a user-controlled device (e.g., a fishing rod) to intersect with an object and reel the object in at a speed proportional to the distance. As another example, an IMVR system can be configured to coast when a user lightly tosses an object, slowly losing momentum before coming to a stop. The user's control can then be released from the thrown object, allowing another user to pick up the object that tossed it. An IMVR system can be configured to allow objects to be scaled using two controllers. For example, a user can point at an instance of an object renderer and use one action to grab or select the object and another action to drag it.For example, the renderer can function such that when a user pulls a trigger associated with a controller, an object is attached to the pointer, allowing the object to be rotated by the user's controller or hand even after the user releases the trigger. The last manipulable object can have a persistent association with the hand controller. Two controllers can be used to double-grip the last selected object. The user can release the hands or pull them together to zoom in or out. The renderer can calculate the distance between the controllers and scale the object based on the change in distance while using a fixed center point for the object being scaled. The fixed center point can be any point of interest; for example, zooming in allows the user to focus on the magnified area without recentering. The user can also rotate the object while changing the magnification.

[0142] Some embodiments use one or more digital anatomical models and neuroanatomical connectivity data to render interactive, functionally representative models of the human nervous system. By rendering simulated neurons visually responsive to the activity of other representative neurons and damaging "lesion" objects, such embodiments can convey to users the spatial relationships of signal pathways, directionality, and / or functional states of neuroanatomical circuits. For example, if a user places a "lesion" object or other representation of neurological damage on a neuroanatomical model, the representative system can respond to the disruption of the affected neurons and display a specific representation of the loss of function experienced by the patient. By utilizing multi-user connectivity, such embodiments enable users from the same or remote locations to interact in real time with the same objects, represented by digital avatars within the same common interactive environment. To enhance the ability to create diagrams and teach in a multi-user environment, tools can be provided to users, allowing them to create three-dimensional line drawings anchored to objects in the virtual environment. For example, instructors can be enabled to conduct and record lectures using three-dimensional diagrams and diagrams in an IMVR environment. In some embodiments, the disclosed features can be utilized to enhance a student's intuitive understanding of cross-section-based medical imaging techniques. The disclosure can be configured to provide two-dimensional cross-sections from any three-dimensional object present in the IMVR environment, allowing users (e.g., students) to simulate the creation of CT and MRI images while maintaining the ability to reference the three-dimensional object represented in the cross-section. Not only do the embodiments provide and / or facilitate real-time interaction (e.g., between students and instructors), the teachings of the present disclosure also enable the creation of self-contained modules. User interactions and created drawings or cross-sections can be recorded / recorded to a transmittable saved file. This can be represented as a virtual instructor performing actions and didactic instructions originally recorded by the instructor.Such embodiments allow student users to review content outside of normal instructional periods and at their own pace. In addition to the cross-section-based medical images described above, the system can import DICOM images, such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography-computed tomography (PET-CT), and / or magnetoencephalography (MEG) data, as well as the initiation of one or more other data sources. This information is imported through various software processes and analyzed by custom mathematical systems and methods that render the image in virtual reality. Additionally, some embodiments provide pseudosegmentation of systems (e.g., vessels, bone, tissue) through innovative transfer functions that allow users to view data of interest (density) while dropping off data of no interest. In some implementations, this can provide clinicians / users the ability to highlight and bring to the forefront structures and systems for detailed analysis in real time. Opaque planes can be inserted into the volume to display data similar to traditional 2D images. Users can adjust the transparency embedded in the transfer function to visualize structures emerging from opaque planes. Additionally, some embodiments provide multi-user interaction in VR / AR environments. Because users do not need to be in the same location, such embodiments enable clinical discussions from anywhere in the world using high-speed networks.

[0143] As noted above, the IMVR system described above can be used to visualize any spatial data series associated with any object from any field of investigation. Figures 51A-51C illustrate an exemplary embodiment of an IMVR system used to volumize and visualize gemstones and / or metals using spatial data series obtained via CT. Figure 51A is an image of a garnet stone obtained by CT scanning, which allows for examination of the stone's inclusions and crystalline structure. Figure 51B is a colorized rendering of the garnet stone in the IMVR environment. Figure 51C is a structural diagram of the garnet stone of Figures 51A and 51B, volumized and rendered in an exemplary IMVR environment by an IMVR system according to one embodiment, showing crystalline structure A and inclusion B labeled in Figure 51C.

[0144] While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. Where the methods and / or schematic diagrams above indicate that certain events and / or flow patterns occur in a particular order, the order of the certain events and / or flow patterns can be changed. While embodiments have been particularly shown and described, it will be understood that various changes in form and detail can be made.

[0145] Although various embodiments have been described as having particular combinations of features and / or components, other embodiments may have any combination of features and / or components from any of the above described embodiments.

[0146] Some embodiments described herein relate to computer storage products comprising a non-transitory computer-readable medium (also referred to as a non-transitory processor-readable medium) having instructions or computer code thereon for performing various computer-implemented operations. The computer-readable medium (or processor-readable medium) is non-transitory in the sense that it does not itself comprise a transient, propagating signal (e.g., a propagating electromagnetic wave carrying information over a transmission medium such as space or a cable). The medium and computer code (also referred to as code) may be designed and constructed for a specific purpose. Examples of non-transitory computer-readable media include, but are not limited to, magnetic storage media such as hard disks, floppy disks, and magnetic tape; optical storage media such as compact discs / digital video discs (CDs / DVDs), compact discs-read-only memory (CD-ROMs), holographic devices; magneto-optical storage media such as optical discs; carrier wave signal processing modules; and hardware devices specially configured to store and execute program code, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), read-only memory (ROM), and random access memory (RAM) devices. Other embodiments described herein relate to computer program products that may include, for example, the instructions and / or computer code discussed herein.

[0147] In this disclosure, reference to a singular item should be understood to include the plural item, and vice versa, unless expressly stated otherwise or clear from the context. Grammatical conjunctions are intended to express any and all logical and conjunction combinations of joined clauses, sentences, words, etc., unless expressly stated otherwise or clear from the context. Thus, the term "or" should generally be understood to mean "and / or," etc. The use of any and all examples or exemplary language (such as, for example, "such as," "including," etc.) provided herein is intended merely to better clarify the embodiments and does not pose limitations on the scope of the embodiments or claims.

[0148] Some embodiments and / or methods described herein may be implemented by software (executed on hardware), hardware, or a combination thereof. Hardware modules may include, for example, general-purpose processors, field-programmable gate arrays (FPGAs), and / or application-specific integrated circuits (ASICs). Software modules (executed on hardware) may be expressed in various software languages ​​(e.g., computer code), including C, C++, Java, Ruby, Visual Basic, and / or other object-oriented, procedural, or other programming languages ​​and development tools. Examples of computer code include, but are not limited to, microcode or microinstructions, machine instructions such as those generated by a compiler, code used to create web services, and files containing high-level instructions executed by a computer using a translator. For example, embodiments may be implemented using imperative programming languages ​​(e.g., C, Fortran, etc.), functional programming languages ​​(e.g., Haskell, Erlang, etc.), logic programming languages ​​(e.g., Prolog), object-oriented programming languages ​​(e.g., Java, C++), or other suitable programming languages ​​and / or development tools. Additional examples of computer code include, but are not limited to, control signals, encrypted code, and compressed code.

[0149] Various embodiments of the present invention have been described and illustrated herein. It should be understood that various other tools, components, elements, means, and / or structures for performing a function and / or obtaining a result and / or obtaining one or more of the results and / or advantages described herein are contemplated. Each such variation and / or modification is considered within the scope of the embodiments described herein. Furthermore, it is understood and appreciated that all parameters, dimensions, materials, and configurations described herein are exemplary, and that the actual parameters, dimensions, materials, and / or configurations may depend on the particular application to which the disclosed embodiments are utilized and / or applied. Those skilled in the art will recognize and / or ascertain using no more than routine experimentation, many equivalents to the specific novel embodiments and embodiments described herein. Accordingly, the foregoing embodiments are presented by way of example only, and it should be understood that, within the scope of the disclosed embodiments and their equivalents, embodiments may be practiced otherwise than as specifically described and / or illustrated. Embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more of such features, systems, articles, materials, kits, and / or methods is included within the scope of this disclosure, provided that such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

[0150] The above-described embodiments can be implemented in any of numerous ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. If implemented in software, the software code can be executed on any suitable processor or collection of processors provided in various ways.

[0151] Additionally, it should be understood that the present disclosure can be used in conjunction with other systems, including medical imaging systems, surgical systems, treatment systems, and data processing systems, any of which can be embodied in any of several ways. Embodiments include methods and apparatus for efficiently rendering and updating interactive, multi-user virtual reality environments that can be provided to multiple users via multiple user devices connected in a multi-user VR system. As used herein, in some embodiments, a VR system can include an augmented reality (AR) system. The present disclosure includes novel and advantageous methods, systems, and apparatus for rendering, managing, recording, and playing back interactive, multi-user VR / AR presentations and experiences, including medical / surgical data, multi-dimensional data converted for VR / AR, and visual data, including real-time and / or near-real-time data, images, video, audio, streaming media, CT data, MRI data, EEG data, MEG data, and the like. In some implementations, some or all of the multi-dimensional data is near-real-time or real-time, and interactions between multiple users can be similarly coordinated to take advantage of such improvements, for example, during surgery or education. Embodiments can be configured to reduce the bandwidth required when multiple users are interacting and manipulating data in a shared VR / AR environment. By way of example, the teachings of the present disclosure may be integrated with the teachings of U.S. Patent Publication Nos. 2013 / 0245461 and / or 2016 / 0191887 and / or U.S. Patent No. 6,256,529, the entireties of each of which are expressly incorporated herein by reference for all purposes.

[0152] The various methods and / or processes outlined herein may be coded as software executable on one or more processors using any one of a variety of operating systems or platforms. Further, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine code or intermediate code that runs on a framework or virtual machine.

[0153] In this regard, the various concepts disclosed may be embodied as a non-transitory computer-readable storage medium (or multiple computer-readable storage media) (e.g., computer memory, one or more floppy disks, compact disks, optical disks, magnetic tapes, flash memories, field programmable gate arrays or other semiconductor devices, or other non-transitory or tangible computer storage medium circuitry) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement various embodiments of the disclosure above. The computer-readable medium or media may be transportable such that the program or programs stored thereon can be loaded onto one or more different computers / computing devices and / or other processors to implement various aspects of the present disclosure described above.

[0154] The terms "program" or "software" are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be used to program a computer or other processor to perform various aspects of the above-described embodiments. Furthermore, it should be understood that, according to one aspect, one or more computer programs that, when executed, perform the methods of the present disclosure can be distributed in a modular manner among many different computing devices / processors to implement various aspects of the present disclosure.

[0155] Processor-executable instructions may be in many forms, such as program modules executed by one or more computing devices, and may include routines, programs, objects, components, data structures, etc. that perform particular tasks and implement particular data types, and the functionality may be combined and / or distributed as appropriate in various embodiments.

[0156] The data structure may be stored in a variety of suitable formats on a processor-readable medium. For ease of explanation, the data structure may be depicted as having fields related by location within the data structure. Such relationships may be achieved by assigning storage to the fields with locations within the processor-readable medium that also convey the relationship between the fields. However, any suitable mechanism / tool ​​may be used to establish relationships between information within fields of the data structure, including the use of pointers, tags, or other mechanisms / tools that establish relationships between data elements.

[0157] Various disclosed concepts may be embodied as one or more methods, examples of which are provided. The acts performed as part of a particular method may be ordered in any suitable manner. Thus, even if some acts are shown as sequential acts in an example embodiment, embodiments may be constructed in which acts are performed in a different order than shown / described, including performing some acts simultaneously.

[0158] All definitions defined and used herein should be understood to control for dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0159] The use of flow diagrams does not imply limitations regarding the order of operations performed. The subject matter described herein may show different components contained within or connected to different other components. It should be understood that such depicted architectures are illustrative examples, and that in fact many other architectures can be implemented in accordance with the teachings herein to achieve the same functionality. In a conceptual sense, an arrangement of components to achieve the same functionality is effectively "associated" such that the desired functionality is achieved. Thus, any two components herein that are combined to achieve a particular function can be considered to be "associated" with each other such that the desired functionality is achieved, regardless of the architecture or intermediate components. Similarly, any two components so associated can also be considered to be "operably connected" or "operably coupled" to each other to achieve the desired functionality. Any two components that can be so associated are "operably coupleable" to each other to achieve the desired functionality. Specific examples of operably coupleable include, but are not limited to, physically coupleable and / or physically interacting components and / or wirelessly interacting and / or wirelessly interacting components and / or logically interacting and / or logically interacting components.

[0160] As used herein, the indefinite articles "a" and "an," unless clearly indicated to the contrary, should be understood to mean "at least one."

[0161] The phrase "and / or" as used herein should be understood to mean "either or both" of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with "and / or" should be construed in the same manner, i.e., "one or more" of the elements so conjoined. Other elements other than the elements specifically identified by the "and / or" clause may optionally be present, whether related to the elements specifically identified. Thus, as a non-limiting example, a reference to "A and / or B," when used in combination with open-ended language such as "comprising," can refer in one embodiment to A only (optionally including elements other than B); in another embodiment to B only (optionally including elements other than A); in yet another embodiment to both A and B (optionally including other elements); etc.

[0162] As used herein, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" shall be interpreted as inclusive, that is, including at least one of a number or list of elements, but including a plurality, and optionally including additional unlisted items. When only terms clearly indicated to the contrary, such as "only one" or "exactly one," or when used in the claims, "consisting of" refers to the inclusion of exactly one element of a number or list of elements. In general, the term "or" as used herein shall be interpreted to indicate exclusive alternatives (i.e., "one or the other, but not both") only when preceded by terms of exclusivity, such as "either," "one," "either," "only one," or "only one." "Consisting essentially of" when used in the claims shall have its ordinary meaning as used in the field of patent law.

[0163] As used herein, the phrase "at least one" in connection with a list of one or more elements should be understood to mean at least one element selected from any one or more elements in the list of elements, but not necessarily including at least one of all elements specifically listed in the list of elements, and not excluding combinations of elements in the list of elements. This definition also allows for elements other than those specifically identified in the list of elements to which the phrase "at least one" refers, whether related to the specifically identified elements or not, can optionally be present. Thus, as a non-limiting example, "at least one of A and B" (or, equivalently, "at least one of A or B," or, equivalently, "at least one of A and / or B") can refer to, in one embodiment, at least one, optionally two or more, including A, and no B (and optionally including elements other than B); in another embodiment, at least one, optionally two or more, including B, and no A (and optionally including elements other than A); in yet another embodiment, at least one, optionally two or more, including A, and at least one, optionally two or more, including B (and optionally including other elements), etc.

[0164] As used herein, all transitional phrases such as "comprise," "include," "carry," "have," "contain," "involve," "hold," "consist," and the like, shall be understood to mean open-ended, i.e., including but not limited to. Only the transitional phrases "consisting of" and "consisting essentially of" shall be closed or semi-closed transitional phrases, respectively, as set forth in Section 2111.03 of the United States Patent Office Manual of Patent Examining Procedure.

Claims

1. a volatile memory; a non-volatile memory; a first processor operatively coupled to the non-volatile memory, the first processor configured to receive a dataset associated with an object and user information associated with a first spatial location of a first user referencing the object in a virtual environment, the dataset including a spatial data sequence of the object, and the first processor configured to receive a set of inputs from the first user, the set of inputs being associated with a set of predetermined properties associated with portions of the object; a second processor operatively coupled to the volatile memory, the second processor configured to render a first instance of the portion of the object in the virtual environment based on the user information and the set of inputs, the first instance of the portion of the object from a first perspective of the first user, the first perspective being based on a first spatial location of the first user; Equipped with at least one of the first processor and the second processor is configured to generate a data package associated with the object and transmit the data package to a user device associated with a second user so that a second instance of the portion of the object can be rendered on the user device associated with the second user; the second processor is configured to receive manipulation information related to a manipulation of the first instance of the portion of the object by the first user; the second processor is configured to render, based on the manipulation information, an updated first instance of the portion of the object that manifests an effect of the manipulation in the virtual environment; at least one of the first processor and the second processor is configured to generate an updated data package including the manipulation information, and transmit the updated data package to the user device associated with the second user so that an updated second instance of the portion of the object can be rendered at the user device associated with the second user, the updated second instance of the portion of the object manifesting the effect of the manipulation; at least one of the first processor and the second processor is configured to record media associated with the rendering of the updated first instance of the portion of the object, the media including spatial information associated with the rendering of the updated first instance of the portion of the object from the first viewpoint of the first user, and the media is configured to be played back to render the updated first instance of the portion of the object from a second viewpoint of the second user at a second spatial location, the second viewpoint being different from the first viewpoint of the first user.

2. the operation information is first operation information associated with a first operation by the first user, the effect of the first operation is a first effect, the updated first instance of the portion of the object is a first updated first instance of the portion of the object, and the updated data package is a first updated data package, and the first processor further: receiving a second updated data package from the user device associated with the second user, the second updated data package including second operation information related to a second operation of the second instance of the portion of the object, the second operation being performed by the second user and related to manifestation of a second effect on the user device associated with the second user; at least one of the first processor and the second processor renders, based on the second manipulation information, a second updated first instance of the portion of the object that manifests the second effect of the second manipulation; The device of claim 1 configured to:

3. at least one of the first processor and the second processor is configured to receive a set of measurement inputs from the first user and, based on the set of measurement inputs, add measurement annotations to the dataset such that the measurement annotations are presented with the rendering of the first instance of the portion of the object; The apparatus of claim 1 , wherein the updated data package includes the data set and the measurement annotations, and the updated second instance of the portion of the object includes a representation of the measurement annotations.

4. 10. The apparatus of claim 1, wherein the spatial data series comprises a series of multidimensional data comprising images or point cloud data collected using at least one of computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography, confocal imaging, optical microscopy, electron microscopy, electroencephalography (EEG), electrocardiography (MCG), and magnetoencephalography.

5. At least one of the first processor and the second processor receiving instructions from the first user relating to sharing of the data package; 2. The apparatus of claim 1, configured to, based on the instructions, transmit the data package to the user device associated with the second user, render the second instance of the portion of the object on the user device associated with the second user, such that a portion of the data package is stored in a volatile memory associated with the user device associated with the second user but not in a non-volatile memory associated with the user device for a period of time, and after the period of time has elapsed, the portion of the data package is erased from the volatile memory associated with the user device associated with the second user.

6. the set of predetermined properties includes a transparency property and a color property, the set of inputs includes a set of keys, each key from the set of keys including a first value associated with the transparency property and a second value associated with the color property; The apparatus of claim 1 , wherein the second processor is configured to generate a lookup table by interpolating information obtained from each key from the set of keys.

7. the data package is a first data package; At least one of the first processor and the second processor generating a second data package associated with the object; configured to receive instructions from the first user relating to sharing of the second data package; the instructions include authorization from the first user and authorization to the third user to store the second data package in a non-volatile memory associated with a user device associated with a third user; At least one of the first processor and the second processor 2. The apparatus of claim 1, configured to: based on the instructions, send the second data package and the permission to the user device associated with the third user, such that for a certain period of time, a portion of the second data package is stored in a volatile memory associated with the user device associated with the third user and also in a non-volatile memory associated with the user device associated with the third user, to render a third instance of the portion of the object on the user device associated with the third user; and after the certain period of time, based on the permission, erase the portion of the second data package from the volatile memory but not from the non-volatile memory.

8. 1. A non-transitory processor-readable medium storing code representing instructions for execution by a processor, the code causing the processor to: receiving data including a medical image associated with the object; extracting a dataset associated with a structure of the object and user information associated with a first spatial position of a first user referencing the object in a virtual environment; receiving a set of inputs from the first user, the set of inputs being associated with a set of predetermined properties associated with portions of the object in the virtual environment; rendering an instance of the portion of the object in the virtual environment from a first perspective of the first user based on the set of inputs and the user information, the first perspective being based on the first spatial location of the first user; generating a data package associated with the object; transmitting the data package to a user device associated with a second user so that a second instance of the portion of the object can be rendered on the user device associated with the second user; receiving manipulation information related to manipulation of the rendered instance of the portion of the object in the virtual environment by the first user; rendering an updated first instance of the portion of the object that manifests the effect of the manipulation in the virtual environment based on the manipulation information; generating an updated data package including the operational information; transmitting the updated data package to the user device associated with the second user so that an updated second instance of the portion of the object may be rendered at the user device associated with the second user, the updated second instance of the portion of the object manifesting the effect of the manipulation; and recording media associated with the manipulation and the rendering of the updated first instance of the portion of the object, the media including spatial information associated with the rendering of the updated first instance of the portion of the object from the first viewpoint of the first user, the media being configured to be played back to render the updated first instance of the portion of the object from a second viewpoint of the second user at a second spatial location, the second viewpoint being different from the first viewpoint of the first user; a non-transitory processor-readable medium containing code for causing a

9. the set of predetermined properties includes transparency, position, visibility, and color, the set of inputs includes a set of keys, each key from the set of keys configured to provide information including a first value associated with a transparency property and a second value associated with a color property; The code causes the processor to: causing the first instance of the portion of the object to be rendered; interpolating information obtained from each key from the set of keys to generate a lookup table associated with the predetermined property from the set of predetermined properties; 9. The non-transitory processor-readable medium of claim 8, further comprising code for causing the rendering of the instance to be based on the lookup table.

10. The code causes the processor to: receiving a set of annotation inputs from the first user; displaying the first instance of the portion of the object along with an annotation associated with the portion of the object based on the annotation input; generating an updated data set, the updated data set including the annotations embedded in association with the portion of the object such that the annotations are presented with a rendering of the portion of the object; storing the updated data set in a non-volatile memory associated with the processor; The non-transitory processor-readable medium of claim 8 , further comprising code for causing:

11. The data package is a first data package, and the code causes the processor to: generating a second data package associated with the object; receiving instructions from the first user relating to sharing of the second data package; transmitting the second data package to the user device associated with a third user and rendering a second instance of the portion of the object on the user device such that, for a period of time, the portion of the second data package is stored in volatile memory associated with the user device associated with the third user and is not stored in non-volatile memory associated with the user device associated with the third user, wherein, after the period of time has elapsed, the portion of the second data package is erased from the volatile memory associated with the user device associated with the third user; The non-transitory processor-readable medium of claim 8 , further comprising code to cause:

12. 10. The non-transitory processor-readable medium of claim 8, wherein the dataset comprises a series of multidimensional data comprising images or point cloud data collected using at least one of computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography, confocal imaging, optical microscopy, electron microscopy, electroencephalography (EEG), electrocardiography (MCG), and magnetoencephalography.

13. The code causes the processor to: receiving device information associated with the processor, the device information including sampling information; defining a scaling factor based on the sampling information, wherein the rendering of the updated first instance of the portion of the object is based on the scaling factor; The non-transitory processor-readable medium of claim 8 , further comprising code to cause:

14. The code causes the processor to: receiving authorization information associated with the data; storing the data in a non-volatile memory associated with the processor based on the authorization information; The non-transitory processor-readable medium of claim 8 , further comprising code to cause:

Images