Reconstructing complex medical environments in virtual reality

Abstract

Techniques for providing, within at least one virtual reality headset, a simulated medical environment are presented. The techniques include: obtaining a video of a medical environment captured by a smart phone; training a neural network based on the video; extracting a surface mesh using the neural network; and providing, within the at least one virtual reality headset, the simulated medical environment based on the surface mesh.

Description

RECONSTRUCTING COMPLEX MEDICAL ENVIRONMENTS IN VIRTUAL REALITYRelated Application

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 569,334, filed March 25, 2024, and entitled, “Reconstructing Complex Medical Environments in Virtual Reality.”Government Support

[0002] This invention was made with government support under grant no. #2239077, awarded by the National Science Foundation. The government has certain rights in the invention.Field

[0003] This disclosure relates generally to virtual reality medical environments.Background

[0004] Procedures, such surgery-related tasks, performed in an operating room, would benefit from advanced preparation. For example, operating room team members may better perform various tasks that require operating room spatial awareness if they were to practice such tasks ahead of the actual procedure. Nonlimiting example tasks include removing operating room hardware, providing operating room hardware, facilitating a surgery, or transferring a patient.

[0005] However, operating room time is a precious resource. Further, planning the space requirements and layout of the needed tools and machinery can be prohibitively difficult and costly. The targeted spaces are often unavailable due to thehigh demand and cost of operating rooms, and because planners or the machinery might still be remote at the time of planning. Therefore, it is typically difficult for operating room team members to rehearse surgery-related tasks in the actual operating room ahead of time.

[0006] Existing virtual reality solutions for this situation, while helpful, have significant limitations. They are typically costly and labor-intensive, making them difficult to adapt or extend with new needs or machines, especially for novices. Creating digital assets of rooms and tools is also expensive and complex. LiDAR scanners and photogrammetry can be precise, but require expertise and postprocessing. SLAM-like methods are fast to create output and straightforward to use, but produce noisy and low-quality output.

[0007] Neural Radiance Fields (NeRFs), another prior art technique, can generate realistic images, but their spatial reconstruction performance can be inadequate, they have poor compatibility with existing rendering pipelines and exhibit high computational demand. In addition, the suitability of state-of-the-art model output as virtual reality environments has not been demonstrated. Moreover, relying on NeRFs directly as an environment representation is challenging, because they have demanding runtime compute requirements and do not explicitly define surfaces. Clear surfaces however are important for collision handling, allowing interaction in virtual reality. Further, the scene geometry is usually modeled by a density neural network. This can make it challenging to export a defined 3D surface to use in other rendering applications, as density thresholding is scene specific and does not always yield the desired results.Summary

[0008] According to various embodiments, a method of providing, within at least one virtual reality headset, a simulated medical environment is presented. The method includes: obtaining a video of a medical environment captured by a smart phone; training a neural network based on the video; extracting a surface mesh using the neural network; and providing, within the at least one virtual reality headset, the simulated medical environment based on the surface mesh.

[0009] Various optional features of the above method embodiments include the following. The video may be less than 30 minutes long. The method may include generating at least one surface texture using a neural surface reconstruction based on the video, and providing the surface texture on at least a portion of the surface mesh. The generating the at least one surface texture may include encoding a plurality of features representing positions within the video, where the training the neural network includes training a signed distance function (SDF) multi-layer perceptron (MLP) using the encoded features. The medical environment may include an operating room. The method may include training at least one operating room team member for an operating room task using the simulated medical environment. The operating room task may include at least one of: removing operating room hardware, providing operating room hardware, facilitating a surgery, or transferring a patient. The simulated medical environment may include at least one movable object. The at least one movable object may include at least one of: a hospital bed, a rolling chair, a surgical robot, a roll-able table, a surgeon avatar, a trolley, a tracking system, a workshop trolley, or a monitor, where a user of the at least one virtual reality headset manipulates the at least one movable object within the simulated medical environment. The method may include including decimating the surface mesh.

[0010] According to various embodiments, a system for providing, within at least one virtual reality headset, a simulated medical environment is presented. The system includes: a non-transitory computer readable medium including instructions; and at least one electronic processor that executes the instructions to perform operations including: obtaining a video of a medical environment captured by a smart phone; training a neural network based on the video; extracting a surface mesh using the neural network; and providing, within the at least one virtual reality headset, the simulated medical environment based on the surface mesh.

[0011] Various optional features of the above system embodiments include the following. The video may be less than 30 minutes long. The operations may further include generating at least one surface texture using a neural surface reconstruction based on the video, and providing the surface texture on at least a portion of the surface mesh. The generating the at least one surface texture may include encoding a plurality of features representing positions within the video, where the training the neural network includes training a signed distance function (SDF) multi-layer perceptron (MLP) using the encoded features. The medical environment may include an operating room. The operations may further include training at least one operating room team member for an operating room task using the simulated medical environment. The operating room task may include at least one of: removing operating room hardware, providing operating room hardware, facilitating a surgery, or transferring a patient. The simulated medical environment may include at least one movable object. The at least one movable object may include at least one of: a hospital bed, a rolling chair, a surgical robot, a roll-able table, a surgeon avatar, a trolley, a tracking system, a workshop trolley, or a monitor, where a user of the at least one virtual reality headset manipulates the at least one movable object within thesimulated medical environment. The operations may further include decimating the surface mesh.

[0012] Combinations, (including multiple dependent combinations) of the above-described elements and those within the specification have been contemplated by the inventors and may be made, except where otherwise indicated or where contradictory.Brief Description of the Drawings

[0013] Various features of the examples can be more fully appreciated, as the same become better understood with reference to the following detailed description of the examples when considered in connection with the accompanying figures, in which:

[0014] Fig. 1 is a flowchart for a method of providing, within at least one virtual reality headset, a simulated medical environment;

[0015] Fig. 2 illustrates a reconstructed operating room with objects in a virtual environment, according to a non-limiting example embodiment analyzed herein;

[0016] Fig. 3 shows a sample photos of reconstructed objects and corresponding reconstructions, according to a non-limiting example embodiment analyzed herein; and

[0017] Fig. 4 illustrates starting objects in a reconstruction according to a nonlimiting example embodiment analyzed herein and a whiteboxed reconstruction.Description of the Examples

[0018] Reference will now be made in detail to example implementations, illustrated in the accompanying drawings. Wherever convenient, the same reference numbers will be used throughout the drawings to refer to the same or like parts. In thefollowing description, reference is made to the accompanying drawings that form a part thereof, and in which is shown by way of illustration specific exemplary examples in which the invention may be practiced. These examples are described in sufficient detail to enable those skilled in the art to practice the invention and it is to be understood that other examples may be utilized and that changes may be made without departing from the scope of the invention. The following description is, therefore, merely exemplary.

[0019] Some embodiments solve the problem of advanced preparation for operating room procedures. For example, some embodiments provide, within one or more virtual reality headsets, a simulated operating room environment. The simulated operating room environment may include one or more user-movable objects, such as a hospital bed, a rolling chair, a surgical robot, a roll-able table, a surgeon avatar, a trolley, a tracking system, a workshop trolley, and / or a monitor. Users may be able to manipulate such virtual object(s) within the simulated operating room environment. Further, because some embodiments accommodate multiple users, operating room team members may practice coordinated surgery-related tasks in a realistic simulated operating room environment prior to an actual operating room procedure, without requiring any actual operating room time.

[0020] Some embodiments generate a simulated operating room environment based on video of the actual operating room captured by a commodity smart phone. Such video may be captured by any person in a short amount of time (e.g., 10-30 minutes) while the operating room is unoccupied. The short video capture time of some embodiments conserves precious operating room time. Moreover, some embodiments allow for operating room team member training and / or preparation within a simulated operating room environment, without time limit, based on the short videocapture of the actual operating room. Further, some embodiments provide realistic textures to surfaces within the simulated operating room environment based on the smart phone video.

[0021] Some embodiments use neural surface reconstruction of smartphone video captures. The reconstructions may be exported as textured polygon meshes, compatible with existing rendering pipelines and various downstream applications. The process may be automated, minimizing the need for specialized 3D capture hardware and post-processing.

[0022] To evaluate a non-limiting example embodiment, the inventors recreated two operating rooms and ten objects through neural reconstruction, then conducted user studies on planning tasks in the resulting virtual reality environments. An analysis of the non-limiting example embodiment, including analyses of task load, presence, perceived utility, and exploration and interaction behavior, as compared to low visual complexity versions of the same environments, is presented throughout this disclosure. These analyses demonstrate significantly increased perceived utility and presence using the neural reconstruction-based environments, combined with higher perceived workload and exploratory behavior.

[0023] Embodiments are not limited to operating room medical environments. For example, embodiments may be used to simulate any of a variety of medical environments, including operating rooms, emergency rooms, patient rooms, etc.

[0024] These and other features and advantages are presented herein in reference to the figures.

[0025] Fig. 1 is a flowchart for a method 100 of providing, within at least one virtual reality headset, a simulated medical environment. The method 100 may include neural surface reconstruction techniques that combine neural rendering with explicitand realistic surface reconstruction. The techniques may provide high-fidelity, detailed 3D virtual objects by optimizing a signed distance field, which provides sharp boundaries, as opposed to prior art density approximators, which provide only fuzzy probabilistic locations. The method 100 may provide a robust and portable solution for reconstructing a complex environment for digital twin-based modeling and planning, providing a practical solution to visualize and navigate Ors, for example.

[0026] At 102, the method 100 includes obtaining a video of a medical environment captured by a smart phone. For the non-limiting example embodiment that is analyzed herein, an Apple iPhone was used, and the temporal length of the video was about 10-30 minutes.

[0027] At 104, the method 100 includes training a neural network based on the video. For the non-limiting example embodiment analyzed herein, a coordinate-based multi-layer perceptron (MLP) was used to represent the scene as signed distance functions (SDF). The non-limiting example embodiment sampled 3D locations along camera view directions and used a multi-resolution hash encoding to encode the positions. The encoded features were input to an SDF MLP and a color MLP to composite images using SDF-based volume rendering.

[0028] In particular, the non-limiting example embodiment employed Instant NGP (Neural Graphics Primitives) as a neural SDF representation of the underlying 3D scene, which was optimized from multi-view image observations via neural surface rendering. The Instant NGP used a hybrid 3D grid structure with a multi-resolution hash encoding and a lightweight MLP that is more expressive with a memory footprint loglinear to the resolution. The hybrid representation greatly increased the representation power of neural fields and successfully represented very fine-grained details for a wide variety of tasks, such as object shape representation and novel viewsynthesis problems. The usage of an SDF MLP, with Instant NGP multi-resolution hash encoding allows for converting short smartphone-based videos into detailed realist virtual reality renderings of operating room environments.

[0029] Training the neural network of the non-limiting example embodiment used numerical gradients to compute higher-order derivatives (e.g., surface normals for the eikonal regularization) which stabilized the optimization. Further, training the non-limiting example embodiment used a progressive optimization schedule, which helped recovering 3D structures at different levels of details.

[0030] Collectively, the techniques used in the neural network of the non-limiting example embodiment leveraged the inherent continuity of MLPs and neural volume rendering, allowing the optimized surfaces to meaningfully interpolate between spatial locations, resulting in smooth and complete surface representations.

[0031] At 106, the method 100 includes extracting a surface mesh using the neural network. The actions of 106 include extracting a surface mesh using the neural network trained per 104.

[0032] At 108, the method 100 includes cutting off excess geometry from the surface mesh. This optional action may be automated, and may be used to remove any erroneous geometry from the surface mesh of 106.

[0033] At 110, the method 100 includes decimating the surface mesh. The actions of 110 may include decimating the mesh so that the target virtual reality device, e.g., the virtual reality headset, can render it using limited memory and computational capabilities. For the non-limiting example embodiment, the mesh decimation depended on the object complexity; objects had 20k to 60k faces.

[0034] At 112, the method 100 includes generating at least one surface texture using a neural surface reconstruction based on the video. For the non-limitingexample embodiment, to optimize the visuals for virtual reality hardware, the meshes were split into two parts of around 400k faces, each with a 4k texture. This improved the visual quality and reduced the creation time and hardware requirements. In general, the surface texture may be generated by encoding features representing positions within the video as described in reference to 104, where the neural network is trained used such encoded features

[0035] At 114, the method 100 includes providing the surface texture on at least a portion of the surface mesh. Note that the surface texture may be applied to only a portion of the mesh, according to various embodiments.

[0036] At 116, the method 100 includes providing, within the at least one virtual reality headset, the simulated medical environment based on the surface mesh. According to some embodiments, the medical environment may include an operating room. The method may further include training at least one operating room team member for an operating room task using the simulated medical environment. The operating room task may include one or more of: removing operating room hardware, providing operating room hardware, facilitating a surgery, or transferring a patient.

[0037] According to some embodiments, the simulated medical environment includes at least one movable object, such as a hospital bed, a rolling chair, a surgical robot, a roll-able table, a surgeon avatar, a trolley, a tracking system, a workshop trolley, or a monitor. The simulation may allow a user to manipulate the movable object within the simulated medical environment.

[0038] This disclosure proceeds to describe a non-limiting example embodiment and an analysis thereof. For the analysis of the non-limiting example embodiment, the inventors performed an experiment in which participants evaluated two virtual reality reconstruction types as applied to two environments. The twoenvironments were a real operating room and a mock operating room, which are shown and described herein in reference to Fig. 2 and Fig. 4 (reference 402), respectively. The two virtual reality reconstruction types were the non-limiting example embodiment, implemented as shown and described herein in reference to Fig. 1 and developed with Unity 2022.3.7, and a “whiteboxed” reconstruction, implemented using a modified embodiment to represent the quality level one could realistically achieve using CAD modeling software in reasonable time without resorting to specialized hardware or relying on experts to truthfully model the environment. The experiment included performing within-subject repeated measures with the reconstruction type and environment type as independent variables. Details of the experiment and analysis follow in reference to Figs. 2, 3, and 4.

[0039] Fig. 2 illustrates a reconstructed operating room with objects in a virtual environment 200, according to a non-limiting example embodiment analyzed herein. The analysis presented herein considers an embodiment as described in reference to Fig. 1 as applied to two environments: a real operating room in daily use in a hospital, and a mock operating room. For the non-limiting example embodiment, the rooms were reconstructed from around 20-25 minutes of wide-angle video each, depending on the complexity and details. An impression from the reconstructions of the real operating room is illustrated by Fig. 2. Fig. 4 (reference 402) illustrates impression from the reconstruction of the mock operating room.

[0040] Fig. 3 shows a sample photos of reconstructed objects (top) and corresponding reconstructions (bottom), according to the non-limiting example embodiment analyzed herein. The reconstructions were implemented in the virtual reality environment as individual and separately manipulable objects, removed from the background shown in the photos. The embodiment and analysis thereof utilizedten different reconstructions of objects found in or around the rooms: a hospital bed, a rolling chair, a surgical robot, a roll-able table with equipment, a surgeon lookalike, a sterile cart, a roll-able tracking system, a tracking system on tripod, a workshop trolley, and a roll-able digital monitor. The objects were reconstructed from around two to four minutes of video. Mesh decimation depended on the object complexity; objects had 20k to 60k faces and a 2k texture. Samples of input images and the resulting 3D models are shown in Fig. 3 for the hospital bed, the tracking system on tripod, the sterile cart, and the workshop trolly.

[0041] Fig. 4 illustrates starting objects in a reconstruction 402 according to a non-limiting example embodiment analyzed herein and a whiteboxed reconstruction 404. Both reconstructions were presented within a virtual reality environment in the experiment. Both environments included the interior of an operating room, as well as ten individually and separately manipulable objects, including, as shown in Fig. 4 by way of example, two tripods, a medical resident, and a table. As clearly depicted in Fig. 4, the reconstruction according to the non-limiting example embodiment is more realistic.

[0042] For the experiment, in addition to the two rooms and ten items, the applications contain a training scene, shown before the actual experiment starts. Users could look around and move freely in the virtual environment, physically within around one square meter, virtually additionally via teleportation. It was possible to create, move (near and far interaction, one- and two-handed), and delete objects. The virtual reality application mirrored the user and objects’ positions onto the supervisor’s laptop.

[0043] For presentation of the virtual reality environment in the experiment, the experimental application was developed to run locally on a Meta Quest 2 headmounted display (HMD) device, using controllers to allow user input. In the experimental setup, a secondary laptop was used to let the study supervisor issue commands and load environments remotely on the headset. Communication between the laptop and HMD was wireless via Wi-Fi.

[0044] Participants in the experiment included 21 individuals, including nine that identified as female and twelve as male. The mean age was M = 24.57 (SD = 1.66) year. Nine participants had little virtual reality experience (less than 5 times overall), nine had some virtual reality experience with occasional use, and three reported high experience with regular use.

[0045] To engage participants in the experiment in the virtual environment, they were tasked to do a simple planning task for the operating room. They had to place, move and remove objects while considering the objective at hand. The instructions were displayed next to the application menu. Included were information about target objects, which constraints to be applied, and the objectives. Additionally, some objects were already placed in the scene, exemplary as a current setup. These could be moved or removed. Actual planning outcomes were not quantified. Examples of the tasks follow.

[0046] Mock Operating Room

[0047] • Task: Update Hardware. Needed: a surgical robot, one or more tracking carts, a TV cart, space for at least two surgeons. Goals: Remove old tripods and place new tracking carts with clean line of view of the patient. Surgeons can easily manipulate the robot and see the monitor. Robot can reach the patient.

[0048] • Task: Prepare long procedure. Needed: Sterile cart, trolley, two or more surgeons, chairs and place to lay down tools. Goal: Surgeons should be able tomove freely, while being able to easily access tools in the trolley and lay them on the table. Chairs should be accessible as needed to sit down without obstructing paths.

[0049] Real Operating Room

[0050] • Task: Plan Surgery. Needed: a surgical robot, a TV cart, tracking trolley, table and space for at least two surgeons. Goal: Surgeons can easily manipulate the robot and reach patient, table and trolley. The TV should be well visible, but not in the way.

[0051] • Task: Transfer Patient. Needed: a surgical robot, a tracking tripod, a monitor and space for patient table, sterile material cart and medical professionals. Goal: Make sure a patient bed can move right next to operating table. Find the best layout to transfer patient, then back to surgery easily, while having access to sterile material.

[0052] The experiment took approximately one hour to complete. At first, the study information was read to the participants, and they then gave verbal consent. Next, participants did a short virtual reality training session for around five minutes, to familiarize themselves with the virtual reality application. Afterward, they started with one of the four conditions in a counterbalanced order. Participants worked on a task in the virtual environment for five minutes. The instructions were given within the environment using text displays next to the menu. After the five minutes were over, participants took off the virtual reality HMD and proceeded to answer the postexperiment questionnaire on a computer. This was repeated four times so that participants experienced both reconstructions in both rooms. After the last condition was completed, they had the option to leave comments on their experience.

[0053] The questionnaire asked the participants’ to evaluate the non-limiting example embodiment and the whiteboxed implementation according to severalcriteria, including the following categories: perceived utility, presence, perceived task load, and familiarity. For perceived utility, a custom scale was used to assess the perceived utility of the application. It included eight questions with the three subscales confidence, memorability, and utility. All questions were to be answered on a 7-point Likert scale with endpoints fully disagree (1 ) and fully agree (7). For presence, the experiment used the Igroup Presence Questionnaire to measure presence in the virtual environment. It included 14 items on four subscales (general, spatial presence, involvement, perceived realness) on a 7-point Likert scale with varying endpoints. To assess perceived task load and performance, the NASA Task load index (NASA-TLX) was used, including all subscales (mental, physical, and temporal demand and performance, effort, and frustration), without pairwise comparisons (raw TLX). Lastly, the participants were asked to rate their familiarity with the presented virtual environment on a 7-point Likert scale. The endpoints of the scale were completely unknown (1 ) and very familiar (7). The Table below sets forth the experimental results for perceived utility, presence, and perceived task load.Table: Analysis of Non-Limiting Example Embodiment

[0054] In the Table, “M” represents mean and “SD” represents standard deviation. As set forth in the Table, the experiment established the superiority of the non-limiting example embodiment according to multiple criteria. For perceived utility, all three subscales were rated significantly higher for the non-limiting example embodiment as compared to the whiteboxed implementation. For presence, all four subscales were rated significantly higher for the non-limiting example embodiment as compared to the whiteboxed implementation. For perceived task load, mental, physical, and effort subscales were rated significantly higher for the non-limiting example embodiment as compared to the whiteboxed implementation. Finally, for familiarity with environment, the participants rated their familiarity with the non-limiting example embodiment mock operating room M = 5.81 (SD = 1 .08, with the whiteboxedmock or M = 4.14 (SD = 1.90), with the non-limiting example embodiment real operating room M = 5.29 (SD = 1.31 ) and with the whiteboxed real operating room M = 3.33 (SD = 1 .88). These results indicate that the non-limiting example embodiment provided users with worthwhile additional details and information to make more informed decisions. Further, the results support the suitability of embodiments for medical planning tasks. Utilizing such visualizations increased perceived utility and presence compared to a low visual fidelity environment, for example.

[0055] Certain examples can be performed using a computer program or set of programs. The computer programs can exist in a variety of forms both active and inactive. For example, the computer programs can exist as software program(s) comprised of program instructions in source code, object code, executable code or other formats; firmware program(s), or hardware description language (HDL) files. Any of the above can be embodied on a transitory or non-transitory computer readable medium, which include storage devices and signals, in compressed or uncompressed form. Exemplary computer readable storage devices include conventional computer system RAM (random access memory), ROM (read-only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), flash memory, and magnetic or optical disks or tapes.

[0056] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented using computer readable program instructions that are executed by an electronic processor.

[0057] These computer readable program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the electronic processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0058] In embodiments, the computer readable program instructions may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, statesetting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the C programming language or similar programming languages. The computer readable program instructions may execute entirely on a user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.

[0059] As used herein, the terms “A or B” and “A and / or B” are intended to encompass A, B, or {A and B}. Further, the terms “A, B, or C” and “A, B, and / or C” areintended to encompass single items, pairs of items, or all items, that is, all of: A, B, C, {A and B}, {A and C}, {B and C}, and {A and B and C}. The term “or” as used herein means “and / or.”

[0060] As used herein, language such as “at least one of X, Y, and Z,” “at least one of X, Y, or Z,” “at least one or more of X, Y, and Z,” “at least one or more of X, Y, or Z,” “at least one or more of X, Y, and / or Z,” or “at least one of X, Y, and / or Z,” is intended to be inclusive of both a single item (e.g., just X, or just Y, or just Z) and multiple items (e.g., {X and Y}, {X and Z}, {Y and Z}, or {X, Y, and Z}). The phrase “at least one of” and similar phrases are not intended to convey a requirement that each possible item must be present, although each possible item may be present.

[0061] The techniques presented and claimed herein are referenced and applied to material objects and concrete examples of a practical nature that demonstrably improve the present technical field and, as such, are not abstract, intangible or purely theoretical. Further, if any claims appended to the end of this specification contain one or more elements designated as “means for [perform]ing [a function]...” or “step for [performing [a function]...”, it is intended that such elements are to be interpreted under 35 U.S.C. § 112(f). However, for any claims containing elements designated in any other manner, it is intended that such elements are not to be interpreted under 35 U.S.C. § 112(f).

[0062] While the invention has been described with reference to the exemplary examples thereof, those skilled in the art will be able to make various modifications to the described examples without departing from the true spirit and scope. The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. In particular, although the method has been described by examples, the steps of the method can be performed in a different order than illustrated orsimultaneously. Those skilled in the art will recognize that these and other variations are possible within the spirit and scope as defined in the following claims and their equivalents.

Claims

What is claimed is:1 . A method of providing, within at least one virtual reality headset, a simulated medical environment, the method comprising: obtaining a video of a medical environment captured by a smart phone; training a neural network based on the video; extracting a surface mesh using the neural network; and providing, within the at least one virtual reality headset, the simulated medical environment based on the surface mesh.

2. The method of claim 1 , wherein the video is less than 30 minutes long.

3. The method of claim 1 , further comprising generating at least one surface texture using a neural surface reconstruction based on the video, and providing the surface texture on at least a portion of the surface mesh.

4. The method of claim 3, wherein the generating the at least one surface texture comprises encoding a plurality of features representing positions within the video, wherein the training the neural network comprises training a signed distance function (SDF) multi-layer perceptron (MLP) using the encoded features.

5. The method of claim 1 , wherein the medical environment comprises an operating room.

6. The method of claim 5, further comprising training at least one operating room team member for an operating room task using the simulated medical environment.

7. The method of claim 6, wherein the operating room task comprises at least one of: removing operating room hardware, providing operating room hardware, facilitating a surgery, or transferring a patient.

8. The method of claim 1 , wherein the simulated medical environment comprises at least one movable object.

9. The method of claim 8, wherein the at least one movable object comprises at least one of: a hospital bed, a rolling chair, a surgical robot, a roll-able table, a surgeon avatar, a trolley, a tracking system, a workshop trolley, or a monitor, and wherein a user of the at least one virtual reality headset manipulates the at least one movable object within the simulated medical environment.

10. The method of claim 1 , further comprising decimating the surface mesh.

11. A system for providing, within at least one virtual reality headset, a simulated medical environment, the system comprising: a non-transitory computer readable medium comprising instructions; and at least one electronic processor that executes the instructions to perform operations comprising:obtaining a video of a medical environment captured by a smart phone; training a neural network based on the video; extracting a surface mesh using the neural network; and providing, within the at least one virtual reality headset, the simulated medical environment based on the surface mesh.

12. The system of claim 11 , wherein the video is less than 30 minutes long.

13. The system of claim 11 , wherein the operations further comprise generating at least one surface texture using a neural surface reconstruction based on the video, and providing the surface texture on at least a portion of the surface mesh.

14. The system of claim 13, wherein the generating the at least one surface texture comprises encoding a plurality of features representing positions within the video, wherein the training the neural network comprises training a signed distance function (SDF) multi-layer perceptron (MLP) using the encoded features.

15. The system of claim 11 , wherein the medical environment comprises an operating room.

16. The system of claim 15, wherein the operations further comprise training at least one operating room team member for an operating room task using the simulated medical environment.

17. The system of claim 16, wherein the operating room task comprises at least one of: removing operating room hardware, providing operating room hardware, facilitating a surgery, or transferring a patient.

18. The system of claim 11 , wherein the simulated medical environment comprises at least one movable object.

19. The system of claim 18, wherein the at least one movable object comprises at least one of: a hospital bed, a rolling chair, a surgical robot, a roll-able table, a surgeon avatar, a trolley, a tracking system, a workshop trolley, or a monitor, and wherein a user of the at least one virtual reality headset manipulates the at least one movable object within the simulated medical environment.

20. The system of claim 11 , wherein the operations further comprise decimating the surface mesh.

Images