Mobile virtual reality system for surgical robotic system

Abstract

A mobile virtual reality system for simulating, training, or demonstrating a surgical robotic system can include a virtual reality processor. The processor can generate a virtual surgical robot and render the virtual surgical robot on a display. The virtual surgical robot can include virtual surgical tools. Hand-held user input devices (UIDs) can sense hand inputs from hands. Foot input devices can sense foot inputs from feet. The virtual reality processor can be configured to control movement or action of the virtual surgical robot based on the hand inputs and change which virtual surgical instrument is controlled by one or more hand-held UIDs based on the foot inputs. Other embodiments and aspects are disclosed and claimed.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to surgical robotic systems, and more particularly to a mobile virtual reality system for simulating, training, or demonstrating a surgical robotic system and / or procedure. Other embodiments are described. BACKGROUND

[0002] Minimally invasive surgery (MIS), such as laparoscopic surgery, involves techniques intended to reduce tissue damage during a surgical procedure. For example, a laparoscopic procedure typically involves forming a plurality of small incisions in a patient (e.g., in the abdomen), and introducing one or more tools and at least one camera into the patient’s body through the incisions. A surgical procedure can then be performed by using the introduced surgical tools, with visualization assistance provided by the camera.

[0003] Generally, MIS provides multiple benefits, such as reduced patient scarring, reduced patient pain, shortened patient recovery periods, and reduced medical costs associated with patient recovery. MIS can be performed with a surgical robotic system that includes one or more robotic arms for manipulating surgical tools based on commands from a remote operator. The robotic arms can support, for example, at their distal ends, various devices such as surgical end effectors, imaging devices, cannulas for providing access to a patient’s body cavities and organs, and the like. Thus, surgical robotic arms can assist in performing surgical procedures.

[0004] Control of such robotic systems can require a user (e.g., a surgeon or other operator) to make control inputs via one or more user interface devices that translate the user’s manipulations or commands into control of the robotic system. For example, a tool driver having one or more motors can actuate one or more degrees of freedom of a surgical tool in response to user commands when the surgical tool is positioned at a surgical site of a patient. SUMMARY

[0005] In one aspect, it can be desirable to use a virtual environment (e.g., virtual reality, mixed reality, or augmented reality) to simulate, train, or demonstrate a surgical robotic system and / or procedure. As such, a medical professional (e.g., a surgeon) can advantageously become familiar with a surgical robotic system and procedure in a virtual environment without the need for an entire physical surgical robotic system (e.g., surgical robotic arms, platforms, control stations, operating rooms, etc.). A virtual reality system that simulates a physical surgical robotic system can be utilized to simulate a surgical robotic system and procedure, including an immersive virtual environment (e.g., a 3D display in a headset). A mature virtual reality system can be bulky and require external sensors (e.g., cameras placed in one or more areas of a physical environment to detect and sense a user), which can make setup difficult. Such systems can also require wiring and routing of wires / cables. As such, such systems are not conveniently mobile as they are difficult to move from one location to another. Thus, it is advantageous to provide a virtual reality system that is mobile (e.g., with minimal component and setup complexity) such that the system can be effectively transported from one location to another, e.g., to train a medical professional.

[0006] Mobile virtual reality can run a fully implemented patient simulation using real hardware and inside-out tracking. Such a system can not have external trackers (e.g., cameras). Foot pedals or foot tracking (e.g., optical tracking of the feet with sensors or inside-out cameras) can be designed into such a system. The simulation can be driven with a real robot model that indicates a real surgical robotic system used in a real procedure. Such a system can be easily assembled and passed between medical professionals for training.

[0007] In one aspect, a mobile virtual reality system for simulating, training, or demonstrating a surgical robotic system includes a processor, a display for receiving and showing a virtual surgical robot based on a data stream generated by the processor, the virtual surgical robot including a plurality of virtual surgical instruments, one or more handheld user input devices (UIDs) that sense hand input from a hand, and one or more foot input devices that sense foot input from a foot. The processor can be configured to control movement of the virtual surgical robot system based on the hand input and to change which virtual surgical instrument is controlled by the one or more handheld UIDs. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 A surgical robotic system is shown in accordance with one embodiment.

[0009] Figure 2A virtual reality system for simulating, training, or demonstrating a surgical robotic system is shown in accordance with one embodiment.

[0010] Figures 3A to 3C A foot input device is shown in accordance with various embodiments.

[0011] Figure 4 A virtual reality system for simulating, training, or demonstrating a surgical robotic system is shown in accordance with one embodiment.

[0012] Figure 5 A virtual interface is shown in accordance with one embodiment.

[0013] Figure 6 A flowchart of a user interface or process is shown in accordance with one embodiment. DETAILED DESCRIPTION

[0014] Examples of various aspects and variations of the present invention are described herein and shown in the accompanying drawings. The following description is not intended to limit the present invention to these embodiments, but rather to enable a person skilled in the art to make and use the present invention.

[0015] The following specification and drawings are illustrative of the present disclosure and are not intended to be limiting of the present disclosure. Numerous specific details are described to provide a thorough understanding of various embodiments of the present disclosure. However, in certain instances, well-known or conventional details are not described in order to make the discussion of embodiments of the present disclosure more concise.

[0016] Reference in the specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the disclosure. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.

[0017] Reference Figure 1 is a pictorial view of an exemplary surgical robotic system 1 in a surgical setting. The robotic system 1 includes a user console 2, a control tower 3, and one or more surgical robotic arms 4 at a surgical robotic platform 5 (e.g., table, bed, etc.). The system 1 can incorporate any number of devices, tools, or accessories used to perform a surgical procedure on a patient 6. For example, the system 1 can include one or more surgical tools 7 used to perform a surgical procedure. The surgical tools 7 can be end effectors attached to a distal end of the surgical arms 4 for performing a surgical procedure.

[0018] Each surgical tool 7 can be manually manipulated, robotically manipulated, or both, during a surgical procedure. For example, the surgical tools 7 can be tools for accessing, viewing, or manipulating the internal anatomy of the patient 6. In one embodiment, the surgical tools 7 are graspers that can grasp tissue of the patient. The surgical tools 7 can be manually controlled by the bedside operator 8; or they can be robotically controlled via actuated movement of the surgical robotic arms 4 to which they are attached. The robotic arms 4 are shown as an over-table installation system, but in other configurations, the arms 4 can be installed on a cart, ceiling, or sidewall, or in another suitable structural support.

[0019] In general, a remote operator 9, such as a surgeon or other operator, can use the user console 2 to remotely manipulate the arms 4 and / or attached surgical tools 7, e.g., teleoperation. The user console 2 can be located in the same operating room as the rest of the system 1, as shown. However, in other environments, the user console 2 can be located in an adjacent or nearby room, or it can be located at a remote location, e.g., in a different building, city, or country. The user console 2 can include a seat 10, foot controls 13, one or more handheld user input devices (UIDs) 14, and at least one user display 15 configured to display, e.g., a view of a surgical site within the patient 6. In the example user console 2, the remote operator 9 sits in the seat 10 and views the user display 15 while manipulating the foot controls 13 and handheld UIDs 14 in order to remotely control the arms 4 and surgical tools 7 (which are mounted on distal ends of the arms 4). Figure 1

[0020] In some variations, the bedside operator 8 can also operate the system 1 in a “bedside” mode, in which the bedside operator 8 (user) is now located at the side of the patient 6 and simultaneously manipulates both the robotically-driven tools (end effectors attached to the arms 4), e.g., holding a handheld UID 14 in one hand and a manual laparoscopic tool in the other hand. For example, the bedside operator’s left hand can manipulate the handheld UID to control the robotic components, while the bedside operator’s right hand manipulates the manual laparoscopic tool. Thus, in these variations, the bedside operator 8 can perform both robotically-assisted minimally invasive surgical procedures and manual laparoscopic surgical procedures on the patient 6.

[0021] ​During an example procedure (surgery), a patient 6 is prepared for surgery and draped in a sterile manner to enable anesthesia. Initial access to the surgical site can be performed manually (to facilitate access to the surgical site) while the arms of the robotic system 1 are in a stowed configuration or a retracted configuration. Once access is complete, initial positioning or preparation of the robotic system 1 including its arms 4 can be performed. The surgery then continues with a remote operator 9 at the user console 2 manipulating various end effectors and possibly an imaging system with foot-operated controls 13 and a UID 14 to perform the surgery. Human assistance can also be provided at the surgical bed or table by a bed-side personnel (e.g., a bed-side operator 8) who can be in a sterile surgical gown, who can perform tasks such as retracting tissue, performing manual repositioning, and tool changes on one or more of the robotic arms 4. There can also be non-sterile personnel to assist the remote operator 9 at the user console 2. When the procedure or surgery is complete, the system 1 and user console 2 can be configured or set into a state to facilitate completion of post-operative procedures, such as cleaning or sterilization and inputting or printing health records via the user console 2.

[0022] In one embodiment, the remote operator 9 holds and moves the UID 14 to provide input commands to move the robotic arm actuators 17 in the robotic system 1. The UID 14 can be communicatively coupled to the rest of the robotic system 1, for example, via the console computer system 16. The UID 14 can generate a spatial state signal corresponding to the movement of the UID 14, for example, the position and orientation of the handheld housing of the UID, and the spatial state signal can be an input signal that controls the motion of the robotic arm actuators 17. The robotic system 1 can use control signals derived from the spatial state signal to control the proportional motion of the actuators 17. In one embodiment, a console processor of the console computer system 16 receives the spatial state signal and generates a corresponding control signal. Based on these control signals of how to energize the actuators 17 to move segments or links of the arms 4, the movement of the corresponding surgical tools attached to the arms can mimic the movement of the UID 14. Similarly, interactions between the remote operator 9 and the UID 14 can generate, for example, a clamp control signal that causes the jaws of a grasper of the surgical tool 7 to close and clamp tissue of the patient 6.

[0023] The surgical robotic system 1 can include several UIDs 14, with a respective control signal being generated for each UID that controls an actuator of a respective arm 4 and a surgical tool (end effector). For example, the remote operator 9 can move a first UID 14 to control the movement of an actuator 17 located in a left robotic arm, with the actuator responding by moving links, gears, etc. in the arm 4. Similarly, movement of the second UID 14 by the remote operator 9 controls the movement of another actuator 17, which in turn moves other links, gears, etc. of the robotic system 1. The robotic system 1 can include a right arm 4 that is fixed to a bed or table on the right side of the patient, and a left arm 4 that is located on the left side of the patient. The actuator 17 can include one or more motors that are controlled so that they drive a joint of the arm 4 to rotate, for example, to change the orientation of an endoscope or grasper of a surgical tool 7 attached to the arm relative to the patient. The movement of several actuators 17 in the same arm 4 can be controlled by spatial state signals generated from a particular UID 14. The UID 14 can also control the movement of a respective surgical tool grasper. For example, each UID 14 can generate a respective clamp signal to control the movement of an actuator (e.g., a linear actuator) that opens or closes the jaws of a grasper at a distal end of the surgical tool 7 to grasp tissue within the patient 6.

[0024] In some aspects, communication between the platform 5 and the user console 2 can pass through a control tower 3, which can translate user commands received from the user console 2 (and more specifically from the console computer system 16) into robotic control commands that are transmitted to the arms 4 on the robotic platform 5. The control tower 3 can also transmit status and feedback from the platform 5 back to the user console 2. The communication connections between the robotic platform 5, the user console 2, and the control tower 3 can be via wired and / or wireless links, using any suitable data communication protocol of various data communication protocols. Any wired connections can optionally be built into the floor and / or walls or ceiling of the operating room. The robotic system 1 can provide video output to one or more displays, including displays within the operating room as well as remote displays that can be accessed via the internet or other network. The video output or feed can also be encrypted to ensure privacy, and all or part of the video output can be saved to a server or electronic health record system.

[0025] Surgical robotic arms can have movable, articulated, and / or motorized members with multiple degrees of freedom that can hold various tools or accessories at a distal end. An example system includes the da Vinci® Surgical System, which can be used for minimally invasive surgical procedures (e.g., urological procedures, general laparoscopic surgical procedures, gynecological laparoscopic surgical procedures, general non- cardiovascular thoracoscopic surgical procedures, and thoracoscopic-assisted open-heart procedures). A “virtual surgical robotic arm” can be a computer-generated model of a robotic arm rendered on a user-set captured video. The virtual surgical robotic arm can be a complex 3D model of a real robotic arm. Alternatively or additionally, the virtual surgical robotic arm can include visual aids such as arrows, tool tips, or other representations directed to providing information about the pose of the robotic arm, such as a geometrically simplified version of the real robotic arm.

[0026] Mobile virtual reality system

[0027] Reference is made to Figure 2 A mobile virtual reality system 40 for simulating, training, or demonstrating a surgical robotic system can include a virtual reality processor 42; a display 44 for receiving and showing a virtual reality environment generated by the processor, the virtual reality environment including a virtual surgical robotic system (e.g., the surgical robotic system shown in Figure 1 Specifically, the processor can generate a data stream including a virtual surgical robot (e.g., one or more surgical robotic arms 4 and / or platforms 5 shown in Figure 1 and one or more virtual surgical instruments (e.g., the surgical instruments shown in Figure 1The virtual reality processor is configured to control movement or action of the virtual surgical robots (including instruments) based on hand input and foot input commanded through sensors on handheld controllers (UIDs). For example, based on hand input, a virtual surgical robot arm or tool in the virtual reality environment can move, thereby simulating a real surgical procedure with the same or similar hand input. Foot input can change which virtual surgical instrument is controlled by one or more handheld UIDs. For example, foot input can switch control from one virtual surgical instrument attached to one virtual surgical robot arm to another virtual surgical instrument attached to another virtual surgical robot arm. In other cases, foot input can switch control between different virtual robot arms or between components of the same virtual robot arm. In other words, with foot input, a user can switch between which device component is actively controlled by hand input. In some embodiments, foot input can act as a clutch to pause control of the virtual surgical robots (including one or more arms and instruments). For example, when foot input indicates that the foot is pressing the foot pedal, all input from the handheld UIDs will be ignored. When foot input indicates that the foot pedal is released, then control of the virtual surgical robots can resume to where it was before the clutch was activated. In one embodiment, the various devices can communicate over a network (e.g., handheld input, processor, foot input, and display). The network can be wired or wireless, and can use known communication protocols (e.g., TCP / IP, CAN, RS-232, etc.).

[0028] In one embodiment, the mobile virtual reality system does not include external stationary components of the tracking system (e.g., cameras). Advantageously, there is no need to install any components of the tracking system in the simulation room, which can reduce the time to prepare a surgical simulation. The lack of external components of the tracking system also provides a more compact and mobile virtual training system. As discussed in other sections, the system can use built-in tracking devices, such as inward-facing cameras, or cameras mounted on a laptop or head-mounted device, rather than external components.

[0029] In one embodiment, the processor and display are integrated with a computing device, such as a laptop. In one embodiment, the display is a two-dimensional screen or a three-dimensional screen (e.g., a multi-view 3D display, a volumetric 3D display, or a digital hologram display). In one embodiment, the display is a 3D wearable display worn on the head of the user. The processor can be integrated with the display, with or without an external computer (e.g., housed in a device such as a laptop or head-mounted computing device).

[0030] Handheld input

[0031] The mobile system may include one or more handheld user input devices (UIDs) 46 that can sense hand input from hand movements. For example, a user's hand may squeeze, rotate, or translate the UID. These hand inputs can be sensed by the UID.

[0032] In one implementation scheme, such as Figure 2 As shown, one or more handheld UIDs 46 include an inside-out tracking module 48 and / or an inertial measurement unit (IMU) 47. The inside-out tracking module 48 may include one or more cameras or other sensors capable of mapping objects outside the UID and their movement. The inside-out tracking module 48 may have different locations on the UID, such as, but not necessarily, a frontal location on the handheld UID (e.g., ...). Figure 2 (As shown), away from where the UID is held. Images from one or more cameras or sensors can be processed to determine the movement and position of the UID, which can be used to control the virtual surgical robot system. IMU 47 may include an accelerometer, a gyroscope, or a combination thereof. In one embodiment, the UID may include one or more switches and / or one or more buttons for squeezing. In one embodiment, the processor is configured to determine hand input (e.g., movement, position, translation, or orientation of the UID) based on input from an inside-out tracking module (e.g., a camera) and / or the IMU.

[0033] Foot input

[0034] The mobile system may include one or more foot input devices that sense foot input from one or more feet. The foot input devices may take different forms. In one embodiment, Figure 3A An optical sensor or external camera 52 is shown for capturing visual data including foot input. A processor (e.g., a VR processor 42 or a dedicated foot input processor) may be configured to determine foot input based on recognizing and tracking foot movement, position, or orientation (e.g., using machine learning and / or trained neural networks).

[0035] In one embodiment, the optical sensor or tracking module 48 may include a camera housed in one or more handheld UIDs 46, such as Figure 2 As shown. The camera may have a wide-angle view or lens, for example, capable of viewing 170° or greater, to capture visual data including foot input from the handheld position. Therefore, in such embodiments, the system can eliminate the need for separate foot input hardware, as the handheld UID and inside-out tracking module can work together to sense foot input.

[0036] In one implementation scheme Figure 3BA foot pedal 50 is shown having one or more sensors, additional proximity / hover sensors, and / or encoders that can sense pressure input from the foot or foot pedal position modified by the foot. Alternatively, a representation of the foot pedal can be projected onto the floor, e.g., from a 3D display or 3D head-mounted display. In one embodiment, as shown in Figure 3C The one or more foot input devices include one or more tracking sensors 54, e.g., inertial measurement units (IMUs). The tracking sensors can include accelerometers, gyroscopes, or combinations thereof. The sensors can be affixed to the foot (e.g., as a shoe, sock, sticker, or strap) to detect movement of the foot. These inputs can be processed by the processor to switch control between controllable members of the virtual surgical robotic system.

[0037] Sensed hand input and foot input

[0038] In one embodiment, a mobile virtual reality system for simulating, training, or demonstrating a surgical robotic system is shown in Figure 4 The system can include a virtual reality processor 62; a display 64 for receiving and showing a virtual reality environment generated by the processor. The virtual reality environment can include a) a virtual surgical robotic system (e.g., a system in a virtual operating room as shown in Figure 1 b) a device for receiving hand input from a hand, and c) a device for receiving foot input from a foot. The virtual reality processor can be configured to control movement of the virtual surgical robotic system based on the hand input; and to use foot motion and input to control other functions of the robotic system, such as switching between robotic instruments, endoscopes, or arms of the virtual surgical robotic system.

[0039] In one embodiment, the system includes a camera or sensor 66 in communication with the processor, where the camera / sensor is positioned (e.g., positioned and oriented) to capture image data containing hand input and foot input from a user. The camera can transmit the image data to the processor (e.g., over a data bus, wire, network, etc.). The processor can be configured to identify hand input by identifying and tracking a hand or hand-held user interface device (UID) in the image data, and configured to identify foot input by identifying and tracking a knee, leg, and / or foot and its movements in the image data.

[0040] In one embodiment, the processor can recognize hand inputs through machine learning (e.g., one or more trained artificial neural networks). For example, the processor can use object recognition and / or computer vision techniques to recognize and track the console UID 68. In this case, the console UID can be a passive console UID that does not have internal tracking elements such as a camera or an IMU. Alternatively or additionally, the passive console UID can have one or more fiducial markers (e.g., affixed to a surface of the console UID) and / or markers to help recognize and track the console UID and its position and orientation. Thus, the UID can advantageously be a passive UID that does not require power or communication with the processor. This can shift the responsibility for hand inputs to the processor, simplifying the development and design of the system. This can also allow for dynamic programming of inputs, e.g., the processor can be programmed to receive new types of hand inputs without the need to redesign the hardware of the handheld UID.

[0041] In one embodiment, the system does not include additional sensors (e.g., cameras, foot pedals, buttons, etc.) to sense hand inputs and foot inputs. Thus, additional sensor setups are not needed to provide the virtual training environment, improving the mobility of the system. In one embodiment, the processor, camera, and display are integrated with a wearable device such as the head-mounted device 60, with the display placed over the eyes, providing an immersive virtual experience. In one aspect, the camera is housed in a gimbal located on the wearable device.

[0042] Interface with cross-section

[0043] In one aspect, as Figure 5 shown, an immersive virtual interface 70 for simulation and teleoperation of a surgical robotic system can include a display 71 having a first view segment 72 showing a virtual view of a surgical procedure feed 74 and a second view segment 76 showing a user’s control of the surgical procedure. The surgical procedure feed 74 can be a feed from a physical surgical procedure (e.g., from a camera in a physical surgical room) or a virtual surgical procedure with a virtual patient model and virtual surgical robotic system (e.g., virtual surgical robotic arms, platforms, patient models, etc.). The virtual interface can be integrated with a head-mounted device, with the display worn over the user’s eyes, providing an immersive visual and / or virtual environment. In one embodiment, one or more cameras of the head-mounted device can provide data for the second view segment 76.

[0044] In one embodiment, the first view segment and the second view segment are rendered onto the display by the processor. For example, the second view can be a camera feed or a virtual rendering. In one embodiment, the second view segment has a camera feed showing a physical pedal. Alternatively, the camera view can have an augmented / virtual pedal rendered on the image from the camera feed. As mentioned, the camera feed can be generated by a camera of the headset (e.g., positioned to capture the body of the wearer of the device). In one embodiment, the second view segment is a virtualized view, rather than a camera feed. The virtualized view can be generated based on the camera feed and / or other sensor data. The virtualized view can show a virtual representation of the user’s body, including the feet. A virtual control (e.g., a virtual foot pedal) can be generated. In one embodiment, the user controls of the second view segment include one or more of: a foot pedal, a handheld user interface device, the user’s feet, the user’s hands. In one embodiment, a virtual control (e.g., a virtual foot pedal or a virtual handheld UID) is generated in the user control view and rendered with a view of the user’s body. Advantageously, the user can see the virtual controls handled by the user’s limbs through the user control view to improve control.

[0045] Alternatively, in one embodiment, the first view segment is rendered onto the display by the processor and the second view segment includes an unobstructed opening of the display, the opening shaped and sized to allow a true working view of the user controls. The opening in the display is below the first view segment or at a bottom portion of the display (e.g., towards the user’s nose when the display is above the eyes on the headset) and provides a view of the user’s body and controls.

[0046] User interface flow

[0047] In Figure 6 In one embodiment, a user interface for simulation and teleoperation of a surgical robotic system (or a process performed by the user interface) 90 is shown, in accordance with one embodiment. The process can be initiated by user input, such as the user picking up a handheld UID, for example, at block 91. The process can sense the picking up or movement of the UID by techniques described in other parts of the disclosure. A display or screen (e.g., on a wearable device, a laptop, or a standalone monitor) can activate and open in response to the initiation.

[0048] At block 92, the process can prompt the user to enter login information (e.g., with input fields) and / or receive login information from the user. A user profile can be retrieved based on the login information. One or more user profiles 93 can be stored in a database and referenced as needed. Each user profile can have stored settings associated with the user profile. At block 94, the process can initiate a session, including synchronizing the user profile with the login information. In the case of a new user, a new user profile can be generated. The process can initiate a session based on the user login information.

[0049] At block 95, the process can provide selectable drill options. In one embodiment, the process can present one or more of the following options 106: a) simulate one or more surgical procedures in a virtual environment and / or b) teleoperate one or more surgical robotic procedures. Advantageously, the same login information and settings can be used for training and simulating surgical robotic procedures as well as real surgical robotic procedures. In one aspect, one or more drills can be assigned to a user by a user profile (e.g., by a second user or an automated scheduling system), the assigned drills to be performed by the user of the user profile. Each drill can specify a procedure type 108 (e.g., laparoscopic surgery of the abdomen). Thus, when the user logs in, the user can select from the assigned drills, train / simulate, and conduct real procedures. Updates and messages can also be presented based on the user profile.

[0050] At block 96, the process can provide reconfigurable settings (e.g., preferences) for a surgical procedure. Such settings can include selecting equipment items for a surgical robotic system (e.g., a surgical robotic type or a tool type 104). Additionally or alternatively, equipment can be automatically selected based on the drill (e.g., procedure type). In one aspect, the user can select one or more metrics 112 to be measured during simulation or teleoperation. In one embodiment, the settings can include tissue characteristics and interactions 110 (e.g., thickness, softness, or strength of tissue). In one aspect, the preferences / settings can include feedback types, such as visual, audible, and / or haptic feedback 114, such as visual indications showing the surgical workspace (e.g., highlighted on a display), warnings of collisions and possible collisions, depth of surgical tools in a patient or patient model, and tissue interactions. In one embodiment, the user can select a patient model 118 or model type (e.g., based on patient size or shape / size). The process advantageously allows the user to select between different training simulations and become proficient with different surgical robotic models.

[0051] At block 98, the process can execute a simulated run-through of a surgical robotic procedure in a virtual environment (e.g., a surgical robotic system and patient model in a virtual operating room) or teleoperation of a surgical robotic procedure through a live feed. The process can provide visual, audible, or haptic feedback to the user during the simulated procedure, which is provided for one or more of: a workspace (e.g., a surgical workspace in a patient or patient model), a fault, a collision, a warning, a depth (e.g., a depth of a surgical tool in a patient / model), a tissue property or interaction. The haptic or tactile feedback can be provided through one or more motors, vibrators, and / or actuators (e.g., they are housed in a headset or handheld UID). The visual feedback can be generated on a display. The audible feedback can be generated through one or more speakers (e.g., on a headset, a standalone speaker, or a computer laptop speaker). The process and system 90 can be executed by a processor through a display (e.g., a headset display, a laptop, a tablet, a desktop computer, and other equivalent technologies). In one embodiment, the user interface and process 90 are provided to simulate and operate Figures 1 to 5 the systems shown (e.g., executed by the processors, displays, and UIDs mentioned therein).

[0052] In one embodiment, the process can determine a score and / or other feedback of the simulated surgical procedure at block 100 (e.g., a run-through result, a measured metric, a case where a depth of a tool was exceeded, or a case where an incision can be too large). The score and other feedback can be provided to the user, e.g., through a user interface or display. In one embodiment, a user profile or data associated with the user profile can be updated based on the session (e.g., settings / preferences of the session can be saved in association with the user profile).

[0053] In one embodiment, Figures 1 to 5 Each of the systems shown can form a mobile kit. For example, a handheld UID can be packaged (e.g., bundled) with a headset or laptop. The kit can provide a mobile training system or teleoperation for a surgical robotic system, which can be easily shared between medical personnel without the need to set up external sensors that can be common in virtual reality systems.

[0054] In one embodiment, the method includes displaying a virtual surgical environment. For example, the virtual surgical environment can be displayed to a user display on a user console (e.g., a headset display, a laptop display, a tablet display, a desktop computer display, and other equivalent technologies) as Figure 1The virtual surgical environment can be displayed on a stadium view, floor plan, first-person view, or other view, or on any local or remote display. The display can be driven by a data transmission protocol (e.g., TCP / IP, Ethernet, UDP, etc.) between nodes on a network (e.g., computing devices). In one embodiment, the virtual surgical environment is displayed on a head-mounted display. The wearer of the head-mounted display can be tracked, allowing the wearer to move throughout the virtual surgical environment to gain a three-dimensional understanding of the position and orientation of various pieces of equipment, as well as unoccupied spaces and walkways within the virtual surgical environment. In one embodiment, the virtual surgical environment is interactive, allowing the user to adjust the orientation and / or position of objects within the virtual surgical environment (e.g., the angle or height of the virtual surgical robot arm, control tower, surgical robot platform, display angle, etc.).

[0055] In one implementation, the system's processors (e.g., VR processors, robot controllers, cameras, displays, and robotic arms) may include microprocessors and memory. Each processor may include a single processor or multiple processors, including a single processor core or multiple processor cores. Each processor may represent one or more general-purpose processors, such as microprocessors, central processing units (CPUs), etc. More specifically, each processor may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing combinations of instruction sets. Each processor may also be one or more special-purpose processors, such as application-specific integrated circuits (ASICs), cellular or baseband processors, field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, graphics processors, communication processors, cryptographic processors, coprocessors, embedded processors, or any other type of logic capable of processing instructions.

[0056] Modules, components, and other features such as the algorithm or method steps described herein may be implemented by a microprocessor, discrete hardware components, or integrated into the functionality of hardware components such as ASICs, FPGAs, DSPs, or similar devices. Furthermore, such features and components may be implemented as firmware or functional circuitry systems within a hardware device; however, such details are not materially relevant to the embodiments of this disclosure. It should also be understood that network computers, handheld computers, mobile computing devices, servers, and / or other data processing systems with fewer or potentially more components may also be used with embodiments of this disclosure.

[0057] Some portions of the preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self- consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities.

[0058] It should be borne in mind, however, that these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as those set forth in the claims below refer to the actions or processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0059] Embodiments of the disclosure also relate to an apparatus for performing the operations herein. Such computer program is stored in a non-transitory computer-readable medium. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, a machine-readable (e.g., computer-readable) medium includes a machine- (e.g., a computer-) readable storage medium (e.g., read only memory ("ROM"), random access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices).

[0060] The processes or methods depicted in the preceding figures can be performed by processing logic that includes hardware (e.g., circuitry, dedicated logic, etc.), software (e.g., embodied on a non-transitory computer-readable medium), or a combination of both. While the processes or methods are described above in terms of some sequential operations, it should be appreciated that some of the operations described can be performed in different order than described. Moreover, some operations can be performed in parallel rather than sequentially.

[0061] Embodiments of the disclosure have not been described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of embodiments of the disclosure as described herein.

[0062] In the foregoing specification, embodiments of the disclosure have been described with reference to specific exemplary embodiments thereof. It will be evident that various modifications can be made to the disclosure without departing from the broader spirit and scope of the disclosure as set forth in the following claims. The Specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense. For the purposes of explanation and illustration, specific nomenclature has been set forth to provide a thorough understanding of the application. However, it will be apparent to one skilled in the art that specific details need not be used to practice the present application. The foregoing specification has been provided in connection with the exemplary embodiments of the present application. They are presented not with the intention of limiting the application, but to enable the full and accurate understanding of the application. The description is thus to be regarded as illustrative rather than restrictive. The description is provided to enable any person skilled in the art to practice the application during the course of its best mode. Those skilled in the art will recognize that bounds of the application are by no means limited to the built-up specific embodiments disclosed and that changes and modifications could be practiced within the scope of the appended claims. The claims are intended to cover all such changes and modifications that complies with the principles of the application.

Claims

1. A mobile virtual reality system for simulating, training, or demonstrating a surgical robotic system, the mobile virtual display system comprising: a processor; a display for receiving and showing a virtual surgical robot from the processor, the virtual surgical robot comprising a plurality of virtual surgical instruments; one or more handheld user input devices (UIDs) that sense hand inputs from a hand; and one or more foot input devices that comprise an optical sensor for capturing visual data that senses a foot, wherein the processor is configured to: control movement of the virtual surgical robot based on the hand inputs, determine a foot input based on detecting movement or position of the foot in the visual data, and change which of the virtual surgical instruments is controlled by the one or more handheld UIDs based on the foot input.

2. The mobile virtual reality system of claim 1, wherein, The mobile virtual reality system does not have an external stationary camera.

3. The mobile virtual reality system of claim 1, wherein, The processor and the display are integrated with a laptop computer.

4. The mobile virtual reality system of claim 1, wherein, The display is a two-dimensional screen or a three-dimensional screen.

5. The mobile virtual reality system of claim 1, wherein, The display is a 3D wearable display that is worn on a user's head.

6. The mobile virtual reality system of claim 5, wherein, The processor is integrated with the display housed in a device.

7. The mobile virtual reality system of claim 1, wherein, The one or more handheld UIDs comprise an inward-outward tracking module, and the processor is configured to determine the hand inputs based on data indicative of movement, position, or orientation of the one or more handheld UIDs sensed by the inward-outward tracking module.

8. The mobile virtual reality system of claim 7, wherein, The one or more handheld UIDs comprise an inertial measurement unit (IMU), and the processor is configured to determine the hand inputs based on data indicative of movement, position, or orientation of the one or more handheld UIDs sensed by the IMU.

9. The mobile virtual reality system of claim 1, wherein, The one or more foot input devices comprise a foot pedal.

10. The mobile virtual reality system of claim 1, wherein, The one or more foot input devices comprise a tracking sensor.

11. The mobile virtual reality system of claim 10, wherein, The tracking sensor comprises an accelerometer, a gyroscope, or a combination thereof.

12. The mobile virtual reality system of claim 1, wherein, The processor is configured to detect and track the foot in the visual data by using machine learning.

13. The mobile virtual reality system of claim 12, wherein, The optical sensor comprises a camera housed in the one or more handheld UIDs.

14. The mobile virtual reality system of claim 13, wherein, The camera has a field of view of 170° or greater to capture the visual data that includes the foot input.

15. A method for simulating a surgical robotic system, comprising: presenting a virtual surgical robot comprising a plurality of virtual surgical instruments to a display; sensing hand inputs from a hand with one or more handheld user input devices (UIDs); and sensing foot inputs from a foot with one or more foot input devices, including capturing visual data of the foot with an optical sensor and detecting movement or position of the foot in the visual data to determine the foot inputs, controlling movement of the virtual surgical robot based on the hand inputs, and changing which of the virtual surgical instruments is controlled by the one or more handheld UIDs based on the foot inputs.

16. The method of claim 15, wherein, The surgical robotic system does not have an external stationary camera.

17. The method of claim 15, wherein, The display is integrated with a laptop computer.

18. The method of claim 15, wherein, The display is a two-dimensional screen or a three-dimensional screen.

19. The method of claim 15, wherein, The display is a 3D wearable display worn on a user's head.

20. The method of claim 19, wherein, The presenting the virtual surgical robot, the sensing the hand input, the sensing the foot input, the controlling the movement, and the changing are performed by at least one processor integrated with the display housed in a device.

Images