Surgical virtual reality user interface

Abstract

A surgical virtual reality user interface generation system comprising: a sensing and tracking unit that senses and tracks a position of a user and generates position data based on movement of the user; and a computational unit that receives the position data and processes the position data to generate a control signal.SOLUTION: The system further comprises a surgical robot system for receiving the control signals, the surgical robot system having a camera assembly for generating image data, and a virtual reality computing unit for generating a virtual reality world. The virtual reality computation unit includes a virtual reality rendering unit that generates an output rendering signal for rendering the image data for display, and a virtual reality object generation unit that generates a virtual reality information object and positions the information object in the virtual reality world. A display unit is provided for displaying the virtual reality world and the information objects to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 62 / 933,873, entitled "Surgical Virtual Reality User Interface," filed November 11, 2019, and U.S. Provisional Patent Application No. 62 / 930,922, entitled "Hand Controller For Surgical Robotic System," filed November 5, 2019, the contents of which are incorporated herein by reference. [Background technology]

[0002] Background of the Invention The present invention relates generally to minimally invasive surgery, surgical robotic devices, and related user interfaces, and more particularly to user interfaces for use in virtual reality minimally invasive surgical systems.

[0003] During a surgical procedure, a surgeon requires access to a myriad of information, including patient data, the patient's vital signs, surgical procedures and plans, and equipment status, among other information and data. In addition, it is also useful for the surgeon to have access to various views of the surgical site and views of the operating room. In traditional minimally invasive surgery and existing robotic surgical systems, the surgeon can easily access this information by viewing various screens located around the operating room and by having an assistant relay the required or needed information.

[0004] In virtual reality-assisted surgery, surgeons use sophisticated virtual reality software and hardware to feel as if they are miniaturized and inside the surgical site on a patient's body. As used herein, the term "virtual reality surgery" is intended to refer to a surgical procedure in which a virtual representation of reality is presented to the surgeon using one or more devices and associated systems. A robotic system can use one or more cameras to provide video data of the surgical site and other necessary surrounding locations, and the video data is combined with or overlaid with other visual elements to provide a virtual representation of various sites and surrounding environments presented to the surgeon to create or form a virtual world. In known systems, a surgical robot can be placed within the patient and configured to replicate selected movements, such as those associated with the surgeon's head, arms, and hands. Combined with three-dimensional visualization provided by a virtual reality display device, such as a head-mounted display (HMD), the surgeon can view the surgical site and interact with the surgical robot as if the robotic arms were the surgeon's arms and hands. Successful outcomes may be achieved predicated on maintaining an immersive and natural-looking virtual reality user interface during virtual reality surgery, allowing the surgeon to focus on the surgical procedure. However, when immersed in the user interface, the surgeon may feel disconnected from the external environment and unable to access necessary information and views while performing the surgery. Summary of the Invention

[0005] To maintain an immersive and natural-looking virtual reality user interface and to allow the surgeon to access any desired information, the system of the present invention can use a user interface that allows the surgeon to interact with the surgical robot and access desired information and data without removing themselves from interaction with the virtual environment.

[0006] The present invention relates to a surgical virtual reality user interface generation system that uses a VR object generation unit to generate virtual objects for placement within a virtual reality landscape or world. The virtual objects provide information related to selected aspects of the system for presentation to a system user immersed in the virtual world. For example, the objects may provide image data, including image or video feed data from a robotic camera assembly, patient-specific data such as MRI or X-ray data, and environmental data, such as data related to the patient and user's environment. The objects may be manipulated by the user and may be switched between various states and modes. The objects may also be placed or reside in a docking station for easy user access. The virtual world also displays a master list of objects to the user, enumerating or describing all available objects.

[0007] The present invention relates to a surgical virtual reality user interface generation system comprising: a sensing and tracking unit configured to sense and track a position of a portion of a user in a space and generate at least position data based on a movement of the user; a computing unit configured to receive the position data, the computing unit having a processor configured to process the position data; and a control unit configured to generate control signals in response to the processed position data. The system further comprises: a surgical robot system coupled to the computing unit and configured to receive the control signals, the surgical robot system having a camera assembly with a pair of axially spaced cameras configured to generate image data; a virtual reality computing unit configured to generate a virtual reality world, the virtual reality rendering unit configured to receive at least the image data from the camera assembly and generate output rendering signals for rendering the image data for display; and a virtual reality generation unit configured to generate one or more virtual reality information objects and position the information objects within the virtual reality world. The system further comprises a display unit configured to display the virtual reality world and the information objects to the user.

[0008] The surgical robot system includes: one or more robotic arms; a motor unit coupled to the camera assembly and the robot arm for selectively moving the camera assembly and the robot arm; Further includes:

[0009] 3. The system described in 1., wherein the sensing and tracking unit includes a hand controller or a head-mounted display.

[0010] 4. The system of claim 3, wherein the hand controller includes an elongated body having a movable lever switch coupled thereto.

[0011] 5. The movable lever switch is rotatable around the body, and the body is a channel formed in the body, the lever switch being pivotally mounted within the channel; a finger loop attached to the body; 4. The system according to claim 4, comprising:

[0012] 6. A rest protrusion formed on the body; a plurality of actuatable buttons formed on the body, the plurality of actuatable buttons comprising: first and second elbow buttons that, when activated, allow the user to manipulate an elbow joint region of the robotic arm; an actuatable pause button that, when activated, allows the user to disengage the robotic arm's movement from the user's arm's movement; a lock button that allows the user to lock the lever switch to the main body; a plurality of actuatable buttons, including: 5. The system described in 5., further comprising:

[0013] 7. The system described in 1., wherein the surgical robot system generates camera data representing the position and orientation of the camera of the camera assembly, and the image rendering unit uses the camera data to render the image data.

[0014] 8. The system described in 7., wherein the display unit generates display data indicating the position and orientation of the user's head, and the VR rendering unit uses the display data to render the image data.

[0015] 9. The system described in 1., wherein the virtual reality object generation unit is configured to receive information data from a data source and then embed the information data within the information object.

[0016] 10. The system of claim 9, wherein the information data includes data from the surgical robot system.

[0017] 11. The system described in 10., wherein the information data further includes data from one or more external data sources, and the information data from the external data sources can include one or more video data from an external camera and information data from one or more external medical devices.

[0018] 12. The system described in 9., wherein the information object is configured to be displayed within the virtual reality world in a free mode in which the information object is placed at a selected fixed position, a docked mode in which the information object is placed in a docking station, or an attached mode in which the information object is positioned to follow the user within the virtual reality world.

[0019] 13. The system described in 12., wherein the virtual reality object generation unit is configured to generate the docking station within the virtual reality world, and the docking station is configured to include multiple of the information objects.

[0020] 14. The system of claim 13, wherein a plurality of virtual slots are formed within the docking station.

[0021] 15. The system described in 12., wherein the virtual reality computing unit is configured to generate the docking station within the virtual reality world, the docking station is configured to include a plurality of slots each configured to accommodate one or more of the plurality of information objects, and when the information object is removed from the slot by the user, the information object automatically switches to the free mode.

[0022] 16. The system described in 1., wherein the virtual reality object generation unit generates an object list including a list of the information objects, and each of the plurality of information objects in the object list includes a title of the object.

[0023] 17. The system of claim 1, wherein the information object includes a title bar and a content area for displaying data, the title bar including a title for the information object and a plurality of action buttons.

[0024] 18. The system described in 17., wherein the plurality of action buttons include two or more of an auto-visualization button that allows the user to determine whether the object is visible in the virtual world, an object mode button that allows the user to switch object mode, a docking station button that allows the user to move the object into the docking station, and a close button that hides the object.

[0025] 19. The system described in 1., wherein the virtual reality computing unit further includes a spherical image generation unit for generating a spherical image from the image data.

[0026] 20. The system described in 19., wherein the control unit is configured to generate an autoscan signal that is received by the camera assembly, and in response, the camera assembly operates in an autoscan mode in which the camera in the camera assembly autonomously rotates through an entire operating range to capture the image data.

[0027] 21. The system described in 1., wherein the virtual reality computing unit further includes a scene graph generation unit that generates a scene graph, the scene graph including a plurality of nodes arranged in a tree graph structure.

[0028] 22. A method for generating one or more information objects, comprising: sensing and tracking a position of a portion of a user within a space and generating position data based on movement of the user; providing an arithmetic unit for receiving and processing the position data and generating control signals in response to the processed position data; providing a surgical robotic system that receives the control signals, the surgical robotic system having a camera assembly that includes a pair of axially spaced cameras that generate image data; generating a virtual reality world using a virtual reality computing unit, the virtual reality computing unit comprising: a virtual reality rendering unit that receives at least the image data from the camera assembly and generates an output rendering signal for rendering the image data for display; a virtual reality generation unit that generates one or more virtual reality information objects and places the information objects within the virtual reality world; and displaying the virtual reality world and the information object to the user; A method comprising:

[0029] 23. The method described in 22., wherein the surgical robot system further includes one or more robotic arms and a motor unit coupled to the camera assembly and the robotic arm, for selectively moving the camera assembly and the robotic arm in response to the control signal.

[0030] 24. The method of claim 22, wherein the sensing and tracking unit comprises a hand controller or a head-mounted display.

[0031] 25. The method of claim 24, wherein the hand controller includes an elongated body having a movable lever switch coupled thereto, the movable lever switch being rotatable about the body, the body having a channel formed therein, the lever switch being rotatably mounted within the channel.

[0032] 26. The method further comprising providing a plurality of actuatable buttons formed on the body, the plurality of actuatable buttons comprising: first and second elbow buttons that, when activated, allow the user to manipulate an elbow joint region of the robotic arm; an actuatable pause button that, when activated, allows the user to disengage the robotic arm's movement from the user's arm's movement; a lock button that allows the user to lock the lever switch to the main body; 25. The method according to claim 25, comprising:

[0033] 27. The method of claim 22, wherein the virtual reality object generation unit is configured to receive information data from a data source and then embed the information data within the information object.

[0034] 28. The method of claim 27, wherein the information data includes data from the surgical robotic system.

[0035] 29. The method of claim 28, wherein the information data further includes data from one or more external data sources, and the information data from the external data sources can include one or more video data from an external camera and information data from one or more external medical devices.

[0036] 30. The method described in 27, wherein the information object is configured to be displayed in the virtual reality world in a free mode in which the information object is placed at a selected fixed position, a docked mode in which the information object is placed in a docking station, or an attached mode in which the information object is positioned to follow the user in the virtual reality world.

[0037] 31. The method described in 30., wherein the virtual reality object generation unit is configured to generate the docking station within the virtual reality world, and the docking station is configured to include multiple of the information objects.

[0038] 32. The method of claim 31, wherein a plurality of virtual slots are formed in the docking station.

[0039] 33. The method described in 30., wherein the virtual reality computing unit is configured to generate the docking station within the virtual reality world, the docking station is configured to include a plurality of slots each configured to accommodate one or more of the plurality of information objects, and when the information object is removed from the slot by the user, the information object automatically switches to the free mode.

[0040] 34. The method of claim 22, wherein the virtual reality object generation unit generates an object list including a list of the information objects, and each of the plurality of information objects in the object list includes a title of the object.

[0041] 35. The method of claim 22, wherein the information object includes a title bar and a content area for displaying data, the title bar including a title for the information object and a plurality of action buttons.

[0042] 36. The method of claim 35, wherein the plurality of action buttons include two or more of an auto-visualization button that allows the user to determine whether the object is visible in the virtual world, an object mode button that allows the user to switch object mode, a docking station button that allows the user to move the object into the docking station, and a close button that hides the object.

[0043] 37. The method according to claim 22, wherein the virtual reality computing unit includes a spherical image generation unit for generating a spherical image from the image data.

[0044] 38. The method of claim 37, wherein the control unit is configured to generate an autoscan signal that is received by the camera assembly, and in response, the camera assembly operates in an autoscan mode in which the camera in the camera assembly autonomously rotates through an entire operating range to capture the image data. [Brief explanation of the drawings]

[0045] These and other features and advantages of the present invention will be more fully understood from the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals refer to like elements throughout the different views, and in which the drawings illustrate the principles of the invention and show relative dimensions, although not to scale. [Figure 1] FIG. 1 is a simplified block diagram of a surgical virtual reality user interface generation system in accordance with the teachings of the present invention. [Figure 2] FIG. 2 is a diagram of a hand controller used by a surgeon to control the movement of a robotic arm in accordance with the teachings of the present invention. [Figure 3] 3A and 3B are perspective views of a hand controller in accordance with the teachings of the present invention. [Figure 4] FIG. 4 is a partial perspective view of the hand controller of the present invention. [Figure 5] FIG. 5 is a schematic diagram of a virtual reality world as viewed by a user through a display unit, showing information objects and associated docking stations, in accordance with the teachings of the present invention. [Figure 6] FIG. 6 is a diagram of a master list of virtual reality based information objects in accordance with the teachings of the present invention. [Figure 7]FIG. 7 is a schematic diagram of an information object according to the teachings of the present invention. [Figure 8] FIG. 8 is a diagram of a virtual reality world generated by the system of the present invention, displaying an object list and associated information objects therein. [Figure 9] 9A and 9B are schematic diagrams illustrating the principles of the coupling or clutch-in and discoupling or clutch-out modes of operation in accordance with the teachings of the present invention. [Figure 10A] FIG. 10A is an illustration of a virtual reality world generated by the system of the present invention, showing graphical elements that can be manipulated by a user as the user moves their arm in a manner selected to coordinate with the robotic arm. [Figure 10B] FIG. 10B is an illustration of the virtual reality world generated by the system of the present invention, showing graphical elements that can be manipulated by the user as the user moves their arm in a manner selected to coordinate with the robotic arm. [Figure 11] FIG. 11 is a schematic diagram of a virtual reality computing unit illustrating a spherical image generator according to the teachings of the present invention. [Figure 12A] FIG. 12A is an illustration of a virtual world utilizing spherical images in accordance with the teachings of the present invention. [Figure 12B] FIG. 12B is another illustration of a spherical image in accordance with the teachings of the present invention. [Figure 13] FIG. 13 is a schematic diagram of a scene graph in accordance with the teachings of the present invention. [Figure 14] FIG. 14 is a schematic block diagram of another embodiment of a surgical virtual reality user interface generation system in accordance with the teachings of the present invention. [Figure 15] FIG. 15 is a schematic block diagram of yet another embodiment of a surgical virtual reality user interface generation system in accordance with the teachings of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0046] Detailed Description of the Invention In the following detailed description, numerous details are set forth about the systems and methods of the present invention, as well as the environments in which the systems and methods of the present invention can operate, in order to provide a thorough understanding of the subject matter of the present disclosure. However, it will be apparent to those skilled in the art that the subject matter of the present disclosure can be practiced without such specific details, and that certain features well known in the art have not been described in detail to avoid obscuring the subject matter of the present disclosure. In addition, it will be understood that any examples described below are merely illustrative and should not be construed as limiting, and that other systems, devices, and / or methods may be used to implement the teachings of the present disclosure and are considered to be within the scope of the present invention.

[0047] Although the systems and methods of the present invention may be designed for use with one or more surgical robotic systems used as part of a virtual reality surgery, the systems of the present invention may also be used in conjunction with any type of surgical system, such as robotic surgical systems, straight stick type surgical systems, laparoscopic systems, etc. Additionally, the systems of the present invention may be used in other non-surgical systems that require a user to access a large amount of information while controlling a device or apparatus.

[0048] The systems and methods of the present disclosure may be used in combination with, for example, robotic surgical devices and systems disclosed in U.S. Patent No. 10,285,765 and PCT Patent Application No. PCT / US20 / 39203, and / or camera systems disclosed in U.S. Patent Application Publication No. 2019 / 0076199, the teachings of which are all incorporated herein by reference. In some embodiments, the surgical virtual reality user interface generation system may be embodied and utilized with existing and future surgical robotic systems and devices using known visualization technologies, including, for example, virtual reality and / or augmented reality visualization technologies. The present invention may employ one or more surgical virtual reality user interface (SVRUI) generation systems designed to enable users to place informational objects or widgets within a virtual reality (VR) environment that can be used to embody and control one or more surgical robotic devices. As used herein, the terms “informational object” or “widget” are intended to include any type of data, such as image data, informational data, etc., that is related to or associated with one or more real-world objects. An object or widget can be manipulated by a user and can be displayed in one or more states. The object may be a virtual reality (VR) object. The object may also be multidimensional, e.g., displaying information in three-dimensional space. Furthermore, the object may include information that does not form part of the image data generated by the camera assembly. The data may include, for example, a virtual representation of equipment, graphic data, patient data, 3D scans, or anatomical models based on a three-dimensional (3D) model. User interface elements, floating virtual menus, or screens may be active or passive elements within which the user can interact, or the object may display information from another data source.As described further herein, in some embodiments, the placement of the information object may be determined, for example, by a user using one or more selected controllers, such as a handheld controller, a head-mounted controller, and an eye-tracking controller, or by some other type of user input device, and the information object may be automatically placed by the system in a selected manner or at any selected location within the virtual environment, such as, for example, in a docking station. The docking station may have a fixed or movable position relative to real-world coordinates on a selected work or surgical site and / or mounted directly on the user's head to maintain a constant position and orientation relative to a display device or unit, such as, for example, a display, a head-mounted display (HMD), or a screen, such as a 3D screen. [Patent Document 1] U.S. Patent No. 10,285,765 [Patent Document 2] International Patent Application No. PCT / US2020 / 39203

[0049] In some embodiments, the system of the present invention is part of a larger surgical system, allowing a user, such as a surgeon, to interact with the VR world and the surgical robotic device while performing a virtual reality procedure using the surgical robotic device.

[0050] FIG. 1 is a schematic block diagram of a surgical virtual reality user interface generation system 10 according to the teachings of the present invention. The system 10 includes a display device unit 12, a virtual reality (VR) computing unit 14, a sensing and tracking unit 16, a computing unit 18, and a surgical robotic system 20. The display unit 12 may be any selected type of display for displaying information, images, or videos generated by the VR computing unit 14, the computing unit 18, and / or the surgical robotic system 20. The display unit 12 may include, for example, a head-mounted display (HMD), a screen or display, a three-dimensional (3D) screen, etc. The sensing and tracking unit 16 may include one or more sensors or detectors coupled to a user of the system, such as a nurse or surgeon. The sensors may be coupled to the user's arms, and if a head-mounted display is not used, additional sensors may be coupled to the user's head and / or neck regions. If the user uses a head-mounted display, eye, head, and / or neck sensors and tracking technology may be incorporated into or used in conjunction with the device. The sensors coupled to the surgeon's arm may be preferably coupled to selected regions of the arm, such as the shoulder region, elbow region, wrist or hand region, and, if desired, the fingers. The sensors generate position data indicative of the position of selected parts of the user. The sensing and tracking unit 16 utilized to control the camera assembly 44 may be separate from the sensing and tracking unit used to control the robotic arm. The position data 34 generated by the sensors may be transmitted to the computing unit 18 for processing by the processor 22. The computing unit 20 may determine or calculate the position and / or orientation of each part of the surgeon's arm from the position data and transmit this data to the surgical robotic system 20. In another embodiment, the sensing and tracking unit 16 may use sensors coupled to the surgeon's torso or any other body part.Furthermore, in addition to the sensors, the sensing and tracking unit 16 may use an Inertial Momentum Unit (IMU) having, for example, an accelerometer, a gyroscope, a magnetometer, and a motion processor. The addition of a magnetometer is standard practice in the art because the magnetic orientation allows for mitigation of sensor drift about a vertical axis. Alternative embodiments may further include sensors located in surgical equipment such as gloves, a surgical gown, or a surgical robes. The sensors may be reusable or disposable. Additionally, the sensors may be located external to the user, for example, in a fixed location in a room such as an operating room.

[0051] In an embodiment in which the display is an HMD, the display unit 12 may be a virtual reality head-mounted display, such as, for example, Oculus Rift, Varjo VR-1, or HTC Vive Pro Eye. The HMD may provide the user with a head-mounted display, lenses that allow the image on the display to be focused, and a sensor and / or tracking system for tracking the position and orientation of the display. The position and orientation sensor system may include, for example, accelerometers, gyroscopes, magnetometers, motion processors, infrared tracking, eye tracking, computer vision, emitting and sensing alternating magnetic fields, and any other method for tracking position and / or orientation, or any combination thereof. As is known, the HMD may provide image data to the surgeon's left and right eyes from a camera assembly 44. To maintain the surgeon's virtual reality experience, the sensor system may track the position and orientation of the surgeon's head and then relay this data to the computing unit 18. The computing unit 18 may also adjust the pan and tilt of the robot's camera assembly 44 to follow the user's head movements.

[0052] If a sensor is associated with the display unit 12, sensor or position data 32 generated by the sensor may be transmitted to the computing unit 18. For simplicity, the sensor data 32 is illustrated as being transmitted to the sensing and tracking unit 16, but one skilled in the art will readily understand that the tracking and position data 32 may be transmitted directly to the computing unit 18. Alternatively, the tracking and position data 32 may be transmitted to the VR computing unit 14 and then to the computing unit 18 for further processing. Similarly, tracking and position data 34 generated by other sensors in the system, such as from the sensing and tracking unit 16, which may be associated with the user's arms and hands, may be transmitted to the computing unit 18. The tracking and position data 32, 34 may be processed by the processor 22 and stored, for example, in the memory unit 24. The tracking and position data 32, 34 may be used by the control unit 26, which may responsively generate control signals for controlling one or more portions of the surgical robotic system 20. The surgical robotic system 20 can comprise a surgical system including a user workstation, a robot support system (RSS), a motor unit 40, and an implantable surgical robot including one or more robotic arms 42 and one or more camera assemblies 44. The implantable robotic arms and camera assemblies may form part of a single support axis robotic system such as that disclosed and described in U.S. Pat. No. 10,285,765, or may form part of a split-arm configuration robotic system such as that disclosed and described in PCT Patent Application No. PCT / US20 / 39203. [Patent Document 3] U.S. Patent No. 10,285,765 [Patent Document 4] International Patent Application No. PCT / US2020 / 39203

[0053] The control signals generated by the control unit 26 may be received by a motor unit 40 of the surgical robotic system 20. The motor unit 40 may include a series of servo motors configured to separately drive the robotic arm 42 and the camera assembly 44. The robotic arm 42 may be controlled to follow the scaled movements of the surgeon's arm as sensed by associated sensors. The robotic arm 42 may have portions or regions that can be associated with the movements of a user's shoulder, elbow, wrist, and fingers. For example, a robotic elbow may follow the position and orientation of a human elbow, and a robotic wrist may follow the position and orientation of a human wrist. The robotic arm 42 may have an associated end region that may terminate in an end effector that follows the movement of one or more of the user's fingers, such as the index finger when the user places the index finger and thumb together. While the robotic arm follows the movement of the user's arm, the robotic shoulder is in a fixed position. In one embodiment, the position and orientation of the user's torso is subtracted from the position and orientation of the user's arm. This subtraction allows the user to move their torso without moving the robotic arm.

[0054] The user may also use a hand controller, which may be associated with one or more sensors or detectors, as part of the sensing and tracking unit 16. Examples of types of hand controllers suitable for use with the surgical robotic system 20 are shown in FIGS. 2-4. Those skilled in the art will readily appreciate that other known types of hand controllers may also be used. The illustrated hand controller 50 has a relatively slim profile and an elongated body 52 sized and shaped to fit comfortably within the hand of a user, such as a surgeon. As shown in FIG. 2, for example, the hand controller 50 is typically held between the surgeon's thumb and index finger. The hand controller is adapted to generate signals that are processed by the computing unit 18. In response to the processed signals, the controller 26 generates and transmits control signals to the motor unit 40, which then controls the robotic arm 42, e.g., the end effector region of the robotic arm 42. In this manner, the surgeon controls the grasping action of the end effector using any selected drive mechanism, such as a switch or lever, using any selected finger of the surgeon. In this manner, for example, grasping (e.g., closing) and releasing (e.g., opening) the end effector, moving the end effector axially outward or inward, and rotating the end effector are all possible using the hand controller 50 of the present invention, which can serve to manipulate the end effector of a robotic system.

[0055] The hand controller 50 further includes a movable lever switch 54 that is movable between a released position ( FIG. 3A ), in which the lever switch 54 is disengaged and the end effector of the robotic arm is open, and an engaged position ( FIG. 2 ), in which the lever switch 54 is movable toward the body 52 of the hand controller 50, which then drives the end effector to close or grasp an object. In particular, when a user or surgeon squeezes the lever switch 54 relative to the body 52 of the hand controller, the robotic arm can close the end effector or perform other robotic operations.

[0056] The lever switch 54 may be coupled to a slip ring (not shown) or other sliding mechanism that allows the lever switch 54 to rotate about the primary axis of the body 52. ​​The lever switch 54 is coupled to a rotatable connector 56 that allows the lever switch to move within a channel 58 formed in the body. Opposite edges of the channel 58 define the range of rotational motion of the lever switch 54. The channel 58 allows the surgeon to grasp the end effector and then rotate or pivot the lever switch 54 about the body 52 of the controller and within the channel 58 while holding the controller in a stationary position, thereby rotating the end effector beyond the limits of human wrist movement.

[0057] The body 52 of the hand controller 50 may further include a finger loop 60 that allows a surgeon to insert a finger therein (FIG. 2) to provide a selected degree or amount of stability to the hand controller 50. If desired, the finger loop 60 may be associated with an adjustment mechanism to allow the opening of the finger loop to be adjusted. This allows the surgeon to customize the size of the opening of the finger loop 60 to better fit the inserted finger.

[0058] The hand controller 50 may further include a series of actuators or buttons that allow the surgeon to manipulate or control the movement of the robotic arm 42. For example, the body 52 of the hand controller 50 may further include one or more elbow buttons 64A, 64B that allow the surgeon to manipulate an elbow joint range or region of the robotic arm. Thus, the elbow buttons may allow the surgeon to bend the elbow region of the robotic arm in selected opposing directions by selectively activating the buttons. Furthermore, the illustrated hand controller 50 may further include an optional rest position protrusion or detent 66 sized and positioned to allow the surgeon to rest one or more fingers on the protrusion or detent to avoid accidentally contacting one of the other actuatable buttons during use. In another embodiment, the protrusion 66 may be configured as an actuatable button that, when activated, allows the surgeon to decouple the movement of the robotic arm from the movement of the surgeon's arm. In this manner, the surgeon can activate the button by pressing it to decouple the movement of the hand controller from the robotic arm. The hand controller may further include an optional lock button 68 that allows the surgeon to lock the lever switch 54 to prevent accidental movement of the end effector of the robotic arm.

[0059] Additionally, the computing unit 18 can translate the surgeon's arm movements into robotic arm 42 movements with straightforward scaling. However, other embodiments may include adjustable scaling of the movements. That is, scaling may be adjusted up or down. For example, a 10-degree movement of the surgeon's elbow may result in a corresponding 5-degree movement of the device's elbow. This scaling allows for increased dexterity at the expense of a decreased natural feel of the device.

[0060] The robotic camera assembly 44 is configured to provide the surgeon with image data 48, such as a live video feed of the procedure or surgical site, and to allow the surgeon to operate and control the cameras that make up the camera assembly 44. The camera assembly 44 preferably includes a pair of cameras whose optical axes are axially separated by a selected distance, known as the inter-camera distance, to provide a stereoscopic view of the surgical site. The surgeon can control the camera movement by movement of a head-mounted display, through sensors coupled to the surgeon's head, or by using hand controllers or sensors that track the user's head or arm movements, allowing the surgeon to obtain a desired view of the surgical site in an intuitive and natural manner. The cameras are movable in multiple directions, including, for example, known yaw and pitch directions. In some embodiments, the cameras are also movable in roll. Components of the stereoscopic camera may be configured to provide a user experience that feels natural and comfortable. In some embodiments, the inter-axial distance between the cameras may be changed to adjust the user's perceived depth of the surgical site.

[0061] The camera assembly 44 is actuated by the surgeon's head movements. For example, if during surgery the surgeon wishes to view an object located above the current field of view, the surgeon looks upward, causing the stereoscopic camera to rotate about a pitch axis upward from the user's line of sight. Images or video data 48 generated by the camera assembly 44 may be displayed on the display unit 12. If the display unit 12 is a head-mounted display, the display may include an embedded tracking and sensor system that captures raw orientation data in the yaw, pitch, and roll directions of the HMD, as well as position data in Cartesian space (x, y, z) of the HMD. However, a separate tracking system may be used to provide supplemental position and orientation tracking data for the display instead of, or in addition to, the HMD's embedded tracking system. An example of a camera assembly suitable for use in the present invention includes the camera assembly disclosed in commonly assigned U.S. Pat. No. 10,285,765 and U.S. Patent Application Publication No. 2019 / 0076199, the contents of which are incorporated herein by reference.

[0062] The image data generated by the camera assembly 44 may be transmitted to the virtual reality (VR) computing unit 14 and processed by the VR or image rendering unit 30. The image data 48 may include still photographic data or image data as well as video data. The VR rendering unit 30 may include suitable hardware and software for processing the image data and then rendering the image data for display by the display unit 12 as known in the art. Additionally, the VR rendering unit 30 may combine the image data received from the camera assembly 44 with information related to the position and orientation of the cameras in the camera assembly and information related to the position and orientation of the surgeon's head. The VR rendering unit 30 may use this information to generate an output video or image rendering signal and send this signal to the display unit 12. That is, the VR rendering unit 30 renders readings of the position and orientation of the hand controllers and the surgeon's head position for display on a display unit, such as an HMD worn by the surgeon.

[0063] The virtual reality (VR) object generation unit 28 of the VR computing unit 14 may be used to generate information objects 82 for placement within the virtual reality world 100 that is displayed to the surgeon via the display unit 12. The information objects 82 may be used to input information data into the VR world, allowing the surgeon to easily access desired information while performing surgery. The information objects and the associated information data contained therein are transmitted to the VR computing unit 14 and then rendered on the display unit 12. In some embodiments of the present invention, the information objects 82 may be contained entirely within the VR computing unit 14. In other embodiments of the present invention, the information objects rely on information from other data sources within the system 10, such as from the surgical robot system 20, or data 36 from a third party or external source, to render or animate the display of a robotic arm or other surgical device. In yet other embodiments, the data source is an external data source that communicates with the VR computing unit 14 and may introduce into the VR computing unit 14 a video stream from an external camera located in the external environment, such as from a camera located in an operating room or at a nurse's station. In another embodiment, the data source 36 may include data from medical devices or systems such as an MRI machine, one or more patient monitors (e.g., blood pressure levels, heart rate), etc. Data from various data sources may be packaged into information objects, which may be inserted or rendered into the surgeon's virtual reality world.

[0064] The surgical virtual reality user interface generation system 10 of the present invention may be configured to allow a user to interact with a VR world 100 generated by the VR computing unit 14 while performing a virtual reality surgical procedure. According to one embodiment, the VR world 100 may be projected onto the screen of a head-mounted display worn by the user, or may be projected onto an interactive display or screen, such as a monitor in an operating room or the screen of a user workstation. In the VR world 100, an information object 82 is displayed on an HMD or a 3D screen. The information object 82 is preferably a virtual reality object that can be manipulated by the user. According to one embodiment, the information object can display three-dimensional medical information, such as MRI information or CT images, image data, such as live video feed data or still images, and / or virtual objects. The virtual object may be a computer-generated three-dimensional representation of information data, objects, or the like, placed within the computer-generated virtual reality world 100, and the user can interact with the virtual object and other objects within the VR world 100. If desired, the virtual objects may be visible to the user through the display unit and may include virtual screens, virtual menus, virtual clocks, reference anatomical models, overlays of 3D trajectories to follow, markers that can be placed by the user, measuring devices, etc., or any tool or object that can be used in the real world and that may have a virtual analog that can be useful in the virtual world while operating on a patient. Essentially, the system 10 of the present invention can provide an augmented telepresence experience in a virtual surgical environment, where the telepresence can include a live camera feed and the augmentation is any virtual overlay or interaction that can be provided along with the camera feed.

[0065] Informational objects may be displayed in different states depending on the user's desired configuration. According to one embodiment of the present invention, informational objects can be displayed or configured in a free mode or state, a docked mode or state, and an attached mode or state. In the free state mode or state, the informational object remains fixed at a specific location within the VR space. The object can be placed at a specific location within the VR world 100 using the hand controller 50 or by the surgeon's hand or arm movements. The free state mode allows the surgeon to place a specific object at a specific location and a specific distance relative to the robotic arm. Thus, from the user's perspective, the informational or virtual object remains in a constant position within the surgical field even as the camera FOV is moved relative to the surgical field. This relative constancy of the object creates the feeling that the object is part of the surgical field. For example, if the surgeon wants to refer to a preoperative CT scan at some point in the surgery, the surgeon can temporarily place a CT-specific informational object near the specific portion of the surgical field that the CT scan highlights or refers to. The state or mode of the informational object may be changed by selecting one or more soft buttons.

[0066] According to the teachings of the present invention, information objects may be placed or placed in a docking mode or state. FIG. 5 illustrates display unit 12 displaying to a user a rendered view of a VR world 100 generated by VR computing unit 14. VR world 100 displays video data from camera assembly 44 and an associated field of view (FOV) 70. In some embodiments, FOV 70 occupies the entire screen of display unit 12, while in other embodiments, it occupies only a portion of the screen of display unit 12. In docking mode, information objects or widgets 82 may be automatically placed or placed within a docking station or container 80. Docking station 80 may be of any selected shape or size and is preferably manipulated by the user. Alternatively, the location of docking station 80 may be predetermined. According to one embodiment, docking station 80 may be arch- or halo-shaped, allowing a user to select one or more information objects from a list of information objects to be placed within docking station 80. Alternatively, system 10 may place one or more pre-selected informational objects 82 in docking station 80 via VR object generation unit 28 or VR computing unit 14. The user can place the informational objects 82 that the user wants to remain visible in selected locations within VR world 100 that do not obstruct or block the view of the surgical site, without the user having to manually place each individual informational object 82 within VR world 100. Docking station 80 may be configured with multiple pre-defined locations or slots 84 for placing selected informational objects 82. Informational objects 82 may be dragged and repositioned from slot to slot within docking station 80 by the user using a hand controller or any other selection method or technique.In one embodiment, the user can see a representation of a hand controller in the VR world 100, move the hand controller representation over the information object 82, press and hold a button on the hand controller, and then move the hand controller to drag the information object. Once the desired location is reached, the user may release the button and the information object is dropped into place in the VR world 100. The same action may be achieved with other similar interface elements known in the art, including but not limited to gesture tracking, gaze and blink tracking, keyboard, mouse, voice activation, etc.

[0067] As the user drags the information object 82 into the display 12, a preview of the slot area that the information object 82 can occupy is shown. In one embodiment, a preview of the slot 84 is shown in front of the docking station 80, and the preview can have any selected shape or size, such as, for example, a sphere. As shown in FIG. 5 , when the information object 82 is dragged out of the slot 84 of the docking station 80, the information object 82 automatically switches to free mode. If the information object is in free mode and the user wants to drag the information object 82 back into a selected slot 84 of the docking station 80, the docking station 80 may optionally be highlighted and a preview of one of the slots 84 may be displayed; when the information object 82 is dropped into the selected slot 84, the information object switches from free mode to docked mode.

[0068] The docking station 80 may be two-dimensional or three-dimensional, preferably configured in an open ring shape. In one embodiment, the shape of the docking station 80 is generated by appropriate hardware and software in the VR object generation unit 28. The VR object generation unit 28 may include appropriate hardware and software for generating the docking station and information objects or widgets, and may also include hardware for receiving external data 36. With respect to the docking station 80, the VR object generation unit 28 may be programmed to generate a docking station having a predetermined shape and size, and the size of the docking station may be scaled by the user during use. Once the shape of the docking station 80 is determined, the VR object generation unit 28 may generate the slots 84 by dividing the space of the docking station 80 in a predetermined manner to generate a selected number of slots 84 with a selected spacing between them. The docking station 80 may remain in a fixed position on the work area within the display, although in another embodiment, the docking station is draggable by the user, allowing the user to reposition the docking station and the widgets contained therein. The docking station 80 may be attached to and / or docked with a virtual object in the VR world 100, such as a virtual workstation model.

[0069] The information object 82 may also be positioned in an attached mode or state. In attached mode, the information object maintains a specific position and orientation relative to the display unit 12 (e.g., appears to move relative to the user), and the position and orientation of the information object 82 are specified by the user. In this mode, the information object 82 is always visible to the user, regardless of how the user adjusts the user's camera or view. In embodiments with an HMD, the user has the appearance and feeling of something attached to their head; as the user moves their head, the information object moves with them, so the information object always remains in the same position and portion of the user's field of view. Attached mode also allows the user to place the information object in their field of view so that they can always see the information object without having to select a widget from the docking station 80, or to position the information object 82 by moving their head, as would be required if the information object were in free mode. In embodiments with a stationary screen, attached mode locks the information object in a specific position and orientation relative to the screen, similar to a head-up display.

[0070] In the present system 10, informational objects 82 may be preconfigured in any of the above modes, and one or more informational objects 82 may be preconfigured in one mode or state while other informational objects 82 are configured in a different mode or state. For example, the surgeon may configure an informational object 82 in a free mode so that the object can be placed or positioned at a selected location in the VR world 100, such as the bottom of the work area, and other informational objects may be configured in an attached mode so that the object is always within the surgeon's field of view. Other informational objects may be placed in a docking station 80.

[0071] Additionally, information objects 82 may be generated on demand during a surgical procedure using an object list or object palette. Figures 5 and 6 show a schematic block diagram of an object list 90 according to the teachings of the present invention. The object list 90 is generated by the VR object generation unit 20 and may include a list of available information objects 82 corresponding to selected information or data generated by the system or associated devices that can be introduced into the VR world 100. Each information object 82 in the object list 90 may include an optional preview and title 84 of the object. Alternatively, the list may include any suitable type of graphical representation or icon 92 instead of a title. The object list 90 is a master list of information objects 82 available to the user. Some of the information objects listed in the object list 90 may be predetermined, while some information objects may be dynamically generated or discovered during startup or use. According to one embodiment, one or more information objects associated with a video feed generated by a camera in the camera assembly 44 may be automatically generated by the VR object generation unit 28, automatically listed in the object list 90, and optionally located in the docking station 80. The list of information objects may be presented as a simple object list 90 with small two-dimensional icons, although in other embodiments, the object list 90 may be a three-dimensional object list that serves as a container for smaller three-dimensional objects that may visually represent the information objects. In another embodiment, the information objects may be placed in a virtual drawer generated by the VR computing unit 14 attached to the virtual surgeon workstation. The surgeon can virtually manipulate the drawer (e.g., open or close the drawer) and access the information objects placed therein.

[0072] According to another embodiment of the present invention, system 10 may process selected data from selected external data sources, which may be introduced into VR computing unit 14, for inclusion in VR world 100, or may process data pre-stored in storage unit 24. Data 36 may include, for example, a representation of a workstation used by a surgeon, and may also include additional virtual elements, including a virtual drawer of user interface elements, a virtual tool belt showing one or more information objects as a virtual semicircular representation, camera-specific data such as zooming in and out of a camera feed, medical device controls, etc. A user may interact with the virtual drawer using a hand controller 50 or other user input device by grasping the drawer handle and opening it in the same manner as one would use to open a drawer in the real world. In some embodiments, the virtual drawer may be invisible to the user in VR world 100 until the user moves it into position across a threshold. Examples of suitable thresholds may include, but are not limited to, the side of the body of the surgeon's workstation or other defined locations within VR world 100.

[0073] Furthermore, an instance of the information object 82 may be generated by selecting and dragging the information object 82 into the VR world 100. For example, the information object 82 may be generated when application software associated with the VR object generation unit 28 is executed by making the information object visible and setting or defining the object's position in the virtual world to match the user's selection. According to a further example, a placeholder information object may be used when the information object 82 is dragged out of the object list 90 by the virtual representation of the user-selected device, and an information object is generated to replace the placeholder when the placeholder is dropped into the VR world 100. The virtual representation of the user-selected device may include a virtual laser pointer beam, reticle, or the like. The virtual user-selected device is controlled by the sensing and tracking unit 16 using a hand controller (e.g., hand controller 50) or a head-mounted controller.

[0074] This embodiment of the present invention may be used for informational objects 82 that are computationally intensive, such as live video feeds, but may also be used for any type of informational object. Furthermore, instead of the user placing the informational object 82 in free mode, a soft button associated with the informational object in the object list 90 may be used to instantly place the informational object 82 in the docking station 80 or in attached mode. The object list 90 may be configured to only allow one instance of a particular informational object, allowing for the same creation process detailed above and allowing the surgeon to easily locate a lost object. For example, if the surgeon accidentally moves an object directly behind him in the VR world 100 and loses track of it, the surgeon can simply drag the object out of the object list 90 again to reposition the object in the VR world 100, rather than searching for the informational object 82.

[0075] While the user is in the VR world 100, the user can use any type of controller, such as a head controller, foot controller, or hand controller, to move and place the information object 82 to a desired position in the VR world 100. A hand controller may be used to move and place the information object in the VR world 100. The hand controller may include, for example, the hand controller 50 described herein, but may also include other types of user input devices, such as a joystick, a laser pointer, or a computer mouse. Alternatively, a head controller, such as a head-mounted display, may be worn by the user and may include sensors and / or an eye-tracking system that track the user's head or pupil movements, respectively. When using the HMD, the user can select, move, or arrange one or more information objects in various ways, such as a gaze timeout, in which the user gazes at an object and selects the object after a predetermined period of time has elapsed with the user's eyes fixed on the selected object. The user may also highlight the object by fixing their eyes on it and then confirm the selection by pressing a button on the hand controller or a foot pedal. The user's gaze may serve to position a reticle over the selected element, and a hand controller may be used to select, drag, and drop objects. Furthermore, when selecting an information object, the object may be virtually grasped using any suitable actuator, for example, by a trigger button on the hand controller or a similar button on the user input device. The information object 82 maintains its position and orientation relative to the controller within the VR world 100 until the trigger button is released, thereby placing the object in the desired location.In some embodiments, any part of an information object can be used for selection, but in other embodiments where the information object includes an actuatable part such as a virtual button, slider, or other user interface control, only the title bar may be used to move the object.

[0076] An example of an information object 82 configuration suitable for use in the present invention is shown in FIG. 7. The illustrated information object 82 includes a title bar 86 that may include an object title 94 and one or more user-selectable action buttons 88. The title 94 may preferably indicate the contents of the information object. The information object 82 may further display information or content 96 associated with the object 82. For example, if the virtual object generated by the VR object generation unit 28 relates to image data received from the camera assembly 44, the object's content 96 may include the image data displayed therein. The title bar 86 may optionally be hidden so that it is only visible when the object is selected by the user, thus providing an auto-hidden mode. According to one embodiment, the title bar 86 may be automatically hidden when the information object 82 is in docked mode. The information object's control buttons 88 are operable, allowing the user to perform a selected action. These actions may include, for example, an auto-visualize button that allows the user to determine whether an object is visible, an object mode button that allows the user to switch modes, such as between free mode and attached mode, a docking station button that allows the user to move an object into a slot 84 in the docking station 80, a close button that hides the object 82 until the user creates the object or selects it from the object list 90, etc. Those skilled in the art will readily appreciate that any selected number of action buttons 88 may be provided and that each button may be assigned any selected action. The action buttons 88 may be customizable by the user, or may be predefined by the system.

[0077] The VR object generation unit 28 can generate any selected type of object for displaying, presenting, or arranging data (e.g., data 36) generated by the system 10 or introduced into the system from an external data source. Another example of an information object 82 that can be generated by the unit 28 and used by the system 10 may be a surgical system-specific information object (e.g., a "mini-world") as shown in FIG. 8. FIG. 8 shows a virtual reality world 100 including an object list 90 containing a list of information objects 82. The object list includes, among the listed information objects, a surgical system information object 102A. The information object 102A can be selected from the object list 90 using any type of selection device, including, for example, a reticle (e.g., a graphical element in the form of a crosshair or a dot) or the illustrated virtual laser beam graphical element 104. The laser beam 104 may point to and hover over the title of the mini-world object in the object list 90, allowing a user to select the object 102A using a suitable controller, such as a hand controller. The information object 102A can be dragged from the object list 90, causing the information object 102A to appear as shown in FIG. 7. The information object 102A includes a title bar 94 and a content area 96 that displays selected types of data, including, for example, a computer-generated virtual reality image 106 of the surgical system being used by the surgeon, which mimics the movement of the surgical system in real time. In this manner, the surgical system information object 106 can display content related to a surgical robot, motor unit, insertion arm, trocar, robotic support system (RSS), operating table, surgeon workstation, etc. The surgical system information object 102A allows the surgeon to visualize the surgical system, even parts located external to the patient that are not visible to the surgical camera assembly 44 inside the patient's body. This may also include virtual representations of a standardized patient and operating table for reference.The virtual reality world 100 may further include additional information objects, such as an MRI information object 102B that includes MRI-related information for a patient therein.

[0078] Additionally, the surgical system information object 102A can display multiple data sets via the display unit 12, providing a frame of reference for the surgeon. In particular, the information object may display images or videos of one or more components of the surgical system, while the object may display representations of one or more components of the surgical system, such as, for example, a robotic arm and camera assembly. These representations may be displayed within a representative virtual patient to provide the surgeon with more robust criteria. Display of this type of information allows the user to determine the precise location and orientation of system components, such as, for example, a robotic arm.

[0079] According to another aspect of the present invention, the surgical system information object 102A allows a surgeon or user to visualize transformations (such as translation, rotation, and scaling) between the human movement of the hand controller within the surgeon workstation space or frame of reference and the movement of the robotic arm 42 and associated end effector within the virtual world 100. This automatic visualization or coordination technique can be activated when the hand controller switches from controlling the robotic arm 42 to modifying the transformation. The surgical system information object 102A may be automatically activated and displayed to the user. Additionally, the surgical system information object 102A may be used to visualize off-camera portions of the robot during some movements. For example, when a hand controller is used to raise or lower the nominal elbow height adjustment of the robotic arm 42, the reference frame of the surgical system information object 110 can visualize the elbow offset by simultaneously displaying the reference robotic arm and its matching human arm. This may be supplemented by displaying the robotic arm moving within a representative virtual patient. In this embodiment, auto-visualization techniques may be used to cause the surgical system information object 102A to automatically appear whenever the user touches the elbow adjustment toggle on the hand controller.

[0080] The surgical system information object 102A may also be used to visualize portions of the robotic system that are entirely external to the patient, including features such as a support arm connected to the RSS that independently move the robotic arm and robotic camera to different orientations, and may be visualized when the user uses options to change the position of these components. For example, as the surgeon moves the attached robotic system by moving the RSS, the current position (e.g., yaw, pitch, and roll) of the system may be represented in full view based on data 36 introduced into the system, thereby showing the current rotation of each joint of the RSS. The object 102A also allows the user to visualize the entire surgical system, if desired, while also allowing the user to highlight specific components of the system.

[0081] In addition to the surgical system information objects, the system 10 of the present invention can include or use additional information objects that provide the user with additional information, such as camera-related information. For example, the additional information objects can include or be provided for display: live camera data from the operating room; live video outputs from one or more patient monitors (e.g., blood pressure, heart rate, oxygen level, and other patient vitals); live video outputs from any necessary medical devices (e.g., cautery or injectors); and / or live video outputs from a computer used by a circulating scrub nurse. Additionally, in some embodiments, the information objects can provide the user with two-dimensional and / or three-dimensional data 36 from external sources, such as preoperative or live patient images, such as X-ray scans, CT scans, MRI scans, etc. In some embodiments, the system 10 of the present invention can include status information objects that indicate, for example, the status of various elements of the robot (e.g., robotic arm, camera clutch-in status, etc.). Additionally, the object list 90 can include a clock information object that can be used to display the current time, elapsed procedure time, and / or total robot clutch-in time. Additionally, patient record information objects can be presented that indicate relevant patient information, such as age, weight, allergies, etc. Additionally, in some embodiments, a virtual ruler information object may be shown that allows the surgeon to measure the actual patient's anatomy using a virtual tape measure or laser rangefinder. Note that in some embodiments, system 10 can include other types of information objects with desired functionality or information data requested by the user.

[0082] The surgical virtual reality user interface generation system 10 of the present invention can also use information objects 82 to enable a user to interact with or engage and control one or more components of the surgical robotic system 20. For example, as shown in FIGS. 9A and 9B , the VR object generation unit may generate a robot information object 110 that can display information associated with, for example, the robotic arm 42. According to one embodiment, the content region 96 of the object 110 can display a representation of the current position 112 of the robotic arm and a separate representation of the actual position 114 of the user's arm and hand. According to one embodiment, the surgical system information object 110 displays to the user a representation of the nominal starting position 112 of the robotic arm 42 when the surgeon first initiates control and a representation of the current position 114 of the robotic arm, allowing the user to visualize the difference in position. In order for a user to begin interacting with and driving the surgical robotic system 20, the user must align their hand to match the current position and orientation of the robotic arm 42 before the user drives the robot. This coupling or clutching procedure is performed to prevent any unintended movement by the robot arm 42 before the robot can follow the surgeon's arm movement. When used, this coupling procedure prevents the robot from moving to follow the user's arm until the user generally matches the current position and orientation of the robot arm 42, which may include, for example, the angle of the robot arm's gripper or end effector, so that if the robot is disengaged or clutched out while holding an object, it will re-couple when the same amount of force is applied, preventing the object in the end effector from falling into the surgical site. The robot information object 110 may provide arm coupling or clutching guidance and feedback to assist the user in properly aligning themselves with the current position of the robot arm, allowing the user to couple with and drive the robot.

[0083] For example, as shown in FIG. 9A , the actual position of the robotic arm 112 is shown within the information object 110. The actual position of the arm of the surgeon 114 is also shown. The positions of the robotic arm and the surgeon's arm can be indicated or displayed using any desired visual aids or cues; in this example, a representation of the robotic arm is shown with a solid line 112 and a representation of the surgeon's arm is shown with a dotted line 114. A controller associated with the surgeon may be placed in a disengaged or clutched-out mode in which movements of the surgeon's head and / or arm are not communicated to the surgical robotic system 20, and therefore the robotic system does not act in response. The controller may then be placed in a coupled or clutched-in mode in which the surgeon's movements are communicated to the robotic system, and the robotic system acts in response. Thus, during the coupled mode in which the robot moves in response to the surgeon's movements, it is important that the surgeon's head and / or arm are aligned with the position and orientation of the robotic arm and camera assembly. Information object 110 displays the surgeon's alignment information, and as the surgeon moves, the surgeon's arm and head positions 114 are reflected in content area 96, and the surgeon continues to move his or her arm until representation 114 of his or her arm is aligned with representation 112 of the robotic arm, as shown in Figure 9B. Once they are aligned, the surgeon can interface with the controller to operate the robotic system accordingly.

[0084] The present invention contemplates the use of any selected type of model or visual cue, such as a hand model that can be displayed within the VR world 100 as, for example, a two-dimensional or three-dimensional (3D) model of the actual physical hand controllers held by the user or attached to a workstation. In another embodiment, the hand positions may be displayed within the VR world 100 as 3D models of human hands. In yet another embodiment, the hand model is a combination of both 3D models of human hands and 3D models of hand controllers. In an alternative embodiment, the hand model may be shown as a separate 3D design or object that displays how the user's hands and the controllers they hold are positioned and oriented. In yet another embodiment, the hand model is configured in the shape of a ring that can be positioned to match a corresponding ring representing the robot's position and orientation. Orientation cues may be represented by color-coded segments on the position cue ring, as shown in FIGS. 10A and 10B .

[0085] Additionally, the gripper guidance cue may be configured as a sphere or ball attached to the position cue ring and the virtual robotic arm. FIGS. 10A and 10B exemplarily illustrate a virtual world 100 showing image data associated with a surgical site captured by the camera assembly 14. The surgical site may be, for example, an interior portion of a patient's abdominal cavity. The image data also includes data associated with an actual robotic arm 42. In addition to the image of the actual robotic arm 42, the virtual world 100 also displays a virtual representation 42A of the robotic arm that the user can manipulate using a hand controller to move a graphical component 118A toward a cue or target graphical component 118B. The virtual world 100 also displays a robotic arm information object 120 that displays the robotic arm within the content area. The information object 120 may further include directional arrows that allow the user to move the robotic arm and camera assembly in yaw and pitch directions. The virtual robotic arm 42A can be manipulated to move the graphical component 118A toward a target graphical component 118B. Once the ring portion of graphical component 118A is aligned with the ring portion of target graphical component 118B, the user can adjust the position of the spherical portion of graphical component 118B toward the hole in the central region of target graphical component 118B. The user adjusts the position of the spherical portion of graphical component 118B by adjusting the commanded angle of the grip of the robotic arm, typically controlled by a trigger on the hand controller 50. The user attempts to align the spherical portion of graphical component 118A with the hole in the central region of target graphical component 118B, and may be assisted by other graphical elements, such as alignment graphical component 118C. The graphical elements 118A are positioned and oriented to reflect or mimic the actual position of the robotic arm 42.When the spherical portion of graphical component 118A is aligned with and properly inserted into the hole in the central region of graphical component 118B, the color of graphical element 118 may change to indicate that both the spherical and ring portions of graphical component 118A are properly aligned with the target graphical component 118B, thereby indicating both a proper grip of the arm's gripping portion and that the surgeon's arm is properly aligned with the position of robotic arm 42. That is, the user can properly control the robotic arm from the moment the user properly matches the robot's position and orientation. To summarize this process, as the surgeon's arm moves, the hand controller can correspondingly move virtual robotic arm 42A and graphical component 118A, indicating the proper coordination position when graphical component 118A is inserted into the center of target graphical component 118B, as shown in FIG. 10B. The robot drive application software associated with the surgical robotic system 20 indicates to the user that their arm is “close enough” when the position, orientation, and gripper tolerances of the virtual robotic arm 42A all match, and after a timed delay, the user can engage (e.g., clutch in) with the real robotic arm 42 and move the robotic arm. In the VR world 100, “close enough” is indicated by any selected visual cue that provides feedback to the user as to how they need to move their hand to match the robot’s position, orientation, and gripper angle. Those skilled in the art will readily recognize that in this example, the left virtual robot is being operated. The same sequence may occur for the right virtual robotic arm, using similar graphical elements to engage or clutch in with the right robotic arm.

[0086] Furthermore, the system 10 can be configured to clutch or engage the robotic camera assembly 44 in addition to the robotic arm 42. To prevent unintended movement of the camera assembly 44, the user aligns their head with the direction of the camera assembly 44 before engaging the camera assembly and activating the assembly's camera. In one embodiment, a visual target is presented to the user within the VR world 100 while the user's responses are measured using any associated VR head tracking system. In another embodiment, a camera view is shown within the VR world 100, serving as a window into the patient's body. In this embodiment, the user can grasp this information object using the laser beam graphical element 104 and move it in any direction, with the camera of the camera assembly 44 moving correspondingly. By moving the information object, the user can move the camera so that it is aligned with the user's head.

[0087] When the user's head and the camera are aligned and "close enough" to engage or clutch in with the camera assembly 44, a proximity sensor located within the HMD worn by the user communicates with the robotic drive system to determine that the user is wearing the HMD. Once this determination is made, the robotic drive system automatically initiates clutch in. In another embodiment, a laser pointer interface is used and the user selects a virtual button attached to the clutch object, while in another embodiment, the user selects the clutch object itself using the laser pointer interface to manually request clutch in from the back-end robotic drive.

[0088] According to another aspect of the present invention, image data captured by the camera assembly 44 may be streamed from the camera assembly to the VR computing unit 14 over a network. Additionally, the surgical robotic system 20 may publish or broadcast additional camera-related data, including, for example, position, orientation, field of view, and inter-camera distance. The VR computing unit 14 can use this data to place a simulated virtual scene camera in the VR world 100 at a corresponding position and orientation, and with the same inter-camera distance. The virtual scene camera is a software camera that captures images and / or a live feed of the VR world and presents them to the user via a screen in the HMD. In one embodiment, image data from the left and right cameras is rendered on a quad (e.g., a virtual screen) attached directly to the VR camera. In these embodiments, the quad is configured to move with the user's head. In another embodiment, the quad is positioned within the VR world 100 based on its position and orientation detected by the camera driver. In the event of a camera failure, the user can select a monoscopic view (monoview) instead of a stereo view, using image data from either the left or right camera and streaming the monoview to both eyes.

[0089] The surgical virtual reality user interface generation system 10 of the present invention is also configured to generate data from the surgical robot system 20 and generate objects for display within the VR world 100. The present invention is particularly configured to capture image data from the camera assembly 44 and generate a spherical image of the entire environment surrounding the camera assembly using a spherical image generation unit. As shown in FIGS. 11-13 , the VR computing unit 14 can include a spherical image generation unit 128 for generating the spherical image 130. Those skilled in the art will readily appreciate that the spherical image generation unit 128 can also form part of the computing unit 18. Known spherical images 130 can include multiple interactive wide-angle panoramic images that generally encompass a 360-degree circular or spherical view for the user, if desired. The spherical image 130 can form a complete scene from multiple images from the image data viewed when rotated around a single center point or location. If these images are formed from multiple photographs or images, they may be stitched together according to known conventional techniques. Alternatively, the spherical image 130 may be formed from partial or complete virtual reality images generated by the VR computing unit 14, or from a combination of photographs and computer-generated objects. This is similar to panoramic photography on modern smart devices, but instead of being limited to rotating the camera around one axis, it is rotated around two axes, and a sphere of images is stitched for each of the two cameras in a stereo camera.

[0090] According to the present invention, a celestial sphere image 130 can be generated by a celestial sphere image generation unit 128 and configured to be rendered into a cube map used as a background for the VR world 100. The system of the present invention can also use a virtual reality (VR) camera unit 38 to generate one or more virtual reality (VR) cameras for use or placement within the VR world 100. The system can also use a scene graph unit 154 to generate a scene graph 140 having a main virtual camera and one or more VR cameras that render images for the head-mounted display as peers or children of the node. This allows the VR camera to always render into the cube map the same view as seen by a user wearing the head-mounted display. While one embodiment may use a single VR camera, another embodiment may use separate right-eye and left-eye VR cameras to render into separate right-eye and left-eye cube maps to provide a stereo view. The FOV setting of the VR camera may self-configure itself to the FOV exposed by the camera assembly 44. In addition to providing a contextual background for live camera views or image data, cube maps can be used to generate dynamic reflections on virtual objects. This effect allows reflections from cubemaps to be projected onto the reflective surfaces of virtual objects, making these objects appear to the user as if they actually reflect the real-world environment.

[0091] Furthermore, the spherical image 130 is typically only valid at the location where the spherical image 130 was captured. Images captured by the camera assembly 44 will no longer be consistent with the images in the spherical image 130 if the camera assembly 44 is moved. When the camera or a robotic assistance system (RSS) holding the camera is moved, the updated location may be published on the network as an inter-process communication (IPC) message. This instance of the spherical image may then completely delete the cube map and set all pixels to black so that no old image remains in the spherical image. In one example, the system may perform an auto-scan each time the camera is moved, and an entirely new spherical image may be constructed and made available to the user.

[0092] In accordance with the teachings of the present invention, the orientation of the camera assembly 44 may be driven by the orientation of a head-mounted display (HMD) such that the camera of the camera assembly 44 is positioned and oriented in the direction the person wearing the HMD is looking, and the image from the camera is displayed within the display portion of the HMD. The present invention may be utilized when the field of view (FOV) of the camera assembly 44 is smaller than the field of view of the HMD on which it is displayed. The generated spherical image is displayed outside the camera's FOV. The generated spherical image provides historical information that has not been updated since it last entered the camera assembly's field of view. In this manner, the spherical image 130 provides environmental context and effectively extends the user's peripheral vision. In one embodiment, movement of the HMD may serve to update the spherical image 130 with additional image or video data. The historical, non-live nature of the spherical image may be indicated by adjusting the color or tone of the image. In one embodiment, the spherical image is displayed in grayscale.

[0093] The system of the present invention may also employ an auto-scan mode in which the control unit 26 generates control signals received by the camera assembly 44, causing the camera to autonomously rotate throughout its entire range of motion to capture images of the workspace or surgical site within the patient. The cameras of the camera assembly 44 generate image data that may be transmitted to the spherical image generation unit 128 to generate the spherical image 130 or to update the image data within the spherical image 130. Thus, the spherical image 130 may include image data from the cameras. After the image data is captured in the spherical image 130, the spherical image generation unit 128 may continuously or periodically update the image data within the spherical image with image data received from the cameras. The auto-scan mode may also be configured to scan only the area necessary to fill the portion of the HMD's FOV not covered by the camera FOV, based on the user's current gaze direction. During the auto-scan mode, it appears to the user wearing the HMD that the camera has stopped moving in accordance with their head movements. Thus, the user can freely view any portion of the spherical image 130 via the HMD or display unit 12, and as the user views the image data in the spherical image 130, the user can observe how the image data in the spherical image is updated (e.g., colored) as the camera assembly 44 is moved throughout its range of motion. In one embodiment, the auto-scan mode may be used to capture images from the stereo cameras of the camera assembly 44 and to create a rendering of the surgical site to allow the user to obtain an image of the site.

[0094] The system of the present invention may also be used to support multiple observers of the robotic camera assembly 44 and to support post-operative evaluation of image data generated by the camera assembly. In either of these cases, simply playing back recorded image data in the HMD is typically uncomfortable for the observer because the image camera's FOV typically does not match the observer's head movement while viewing. To the observer, the image appears to rotate around the surgical field based solely on what the user is doing, without any control from the observer. The discrepancy between what the observer sees with their eyes (i.e., panning the camera image) and what the observer's vestibular system experiences (e.g., not moving their head) can cause simulator sickness, dizziness, or nausea. However, in the present invention, the observer can view a spherical image 130, thus presenting a 360-degree view in which the observer can freely move their head to look in any direction and not be uncomfortable due to a discrepancy between the observed head movement and their actual head movement. In one embodiment, only a portion of the spherical image 130 represents live image data from the camera assembly 44, which may be incorporated into the spherical image 130 in a contextual relationship with all other recorded images. For example, as shown in FIG. 12B , the generated spherical image 130 includes a virtual representation of a robotic arm 42A positioned within the VR world of the spherical image. A live camera feed or window may be presented as part of the image data field 132 and may include video of the actual robotic arm 42 performing a surgical procedure on a human organ 160. The organ may be presented as part of the live video feed and as part of the spherical image 130.

[0095] The illustrated spherical image generation unit 128 can also be configured to differentiate live image data captured by the camera assembly 44 from older image data in the spherical image 130. In one embodiment, the system 10 provides a seamless spherical image 130 with as few image artifacts as possible. One way to achieve this is to use standard image blending techniques, such as dissolve, multiply, dodge, and burn techniques. These techniques can help soften hard edges of each image rendered on the spherical image and edges of the current image against the spherical image background. Another technique is to vary the frequency at which images are rendered by the VR camera and applied to the spherical image. Slow rendering means that the spherical image data has fewer blended edges and displays slightly older images, while fast rendering means that there are more edges blended with lower latency. In other embodiments, smart blending or image stitching techniques capable of understanding the scene and the objects therein can improve the smoothness and quality of the composite image. For example, if the system can recognize that an object in one image is the same as an object in a second image, it can smoothly and continuously mesh the contours of the objects. This technology may use artificial intelligence, machine learning, and computers for these purposes.

[0096] The present invention also provides a clear depiction of which images are “live” and which images are older and form part of the spherical image 130. Older images can pose a safety concern, for example, in a surgical system if a patient begins bleeding in an area that is not being captured live but is only visualized as part of the spherical image 130. In this example, the surgeon would not know the current state of the bleeding until he or she moves his or her head to return the camera to the area and acquire an updated image. For external observers or post-operative evaluation, it is desirable to know in which direction the surgeon is looking. Several techniques can be applied to make this more obvious. In one embodiment, the cubemap texture resolution may be reduced so that the pixel density of the area of ​​the HMD's FOV displayed as part of the spherical image 130 is significantly smaller than the pixel density of the camera quad onto which the live image is mapped. This may give the spherical image 130 a blocky, pixelated appearance, which may be softened by applying a blur filter to the cubemap. In another embodiment, the brightness of the cube-map texture is reduced so that the celestial spherical image 130 appears darker and the live, updated camera quad appears significantly brighter. In yet another embodiment, a frame object is placed around the camera quad, so that there is a visible boundary between the live data and the celestial spherical image 130. An example of this technique is demonstrated in FIGS. 12A and 12B . The celestial spherical image 130 generated by the celestial spherical image generation unit 128 uses image data generated by the camera assembly. The celestial spherical image 130 includes an image data field 132 differentiated in a known manner from the remainder of the celestial spherical image 134 to present live image data of the surgical site being captured by the camera assembly. The remainder of the celestial spherical image may be visually differentiated from the image data field 132, for example, by redacting the remainder of the celestial spherical image or portions of the celestial spherical image immediately adjacent to the image data field 132.For ease of reference, the spherical image 130 may also be configured to include a surgeon workstation area 136 including a visual representation of the surgical system and a visual representation of a hand controller 136A. In one embodiment, a visual representation of a patient 136B may also be included. The virtual world 100 may also include a second type of docking station configured in the form of a tool belt-style docking station 138, if desired. The tool belt docking station 138 may include multiple information objects 82, if desired. The tool belt docking station may be positioned within the virtual reality world 100 around the user's waist region, and thus may be in the user's view when the user looks downward and out of the user's view when the user looks upward.

[0097] In yet another embodiment specifically designed for remote post-operative assessment, directional indicators, such as arrows, are placed in front of the user. These indicators guide the user's gaze in the current direction the camera quad and robotic camera are pointing. These are either attached to the VR camera (effectively attached to the user's head) or added to the scene graph 140 as objects that may be placed at fixed locations within the virtual reality world 100. Such objects may be set to be always visible, or to be visible only when the HMD is moving beyond a certain threshold. The latter mode allows the user to look around and focus on something without being interrupted by the arrows, and then when the user looks around to return to the "live" camera quad, the arrows appear to assist them.

[0098] The user directly controlling the surgical robotic system 20 is herein understood as the primary operator, user, or surgeon, and all other observers are herein understood as third-party observers. The third-party observer may observe a live surgical session or replay a previously recorded session for post-operative evaluation. The primary observer may participate on a local area network or remotely over a wide area network. The observer may connect to both the video stream and inter-process communication (IPC) messages and associated data. The remote observer may wear an HMD other than the one the observer uses to control the robotic camera. Therefore, the observer's camera quad must be driven by IPC messages that describe the commanded position of the robotic camera. Thus, the live image transmitted from the camera appears on the quad positioned based on where the primary operator is pointing the robotic camera.

[0099] Additionally, post-operative evaluation may be performed by recording timestamps when recording both the video stream data and the IPC data. In one embodiment, the timestamps are encoded in the video stream metadata, and the IPC data is written to disk with the timestamps in JSON format. A playback application may simultaneously expose both the video data and the IPC data, maintaining a correlated timestamp between the two data sets. Post-operative evaluation may be performed in the same VR application used by the live observer, only different in that the data is received from the playback application rather than from the live session.

[0100] There are objects of various scales that the VR computing unit 14 can use, for example, by using the optional VR camera unit 38, to render and present to the user via the HMD. For example, there may be objects rendered at robot scale in the VR world 100, in addition to objects rendered at human scale. In this example, as is customary in most virtual reality applications today, the user may be presented with a virtual representation of the user's hand controller and the objects with which the hand controller can interact. From the user's perspective, the hand controller and the objects with which the hand controller can interact appear to be at the same scale as in the real world. Within the same virtual world, a rendering of a virtual representation of a robotic arm may exist at a scale that differs from the apparent scale of the real world from the user's perspective. The different scales of these two sets of objects can make it difficult for them to interact or to be presented to the user so that they appear to be at the same scale. To solve this problem, one set of objects may be scaled up or down an appropriate amount to match the scale of the other set of objects. To this end, the system of the present invention may also be configured to enable the virtual cameras used to generate views to each eye in the head-mounted display (HMD) to use the same inter-camera distance (ICD) as the stereoscopic robotic camera assembly 44. Even if the ICD is substantially different from the user's inter-pupillary distance (IPD), the present invention maintains correct scaling of both robot-scale and human-scale rendered objects by approximating a scale factor equal to the ratio of the user's IPD (in distance units) to the camera assembly's ICD (in the same distance units). This scale factor is then applied to enlarge the object to robot scale, or the inverse of this scale factor is applied to reduce the object to human scale.This allows the system to switch between displaying objects at robot scale, displaying objects at human scale, or displaying objects at both scales simultaneously, so that all objects appear to the user at human scale at all times. For example, if the user's IPD is 65 mm and the camera's ICD is 13 mm, the scale factor may be estimated to be 5 mm by dividing the IPD by the ICD (IPD / ICD), and this scale factor may then be used to magnify the virtual representation of the robot arm at robot scale by a factor of 5 and present it to the user via the HMD so that it appears at human scale. If available, other measurements of the user's anatomy may also be used to estimate the scale factor. For example, the user may hold their arm outstretched, and the distance between the hand controller and the HMD may be taken as an estimate of the user's arm length. The scale factor may then be estimated as the ratio of the estimated length of the user's arm (in distance units) to the known length of the robot arm (in the same distance units). This scaling can also be applied to head tracking, so that as the HMD is tracked at human scale, its movements are scaled down and applied to the virtual camera at robot scale.

[0101] The present invention also accommodates robotic camera assemblies 44, which typically have fewer degrees of freedom (DOF) than a human wearing a head-mounted display. For example, if the camera assembly 44 can only rotate in yaw and pitch directions (e.g., two degrees of freedom), it can be problematic that the movement of a human's head being tracked within the VR world 100 can also rotate in rotational and translational directions (e.g., x, y, z) in addition to yaw and pitch directions, for a total of six DOF. The camera assembly 44 can also transmit data over the system network regarding selected types of parameters, including, for example, the current position and placement (e.g., pose) of the camera within the camera assembly 44, the ICD, and the field of view (FOV). In this case, the VR computing unit 14 may be configured to align the VR world 100 with, for example, a virtual camera therein. Furthermore, as is known in the art, the VR world 100 may be represented by a series of data structures organized hierarchically as part of a scene graph. A scene graph arranges a logical, and often spatial, representation of a virtual reality graphical scene and may be represented by a series of nodes in a graph or tree structure. An example of a tree-type scene graph representing a VR world 100 is shown in FIG. 13. The illustrated scene graph 140 may be generated by the scene graph generation unit 154 as described above and may include a parent node 142 for a head tracking sensor associated with the display unit 12, specifically a head-mounted display. The scene graph 140 is configured such that all child nodes 144, 146, and 148 conform to the characteristics of the parent node 142. The head tracking parent node 142 receives tracking data from the camera assembly 44 via the VR object generation unit 28 and the VR computation unit 14 and applies this data, which typically represents the position of the HMD, to the child nodes. By way of example only, the scene graph 140 may include a child node 144 for positional tracking data only, a child node 146 for rotational tracking data only, and a child node 148 for positional and rotational tracking data.

[0102] In the VR world 100, the virtual camera generated by the virtual camera unit 38 of FIG. 1 may be configured to directly track the position of the HMD. A properly embodied neck model is important for user comfort, where the pose of the virtual camera matches the user's eye movement as the user tilts their head. If the IPD is reduced but head tracking is not adjusted, the user will see their head move through a larger arc as they tilt. To correct for this ratio and mitigate or eliminate the occurrence of this phenomenon, the scene graph generation unit 154 may generate a scene graph 140 that includes an IPD scaler unit 156 that can scale position and rotation data received from the HMD so that data related to the HMD is robot-scaled rather than human-scaled. Each child node in the scene graph 140 is also constrained by the IPD scaler unit 156, with the inverse scale parameter enabled. This allows objects located below these nodes (e.g., child nodes) to appear at their default scale and not inherit any scaling intended to compensate for the neck model. In a default configuration, IPD scaler unit 156 can generate and apply a scale factor that scales a human-scale object to robot-scale. IPD scaler unit 156 may also be configured to perform inverse scaling, which allows a child node to reverse the scaling applied by the IPD scaler unit of a parent node. Scaling may also optionally be performed by VR camera unit 38.

[0103] Additionally, for surgical robotic system 20, the ICD is typically very small compared to a normal human IPD due to the robot's need to enter the patient. IPD scaler unit 156 can address this discrepancy by reading the ICD value published on the network by camera assembly 44. Scaler unit 156 can also read the HMD's IPD data published by the associated virtual reality API (e.g., OpenVR, OpenXR, etc.) and apply this to the headset position to determine the distance between the virtual scene cameras, understood herein as the virtual camera ICD. A scale factor applied to a parent node and generated by IPD scaler unit 156 can be determined by dividing the robot camera ICD by the virtual camera ICD.

[0104] Image data generated by the robotic camera assembly 44 may be used as texture for (i.e., rendered onto) one or more quads in a head-mounted display. In an HMD, at least one quad is associated with a display for each eye. The FOV values ​​broadcast by the camera assembly 44, one for each eye of the HMD, can be positioned and oriented using two different techniques. Both techniques require the camera assembly 44 to expose an FOV value. The FOV value allows the size of the quad or the distance between the quad and the virtual camera to be adjusted to occupy the same FOV in the virtual world as in the real world.

[0105] In one embodiment, the size and position of the quads in the HMD can be configured to always appear directly in front of the virtual camera and therefore directly in front of the user when looking through the HMD, and further, the relative positions of the quads can be configured to maintain them from rotating in selected ways that would not be possible with a real camera.

[0106] In another embodiment of the inventive system, the camera of the camera assembly 44 has a position sensor that detects the camera's pose in all degrees of freedom (DOF) along which the camera can be moved and rotated by the user. These values ​​are sent from the camera to the VR application over the network and applied to a "camera-driven camera quad" node. This allows the quad to be displayed at a position corresponding to the camera's real-world pose. This can be advantageous when the robotic camera moves slower than the human head (tracked by the HMD). The quad, and therefore the camera image, may be displayed at the position reported by the camera in the virtual scene such that if the human head moves faster than the camera assembly can move, the quad position will follow the camera assembly's movement and eventually match the user's pose after the user's head movement stops. In some robotic camera systems, the sensed pose data may come in at a much faster rate and / or with lower latency than the video feed. In such cases, application of the sensed data to the "camera drive camera quad" node may be delayed by an asynchronous standby function (or some equivalent) to allow the sensed data to better correlate with where the video feed is being displayed.

[0107] In one simple example, the surgical site may be viewed by a camera assembly that is driven to match the pose (e.g., yaw, pitch, and roll) of the user wearing the HMD. The camera assembly 44 may include a left camera module and a right camera module separated by a distance known as the inter-camera distance (ICD). Each camera module consists of at least a lens or lens stack, an image sensor, and a housing that holds everything in place. Each image sensor is exposed to light through the lens or lens stack to generate an image, which is then transmitted to the system 10 and displayed in the display unit 12. The entire image sensor may be exposed to light, or only a portion or subsection of the image sensor may be exposed to light. The portion of the image sensor used to generate the image is understood to be the active portion of the sensor. Each camera module is configured to have a known set of parameters, such as focal length, field of view (FOV), and sensor size, and should have low distortion and aberrations. The FOV of a camera may be measured as a diagonal FOV, but may also be measured separately as a horizontal FOV or a vertical FOV with corresponding aspect ratios (e.g., the ratio of the height to width of the image).

[0108] The system may be configured such that the VR computing unit 14 generates a virtual world 100 that is displayed on and viewable through the quads of the HMD. The virtual world may include at least one virtual camera and one finite plane (e.g., a quad) per eye. In one embodiment, there is one quad visible only to the right virtual camera and another quad visible only to the left virtual camera. Each virtual camera functions similarly to an actual camera in the real world in that it generates an image based on the virtual scene presented to the camera. The image collected by the left virtual camera is observed by the user's left eye through the optics of the HMD. The image collected by the right virtual camera is observed by the user's right eye through the optics of the HMD. The quads are positioned perpendicular to the visual axis of each virtual camera so that the user can observe the height and width of the quad but not the depth. The right quad displays an image from the camera's right camera module. The left quad displays an image from the camera's left camera module. The virtual dimensions of the quads are such that the height-to-width ratio of each quad is equal to the height (number of pixels) to width (number of pixels) ratio of the desired image from the respective camera module.

[0109] When designing the virtual world 100, known camera parameters are taken into account to create a comfortable and scaled representation for the user to observe the surgical scene through the HMD. The field of view of each camera module is a function of the dimensions of the active portion of the image sensor and its focal length. The field of view of each eye perceived through the HMD is a function of the dimensions of the quad, the depth of the quad's location from each virtual camera, and several dimensional ratios derived from these values. The ratio of the width of the active portion of the image sensor (in mm) to the focal length (f) of the camera module (in mm) must be equal to the ratio of the width of the quad (in distance units) to the depth of the quad's location from each virtual camera (in the same distance units). In addition, the ratio of the height of the active portion of the sensor (in mm) to the focal length (f) of the camera module (in mm) must be equal to the ratio of the height of the quad (in distance units) to the depth of the quad's location from each virtual camera (in the same distance units). The dimensional ratios can be expressed as follows: (Width of the active part of the sensor) / f = (Width of the quad) / (Depth of the quad) (1) and (Height of the active part of the sensor) / f = (Height of the quad) / (Depth of the quad) (2)

[0110] By maintaining the ratio of equations (1) and (2), the horizontal and vertical fields of view of the image from each camera module will be equal to the horizontal and vertical fields of view perceived by each eye in the HMD. Alternatively, this may be achieved by maintaining the following equations: (Height of the active area of ​​the sensor) / (Width of the active area of ​​the sensor)=(Height of the quad) / (Width of the quad) (3) and either one of the following formulas (4) and (5): (Quad depth)=(Quad height) / (2 * tan(1 / 2 * Vertical FOV)(4) or (Quad depth)=(Quad width) / (2 * tan(1 / 2 * horizontal FOV)(5)

[0111] According to one embodiment, the aspect ratio of the active portions of each of the left and right camera modules is replicated for the left and right quads viewed through the HMD, respectively, according to the above formula. Thus, each quad may be flat and rectangular, and each may be arbitrarily sized. The depth of each quad associated with the corresponding virtual camera is set and calculated using one or more of the above formulas.

[0112] According to another embodiment, the quad may have a rectangular surface curved into a cylinder (e.g., similar to a curved television), and the corresponding angular measurements of the vertical or horizontal FOV may be used to define the angular cross-sectional area of ​​the cylindrical portion that defines the quad. The virtual camera may be located at the center of the cylindrical portion. The quad may consist of a surface of any shape and curvature, such as a portion of a sphere. The curvature of the quad may be used to create some perceived effect on the image, such as lens distortion correction or enhancement. The shape of the quad may affect the user's comfort level while the user moves their head.

[0113] There are several potential drawbacks that can arise when there is a mismatch between the camera module's size ratio or field of view and the perceived size ratio or field of view within the HMD. For example, if the perceived field of view is larger than the camera module's actual field of view, the user may perceive the world as being too large or too zoomed in. Additionally, in this configuration, when the user moves their head, the perceived angular velocity of the movement may differ from the actual angular velocity of the user's head movement. This velocity discrepancy may cause the user to feel nauseous or nauseous.

[0114] Additionally, the ratio of the ICD (mm) to the desired working depth (mm) in the surgical scene must be equal to the ratio of the IPD (mm) of the user's eyes to the desired working depth (mm) of the user. In one embodiment, the desired working depth in the surgical scene is equal to the length of the robotic arm, and the user's desired working depth is equal to the arm length of the user or a typical user. In another embodiment, the desired working depth in the surgical scene is equal to a set fraction of the length of the robotic arm, and the user's desired working depth is equal to a similarly set fraction of the arm length of the user or a typical user. If these ratios do not match, the user will experience differences in the degree or strength of depth perception due to different amounts of image parallax (binocular parallax). For example, if the ICD is larger than it should be based on these ratios, the user will experience an unnaturally strong depth perception (from parallax). Additionally, as a user moves an object closer to the camera, the distance the stereoscopic images diverge increases (similar to cross-eyedness when the brain cannot construct a 3D view of the scene, causing the user to experience double vision), meaning that a smaller ICD means that users cannot work comfortably as close to the camera as possible. Conversely, if the ICD is smaller than it should be based on these ratios, the user will experience unnaturally weak depth perception (from parallax). The smaller the ICD, the less depth perception the user will experience from parallax. In theory, an ICD of 0 mm would result in the user experiencing no depth perception from parallax, which would be the same as viewing a scene through a standard 2D television. It should be noted that humans use many cues to give them depth perception, and parallax is only one of them. Other cues include accommodation, vergence, monocular disparity, retinal image size, linear perspective, and gradient.

[0115] Aspects of the subject matter described herein can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including components disclosed herein and structural equivalents thereof, or in combinations of these. Furthermore, aspects of the subject matter described herein can be implemented using one or more computer program products, such as one or more computer programs tangibly embodied in an information medium (e.g., in a machine-readable storage device) or in a propagated signal, for execution by or control of the operation of any suitable data processing device (e.g., a programmable processor, controller, computer, or multiple computers). Computer programs (also known as programs, software, software applications, or code) may be written in any form of programming language, including compiled or interpreted language, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program need not necessarily correspond to a file. A program may be stored in a portion of a file that holds other programs or data, in a single file dedicated to the program, or in multiple linked files (e.g., files containing one or more modules, subprograms, or code portions).

[0116] The processes and logic flows described herein, including the method steps of the subject matter described herein, may be performed by one or more programmable processors executing one or more computer programs that perform the functions of the subject matter described herein by processing input data and generating output. The processes and logic flows may also be embodied as special purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0117] Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. A processor may be used by any component, unit, or module of the surgical virtual reality user interface generation system 10 of the present invention. Generally, a processor receives instructions and data from a suitable memory or storage device, such as, for example, a read-only memory, a random-access memory, or both. Each of the units of the surgical virtual reality user interface generation system 10 may consist of or include one or more computing devices, which may include a processor for executing instructions and a storage device for storing instructions and data. Generally, a computing device may also include, or be operatively connected to, one or more mass storage devices, such as, for example, magnetic disks, magneto-optical disks, or optical disks, for storing data, and may receive, transmit, or both data from or to these devices. Suitable information media for embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices (e.g., EPROMs, EEPROMs, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks, and optical disks (e.g., CDs and DVD disks). The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0118] System 10 may use or include any selected type or form of user input devices and display units. The units or components of system 10 may also communicate over any selected type of network using one or more computing devices, such as client devices or servers, as is known. The network may include any selected type of network, such as a local area network (LAN), a wide area network (WAN), the Internet, etc. System 10 may also communicate over various other communication forms known in the art for communication between disparate parts of a system, such as SPI, USART, UART, USB, I2C, and asynchronous methods.

[0119] FIG. 14 illustrates another embodiment of the surgical virtual reality user interface generation system 10 of the present invention. Like reference numerals represent like parts throughout the various views. In this embodiment, the system 10 uses a head-mounted display 170 that can include a display unit 12 and a first sensing and tracking unit 16A of a pair of sensing and tracking units. The head-mounted display 170 therefore tracks the user's head and eye movements to generate tracking and position data 34A that is transmitted to the VR computing unit 14. The tracking and position data 34A is processed by the VR computing unit 14 and the computing unit 18, and the control unit 26 responsively generates control signals for controlling the operation of the camera assembly 44 of the surgical robotic system 20. The system also uses a second sensing and tracking unit 16B that senses and tracks the position of the user's arms and hands as described above. The sensing and tracking unit 16B generates tracking and position data 34B that is transmitted to and processed by the computing unit 18. In response, the control unit 26 generates control signals to control the movement of the robot arm 42 .

[0120] 15 illustrates yet another embodiment of the surgical virtual reality user interface generation system of the present invention. Like reference numerals represent like parts throughout the various views. In this embodiment, the sensing and tracking unit 16 and the display unit 12 form part of a head-mounted display 170. The head-mounted display 170 therefore tracks the user's head and eye movements to generate tracking and position data 34A that is transmitted to the VR computing unit 14, which then relays the tracking and position data 34A to the computing unit 18 for further processing. In response, the control unit 26 generates control signals for controlling the surgical robot 20.

[0121] The inventive systems, apparatus, methods, and processes of the present disclosure are intended to encompass variations and applications developed using information from the embodiments described herein. Applications and / or modifications of the systems, apparatus, methods, and processes described herein may be practiced by those skilled in the relevant art.

[0122] Throughout the specification, when articles, devices, and systems are described as having, including, or comprising particular components, or when processes and methods are described as having, including, or comprising particular steps, it is contemplated that there are also articles, devices, and systems of the present disclosure that consist essentially of or consist of such components, and that there are processes and methods according to the present disclosure that consist essentially of or consist of such processing steps.

[0123] It should be understood that the order of steps or order for performing certain actions is immaterial so long as the present disclosure remains operable. Moreover, two or more steps or actions may be performed simultaneously. The reference to any publication in the Background of the Invention section is not an admission that such publication is prior art to any claim described herein. The Background of the Invention section is presented for purposes of clarity and is not meant to be a description of prior art to any claim.

[0124] It is to be understood that the subject matter of the present disclosure is not limited to the use of the details of construction and arrangements of parts described above or illustrated in the drawings. The subject matter of the present disclosure may be embodied in other forms and may be practiced and carried out in various ways. It is further to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. Thus, those skilled in the art will appreciate that the conception upon which the present disclosure is based may readily be utilized as a basis for the designing of other structures, methods and systems for carrying out some of the objectives of the subject matter of the present disclosure.

Claims

[Claim 1] 1. A system for providing an augmented telepresence experience in a virtual surgical environment, comprising: a camera assembly for capturing live images and video; A display unit, a spherical image including the live image and video; and displaying augmented information as a virtual overlay provided with the images and video from the camera; a display unit configured to differentiate the virtual overlay from the live image and video within the spherical image; and 1. A virtual reality generation unit, comprising: receiving information data from one or more external data sources; generating the virtual overlay based on the received information data; the virtual overlay includes a plurality of virtual reality information objects associated with a virtual surgical environment; Each of the plurality of information objects is configured to be displayed in the virtual reality world in a free mode in which the virtual reality information object is placed at a selected fixed position, a docked mode in which the virtual reality information object is placed in a docking station, or an attached mode in which the virtual reality information object is placed to follow the user in the virtual reality world. a virtual reality generation unit; 1. A system for providing an augmented telepresence experience in a virtual surgical environment, comprising:

Images