Immersion menu access

Abstract

In examples, a surgical robotic system includes an immersive display; an end effector mounted on a distal end of a robotic arm; and one or more human interface devices (HIDs) for controlling the end effector and the robotic arm. The system also includes a processor configured to: render a surgical site including the end effector in the immersive display; responsive to a single user input, disengage the HIDs from controlling the end effector and the robotic arm; and, responsive to the single user input, overlay a menu on the rendered surgical site in the immersive display.

Description

IMMERSION MENU ACCESS PRIORITY

[0001] This application claims priority to U.S. Provisional Application No. 63 / 551,567, filed February 9, 2024, entitled “IMMERSION MENU ACCESS,” the disclosure of which is incorporated by reference herein, in its entirety. BACKGROUND

[0002] A surgical robotic system may be useful for performing various medical procedures, including both minimally invasive procedures (e.g., laparoscopy) and non-invasive procedures (e.g., endoscopy). The surgical robotic system may include an operating table, robotic arms configured to control the movement of surgical tools, and a physician console through which a medical provider may control the movements of the robotic arms and / or surgical tools. SUMMARY

[0003] In examples, a surgical robotic system includes an immersive display; an end effector mounted on a distal end of a robotic arm; and one or more human interface devices (HIDs) for controlling the end effector and the robotic arm. The system also includes a processor configured to: render a surgical site including the end effector in the immersive display; responsive to a single user input, disengage the HIDs from controlling the end effector and the robotic arm; and overlay a menu on the rendered surgical site in the immersive display.

[0004] In examples, a surgical robotic system comprises an immersive display; an end effector mounted on a distal end of a robotic arm; and one or more human interface devices (HIDs) for controlling the end effector and the robotic arm. The system also comprises a processor configured to: render a menu through the immersive display responsive to receipt of a single, first user signal; modify the rendered menu to identify a selected menu item responsive to receipt of a second user signal; finalize the selected menu item responsive to a third user signal; and perform an action based on the finalized selected menu item.

[0005] The various examples described above can be combined with any other examples described herein. The features and advantages described in the specification are not all inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used in the specification has been principally selected for readabilityand instructional purposes and may not have been selected to delineate or circumscribe the inventive subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG.1 illustrates a surgical robotic system according to some examples.

[0007] FIG. 2 illustrates components of a surgical robotic system in accordance with some examples.

[0008] FIGS. 3A to 3C illustrate various views of a surgical robotic system footboard, in accordance with some examples.

[0009] FIGS. 4A to 4C illustrate various views of a surgical robotic system footboard, in accordance with some examples.

[0010] FIGS. 5A to 5C illustrate various views of a surgical robotic system footboard, in accordance with some examples.

[0011] FIGS. 6A to 6C illustrate various views of a surgical robotic system footboard, in accordance with some examples.

[0012] FIG. 7 illustrates human interface devices (HIDs) of a surgical robotic system, in accordance with some examples.

[0013] FIG. 8 is a block diagram of components of a surgical robotic system, in accordance with some examples.

[0014] FIG.9 is a flow diagram of a process for providing immersion menu access to a medical provider operating a surgical robotic system, in accordance with some examples.

[0015] FIGS. 10A to 10F provide a process flow for providing a menu to a medical provider operating a surgical robotic system without breaking immersion, in accordance with some examples.

[0016] FIG.11A and 11B illustrate immersion menus in accordance with some examples.

[0017] FIGS.12A to 12C illustrate immersion menus in accordance with various examples.

[0018] FIGS.13A to 13C illustrate immersion menus in accordance with various examples.

[0019] FIG.14A illustrates an immersion menu in accordance with various examples.

[0020] FIG.14B illustrates an immersion menu in accordance with various examples.

[0021] FIG.15 illustrates an energy adjustment menu in accordance with various examples.

[0022] FIGS. 16A to 16F illustrate the use of a footboard to operate an immersion menu, in accordance with various examples.

[0023] FIG.17A is a perspective view of a HID useful to render and / or navigate a menu on an immersive display of a headset, in accordance with various examples.

[0024] FIG. 17B is a profile view of a HID useful to render and / or navigate a menu on an immersive display of a headset, in accordance with various examples.

[0025] FIG.18 illustrates a physician console headset, in accordance with various examples. DETAILED DESCRIPTION A. Overview

[0026] As described above, a surgical robotic system may be useful for performing various medical procedures, including both minimally invasive procedures (e.g., laparoscopy) and non- invasive procedures (e.g., endoscopy). The surgical robotic system may include an operating table, robotic arms configured to control the movement of surgical tools, and a physician console through which a medical provider may control the movements of the robotic arms and / or surgical tools.

[0027] The physician console may include a display that is configured to display images (e.g., individual images or video streams) captured by cameras on the robotic arms or surgical tools of the surgical robotic system. These images are useful to medical providers using the surgical robotic system to perform medical procedures. To provide the medical provider with comfort, minimize distractions, and enhance image quality and viewability, the display may include a face rest against which the medical provider rests her head while viewing the display. Together, the display and the face rest form a headset. When the medical provider is viewing the display using the headset during a medical procedure, the medical provider is said to be “immersed” or “in immersion.”

[0028] During a medical procedure, it may be necessary or desirable for an immersed medical provider to adjust a feature of the surgical robotic system. For example, the medical provider may wish to adjust an aspect of the display (e.g., brightness, contrast, hue, saturation, image size or scale, orientation), an aspect of the physician console human interface devices (HIDs) (hand controls) that are configured to control the robotic arms and / or the surgical tools (e.g., the functions of certain actuators (e.g., buttons) on the HIDs, the actions taken responsive to the medical provider grasping the actuators with a certain amount of force, the actions taken responsive to the HIDs being moved or rolled in a certain manner), an aspect of the robotic arms or the surgical tools, and so on. Typically, to adjust such features, the medical provider must access a graphical user interface, such as a menu. But to access such menus, the medical provider is required to break immersion, meaning that the medical provider must disengage from the HIDs, remove her head from the display headset, interact with a separate device (e.g., a menu display or controller on the armrest of the physician console) to adjust the desiredfeatures, reengage with her headset, and reengage with the HIDs. To reengage with the HIDs, the medical provider may be required to confirm that the positions and / or orientations of the HIDs matches the desired positions and / or orientations of the robotic arms and / or the surgical tools. Accessing menus in this manner interrupts surgical workflow and is a time-consuming and potentially dangerous process, as the medical provider is unable to quickly react to a medical emergency while she has broken immersion. In alternative systems, the medical provider may issue voice commands to adjust certain features, but such voice-based systems are vulnerable to miscommunication and do not show the medical provider the full range of options available to her as would be seen in a display-rendered menu.

[0029] This disclosure describes various examples of a surgical robotic system that enables a medical provider using the system to access menus during medical procedures without breaking immersion, thus mitigating the challenges posed by menu access in other surgical robotic systems. An example surgical robotic system includes a robotic arm coupled to an operating table support and adapted to couple to a surgical instrument including a camera. The system also includes a physician console configured to control the robotic arm. The physician console comprises a HID configured to control the robotic arm, a foot actuator (such as a foot pedal), and a display configured to display an image (e.g., a discrete image or an image that is part of a video feed) captured by the camera and display a menu responsive to a single user input (e.g., actuation of a foot actuator or a finger actuator). Such a surgical robotic system provides an efficient and robust way to access menus that enable the medical provider to adjust various features of the system without breaking immersion and raising patient risk. Such a system is superior to voice-based systems because the medical provider is provided with a full range of options from which to select, greater granularity of control over various system features, and greater reliability relative to voice-based commands. B. Robotic System

[0030] FIG. 1 illustrates an exemplary surgical robotic system 200 according to some examples. In some examples, the surgical robotic system 200 is a robotic surgery system. In the example of FIG. 1, the surgical robotic system 200 comprises a patient support platform 202 (e.g., a patient platform, a table, a bed, etc.). The two ends along the length of the patient support platform 202 are respectively referred to as “head” and “leg”. The two sides of the patient support platform 202 are respectively referred to as “left” and “right.” The patient support platform 202 includes a support 204 (e.g., a rigid frame) for the patient support platform 202.

[0031] The surgical robotic system 200 also includes a base 206 for supporting the surgical robotic system 200. The base 206 includes wheels 208 that allow the surgical robotic system 200 to be easily movable or repositionable in a physical environment. In some examples, the wheels 208 are omitted from the surgical robotic system 200 or are retractable, and the base 206 can rest directly on the ground or floor. In some examples, the wheels 208 are replaced with feet.

[0032] The surgical robotic system 200 includes one or more robotic arms 210. In some examples, the robotic arms 210 can be configured to perform robotic medical procedures. Although FIG.1 shows five robotic arms 210, it should be appreciated that the surgical robotic system 200 may include any number of robotic arms, including less than five or six or more.

[0033] The surgical robotic system 200 also includes one or more bars 220 (e.g., adjustable arm support or an adjustable bar) that support the robotic arms 210. Each of the robotic arms 210 is supported on, and movably coupled to, a bar 220, by a respective base joint of the robotic arm. In some examples, bar 220 can provide several degrees of freedom, including lift, lateral translation, tilt, etc. In some examples, each of the robotic arms 210 and / or the adjustable arm supports 220 is also referred to as a respective kinematic chain.

[0034] FIG.1 shows three robotic arms 210 supported by the bar 220 that is in the field of view of the figure. The two remaining robotic arms are supported by another bar that is located across the other length of the patient support platform 202.

[0035] In some examples, the adjustable arm supports 220 can be configured to provide a base position for one or more of the robotic arms 210 for a robotic medical procedure. A robotic arm 210 can be positioned relative to the patient support platform 202 by translating the robotic arm 210 along a length of its underlying bar 220 and / or by adjusting a position and / or orientation of the robotic arm 210 via one or more joints and / or links. In some examples, the bar pose can be changed via manual manipulation, teleoperation, and / or power assisted motion.

[0036] In some examples, the adjustable arm support 220 can be translated along a length of the patient support platform 202. In some examples, translation of the bar 220 along a length of the patient support platform 202 causes one or more of the robotic arms 210 supported by the bar 220 to be simultaneously translated with the bar or relative to the bar. In some examples, the bar 220 can be translated while keeping one or more of the robotic arms stationary with respect to the base 206 of the surgical robotic system 200.

[0037] In the example of FIG. 1, the adjustable arm support 220 is located along a length of the patient support platform 202. In some examples, the adjustable arm support 220 may extendacross a partial or full length of the patient support platform 202, and / or across a partial or full width of the patient support platform 202.

[0038] During a robotic medical procedure, one or more of the robotic arms 210 can also be configured to hold medical tools 212 (e.g., robotically controlled medical instruments or tools, such as an endoscope and / or any other instruments (e.g., sensors, illumination instrument, cutting instrument, etc.) that may be used during surgery), and / or be coupled to one or more accessories, including one or more cannulas, in accordance with some examples.

[0039] Although FIG. 1 illustrates a particular configuration of the surgical robotic system integrated with a surgical table or patient platform, in some examples, the surgical robotic system may be configured as a cart-based system, where one or more robotic arms are supported on one or more carts separate from the table.

[0040] In some examples, the surgical robotic system 200 includes a tower 230 (e.g., tower viewer) or a physician console 240 (or both), as illustrated in FIG. 2. The tower 230 may provide support for controls, electronics, fluidics, optics, sensors, and / or power for the patient support platform 202 and the physician console 240. In some examples, the tower 230 includes a display device 232. The display device 232 can include a user interface for displaying a surgical view obtained by one or more cameras of the surgical robotic system and / or one or more notifications to an operator of the surgical robotic system 200. In some examples, the physician console 240 can include a headset 242 having a user interface used by the physician operator for operating the patient support platform 202. For example, the headset 242 may include a user interface for displaying a surgical view obtained by one or more cameras of the surgical robotic system and / or one or more notifications to an operator of the surgical robotic system 200. The physician console 240 can provide both robotic controls and pre-operative and real-time information of a medical procedure to a physician operator. In some examples, the physician console 240 includes one or more input devices (e.g., buttons, switches, touch- sensitive surfaces, gimbals, etc.), such as a foot actuator 244. In some examples, the physician console 240 includes one or more haptic interface devices (HID) that provide force and tactile feedback to a user as the user interacts with the physician console 240. C. Footboards

[0041] FIGS. 3A-3C illustrate various views of a surgical robotic system footboard, in accordance with examples. In particular, FIG. 3A is a top-down view of an example surgical robotic system footboard 1000, such as the footboard including the foot actuator 244 of FIG. 2. The footboard 1000 includes foot actuators 244, and the footboard 1000 also includes a foot actuator 1002. The foot actuator 1002 may perform a different function than any of the footactuators 244. In some instances, the function that the foot actuator 1002 performs may be relevant to patient safety. For example, the foot actuator 1002 may be dedicated to on-display menu access, meaning that actuating the foot actuator 1002 (e.g., by depressing or releasing the foot actuator 1002) may cause a menu to appear or disappear on a display of the headset 242. When the menu appears on the display of the headset 242, the robotic arms 210 and / or medical tools 212 may disengage from HIDs (hand controls) of the physician console 240. Accidentally actuating an unintended foot actuator 244 or foot actuator 1002 during a medical procedure can raise patient risk.

[0042] Accordingly, one or more features of the foot actuator 1002 may make the foot actuator 1002 easily distinguishable from the foot actuators 244 to prevent the medical provider from inadvertently interacting with the foot actuator 1002 when the medical provider intended to interact with one of the foot actuators 244, or from inadvertently interacting with one of the foot actuators 244 when the medical provider intended to interact with the foot actuator 1002. Such distinguishing features may include the placement of the foot actuator 1002 on the footboard 1000, the size of the foot actuator 1002 relative to the foot actuators 244, the shape of the foot actuator 1002 relative to the foot actuators 244, the style or type of foot actuator 1002 relative to the foot actuators 244, and so on. In the example of FIG.3A, the foot actuator 1002 may be of the same type and style as the foot actuators 244, but the foot actuator 1002 has a dedicated location, such as in the top-right corner of the footboard 1000. In such examples, the medical provider can be certain that the top-right most item on the footboard 1000 is the foot actuator 1002, and thus the medical provider can be certain about the consequences of actuating the foot actuator 1002 during a medical procedure. Dedicated locations for the foot actuator 1002 are not restricted to the top-right corner of the footboard 1000, and the scope of this disclosure contemplates dedicated locations being in any of a variety of locations on the footboard 1000.

[0043] When pressure is applied to the foot actuator 1002, the foot actuator 1002 transitions from an elevated state to a depressed state. In some examples, when the pressure is relieved, the foot actuator 1002 automatically releases and transitions from the depressed state back to the elevated state. In other examples, when the pressure is relieved, the foot actuator 1002 remains in a depressed state unless and until pressure is applied again to the foot actuator 1002, whereupon the foot actuator 1002 is released and transitions from the depressed state back to the elevated state. FIG.3B is a profile view of the footboard 1000 of FIG.3A, and FIG.3C is a perspective view of the footboard 1000 of FIG.3A.

[0044] FIGS. 4A-4C illustrate various views of a surgical robotic system footboard, in accordance with some examples. Specifically, FIG. 4A is a top-down view of an example footboard 1000 on which a foot actuator 1004 is of a different style and type than the foot actuators 244. (Style and type refer to features such as mechanical principle of operation, size, shape, texture, etc.) As shown, the foot actuator 1004, like the foot actuators 244, is a type of foot pedal, but the size and shape of the foot actuator 1004 is distinguishable from those of the foot actuators 244. The scope of this disclosure is not limited to any particular style and type of foot actuator. Further, in examples, the foot actuator 1004 has a dedicated location in the top-right corner of the footboard 1000, but the style and type of the foot actuator 1004 distinguish the foot actuator 1004 from the foot actuators 244 even if the foot actuator 1004 does not have a dedicated location on the footboard 1000. When pressure is applied to the foot actuator 1004, the foot actuator 1004 transitions from an elevated state to a depressed state. In some examples, when the pressure is relieved, the foot actuator 1004 automatically releases and transitions from the depressed state back to the elevated state. In other examples, when the pressure is relieved, the foot actuator 1004 remains in a depressed state unless and until pressure is applied again to the foot actuator 1004, whereupon the foot actuator 1004 is released and transitions from the depressed state back to the elevated state. FIG.4B is a profile view of the footboard 1000 of FIG. 4A, and FIG. 4C is a perspective view of the footboard 1000 of FIG. 4A.

[0045] FIGS. 5A-5C illustrate various views of a surgical robotic system footboard, in accordance with some examples. Specifically, FIG. 5A is a top-down view of an example footboard 1000 on which a foot actuator 1006 is of a different style and type than the foot actuators 244. In particular, the foot actuator 1006 is a spherical trackball, meaning that the foot actuator 1006 is positioned so that the top surface of the footboard 1000 is above the equatorial line (e.g., the horizontal plane with the widest diameter) of the foot actuator 1006. In this way, the footboard 1000 at least partially obscures the foot actuator 1006. The interior of the footboard 1000 includes multiple bearings to detect movements of the trackball in various directions, and these detected movements are provided to a controller (such as the controller described with respect to FIG. 8) in the form of electrical signals for interpretation and further processing. In examples, the foot actuator 1006 is textured with grips 1008 to increase traction with the medical provider’s foot, thus improving safety. The grips 1008 also help the medical provider distinguish the foot actuator 1006 from the foot actuators 244, as do the spherical shape and trackball-style movement of the foot actuator 1006. When pressure is applied to the foot actuator 1006, the foot actuator 1006 transitions from an elevated state to adepressed state. In some examples, when the pressure is relieved, the foot actuator 1006 automatically releases and transitions from the depressed state back to the elevated state. In other examples, when the pressure is relieved, the foot actuator 1006 remains in a depressed state unless and until pressure is applied again to the foot actuator 1006, whereupon the foot actuator 1006 is released and transitions from the depressed state back to the elevated state. Regardless of whether the foot actuator 1006 is in a depressed state or elevated state, the foot actuator 1006 may be rolled in any direction, for example to navigate a menu displayed on a display of the headset 242. FIG. 5B is a profile view of the footboard 1000 of FIG. 5A, and FIG.5C is a perspective view of the footboard 1000 of FIG.5A.

[0046] FIGS. 6A-6C illustrate various views of a surgical robotic system footboard, in accordance with some examples. Specifically, FIG. 6A is a top-down view of an example footboard 1000 on which a foot actuator 1010 is positioned on a side surface 1012 of the footboard 1000. The foot actuator 1010 may have the style and type of a foot pedal (e.g., the foot actuator 1004 in FIG.4A), a trackball (e.g., foot actuator 1006 in FIG.5A), a button, etc. In the particular example of FIG. 6A, the foot actuator 1010 is depicted as a button, but the scope of this disclosure is not limited as such. When pressure is applied to the foot actuator 1010, the foot actuator 1010 transitions from an elevated state to a depressed state. In some examples, when the pressure is relieved, the foot actuator 1010 automatically releases and transitions from the depressed state back to the elevated state. In other examples, when the pressure is relieved, the foot actuator 1010 remains in a depressed state unless and until pressure is applied again to the foot actuator 1010, whereupon the foot actuator 1010 is released and transitions from the depressed state back to the elevated state. Pressure may be applied to the foot actuator 1010, for instance, using a right lateral side of the right foot or the right lateral side of the left foot. FIG.6B is a profile view of the footboard 1000 of FIG.6A, and FIG.6C is a perspective view of the footboard 1000 of FIG.6A. D. Hand controls

[0047] FIG.7 illustrates HIDs of a surgical robotic system, in accordance with some examples. In particular, FIG.7 depicts HIDs 1400, which, as mentioned above, may be hand controls that a medical provider uses to operate the surgical robotic system. The HIDs 1400 include finger actuators 1402, 1404, and 1406. The finger actuators 1402, 1404, and 1406 may include depressible buttons, pressure-sensitive surfaces, optical sensors, or any other suitable devices configured to detect manual inputs. In examples, the finger actuators 1402 are configured to cause the surgical robotic system to perform the same function. For instance, the finger actuators 1402 may be clutches which, when actuated, disengage the HIDs 1400 from therobotic arms. When depressed, finger actuator 1404 may cause the surgical robotic system to perform a different function than that effected by the finger actuators 1402 and 1406. Similarly, when depressed, finger actuator 1406 may cause the surgical robotic system to perform a different function than that effected by the finger actuators 1402 and 1404. Example functions caused by the finger actuators 1404 and 1406 may include the display of a menu on the headset (e.g., headset 242) of the surgical robotic system. In some examples, display of the menu requires simultaneous actuation of both finger actuators 1404 and 1406. In examples, display of the menu requires at least one of the finger actuators 1404 or 1406 to be actuated. In examples, the menu is displayed and the HIDs 1400 are disengaged from the robotic arms so long as the finger actuator(s) remains depressed, and upon release, the menu is no longer displayed and the HIDs 1400 are reengaged with the robotic arms. In other examples, the menu is displayed after a single actuation of the finger actuator(s) and continues to be displayed until a second actuation of the finger actuator(s).

[0048] While a menu is displayed, one or both of the HIDs 1400 may be used to navigate the menu. For instance, the HIDs 1400 may be rolled to the left or to the right, up or down, in any diagonal direction, etc. Rolling HIDs 1400 in this manner causes different items in a menu to be selected in a sequential fashion (e.g., by a colored rectangle or other identifier encircling or highlighting the selected menu item), and once a desired menu item is selected, a finger actuator 1402, 1404, or 1406 may be pressed to finalize the selected menu item. Alternatively, one or both of the HIDs 1400 may be grasped by applying pressure to the HID(s) 1400, thus causing the selected menu item to be finalized. Menu navigation is described in greater detail below with respect to the drawings. E. Physician console

[0049] FIG. 8 is a block diagram of components of a physician console 240, in accordance with various examples. The physician console 240 includes a controller 1500, such as a microprocessor or embedded controller. The controller 1500 is coupled to a bus 1502. The bus 1502, in turn, is coupled to robotic arms and surgical tools 1504, audio equipment 1506 (e.g., microphone, speaker), memory 1508 (e.g., random access memory, read-only memory) storing executable code 1510 executable by the controller 1500, the headset 242, the footboard 1000, the HIDs 1400, and miscellaneous equipment 1512. The controller 1500 communicates with the various components coupled to the bus 1502 by way of the bus 1502. Execution of the executable code 1510 by the controller 1500 causes the controller 1500 to perform one or more of the actions attributed herein to the controller 1500.

[0050] In examples, the controller 1500 may receive inputs from various foot actuators (e.g., foot pedals) and / or from finger actuators on the HIDs 1400 and may take one or more actions responsive to such inputs. For instance, actuating a foot actuator and / or finger actuator of any type in any of the ways described above may cause the controller 1500 to render a menu on the display of the headset 242. In examples, the controller 1500 renders a menu on the display of the headset 242 as long as the foot and / or finger actuator is actuated. In examples, the controller 1500 renders a menu on the display of the headset 242 once the foot and / or finger actuator is actuated and continues rendering the menu until the foot and / or finger actuator is actuated again. In examples, actuating a foot and / or finger actuator causes the actuator to remain depressed, and the controller 1500 renders the menu on the display of the headset 242 so long as the actuator remains depressed (e.g., until the actuator is released by another actuation, at which point the controller 1500 ceases to render the menu). In examples, actuating a predetermined combination of foot and / or finger actuators (e.g., simultaneous actuation of two finger actuators, simultaneous actuation of a finger and a foot actuator, simultaneous actuation of multiple finger and multiple foot actuators, simultaneous activation of multiple foot actuators, sequential activation of finger actuators, foot actuators, or a combination thereof) causes the controller 1500 to render the menu or to cease rendering the menu on the headset 242.

[0051] The controller 1500 is configured to control the robotic arms and surgical tools 1504 in accordance with input received from the footboard 1000 and / or the HIDs 1400. The robotic arms may have end effectors mounted thereto. In examples, actuation of a clutch (e.g., the finger actuators 1402) or of a foot actuator in any of the ways described herein may cause the controller 1500 to disengage the HIDs 1400 from the robotic arms and surgical tools 1504, for instance, by ceasing to transmit input signals from the HIDs 1400 to the robotic arms and surgical tools 1504. Further, the controller 1500 may be configured to perform various actions responsive to audio inputs received from the audio equipment 1506, such as oral commands issued by the medical provider operating the physician console 240 (e.g., to disengage the robotic arms and surgical tools 1504 from the HIDs 1400, to display a menu on the headset 242, to select and / or finalize particular menu items, etc.). F. Immersion menu display and navigation

[0052] FIG.9 is a flow diagram of a method 1600 for providing a menu to a medical provider operating a surgical robotic system without breaking immersion, in accordance with some examples. FIGS.10A-10E provide a process flow for providing a menu to a medical provider operating a surgical robotic system without breaking immersion, in accordance with someexamples. Accordingly, the method 1600 is described in parallel with the process flow of FIGS. 10A-10E. The method 1600 begins with a controller (e.g., the controller 1500) receiving images (e.g. static images, or a series of images forming a video stream) from one or more cameras on a surgical tool (1602). For example, as a medical provider performs a surgical procedure using the robotic surgical system, cameras capture images of soft tissue and provide such images to the controller. The method 1600 includes the controller rendering the images on a headset display (1604). For example, the controller receives the soft tissue images captured by the cameras and renders the images on a display of a physician console headset (e.g., headset 242), such as in the form of a live video stream. FIG. 10A depicts an example view provided by the headset 242, which includes an image 1700 (e.g., an image of a live video stream) and a dashboard 1702 providing information and / or control options that may be useful to the medical provider during a surgical procedure.

[0053] The method 1600 continues with the controller receiving a menu access signal from a foot or finger actuator indicating that the medical provider has issued a menu access request (1606). As described above, the medical provider may actuate one or more foot and / or finger actuators in any of the ways described herein and in any sequence and / or combination to command the controller to render a menu on the headset 242. Responsive to receiving this menu access signal, the controller disengages the HIDs from the robotic arms and surgical tools (e.g., robotic arms and surgical tools 1504) (1608) and displays a menu on the headset display (1610). Optionally, in examples, responsive to receiving the menu access signal, the controller also may maintain gimbals associated with the HIDs in a static state, such that the medical provider is able to freely manipulate the HIDs for menu navigation and, upon completing such menu navigation, is able to reengage the HIDs with the robotic arms and surgical tools (e.g., by releasing a foot actuator) without being required to confirm that the positions and / or orientations of the HIDs match the desired positions and / or orientations of the robotic arms and / or the surgical tools (i.e., the medical provider is not required to go through the tedious process of “matching grips”). The controller may render the menu and the captured camera images such that the captured images are overlaid with the menu. In some examples, the controller is configured to take no action upon actuation of a foot and / or finger actuator other than rendering the menu. The menu access signal may be generated in response to a single user input, such as a single actuation of a finger actuator or a foot actuator, a single verbal command captured by a microphone, etc.

[0054] FIG.10B shows the view provided by the headset 242, which includes the image 1700 overlaid with a menu 1704. In this particular example, the menu 1704 provides the medicalprovider with options pertaining to illumination, scope orientation, image capture, video recording, and picture-in-picture (PIP) views, although the scope of this disclosure is not limited to menus providing any particular set or combination of options. Other options (e.g., fluorescence, telestration, instrument settings, file selection for PIP viewing) are contemplated and are non-limiting examples that may be included in the scope of this disclosure. For example, in some instances a fluorescence or near infrared (NIR) imaging setting may be presented in the menu 1704. Such a setting may provide an option to toggle indocyanine green (ICG) fluorescence imaging on or off, or change an overlay visualization of how detected ICG die is presented on the image (e.g., color visualization, illuminated overlay onto a grayscale image, and the like). In some examples, the menu may include an option to adjust energy settings and / or adjust intensity levels for an energy delivery instrument (e.g., RF energy, ultrasonic energy, etc.). In some examples, the menus may include an option to adjust a digital zoom level. In some examples, the menu 1704 is transparent or translucent such that the medical provider may still view at least part of the image 1700 through the menu 1704.

[0055] The method 1600 includes the controller navigating the menu responsive to foot actuator, finger actuator, and / or HID navigation signals (1612). For example, when the menu is rendered, the medical provider may navigate the menu by scrolling through the various menu items. To navigate the menu, the medical provider may repeatedly actuate a foot and / or finger actuator, with each actuation causing a different menu item to be selected. Alternatively, the medical provider may pull or push either of the HIDs to navigate the menu, and may use one of the foot and / or finger actuators to finalize a selected menu item. In some examples, one or both HIDs may be sensitive to pressure, and thus more forcefully grasping the HID(s) causes a desired menu item to be finalized. The finalization of a menu item causes the controller to perform an action, such as with a hardware device of the surgical robotic system (e.g., flipping the scope), or by a software aspect of the surgical robotic system (e.g., displaying a sub-menu, displaying analog controls to adjust brightness or another aspect of the system, displaying a picture-in-picture).

[0056] The HIDs may be pushed and / or pulled in different ways to navigate the menu. In some examples, a HID may be rolled in any suitable direction to navigate the menu and grasped to finalize a menu item selection. In such examples, every degree of freedom of the HIDs may be deactivated except rolling and grasping. In some examples, each HID may be moved in two directions along a fixed axis and over a set distance (e.g., 2 cm to the right to move forward in the menu, and 2 cm to the left to move back in the menu) to navigate the menu and grasped to finalize a menu item selection. In some examples, each HID may be moved in four directionsover a set distance on two axes that are perpendicular to each other and lie in the same plane to navigate menus that are more complex (e.g., menus rendered in a grid pattern). Thus, in such examples, moving a HID 2 cm to the right, 2 cm to the left, 2 cm up, or 2 cm down will result in different menu item selections. Grasping results in the controller finalizing a selected menu item.

[0057] In some examples, a HID can be moved freely along an axis over a set distance, such that the distance the HID travels is scaled to match the range of the menu. For example, moving the HID a large distance along the axis causes several menu items to be traversed, while moving the HID a short distance along the axis causes fewer menu items to be traversed. Grasping results in the controller finalizing a selected menu item. In some examples, a HID may be moved freely in a fixed plane, where the HID motion is projected onto an axis of a set distance in that plane whereby sections of the axis map to menu items that can be selected. Because the menu items are discrete, specific sections of the axis map to specific menu items. In some examples, a HID may be moved freely in a fixed plane where specific square sections of the plane map to a menu that is rendered in a grid-like format. For instance, navigating the HID toward the top and left causes a top-left menu item to be selected from a menu item grid, while navigating the HID toward the bottom and right causes a bottom-right menu item to be selected from the menu item grid.

[0058] In some examples, a foot actuator, such as the foot actuator 1006 (e.g., a spherical trackball), may be useful to navigate the menu, and the foot actuator may be actuated to finalize a selected menu item. When a spherical actuator is used to navigate a menu, the free, three- dimensional motion of the actuator corresponds to the menu items that are selected. For example, the diameter of the sphere may be scaled to the quantity of the menu items that can be selected, such that all menu items can be selected using the sphere.

[0059] FIGS. 10C-10E depict menu navigation as described above. For instance, FIG. 10C shows an illumination menu item 1706 having been selected. The selected item is identified by a white border encircling the item, and as the medical provider navigates the menu, different menu items are selected (encircled by the white border), demonstrating menu navigation. Identifiers other than white borders are contemplated and included in the scope of this disclosure. FIG.10D shows a flip-scope menu item 1708 having been selected. FIG.10E shows the illumination menu item 1706 having been selected and finalized, as the blue circle indicates. The blue circle differs from the white border to distinguish between menu items that have been selected and selected menu items that have been finalized for adjustment. Because the illumination menu item 1706 has been finalized, the medical provider is provided with ananalog control bar 1710 by which she may adjust the brightness of the display in the headset 242. In other situations, finalizing a menu item may cause the controller to display a sub-menu for further navigation. In examples, the controller may provide the medical provider with audio feedback (e.g., by way of the audio equipment 1506) and / or haptic feedback (e.g., by way of the HIDs 1400) responsive to any or all types of menu interaction, such as accessing the menu or a sub-menu, selecting a menu item, finalizing the menu item, etc. The haptic feedback may be enabled or disabled by a medical provider or other user at any time (e.g., by navigating an appropriate menu as described herein).

[0060] Although FIGS. 10B-10E depict the menu 1704 as being rendered on the left side of the display, in some examples, the menu 1704 may be rendered elsewhere on the display. For instance, the menu 1704 may be rendered along a top, right, or bottom edge of the display. However, it may be useful to render the menu 1704 in an area of the display that would not require the menu 1704 to be scaled down in size (and, thus, result in diminished readability) when the PIP option is finalized. FIG.10F shows the menu 1704 in such a location. Although PIP images 1712 are displayed, the menu 1704 is not reduced in size and thus maintains readability.

[0061] The scope of disclosure is not limited to the particular style of the menu 1704. For example, the controller may render a carousel-style menu, which is particularly useful to provide the medical provider with numerous menu items without having to scale down the menu size and thus avoids the use of excessively small text font. FIG.11A shows an example carousel menu 1800 having a selected menu item 1802 and non-selected menu items 1804. The selected menu item 1802 is shown as being centered and enlarged relative to the non-selected menu items 1804, which are non-centered and are relatively small compared to the selected menu item 1802. As the medical provider navigates through the carousel menu 1800, new items appear on one end of the menu 1800 and previously displayed items disappear on the opposite end of the menu 1800. The carousel menu 1800 may be rendered elsewhere, such as in a corner of the display, as FIG.11B shows.

[0062] The method 1600 includes the controller receiving a menu closure signal from an actuator (1614), as described above (e.g., by releasing a depressed foot actuator, by relieving pressure applied to keep a foot actuator depressed, etc.). Responsive to receiving a menu closure signal, the controller ceases to render the menu (1616), and the controller further reengages the HIDs 1400 with the robotic arms and surgical tools 1504 (1618). Optionally, in the event that the gimbals were maintained in a static state as described above, the gimbals may be released. In some examples, steps 1614, 1616, and 1618 may be performed responsive to asingle action by the medical provider. For instance, the medical provider may be navigating a rendered menu and simultaneously monitoring a live video feed that is continuously rendered in conjunction with the menu on the display. Upon determining that an emergency condition exists (e.g., bleeding), the medical provider may need to quickly exit the menu, reengage the HIDs 1400 with the robotic arms and surgical tools 1504, and resume surgery. Accordingly, to provide such a quick-exit option, the controller may perform steps 1614, 1616, and 1618 responsive to actuation of a predetermined foot and / or finger actuator(s) in a particular sequence and / or combination. In some examples, a foot actuator may remain depressed upon a first actuation and, while the foot actuator remains depressed, the controller renders a menu on the display of the headset 242. Upon a second actuation, the foot actuator may be released, and upon the foot actuator being released, the controller may immediately perform steps 1616 and 1618. In some examples, after steps 1614, 1616, and 1618, actuation of a designated foot actuator may cause the activation of the most recently selected and finalized menu item. For example, if the most recently selected and finalized menu item was the capture of an image, actuation of this designated foot actuator after steps 1614, 1616, and 1618 may cause the capture of an image. In some examples, the activation of such a feature may depend on the speed and / or force with which the foot actuator is actuated, with a threshold speed and / or force being required to result in the activation of the most recently selected and finalized menu item.

[0063] In some examples, the medical provider may be provided the option of confirming the HIDs 1400 and the tips of the surgical tools 1504 have matching orientations prior to the controller reengaging the HIDs 1400 with the robotic arms and surgical tools 1504. Absent such matching orientations, the medical provider may be provided the option of orienting the HIDs 1400 so the HIDs 1400 and tips of the surgical tools 1504 have matching orientations prior to the controller reengaging the HIDs 1400 with the robotic arms and surgical tools 1504.

[0064] FIGS. 12A-12C illustrate immersion menus in accordance with various examples. In particular, FIG.12A depicts an image rendered on a display of the headset 242. A menu 1900 is rendered on the image. The menu 1900 may be rendered responsive to the actuation of one or more actuators in any sequence or combination, as described in detail above. The menu 1900 may be sized and shaped to obscure as little of the underlying image as possible. The menu includes menu items 1902, 1904, 1906, and 1908 rendered on a shaded background 1910. For example, the menu item 1902 may facilitate image captures using a camera; menu item 1904 may facilitate a video recording using the camera; menu item 1906 may facilitate the “flip scope” feature; and menu item 1908 may facilitate a PIP feature. A medical provider may navigate the menu 1900 using one or more actuators as described in detail above. The menu1900 may be rendered on different areas of the image, including the top of the image, the mid- section of the image, and the bottom of the image, the latter two of which are shown in FIGS. 12B and 12C, respectively. The menu 1900 may be depicted in another areas of the image not expressly shown. The menu 1900 may be oriented vertically, horizontally, diagonally, or in any other suitable manner. Any and all such features may be controlled using one or more of the actuators described herein.

[0065] FIGS.13A-13C illustrate immersion menus in accordance with various examples. More specifically, FIGS.13A-13C are identical to FIGS.12A-12C, respectively, but FIGS.13A-13C omit the background 1910. Omitting the background 1910 provides the medical provider with a view of additional areas of the underlying image.

[0066] The scope of this disclosure is not limited to menus that include a linear array of menu items. As described above, for example, menu items may be positioned in an arc. FIG. 14A illustrates an arc-style immersion menu 2100 in accordance with various examples. The menu 2100 is shaped as an arc or rotary dial and includes multiple menu items 2102, 2104, 2106, 2108, and 2110 on a shaded background 2112. In examples, the menu item 2102 is useful to control a “flip scope” option; the menu item 2104 is useful to control a PIP option; the menu item 2106 is useful to capture an image using a camera; the menu item 2108 is useful to record video using the camera; and the menu item 2110 is useful to adjust brightness of the image. A medical provider may navigate the menu 2100 using actuators as described above. If the medical provider has selected the right-most menu item on the menu 2100 and navigates further to the right, additional menu item(s) may be displayed and the left-most menu item on the menu 2100 may cease to be displayed. Similarly, if the medical provider has selected the left-most menu item on the menu 2100 and navigates further to the left, additional menu item(s) may be displayed and the right-most menu item on the menu 2100 may cease to be displayed. Although FIG. 14A shows the menu 2100 rendered near the bottom of the underlying image, the menu 2100 may be rendered anywhere on the underlying image.

[0067] FIG.14B illustrates another arc-style immersion menu 2200 in accordance with various examples. The menu 2200 is shaped as an arc and includes multiple menu items 2202, 2204, 2206, 2208, and 2210. In examples, the menu item 2202 is useful to capture an image using a camera; the menu item 2204 is useful to record a video using the camera; the menu item 2206 is useful to adjust the brightness of the image; the menu item 2208 is useful to control a “flip scope” option; and the menu item 2210 is useful to control a PIP option. A medical provider may navigate the menu 2200 using actuators as described above. If the medical provider has selected the top-most menu item on the menu 2200 and navigates further upward, additionalmenu item(s) may be displayed and the bottom-most menu item on the menu 2200 may cease to be displayed. Similarly, if the medical provider has selected the bottom-most menu item on the menu 2200 and navigates further downward, additional menu item(s) may be displayed and the top-most menu item on the menu 2200 may cease to be displayed. Although FIG.15 shows the menu 2200 rendered near the left side of the underlying image, the menu 2200 may be rendered anywhere on the underlying image.

[0068] In some examples, the menu items or icons presented may be dynamic or context sensitive based on the state of the system. The immersion menu may be smaller or larger to present more or fewer available items or icons depending on the types of instruments that are loaded on to the robotic arms. For example, the controller may display the “flip scope” or scope orientation option when the system detects that a 30 degree laparoscope or endoscope (or scope having an angled tip) is mounted to a robotic arm, such that the flip scope option is available to flip or adjust the orientation of the scope. Then, in instances where the system detects that a zero degree laparascope or endoscope (or scope having a non-angled tip) is mounted to the robot, the controller may present the menu without the “flip scope” or scope orientation option, to reflect the state of the system where such an option would not be useful or would not be available to the user. In some examples, the controller may display an energy adjustment item or icon in instances where the system detects that an energy instrument is loaded onto the robot. Then, in instances where only non-energy instruments are loaded to the robot (or no energy instruments are loaded to the robot), the controller may display the menu without such an energy item or icon.

[0069] FIG. 15 illustrates an interface that allows the user to adjust energy in the menu, in accordance with some examples. In particular, FIG.15 illustrates energy setting interface 1599, which can display adjustable and / or interactive energy settings in accordance with some examples. In some examples, the energy setting interface 1599 can be displayed in response to user selection of a energy menu item from a main immersive menu (e.g, menu 1704 from FIGS. 10B where one of the a selectable menu items is an energy menu item. In other examples, the energy setting interface 1599 is itself the primary interface that is displayed upon user activation of the immersive menu.

[0070] Energy settings interface 1599 can provide one or more user interactive elements or icons that allow the user to adjust energy settings (e.g., in accordance with any of the input modalities described herein). For example, energy settings interface 1599 can include an increase item 1597 (e.g., increase button) operable by the user to increase an energy intensity or setting associated with an energy instrument, and the interface can include a decrease item1593 (e.g., a decrease button) operable by the user to decrease an energy intensity or setting associated with the energy instrument. Alternatively, or in combination, the energy settings interface 1599 can be operable to cycle through or change a type of mode of operation of the instrument (e.g,, cutting mode, coagulation mode 1, coagulation mode 2, etc.). In some examples, the energy settings interface 1599 can include a status element 1589 that display a current setting associated with the energy instrument, such as a current energy intensity, a current energy modality, or a combination thereof. In some examples, the energy setting interface 1599 can include a navigation item 1591 (e.g., a navigation button), that allows a user to navigate to a different interface, such as returning to a main menu (e.g., menu 1704 from FIG.10B).

[0071] FIGS. 16A-16F illustrate the use of a footboard to navigate an immersion menu, in accordance with various examples. In particular, FIG. 16A depicts a footboard 2300 that includes foot actuators 2302, 2304, 2306, 2308, 2309, and 2310. The foot actuators of the footboard 2300 may be similar to those described above with reference to footboard 1000. The foot actuators 2302, 2304, 2306, 2308, 2309, and 2310 may perform differing functions. In examples, actuating the foot actuator 2302 may cause a controller, such as the controller 1500 described above, to render a menu on an underlying image of a surgical site. FIG.16B depicts one such menu 2311, which includes menu items 2312, 2314, 2316, 2318, and 2320. The menu item 2312 may be useful to capture an image using a camera; the menu item 2314 may be useful to control a “flip scope” option; the menu item 2316 may be useful to record a video using the camera; the menu item 2318 may be useful to control a PIP option; and the menu item 2320 may be useful to control a brightness of the underlying image.

[0072] Each of the foot actuators on the footboard 2300 may be intuitively assigned to a corresponding menu item. For example, the foot actuators 2304, 2306, 2308, and 2310 form a grid, and the menu items 2312, 2314, 2316, and 2318 also form a grid. Accordingly, the foot actuator 2304 may be useful to control the menu item 2312, since they are both located in the top-left corners of their respective grids (as FIG. 16B shows); the foot actuator 2306 may be useful to control the menu item 2314, since they are both located in the top-right corners of their respective grids (as FIG.16C shows); the foot actuator 2308 may be useful to control the menu item 2316, since they are both located in the bottom-left corners of their respective grids (as FIG.16D shows); and the foot actuator 2310 may be useful to control the menu item 2318, since they are both located in the bottom-right corners of their respective grids (as FIG. 16E shows). Further, the foot actuator 2309 is positioned to the right of the grid of foot actuators on the footboard 2300, and, similarly, the menu item 2320 is positioned to the right of the grid ofmenu items in the menu 2311. Thus, the foot actuator 2309 may be useful to control the menu item 2320, as FIG.16F shows.

[0073] In examples, actuating the foot actuator 2302 causes the menu 2311 to be rendered, and the menu 2311 remains rendered so long as the foot actuator 2302 is continuously depressed by the medical provider, until the foot actuator 2302 is released from a depressed state by the medical provider, or until the foot actuator 2302 is actuated a second time to cause the menu 2311 to cease being rendered. During the time period the menu 2311 is rendered, actuating any of the foot actuators on the footboard 2300 causes a corresponding menu item to be selected, and actuating that same foot actuator a second time causes the selected menu item to be finalized. Other schemes for navigating the menu 2311 are contemplated and included in the scope of this disclosure.

[0074] FIG. 17A is a perspective view of an example HID 2400 useful to render and / or navigate a menu on the immersive display of the headset 242, in accordance with various examples. FIG. 17B is a profile view of the HID 2400 of FIG. 17A. The HIDs 2400 may be examples of the HIDs 1400 (e.g., FIG. 8) described herein. The example HID 2400 includes one or more actuators 2402 coupled to a shaft 2402. In examples, multiple actuators 2402 couple to the shaft 2402 and encircle the shaft 2402. The actuators 2402 may be rotated about the shaft 2402. In some examples, when rotated from a first position to a second position and released, the actuators 2402 may automatically rotate back to the first position. When grasped, the actuators 2402 may collapse onto or toward the shaft 2402, thereby triggering the generation of a user signal that is received by a processor, such as the controller 1500 described herein. The user signal may indicate that the user wishes to finalize a selected menu item, for example. Finger straps 2406 are coupled to the actuators 2402. One or more of the actuators 2402 may include one or more actuators 2408. An actuator 2408 may provide a user signal to the controller 1500 responsive to being retracted and released, at which point the actuator 2408 may automatically return to a non-retracted position. In examples, the actuator 2408 may provide a user signal to the controller 1500 responsive to being retracted, with the actuator 2408 being released after the user signal has been provided to the controller 1500.

[0075] The controller 1500 may be configured to render a menu on an immersive display of the headset 242 responsive to the receipt of a single user signal. For example, the HID 2400 may generate a single user signal when a user actuates the actuators 2402 and / or 2408, or when a user actuates a foot actuator, such as those included on the footboard 2300 described herein. The menu may be similar to those described above, such as menu 1704 (FIGS.10B-10E), with multiple menu items, such as menu items 1706, 1708, for example. In examples, the controller1500 is configured to disengage the HID 2400 from the robotic arms, such as the robotic arms and surgical tools 1504 (FIG. 8), responsive to receipt of the single user signal, such that movement of the HID 2400 does not cause movement of any robotic arm. Actuation of one of the actuators 2402, 2408 or a foot actuator on footboard 2300 may generate another user signal, which may, in turn, cause the controller 1500 to modify the rendered menu 1704 to identify a selected menu item. As already described, a selected menu item is a menu item that is set apart from other menu items in some way, such as by a cursor pointing to the menu item, a colored or glowing indicator encircling the menu item, etc. Another user signal may be generated and provided to the controller 1500 responsive to actuation of one of the actuators 2402, 2408 or a foot actuator on footboard 2300, upon which the controller 1500 again modifies the rendered menu 1704 to finalize the selected menu item and performs an action based on the finalized, selected menu item. For example, if the illumination menu item 1706 is selected and is subsequently finalized, a sub-menu pertaining to illumination may be displayed, or a selected illumination level may be implemented by brightening or dimming the immersive display. The controller 1500 may subsequently receive another user signal generated by the actuator 2402, 2408 or a foot actuator on footboard 2300, and responsive to this user signal, the controller 1500 may cease to render the menu 1704 on the immersive display of the headset 242, and the controller 1500 may likewise cease to render any sub-menus or related items on the immersive display.

[0076] The above-described actuations by the actuators 2402, 2408 and / or the foot actuator(s) on footboard 2300 may be performed by any other suitable actuators. However, in the event the HID 2400 is to be used to navigate a menu such as the menu 1704, the following examples describe specific ways in which the HID 2400 may be operated to navigate the menu. In some examples, the actuators 2402 are capable of being rotated about the shaft 2404. As the actuators 2402 are rotated, the controller 1500 modifies the rendered menu 1704 so that the menu item that is selected changes. The result is a “scrolling” effect, in which a cursor, highlighter, etc. visually appears to scroll through the various menu items in the menu 1704. When a target menu item is selected, the selection may be finalized responsive to the user grasping the actuators 2402 so the actuators 2402 collapse toward the shaft 2404. In examples, when a target menu item is selected, the selection may be finalized responsive to the user retracting and / or releasing the actuator 2408. In some examples, so long as the actuators 2402 are rotated a minimum threshold amount, the rotation produces the same result irrespective of the degree of rotation, which is the controller 1500 modifying the rendered menu 1704 to select another menu item. Releasing the medical provider’s grasp causes the actuators 2402 to spring back aboutthe shaft 2404 to their original starting position, at which point another rotation may be performed to further select additional menu items. In examples, the actuators 2402 may be rotated in a different direction about the shaft 2404 to navigate the menu differently (i.e., go backward through the menu). Actuation of the actuators 2402, 2408, and / or a foot actuator on the footboard 2300 may cause a selected menu item to be finalized. In yet other examples, the actuator 2408 may be repeatedly actuated (e.g., retracted and released) to change the menu item that is selected, and the actuators 2402 may be grasped to finalize a selected menu item. In examples, the actuator 2408 may be used to finalize a selected menu item.

[0077] In examples, the controller 1500 is configured to cause an identifier (e.g., a cursor, a highlight, a circle, or other identifier) to scroll through menu items of the menu 1704 in a first direction responsive to the HID 2400 being rotated in a second direction (e.g., clockwise) by a threshold amount and held in place. The controller 1500 is configured to provide haptic feedback to the HID 2400 during the scrolling, for example, to indicate that a previously selected menu item has been deselected and a next menu item has been selected on the rendered menu 1704. The controller 1500 is configured to accelerate the scrolling responsive to further rotation of the HID 2400 in the second direction (e.g., clockwise). For example, the HID 2400 may have been rotated clockwise to cause the controller 1500 to scroll through the menu items in a particular direction, and responsive to the HID 2400 being rotated clockwise even further, the controller 1500 may scroll through the menu items in the same direction, but at a faster rate. The controller 1500 may be configured to decelerate the scrolling responsive to decreased rotation of the HID 2400 in the second direction, for example, by rotation of the HID 2400 in a third direction opposite the second direction (e.g., counter clockwise). The controller 1500 may be configured to cause the identifier to scroll through the menu items of the menu in a fourth direction opposite the first direction responsive to the HID 2400 being rotated in the third direction opposite the second direction and held in place. Releasing the grasp causes the HID 2400 to spring back about the shaft 2404 to a starting position. G. Headset

[0078] FIG. 18 illustrates an example physician console headset 242. In some examples, the headset 242 has a single display that is configured to display the various images (e.g., live video streams) and menus described herein. During immersion, a medical provider may rest her head against the headset 242 and view these images, such as during a surgical procedure. In other examples, the headset 242 has multiple displays 2500 and 2502 that are offset from each other by an opaque area 2504. The separate displays 2500 and 2502 provide the medical provider with stereoscopic vision, thereby providing the medical provider with the ability to perceivedepth. The images rendered and displayed to the medical provider thus appear to be three- dimensional. The added depth perception assists the medical provider in performing surgical procedures. In examples, a menu rendered on the headset 242 may be rendered atop the underlying image of the surgical site, and when viewed with stereoscopic vision, the menu may appear to be closer to the medical provider than the underlying image. H. Alternatives

[0079] The scope of this disclosure includes every possible combination of any or all of the following examples.

[0080] In examples, a surgical robotic system comprises a robotic arm coupled to an operating table support and adapted to couple to a surgical instrument including a camera. The system also includes a physician console configured to control the robotic arm. The physician console comprises a hand control configured to control the robotic arm; a foot actuator; and a display. The display is configured to display an image captured by the camera, and display a menu responsive to the foot actuator being actuated.

[0081] In examples, the display is configured to overlay the image with the menu.

[0082] In examples, the foot actuator is configured to transition from an elevated state to a depressed state responsive to a first actuation, remain in a depressed state irrespective of a duration of the first actuation, and transition from the depressed state to the elevated state responsive to a second actuation.

[0083] In examples, the display is configured to display a live video stream from the camera while the foot actuator is in the depressed state.

[0084] In examples, the foot actuator is spherical, includes a foot traction grip, and is at least partially concealed by a housing. The physician console is configured to navigate the menu according to a rolling motion of the foot actuator.

[0085] In examples, the physician console includes a controller configured to disengage the hand control from the robotic arm responsive to the foot actuator being actuated and the menu being displayed.

[0086] In examples, while the hand control is disengaged from the robotic arm, the hand control is configured to control navigation of the menu.

[0087] In examples, the physician console is configured to take no action in response to the foot actuator being actuated other than displaying the menu.

[0088] In examples, the physician console is configured to display the menu so long as the foot actuator is held in a depressed state and to not display the menu so long as the foot actuator is in an elevated state.

[0089] In examples, a surgical robotic system comprises a robotic arm coupled to an operating table support and adapted to couple to a surgical instrument including a camera, and a physician console configured to control the robotic arm. The physician console comprises a hand control configured to control the robotic arm, the hand control including a finger actuator and a display configured to display images captured by the camera, and display a menu responsive to the finger actuator being actuated.

[0090] In examples, the hand control includes a second finger actuator configured to disengage the hand control from the robotic arm.

[0091] In examples, the system further comprises a second hand control configured to control a second robotic arm, the second hand control including a second finger actuator, the first and second finger actuators configured to disengage the hand control from the robotic arm. The display is configured to display the menu responsive to the second finger actuator being actuated at the same time that the finger actuator is being actuated.

[0092] In examples, the physician console is configured to navigate the menu according to movement of the hand control.

[0093] In examples, the physician console is configured to disengage the hand control from the robotic arm responsive to the finger actuator being actuated, and to re-engage the hand control and the robotic arm responsive to the finger actuator being released.

[0094] In examples, the physician console includes a foot actuator, and the display is configured to display the menu responsive to simultaneous actuation of the finger and foot actuators.

[0095] In examples, a non-transitory, computer-readable medium stores instructions which, when executed by a processor, cause the processor to receive an image from a camera coupled to a surgical instrument supported by a robotic arm; display the image on a display of a physician console; and display a menu on the display responsive to a hardware actuator being actuated, the image overlaid with the menu.

[0096] In examples, a portion of the menu is transparent.

[0097] In examples, the hardware actuator includes a finger actuator on a hand control of the physician console or a foot actuator of the physician console.

[0098] In examples, execution of the instructions causes the processor to display the menu responsive to the hardware actuator being actuated and to continue displaying the menu so long as the hardware actuator remains actuated, and further to cease displaying the menu responsive to the hardware actuator being released, and, while the menu is displayed, navigate the menuresponsive to movements of a hand control of the physician console or movements of a spherical foot actuator.

[0099] In examples, execution of the instructions causes the processor to, responsive to actuation of the hardware actuator, disengage the robotic arm from a hand control configured to control the robotic arm, and, responsive to release of the hardware actuator, re-engage the robotic arm and the hand control.

[0100] The following examples relate to various non-exhaustive ways in which the teachings herein may be combined or applied. It should be understood that the following examples are not intended to restrict the coverage of any claims that may be presented at any time in this application or in subsequent filings of this application. No disclaimer is intended. The following examples are being provided for nothing more than merely illustrative purposes. It is contemplated that the various teachings herein may be arranged and applied in numerous other ways. It is also contemplated that some variations may omit certain features referred to in the below examples. Therefore, none of the aspects or features referred to below should be deemed critical unless otherwise explicitly indicated as such at a later date by the inventors or by a successor in interest to the inventors. If any claims are presented in this application or in subsequent filings related to this application that include additional features beyond those referred to below, those additional features shall not be presumed to have been added for any reason relating to patentability.

[0101] Example 1: A surgical robotic system, comprising: an immersive display; an end effector mounted on a distal end of a robotic arm; one or more human interface devices (HIDs) for controlling the end effector and the robotic arm; and a processor configured to: render a surgical site including the end effector in the immersive display; responsive to a single user input, disengage the HIDs from controlling the end effector and the robotic arm; and , responsive to the single user input, overlay a menu on the rendered surgical site in the immersive display.

[0102] Example 2: The surgical robotic system of Example 1, further comprising a dedicated foot pedal configured to provide the single user input.

[0103] Example 3: The surgical robotic system of Example 1 or Example 2, further comprising a dedicated button on the one or more HIDs configured to provide the single user input.

[0104] Example 4: The surgical robotic system of any one of Examples 1-3, wherein the single user input includes a voice command detected at a microphone of the one or more HIDs.

[0105] Example 5: The surgical robotic system of any one of Examples 1-4, further comprising a foot actuator configured to provide the single user input, and wherein the foot actuator is further configured to: transition from an elevated state to a depressed state responsive to a first actuation; remain in a depressed state irrespective of a duration of the first actuation; and transition from the depressed state to the elevated state responsive to a second actuation.

[0106] Example 6: The surgical robotic system of any one of Examples 1-5, wherein the immersive display is configured to display a live video stream from a camera while the foot actuator is in the depressed state.

[0107] Example 7: The surgical robotic system of any one of Examples 1-6, further comprising a foot actuator configured to provide the single user input, and wherein the foot actuator is further configured to: transition from an elevated state to a depressed state responsive to an actuation; and automatically return to the elevated state upon release of the actuation.

[0108] Example 8: The surgical robotic system of any one of Examples 1-7, further comprising a foot actuator configured to provide the single user input, and wherein the foot actuator is spherical, includes a foot traction grip, and is at least partially concealed by a housing, the processor configured to navigate the menu according to a rolling motion of the foot actuator.

[0109] Example 9: The surgical robotic system of any one of Examples 1-8, further comprising an actuator configured to provide the single user input, and wherein the processor is configured to disengage a hand control of the robotic arm from the robotic arm responsive to the actuator being actuated and the menu being displayed.

[0110] Example 10: The surgical robotic system of any one of Examples 1-9, wherein, while the hand control is disengaged from the robotic arm, the hand control is configured to control navigation of the menu.

[0111] Example 11: The surgical robotic system of any one of Examples 1-10, wherein the physician console is configured to take no action in response to the foot actuator being actuated other than displaying the menu.

[0112] Example 12: The surgical robotic system of any one of Examples 1-11, further comprising a foot actuator configured to provide the single user input, and wherein the immersive display is configured to display the menu so long as the foot actuator is held in a depressed state and to not display the menu so long as the foot actuator is in an elevated state.

[0113] Example 13: A surgical robotic system, comprising: an immersive display; an end effector mounted on a distal end of a robotic arm; one or more human interface devices (HIDs) for controlling the end effector and the robotic arm; and a processor configured to: render a menu through the immersive display responsive to receipt of a single, first user signal; modify the rendered menu to identify a selected menu item responsive to receipt of a second user signal; finalize the selected menu item responsive to a third user signal; and perform an action based on the finalized selected menu item.

[0114] Example 14: The surgical robotic system of Example 13, wherein the processor is configured to disengage the one or more HIDs from the robotic arm responsive to receipt of the single, first user signal, such that movement of the one or more HIDs does not cause movement of the robotic arm.

[0115] Example 15: The surgical robotic system of Example 13 or Example 14, wherein, to perform the action, the processor is configured to activate a setting for the system.

[0116] Example 16: The surgical robotic system of any one of Examples 13-15, wherein, to perform the action, the processor is configured to render a sub-menu of the menu.

[0117] Example 17: The surgical robotic system of any one of Examples 13-16, wherein the processor is configured to cease rendering the menu responsive to receipt of a single, fourth user signal.

[0118] Example 18: The surgical robotic system of any one of Examples 13-17, wherein the one or more HIDs is capable of being rotated and further includes an actuator, and wherein the processor is configured to modify the rendered menu to identify another selected menu item responsive to rotation of the one or more HIDs.

[0119] Example 19: The surgical robotic system of any one of Examples 13-18, wherein the processor is configured to finalize the selected menu item responsive to a user actuating the actuator by grasping the actuator.

[0120] Example 20: The surgical robotic system of any one of Examples 13-19, wherein the processor is configured to finalize the selected menu item responsive to a user actuating the actuator by retracting and releasing the actuator.

[0121] Example 21: The surgical robotic system of any one of Examples 13-20, wherein the one or more HIDs is configured to automatically return to an initial position after being rotated and released.

[0122] Example 22: The surgical robotic system of any one of Examples 13-21, wherein the processor is configured to modify the rendered menu to identify another selected menu item in the same way irrespective of a degree to which the one or more HIDs is rotated, as long as the one or more HIDs is rotated at least a minimum amount.

[0123] Example 23: The surgical robotic system of any one of Examples 13-22, wherein the one or more HIDs includes first and second actuators, and wherein the processor is configured to modify the rendered menu to identify another selected menu item responsive to a user actuating the first actuator by retracting and releasing the first actuator, and wherein the processor is configured to finalize the selected menu item responsive to a user actuating the second actuator by grasping the second actuator.

[0124] Example 24: The surgical robotic system of any one of Examples 13-23, wherein the one or more HIDs is capable of being rotated, and wherein the processor is configured to: cause an identifier to scroll through menu items of the menu in a first direction responsive to the one or more HIDs being rotated in a second direction by a threshold amount and held in place; provide haptic feedback to the one or more HIDs during the scrolling; accelerate the scrolling responsive to further rotation of the one or more HIDs in the second direction; decelerate the scrolling responsive to rotation of the one or more HIDs in a third direction opposite the second direction; and cause the identifier to scroll through the menu items of the menu in a fourth direction opposite the first direction responsive to the one or more HIDs being further rotated in the third direction and held in place.

[0125] Example 25: The surgical robotic system of any one of Examples 13-24, wherein the one or more HIDs is configured to automatically return to a starting position when released by a user.

[0126] The previous description of the disclosed examples is provided to enable any person skilled in the art to make or use the features described in the present disclosure. Various modifications to these examples will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other examples without departing from the scope of the disclosure. For example, one of ordinary skill in the art will be able to employ a number of corresponding alternative and equivalent structural details, such as equivalent ways of fastening, mounting, coupling, or engaging tool components, equivalent mechanisms for producing particular actuation motions, equivalent ways to render images, and equivalent mechanisms for delivering electrical energy. Thus, the present disclosure is not intended to belimited to the examples shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

CLAIMS The following is claimed:

1. A surgical robotic system, comprising: an immersive display; an end effector mounted on a distal end of a robotic arm; one or more human interface devices (HIDs) for controlling the end effector and the robotic arm; and a processor configured to: render a surgical site including the end effector in the immersive display; responsive to a single user input, disengage the HIDs from controlling the end effector and the robotic arm; and, responsive to the single user input, overlay a menu on the rendered surgical site in the immersive display.

2. The surgical robotic system of claim 1, further comprising a dedicated foot pedal configured to provide the single user input, a dedicated button on the one or more HIDs configured to provide the single user input, or the single user input includes a voice command detected at a microphone of the one or more HIDs.

3. The surgical robotic system of claim 1 or 2, further comprising a foot actuator configured to provide the single user input, and wherein the foot actuator is further configured to: transition from an elevated state to a depressed state responsive to a first actuation; remain in a depressed state irrespective of a duration of the first actuation; andtransition from the depressed state to the elevated state responsive to a second actuation, wherein the immersive display is configured to display a live video stream from a camera while the foot actuator is in the depressed state.

4. The surgical robotic system of claim 1 or 2, further comprising a foot actuator configured to provide the single user input, and wherein the foot actuator is further configured to: transition from an elevated state to a depressed state responsive to an actuation; and automatically return to the elevated state upon release of the actuation.

5. The surgical robotic system of any of claims 1-3, further comprising a foot actuator configured to provide the single user input, and wherein the foot actuator is spherical, includes a foot traction grip, and is at least partially concealed by a housing, the processor configured to navigate the menu according to a rolling motion of the foot actuator.

6. The surgical robotic system of any of claims 1-4, wherein, while the HID is disengaged from controlling the end effector and the robotic arm, the HID is configured to control navigation of the menu.

7. The surgical robotic system of any of claims 1-6, further comprising a foot actuator configured to provide the single user input, and wherein the immersive display is configured to display the menu so long as the foot actuator is held in a depressed state and to not display the menu so long as the foot actuator is in an elevated state.

8. A surgical robotic system, comprising:an immersive display; an end effector mounted on a distal end of a robotic arm; one or more human interface devices (HIDs) for controlling the end effector and the robotic arm; and a processor configured to: render a menu through the immersive display responsive to receipt of a single, first user signal; modify the rendered menu to identify a selected menu item responsive to receipt of a second user signal; finalize the selected menu item responsive to a third user signal; and perform an action based on the finalized selected menu item.

9. The surgical robotic system of claim 8, wherein the processor is configured to disengage the one or more HIDs from the robotic arm responsive to receipt of the single, first user signal, such that movement of the one or more HIDs does not cause movement of the robotic arm.

10. The surgical robotic system of claim 8 or 9, wherein, to perform the action, the processor is configured to activate a setting for the system or render a sub-menu of the menu.

11. The surgical robotic system of any of claims 8-10, wherein the one or more HIDs is capable of being rotated and further includes an actuator, and wherein the processor is configured to modify the rendered menu to identify another selected menu item responsive to rotation of the one or more HIDs.

12. The surgical robotic system of claim 11, wherein the processor is configured to finalize the selected menu item responsive to: a user actuating the actuator by grasping the actuator; or a user actuating the actuator by retracting and releasing the actuator.

13. The surgical robotic system of claim 11 or 12, wherein the one or more HIDs is configured to automatically return to an initial position after being rotated and released, wherein the processor is configured to modify the rendered menu to identify another selected menu item in the same way irrespective of a degree to which the one or more HIDs is rotated, as long as the one or more HIDs is rotated at least a minimum amount.

14. The surgical robotic system of any of claims 10-13, wherein the one or more HIDs includes first and second actuators, and wherein the processor is configured to modify the rendered menu to identify another selected menu item responsive to a user actuating the first actuator by retracting and releasing the first actuator, and wherein the processor is configured to finalize the selected menu item responsive to a user actuating the second actuator by grasping the second actuator.

15. The surgical robotic system of any of claims 10-14, wherein the one or more HIDs is capable of being rotated, and wherein the processor is configured to: cause an identifier to scroll through menu items of the menu in a first direction responsive to the one or more HIDs being rotated in a second direction by a threshold amount and held in place; provide haptic feedback to the one or more HIDs during the scrolling;accelerate the scrolling responsive to further rotation of the one or more HIDs in the second direction; decelerate the scrolling responsive to rotation of the one or more HIDs in a third direction opposite the second direction; and cause the identifier to scroll through the menu items of the menu in a fourth direction opposite the first direction responsive to the one or more HIDs being further rotated in the third direction and held in place.

Images