Object detection and visual feedback system

Abstract

An object detection and visual feedback system includes a button having a selection surface positioned for selection by an object. The ranging sensor has a field of view across a selection surface of the button. A ranging sensor measures a distance of an object within a field of view. The user interface performs a first action when the measured distance is less than or equal to a first threshold distance. When the measured distance is at least a second threshold distance, the user interface performs a second action. The second threshold distance is greater than the first threshold distance. The first threshold distance is at an edge of the button. The second threshold distance is not coextensive with the button and is spaced apart from an edge of the button by a predetermined distance. The predetermined distance is sufficient to mitigate accidental selection of the button by the object.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims the priority and benefit of U.S. Provisional Patent Application No. 63 / 426,594, filed on November 18, 2022, which is incorporated herein by reference in its entirety. Background Art

[0003] Minimally invasive medical techniques are designed to reduce the amount of extraneous tissue damaged during diagnostic or surgical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques can be performed through natural orifices in a patient's anatomy or through one or more surgical incisions. Through these natural orifices or incisions, a clinician can insert medical tools to reach a target tissue location. Minimally invasive medical tools include instruments such as treatment instruments, diagnostic instruments, and surgical instruments. Minimally invasive medical tools can also include imaging instruments (such as endoscopic instruments) that provide a field of view within a patient's anatomy to the user.

[0004] Some minimally invasive medical tools may be robot - assisted, including remotely operated, teleoperated, or otherwise computer - assisted. During a medical procedure, a graphical user interface can be provided to a clinician, which includes an image of a three - dimensional field of view of a patient's anatomy. To enhance the clinician's experience and efficiency, various indicators may be required to provide additional information about medical tools within the field of view, medical tools obscured within the field of view, and components outside the field of view. Summary of the Invention

[0005] A first aspect of the present disclosure includes an object detection and visual feedback system. The system includes a button positioned for an object to select. The button has a selection surface. The system includes a range sensor having a field of view spanning the selection surface of the button. The range sensor is configured to measure the distance of an object within the field of view. The system includes a user interface configured to perform a first action when the distance measured by the range sensor is less than or equal to a first threshold distance. The user interface is also configured to perform a second action when the distance measured by the range sensor is greater than or equal to a second threshold distance.

[0006] In some embodiments of the first aspect of the present disclosure, the second threshold distance is greater than the first threshold distance.

[0007] In any of the above - mentioned embodiments of the first aspect of the present disclosure, the first threshold distance is at the edge of the button.

[0008] In any of the above embodiments of the first aspect of the present disclosure, the second threshold distance does not co - extend with the button and is spaced apart from the edge of the button by a predetermined distance.

[0009] In any of the above embodiments of the first aspect of the present disclosure, the predetermined distance is sufficient to reduce the accidental selection of the button by the object.

[0010] In any of the above embodiments of the first aspect of the present disclosure, the predetermined distance is in the range of 10 mm to 30 mm.

[0011] In any of the above embodiments of the first aspect of the present disclosure, the object is any one of a non - hand limb, a foot, a leg, a knee, a head, an elbow, or an extension from the human anatomy.

[0012] In any of the above embodiments of the first aspect of the present disclosure, the range - finding sensor is any one of the following: a time - of - flight distance sensor, a triangulation distance sensor, an optical distance sensor, an acoustic distance sensor, an inductive distance sensor, a capacitive distance sensor, a photoelectric distance sensor, a camera, an infrared distance sensor, a laser rangefinder, or a light detection and ranging sensor.

[0013] In any of the above embodiments of the first aspect of the present disclosure, the direction of the field of view is parallel to the direction of the distance measured by the range - finding sensor.

[0014] In any of the above embodiments of the first aspect of the present disclosure, the direction of the field of view is orthogonal to the direction of the distance measured by the range - finding sensor.

[0015] In any of the above embodiments of the first aspect of the present disclosure, the system further includes a second button that is positioned for selection by the object. The second button has a second selection surface. The system further includes a second range - finding sensor that has a second field of view across the second selection surface of the second button. The second range - finding sensor is configured to measure the distance of an object within the second field of view. The user interface is configured to perform a third action when the distance measured by the second range - finding sensor is less than or equal to a third threshold distance.

[0016] In any of the above embodiments of the first aspect of the present disclosure, the third action is different from the first action.

[0017] In any of the above embodiments of the first aspect of the present disclosure, the first action is to display an indication of the button.

[0018] In any of the above embodiments of the first aspect of the present disclosure, the second action is to stop displaying the indication of the button.

[0019] In any of the above-described embodiments of the first aspect of the present disclosure, the indication of the button is selected from a group of indications consisting of: an indication of a function to be performed when the button is selected; a change in the intensity of an icon displayed on the user interface; and a graphic of an object relative to the layout of the selection button, the layout including the button.

[0020] In any of the above-described embodiments of the first aspect of the present disclosure, the first action is to emit a first audible alarm, and the second action is to emit a second audible alarm.

[0021] In any of the above-described embodiments of the first aspect of the present disclosure, the first action is to provide a first haptic feedback, and the second action is to provide a second haptic feedback.

[0022] In any of the above-described embodiments of the first aspect of the present disclosure, the first haptic feedback and the second haptic feedback are provided to the hand controller.

[0023] A second aspect of the present disclosure includes a robotic surgical system. The system includes a user interface and a footrest. The footrest includes a button that is positioned for selection by a user's foot. The button has a selection surface. The footrest further includes a range sensor that is positioned to have a field of view that spans the selection surface of the button. The range sensor is configured to measure the distance of an object within the field of view. The user interface is configured to perform a first action associated with the button when the distance measured by the range sensor is less than or equal to a first threshold distance. The user interface is further configured to perform a second action associated with the button when the distance measured by the range sensor is greater than or equal to a second threshold distance.

[0024] In some embodiments of the second aspect of the present disclosure, the second threshold distance is greater than the first threshold distance.

[0025] In any of the above-described embodiments of the second aspect of the present disclosure, the footrest is positioned on the base of the robotic surgical system.

[0026] In any of the above-described embodiments of the second aspect of the present disclosure, the first threshold distance is at or near the edge of the button.

[0027] In any of the above-described embodiments of the second aspect of the present disclosure, the second threshold distance does not co-extend with the button and is spaced apart from the edge of the button by a predetermined distance.

[0028] In any of the above-described embodiments of the second aspect of the present disclosure, the predetermined distance is sufficient to reduce the accidental selection of the button by the user's foot.

[0029] In any of the above-described embodiments of the second aspect of the present disclosure, the predetermined distance is in the range of 10 mm to 30 mm.

[0030] In any of the above embodiments of the second aspect of the present disclosure, the ranging sensor is any one of the following: a time-of-flight distance sensor, a triangulation distance sensor, an optical distance sensor, an acoustic distance sensor, an inductive distance sensor, a capacitive distance sensor, a photoelectric distance sensor, a camera, an infrared distance sensor, a laser rangefinder, or a light detection and ranging sensor.

[0031] In any of the above embodiments of the second aspect of the present disclosure, the direction of the field of view is parallel to the direction of the distance measured by the ranging sensor.

[0032] In any of the above embodiments of the second aspect of the present disclosure, the direction of the field of view is orthogonal to the direction of the distance measured by the ranging sensor.

[0033] In any of the above embodiments of the second aspect of the present disclosure, the footrest further includes a second button that is positioned for selection by the user's foot. The second button includes a second selection surface. The footrest further includes a second ranging sensor that is positioned to have a second field of view across the second selection surface of the second button. The second ranging sensor is configured to measure a second distance of an object within the second field of view. The user interface is further configured to perform a third action associated with the second button when the second distance measured by the second ranging sensor is less than or equal to a third threshold distance.

[0034] In any of the above embodiments of the second aspect of the present disclosure, the third action is different from the first action.

[0035] In any of the above embodiments of the second aspect of the present disclosure, the user interface includes a display.

[0036] In any of the above embodiments of the second aspect of the present disclosure, the system further includes a headrest, wherein the display is incorporated into the headrest.

[0037] In any of the above embodiments of the second aspect of the present disclosure, the display is a stereoscopic display.

[0038] In any of the above embodiments of the second aspect of the present disclosure, the first action is to display an indication of the button.

[0039] In any of the above embodiments of the second aspect of the present disclosure, the second action is to stop displaying the indication of the button.

[0040] In any of the above embodiments of the second aspect of the present disclosure, the indication of the button is selected from a group of indications consisting of: an indication of a function to be performed when the button is selected; a change in the intensity of an icon displayed on the user interface; and a graphic of the user's foot relative to the layout of a plurality of buttons, the layout including the button.

[0041] In any of the above embodiments of the second aspect of the present disclosure, the first action is to emit a first audible alarm, and the second action is to emit a second audible alarm.

[0042] In any of the above embodiments of the second aspect of the present disclosure, the first action is to provide a first haptic feedback, and the second action is to provide a second haptic feedback.

[0043] In any of the above embodiments of the second aspect of the present disclosure, the first haptic feedback and the second haptic feedback are provided to the hand controller.

[0044] A third aspect of the present disclosure includes a method of providing feedback upon detection of an object. The method includes measuring the distance of an object within the field of view of a ranging sensor. The ranging sensor is positioned to have a field of view across a selection surface of a button. The button is positioned for selection by the object. The method includes performing a first action using a user interface when the distance measured by the ranging sensor is less than or equal to a first threshold distance. The method includes performing a second action using the user interface when the distance measured by the ranging sensor is greater than a second threshold distance.

[0045] In various embodiments of the third aspect of the present disclosure, the second threshold distance is greater than the first threshold distance.

[0046] In any of the above embodiments of the third aspect of the present disclosure, the first threshold distance is at the edge of the button.

[0047] In any of the above embodiments of the third aspect of the present disclosure, the second threshold distance does not co - extend with the button and is spaced apart from the edge of the button by a predetermined distance.

[0048] In any of the above embodiments of the third aspect of the present disclosure, the predetermined distance is sufficient to mitigate an accidental selection of the button by the object.

[0049] In any of the above embodiments of the third aspect of the present disclosure, the predetermined distance is in the range of 10 mm to 30 mm.

[0050] In any of the above embodiments of the third aspect of the present disclosure, the object is any one of a non - hand limb, foot, leg, knee, head, elbow, or an extension from the human anatomy.

[0051] In any of the above embodiments of the third aspect of the present disclosure, the ranging sensor is any one of the following: a time - of - flight distance sensor, a triangulation distance sensor, an optical distance sensor, an acoustic distance sensor, an inductive distance sensor, a capacitive distance sensor, a photoelectric distance sensor, a camera, an infrared distance sensor, a laser rangefinder, or a light detection and ranging sensor.

[0052] In any of the above embodiments of the third aspect of the present disclosure, the direction of the field of view is parallel to the direction of the distance measured by the distance measurement sensor.

[0053] In any of the above embodiments of the third aspect of the present disclosure, the direction of the field of view is orthogonal to the direction of the distance measured by the distance measurement sensor.

[0054] In any of the above embodiments of the third aspect of the present disclosure, the method further includes measuring a second distance of a second object within a second field of view of a second distance measurement sensor. The second distance measurement sensor is positioned to have a second field of view across a second selection surface of a second button. The second button is positioned for an object to select. The method further includes performing a third action using a user interface when the second distance measured by the distance measurement sensor is less than or equal to a third threshold distance. The method further includes performing a fourth action using the user interface when the second distance measured by the second distance measurement sensor is greater than a fourth threshold distance. The fourth threshold distance is greater than the third threshold distance.

[0055] In any of the above embodiments of the third aspect of the present disclosure, the third action is different from the first action.

[0056] In any of the above embodiments of the third aspect of the present disclosure, the first action is to display an indication of the button.

[0057] In any of the above embodiments of the third aspect of the present disclosure, the second action is to stop displaying the indication of the button.

[0058] In any of the above embodiments of the third aspect of the present disclosure, the indication of the button is selected from a group consisting of: an indication of a function to be performed when the button is selected; a change in the intensity of an icon displayed on the user interface; and a graphic of the layout of the object relative to the selection button, the layout including the button.

[0059] In any of the above embodiments of the third aspect of the present disclosure, the first action is to emit a first audible alarm, and the second action is to emit a second audible alarm.

[0060] In any of the above embodiments of the third aspect of the present disclosure, the first action is to provide a first haptic feedback, and the second action is to provide a second haptic feedback.

[0061] In any of the above embodiments of the third aspect of the present disclosure, the first haptic feedback and the second haptic feedback are provided to the hand controller.

[0062] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and the claims. Description of the Drawings

[0063] For a more complete understanding of the present disclosure, reference is now made to the following brief description, taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like components.

[0064] Figure 1A is a schematic diagram of a medical system according to aspects of the present disclosure.

[0065] Figure 1B is a perspective view of an assembly according to aspects of the present disclosure.

[0066] Figure 1C is a perspective view of a surgeon's console for a medical system in accordance with aspects of the present disclosure.

[0067] Figure 2 is a perspective view of a user input pad according to some embodiments.

[0068] Figure 3 is a cross-sectional view of a user input pad according to some embodiments.

[0069] Figures 4A - 4F A graphical user interface with icons providing status information about user input devices in a user input tray is shown according to some embodiments.

[0070] Figure 5A , Figure 5B and Figure 5C A graphical user interface with composite indicators providing status information about user input devices associated with on-screen tools is shown in accordance with some embodiments.

[0071] Figure 6A , Figure 6B , Figure 6C and Figure 6D A graphical user interface with composite indicators providing status information about user input devices associated with on-screen tools is shown in accordance with some embodiments.

[0072] Figure 7A , Figure 7B , Figure 7C and Figure 7D A graphical user interface with composite indicators that may conditionally move to remain visible as a component or endoscope generating a field of view is moved is shown in accordance with some embodiments.

[0073] Figure 8A is a flow chart of the operation of a control system according to various embodiments described herein.

[0074] Figure 8B is a flow chart of an example threshold-based hysteresis of a control system according to various implementations described herein.

[0075] Figure 9 is a flowchart of a calibration operation according to various embodiments described herein.

[0076] Figure 10 is a flowchart of user intention determination according to various embodiments described herein.

[0077] Figure 11 Shows an exemplary computer system. Detailed embodiments

[0078] It should be understood at the outset that although illustrative embodiments of one or more embodiments are shown below, the disclosed systems and methods may be implemented using any number of (whether currently known or existing) technologies. The present disclosure should not in any way be limited to the illustrative embodiments, drawings, and technologies shown below, but may be modified within the scope of the appended claims and their full equivalents. The phrase "and / or" is used to indicate that any one or any combination from a list of options may be used. For example, "A, B, and / or C" means "A", or "B", or "C", or "A and B", or "A and C", or "B and C", or "A and B and C".

[0079] In a robot-assisted medical procedure, an endoscopic image of the surgical environment can provide a clinician with a field of view of the patient's anatomy and any medical tools located within the patient's anatomy. Enhancing the endoscopic image with various indicators can allow the clinician to access information while maintaining the field of view. Such indicators can include indicators of components outside the field of view.

[0080] For example, when using an operator input system during a surgical procedure, the surgeon places their head against the viewing module to view the endoscopic view of the patient's internal body cavity. It is challenging to operate the user input device with a non-hand limb (e.g., operate a foot pedal in a pedal tray with the foot) without looking at the user input device. The object presence sensor provides the surgeon with UI indications (e.g., display, audio, or tactile feedback) of which pedal their foot is positioned on, such that the surgeon does not need to observe their foot or use tactile feedback to identify the orientation of their foot before using the pedal during the surgical procedure. This feature is designed to save the surgeon's time, reduce the surgeon's task switching and cognitive load, and help reduce the likelihood of accidental pedal presses due to inaccurate foot placement during pedal use.

[0081] Figure 1A , Figure 1B and Figure 1CCollectively provide an overview of a medical system 10 that can be used for medical procedures such as, for example, diagnostic, therapeutic, or surgical procedures. The medical system 10 is located in a medical environment 11. In Figure 1A , the medical environment 11 is depicted as an operating room. In other embodiments, the medical environment 11 can be an emergency room, a medical training environment, a medical laboratory, or some other type of environment in which any number of medical procedures or medical training procedures can be performed. In still other embodiments, the medical environment 11 can include an operating room and a control area located outside the operating room.

[0082] In one or more embodiments, the medical system 10 can be a robot-assisted medical system under the remote operation control of a surgeon. In alternative embodiments, the medical system 10 can be under partial control of a computer programmed to perform a medical procedure or a subroutine. In still other alternative embodiments, the medical system 10 can be a fully automated medical system under the full control of a computer programmed to perform a medical procedure or a subroutine with the medical system 10. An example of a medical system 10 that can be used to implement the systems and techniques described in the present disclosure is the da Surgical System manufactured by Intuitive Surgical, Inc. of Sunnyvale, California, USA.

[0083] As Figure 1A shown, the medical system 10 generally includes an assembly 12 that can be mounted on or positioned near an operating table O on which a patient P is located. The assembly 12 can be referred to as a patient-side cart, a surgical cart, or a surgical robot. In one or more embodiments, the assembly 12 can be a remotely operated assembly. A remotely operated assembly can be referred to as, for example, a remotely operated arm cart. A medical device system 14 and an endoscopic imaging system 15 are operably coupled to the assembly 12. An operator input system 16 allows a surgeon S or other type of clinician to view or present images of the surgical site and control the operation of the medical device system 14 and / or the endoscopic imaging system 15.

[0084] The medical device system 14 can include one or more medical devices. In embodiments where the medical device system 14 includes multiple medical devices, the multiple medical instruments can include multiple identical medical devices and / or multiple different medical devices. Similarly, the endoscopic imaging system 15 can include one or more endoscopes. In the case of multiple endoscopes, the multiple endoscopes can include multiple identical endoscopes and / or multiple different endoscopes.

[0085] The operator input system 16 can be located at the surgeon's console, which can be in the same room as the operating table O. In some embodiments, the surgeon S and the operator input system 6 can be in different rooms or in completely different buildings from the patient P. The operator input system 16 generally includes one or more control devices for controlling the medical device system 14. The control devices can include one or more of any number of various input devices, such as hand grips, joysticks, trackballs, data gloves, trigger guns, foot pedals, manual controllers, voice recognition devices, touchscreens, body movement or presence sensors, and other types of input devices.

[0086] In some embodiments, the control devices will be provided with the same degrees of freedom as the medical devices of the medical device system 14 to provide the surgeon with telepresence, which is the perception that the control device and the instrument are integrated, such that the surgeon has a strong sense of directly controlling the instrument as if present at the surgical site. In other embodiments, the control devices can have more or fewer degrees of freedom than the associated medical devices and still provide the surgeon with telepresence. In some embodiments, the control devices are manual input devices that move in six degrees of freedom and can also include an actuatable handle for actuating the instrument (e.g., for closing the jaws of a grasping clamp end effector, applying an electrical potential to an electrode, delivering a drug treatment, and actuating other types of instruments).

[0087] When the surgeon S views the surgical site through the operator input system 16, the assembly 12 supports and manipulates the medical device system 14. Images of the surgical site can be obtained by the endoscopic imaging system 15, which can be manipulated by the assembly 12. The assembly 12 can include the endoscopic imaging system 15 and can similarly include multiple medical device systems 14. The number of single-use medical device systems 14 will typically depend on factors such as the diagnostic or surgical procedure to be performed and the space constraints within the operating room. The assembly 12 can include one or more non-servo-controlled linkages (e.g., one or more linkages that can be manually positioned and locked in place, which are commonly referred to as setting structures) and the kinematic structure of the manipulator. When the manipulator takes the form of a remote operating manipulator, the assembly 12 is a remote operating assembly. The assembly 12 includes multiple motors that drive inputs on the medical device system 14. In one embodiment, these motors move in response to commands from a control system (e.g., control system 20). The motors include drive systems that, when coupled to the medical device system 14, can advance the medical device into a naturally formed or surgically created anatomical orifice. Other electric drive systems can move the distal end of the medical device in multiple degrees of freedom, which can include three linear degrees of movement (e.g., linear movement along the X, Y, Z Cartesian axes) and three rotational degrees of movement (e.g., rotation about the X, Y, Z Cartesian axes). Additionally, the motors can be used to actuate an articulating end effector of the medical device to grasp tissue in the jaws of a biopsy device or the like. The medical device of the medical device system 14 can include an end effector having a single working member (e.g., a scalpel, a blunt blade, an optical fiber, or an electrode). Other end effectors can include, for example, forceps, graspers, scissors, or clip appliers.

[0088] The medical system 10 also includes a control system 20. The control system 20 includes at least one memory 24 and at least one processor 22 for implementing control between the medical device system 14, the operator input system 16, and other auxiliary systems 26, which can include, for example, an imaging system, an audio system, a fluid delivery system, a display system, a lighting system, a steering control system, a flushing system, and / or a suction system. The clinician can circulate within the medical environment 11 and can access, for example, the assembly 12 during a setup procedure or view the displays of the auxiliary systems 26 from the patient's bedside.

[0089] Although in Figure 1Ais depicted as being external to the other components of the medical system 10, but in some embodiments, the control system 20 can be fully or partially housed within any one of the assembly 12, the operator input system 16, or the assistance system 26. The control system 20 also includes programming instructions (e.g., stored on a non-transitory computer-readable medium) to implement some or all of the methods described in accordance with some aspects disclosed herein. Although the control system 20 is shown as a single block in the Figure 1A simplified schematic diagram of, the control system 20 can include two or more data processing circuits, with a portion of the processing optionally performed on or near the assembly 12 and another portion of the processing performed at the operator input system 16, and so on.

[0090] Any of a variety of centralized or distributed data processing architectures can be employed. Similarly, the programming instructions can be implemented as several independent programs or subroutines, or they can be integrated into several other aspects of the systems described herein (including remote operating systems). In one embodiment, the control system 20 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry.

[0091] The control system 20 communicates with a database 27, which can store a list of one or more clinician profiles, patients and patient profiles, a list of procedures to be performed on a patient, a list of clinicians scheduled to perform the procedures, other information, or a combination thereof. A clinician profile can include information about a clinician, which includes how long the clinician has worked in the medical field, the level of education the clinician has obtained, the clinician's level of experience with the medical system 10 (or a similar system), or any combination thereof.

[0092] The database 27 can be stored in the memory 24 and can be dynamically updated. Additionally or alternatively, the database 27 can be stored on a device such as a server or a portable storage device that can be accessed by the control system 20 via an internal network (e.g., a secure network of a medical facility or a remote operating system provider) or an external network (e.g., the Internet). The database 27 can be distributed at two or more locations. For example, the database 27 can exist on multiple devices, which can include devices of different entities and / or cloud servers. Additionally or alternatively, the database 27 can be stored on a portable user-assigned device (e.g., a computer, a mobile device, a smartphone, a laptop computer, an electronic badge, a tablet computer, a pager, and other similar user devices).

[0093] In some embodiments, the control system 20 may include one or more servo controllers that receive force and / or torque feedback from the medical device system 14. In response to the feedback, the servo controllers transmit signals to the operator input system 16. The servo controllers may also transmit signals indicating the movement of the assembly 12 to move the medical device system 14 and / or the endoscopic imaging system 15, which extend through an opening in the body to an internal surgical site within the patient. Any suitable conventional or dedicated servo controller may be used. The servo controller may be separate from or integrated with the assembly 12. In some embodiments, the servo controller and the assembly 12 are provided as part of a remote operating arm cart positioned near the patient's body.

[0094] The control system 20 may be coupled to the endoscopic imaging system 15 and may include a processor to process the captured images for subsequent display, such as on a surgeon's console or on another suitable display located locally and / or remotely for display to the surgeon. For example, in the case of using a stereoscopic endoscope, the control system 20 may process the captured images to present a coordinated stereoscopic image of the surgical site to the surgeon. Such coordination may include alignment between the relative images and may include adjusting the stereoscopic working distance of the stereoscopic endoscope.

[0095] In alternative embodiments, the medical system 10 may include more than one assembly 12 and / or more than one operator input system 16. The exact number of assemblies 12 will depend on factors such as the surgical procedure and space constraints within the operating room. The operator input systems 16 may be collocated, or they may be located in separate positions. Multiple operator input systems 16 allow more than one operator to control one or more assemblies 12 in various combinations. The medical system 10 may also be used for training and rehearsing medical procedures.

[0096] Figure 1Bis a perspective view of an embodiment of the assembly 12, which may be referred to as a patient-side cart, a surgical cart, a remote manipulator arm cart, or a surgical robot. The illustrated assembly 12 provides manipulation of three surgical tools 30a, 30b, and 30c (e.g., a medical device system 14) and an imaging device 28 (e.g., an endoscopic imaging system 15) (e.g., a stereoscopic endoscope for capturing images of the site of the procedure). The imaging device can transmit signals to the control system 20 via a cable 56. Manipulation is provided by a remote manipulator mechanism having a number of joints. The imaging device 28 and the surgical tools 30a-c can be positioned and manipulated through an incision in a patient's body such that a kinematic remote center is maintained at the incision to minimize the size of the incision. When the distal ends of the surgical tools 30a-c are positioned within the field of view of the imaging device 28, the image of the surgical site can include the images of the distal ends of the surgical tools 30a-c.

[0097] The assembly 12 includes a drivable base 58. The drivable base 58 is connected to a telescoping column 57 that allows adjustment of the height of the arm 54. The arm 54 can include a rotary joint 55 that rotates and moves up and down. Each arm 54 can be connected to an orientation platform 53. The arms 54 can be marked to facilitate troubleshooting. For example, each arm 54 can be decorated with a different number, letter, symbol, other identifier, or a combination thereof. The orientation platform 53 can be capable of 360-degree rotation. The assembly 12 can also include a telescoping horizontal boom 52 for moving the orientation platform 53 in the horizontal direction.

[0098] In this example, each arm 54 is connected to a manipulator arm 51. The manipulator arm 51 can be directly connected to a medical device (e.g., one of the surgical tools 30a-c). The manipulator arm 51 can be remotely operable. In some examples, the arms 54 connected to the orientation platform 53 may not be remotely operable. Instead, these arms 54 can be positioned as needed before the surgeon S begins operating with the remote operating components. During the entire surgical procedure, the medical device can be removed and replaced with other instruments such that the association of the instrument with the arm may change during the procedure.

[0099] Endoscopic imaging systems (e.g., endoscopic imaging system 15 and imaging device 28) can be provided in a variety of configurations including rigid or flexible endoscopes. A rigid endoscope includes a rigid tube that houses a relay lens system for transmitting an image from the distal end of the endoscope to the proximal end of the endoscope. A flexible endoscope uses one or more flexible optical fibers to transmit an image. Digital image-based endoscopes have a "chip on the tip" design, where a distal digital sensor (such as one or more charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) devices) stores image data. Endoscopic imaging systems can provide two-dimensional or three-dimensional images to a viewer. Two-dimensional images can provide limited depth perception. Three-dimensional stereoscopic endoscopic images can provide a viewer with a more accurate depth perception. Stereoscopic endoscopic instruments employ a stereoscopic camera to capture stereoscopic images of a patient's anatomy. The endoscopic instrument can be a fully sterilizable assembly where the endoscopic cable, handle, and shaft are all rigidly coupled and hermetically sealed.

[0100] Figure 1C FIG. 4 is a perspective view of one implementation of an operator input system 16 at a surgeon's console. The operator input system 16 includes a left-eye display 32 and a right-eye display 34 for presenting a coordinated stereoscopic view of a surgical environment enabling depth perception to a surgeon S (e.g., the left-eye display 32 and the right-eye display 34 are stereoscopic displays). The left-eye display 32 and the right-eye display 34 can be components of a display system 35. The left-eye display 32 and the right-eye display 34 can be incorporated into a headrest 39. The surgeon S can place their head on the headrest 39 to view the left-eye display 32 and the right-eye display 34. In some implementations, the display system 35 can include one or more other types of displays. The display system 35 can present, for example, an image captured by the imaging system 15 to show the endoscopic field of view to the surgeon S. The endoscopic field of view can be enhanced with virtual or synthetic menus, indicators, and / or other graphical or textual information to provide additional information to the viewer. In some implementations, the display system 35 can include one or more other user feedback devices, such as a lighting system, a speaker, a haptic feedback device, or other user interface devices for communicating information to the surgeon S. In some implementations, other user interface devices or displays can be positioned separately from the display system 35 on the operator input system 16, the assembly 12, or the control system 20.

[0101] The operator input system 16 also includes one or more input control devices 36 which in turn cause the assembly 12 to manipulate one or more instruments of the endoscopic imaging system 15 and / or the medical instrument system 14. The input control devices 36 can provide the same degrees of freedom as the associated instruments to provide telepresence to the surgeon S, or provide the perception that the input control device 36 is integral with the instrument such that the surgeon has a strong sense of directly controlling the instrument. To this end, orientation, force, and haptic feedback sensors (not shown) can be employed to transmit orientation, force, and haptic sensations from the medical instrument (such as the surgical tools 30a-c or the imaging device 28) back to the surgeon's hand through the input control device 36. Some aspects of the operator input system 16, the assembly 12, and the assist system 26 can be adjustable and customizable to meet the physical needs, skill level, or preferences of the surgeon S.

[0102] The input control device 37 is a foot pedal that receives input from the user's foot. The input control device 37 is positioned at the base of the operator input system 16 in the user input tray 38. The input control device 37 can control the functions of a remotely operated assembly (such as the medical system 10, the assembly 12) coupled to the arm 54 of the assembly 12 and / or a medical tool (such as the surgical tools 30a-c or the imaging device 28).

[0103] Although the input control device 37 is described as a foot pedal in the examples presented herein, the input control device 37 can include any suitable pedal, button, or other user input device to be manipulated by a non-hand limb different from the hand (such as the foot, leg, knee, arm, elbow, head, or other part of the human anatomy or an extension from the human anatomy (such as a selection wand, cane, or other selection tool)), which has relatively reduced tactile dexterity and / or sensitivity.

[0104] During a medical procedure performed using the medical system 10, the surgeon S or another clinician may need to access a medical tool in the patient's anatomy outside the field of view of the imaging system 15, may need to engage the input control device 37 (such as a foot pedal) to activate the medical tool or perform other system functions, and / or may need to identify a tool occluded in the field of view. Additionally, in the case of a stereoscopic field of view, it may be desirable for synthetic elements presented in the field of view to be displayed at a depth corresponding to the tissue or component indicated by the synthetic element. Thus, the synthetic elements may appear attached to the components in the field of view rather than floating in front of the field of view. The various embodiments described below provide some methods and systems that allow the surgeon S to view depth perception graphic indicators, indicators of components outside the field of view, and indicators of components occluded in the field of view.

[0105] Figure 2 is a perspective view of a user input disk 100 according to some embodiments. The user input disk 100 may be implemented as the user input disk 38 described above. The user input disk 100 includes a plurality of user input devices 102a - 102f (collectively or individually referred to as user input devices 102). The user input devices 102 are arranged in a certain layout on one or more surfaces of the user input disk 100. In one embodiment, the user input disk 100 is arranged in a certain layout at the base of the operator input system 16 to facilitate a surgeon S to select the user input devices 102 with the surgeon S's foot while observing the field of view of the imaging system 15 in the display system 35.

[0106] The user input disk 100 has a first side 106 (also referred to as the front side) and a second side 108 (also referred to as the rear side), where the second side 108 is opposite to the first side 106. The user input disk 100 also has a third side 110 (also referred to as the left side) and a fourth side 112 (also referred to as the right side), where the fourth side 112 is opposite to the third side 110. The third side 110 and the fourth side 112 form angles with the first side 106 and the second side 108 respectively to form the perimeter of the user input disk 100. The user input disk 100 also has a first base 114 and a second base 116. The second base 116 is spaced apart from the first base 114 to form a step within the user input disk 100. Sidewalls extend between the first base 114 and the second base 116 to form a step within the user input disk 100. The sidewalls extend from the second base 116 along the second side 108. The third side 110 and the fourth side 112 include sidewalls extending from the first base 114 and the second base 116 to form a partially enclosed area within the user input disk 100. In the illustrated example, the first side 106 does not include sidewalls. In the illustrated example, the sidewalls on the third side 110 and the fourth side 112 are tapered towards the first side 106.

[0107] Figure 3 is a cross - sectional view of the user input disk 100 according to some embodiments. The user input devices 102 may be referred to as buttons, pedals, touch sensors, or any other device for receiving user selection inputs. Each user input device 102 has a leading edge 118 and a selection surface 120. Each user input device 102 communicates with a control system 20 for recording selections of one or more of the user input devices 102 and performing associated actions to control the functions of a remote operation assembly (e.g., medical system 10, assembly 12) coupled to the arm 54 of the assembly 12 and / or a medical tool (e.g., surgical tools 30a - c or imaging device 28) (e.g., performing one or more functions of the imaging device 28 and / or the surgical tools 30a - c).

[0108] When the user input device 102 is selected, the leading edge 118 of the user input device is positioned to face the expected approach direction 122 of an object (e.g., a non-hand limb such as the foot of surgeon S). For example, when the user input device 102 is selected, the expected object (e.g., a non-hand limb such as the foot of surgeon S) approaches the user input device 102 from the first side 106 of the user input pad 100 towards the second side 108 of the user input pad 100. In Figure 3 the example shown, the leading edge 118a of the user input device 102a is positioned closest to the first side 106 of the user input pad 100. Similarly, the leading edge 118f of the user input device 102f is positioned closest to the first side 106 of the user input pad 100. However, the position of the leading edge 118f of the user input device 102f is at a different position from the leading edge 118a of the user input device 102a.

[0109] The selection surface 120 of the user input device 102 is configured to record a selection event. For example, when an object (e.g., a non-hand limb such as the foot of surgeon S) presses on or remains in contact with the selection surface 120 for a predetermined period of time, the selection event is recorded on the user input device 102. As described above, after the selection event is recorded on one of the user input devices 102, the control system 20 controls the relevant functions of the medical system 10, the assembly 12, the arm 54 of the assembly 12, the surgical tools 30a-c, and / or the imaging device 28. In various examples, the user input device 102 includes one or more sensors (not shown) for recording selection events. For example, the one or more sensors can include pressure sensors, tactile sensors, displacement sensors, switches, buttons, capacitive sensors, or other types of sensors that detect that one or more of the user input devices have been activated or engaged.

[0110] In various embodiments, the one or more sensors in the user input device 102 for recording selection events can distinguish between a hover event and a selection event. That is, the one or more sensors in the user input device 102 (alone or in conjunction with the range sensor 104) distinguish an object that is positioned to select the selection surface 120, is being placed on the selection surface 120, or is in contact with the selection surface 120 (e.g., a hover event) from an object that is pressing on the selection surface 120 (such as a selection event). For example, when surgeon S moves their foot from one of the user input devices 102 to another, a hover event can be detected.

[0111] For example, one or more sensors may use a combination of a first sensor (e.g., a pressure sensor or a capacitance sensor) and a second sensor (e.g., a haptic sensor, a displacement sensor, a switch, or a button), where the first sensor is used to detect contact with the selection surface 120, and the second sensor is used to detect a selection of the selection surface 120.

[0112] In another example, one or more sensors may use the same sensor to detect both contact with the selection surface 120 and a selection of the selection surface 120. For example, a first sensor may detect contact with the selection surface 120 (e.g., a pressure sensor detects a first threshold pressure amount, a multi-stage switch detects a first stage of the multi-stage switch, etc.), and the first sensor may also detect a selection of the selection surface 120 (e.g., the pressure sensor detects a second threshold pressure amount greater than the first threshold pressure, the multi-stage switch detects a second stage of the multi-stage switch, etc.).

[0113] In various embodiments, the selection surface 120 may include a protrusion 124. The protrusion 124 is positioned on the selection surface 120 along the leading edge 118. For example, the protrusion 124a is positioned on the selection surface 120a along the leading edge 118a. The protrusion 124 provides haptic feedback to assist in positioning an object for selection on the selection surface of the user input device 102. The protrusion 124 also facilitates gripping onto the selection surface 120 of the user input device 102. For example, the protrusion 124 increases frictional engagement with a shoe or other selection object to assist in positive selection of the user input device 102. Although each of the user input devices 102 is depicted as having a protrusion 124, in some embodiments, only one or some of the user input devices 104 have a protrusion 124.

[0114] After detecting a hover event or a selection event of one or more of the user input devices 102, the control system 20 controls the functions of the remote operation assembly (e.g., the medical system 10, the assembly 12) and / or the medical tool (e.g., the surgical tools 30a-c or the imaging device 28) coupled to the arm 54 of the assembly 12. For example, after detecting a selection event of the user input device 102f, the control system 20 may control the imaging system 15 to capture an image of the current field of view. The present disclosure contemplates the control of other tools and functions performed by the control system 20, such as the tools and functions described below in connection with Figures 5A - 7D the description.

[0115] Reference Figure 2 and Figure 3, multiple ranging sensors 104a - 104f (collectively referred to as ranging sensors 104) are positioned to have a field of view 126 across the selection surface 120 of the user input device 102. The ranging sensors 104 are configured to measure the distance to an object within the field of view 126. Each ranging sensor 104 communicates with the control system 20 for controlling the display system 35 and / or one or more other user feedback devices (such as a lighting system, a speaker, a haptic feedback device, or other user interface devices for communicating information to the surgeon S).

[0116] The ranging sensor 104 can be any type of sensor for measuring the distance of an object relative to the selection surface 120. For example, the ranging sensor can be a time - of - flight distance sensor, a triangulation distance sensor, an optical distance sensor, an acoustic distance sensor, an ultrasonic sensor, an inductive distance sensor, a capacitive distance sensor, a photoelectric distance sensor, a camera, an infrared distance sensor, a laser rangefinder, a light detection and ranging sensor, or any other type of ranging sensor.

[0117] In the example shown, one of the ranging sensors 104 is positioned to have a field of view 126 across the selection surface 120 of each user input device 102. For example, the ranging sensor 104a is positioned on a sidewall extending from the second base 116 along the second side 108 and facing the first side 106 such that the field of view 126a spans the selection surface 120a of the user input device 102a.

[0118] In the example shown, the ranging sensors 104 are positioned to face the expected approach direction 122 of the object. Thus, the direction of the field of view 126 is parallel to the direction of the distance measured by the ranging sensors 104. That is, the field of view 126 spans the selection surface 120 and extends towards the leading edge 118 of the user input device 102. In other words, the ranging sensors 104 are positioned to face the direction from the second side 108 of the user input disc 100 towards the first side 106 of the user input disc 100.

[0119] In some embodiments, one or more of the ranging sensors 104 can be positioned to face a direction orthogonal to the expected approach direction 122 of the object. In other words, one or more of the ranging sensors can be positioned to have a field of view 126 extending in a direction parallel to the leading edge 118 of the user input device 102. For example, instead of being positioned on the sidewall along the second side 108 of the user input disc 100, one or more of the ranging sensors 104 can be positioned on the sidewall of the third side 110 or the fourth side 112 of the user input disc 100.

[0120] In one example, a camera can be positioned on the sidewall along the second side 108 of the user input disc 100 to capture having the same asFigure 3 Images of similar fields of view as shown. Thus, the direction of the field of view is orthogonal to the direction of the distance measured by the rangefinder sensor. The control system 20 can perform image processing of the images to determine the distance of the object relative to the selection surface 120. The camera can be used in combination with the rangefinder sensor or in combination with a second camera to generate depth information that indicates where the object may be located along the length of the first side 106 or the second side 108 of the user input pad 100 (e.g., in front of the user input device 102a, in front of the user input device 102b, or in front of the user input device 102c).

[0121] The present disclosure contemplates other placements and orientations of the rangefinder sensor 104. For example, the rangefinder sensor 104 can be placed on the user input device 102 itself, with its field of view pointing away from the selection surface 120. The rangefinder sensor 104 can also be placed above the user input pad 100, with its field of view directed downward toward the user input device 102. The rangefinder sensor 104 can also be placed on the sidewall on the third side 110 or the fourth side 112 of the user input pad 100, with its field of view spanning across the user input pad 100.

[0122] The user input device 102g is positioned on the sidewall on the third side 110 of the user input pad 100. Thus, range data from the rangefinder sensor 104a and / or the rangefinder sensor 104f can be used to detect an object near the selection surface 120 of the user input device 102g. More generally, one or more of the rangefinder sensors 104 can be positioned to have a field of view 126 that extends across the selection surfaces 120 of multiple user input devices 102.

[0123] Although only one rangefinder sensor 104 is shown as being positioned to have a field of view 126 across the selection surface 120 of the user input device 102, in some embodiments, multiple (e.g., two or more) rangefinder sensors can be used to increase the redundancy of object detection and / or increase the field of view 126 across the selection surface 120 of the user input device 102.

[0124] In operation, the rangefinder sensor 104 is configured to measure the distance of the object relative to the selection surface 120. At each of a plurality of threshold distance values, different user interface actions are performed by the control system 20. The different user interface actions provide feedback (e.g., tactile, visual, auditory) to the surgeon S regarding the placement of the object (e.g., a non - hand limb, such as the surgeon S's foot) relative to the user input device 102 and its activation.

[0125] At Figure 3In the example shown, there are two threshold distances associated with each range sensor 104. For example, for range sensor 104a, there is a first threshold distance 128a and a second threshold distance 130a, where the second threshold distance 130a is greater than the first threshold distance 128a.

[0126] Similarly, for range sensor 104f, there is a first threshold distance 128f and a second threshold distance 130f, where the second threshold distance 130f is greater than the first threshold distance 128f. In various embodiments, the first threshold distance 128a may be the same as or different from the first threshold distance 128f. The second threshold distance 130a may be the same as or different from the second threshold distance 130f.

[0127] More generally, the first threshold distances 128a, 128f are individually or collectively referred to as the first threshold distance 128, and the second threshold distances 130a, 130f are individually or collectively referred to as the second threshold distance 130, where the second threshold distance 130 is greater than the first threshold distance 128.

[0128] The first threshold distance 128 is located at or near the leading edge 118 of the user input device 102. The first threshold distance 128 located near the leading edge 118 of the user input device 102 is a location where an intentional or accidental selection of the user input device 102 may occur. The second threshold distance 130 does not co - extend with the user input device 102 (e.g., is not located along the selection surface 120) and is spaced a predetermined distance from the leading edge 118 of the user input device 102. In Figure 3 the example shown, there is a predetermined distance 132a between the first threshold distance 128a and the second threshold distance 130a. Similarly, there is a predetermined distance 132f between the first threshold distance 128f and the second threshold distance 130f. More generally, the predetermined distances 132a, 132f are individually or collectively referred to as the predetermined distance 132.

[0129] In various embodiments, the placement of the first threshold distance 128 and the second threshold distance 130 may vary according to other presence sensors on the medical system 10, the assembly 12, and / or the operator input system 16. For example, a hand presence sensor on the input control device 36 or a head presence sensor on the display system 35 and / or the headrest 39 may be used to adjust the first threshold distance 128 and the second threshold distance 130 for detecting an object. If no head and / or hand is detected, the first threshold distance 128 and the second threshold distance 130 may be adjusted such that it is more difficult to detect an object (e.g., a more certain measurement of the object's presence is required before providing an indication of the object's presence).

[0130] In various embodiments, the predetermined distance 132 is between 5 mm and 50 mm (including the endpoints). In some embodiments, the predetermined distance 132 is between 10 mm and 30 mm (including the endpoints). In some embodiments, the predetermined distance 132 is between 15 mm and 25 mm (including the endpoints). In one embodiment, the predetermined distance 132 is 20 mm. All provided values are considered to have a variation of up to 25% of the provided value.

[0131] In various embodiments, the predetermined distance 132 is the same for all user input devices 102. In some embodiments, depending on the geometry of the user input device 102, the predetermined distance 132 is different for one or more of the user input devices 102. More generally, the predetermined distance 132 is set to be a sufficient distance away from the user input device 102 to mitigate accidental selection of the user input device 102 by an object.

[0132] As described above, the control system 20 determines the distance to an object (e.g., a non - hand limb, such as the foot of surgeon S) within the field of view 126 measured by one or more of the range sensors 104. After the control system 20 determines that the measured distance is less than or equal to the first threshold distance 128, the control system 20 performs a first user interface action. After the control system 20 determines that the measured distance is greater than or equal to the second threshold distance 130, the control system 20 performs a second user interface action. For example, for range sensor 104f, after the control system 20 determines that the measured distance is less than or equal to the first threshold distance 128f, the control system 20 performs a first user interaction. Similarly, after the control system 20 determines that the measured distance of range sensor 104f is greater than or equal to the second threshold distance 130f, the control system 20 performs a second user interaction.

[0133] The first and second user interface actions provide feedback (e.g., tactile, visual, auditory) to surgeon S regarding the placement of the object (e.g., a non - hand limb, such as the foot of surgeon S) relative to the user input device 102. The first user interface action provides feedback that the object is above or otherwise positioned to facilitate selection of one of the user input devices 102. The second user interface action provides feedback that the object is no longer above or otherwise positioned to facilitate selection of one of the user input devices 102. In various embodiments, for different user input devices 102, the first user interface action can be the same or different. Similarly, for different user input devices 102, the second user interface action can be the same or different.

[0134] Following the example associated with the ranging sensor 104f above, the first user interface action provides feedback that an object is within the field of view 126f and located at or closer than the first threshold distance 128f, in order to be positioned above or otherwise to facilitate selection of the selection surface 120f of the user input device 102f. The second user interface action provides feedback that the object is at or farther than the second threshold distance 130f, in order to no longer be positioned above or otherwise to facilitate selection of the selection surface 120f of the user input device 102f.

[0135] In various embodiments, the control system 20 records hover events and selection events of the user input device 102 according to the order in which an object is detected within the first threshold distance 128. For example, after an object is first detected within the first threshold distance 128a, the control system 20 may ignore subsequent detections of the object within the first threshold distance 128f or hover events or selection events of the user input device 128e until the object first detected within the first threshold distance 128a is at or farther than the second threshold distance 130a.

[0136] In various embodiments, if the control system determines that an object is within the first threshold distance 128 for more than one user input device 102, the control system 20 may determine that the object is located at the user input device 102 with a closer ranging reading (e.g., higher signal strength) from the ranging sensor 104. For example, a first ranging reading is provided from the ranging sensor 104a, and a second ranging reading is provided from the ranging sensor 104f, and both are within their respective first threshold distances 128a, 128f. If the first ranging reading is closer (e.g., has a higher signal strength) than the second ranging reading, the control system 20 may determine that the object is located at the user input device 102a. In such an embodiment, the first user interface action is provided for the user input device 102a rather than for the user input device 102f.

[0137] Alternatively, the first user interface action may be provided for all user input devices 102 for which an object is detected within the first threshold distance. Alternatively, the control system 20 may issue an error or warning when an object is detected within the threshold distance 128 of more than one user input device 102.

[0138] In various embodiments, the control system 20 records hover events and selection events of the user input device 102 based on whether an object is detected within the first threshold distance 128. For example, after detecting a hover event or a selection event of the user input device 102a, if an object is not detected within the first threshold distance 128a, the control system 20 ignores the hover event or the selection event. Thus, the ranging sensor 104 provides redundancy to prevent accidental or unintentional selection of one or more of the user input devices 102.

[0139] Figures 4A - 4F FIG. shows a graphical user interface 300 that can be displayed, for example, on the display system 35. The graphical user interface 300 includes icons 302a - 302g (individually or collectively referred to as icons 302), which correspond to the user input devices 102a - 102f in the user input pad 100. The icons 302 provide visual feedback for first and second user interface actions on the display system 35. In the example shown, the icons 302 are arranged in the layout of the user input devices 102 in the footrest 100. In some embodiments, the icons 302 can be arranged in any layout or only displayed when performing the first or second user interface action. In some embodiments, the graphical user interface 300 further includes an icon 304 indicating the shape of the user input pad 100.

[0140] In some embodiments, the graphical user interface 300 can be displayed within a portion of a larger graphical user interface (not shown). For example, as Figures 5A - 7D shown and described, the larger graphical user interface can include a field - of - view portion for displaying an image of the field of view of the surgical environment captured by an imaging system (such as the imaging system 15). The larger graphical user interface can also include information blocks for displaying information about the medical tools and information blocks for displaying information about the imaging system that captured the image in the field - of - view portion. The graphical user interface 300 can be included as an additional information block within the larger graphical user interface or superimposed on the field - of - view portion of the larger graphical user interface.

[0141] In some embodiments, the first user interface action is to modify the display of one of the associated icons 302 to indicate that an object (e.g., a non - hand limb, such as the foot of the surgeon S) is above or otherwise positioned to facilitate selection of one of the user input devices 102. In other words, the first user interface action is the first action to display an indication of the user input device 102 in which the object is positioned for selection.

[0142] In some embodiments, the second user interface action is to modify the display of one of the associated graphs in icon 302 to indicate that the object (e.g., a non - hand limb, such as the foot of surgeon S) is no longer above or otherwise positioned to facilitate the selection of one of the user input devices 102. In various embodiments, the second user interface action is simply to abort the display of the first user interface action.

[0143] In Figure 4B the example shown, the indication of the user input device 102 in which the object is positioned for selection is displayed as the foot icon 306 in an overlapping manner with icon 302f, to indicate that the object is positioned above or otherwise positioned to facilitate the selection of user input device 102f. Thus, the foot icon 306 serves as a graphic of the object (e.g., a non - hand limb, such as the foot of surgeon S) relative to the layout of icon 302.

[0144] In other embodiments, the indication of the user input device 102 in which the object is positioned for selection is displayed as an indication of the function to be performed after the selection of the user input device 102. For example, for user input device 102f, an icon of a camera may be displayed in icon 302f or in the larger graphical user interface described above, to indicate that the object is positioned to facilitate the activation of the camera function after the selection of user input device 102f.

[0145] In further embodiments, the indication of the user input device 102 in which the object is positioned for selection is displayed as a change in the intensity, color, highlighting, or other visually distinct change of icon 302. For example, as Figure 4C shown, icon 302f is shown to be displayed in a pattern. In this example, icon 302f is also shown together with the foot icon 306, but in other examples, the foot icon 306 may be omitted.

[0146] In another embodiment, the graphical user interface 300 can be modified based on sensor readings from both one or more sensors of the ranging sensor 104 and the user input device 102. For example, the graphical user interface 300 can first display the indication of the user input device 102 in which the object is positioned for selection. Additionally, after detecting that the object stays on or contacts the selection surface 120 (e.g., a hover event), icon 302 can be further modified to indicate the hover event. For example, as Figure 4B shown, the indication of the user input device 102 in which the object is positioned for selection can be displayed. Subsequently, after detecting a hover event on user input device 102f, icon 302f is then modified to show as Figure 4Cthe pattern shown, or be modified to have a unique visual appearance. Additionally or alternatively, after detecting an object pressing on the selection surface 120 (e.g., a selection event), the icon 302 can be further modified to indicate the selection event. For example, after detecting a selection event on the user input device 102f, the icon 302f is subsequently modified to show the filled pattern as Figure 4D shown, or be modified to have a unique visual appearance.

[0147] In various embodiments, one or more of the above examples can be used in combination with each other as an indication of the user input device 102 in which an object is positioned for selection, an indication of a hover event, and / or an indication of a selection event.

[0148] In some embodiments, the control system 20 tracks and evaluates the ranging data from the ranging sensor 104 as a time series. Thus, the control system 20 is additionally able to determine the direction and / or speed of movement of the object (even beyond the above threshold distance). This time series of ranging data can facilitate determining the intention of the surgeon S based on the speed and / or direction of movement. Based on the determined intention, the control system 20 can modify the user interface actions or operations performed by the control system 20.

[0149] In some embodiments, the control system 20 determines the three-dimensional trajectory of an object based on a time series of data using the ranging data from one or more ranging sensors 104. For example, the ranging sensor 104 can capture three-dimensional ranging data. Alternatively or additionally, the control system 20 can integrate the ranging data from more than one ranging sensor 104 at different locations to resolve the three-dimensional orientation, direction of movement, and / or speed of the object.

[0150] For example, after tracking the time series of ranging data, the control system 20 can determine that the surgeon S is moving their feet quickly (e.g., at a speed greater than a first predetermined threshold) and / or moving their feet in a direction away from the user input device 102. Thus, the control system 20 can determine that the surgeon S intends to no longer use the user input device 102. Thus, any incidental hover events or selection events on any of the user input devices 102 can be ignored by the control system 20 or require verification from the surgeon S.

[0151] In another example, after tracking the time series of ranging data, the control system 20 can determine that the surgeon S is moving their feet slowly (e.g., at a speed less than a second predetermined threshold) and / or moving their feet in a direction towards one or more user input devices 102. Thus, the control system 20 can determine that the surgeon S intends to select the user input device 102 in the detected direction.

[0152] After determining the intention to select one of the user input devices 102, the control system 20 can initiate one or more control actions associated with the user input device 101 in the detected direction, which may require a lead time to be executed in order to reduce the lag time between the selection of the user input device and the execution of the control action. Alternatively or additionally, the control system 20 can change the power state of the medical tool associated with the user input device 102 in the detected direction, such that the tool can be switched from a lower power consumption mode to a higher power consumption mode. Alternatively or additionally, the control system 20 can provide user interface feedback to inform the surgeon S which one of the user input devices 102 their foot is currently moving towards. The second predetermined speed threshold is the same as or different from the first predetermined speed threshold. Other intentions and actions are contemplated by the present disclosure.

[0153] In another example, after tracking a time series of ranging data, the control system 20 can modulate the time period during which the user input device 102 can be selected. For example, after tracking a time series of ranging data, the control system 20 can determine that the surgeon S is moving their foot rapidly (e.g., at a speed greater than a predetermined threshold), and any detected selection events within a predetermined time period can be ignored or require verification from the surgeon S.

[0154] Additionally or alternatively, after tracking a time series of ranging data, the control system 20 can animate the graphical user interface 300 or otherwise modify the graphical user interface 300 to provide an indication of the speed and direction of an object relative to the user input device 102. As Figure 4E and Figure 4F shown in the example of, even outside the threshold distance, the graphical user interface 300 can animate the position of the object. In some embodiments, the graphical user interface 300 can animate a plurality of objects located within an icon 304 that indicates the shape of the user input pad 100. For example, the graphical user interface 300 can animate both the left foot and the right foot of the surgeon S (as indicated by the foot icon 306 and the foot icon 312).

[0155] As Figure 4E shown, the foot icon 306 can be animated to move in the direction towards the icons 302a and 302f together with an indicator 308 that represents the direction and / or speed of the movement of the object. For example, by being located on the rear side of the foot icon 306, the indicator 308 represents the movement of the object towards the user input device 102. The speed can be represented by the indicator 308, which uses longer lines to indicate higher speeds and shorter lines to indicate slower speeds. Other visual representations of direction and speed are contemplated by the present disclosure.

[0156] Similarly, as Figure 4F shown, the foot icon 306 can be animated to move in a direction away from the icons 302a and 302f along with an indicator 310 representing the direction and / or speed of the movement of the object. For example, by being positioned on the front side of the foot icon 306, the indicator 310 represents the movement of the object away from the user input device 102. The speed can be represented by the indicator 310, which uses longer lines to indicate higher speeds and shorter lines to indicate slower speeds. Other visual representations of direction and speed are contemplated by the present disclosure.

[0157] In various embodiments, the control system 20 can animate the graphical user interface 300 linearly, two-dimensionally, or three-dimensionally with an indication of the object (e.g., the foot icon 306) based on the sensitivity and resolution of the range sensor 104. For linear animation, the foot icon 306 can simply travel back and forth in a line intersecting the plurality of icons 302. For example, as Figure 4E shown, the foot icon 306 can animate the movement of the left foot of the surgeon S in a line intersecting the icons 302a and 302f. For two-dimensional animation, the foot icon 306 can be animated to be positioned at any corresponding orientation of the object within the user input pad 100 (e.g., anywhere within the icon 304 indicating the shape of the user input pad 100).

[0158] In the examples described above with reference to Figures 4A - 4F the first and second user interface actions are to modify the display on the display system 35. In other examples, the first and second user interface actions can be to provide auditory or tactile feedback to the surgeon S.

[0159] For example, for auditory feedback, a first audio indication (e.g., tone, sound effect, music, etc.) can be output from a speaker as the first user interface action. A second audio indication can be output from the speaker as the second user interface action. The first audio indication can be the same as or different from the second audio indication. Additionally, different user input devices 102 can have different sets of first and second audio indications.

[0160] For example, a first audio indication may be provided as a first user interface action associated with user input device 102a, and a second audio indication may be provided as a second user interface action associated with user input device 102a. Similarly, a third audio indication may be provided as a first user interface action associated with user input device 102f, and a fourth audio indication may be provided as a second user interface action associated with user input device 102f. Although only two of user input devices 102 are discussed in this example, any one of user input devices 102 may have the same or different audio indications for the first and second user interface actions.

[0161] Likewise, for tactile feedback, different modes of feedback (e.g., pulses, sequences, etc.) may be provided to the surgeon S. Tactile feedback may be provided to the surgeon S via a tactile feedback transducer (not shown) coupled to any user input device, via a tactile feedback transducer (not shown) coupled to the headrest 39 , via tactile feedback provided via one or more of the input control devices 36 (e.g., hand controllers), or via tactile feedback provided anywhere on the operator input system 16 .

[0162] For example, a first tactile feedback pattern may be provided to surgeon S as a first user interface action. A second tactile feedback pattern may be provided to surgeon S as a second user interface action. The first tactile feedback pattern may be the same as or different from the second tactile feedback pattern. In addition, different user input devices 102 may have different sets of first and second tactile feedback patterns.

[0163] For example, the first tactile feedback pattern can be provided as a first user interface action associated with user input device 102a, and the second tactile feedback pattern can be provided as a second user interface action associated with user input device 102a. Similarly, the third tactile feedback pattern can be provided as a first user interface action associated with user input device 102f, and the fourth tactile feedback pattern can be provided as a second user interface action associated with user input device 102f. Although only two of the user input devices 102 are discussed in this example, any one of the user input devices 102 can have the same or different tactile feedback patterns for the first and second user interface actions.

[0164] Figure 5A , Figure 5B and Figure 5CShows a graphical user interface 200 that can be displayed, for example, on a display system 35. The graphical user interface 200 can include a field-of-view portion 202 for displaying an image of the field of view of a surgical environment captured by an imaging system (e.g., imaging system 15). The surgical environment can have a Cartesian coordinate system Xs, Ys, Zs. The image in the field-of-view portion 202 can be a three-dimensional stereoscopic image and can include patient tissue and surgical components (including instruments such as medical tools 400 and medical tool 402). The graphical user interface 200 can also include an information block 210 for displaying information about the medical tool 400, a message block 212 for displaying information about the imaging system (e.g., imaging system 15) that captured the image in the field-of-view portion 202, an information block 214 for displaying information about the medical tool 402, and an information block 216 for displaying information indicating that a fourth medical tool is not installed. The information blocks 210, 212, 214, 216 can include tool type, the number of manipulator arms coupled to the tool, status information of the arm or tool, and / or operation information of the arm or tool.

[0165] The medical tool 400 and the medical tool 402 are visible in the field-of-view portion 202. The function of the medical tool can be initiated by engaging a corresponding user input device 102 (e.g., a foot pedal) on the user input pad 100. For example, the medical tool 400 can be operated by the manipulator arm 1 as indicated by the information block 210 and can be a vessel sealer that can perform a cutting function when the user input device 102b is engaged and a sealing function when the user output device 102e is engaged. As Figure 5AAs shown, the tool 400 can be marked with a composite indicator 404. In this embodiment, the composite indicator 404 can be a generally circular badge including an upper semi-circular portion 406 and a lower semi-circular portion 408. The upper semi-circular portion 406 includes a contour portion 410 and a central portion 412, and the lower semi-circular portion 408 includes a contour portion 414 and a central portion 416. The upper semi-circular portion 406 can correspond to the function of the secondary user input device 102b and can indicate the engagement state (e.g., hover, activation) of the user input device 102b. The lower semi-circular portion 408 can correspond to the function of the primary user input device 102e and can indicate the engagement state (e.g., hover, activation) of the user input device 102e. The spatial relationship between the upper semi-circular portion 406 and the lower semi-circular portion 408 can have the same or a similar spatial relationship as the user input devices 102b, 102e. When the range sensor 104 and / or the sensors of the user input device 102 detect that the operator's foot is hovering above the user input device 102b or otherwise within a threshold distance from the user input device 102b, the contour portion 410 of the upper semi-circular portion 406 can change its appearance (e.g., change color, become animated) to indicate to the operator that the operator's foot is approaching the user input device 102b. Thus, the operator can determine the foot orientation while the operator's line of sight remains directed towards the graphical user interface 200. When the operator engages the user input device 102b (e.g., steps on or presses down the pedal), the central portion 412 of the upper semi-circular portion 406 can change its appearance (e.g., change color, become animated) to indicate to the operator that the operator's foot has engaged the user input device 102b and the function of the user input device 102b (e.g., cutting) has been activated. In some embodiments, the same or a similar graphical indicator can be used to indicate the hover or engagement state of the user input device 102b in the information block 210. The left bank of the user input device 102 (e.g., user input devices 102b, 102e) can be associated with the left hand input control device, and the right bank of the user input device (e.g., user input devices 102c, 102d) can be associated with the right hand input control device. Each hand can be associated with controlling any of the instrument arms. The co-located composite indicator reflects this association of the instrument with the respective hand and foot. In some configurations, the posture of the instrument relative to the endoscopic field of view might otherwise appear ambiguously related to the left or right side, so the co-located composite indicator clarifies this association.

[0166] As Figure 5CAs shown, the lower semi-circular portion 408 can function as an indicator of hovering and engagement of the user input device 102e similarly to the upper semi-circular portion 406. When the operator engages the primary user input device 102e (e.g., steps on or presses down the pedal), the central portion of the lower semi-circular portion 408 can change its appearance (e.g., change color, become animated) to indicate to the operator that the operator's foot has engaged the user input device 102e and the function of the user input device 102.e (e.g., sealing) has been activated. The user input devices 102 at the surgeon's console can be color-coded. For example, the primary user input devices 102e, 102d can be colored blue, and the secondary user input devices 102b, 102c can be colored yellow. This color-coding is reflected in the associated highlighting and fill colors of the pedal function synthesis indicators on the graphical user interface.

[0167] As Figure 5B shown, the tool 402 can be marked with a synthesis indicator 420. In this embodiment, the synthesis indicator 420 can be substantially similar in appearance and function to the synthesis indicator 404, but can provide information about the set of user input devices 102c, 102d. The tool 402 can be operated by the manipulator arm 3 as indicated by the information block 214, and can be a monopolar cautery instrument that can perform the function of delivering energy for cutting when the user input device 102c is engaged, and can perform the function of delivering energy for coagulation when the user input device 102d is engaged. When the range sensor 104 and / or the sensors of the user input device 102 detect that the operator's foot is hovering above the user input device 102c or otherwise within a threshold distance from the user input device 102c, the contour portion of the upper semi-circular portion can change its appearance to indicate to the operator that the operator's foot is close to the user input device 102c. When the range sensor 104 and / or the sensors of the user input device 102 determine that the operator has engaged or activated the user input device 102c, the central portion of the upper semi-circular portion can change its appearance to indicate to the operator that the operator's foot has engaged the user input device 102c and the function of the user input device 102c (e.g., delivering energy for cutting) has been activated. In some embodiments, the same or similar graphical indicators can be used to indicate the hovering or engagement state of the secondary user input device 102b in the information block 214.

[0168] In some embodiments, the lower semi-circular portion of the indicator 420 can be used as an indicator for hovering and engaging the primary user input device 102d similar to the upper semi-circular portion. When the operator engages the primary user input device 102d, the central portion of the lower semi-circular portion can change its appearance to indicate to the operator that the operator's foot has engaged the primary user input device 102d and the function of the user input device 102d (e.g., delivering energy for coagulation) has been activated.

[0169] The orientation and alignment of the composite indicators 404, 420 can be determined to create the appearance that the composite indicators are, for example, decals adhered to the tool U-bolt or shaft. As the tool or endoscope providing the field of view moves, the composite indicators 404, 420 can change their orientation in three-dimensional space to maintain tangency with the tool surface and retain the spatial understanding of the upper and lower pedals.

[0170] Composite indicators of various types, shapes, and configurations can be displayed to provide information about the state of engagement of the user input device 102. In alternative embodiments, as Figure 6A , Figure 6B , Figure 6C and Figure 6D shown, the graphical user interfaces 200 with the medical tools 400, 402 are visible in the field of view portion 202. In this embodiment, the composite indicators 450, 452, 454, 456 can take the form of elongated strips extending along the perimeter 219.

[0171] In this example, the composite indicators 450 - 456 are within the boundaries of the perimeter 219, but in alternative embodiments can be outside the perimeter 219 of the field of view 202. In this embodiment, the composite indicators 450, 452 can perform a function similar to that of the composite indicator 404 when providing information about the set of user input devices 102b, 102e. As Figure 6A shown, when the range sensor 104 and / or the sensors of the user input device 102 detect that the operator's foot is hovering above the user input device 102d or otherwise within a threshold distance from the user input device 102d, the composite indicator 456 is outlined to indicate to the operator that the operator's foot is near the primary user input device 102d. As Figure 6B shown, when the operator engages the user input device 102d, the composite indicator 456 can change to a filled bar to indicate to the operator that the operator's foot has engaged the user input device 102d and the function of the user input device 102d has been activated. In some embodiments, the same or similar graphical indicators can be used to indicate the hovering or engagement state of the user input device 102d in the information block 214.

[0172] As Figure 6C shown, when the rangefinder sensor 104 and / or the sensors of the user input device 102 detect that the operator's foot is hovering over or otherwise within a threshold distance from the secondary user input device 102b, the composite indicator 450 is outlined to indicate to the operator that the operator's foot is close to the user input device 102b. As Figure 6D shown, when the operator engages the user input device 102b, the composite indicator 456 can change to a filled bar to indicate to the operator that the operator's foot has engaged the user input device 102b and the function of the user input device 102b has been activated. In some embodiments, the same or similar graphical indicators can be used to indicate the hovering or engagement state of the user input device 102b in the information block 210.

[0173] In alternative embodiments, an audio cue can be provided to replace or supplement the composite indicator to provide instructions or indicate spatial directions (e.g., up / down / left / right) to move the operator's foot to the hovering orientation of the user input device. The system can distinguish between hovering the foot over the pedal and actuating the pedal, and may have different visual and audio cues for the hovering state versus the engaged or actuated state. The system can also depict when the pedal function is valid or invalid. When the pedal function is invalid (e.g., when the instrument function cable is unplugged or the instrument function is not configured), the highlighted color may appear as gray.

[0174] As Figures 7A - 7D shown, the composite indicator that appears as a badge or label on a component in the field of view portion 202 can appear near the component and can be conditionally moved to remain visible and close to the component as the component or the endoscope generating the field of view moves. The composite indicator can be used for any of the purposes described above, but can also be used to identify a medical tool or other component in the field of view portion 202, identify the manipulator arm to which the medical tool is coupled, provide status information about the medical tool, provide operating information about the medical tool, or provide any other information about the tool or the manipulator arm to which it is coupled.

[0175] As Figure 7A shown, the composite indicator can be associated with the tool 502. In this embodiment, the composite indicator can be a badge (and is thus shown as badge 500) that is configured to have the appearance of a decal on the tool 502. The badge 500 can appear near the jaws 504a, 504b of the tool 502, but can be positioned to avoid obscuring the jaws. Based on the orientation uncertainty of the underlying kinematic tracking technology, this placement may include a deviation away from the jaws. The default position of the badge 500 can be at a predetermined key point 501 on the tool 502. As Figure 7AAs shown, the badge 500 can be placed at the key point 501 located at the U - clip of the tool. As the endoscope or tool 502 moves, the badge 500 can pivot and translate such that the badge 500 remains at the key point and is oriented along the surface of the U - clip. When the surface of the U - clip is no longer visible in the field - of - view portion 202, the badge 500 can be moved to another key point 503, such as Figure 7B shown (at a predetermined joint position) or such as Figure 7D shown (along the axis of the tool 502).

[0176] If the key - point position remains visible in the field - of - view portion 202, the badge 500 can remain at the original key - point position. Referring again to Figure 7B , since the normal of the badge 500 at the original key - point ( Figure 7A ) is no longer within the field - of - view portion 202, the badge 500 can be re - positioned to a second default key - point.

[0177] The orientation of the badge 500 at the key - point can be constrained such that the normal of the badge surface is within the field - of - view portion 202. If the badge 500 may not be oriented at the key - point such that the normal is within the field - of - view portion 202, the badge 500 can be moved to a different key - point. As Figure 7D shown, the orientation of the badge 500 can be pivoted to match the orientation of the tool 502 axis while the surface of the badge 500 remains visible to the viewer. The size of the badge 500 can also change as the distance of the key - point to which it is attached moves closer or farther from the distal end of the endoscope, or when the zoom function of the endoscope is activated. The badge size can be controlled to remain within a maximum threshold and a minimum threshold to avoid becoming too large or too small on the display. As Figure 7C shown, the badge 500 can be smaller because Figure 7C the key - point in Figure 7A is farther from the endoscope than the key - point in

[0178] Figure 8AFIG. 800 is a flowchart of the operation of control system 20 according to various embodiments described herein. At 802, control system 20 detects an object at a first threshold distance 128 or closer to one of user input devices 102 in input device tray 100. For example, range sensor 104 associated with one of the user input devices measures the distance of an object within the field of view 126 of range sensor 104. Range sensor 104 is positioned to have a field of view 126 across selection surface 120 of user input device 102. Control system 20 compares the measured distance from range sensor 104 with first threshold distance 128 to determine if the object is at first threshold distance 128 or within first threshold distance 128 (e.g., determines if the measured distance is less than or equal to first threshold distance 128).

[0179] At 804, control system 20 performs a first user interface action to provide feedback to surgeon S that an object is above or otherwise positioned to facilitate selection of one of user input devices 102. The first user interface action can be to provide visual feedback (e.g., via display system 35), audio feedback, and / or tactile feedback, such as described in the examples provided above.

[0180] At 806, control system 20 detects an object at a second threshold distance 130 or farther from user input device 102. Second threshold distance 130 is greater than first threshold distance 128. For example, range sensor 104 associated with user input device 102 measures the distance of an object within the field of view 126 of range sensor 104. Control system 20 compares the measured distance from range sensor 104 with second threshold distance 130 to determine if the object is at second threshold distance 130 or farther from second threshold distance 130 (e.g., determines if the measured distance is greater than or equal to second threshold distance 130).

[0181] At 808, control system 20 performs a second user interface action to provide feedback to surgeon S that the object is no longer above or otherwise positioned to facilitate selection of user input device 102. The second user interface action can be to discontinue providing visual feedback (e.g., via display system 35), provide another audio feedback, and / or provide another tactile feedback, such as described in the examples provided above.

[0182] Figure 8BFIG. 850 is a flow chart of example threshold-based hysteresis of control system 20 in accordance with various embodiments described herein. At 852, control system 20 determines whether an object is detected at or within a first threshold distance 128 of one of user input devices 102 in input device tray 100. For example, a ranging sensor 104 associated with one of the user input devices measures the distance of an object within the field of view 126 of ranging sensor 104. Ranging sensor 104 is positioned to have a field of view 126 that spans selection surface 120 of user input device 102. Control system 20 compares the measured distance from ranging sensor 104 to first threshold distance 128 to determine whether the object is at or within first threshold distance 128 (e.g., determines whether the measured distance is less than or equal to first threshold distance 128).

[0183] If an object is detected at or within first threshold distance 128 at 852, control system 20 proceeds to 854. At 854, control system 20 indicates that the object is positioned for selection. For example, control system 20 performs a first user interface action to provide the surgeon S with feedback that the object is above or otherwise positioned to facilitate selection of one of user input devices 102. The first user interface action can be providing visual feedback (e.g., via display system 35), audio feedback, and / or haptic feedback, such as described in the examples provided above.

[0184] At 806, control system 20 detects an object at or beyond a second threshold distance 130 of user input device 102. Second threshold distance 130 is greater than first threshold distance 128. For example, a ranging sensor 104 associated with user input device 102 measures the distance of an object within the field of view 126 of ranging sensor 104. Control system 20 compares the measured distance from ranging sensor 104 to second threshold distance 130 to determine whether the object is at or beyond second threshold distance 130 (e.g., determines whether the measured distance is greater than or equal to second threshold distance 130).

[0185] If an object is not detected at or within first threshold distance 128 at 852, control system 20 proceeds to 858. At 858, control system 20 indicates that the object is not positioned for selection. For example, control system 20 may not perform any action when transitioning from 852 to 858. Alternatively or additionally, control system 20 actively indicates that the object is not positioned for selection.

[0186] At 808, the control system 20 performs a second user interface action to provide feedback to the surgeon S that the object is no longer above or otherwise positioned to facilitate selection of the user input device 102. The second user interface action can be to abort providing visual feedback (e.g., via the display system 35), provide another audio feedback, and / or provide another tactile feedback, such as described in the examples provided above.

[0187] Figure 9 is a flowchart of a calibration operation 900 according to various embodiments described herein. At 902, a calibration object is placed within the field of view 126 of one or more rangefinding sensors 104. The calibration object is placed along the leading edge 118 of the user input device 102 such that the calibration object is within the field of view 126 at a first threshold distance 128. In various embodiments, the calibration object is selected to have a reflectivity or characteristic similar to that of the object used to select the user input device 102. For example, when the user input device 102 is a foot pedal, the calibration object is selected to have a reflectivity or characteristic similar to that of a shoe. Alternatively or additionally, the sensor threshold of the rangefinding sensor 104 can be adjusted based on the reflectivity of the shoe worn by the surgeon S. In some embodiments, the calibration object can be the shoe of a given surgeon.

[0188] At 904, the rangefinding sensor 104 of the user input device 102 measures the distance to the calibration object. For example, the rangefinding sensor 104 can generate a signal indicating the distance to the calibration object (e.g., a time signal, a signal strength value, etc.) and / or can generate a measured distance value (e.g., 75 mm). The control system 20 receives a signal indicating the distance and / or the measured distance value from the rangefinding sensor.

[0189] In various embodiments, the control system 20 receives multiple such distance measurements during the calibration operation. The control system 20 then performs an average, median, mean, or other statistical evaluation on the received rangefinding data to determine the measured distance to the calibration object.

[0190] At 906, the control system 20 stores the measured distance to the calibration object as the first threshold distance 128 of the user input device 102. At 908, the control system 20 calculates and stores a second threshold distance 130 based on the first threshold distance 128. For example, the control system 20 adds a predetermined distance 132 to the first threshold distance 128 to determine the second threshold distance 130.

[0191] Although the calibration operation 900 has been described above for one of the user input devices 102, the calibration operation 900 can be repeated for each user input device 102 in the user input panel 100.

[0192] Figure 10 is a flowchart for determining 1000 user intent according to various embodiments described herein. At 1002, the control system 20 tracks range data of one or more range sensors 104 over time as one or more time series of range data.

[0193] At 1004, the control system 20 evaluates the time series to determine user intent relative to one or more user input devices 102. For example, the control system 20 can determine the direction of movement and / or speed of an object relative to one or more user input devices 102 based on the time series. In some embodiments, the control system 20 resolves the three-dimensional orientation, direction of movement, and / or speed of the object. Movement above a first threshold speed and / or in a direction away from the user input device 102 can be determined as an intent not to select one of the user input devices. In contrast, movement below a second threshold speed and / or in a direction towards the user input device can be determined as an intent to select one of the user input devices.

[0194] At 1006, the control system 20 performs user interface actions based on the determined user intent. For example, after determining an intent not to select one of the user input devices 102, the control system 20 can ignore (e.g., for a predetermined period of time) any hover or selection events or otherwise require verification from the surgeon S for any hover or selection events. Alternatively or additionally, the control system 20 can animate the displayed graphical user interface or otherwise modify the displayed graphical user interface to provide an indication of the orientation, direction of movement, and / or speed of the object. Alternatively or additionally, the control system 20 can initiate one or more control actions associated with the user input device 102 (e.g., actions that require lead time, change the power state of a medical tool). Alternatively or additionally, the control system 20 can provide auditory or tactile feedback to the surgeon S.

[0195] It should be understood that the logical operations described herein with respect to the various figures can be implemented as: (1) in a computing device (e.g. Figure 11a sequence of computer-implemented acts or program modules (i.e., software) running on a computing device (as described in ), (2) interconnected machine logic circuits or circuit modules within the computing device (i.e., hardware), and / or (3) a combination of software and hardware of the computing device. Thus, the logical operations discussed herein are not limited to any particular combination of hardware and software. Implementation is a matter of choice depending on the performance and other requirements of the computing device. Thus, the logical operations described herein are variously referred to as operations, structural devices, acts, or modules. These operations, structural devices, acts, and modules may be implemented in software, firmware, special purpose digital logic, and any combination thereof. It should also be understood that more or fewer operations may be performed than those shown and described herein. These operations may also be performed in a different order than those described herein.

[0196] Reference Figure 11 , which shows an example computing device 1200 on which embodiments of the present invention may be implemented. For example, the computer processors located on the medical system 10, assembly 12, operator input system 16, control system 20, or auxiliary system 26 described herein may each be implemented as a computing device (e.g., computing device 1200). It should be understood that example computing device 1200 is only one example of a suitable computing environment on which embodiments of the present invention may be implemented. Optionally, computing device 1200 may be a well-known computing system, including but not limited to a personal computer, server, hand-held or laptop device, multiprocessor system, microprocessor-based system, network personal computer (PC), minicomputer, mainframe computer, embedded system, and / or a distributed computing environment including any of the above systems or devices in multiple. A distributed computing environment enables remote computing devices connected to a communication network or other data transmission medium to perform various tasks. In a distributed computing environment, program modules, applications, and other data may be stored on local and / or remote computer storage media.

[0197] In one embodiment, computing device 1200 may include two or more computers that communicate with each other and cooperate to perform tasks. For example but not limited to, an application may be partitioned in a way that allows for concurrent and / or parallel processing of the application's instructions. Alternatively, the data processed by the application may be partitioned in a way that allows for concurrent and / or parallel processing of different portions of the data set by two or more computers. In one embodiment, computing device 1200 may employ virtualization software to provide the functionality of a number of servers that are not directly bound to a number of computers in computing device 1200. For example, virtualization software may provide twenty virtual servers on four physical computers. In one embodiment, the functions disclosed above may be provided by executing one application and / or multiple applications in a cloud computing environment. Cloud computing may include providing computing services using dynamically scalable computing resources via a network connection. Cloud computing may be at least partially supported by virtualization software. A cloud computing environment may be established by an enterprise and / or leased from a third-party provider as needed. Some cloud computing environments may include cloud computing resources owned and operated by an enterprise, as well as cloud computing resources leased and / or rented from a third-party provider.

[0198] In its most basic configuration, computing device 1200 generally includes at least one processing unit 1220 and system memory 1230. Depending on the exact configuration and type of the computing device, system memory 1230 may be volatile (such as random access memory (RAM)), non-volatile (such as read-only memory (ROM), flash memory, etc.), or some combination of the two. This most basic configuration is shown by the dashed line 1210 in Figure 11 Figure. Processing unit 1220 may be a standard programmable processor that performs the arithmetic and logical operations necessary for the operation of computing device 1200. Although only one processing unit 1220 is shown, multiple processors may exist. Thus, although instructions may be discussed as being executed by a processor, the instructions may be executed by one or more processors simultaneously, serially, or otherwise. Computing device 1200 may also include a bus or other communication mechanism for passing information between the various components of computing device 1200.

[0199] The computing device 1200 may have additional features / functions. For example, the computing device 1200 may include additional storage devices, such as a removable storage device 1240 and a non-removable storage device 1250, which include but are not limited to magnetic disks or optical disks or magnetic tapes. The computing device 1200 may also include a network connection 1280 that allows the device to communicate with other devices, for example, via the communication paths described herein. The network connection 1280 may take the form of: a modem, a modem group, an Ethernet card, a Universal Serial Bus (USB) interface card, a serial interface, a token ring card, a Fiber Distributed Data Interface (FDDI) card, a Wireless Local Area Network (WLAN) card, a radio transceiver card (such as Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), Long Term Evolution (LTE), Worldwide Interoperability for Microwave Access (WiMAX), and / or other air interface protocol radio transceiver cards), and other well-known network devices. The computing device 1200 may also have an input device 1270, such as a keyboard, a keypad, a switch, a dial, a mouse, a trackball, a touch screen, a voice recognizer, a card reader, a paper tape reader, or other well-known input devices. An output device 1260, such as a printer, a video monitor, a Liquid Crystal Display (LCD), a touch screen display, a display, a speaker, etc., may also be included. Additional devices may be connected to the bus to facilitate data communication between the components of the computing device 1200. All of these devices are well known in the art and do not need to be discussed in detail herein.

[0200] The processing unit 1220 may be configured to execute program code encoded in a tangible computer-readable medium. A tangible computer-readable medium refers to any medium that can provide data that causes the computing device 1200 (i.e., a machine) to operate in a particular manner. Various computer-readable media may be used to provide instructions to the processing unit 1220 for execution. Exemplary tangible computer-readable media may include but are not limited to volatile media, non-volatile media, removable media, and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. The system memory 1230, the removable storage device 1240, and the non-removable storage device 1250 are all examples of tangible computer storage media. Exemplary tangible computer-readable recording media include but are not limited to integrated circuits (e.g., Field Programmable Gate Arrays or Application Specific ICs), hard disks, optical disks, magneto-optical disks, floppy disks, magnetic tapes, holographic storage media, solid state devices, RAM, ROM, Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, or other memory technologies, CD-ROM, Digital Versatile Disk (DVD), or other optical storage devices, magnetic tape cartridges, magnetic tapes, magnetic disk storage devices, or other magnetic storage devices.

[0201] For the fields of electrical engineering and software engineering, it is important that functions implementable by loading executable software into a computer can be transformed into hardware implementation means by well-known design rules. The decision between software and hardware implementation concepts usually depends on considerations of design stability and the number of units to be produced, rather than any issues involved in converting from the software domain to the hardware domain. Generally, a design that is still undergoing frequent changes may be more likely to be implemented in software because re-spinning the hardware implementation is more expensive than re-spinning the software design. Generally, a stable design that will be mass-produced may be more likely to be implemented in hardware (e.g., implemented in an application-specific integrated circuit (ASIC)) because for large-scale production runs, the hardware implementation may be cheaper than the software implementation. Generally, a design can be developed and tested in software form and then transformed into an equivalent hardware implementation in an application-specific integrated circuit that hard-wires the instructions of the software according to well-known design rules. In the same way that a machine controlled by a new ASIC is a specific machine or device, similarly, a computer that has been programmed and / or loaded with executable instructions can be regarded as a specific machine or device.

[0202] In an example embodiment, the processing unit 1220 may execute program code stored in the system memory 1230. For example, the bus may transfer data to the system memory 1230, and the processing unit 1220 receives and executes instructions from the system memory 1230. Before or after being executed by the processing unit 1220, the data received by the system memory 1230 may optionally be stored on the removable storage device 1240 or the non-removable storage device 1250.

[0203] It should be understood that the various techniques described herein can be implemented in hardware or software, or in a combination thereof as appropriate. Thus, the methods and apparatuses or certain aspects or portions of the presently disclosed subject matter may take the form of program code (i.e., instructions) embodied in a tangible medium (such as a floppy disk, CD-ROM, hard disk drive, or any other machine-readable storage medium), where, when the program code is loaded into and executed by a machine (such as a computing device), the machine becomes an apparatus for practicing the presently disclosed subject matter. In the case of executing program code on a programmable computer, the computing device generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. One or more programs may implement or utilize the processes described in connection with the presently disclosed subject matter (e.g., by using an application programming interface (API), reusable controls, etc.). Such programs may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the programs may be implemented in assembly language or machine language. In any case, the language may be a compiled or interpreted language, and it may be combined with a hardware implementation.

[0204] Embodiments of the methods and systems may be described herein with reference to block diagrams and flowchart illustrations of methods, systems, apparatuses, and computer program products. It should be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, respectively, can be implemented by computer program instructions. These computer program instructions may be loaded onto a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions executed on the computer or other programmable data processing apparatus create means for implementing the functions specified in one or more flowchart blocks.

[0205] These computer program instructions may also be stored in a computer-readable memory, which can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including computer-readable instructions for implementing the functions specified in one or more flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus, thereby producing a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one or more flowchart blocks.

[0206] Accordingly, each block of the block diagrams and flowchart illustrations supports a combination of devices for performing a particular function, a combination of steps for performing a particular function, and program instruction means for performing a particular function. It should also be understood that each block in the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, can be implemented by a computer system based on dedicated hardware for performing a particular function or step, or by a combination of dedicated hardware and computer instructions.

[0207] Although several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The current examples are to be considered illustrative and not restrictive, and are not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or some features may be omitted or not implemented.

[0208] Furthermore, without departing from the scope of the present disclosure, the techniques, systems, subsystems, and methods described and illustrated as discrete or separate in various embodiments may be combined or integrated with other systems, modules, techniques, or methods. Other items shown or discussed as being directly coupled or communicating with each other may be indirectly coupled or communicating through some interface, device, or intermediate component, whether electrical, mechanical, or otherwise. Those skilled in the art can identify other examples of changes, substitutions, and alterations, and can make such changes, substitutions, and alterations without departing from the spirit and scope disclosed herein.

Claims

1. An object detection and visual feedback system, comprising: A button positioned for selection by an object, the button including a selection surface; A ranging sensor having a field of view across the selection surface of the button, wherein the ranging sensor is configured to measure the distance of an object within the field of view; And A user interface configured to perform a first action when the distance measured by the ranging sensor is less than or equal to a first threshold distance, wherein the user interface is further configured to perform a second action when the distance measured by the ranging sensor is greater than or equal to a second threshold distance, wherein the second threshold distance is greater than the first threshold distance.

2. The system according to claim 1, wherein the first threshold distance is at the edge of the button.

3. The system according to claim 2, wherein the second threshold distance does not co - extend with the button and is spaced apart from the edge of the button by a predetermined distance.

4. The system according to claim 3, wherein the predetermined distance is sufficient to reduce an accidental selection of the button by the object.

5. The system according to claim 4, wherein the predetermined distance is in the range of 10 mm to 30 mm.

6. The system according to any one of claims 1 - 5, wherein the object is any one of a non - hand limb, foot, leg, knee, head, elbow, or an extension from the human anatomy.

7. The system according to any one of claims 1 - 6, wherein the range - finding sensor is any one of the following: a time - of - flight distance sensor, a triangulation distance sensor, an optical distance sensor, an acoustic distance sensor, an inductive distance sensor, a capacitive distance sensor, a photoelectric distance sensor, a camera, an infrared distance sensor, a laser rangefinder, or a light detection and ranging sensor.

8. The system according to any one of claims 1 - 7, wherein the direction of the field of view is parallel to the direction of the distance measured by the range - finding sensor.

9. The system according to any one of claims 1 - 7, wherein the direction of the field of view is orthogonal to the direction of the distance measured by the range - finding sensor.

10. The system according to any one of claims 1 - 9, further comprising: A second button positioned for selection by an object, the second button including a second selection surface; And A second ranging sensor having a second field of view across the second selection surface of the second button, wherein the second ranging sensor is configured to measure the distance of an object within the second field of view; Wherein the user interface is configured to perform a third action when the distance measured by the second ranging sensor is less than or equal to a third threshold distance.

11. The system according to claim 10, wherein the third action is different from the first action.

12. The system according to any one of claims 1 - 11, wherein the first action is to display an indication of the button.

13. The system according to claim 12, wherein the second action is to stop displaying the indication of the button.

14. The system according to any one of claims 12 - 13, wherein the indication of the button is selected from a group of indications consisting of: an indication of the function to be performed when the button is selected; a change in the intensity of an icon displayed on the user interface; and a graphic of the object relative to the layout of the selection button, the layout including the button.

15. The system according to any one of claims 1 - 11, wherein the first action is to issue a first audible alarm, and the second action is to issue a second audible alarm.

16. The system according to any one of claims 1 - 11, wherein the first action is to provide a first tactile feedback, and the second action is to provide a second tactile feedback.

17. The system according to claim 16, wherein the first tactile feedback and the second tactile feedback are provided to a hand controller.

18. A robotic surgical system, comprising: A user interface; And A footrest including: A button positioned for selection by a user's foot, the button including a selection surface; And A ranging sensor positioned to have a field of view across the selection surface of the button, wherein the ranging sensor is configured to measure the distance of an object within the field of view, Wherein the user interface is configured to perform a first action associated with the button when the distance measured by the ranging sensor is less than or equal to a first threshold distance, wherein the user interface is further configured to perform a second action associated with the button when the distance measured by the ranging sensor is greater than or equal to a second threshold distance, wherein the second threshold distance is greater than the first threshold distance.

19. The robotic surgical system according to claim 18, wherein the footrest is positioned on the base of the robotic surgical system.

20. The robotic surgical system according to any one of claims 18 - 19, wherein the first threshold distance is at or near the edge of the button.

21. The robotic surgical system according to claim 20, wherein the second threshold distance does not co - extend with the button and is spaced apart from the edge of the button by a predetermined distance.

22. The robotic surgical system according to claim 21, wherein the predetermined distance is sufficient to reduce the accidental selection of the button by the user's foot.

23. The robotic surgical system according to claim 22, wherein the predetermined distance is in the range of 10 mm to 30 mm.

24. The robotic surgical system according to any one of claims 18 - 23, wherein the range - finding sensor is any one of the following: a time - of - flight distance sensor, a triangulation distance sensor, an optical distance sensor, an acoustic distance sensor, an inductive distance sensor, a capacitive distance sensor, a photoelectric distance sensor, a camera, an infrared distance sensor, a laser rangefinder, or an optical detection and ranging sensor.

25. The robotic surgical system according to any one of claims 18 - 24, wherein the direction of the field of view is parallel to the direction of the distance measured by the range sensor.

26. The robotic surgical system according to any one of claims 18 - 24, wherein the direction of the field of view is orthogonal to the direction of the distance measured by the range sensor.

27. The robotic surgical system according to any one of claims 18 - 26, wherein the footrest further comprises: A second button positioned for selection by a user's foot, the second button including a second selection surface; And A second ranging sensor positioned to have a second field of view across the second selection surface of the second button, wherein the second ranging sensor is configured to measure a second distance of an object within the second field of view, Wherein the user interface is further configured to perform a third action associated with the second button when the second distance measured by the second ranging sensor is less than or equal to a third threshold distance.

28. The robotic surgical system according to claim 27, wherein the third action is different from the first action.

29. The robotic surgical system according to any one of claims 18 - 28, wherein the user interface includes a display.

30. The robotic surgical system according to claim 29, further comprising a headrest, wherein the display is incorporated into the headrest.

31. The robotic surgical system according to claim 30, wherein the display is a stereoscopic display.

32. The robotic surgical system according to any one of claims 18 - 31, wherein the first action is to display an indication of the button.

33. The robotic surgical system according to claim 32, wherein the second action is to stop displaying the indication of the button.

34. The robotic surgical system according to any one of claims 32 - 33, wherein the indication of the button is selected from a group of indications consisting of: an indication of a function to be performed when the button is selected; a change in the intensity of an icon displayed on the user interface; and a graphic of the user's foot relative to the layout of a plurality of buttons, the layout including the button.

35. The robotic surgical system according to any one of claims 18 - 31, wherein the first action is to emit a first audible alarm, and the second action is to emit a second audible alarm.

36. The robotic surgical system according to any one of claims 18 - 31, wherein the first action is to provide a first tactile feedback, and the second action is to provide a second tactile feedback.

37. The robotic surgical system according to claim 36, wherein the first haptic feedback and the second haptic feedback are provided to the hand controller.

38. A method for providing feedback upon detection of an object, the method comprising: Measuring the distance of an object within the field of view of a ranging sensor positioned to have a field of view across a selection surface of a button, wherein the button is positioned for selection by the object; Performing a first action using a user interface when the distance measured by the ranging sensor is less than or equal to a first threshold distance; And Performing a second action using the user interface when the distance measured by the ranging sensor is greater than a second threshold distance, wherein the second threshold distance is greater than the first threshold distance.

39. The method according to claim 38, wherein the first threshold distance is at an edge of the button.

40. The method according to claim 39, wherein the second threshold distance does not co - extend with the button and is spaced apart from the edge of the button by a predetermined distance.

41. The method according to claim 40, wherein the predetermined distance is sufficient to mitigate an accidental selection of the button by the object.

42. The method according to claim 41, wherein the predetermined distance is in a distance range of 10 mm to 30 mm.

43. The method according to any one of claims 38 - 42, wherein the object is any one of a non - hand limb, a foot, a leg, a knee, a head, an elbow, or an extension from the human anatomy.

44. The method according to any one of claims 38 - 43, wherein the range - finding sensor is any one of the following: a time - of - flight distance sensor, a triangulation distance sensor, an optical distance sensor, an acoustic distance sensor, an inductive distance sensor, a capacitive distance sensor, a photo - electric distance sensor, a camera, an infrared distance sensor, a laser rangefinder, or a light detection and ranging sensor.

45. The method according to any one of claims 38 - 44, wherein the direction of the field of view is parallel to the direction of the distance measured by the range - finding sensor.

46. The method according to any one of claims 38 - 45, wherein the direction of the field of view is orthogonal to the direction of the distance measured by the range - finding sensor.

47. The method according to any one of claims 38 - 46, further comprising: Measure a second distance to a second object within a second field of view of a second ranging sensor, the second ranging sensor being positioned to have the second field of view across a second selection surface of a second button, wherein the second button is positioned for selection by the object; When the second distance measured by the ranging sensor is less than or equal to a third threshold distance, perform a third action using the user interface; and When the second distance measured by the second ranging sensor is greater than a fourth threshold distance, perform a fourth action using the user interface, wherein the fourth threshold distance is greater than the third threshold distance.

48. The method according to claim 47, wherein the third action is different from the first action.

49. The method according to any one of claims 38 - 48, wherein the first action is to display an indication of the button.

50. The method according to claim 49, wherein the second action is to stop displaying the indication of the button.

51. The method according to any one of claims 49 - 50, wherein the indication of the button is selected from a group of indications consisting of: an indication of a function to be performed when the button is selected; a change in intensity of an icon displayed on the user interface; and a graphic of the object relative to the layout of the selection button, the layout including the button.

52. The method according to any one of claims 38 - 48, wherein the first action is to emit a first audible alarm, and the second action is to emit a second audible alarm.

53. The method according to any one of claims 38 - 48, wherein the first action is to provide a first tactile feedback, and the second action is to provide a second tactile feedback.

54. The method according to claim 53, wherein the first tactile feedback and the second tactile feedback are provided to a hand controller.

Images