Systems and methods for indicating an event outside of an imaging system field of view

Abstract

A method may comprise detecting an event and determining a location of the event with respect to a frame of reference. The method may also comprise determining that the location of the event is outside a field of view of an endoscopic imaging system and displaying, within the field of view of the endoscopic imaging system, a directional indicator indicating a direction of the location of the event outside the field of view.

Description

Docket No. P06987-WO (70228.978WO01) Customer No. 160596SYSTEMS AND METHODS FOR INDICATING AN EVENT OUTSIDE OF AN IMAGING SYSTEM FIELD OF VIEWCROSS-REFERENCED APPLICATIONS

[0001] This application claims priority to and benefit of U.S. Provisional Applications No. 63 / 724,173 filed November 22, 2024 and entitled “Systems and Methods for Indicating an Event Outside of an Imaging System Field of View,” which is incorporated by reference herein in its entirety.FIELD

[0002] The present disclosure is directed to systems and methods for indicating an event outside of an imaging system field of view.BACKGROUND

[0003] Minimally invasive medical techniques are intended to reduce the amount of extraneous tissue that is damaged during diagnostic or surgical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques may be performed through natural orifices in a patient anatomy or through one or more surgical incisions. Through these natural orifices or incisions, clinicians may insert medical tools to reach a target tissue location. Minimally invasive medical tools include instruments such as therapeutic instruments, diagnostic instruments, and surgical instruments. Minimally invasive medical tools may also include imaging instruments such as endoscopic instruments that provide a viewer with an image of a field of view within the patient anatomy.

[0004] Some minimally invasive medical tools may be robot-assisted including teleoperated, remotely operated, or otherwise computer-assisted. During a medical procedure, the clinician may view an image of a field of view of the patient anatomy. Various graphical user interface elements may be presented with an image of the field of view. Improved systems and methods are needed for presentation of graphical user interface elements that provide an indication of an event outside of the field of view of the imaging system.SUMMARY

[0005] The embodiments of the invention are best summarized by the claims that follow the description.

[0006] In one example, method may comprise detecting an event and determining a location of the event with respect to a frame of reference. The method may also comprise determining that the location of the event is outside a field of view of an endoscopic imaging system and4910-7717-3622 v. l 1Docket No. P06987-WO (70228.978WO01) Customer No. 160596 displaying, within the field of view of the endoscopic imaging system, a directional indicator indicating a direction of the location of the event outside the field of view.

[0007] In another example, a medical system may comprise a display system and

[0008] a control system. The control system may include a processing unit may include one or more processors. The processing unit may be configured to detect an event, determine a location of the event with respect to a frame of reference, determine that the location of the event is outside a field of view of an endoscopic imaging system, and display a directional indicator, within the field of view of the endoscopic imaging system, indicating a direction of the location of the event outside the field of view.

[0009] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.BRIEF DESCRIPTIONS OF THE DRAWINGS

[0010] FIG. 1A illustrates a graphical user interface including a directional indicator for an event outside of a field of view of an imaging system, according to some examples.

[0011] FIG. IB illustrates a graphical user interface including a virtual marker indicating a location of the event outside of a field of view in FIG. 1 A, according to some examples.

[0012] FIG. 2 illustrates a method for providing an indicator of an event outside of the field of view of an imaging system, according to some examples.

[0013] FIG. 3A and 3B illustrate a graphical user interface including a directional indicator for a tool collision event outside of a field of view of an imaging system, according to some examples.

[0014] FIG. 3C illustrates a graphical user interface after a resolution of the tool collision event outside of a field of view in FIGS. 3A and 3B, according to some examples.

[0015] FIG. 3D illustrates a graphical user interface including a field of view image that includes the tool collision event that is outside the field of view in FIGS. 3A and 3B, according to some examples.

[0016] FIG. 4 illustrates a schematic view of a medical system, according to some examples.

[0017] FIG. 5 is a perspective view of a manipulator assembly of the medical system of FIG. 4, according to some examples.4910-7717-3622 v. l 2Docket No. P06987-WO (70228.978WO01) Customer No. 160596

[0018] FIG. 6 is a front elevation view of an operator’s console in a robot-assisted medical system, according to some examples.

[0019] Embodiments of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, wherein showings therein are for purposes of illustrating embodiments of the present disclosure and not for purposes of limiting the same.DETAILED DESCRIPTION

[0020] Medical systems, including robot-assisted medical systems such as medical system 1010 described below, may include an endoscopic imaging system that presents imagery to a viewer, such as a surgeon, via a display that emulates peering through a viewing window into an anatomical space. The endoscopic imaging system may include any of a variety of rigid or flexible endoscopic instruments that extend into a surgical site or through a natural patient orifice to gather images from within the patient anatomy. In some examples, image data from a field of view of an endoscopic imaging system may be displayed to a viewer as separate right and left eye viewing volumes on a stereoscopic display system. For a stereoscopic display, the viewing window may generally correspond to an area at the convergence plane of the right and left eye viewing volumes. Endoscopic images of the anatomical space may provide a clinician with an image of a field of view of the patient anatomy and any medical tools located in the patient anatomy. Various graphical user interface elements may be incorporated with the field of view image. Graphical user interface elements may include a directional indicator that points to the location of an event outside of the field of view image. Such events may include, for example, an instrument shaft collision, an instrument break, an object deposition in the tissue, or a virtual marker deposition.

[0021] FIG. 1A illustrates a graphical user interface 100 including a directional indicator 102 for an event outside of a viewing window or field of view image 104 of an imaging system (e.g., imaging system 1015). A location 106 of the event is outside of the field of view image 104 and thus not visible in FIG. 1A. If the imaging system moves or bends, the field of view of the imaging system may change. FIG. IB illustrates the graphical user interface 100 with the field of view image 104 shifted to view the location 106 of the event that was indicated by the directional indicator 102 in FIG. 1A. In FIG. IB, the location 106 of the event is identified with a virtual marker 108, but in other examples, the location of the event may be detected and recorded (e.g., by the control system 1020) without being marked with a visual marker. As4910-7717-3622 v. l 3Docket No. P06987-WO (70228.978WO01) Customer No. 160596 described in greater detail below, a detected event may include, for example, an interaction between two or more instruments within the patient anatomy or outside the patient anatomy, an interaction between patient tissue and an instrument, a placement or deposition of a physical object within the patient anatomy, a placement of a virtual object at a recorded location relative to the patient anatomy, or any other detectable physical or virtual occurrence.

[0022] As shown in FIG. 1A, the graphical user interface 100 may include the field of view image 104, which may be a stereoscopic image of the field of view, and the overlaid directional indicator 102. In this example, anatomic tissue 1 10 and an instrument 1 12 may be visible in the field of view image. The directional indicator 102 be two-dimensional or three-dimensional in appearance and may include a pointing portion 120 and an icon portion 122. The pointing portion 120 may be oriented (e.g., in 2D or 3D) to point in the direction of the out-of-view event location 106. The angle or pointedness of the pointing portion 120 may be dynamic based on the distance between the directional indicator 102 and the location of the out-of-view event location 106. As the event location 106 gets closer to the field of view, the pointing portion 120 of the indicator 102 may become wider, rounder, or otherwise generally less pointed. As the imaging system moves or bends, the directional indicator 102 may change position and / or the pointing portion 120 may pivot or reorient to remain trained on or pointed in the direction of the out-of-view event location 106.

[0023] In some examples, the icon portion 122 may include a graphic associated with a type of the event (e.g., an instrument collision, a foreign object placement). In some examples, the directional indicator 102 or a portion thereof, such as the icon portion 122, may provide a visual gesture such as an oscillation, a bounce, a flash, a strobe, or other visual effect to attract a viewer’s attention or provide information. In some examples, the directional indicator 102 or icon portion 122 may change color, size, shape, or otherwise alternate a characteristic to attract attention or provide status or indicate a level of alarm. In some examples, the graphic of the icon portion 122 may change based on the detected event (e.g., a sensed or determined value associated with the detected event, a determined characteristic of the detected event, etc.). For instance, the detected event can be a contact between two or more instrument shafts. Contact may include, for example, a collision, sliding, rubbing, or other interaction between the shafts. Contact may include interaction between instruments that are both in motion, both at rest, or with one at rest and one in motion. Sensed or determined values and / or characteristics of the contact can be used in the display of the directional indicator 102 or the icon portion 122. In one example, an amount or level of bending of an instrument shaft due to the detected contact can be estimated. The directional indicator 102 or the icon portion 122 may be displayed based4910-7717-3622 v. l 4Docket No. P06987-WO (70228.978WO01) Customer No. 160596 on the estimated amount or level of bending. The icon portion 122 (or a portion thereof) may be displayed in a first color (e.g., yellow) and / or in accordance with a first display size in the event that the amount or level of bending is below some threshold level and in a second color (e.g., red) and / or in accordance with a second display size (e.g., bigger than the first display size) a when the amount or level of bending exceeds the threshold. In addition, or as an alternative, the system may determine an impact of the contact (e.g., measured by an inertial measurement unit disposed on a manipulator arm assembly) between the two or more instruments and the display of the directional indicator 102 or the icon portion 122 may be based on the measured impact of the contact. Still further, the display of the directional indicator 102 or the icon portion may be based on another determined characteristic of the detected contact between the two or more instruments. For example, if the two or more instruments are in motion and in active contact or collision with one another, a visual gesture (e.g., an oscillation, a bounce, a flash, a strobe, or other visual effect) can be provided in the display of the directional indicator 102 or the icon portion 122. In contrast, if the two or more instruments are in contact but not in motion, the visual gesture may be omitted. In some examples, as the imaging system moves and the pointing portion 120 changes orientation to remain trained on the event location 106, the icon portion 122 may adjust to remain in a consistent presentation orientation such as vertically upright. The orientations of the icon portion 122 and the pointing portion 120 of the directional indicator 102 may be independent of each other.

[0024] The graphical user interface 100 may also include, for example, banner messages, a status information, manipulator arm information, an interactive menu, alerts, user messages, and / and auxiliary windows. For example, a notification banner 124 may provide instructions or alerts to an operator. For example, a status banner 126 may provide information about a state of the medical system, including instruments controlled by the medical system. Information related on the status of manipulator arms of the medical system may be displayed in pods 128.

[0025] FIG. 2 illustrates a flowchart illustrating a method 200 for providing an indicator of an event outside of the field of view or viewing window of an imaging system. The methods described herein are illustrated as a set of operations or processes and are described with continuing reference to additional figures. Not all of the illustrated processes may be performed in all embodiments of the methods. Additionally, one or more processes that are not expressly illustrated in may be included before, after, in between, or as part of the illustrated processes. In some embodiments, one or more of the illustrated processes may be omitted. In some embodiments, one or more of the processes may be implemented, at least in part, in the form4910-7717-3622 v. l 5Docket No. P06987-WO (70228.978WO01) Customer No. 160596 of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors (e.g., the processing units of a control system such as control system 1020) may cause the one or more processors to perform one or more of the processes. In one or more embodiments, the processes may be performed by a control system.

[0026] At a process 202, an event may be detected. An event may include any of a variety of occurrences in a real or virtual medical environment. As an example, an event may be detected within the patient anatomy but outside of the field of view image 104, as illustrated in FIG. 1 A. The event may include, for example, an interaction between two or more instruments within the medical environment, an interaction between patient tissue and an instrument, a placement or deposition of a physical object within the medical environment, a placement of a virtual object at a recorded location within the medical environment, or any other physical or virtual occurrence detectable within the medical environment. In various examples, the event may occur within a patient anatomy, external to a patient anatomy, in a virtual procedure such as a virtual training in which a patient may be absent, or in a medical demonstration in which a patient may be absent.

[0027] Any of a variety of detection modalities may be used for detecting an event. In some examples, the detection modality may include kinematic modeling that describes the position and / or motion of components of a medical system (e.g., the medical system 1010), including arms (e.g., arms of the assembly 1012) and / or instrument systems (e.g., instrument system 1014), within the medical environment. In some examples, the detection modality may include digital image analysis of endoscopic image data that may be registered with tracked instruments that are visible or outside the view of the endoscopic image data. In some examples, the detection modality may include sensors on a robot-actuated medical assembly (e.g., the assembly 1012). In some examples, the detection modality may include sensors on instruments (e.g., instrument systems 1014, 1015) coupled to the robot-actuated medical assembly. Any of a variety of sensor modalities may be used including an electromagnetic position sensor, an inertial measurement unit, an accelerometer, or an optical fiber shape sensor. In some examples, the detection modality may include a record or history of commanded motions or actions by a control system (e.g., control system 1020) such as throwing of a suture stitch, depositing an implant, or delivering of an ablation or other treatment. In some examples, the detection modality may include an operator input at an input control device (e.g., input control device 1036) of an operator console, at a touchscreen (e.g., a display system of the auxiliary system 1026) for input of virtual markers or annotations, or at another type of operator interface. In some examples, the detection modality may include any instrument system (e.g.,4910-7717-3622 v. l 6Docket No. P06987-WO (70228.978WO01) Customer No. 160596 instrument systems 1014, 1015 or auxiliary system 1026) registered to another component of a medical system, a frame of reference of the medical system, or a frame of reference of a patient in the medical environment. Examples of a registered instrument system may include, for example, an ultrasound probe registered to an endoscopic imaging system, an endoluminal instrument system such as the Ion Robotic Bronchoscopy system provided by Intuitive Surgical, Inc. Examples of a registered instrument system may also include, for example, a force feedback sensor system that may detect a force against a tissue or other obstructions encountered by an instrument, including instruments outside of a view of an anatomic imaging system.

[0028] In some examples, a detected event may occur within a patient anatomy but outside of an imaging system field of view. For example, a detected event may include a collision of instrument shafts extending into the patient anatomy. In some examples, the collision event may be detected by an observed bending of the instrument shaft beyond a bend threshold. The bending may be observed from an image analysis, kinematic modeling, sensor data, etc. In some examples, the collision event may be detected by a non-responsiveness or unexpected response of an instrument to a commanded motion. In some examples, the collision event may be detected by instrument shaft contact as observed in kinematic modeling or by instrument sensors.

[0029] In some examples, a detected event may include an object deposited in a real or virtual patient anatomy. For example, an object such as a suture, an implant, or a radiopaque marker may be deposited and the deposition event may be detected by kinematic modeling or the commanded motion record. In some examples, a deposition event may be detected by digital image analysis. For example, an unintentional event, such as an instrument breakage that results in portions of an instrument being deposited in an anatomy, may be detected by image analysis. In some examples, the deposition of the object and a detection that the object has moved out of a field of view of the imaging system may be considered discrete and separate events.

[0030] In some examples, a detected event may include a virtual marker placed in the patient anatomy. The placement of a virtual marker may include a registration of the virtual marker relative to a 2D or 3D frame of reference such as a frame of reference associated with a patient anatomy and / or the imaging system of the field of view. In some examples, an operator input (e.g. a menu selection, a motion of an input control device) associated with an annotation or virtual marker deposition may be recognized as a detected event. As another example, a virtual marker associated with a co-registered imaging modality, such as a fluoroscopic imaging4910-7717-3622 v. l 7Docket No. P06987-WO (70228.978WO01) Customer No. 160596 modality, may be recorded. As another example, a virtual marker may be a planned termination location for an instrument in an instrument exchange procedure. In some examples, the deposition of the virtual object and a detection that the virtual object has moved out of a field of view of the imaging system may be considered discrete and separate events.

[0031] In some examples, a detected event may include a determination that a recognized object has moved out of the field of view of an imaging system either because the object has moved relative to the imaging system or because the imaging system has moved relative to the object. Detection of such an event may include monitoring a position of the object relative to the imaging system field of view.

[0032] At a process 204, a location of the detected event with respect to a frame of reference may be determined. In various examples, the frame of reference may be a two-dimensional or three-dimensional frame of reference for the patient anatomy (e.g., a patient P having patient reference frame Xp, Yp, Zp), a two-dimensional or three-dimensional frame of reference for an imaging system having a field of view (e.g., imaging system 1028 having imaging system reference frame Xi, Yi, Zi), and / or a two-dimensional or three-dimensional frame of reference for a medical system assembly (e.g., assembly 1012 having medical system assembly reference frame XM, YM, ZM). In various examples, one or more frames of reference may be co-registered such that a location of an event known in a frame of reference is also known in co-registered frames of reference. In various examples, determining the location of the detected event may include translating the location of the detected event from one frame of reference to another. In some examples, the imaging system reference frame may be determined based on kinematic information for the imaging system including, for example, a kinematic measure of the distal end portion of the endoscope generating the field of view. A position and / or orientation of the distal end portion may be used to determine relative distance to structures visible in the field of view. Kinematic information may additionally or alternatively include information about orientation, velocity, acceleration, and / or other movement information.

[0033] In some examples, the location of the detected event may be determined and recorded, for example by a control system (e.g. control system 1020) and / or the detection modality, contemporaneously with the occurrence of the event. In some examples, the location of the detected event may be determined and recorded a period of time after the occurrence of the event. In various examples, the location of the event may be different from a predetermined location on or relative to an instrument. For example, the location of a detected collision event between instrument shafts may be at the intersection location or contact location between the instrument shafts, and the location of the detected event may shift as the instruments are4910-7717-3622 v. l 8Docket No. P06987-WO (70228.978WO01) Customer No. 160596 adjusted but remain in a collided state. Rather than associate a location of a collision event with a default or predetermined location of an involved instrument (e.g., a clevis, jaw area, or instrument tip), the location of the detected event may correspond to the location of the collision (e.g., the location of the intersection or contact point) and the location may shift as the location of the detected collision shifts.

[0034] As an example, a location 106 may be determined for the event detected in FIGS. 1 A and IB. The location 106 may be known in a patient frame of reference (e.g., the patient reference frame Xp, Yp, Zp) and / or in a co-registered frame of reference such as imaging system frame of reference (e.g., the imaging system frame of reference Xi, Yi, Zi) or manipulator assembly frame of reference (e.g., the manipulator assembly frame of reference XM, YM, M).

[0035]

[0036] At a process 206, a determination may be made that the location of the detected event is outside a field of view of an imaging system. In some examples, an imaging system (e.g. imaging system 1015) may be, for example a stereoscopic endoscope that is extendable into a patient anatomy to obtain stereoscopic image data of the anatomic issue, instruments, or other structures in a field of view of the imaging system. Image data from a stereo endoscopic imaging system may be displayed to a viewer as separate right and left eye viewing volumes on a stereoscopic display system. The content of the intersection of the right and left eye viewing volumes may be the field of view visible to a viewer. The boundaries of a current field of view of an imaging system, such as an endoscopic imaging system, may be determined in the imaging system frame of reference and compared to the determined location of the detected event in the imaging system frame of reference to determine if the location of the detected event is outside or inside the field of view of the imaging system. In various examples, the boundaries of the current field of view of the imaging system may be translated to registered reference frames in which the location of the detected event is determined. The comparison of the location of the detected event may then be made in the registered frame of reference. If an event is outside the field of view of the imaging system, the viewer may be unaware of the event and any associated consequences or risks that may affect the patient or the medical procedure.

[0037] In the example of FIGS. 1 A and IB, the location 106 may be determined to be outside the field of view image 104 by comparing the boundaries of the field of view image 104 to the location 106 in a common frame of reference.

[0038] At a process 208, a directional indicator may be displayed at a display location within the field of view to indicate the direction of the determined location of the detected event4910-7717-3622 v. l 9Docket No. P06987-WO (70228.978WO01) Customer No. 160596 outside of the field of view. In some examples, the field of view and the directional indicator may be displayed on a display system at an operator console (e.g. display system 1035) or a display system at an auxiliary system (e.g., auxiliary system 1026) in the medical environment. In some examples, displaying the directional indicator may include determining a location, such as a two or three dimensional location, within the field of view to display the directional indicator. Various constraints may be applied to determine the location or a set of appropriate locations for displaying the directional indicator. For example, the location of the directional indicator may be constrained to a location within the boundaries of both the right and left eye viewing volumes. Additionally or alternatively, the three dimensional display of the directional indicator may be constrained so that pointing portion is visible in the field of view. For example, the directional indicator 102 may be constrained so that pointing portion 120 is not displayed exactly parallel to the Z-axis. In various examples, a line may be determined between a central point of the field of view and the determined location of the detected event. The directional indicator may be displayed along the line in a region within the right and left eye viewing volumes. In some examples, the directional indicator may be displayed along the line at a predetermined distance from the closest boundary of the combined left and right viewing volumes. In some examples, the directional indicator may be, for example, semi-translucent or opaque. In some examples, the directional indicator may be displayed after a determination is made that the event has persisted for a predetermined period of time.

[0039] In the example of Fig. IB, the directional indicator may be, for example, the directional indicator 102 and may be overlaid on the image of the field of view 104. The pointing portion 120 of the directional indicator 102 may be directed to or trained on the location 106 of the detected event outside of the field of view image 104.

[0040] In some examples, the method 200 may include a process of adjusting a position, orientation, and / or zoom factor of the directional indicator as the field of view changes so that the directional indicator remains trained on or pointed at the location of the detected event. For example, movement of an endoscope within the patient anatomy may cause the field of view to change. As the position, orientation, or zoom factor of the endoscope changes with respect to the frame of reference in which the detected event is recorded, the position and / or orientation of the directional indicator, including the pointing portion, may be adjusted to remain directed at location of the detected event.

[0041] In some examples, the method 200 may include a process of terminating a display of the directional indicator when termination criteria are met. For example, termination criteria may include a determination that the location of the detected event is within the field of view.4910-7717-3622 v. l 10Docket No. P06987-WO (70228.978WO01) Customer No. 160596Movement of an endoscope within the patient anatomy may cause the field of view to change. As the position or orientation of the endoscope changes with respect to the frame of reference in which the detected event is recorded, the location of the detected event may become located within the field of view of the imaging system. When the location of the detected event is determined to be within the field of view of the imaging system, the display of the directional indicator 102 may be terminated or suppressed. In some examples, a field of view indicator 108, such as a virtual marker overlay, may be displayed at the location 106 of the detected event when the location of the detected event is within the field of view of the imaging system. The field of view indicator 108 may include a modified version of the directional indicator 102 that includes some features of the directional indicator such as characterizing information but without a pointing portion or other directional characteristics. In some examples, the field of view indicator may include a caution marker indicating a detected foreign object. In other examples, particularly when the event persists or is self-evident, no virtual marker may be displayed at the location of the detected event. In the example of FIG. IB, the directional indicator 102 is terminated or omitted as the event location 106 of the detected event moves into the field of view image 104 and the virtual marker 108 may be displayed at the event location.

[0042] In some examples, a termination criteria for terminating a display of the directional indicator may include a determination that the detected event has been resolved. A resolution may include, for example, separation of instruments involved in a detected collision event, removal of an object from the patient anatomy, removal of a virtual object in the frame of reference, or any detected terminating event that obviates the detected event associated with the directional indicator.

[0043] FIGS. 3A-3D illustrate various examples of the processes of method 200 in which the detected event is a collision of instrument shafts. As shown in FIG. 3A, the graphical user interface 100 displays a directional indicator 302 that indicates that an instrument collision has been detected at a location outside of the field of view 104. The instrument collision may be between instruments 311 (not visible in FIG. 3 A) and 312. In this example, an icon portion 322 of the directional indicator 302 displays a symbol indicative of an instrument collision and a pointing portion 320 of the directional indicator 302 is pointed toward an out-of- view, detected location of the instrument collision. In some examples, the instrument collision may be detected using kinematic modelling, instrument sensors, or any of the other detection modalities previously described. The location of the instrument collision may be determined by the detection modality and / or a control system (e.g., the control system 1020) in the frame of4910-7717-3622 v. l 11Docket No. P06987-WO (70228.978WO01) Customer No. 160596 reference for the imaging system capturing the field of view 104 (e.g., the imaging system frame of reference Xi, Y i, Zi) or another co-registered frame of reference. The location of the instrument collision may be compared to the boundaries of the field of view 104 to determine that the collision is located outside of the view and to determine that the directional indicator should be displayed because the collision is located outside of the field of view.

[0044] As shown in FIG. 3B, the location, orientation, and / or zoom factor of the imaging system may be changed, thus causing a change in the image of the field of view. As the image of the field of view changes, the directional indicator 302 may change accordingly. The pointing portion 320 may pivot to remain trained on the out-of-view location of the instrument collision. In some examples, the location of the instrument collision may change due to the change in the imaging system, and the detected location of the instrument collision may be updated, resulting in a change in the directional indicator. In some examples, adjusting the directional indicator 302 may include adjusting a position of the directional indicator relative to the boundaries of the image of the field of view 104.

[0045] As shown in FIG. 3C, the directional indicator 302 may disappear or be removed from overlay on the field of view 104 when the instrument collision is resolved or no longer detected. For example, a detected collision between instruments 312 and 311 may have caused the directional indicator 302 to display in FIG. 3B. A separation of the instruments, as shown in FIG. 3C, may cause a resolution or termination of the detected collision event, and accordingly, the display of the directional indicator 302 may be discontinued.

[0046] Alternatively, as shown in FIG. 3D, the directional indicator 302 may disappear or be removed from overlay on the field of view 104 when the location of the instrument collision is within the field of view 104. As the position, orientation, and / or zoom factor of the endoscope changes, the location of the detected event may become located within the field of view 104 of the imaging system. When a location 306 of the detected collision of instruments 311, 312 is within the field of view of the imaging system, the display of the directional indicator 302 may be discontinued.

[0047] FIGS. 4-6 together provide an overview of a medical system 1010 that may be used in, for example, medical procedures including diagnostic, therapeutic, or surgical procedures. The directional indicator examples provided above may be used in the context of the medical system 1010. The medical system 1010 is located in a medical environment 1011. The medical environment 1011 is depicted as an operating room in FIG. 4. In other embodiments, the medical environment 1011 may be an emergency room, a medical training environment, a medical laboratory, or some other type of environment in which any number of medical4910-7717-3622 v. l 12Docket No. P06987-WO (70228.978WO01) Customer No. 160596 procedures or medical training procedures may take place. In still other embodiments, the medical environment 1011 may include an operating room and a control area located outside of the operating room.

[0048] In one or more embodiments, the medical system 1010 may be a robot-assisted medical system that is under the teleoperational control of an operator (e.g., a surgeon, a clinician, a physician, etc.). In alternative embodiments, the medical system 1010 may be under the partial control of a computer programmed to perform the medical procedure or subprocedure. In still other alternative embodiments, the medical system 1010 may be a fully automated medical system that is under the full control of a computer programmed to perform the medical procedure or sub-procedure with the medical system 1010. One example of the medical system 1010 that may be used to implement the systems and techniques described in this disclosure is the da Vinci® Surgical System manufactured by Intuitive Surgical, Inc. of Sunnyvale, California.

[0049] As shown in FIG. 4, the medical system 1010 generally includes an assembly 1012, which may be mounted to or positioned near an operating table T on which a patient P is positioned. The patient P may have a patient reference frame Xp, Yp, Zp. The assembly 1012 may be referred to as a patient side cart, a surgical cart, a manipulator assembly, or a surgical robot. In one or more embodiments, the assembly 1012 may be a teleoperational assembly. The teleoperational assembly may be referred to as, for example, a teleoperational arm cart. A medical instrument system 1014 and an endoscopic imaging system 1015 are operably coupled to the assembly 1012. An operator input system 1016 allows an operator O or other type of clinician to view images of or representing the surgical site and to control the operation of the medical instrument system 1014 and / or the endoscopic imaging system 1015.

[0050] The medical instrument system 1014 may comprise one or more medical instruments. In embodiments in which the medical instrument system 1014 comprises a plurality of medical instruments, the plurality of medical instruments may include multiple of the same medical instrument and / or multiple different medical instruments. Similarly, the endoscopic imaging system 1015 may comprise one or more endoscopes. In the case of a plurality of endoscopes, the plurality of endoscopes may include multiple of the same endoscope and / or multiple different endoscopes.

[0051] The operator input system 1016 may be located at an operator's control console, which may be located in the same room as operating table T. In some embodiments, the operator O and the operator input system 1016 may be located in a different room or a completely different building from the patient P. The operator input system 1016 generally includes one or more4910-7717-3622 v. l 13Docket No. P06987-WO (70228.978WO01) Customer No. 160596 control device(s) for controlling the medical instrument system 1014. The control device(s) may include one or more of any number of a variety of input devices, such as hand grips, joysticks, trackballs, data gloves, trigger-guns, foot pedals, hand-operated controllers, voice recognition devices, touch screens, body motion or presence sensors, and other types of input devices.

[0052] In some embodiments, the control device(s) will be provided with the same degrees of freedom as the medical instrument(s) of the medical instrument system 1014 to provide the operator with telepresence, which is the perception that the control device(s) are integral with the instruments so that the operator has a strong sense of directly controlling instruments as if present at the surgical site. In other embodiments, the control device(s) may have more or fewer degrees of freedom than the associated medical instruments and still provide the operator with telepresence. In some embodiments, the control device(s) are manual input devices that are movable with six degrees of freedom, and which may also include an actuatable handle for actuating instruments (for example, for closing grasping jaw end effectors, applying an electrical potential to an electrode, delivering a medicinal treatment, and actuating other types of instruments).

[0053] The assembly 1012 may support and manipulate the medical instrument system 1014 while the operator O views the surgical site through the operator input system 1016. An image of the surgical site may be obtained by the endoscopic imaging system 1015, which may be manipulated by the assembly 1012. The assembly 1012 may comprise endoscopic imaging systems 1015 and may similarly comprise multiple medical instrument systems 1014 as well. The number of medical instrument systems 1014 used at one time will generally depend on the diagnostic or surgical procedure to be performed and on space constraints within the operating room, among other factors. The assembly 1012 may include a kinematic structure of one or more non-servo controlled links (e.g., one or more links that may be manually positioned and locked in place, generally referred to as a set-up structure) and a manipulator. When the manipulator takes the form of a teleoperational manipulator, the assembly 1012 is a teleoperational assembly. The assembly 1012 includes a plurality of motors that drive inputs on the medical instrument system 1014. In an embodiment, these motors move in response to commands from a control system (e.g., control system 1020). The motors include drive systems which when coupled to the medical instrument system 1014 may advance a medical instrument into a naturally or surgically created anatomical orifice. Other motorized drive systems may move the distal end of said medical instrument in multiple degrees of freedom, which may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes)4910-7717-3622 v. l 14Docket No. P06987-WO (70228.978WO01) Customer No. 160596 and three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). Additionally, the motors may be used to actuate an articulable end effector of the medical instrument for grasping tissue in the jaws of a biopsy device or the like. Medical instruments of the medical instrument system 1014 may include end effectors having a single working member such as a scalpel, a blunt blade, an optical fiber, or an electrode. Other end effectors may include, for example, forceps, graspers, scissors, or clip appliers.

[0054] The medical system 1010 also includes a control system 1020. The control system 1020 includes at least one memory 1024 and at least one processor 1022 for effecting control between the medical instrament system 1014, the operator input system 1016, and other auxiliary systems 1026 which may include, for example, imaging systems, audio systems, fluid delivery systems, display systems, illumination systems, steering control systems, irrigation systems, and / or suction systems. A clinician may circulate within the medical environment 1011 and may access, for example, the assembly 1012 during a set up procedure or view a display of the auxiliary system 1026 from the patient bedside.

[0055] Though depicted as being external to the assembly 1012 in FIG. 4, the control system 1020 may, in some embodiments, be contained wholly within the assembly 1012. The control system 1020 also includes programmed instructions (e.g., stored on a non-transitory, computer- readable medium) to implement some or all of the methods described in accordance with aspects disclosed herein. While the control system 1020 is shown as a single block in the simplified schematic of FIG. 4, the control system 1020 may include two or more data processing units or circuits with one portion of the processing optionally being performed on or adjacent the assembly 1012, another portion of the processing being performed at the operator input system 1016, and the like.

[0056] Any of a wide variety of centralized or distributed data processing architectures may be employed. Similarly, the programmed instructions may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the systems described herein, including teleoperational systems. In one embodiment, the control system 1020 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry.

[0057] In some embodiments, control system 1020 may include one or more servo controllers that receive force and / or torque feedback from the medical instrument system 1014. Responsive to the feedback, the servo controllers transmit signals to the operator input system 1016. The servo controller(s) may also transmit signals instructing assembly 1012 to move the medical instrument system(s) 1014 and / or endoscopic imaging system 1015 which extend into4910-7717-3622 v. l 15Docket No. P06987-WO (70228.978WO01) Customer No. 160596 an internal surgical site within the patient body via openings in the body. Any suitable conventional or specialized servo controller may be used. A servo controller may be separate from, or integrated with, assembly 1012. In some embodiments, the servo controller and assembly 1012 are provided as part of a teleoperational arm cart positioned adjacent to the patient's body.

[0058] The control system 1020 can be coupled with the endoscopic imaging system 1015 and can include a processor to process captured images for subsequent display, such as to an operator on the operator's control console, or on another suitable display located locally and / or remotely. For example, where a stereoscopic endoscope is used, the control system 1020 can process the captured images to present the operator with coordinated stereo images of the surgical site. Such coordination can include alignment between the opposing images and can include adjusting the stereo working distance of the stereoscopic endoscope.

[0059] In alternative embodiments, the medical system 1010 may include more than one assembly 1012 and / or more than one operator input system 1016. The exact number of assemblies 1012 will depend on the surgical procedure and the space constraints within the operating room, among other factors. The operator input systems 1016 may be collocated or they may be positioned in separate locations. Multiple operator input systems 1016 allow more than one operator to control one or more assemblies 1012 in various combinations. The medical system 1010 may also be used to train and rehearse medical procedures.

[0060] FIG. 5 is a perspective view of one embodiment of an assembly 1012 which may be referred to as a patient side cart, surgical cart, teleoperational arm cart, manipulator assembly or surgical robot. The assembly 1012 may have a manipulator assembly reference frame XM, YM, ZM. In some examples, if the patient is stationary throughout the procedure, the manipulator assembly reference frame may be equivalent to the manipulator assembly reference frame. The assembly 1012 shown provides for the manipulation of three surgical tools 1030a, 1030b, and 1030c (e.g., medical instrument systems 1014) and an imaging system 1028 (e.g., endoscopic imaging system 1015), such as a stereoscopic endoscope used for the capture of images of the site of the procedure. The distal end of the imaging system 1028 may have an imaging reference frame Xi, Yi, Zi that moves with the distal end, relative to the manipulator assembly reference frame. The imaging device may transmit signals over a cable 1056 to the control system 1020. Manipulation is provided by teleoperative mechanisms having a number of joints. The imaging system 1028 and the surgical tools 1030a-c can be positioned and manipulated through incisions in the patient so that a kinematic remote center is maintained at the incision to minimize the size of the incision. Images of the surgical site can include4910-7717-3622 v. l 16Docket No. P06987-WO (70228.978WO01) Customer No. 160596 images of the distal ends of the surgical tools 1030a-c when they are positioned within the field of view of the imaging system 1028.

[0061] The assembly 1012 includes a drivable base 1058. The drivable base 1058 is connected to a telescoping column 1057, which allows for adjustment of the height of arms 1054. The arms 1054 may include a rotating joint 1055 that both rotates and moves up and down. Each of the arms 1054 may be connected to an orienting platform 1053. The arms 1054 may be labeled to facilitate trouble shooting. For example, each of the arms 1054 may be emblazoned with a different number, letter, symbol, other identifier, or combinations thereof. The orienting platform 1053 may be capable of 1060 degrees of rotation. The assembly 1012 may also include a telescoping horizontal cantilever 1052 for moving the orienting platform 1053 in a horizontal direction.

[0062] In the present example, each of the arms 1054 connects to a manipulator arm 1051. The manipulator arms 1051 may connect directly to a medical instrument, e.g., one of the surgical tools 1030a-c. The manipulator arms 1051 may be teleoperable. In some examples, the arms 1054 connecting to the orienting platform 1053 may not be teleoperable. Rather, such arms 1054 may be positioned as desired before the operator O begins operation with the teleoperative components. Throughout a surgical procedure, medical instruments may be removed and replaced with other instruments such that instrument to arm associations may change during the procedure.

[0063] Endoscopic imaging systems (e.g., endoscopic imaging system 1015 and imaging system 1028) may be provided in a variety of configurations including rigid or flexible endoscopes. Rigid endoscopes include a rigid tube housing a relay lens system for transmitting an image from a distal end to a proximal end of the endoscope. Flexible endoscopes transmit images using one or more flexible optical fibers. Digital image-based endoscopes have a “chip on the tip” design in which a distal digital sensor such as a one or more charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) device store image data. Endoscopic imaging systems may provide two- or three- dimensional images to the viewer. Two-dimensional images may provide limited depth perception. Three-dimensional stereo endoscopic images may provide the viewer with more accurate depth perception. Stereo endoscopic instruments employ stereo cameras to capture stereo images of the patient anatomy. An endoscopic instrument may be a fully sterilizable assembly with the endoscope cable, handle, and shaft all rigidly coupled and hermetically sealed.

[0064] FIG. 6 is a perspective view of an embodiment of the operator input system 1016 at the operator’s control console. The operator input system 1016 includes a display system 10354910-7717-3622 v. l 17Docket No. P06987-WO (70228.978WO01) Customer No. 160596 with a left eye display 1032 and a right eye display 1034 for presenting the operator O with a coordinated stereo view of the surgical environment that enables depth perception. The left and right eye displays 1032, 1034 together may be a viewport and may be components of the display system 1035. In other embodiments, the display system 1035 may include one or more other types of displays. The display system 1035 may present images captured, for example, by the imaging system 1015 to display the endoscopic field of view to the operator. The endoscopic field of view may be augmented by graphical user interface elements such as virtual or synthetic menus, indicators, and / or other graphical or textual information to provide additional information to the viewer.

[0065] The operator input system 1016 further includes one or more input control devices 1036, which in turn cause the assembly 1012 to manipulate one or more instruments of the endoscopic imaging system 1015 and / or medical instrument system 1014. The input control devices 1036 can provide the same degrees of freedom as their associated instruments to provide the operator O with telepresence, or the perception that the input control devices 1036 are integral with said instruments so that the operator has a strong sense of directly controlling the instruments. To this end, position, force, and tactile feedback sensors (not shown) may be employed to transmit position, force, and tactile sensations from the medical instruments, e.g., surgical tools 1030a-c, or imaging system 1028, back to the operator's hands through the input control devices 1036. Input control devices 1039 are foot pedals that receive input from a user' s foot. Aspects of the operator input system 1016, the assembly 1012, and the auxiliary systems 1026 may be adjustable and customizable to meet the physical needs, skill level, or preferences of the operator O.

[0066] Elements described in detail with reference to one embodiment, implementation, or application optionally may be included, whenever practical, in other embodiments, implementations, or applications in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or implementation non-functional, or unless two or more of the elements provide conflicting functions.4910-7717-3622 v. l 18Docket No. P06987-WO (70228.978WO01) Customer No. 160596

[0067] Any alterations and further modifications to the described devices, systems, instruments, methods, and any further application of the principles of the present disclosure are fully contemplated as would normally occur to one skilled in the art to which the disclosure relates. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one embodiment may be combined with the features, components, and / or steps described with respect to other embodiments of the present disclosure. In addition, dimensions provided herein are for specific examples and it is contemplated that different sizes, dimensions, and / or ratios may be utilized to implement the concepts of the present disclosure. To avoid needless descriptive repetition, one or more components or actions described in accordance with one illustrative embodiment can be used or omitted as applicable from other illustrative embodiments. For the sake of brevity, the numerous iterations of these combinations will not be described separately.

[0068] Various systems and portions of systems have been described in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an object or a portion of an object in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian X, Y, Z coordinates). As used herein, the term “orientation” refers to the rotational placement of an object or a portion of an object (three degrees of rotational freedom - e.g., roll, pitch, and yaw). As used herein, the term “pose” refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of the object in at least one degree of rotational freedom (up to six total degrees of freedom).

[0069] Although some of the examples described herein refer to surgical procedures or instruments, or medical procedures and medical instruments, the techniques disclosed optionally apply to non-medical procedures and non-medical instruments. For example, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, and sensing or manipulating non-tissue work pieces. Other example applications involve cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, and training medical or non- medical personnel. Additional example applications include use for procedures on tissue removed from human or animal anatomies (without return to a human or animal anatomy) and performing procedures on human or animal cadavers. Further, these techniques can also be used for surgical and nonsurgical medical treatment or diagnosis procedures.

[0070] A computer is a machine that follows programmed instructions to perform mathematical or logical functions on input information to produce processed output4910-7717-3622 v. l 19Docket No. P06987-WO (70228.978WO01) Customer No. 160596 information. A computer includes a logic unit that performs the mathematical or logical functions, and memory that stores the programmed instructions, the input information, and the output information. The term “computer” and similar terms, such as “processor” or “controller” or “control system,” are analogous.

[0071] While certain exemplary embodiments of the invention have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that the embodiments of the invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art.4910-7717-3622 v. l 20

Claims

Docket No. P06987-WO (70228.978WO01) Customer No. 160596CLAIMSWhat is claimed is:

1. A method comprising : detecting an event; determining a location of the event with respect to a frame of reference; determining that the location of the event is outside a field of view of an endoscopic imaging system; and displaying, within the field of view of the endoscopic imaging system, a directional indicator, the directional indicator indicating a direction of the location of the event outside the field of view.

2. The method of claim 1, wherein the event is within a patient anatomy.

3. The method of claim 2, wherein detecting the event includes detecting contact between two or more instruments in the patient anatomy.

4. The method of claim 2, wherein detecting the event includes detecting presence of an object in the patient anatomy.

5. The method of claim 4, wherein the object includes an instrument portion and the event includes an instrument breakage resulting in the depositing of the instrument portion in the patient anatomy.

6. The method of claim 4, wherein detecting the event includes monitoring a position of the object relative to the field of view of the endoscopic imaging system.

7. The method of claim 2, wherein detecting the event includes receiving user input to place a virtual marker in the patient anatomy.

8. The method of claim 7, wherein detecting the event includes monitoring a position of the virtual marker relative to the field of view of the endoscopic imaging system.4910-7717-3622 v. l 21Docket No. P06987-WO (70228.978WO01) Customer No. 1605969. The method of claim 8. further comprising: terminating a display of the virtual marker when the directional indicator is displayed.

10. The method of claim 1, wherein the event is external to a patient anatomy.

11. The method of claim 1, wherein detecting the event includes analyzing a kinematic model.

12. The method of claim 1, wherein detecting the event includes performing an image analysis.

13. The method of claim 1, wherein detecting the event includes receiving a signal from a sensor.

14. The method of claim 1, wherein detecting the event includes receiving a user input at an input control device.

15. The method of claim 1, wherein detecting the event includes receiving a signal from a registered instrument system.

16. The method of claim 15, wherein detecting the event includes monitoring a position of the registered instrument system relative to the field of view of the endoscopic imaging system.

17. The method of claim 1, wherein the directional indicator includes a pointing portion and an icon portion.

18. The method of claim 17, wherein the pointing portion is oriented toward the direction of the location of the event.4910-7717-3622 v. l 22Docket No. P06987-WO (70228.978WO01) Customer No. 16059619. The method of claim 17, wherein the icon portion includes a graphic associated with a type of the event.

20. The method of claim 17, wherein an orientation of the icon portion is independent of the orientation of the pointing portion.

21. The method of claim 17, wherein a characteristic of the icon portion is based on the detected event.

22. The method of claim 1, wherein the directional indicator is displayed in a three- dimensional form.

23. The method of claim 1, wherein displaying the directional indicator includes alternating a characteristic of the directional indicator to provide a gesture.

24. The method of claim 1 , wherein displaying the directional indicator includes determining that the event has persisted for a predetermined period of time.

25. The method of claim 1, wherein displaying the directional indicator includes determining a display location relative to the field of view to overlay the directional indicator.

26. The method of claim 25, wherein determining the display location to overlay the directional indicator includes establishing a line between a center of the field of view and the location of the event, wherein the display location is along the line.

27. The method of claim 25, wherein the display location is within left and right viewing volumes of the endoscopic imaging system.

28. The method of claim 1 , further comprising adjusting a position or orientation of the directional indicator as the field of view changes.

29. The method of claim 1, further comprising adjusting a position or orientation of the directional indicator as the endoscopic imaging system moves.4910-7717-3622 v. l 23Docket No. P06987-WO (70228.978WO01) Customer No. 16059630. The method of claim 1, further comprising terminating the displaying of the directional indicator when the location of the event is within the field of view of the endoscopic imaging system.

31. The method of claim 30, further comprising displaying a field of view indicator when the location of the event is within the field of view.

32. The method of claim 31, wherein the field of view indicator includes a virtual marker.

33. The method of claim 31 , wherein the field of view indicator includes a caution marker for a detected foreign object.

34. The method of claim 1, further comprising modifying the directional indicator when the location of the event is within the field of view of the endoscopic imaging system.

35. The method of claim 1, further comprising terminating the displaying of the directional indicator when a determination is made that the event has resolved.

36. A medical system comprising: a display system; and a control system, wherein the control system includes a processing unit including one or more processors, and wherein the processing unit is configured to: detect an event; determine a location of the event with respect to a frame of reference; determine that the location of the event is outside a field of view of an endoscopic imaging system; and display a directional indicator, with the field of view of the endoscopic imaging system, indicating a direction of the location of the event outside the field of view.4910-7717-3622 v. l 24

Images