Systems and methods for adapting depth placement of graphical user interface elements with surgical imagery

Abstract

A medical system may comprise a display system and a control system. The control system may include a processing unit including one or more processors. The processing unit may be configured to receive stereoscopic image data for a field of view of an imaging system, determine depth information for the field of view, and determine a target display distance for a graphical user interface element based on the depth information. The processing unit may also be configured to determine a stereo separation distance for stereo presentation of the graphical user interface element based on the determined target display distance and display, on the display system, the graphical user interface element at the target display distance with the stereoscopic image data of the field of view.

Description

SYSTEMS AND METHODS FOR ADAPTING DEPTH PLACEMENT OF GRAPHICAL USER INTERFACE ELEMENTS WITH SURGICAL IMAGERYCROSS-REFERENCED APPLICATIONS

[0001] This application claims priority to and benefit of U.S. Provisional Application No. 63 / 642,545 filed May 3, 2024 and entitled “Systems and Methods for Adapting Depth Placement of Graphical User Interface Elements With Surgical Imagery,” which is incorporated by reference herein in its entirety.FIELD

[0002] The present disclosure is directed to systems and methods for fusing a graphical user interface element with a stereoscopic surgical image.BACKGROUND

[0003] Minimally invasive medical techniques are intended to reduce the amount of extraneous tissue that is damaged during diagnostic or surgical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques may be performed through natural orifices in a patient anatomy or through one or more surgical incisions. Through these natural orifices or incisions, clinicians may insert medical tools to reach a target tissue location. Minimally invasive medical tools include instruments such as therapeutic instruments, diagnostic instruments, and surgical instruments. Minimally invasive medical tools may also include imaging instruments such as endoscopic instruments that provide a viewer with an image of a field of view within the patient anatomy.

[0004] Some minimally invasive medical tools may be robot-assisted including teleoperated, remotely operated, or otherwise computer-assisted. During a medical procedure, the clinician may view an image of a field of view of the patient anatomy that may include one or more of the minimally invasive medical tools. Various graphical user interface elements may be presented with an image of the field of view. Improved systems and methods are needed for stereo presentation of graphical user interface components that minimizes depth conflicts, convergence issues, or other suboptimal stereo fusion experiences.SUMMARY

[0005] The embodiments of the invention are best summarized by the claims that follow the description.

[0006] In one example embodiment, a medical system may comprise a display system and a control system. The control system may include a processing unit including one or more processors. The processing unit may be configured to receive stereoscopic image data for a field of view of an imaging system, determine depth information for the field of view, and determine a target display distance for a graphical user interface element based on the depth information. The processing unit may also be configured to determine a stereo separation distance for stereo presentation of the graphical user interface element based on the determined target display distance and display, on the display system, the graphical user interface element at the target display distance with the stereoscopic image data of the field of view.

[0007] In one example, a method may comprise receiving stereoscopic image data for a field of view of an imaging system, determining depth information for the field of view, and determining a target display distance for a graphical user interface element based on the depth information. The method may also comprise determining a stereo separation distance for stereo presentation of the graphical user interface element based on the determined target display distance and displaying the graphical user interface element at the target display distance with the stereoscopic image data of the field of view.

[0008] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.BRIEF DESCRIPTIONS OF THE DRAWINGS

[0009] FIG. 1A illustrates left and right view volumes and an associated convergence plane, according to some examples.

[0010] FIG. IB illustrates left and right view volumes and an associated convergence plane, according to some examples.

[0011] FIG. 2 illustrates left and right view volumes and an associated convergence plane, according to some examples.

[0012] FIG. 3 illustrates a method for providing stereoscopic fusion of stereoscopic imagery and graphical user interface elements, according to some examples.

[0013] FIG. 4 illustrates a method for determining depth information for a field of view, according to some examples.

[0014] FIG. 5 illustrates a method for modifying a target display distance for a graphical user interface element, according to some examples.

[0015] FIG. 6A illustrates a stereoscopic image of a field of view of an imaging system, according to some examples.

[0016] FIG. 6B illustrates depth information for the field of view of FIG. 6A, according to some examples.

[0017] FIG. 6C illustrates graphical user interface elements displayed at a target distance with the stereoscopic image of FIG. 6A.

[0018] FIG. 7 A illustrates a stereoscopic image of another field of view of the imaging system of FIG. 6A, according to some examples.

[0019] FIG. 7B illustrates depth information for the field of view of FIG. 7 A, according to some examples.

[0020] FIG. 7C illustrates graphical user interface elements displayed at a target distance with the stereoscopic image of FIG. 7A.

[0021] FIG. 8A illustrates a stereoscopic image of another field of view of the imaging system of FIG. 6A, according to some examples.

[0022] FIG. 8B illustrates depth information for the field of view of FIG. 8A, according to some examples.

[0023] FIG. 8C illustrates graphical user interface elements displayed at a target distance with the stereoscopic image of FIG. 8A.

[0024] FIG. 9A illustrates depth information for a field of view of an imaging system, according to some examples.

[0025] FIG. 9B illustrates summary depth information, by region, for the field of view of FIG. 9A, according to some examples.

[0026] FIG. 10 illustrates a relationship between stereo separation distance and target display distance, according to some examples.

[0027] FIG. 11A illustrates stereo display of poorly fused graphical user interface elements, according to some examples.

[0028] FIG. 1 IB illustrates stereo display of properly fused graphical user interface elements, according to some examples.

[0029] FIG. 12 illustrates a method for synthetically adjusting the convergence plane of the stereoscopic image data based on a determined working distance, according to some examples.

[0030] FIG. 13 illustrates a graphical user interface for selecting a display parameter.

[0031] FIG. 14 illustrates a schematic view of a medical system, according to some examples.

[0032] FIG. 15 is a perspective view of a manipulator assembly of the medical system of FIG. 14, according to some examples.

[0033] FIG. 16 is a front elevation view of an operator’s console in a robot- assisted medical system, according to some examples.

[0034] Embodiments of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, wherein showings therein are for purposes of illustrating embodiments of the present disclosure and not for purposes of limiting the same.DETAILED DESCRIPTION

[0035] In robot-assisted medical procedures, stereoscopic endoscope images of the surgical environment may provide a clinician with an image of a field of view of the patient anatomy and any medical tools located in the patient anatomy. Various graphical user interface elements may be incorporated with the field of view image. Graphical user interface elements may include, for example, menus, alerts, user messages, indicators, digital tools, and auxiliary images. As described herein, the three-dimensional depth presentation of these graphical user interface components may be adapted based on the depth of structures in the field of view.

[0036] Image data from a field of view of a stereoscopic imaging system (e.g., imaging system 1015 of FIG. 14) may be displayed to a viewer as separate right and left eye view volumes on a display system (e.g., display system 1035 of FIG. 16). The disparity between the view volumes may determine a plane of convergence, with the depth of the convergence plane increasing as the disparity increases. To minimize distortion, the stereo geometry of the stereo imaging system may correspond to the stereo geometry of the stereo display system. For example, when displaying stereo imagery, matching the ratios of stereo separation and convergence depth for the stereoscopic imaging system and the stereoscopic display system may minimize or eliminate any depth distortion. User interface element presentation may be based on depth values mapped from the stereo imaging system space to the stereo display space. FIG. 1A illustrates a stereo viewing arrangement 100 including a viewing volume 102, which may be a right viewing volume for presenting stereo image data from a stereoscopic imaging system to a viewer’s right eye, and a viewing volume 104, which may be a left viewing volume for presenting the stereo image data from the stereo imaging system to the viewer’ s left eye. The viewing volume 102 may have a central axis 106 and an origin 108. The viewing volume 104 may have a central axis 110 and an origin 1 12. The central axes 106, 1 10 may intersect ata convergence location 114. A convergence plane 116 may extend through the convergence location 114, generally parallel to an origin plane 118 of the origins 108, 112. A convergence depth 120 from the origin plane 118 to the convergence plane 116 may be determined by a stereo separation or disparity 122 between the origins 108, 112. In some examples, the convergence plane 116 may be located at a display surface of a viewport or head-in display system (e.g., display system 1035) and the convergence depth 120 may correspond to the distance between a plane of the viewer’s eyes (e.g., the origin plane 118) and the display surface. For example, in some medical systems, the distance between the display surface and the plane of the viewer’s eyes may be between approximately 0.045 and 0.055 meters, which corresponds to the convergence depth 120. In some examples, the disparity 122 may be generated by a physical separation between eye pieces of a head-in display system. In other examples, the disparity 122 may be synthetically generated or simulated by adjusting the presentation (e.g., shifting the image pixels) of the right and left image data to create a virtual separation, while the eye pieces of the head-in display system remain positionally fixed relative to each other.

[0037] As the disparity is reduced, the convergence depth may also be reduced. FIG. IB illustrates a stereo viewing arrangement 150 including the viewing volume 102 and the viewing volume 104. In the arrangement 150, the origins 108, 112 may have a disparity 152 that is smaller than the disparity 122 of arrangement 100. In the arrangement 150, the central axes 106, 108 may intersect at a convergence location 154. A convergence plane 156 may extend through the convergence location 154, generally parallel to the origin plane 118 of the origins 108, 112. A convergence depth 160 from the origin plane 118 to the convergence plane 116 may be determined by the disparity 122 between the origins 108, 112. Because the disparity 152 is smaller than the disparity 122, the convergence depth 160 may be smaller (e.g., shallower) than the convergence depth 120. Relatedly, the convergence plane 156 may be closer to the origin plane 118 than the convergence plane 1 16. In some examples, the convergence plane 156 may be located at a convergence depth of, for example, approximately 0.02 meters, which may correspond to a minimum focal depth of the imaging system.

[0038] Medical systems, including robot- assisted medical systems such as medical system 1010 described below, may present endoscopic imagery to a viewer, such as a surgeon, via a stereoscopic display that emulates peering through a viewing window at the convergence plane and into an anatomical space beyond the convergence plane. The viewing window may generally correspond to the area of the convergence plane within the viewing volumes. When the endoscope is positioned close to anatomic structures or when instruments are positionedclose to the endoscope tip, the stereo imagery can appear to protrude in front of or to the sides of the viewing window. When content in the viewing volumes intersects the viewing window, known as window violation, it can become difficult for a viewer to fuse the stereoscopic images to perceive a single image. This problem can be exacerbated when two-dimensional graphical user interface elements (which may be at least partially opaque) are overlayed on the protruding endoscopic imagery at the convergence plane. Under such circumstances, the graphical user interface elements may appear to conflict in depth with the endoscopic imagery, since the two- dimensional graphical user interface element is shown at the convergence plane while the field of view content is closer to the origin plane. Methods for resolving window violations, depth conflicts, and other fusion disturbances are described herein. For example, as described in method 300, the depth placement of two-dimensional graphical user interface elements may be adapted based on the depth information associated with content presented in the viewing volumes (e.g., depth information associated with anatomical structures in the field of view). In the method 900, the convergence plane of the stereoscopic image data may be synthetically adjusted based on a determined working distance.

[0039] FIG. 2 illustrates a stereo viewing arrangement 200 including a viewing volume 202, which may be a right viewing volume for presenting stereo image data from a stereoscopic imaging system to a view’s right eye, and a viewing volume 204, which may be a left viewing volume for presenting the stereo image data from the stereo imaging system to the viewer’s left eye. Together the viewing volumes 202, 204 form a composite viewing volume 205 capturing the field of view of the imaging system. The viewing volume 202 may have a central axis 206 and an optical center or origin 208. The viewing volume 204 may have a central axis 210 and an optical center or origin 212. The central axes 206, 210 may intersect at a convergence location 214. A convergence plane 216 may extend through the convergence location 214, generally parallel to an origin plane 218 of the origins 208, 212. A convergence depth 220 from the origin plane 218 to the convergence plane 216 may be determined by the disparity 222 between the origins 208, 212. In some examples, the convergence plane 216 may be located at a display surface of a head-in display system (e.g., display system 1035) and the convergence depth 220 may correspond to the distance between a plane of the viewer’s eyes (e.g., the origin plane 118) and the display surface. A viewing window 207 may generally correspond to the area of the convergence plane 216 within the viewing volumes 202, 204. The composite viewing volume 205 may include a close range 211 that extends generally between a plane of minimum focal depth 230 of the imaging system and the convergence plane 216 and a common range 213 that extends distally from convergence plane 216. Structures in thecomposite viewing volume 205 may include, for example, anatomic tissue 232 and surgical instruments 234, 236. When tissue or an active portion of surgical instrument, such as surgical instrument 236, is located in the close range 211 , window violations, depth conflicts, and / or other compromised stereo fusion conditions may occur. These conditions may be exacerbated if graphical user interface elements are overlayed at the viewing window 207 of the convergence plane 216. As described in FIG. 3, an adapted display plane 240 for displaying graphical user interface elements may be determined to alleviate or at least improve the clarity of the presentation. The adapted display plane 240 may be located between the convergence plane 216 and the origin plane 218 at a target display distance 242 from the origin plane.

[0040] FIG. 3 is a flowchart illustrating a method 300 for providing stereoscopic fusion of stereoscopic imagery and graphical user interface elements. The methods described herein are illustrated as a set of operations or processes and are described with continuing reference to the additional figures. Not all of the illustrated processes may be performed in all embodiments of the methods. Additionally, one or more processes that are not expressly illustrated in may be included before, after, in between, or as part of the illustrated processes. In some embodiments, one or more of the illustrated processes may be omitted. In some embodiments, one or more of the processes may be implemented, at least in part, in the form of executable code stored on non-transitory, tangible, machine-readable media that when run by one or more processors (e.g., the processing units of a control system such as control system 1020) may cause the one or more processors to perform one or more of the processes. In one or more embodiments, the processes may be performed by a control system.

[0041] At a process 302, stereoscopic image data of a field of view may be received from an imaging system. An imaging system (e.g. imaging system 1015) may be, for example a stereoscopic endoscope that is extendable into a patient anatomy to obtain stereoscopic image data of the anatomic issue, instruments, or other structures in a field of view of the imaging system. For example and with reference to FIG. 2, the imaging system may capture stereoscopic viewing volumes 202, 204 that include the anatomic tissue 232 and the instruments 234, 236 in the field of view of the volumes 202, 204. Optionally, the stereoscopic image data of the field of view may be presented or displayed on a display screen. FIGS. 6A, 7A, and 8 A illustrate a display screen 600 presenting stereoscopic image data from an anatomic region 602. FIGS. 6A, 7A, 8A include increasing narrow fields of view 601, 603, 605, respectively, of the anatomic region 602 as, for example, the endoscope advances into the anatomy or the endoscope zooms from a wide field of view in FIG. 6A to a more close-up viewin FIG. 8A. Structures in the anatomic region 602 include instruments 604, 606, 608 and anatomic features 610, 612.

[0042] At a process 304, depth information may be determined for the field of view. Any of several or more inputs may be used to determine depth information for structures in the field of view. Inputs may include depth mapping information, kinematic information for the imaging system, zoom status information for the imaging system, kinematic information for an instrument in the field of view, and / or kinematic information for a manipulator arm operating an instrument or the imaging system. In some examples, determining depth information for the field of view may be achieved with the method 400 as described in FIG. 4.

[0043] At a process 402, depth map information may be determined for the field of view. For example, a depth map information may include a depth value for each image unit (e.g., pixel) may be determined based on the relationship, Z = (B x f) / D, where Z is a depth for a pixel in the field of view (e.g., a depth from the origins 208, 212 to a pixel capturing an image of structures 232, 236). B may represent a distance between optical centers of stereoscopic display devices (e.g., disparity 222), and f may represent a focal length of the imaging system. D may be a difference between an image point in the right eye image and the right optical center and the corresponding image point in the left eye image and the left optical center. To produce a dense depth map, a correspondence maybe evaluated for each pixel location in the left and right eye images while satisfying an epipolar constraint for identifying matching features that are within a tolerance of the same horizontal plane. In some examples, this form of analysis can be achieved at real-time rates using GPU-based parallel processing methods. In some examples, a depth map may be displayed with each pixel of the image colored or-gray-shaded according to its depth. In other examples, the depth map information may be generated but not displayed. FIGS. 6B, 7B, 8B illustrate depth maps 621, 623, 625, respectively, generated for the fields of view of the anatomic region 602 as shown in FIGS. 6A, 7A, 8A, respectively. Depth map information may also or alternatively include summary statistics such as minimum, average, and maximum and percentile-based depth. For example, the wider field of view 601 of FIG. 6A may have a minimum depth 624 (e.g., 0.06 m), an average depth 626 (e.g., 0.08 m), and a maximum depth 628 (e.g., 0.15 m), while the close-up field of view 605 of FIG. 8A may have a minimum depth 624 (e.g., 0.02 m), an average depth 626 (e.g., 0.025 m), and a maximum depth 628 (e.g., 0.05 m).

[0044] At a process 404, depth information for regions of the field of view may be generated. For example, a regionalized depth map may be used to generate depth information (e.g., summary statistics) for a particular region of the field of view. In some examples, regions maybe established based on locations within the field of view for presentation of graphical user interface elements (e.g., regions around the periphery of the field of view where menus, status bars, or other graphical elements may be overlayed). Summary statistics including, for example, minimum, average, mean, and / or maximum depths may be determined for each region. Such statistics for a particular region may correspond to, for example, the minimum, average, mean, and / or maximum depth values of pixels within the particular region. In some examples, for example to conserve computing time or effort, depth information may be determined only for selected regions, such as regions where graphical user interface elements will be overlayed or regions that include actuatable components of instruments but exclude more stationary components such as the instrument shaft. Depth samples may have accompanying quality metrics related to the agreement of match within a neighborhood of pixels. The summary statistics may be limited to depth samples that meet a threshold quality level. For instance, outlier depth samples and / or depths samples associated with poor quality metrics may be ignored in computing summary statistics for the region. In some examples, depth map summary statistics may be filtered with a second order low pass filter to produce smooth and continuous changes in the depth of graphical user interface elements and clamped within predetermined near and far plane distances. In some examples, the filter response may change based on context. For example, the filter response may be slow (e.g., 0.2 Hz) during teleoperation of an instrument and may be more responsive (e.g., 3.0 Hz) during gross endoscope motion.

[0045] FIG. 9A, for example, illustrates a depth map 700, and FIG. 9B illustrates a regionalized depth map 702 that corresponds to the depth map 700. The regionalized map 702 includes regions 704, 706, 708, and 710 that extend along the periphery of the map and aregion 712 in the central region of the map. Regional summary statistics are determined for each region. For example, region 704 may have a minimum depth 714 (e.g., 0.15 m), an average depth 716 (e.g., 0.030 m), and a maximum depth 718 (e.g., 0.045 m).

[0046] With further reference to FIG. 4, at a process 406, determining depth information may additionally or alternatively include receiving kinematic information for the imaging system. Kinematic information for the imaging system may include, for example, a kinematic measure of the distal end portion of the endoscope generating the field of view. A position and / or orientation of the distal end portion may be used to determine relative distance to structures visible in the field of view. Kinematic information may additionally or alternatively include information about orientation, velocity, acceleration, and / or other movement information.

[0047] At a process 408, determining depth information may additionally or alternatively include receiving zoom information for the imaging system. For example, zoom information may include digital or optical zoom information indicating the amount of magnification or the increase or decrease in the size of the subject without changing the position of the camera. Zoom information may include whether or not a zoom has been applied to the image data and the magnitude of the applied zoom. As a consequence of changing digital zoom, the visible field of view of the depth map may be considered when estimating depth for regions of the viewport. For example, the digital zoom may be modeled as an orthographic center-based scale transform applied to the output of the projection transform for both left and right eyes before converting from normalized device coordinates to viewport coordinates.

[0048] At a process 410, determining depth information may additionally or alternatively include receiving kinematic information for an instrument in the field of view. Kinematic information may include information about position, orientation, velocity, acceleration, and / or other movement information about an instrument. The instrument kinematic information may be supplement or correct depth map information by providing a kinematic measure of the distance between a kinematically known location of a distal tip of the imaging system (e.g., the distal tip of the endoscope) and a kinematically known portion of the instrument (e.g., distal tip, end effector tip, clevis). For example, with reference to FIG. 6A, kinematic information may be determined for each of the instruments 604, 606, 608 in the field of view 601. For example a position of a clevis 622 and a distal tip 627 of the instrument 606 may be kinematically known or determined. This information may be used to supplement or correct the depth map 620. In some examples, knowledge of instrument pose and / or velocity may be received or determined and used to exclude depth samples related to rapid motion of instruments passing through the field of view or a region of the field of view to attenuate disturbances to graphical user interface depth placement. For example, depth pixels may be unprojected into the endoscope tip coordinate system and compared against a bounding volume or estimate of the instrument pose to determine if the depth sample corresponds to the instrument. The instrument related depth samples may be excluded if an instrument velocity exceeds a threshold motion or included if the instrument velocity is below the threshold motion.

[0049] At a process 412, determining depth information may additionally or alternatively include receiving kinematic information for one or more manipulator arms that control motion of the imaging system and / or instruments visible in the field of view of the imaging system. For example, a gross motion of the endoscope may result in motion blur and may degrade the quality of the depth map reconstruction. Kinematic information about the movement of theendoscope may be incorporated to improve the responsiveness of adaptive graphical user interface element depth placement during gross endoscope movement. Upon activation of an imaging system control mode, the current scene depth within one or more regions may be sampled and latched along with the current pose of the endoscope tip frame of reference. With each subsequent update, movement of the endoscope tip relative to the latched endoscope tip frame of reference may be evaluated to determine a relative change in viewing depth. This relative change in viewing depth may be added as an offset to the latched scene depth to estimate the current scene depth. This alternative estimate of scene depth may be blended with the depth map estimate of scene depth to produce an estimate that is responsive to the onset of movement while continuously converging toward the actual scene depth from the depth map.

[0050] At a process 414 (which corresponds to the process 304 of FIG. 3), information from one or more of the processes 402, 404, 406. 408, 410, 412 may be used to determine depth information for the field of view.

[0051] Referring again to FIG. 3, at a process 306, a target display distance for a graphical user element may be determined from the depth information. The graphical user interface element may include one or more of banner messages, a status information box, manipulator arm information, an interactive menu, markers, alerts, user messages, indicators, digital tools, and auxiliary video windows. A graphical user interface element may be a two-dimensional graphical user interface element in that the element may appear to lack a thickness and may appear to exist in a plane. The two-dimensional graphical user interface element may, however, be positioned at a depth, such as a target display distance, in the three-dimensional field of view. The graphical user interface element may be, for example, semi-translucent or opaque. The target display distance for the graphical user interface element may be a distance from the imaging system (e.g., from the distal tip of an endoscope) at which the right and left images forming the graphical user interface element fuse or converge. The adapted display plane for the graphical user interface element may be located at the target display distance from the origin plane. For example, with reference to FIG. 2, the adapted display plane 240 may be located between the convergence plane 216 and the origin plane 218 at the target display distance 242 from the origin plane.

[0052] In some examples, the target display distance for locating the adapted display plane may be determined based on the depth information for the full field of view. For example, with reference to FIG. 6C, the target display distance for graphical user interface elements 630, 632 may be the minimum depth 624 (e.g., 0.06 m) for the full field of view which may correspond to the depth of a plane of the structure closest to the imaging system. In other examples, thetarget display distance may be an average depth, a mean depth, a maximum depth, or another distance determined from the depth information. The target display distance may be different for each field of view. The target display distance for displaying the graphical user interface elements overlayed on the field of view images may be different from the convergence depth (e.g. convergence depth 220) for the right and left images of the field of view. For example and with reference to FIG. 8C, the target display distance for graphical user interface elements 630, 632 may be the minimum depth 624 (e.g., 0.02 m) for the full field of view. In some examples, the target display distance may be a median depth or another percentile-based depth for the display or a region of the display. A median depth may provide a more conservative fit that is generally immune to localized peaks in the depth map and depth map errors.

[0053] In some examples, the target display distance for positioning the adapted display plane may be determined based on regional depth information for a specific region of the field of view underlying, near, or adjacent to the location of the graphical user interface elements. For example, with reference to FIG. 9B, the target display distance for a graphical user interface element located in the region 704 may be at the minimum depth 714 (e.g., 0.015 m) for the region 704, and the target display distance a graphical user interface element located in region 708 may be at an average depth for that region (e.g., 0.028 m). Thus, the target display distances for displaying graphical user interface elements in adapted display planes may be different for different regions of a common field of view. The most appropriate display distance may be dependent on the content of the region. For example, text messages, may be located in front of the closest feature of the scene to preserve fusion as the eye scans across the message. For robotic arm information along the bottom and spanning the width of the display screen (e.g., region 708, elements 632), the human eye may tolerate some localized depth conflicts. The display distance may be chosen to reduce sensitivity to localized peaks in the depth map causing the robotic arm information to move. For these elements, an average or median depth estimate may be used because it is not as sensitive to the presence of localized peaks.

[0054] In some examples the target display distance for the graphical user interface element may be adjusted based on a selected display parameter. In some examples, the selected display parameter may be used to manually adjusting a "working distance" disparity of the stereo images to resolve depth conflicts. FIG. 13 illustrates a graphical user interface 950 displayed on a display system 952 (e.g. the display system 1035) for selecting a display parameter. An operator may select a working distance menu option 954 which may allow the operator to select between one or more display parameters 956. An “automatic” display parameter may result in a target display distance that is based entirely or primarily on the determined depth information.A “normal” display parameter may result in a target display distance that in adjusted for a nominal convergence depth of the stereo view volumes. A “close” display parameter may result in a target display distance that in adjusted to a convergence depth that is closer than the nominal depth of the stereo view volumes (e.g. 0.035 m). A “very close” display parameter may result in a target display distance that is adjusted for a minimum convergence depth that aligns with the minimum focal depth of the endoscope (e.g. 0.02 m).

[0055] Referring again to FIG. 3, at a process 308, a stereo separation distance for implementing stereo presentation of the graphical user interface element, overlayed on the field of view, may be determined. The stereo separation distance may be a measure of the effective disparity (including physical optical display separation and synthetic shift generated by shifting pixels in left and / or right eye images) to achieve graphical user interface element fusion or may be a measure of a shift or adjustment relative to the disparity used to achieve fusion of the field of view image data over which the graphical user interface element is overlayed. For example, stereo separation distance of the graphical user interface element may be a measure of the horizontal shift between pixels in the right and left images of the graphical user interface element that provide an adjustment of the graphical user interface element’s depth in the field of view. The stereo separation may be based on the determined target display distance for the graphical user interface element. For example, the stereo separation distance may be determined by a predetermined relationship between disparity adjustment and target display distance adjustments (e.g. as shown in FIG. 10) or by the relationship, Z = (B x Ij / D, where the D may be calculated to adjust the stereo separation distance. In some examples, a first stereo separation distance for presenting the stereoscopic image data (e.g., images of the endoscopic field of view) may be different from a second stereo separation distance for presenting the graphic user interface element.

[0056] FIG. 10 is a graph 800 illustrating a predetermined relationship between depth adjustment 802 (e.g., adjustment relative to the convergence depth 220 to achieve the target display distance 242) and the corresponding viewport disparity shift, per eye 804 (e.g., stereo separation shift relative to the original disparity for image data convergence). In this example, a relatively small viewport disparity (e.g., -6% to +1.5% of viewport width) may achieve + / - 3.0 cm of synthetic displacement of the adapted display plane for displaying a graphical user interface element.

[0057] Referring again to FIG. 3, at a process 310, the graphical user interface element may be displayed at the target display distance overlayed on the stereoscopic image data of the field of view. The graphical user interface element may be presented at the target display distancewhile the stereoscopic image data of the field of view is presented at the convergence depth (or another depth). Displaying the graphical user interface element at the target display distance may include displaying the left and right images of the graphical user interface element in accordance the stereo separation distance. When displayed at the target display distance, the graphical user interface element may appear to be located on the viewing window, with no structures in the field of view protruding from the viewing window. For example, as shown in FIG. 6C, the graphical user interface elements 630, 632 may be displayed at the target display distance (e.g., the minimum depth 0.06 m), overlayed on the imagery of the field of view 601 . As shown in FIG. 7C, the graphical user interface elements 630, 632 may be displayed at the target display distance (e.g. the minimum depth 0.04 m), overlayed on the imagery of the field of view 603. As shown in FIG. 8C, the graphical user interface elements 630, 632 may be displayed at the target display distance (e.g. the minimum depth 0.02 m), overlayed on the imagery of the field of view 605.

[0058] In another example, various graphical user interface elements may be displayed at different target display depths, depending on the region in which the elements are located. FIG. 11A illustrates a stereo display of poorly fused graphical user interface elements 810, 812 when the elements are displayed at the convergence plane of the stereoscopic image data of the field of view 814. In this example the endoscopic image data of the field of view is well fused, but the graphical user interface elements are blurry or unfused. FIG. 11B illustrates stereo display of properly fused (not blurry) graphical user interface elements 810, 812, while maintaining fusion of the endoscopic image data of the field of view 814. Each of the graphical user interface elements 810, 812 is located in a different region and is displayed at a target display depth that corresponds to the respective region (e.g., at a target display depth that is determined based on depth information or summary statistics, such as minimum depth, average depth, mean depth, maximum depth, etc., associated with the respective region). For example, to achieve the fusion in FIG. 1 IB, a target display distance (e.g., a 2.25 cm shift toward the eyes from the convergence plane) was determined for the region of the graphical user interface element 810 and a different target display distance (e.g., a 1.75 cm shift toward the eyes from the convergence plane) was determined for the region of the graphical user interface element 812. Based, for example, on the relationship illustrated in the graph 800 of FIG. 10, to achieve the -2.25 cm shift (toward the eyes), a -3% disparity adjustment between the presentation of the right and left graphical user interface element overlay may be used. To achieve the -1.75 cm shift (toward the eyes), a -2% disparity adjustment between the presentation of the right and left graphical user interface element overlays may be used. To achieve the disparityadjustments, half of the disparity adjustment may be applied to shift each of the left and right images by equal amounts in opposing directions.

[0059] FIG. 5 illustrates a method 500 for modifying a target display distance for a graphical user interface element. If the tissue or instruments in the field of view change position or if the position of the imaging system changes, reevaluation of the depth placement of the graphical user interface element may be warranted. A modification to the depth placement of the graphical user interface elements may be made in a manner that minimizes distraction to the viewer. The method 500 may be performed following the process 310. At a process 502, stereoscopic image data for a second field of view of the imaging system may be received. For example, the stereoscopic image data for a first field of view may be the field of view 601, as shown in FIG. 6A. If the imaging system is further inserted or if a zoom is applied, stereoscopic image data for a second field of view, such as the field of view 603 of FIG. 7A, may be received from the imaging system.

[0060] At a process 504, depth information may be determined for the second field of view. The process 504 may be substantially similar to the process 304, as applied to the second field of view. For example, the closer field of view 603 of FIG. 7 A may have determined depth information such as a minimum depth 624 (e.g., 0.04 m), an average depth 626 (e.g., 0.05 m), and a maximum depth 628 (e.g., 0.10 m). As described previously, the depth information may include kinematic information about changes in position, orientation, velocity, and / or acceleration of the imaging system, instruments, and / or the manipulator arms controlling the imaging system or instruments. The depth information, including depth map information and / or kinematic information, may provide an indication that tissue, instruments, and / or the imaging system have moved.

[0061] At a process 506, a target display distance for the graphical user element may be determined from the depth information for the second field of view. The process 506 may be substantially similar to the process 306, as applied to the second field of view. For example, with reference to FIG. 7C, the target display distance for graphical user interface elements 630, 632 may be the minimum depth 624 (e.g., 0.04 m) for the full field of view which may correspond to the depth of a plane of the structure closest to the imaging system. In other examples, the target display distance may be an average depth, a maximum depth, or another distance determined from the depth information.

[0062] At a process 508, a stereo separation distance for implementing stereo presentation of the graphical user interface element overlayed on the second field of view may be determined. The second stereo separation distance, including any modification from the first stereoseparation distance needed to achieve the second stereo separation distance, may be based on the determined second target display distance for the graphical user interface element. The process 508 may be substantially similar to the process 308, as applied to the second field of view.

[0063] At a process 510, system status or information may be determined or received. System status or information may indicate, for example, a control mode of the system (e.g., an instrument control mode, an imaging system control mode, etc.), a status of the user input system (e.g., 1016) or display system (e.g., 1035), eye gaze tracking information, and the like. Generally, adjustments to stereo separation distance(s) and / or target display distance(s) for displaying one or more graphical user interface elements within the field of view may be performed based at least in part on the system status or information. In certain embodiments, modifying stereo separation distance(s) and / or target display distance(s) may be performed in response to the system status or information (e.g., in response to a display system disengagement indication or signal). In some examples, the depth map may become invalid as the endoscope is moved too close to the scene because too much of the image is out of focus and there are too many poor quality depth samples. In this example, preserving the current disparity adjustments rather than reverting to the nominal disparity may be preferred. As an alternative, or in addition, the system status or information may prevent or inhibit the system from modifying stereo separation distance(s) and / or target display distance(s). Similarly, the rate at which a stereo separation distance and / or target display distance for a graphical user interface element are modified or adjusted (e.g., over a series of frames of endoscopic video captured by the imaging system) may be based on the system status or information. For example, instrument control mode may indicate whether one or more instruments are in an active or controllable state by a control system (e.g., the control system 1020) (e.g., the instrument control mode may be a mode of operation in which one or more input devices of the control system 1020 teleoperationally controls movement and / or operation of one or more instruments supported by manipulator arms). In the instrument control mode, motion of the imaging system may be disabled. Thus, the instrument control mode may indicate that the imaging system is not in motion and therefore, any detected changes in depth are based on instrument movement and manipulation of tissue. Unlike gross motion associated with imaging system motion, instrument or tissue movement may have more subtle effects on the depth perception of graphical user interface elements and consequently, stereo separation distance and / or target display distance may be adjusted slowly and / or progressively (e.g., as compared with in the imaging system control mode). As another example, the imaging system controlmode may indicate whether the imaging system is in an active or controllable state (e.g., the imaging system control mode may be a mode of operation in which one or more input devices of the control system 1020 teleoperationally controls movement and / or operation of one or more components of the imaging system 1015). In the imaging system control mode, the determined changes in depth may be associated with gross motions of the imaging system. Consequently, stereo separation distance may be adjusted more quickly or to match imaging system motion (e.g., as compared with in the instrument control mode). In some examples, a rate of adjustment between the first target display distance and the second target display distance may be determined based on a system status. For example, a maximum rate at which a stereo separation distance and / or target display distance for a graphical user interface element are modified or adjusted (e.g., over a series of frames of endoscopic video captured by the imaging system) may be based on the system status or information. For example, a maximum adjustment rate in an instrument control mode (e.g., where more subtle adjustment is preferred) may be lower than a maximum adjustment rate in an imaging system control mode (e.g., where rapid adjustment may be tolerated). As another example, rapid adjustment may be permitted only while the camera is moving and briefly after camera movement has ceased, in order to improve settling. This settling period may also extend for a brief period after instrument control is resumed.

[0064] As another example, system status or information may include an indication of display system disengagement (e.g., indicating the user’s head has become disengaged from or is not present at a “head-in” display system such as display system 1035) or another transition mode (e.g. instrument exchange being performed) is determined, stereo separation may be adjusted more quickly (e.g., as compared with in the instrument control mode). As another example, the display system 1035 may include one or more eye tracking systems to track or monitor the gaze of the eyes of the user and eye gaze tracking information may indicate a location within the field of view presented on the display system 1035 to which the user’s gaze is directed. And stereo separation distance and / or target display distance of graphical user interface elements at or near (e.g., within a threshold distance of) the user’s gaze location may be adjusted more slowly and / or progressively as compared with graphical user interface elements far from the user’s gaze location. In some examples, process 510 may be omitted and modification of the target display distance may be implemented at process 512, without consideration of system control factors. In some examples, the user’s gaze may also influence the fit of a large planar user interface element to the depth map in order to further reduce depth conflicts for regions where the user is known to be looking.

[0065] At a process 512, the graphical user interface element may be displayed at a modified target display distance. In some examples, the process 512 may be similar to the process 310. The modified target display distance may be the second target display distance or may be a display distance between the first and second target display distance if the transition from the first target display distance to the second target display distance is incremental or gradual. As in process 310, the graphical user interface element may be displayed at the modified target display distance (which in some examples may be the second target display distance) overlayed on the stereoscopic image data of the field of view. Displaying the graphical user interface element at the modified target display distance may include displaying the left and right images of the graphical user interface element at a modified stereo separation distance (which may be the second stereo separation distance associated with the second target display distance). The left and right eye images forming the graphical user interface element may be presented at the modified target display distance while the stereoscopic image data of the field of view is presented at the convergence depth (or another depth). For example, as shown in FIG. 7C, the graphical user interface elements 630, 632 may be displayed at the modified or second target display distance (e.g., the minimum depth 0.04 m), overlayed on the imagery of the field of view 601.

[0066] Modifying the presentation of the graphical user interface element to be displayed at a new target display distance (and the associated stereo separation distance modification) may be conditioned on any of a variety of factors to minimize distraction and maximize comfort for the viewer. For example, the displayed graphical user interface element depth may be modified if a display difference is greater than a predetermined threshold. A display difference may be difference between the second target display distance and the prior target display distance. If the difference is below a predetermined threshold (i.e. relatively little change in depth), no change in depth for the displayed graphical use interface element may be implemented. As another example, the speed of the modification may be conditioned on a variety of factors including the instrument control mode, an imaging system control mode, or a viewer disengagement mode. For example, if the system control mode is an instrument control mode, the target display distance (and the corresponding stereo separation distance) may be adjusted slowly and / or progressively. If the system control mode is an imaging system control mode, the target display distance (and the corresponding stereo separation distance) may be adjusted more quickly to track the imaging system motion. In some examples, head-out transitions (e.g., periods when a user’s head is disengaged from a viewport) may be used to make a faster adjustment while the user is not looking into the viewer.

[0067] Another method for resolving window violations and depth conflicts is described in FIG. 12. The method of FIG. 12 may be used together with any of the methods described above or may be implemented independently. FIG. 12 illustrates a method for synthetically adjusting the fusion or convergence of stereoscopic image data based on a determined working distance. This method may be of particular use to reduce or eliminate window violations or depth conflicts if active instruments or tissue (e.g., where the viewer is likely looking) in the field of view are between the convergence plane 220 and the minimum focal depth 230. At a process 902, stereoscopic image data may be received for a field of view of an imaging system. This process may be substantially similar to process 302.

[0068] At a process 904, depth information for the field of view may be determined. This process may be substantially similar to process 304 and may include the use of depth map and kinematic information for the imaging system and / or instruments. In some examples, a working distance estimate may be obtained based on the minimum non- zero pixel depth found in the depth map across the entire field of view or within a central region of interest. These raw depth values may be conditioned to avoid errors due to temporal noise and false-positive correlation matches. The depth values from the full resolution framebuffer may be temporally and spatially filtered to produce coarse regional depth estimates (e.g., down sampled depth samples reflect > 1 cm sized depth patches and 5 Hz frequency content). This depth map information may be supplemented with kinematic information about the imaging system or the instruments in the field of view. Calibrated kinematics and instrument motion tracking may provide accurate estimates of the instrument tip position and instrument shaft poses (e.g., with approximately 2- 3 mm relative positional uncertainty between the instrument arm and the endoscope tip). In close-up arrangements, the instrument shafts may be a source of window violations that may not be well represented in the depth map since the window violations often correspond to regions of the image where there is not sufficient overlap between the stereo view volumes. Therefore, knowledge of the instrument shaft pose in the imaging system frame of reference may allow computation of an intersection between the shaft segment and the view volume to produce a 3D point that represents where the instrument enters the endoscope field of view. The distance from that intersection point to the endoscope tip minus the radius of the instrument shaft may provide another useful reference depth for working distance. Similarly, the instrument distal clevis may provide another point of reference for estimating working distance, particularly when working close-up and the specular highlight regions from the shiny metal instrument surfaces may cause problems with depth mapping. The distance between the distal clevis point in the endoscope tip frame minus the radius of the instrument distal clevis may bean important supplemental depth measure. The minimum depth of these points of interest may be inversely mapped as a function of focal length and baseline to an image-space disparity that can be used for shifting the left and right stereo video images. Half of the disparity may be applied to shift the two images by equal amounts in opposing directions.

[0069] At a process 906, a working distance for the field of view may be determined from the depth information. The working distance may be a distance from the imaging system (e.g., from the distal tip of an endoscope) at which an active instrument end effector, a moving tissue, a tissue of interest, or other structure of interest in the field of view is located. The process 906 may determine working distance from the depth information in a similar manner as the target display distance is determined from the depth information at process 306.

[0070] At a process 908, a stereo separation distance for stereo presentation of the stereoscopic image data may be determined. The stereo separation may be based on the determined working distance. Similar to the process 308 described above, a stereo separation distance may be determined to display the left and right images of the field of view at the working distance. The stereo separation distance may be a measure of the effective disparity (including physical optical display separation and synthetic shift generated by shifting pixels in left and / or right eye images) to achieve fusion at the working distance. For example, stereo separation distance may be a measure of the horizontal shift between pixels in the right and left images that provide an adjustment of the imaging of the structures at the working distance in the field of view.

[0071] At a process 910, the stereoscopic image data may be displayed at the working distance. When displayed at the working distance, the stereoscopic imagery may appear to be located on the viewing window, with no structures in the field of view protruding from the viewing window. Adjustments to the working distance may be made in a comfortable and nondistracting manner based on system control mode or other conditions as described above. For example, changes in video disparity may be applied in a very slow, smooth, and continuous manner so the user’s eyes naturally accommodate the change without distraction.

[0072] FIGS. 14-16 together provide an overview of a medical system 1010 that may be used in, for example, medical procedures including diagnostic, therapeutic, or surgical procedures. The stereo imagery display examples provided above may be used in the context of the medical system 1010. The medical system 1010 is located in a medical environment 1011. The medical environment 1011 is depicted as an operating room in FIG. 14. In other embodiments, the medical environment 1011 may be an emergency room, a medical training environment, a medical laboratory, or some other type of environment in which any number of medicalprocedures or medical training procedures may take place. In still other embodiments, the medical environment 1011 may include an operating room and a control area located outside of the operating room.

[0073] In one or more embodiments, the medical system 1010 may be a robot-assisted medical system that is under the teleoperational control of an operator (e.g., a surgeon, a clinician, a physician, etc.). In alternative embodiments, the medical system 1010 may be under the partial control of a computer programmed to perform the medical procedure or subprocedure. In still other alternative embodiments, the medical system 1010 may be a fully automated medical system that is under the full control of a computer programmed to perform the medical procedure or sub-procedure with the medical system 1010. One example of the medical system 1010 that may be used to implement the systems and techniques described in this disclosure is the da Vinci® Surgical System manufactured by Intuitive Surgical, Inc. of Sunnyvale, California.

[0074] As shown in FIG. 14, the medical system 1010 generally includes an assembly 1012, which may be mounted to or positioned near an operating table T on which a patient P is positioned. The assembly 1012 may be referred to as a patient side cart, a surgical cart, or a surgical robot. In one or more embodiments, the assembly 1012 may be a teleoperational assembly. The teleoperational assembly may be referred to as, for example, a teleoperational arm cart. A medical instrument system 1014 and an endoscopic imaging system 1015 are operably coupled to the assembly 1012. An operator input system 1016 allows an operator O or other type of clinician to view images of or representing the surgical site and to control the operation of the medical instrument system 1014 and / or the endoscopic imaging system 1015.

[0075] The medical instrument system 1014 may comprise one or more medical instruments. In embodiments in which the medical instrument system 1014 comprises a plurality of medical instruments, the plurality of medical instruments may include multiple of the same medical instrument and / or multiple different medical instruments. Similarly, the endoscopic imaging system 1015 may comprise one or more endoscopes. In the case of a plurality of endoscopes, the plurality of endoscopes may include multiple of the same endoscope and / or multiple different endoscopes.

[0076] The operator input system 1016 may be located at an operator's control console, which may be located in the same room as operating table T. In some embodiments, the operator O and the operator input system 1016 may be located in a different room or a completely different building from the patient P. The operator input system 1016 generally includes one or more control device(s) for controlling the medical instrument system 1014. The control device(s)may include one or more of any number of a variety of input devices, such as hand grips, joysticks, trackballs, data gloves, trigger-guns, foot pedals, hand-operated controllers, voice recognition devices, touch screens, body motion or presence sensors, and other types of input devices.

[0077] In some embodiments, the control device(s) will be provided with the same degrees of freedom as the medical instrument(s) of the medical instrument system 1014 to provide the operator with telepresence, which is the perception that the control device(s) are integral with the instruments so that the operator has a strong sense of directly controlling instruments as if present at the surgical site. In other embodiments, the control device(s) may have more or fewer degrees of freedom than the associated medical instruments and still provide the operator with telepresence. In some embodiments, the control device(s) are manual input devices that are movable with six degrees of freedom, and which may also include an actuatable handle for actuating instruments (for example, for closing grasping jaw end effectors, applying an electrical potential to an electrode, delivering a medicinal treatment, and actuating other types of instruments).

[0078] The assembly 1012 may support and manipulate the medical instrument system 1014 while the operator O views the surgical site through the operator input system 1016. An image of the surgical site may be obtained by the endoscopic imaging system 1015, which may be manipulated by the assembly 1012. The assembly 1012 may comprise endoscopic imaging systems 1015 and may similarly comprise multiple medical instrument systems 1014 as well. The number of medical instrument systems 1014 used at one time will generally depend on the diagnostic or surgical procedure to be performed and on space constraints within the operating room, among other factors. The assembly 1012 may include a kinematic structure of one or more non-servo controlled links (e.g., one or more links that may be manually positioned and locked in place, generally referred to as a set-up structure) and a manipulator. When the manipulator takes the form of a teleoperational manipulator, the assembly 1012 is a teleoperational assembly. The assembly 1012 includes a plurality of motors that drive inputs on the medical instrument system 1014. In an embodiment, these motors move in response to commands from a control system (e.g., control system 1020). The motors include drive systems which when coupled to the medical instrument system 1014 may advance a medical instrument into a naturally or surgically created anatomical orifice. Other motorized drive systems may move the distal end of said medical instrument in multiple degrees of freedom, which may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes).Additionally, the motors may be used to actuate an articulable end effector of the medical instrument for grasping tissue in the jaws of a biopsy device or the like. Medical instruments of the medical instrument system 1014 may include end effectors having a single working member such as a scalpel, a blunt blade, an optical fiber, or an electrode. Other end effectors may include, for example, forceps, graspers, scissors, or clip appliers.

[0079] The medical system 1010 also includes a control system 1020. The control system 1020 includes at least one memory 1024 and at least one processor 1022 for effecting control between the medical instrument system 1014, the operator input system 1016, and other auxiliary systems 1026 which may include, for example, imaging systems, audio systems, fluid delivery systems, display systems, illumination systems, steering control systems, irrigation systems, and / or suction systems. A clinician may circulate within the medical environment 1011 and may access, for example, the assembly 1012 during a set up procedure or view a display of the auxiliary system 1026 from the patient bedside.

[0080] Though depicted as being external to the assembly 1012 in FIG. 14, the control system 1020 may, in some embodiments, be contained wholly within the assembly 1012. The control system 1020 also includes programmed instructions (e.g., stored on a non-transitory, computer- readable medium) to implement some or all of the methods described in accordance with aspects disclosed herein. While the control system 1020 is shown as a single block in the simplified schematic of FIG. 14, the control system 1020 may include two or more data processing units or circuits with one portion of the processing optionally being performed on or adjacent the assembly 1012, another portion of the processing being performed at the operator input system 1016, and the like.

[0081] Any of a wide variety of centralized or distributed data processing architectures may be employed. Similarly, the programmed instructions may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the systems described herein, including teleoperational systems. In one embodiment, the control system 1020 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry.

[0082] In some embodiments, control system 1020 may include one or more servo controllers that receive force and / or torque feedback from the medical instrument system 1014. Responsive to the feedback, the servo controllers transmit signals to the operator input system 1016. The servo controller(s) may also transmit signals instructing assembly 1012 to move the medical instrument system(s) 1014 and / or endoscopic imaging system 1015 which extend into an internal surgical site within the patient body via openings in the body. Any suitableconventional or specialized servo controller may be used. A servo controller may be separate from, or integrated with, assembly 1012. In some embodiments, the servo controller and assembly 1012 are provided as part of a teleoperational arm cart positioned adjacent to the patient's body.

[0083] The control system 1020 can be coupled with the endoscopic imaging system 1015 and can include a processor to process captured images for subsequent display, such as to an operator on the operator's control console, or on another suitable display located locally and / or remotely. For example, where a stereoscopic endoscope is used, the control system 1020 can process the captured images to present the operator with coordinated stereo images of the surgical site. Such coordination can include alignment between the opposing images and can include adjusting the stereo working distance of the stereoscopic endoscope.

[0084] In alternative embodiments, the medical system 1010 may include more than one assembly 1012 and / or more than one operator input system 1016. The exact number of assemblies 1012 will depend on the surgical procedure and the space constraints within the operating room, among other factors. The operator input systems 1016 may be collocated or they may be positioned in separate locations. Multiple operator input systems 1016 allow more than one operator to control one or more assemblies 1012 in various combinations. The medical system 1010 may also be used to train and rehearse medical procedures.

[0085] FIG. 15 is a perspective view of one embodiment of an assembly 1012 which may be referred to as a patient side cart, surgical cart, teleoperational arm cart, manipulator assembly or surgical robot. The assembly 1012 shown provides for the manipulation of three surgical tools 1030a, 1030b, and 1030c (e.g., medical instrument systems 1014) and an imaging device 1028 (e.g., endoscopic imaging system 1015), such as a stereoscopic endoscope used for the capture of images of the site of the procedure. The imaging device may transmit signals over a cable 1056 to the control system 1020. Manipulation is provided by teleoperative mechanisms having a number of joints. The imaging device 1028 and the surgical tools 1030a-c can be positioned and manipulated through incisions in the patient so that a kinematic remote center is maintained at the incision to minimize the size of the incision. Images of the surgical site can include images of the distal ends of the surgical tools 1030a-c when they are positioned within the field of view of the imaging device 1028.

[0086] The assembly 1012 includes a drivable base 1058. The drivable base 1058 is connected to a telescoping column 1057, which allows for adjustment of the height of arms 1054. The arms 1054 may include a rotating joint 1055 that both rotates and moves up and down. Each of the arms 1054 may be connected to an orienting platform 1053. The arms 1054 may belabeled to facilitate trouble shooting. For example, each of the arms 1054 may be emblazoned with a different number, letter, symbol, other identifier, or combinations thereof. The orienting platform 1053 may be capable of 1060 degrees of rotation. The assembly 1012 may also include a telescoping horizontal cantilever 1052 for moving the orienting platform 1053 in a horizontal direction.

[0087] In the present example, each of the arms 1054 connects to a manipulator arm 1051. The manipulator arms 1051 may connect directly to a medical instrument, e.g., one of the surgical tools 1030a-c. The manipulator arms 1051 may be teleoperable. In some examples, the arms 1054 connecting to the orienting platform 1053 may not be teleoperable. Rather, such arms 1054 may be positioned as desired before the operator O begins operation with the teleoperative components. Throughout a surgical procedure, medical instruments may be removed and replaced with other instruments such that instrument to arm associations may change during the procedure.

[0088] Endoscopic imaging systems (e.g., endoscopic imaging system 1015 and imaging device 1028) may be provided in a variety of configurations including rigid or flexible endoscopes. Rigid endoscopes include a rigid tube housing a relay lens system for transmitting an image from a distal end to a proximal end of the endoscope. Flexible endoscopes transmit images using one or more flexible optical fibers. Digital image-based endoscopes have a “chip on the tip” design in which a distal digital sensor such as a one or more charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) device store image data. Endoscopic imaging systems may provide two- or three- dimensional images to the viewer. Two-dimensional images may provide limited depth perception. Three-dimensional stereo endoscopic images may provide the viewer with more accurate depth perception. Stereo endoscopic instruments employ stereo cameras to capture stereo images of the patient anatomy. An endoscopic instrument may be a fully sterilizable assembly with the endoscope cable, handle, and shaft all rigidly coupled and hermetically sealed.

[0089] FIG. 16 is a perspective view of an embodiment of the operator input system 1016 at the operator’s control console. The operator input system 1016 includes a display system with a left eye display 1032 and a right eye display 1034 for presenting the operator O with a coordinated stereo view of the surgical environment that enables depth perception. The left and right eye displays 1032, 1032 together may be a viewport and may be components of a display system 1035. In other embodiments, the display system 1035 may include one or more other types of displays. The display system 1035 may present images captured, for example, by the imaging system 1015 to display the endoscopic field of view to the operator. The endoscopicfield of view may be augmented by graphical user interface elements such as virtual or synthetic menus, indicators, and / or other graphical or textual information to provide additional information to the viewer.

[0090] The operator input system 1016 further includes one or more input control devices 1036, which in turn cause the assembly 1012 to manipulate one or more instruments of the endoscopic imaging system 1015 and / or medical instrument system 1014. The input control devices 1036 can provide the same degrees of freedom as their associated instruments to provide the operator O with telepresence, or the perception that the input control devices 1036 are integral with said instruments so that the operator has a strong sense of directly controlling the instruments. To this end, position, force, and tactile feedback sensors (not shown) may be employed to transmit position, force, and tactile sensations from the medical instruments, e.g., surgical tools 1030a-c, or imaging device 1028, back to the operator's hands through the input control devices 1036. Input control devices 1039 are foot pedals that receive input from a user’ s foot. Aspects of the operator input system 1016, the assembly 1012, and the auxiliary systems 1026 may be adjustable and customizable to meet the physical needs, skill level, or preferences of the operator O.

[0091] Elements described in detail with reference to one embodiment, implementation, or application optionally may be included, whenever practical, in other embodiments, implementations, or applications in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or implementation non-functional, or unless two or more of the elements provide conflicting functions.

[0092] Any alterations and further modifications to the described devices, systems, instruments, methods, and any further application of the principles of the present disclosure are fully contemplated as would normally occur to one skilled in the art to which the disclosure relates. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one embodiment may be combined with the features, components, and / or steps described with respect to other embodiments of the present disclosure. In addition, dimensions provided herein are for specific examples and it is contemplated that different sizes,dimensions, and / or ratios may be utilized to implement the concepts of the present disclosure. To avoid needless descriptive repetition, one or more components or actions described in accordance with one illustrative embodiment can be used or omitted as applicable from other illustrative embodiments. For the sake of brevity, the numerous iterations of these combinations will not be described separately.

[0093] Various systems and portions of systems have been described in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an object or a portion of an object in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian X, Y, Z coordinates). As used herein, the term “orientation” refers to the rotational placement of an object or a portion of an object (three degrees of rotational freedom - e.g., roll, pitch, and yaw). As used herein, the term “pose” refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of the object in at least one degree of rotational freedom (up to six total degrees of freedom).

[0094] Although some of the examples described herein refer to surgical procedures or instruments, or medical procedures and medical instruments, the techniques disclosed optionally apply to non-medical procedures and non-medical instruments. For example, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, and sensing or manipulating non-tissue work pieces. Other example applications involve cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, and training medical or non- medical personnel. Additional example applications include use for procedures on tissue removed from human or animal anatomies (without return to a human or animal anatomy) and performing procedures on human or animal cadavers. Further, these techniques can also be used for surgical and nonsurgical medical treatment or diagnosis procedures.

[0095] A computer is a machine that follows programmed instructions to perform mathematical or logical functions on input information to produce processed output information. A computer includes a logic unit that performs the mathematical or logical functions, and memory that stores the programmed instructions, the input information, and the output information. The term “computer” and similar terms, such as “processor” or “controller” or “control system,” are analogous.

[0096] While certain exemplary embodiments of the invention have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that the embodiments of theinvention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art.

Claims

CLAIMSWhat is claimed is:1 . A medical system comprising: a display system; and a control system, wherein the control system includes a processing unit including one or more processors, and wherein the processing unit is configured to: receive stereoscopic image data for a field of view of an imaging system; determine depth information for the field of view; determine a target display distance for a graphical user interface element based on the depth information; determine a stereo separation distance for stereo presentation of the graphical user interface element based on the determined target display distance; and display, on the display system, the graphical user interface element at the target display distance with the stereoscopic image data of the field of view.

2. The medical system of claim 1 , wherein displaying the graphical user interface element at the target display distance includes displaying the graphical user interface element in stereo presentation in accordance with the stereo separation distance.

3. The medical system of claim 2, wherein displaying the graphical user interface element in stereo presentation in accordance with the stereo separation distance includes displaying a right eye image including the graphical user interface element and displaying a left eye image including the graphical user interface element, wherein the graphical user interface element in the right eye image is shifted by the stereo separation distance relative to the graphical user interface element in the left eye image.

4. The medical system of claim 1, wherein the processing unit is further configured to display the stereoscopic image data in accordance with a second stereo separation distance that is different from the stereo separation distance for the graphical user interface element.

5. The medical system of claim 1, wherein determining depth information includes determining a minimum depth for the field of view.

6. The medical system of claim 1 , wherein determining depth information includes summary depth information for each of a plurality of regions of the field of view.

7. The medical system of claim 6, wherein determining the target display distance for the graphical user interface element includes determining a minimum depth from the summary depth information for a first region of the plurality of regions of the field of view.

8. The medical system of claim 6, wherein determining the target display distance for the graphical user interface element includes determining a percentile-based depth from the summary depth information for a first region of the plurality of regions of the field of view.

9. The medical system of claim 6, wherein displaying the graphical user interface element includes displaying a first graphical user element in a first region of the plurality of regions at a first target display distance and displaying a second graphical user interface element in a second region of the plurality of regions at a second target display distance.

10. The medical system of claim 1, wherein the graphical user interface element is a two- dimensional graphical user interface element.

11. The medical system of claim 1 , wherein determining depth information includes receiving kinematic information for the imaging system.

12. The medical system of claim 11, wherein the kinematic information includes a kinematic position of a distal end portion of an endoscopic instrument of the imaging system.

13. The medical system of claim 11, wherein the kinematic information includes movement information for an endoscopic instrument of the imaging system.

14. The medical system of claim 1, wherein determining depth information includes receiving kinematic information for an instrument in the field of view.

15. The medical system of claim 1, wherein determining depth information includes receiving digital zoom status information for the imaging system.

16. The medical system of claim 1, wherein the target display distance is determined from a minimum depth for the field of view.

17. The medical system of claim 1 , further comprising: adjusting the stereo separation distance in response to a system status indicating one or more of: a display system disengagement or an instrument exchange being performed.

18. The medical system of claim 1 , wherein the target display distance is a first target display distance and wherein the processing unit is further configured to: receive stereoscopic image data for a second field of view of the imaging system; determine second depth information for the second field of view; determine a second target display distance for the graphical user interface element based on the second depth information; determine a second stereo separation distance for stereo presentation of the graphical user interface element based on the determined second target display distance; and modify the first target display distance for the graphical user interface if a display difference between the first target display distance and the second target display distance is greater than a predetermined threshold.

19. The medical system of claim 18, wherein the processing unit is further configured to: receive a system control mode, wherein modifying the first target display distance includes adjusting a speed of the modification based on the system control mode.

20. The medical system of claim 18, wherein the processing unit is further configured to: determine a stereo separation distance modification based on the display difference.

21. The medical system of claim 20, wherein the processing unit is further configured to: display the graphical user interface element in stereo presentation at the stereo separation distance, adjusted by the stereo separation distance modification.

22. The medical system of claim 18, wherein a rate of adjustment between the first target display distance and the second target display distance is determined based on a system status.

23. The medical system of claim 1, wherein the processing unit is further configured to receive a user input indicating a selected display parameter and wherein determining the target display distance for the graphical user interface element is based on the selected display parameter.

24. The medical system of claim 23, wherein the selected display parameter generates a target display distance that is at a minimum depth for the field of view.

25. The medical system of claim 23, wherein the selected display parameter generates a target display distance that is less than a minimum depth for the field of view.

26. A method comprising: receiving stereoscopic image data for a field of view of an imaging system; determining depth information for the field of view; determining a target display distance for a graphical user interface element based on the depth information; determining a stereo separation distance for stereo presentation of the graphical user interface element based on the determined target display distance; and displaying the graphical user interface element at the target display distance with the stereoscopic image data of the field of view.

27. The method of claim 26, wherein displaying the graphical user interface element at the target display distance includes displaying the graphical user interface element in stereo presentation in accordance with the stereo separation distance.

28. The method of claim 27, wherein displaying the graphical user interface element in stereo presentation in accordance with the stereo separation distance includes displaying a right eye image including the graphical user interface element and displaying a left eye image including the graphical user interface element, wherein the graphical user interface element in the right eye image is shifted by the stereo separation distance relative to the graphical user interface element in the left eye image.

29. The method of claim 26, further comprising displaying the stereoscopic image data in accordance with a second stereo separation distance that is different from the stereo separation distance for the graphical user interface element.

30. The method of claim 26, wherein determining depth information includes determining a minimum depth for the field of view. 1 . The method of claim 26, wherein determining depth information includes determining summary depth information for each of a plurality of regions of the field of view.

32. The method of claim 31 , wherein determining the target display distance for the graphical user interface element includes determining a minimum depth from the summary depth information for a first region of the plurality of regions of the field of view.

33. The method of claim 31 , wherein displaying the graphical user interface element includes displaying a first graphical user element in a first region of the plurality of regions at a first target display distance and displaying a second graphical user element in a second region of the plurality of regions at a second target display distance.

34. The method of claim 26, wherein the graphical user interface element is a two- dimensional graphical user interface element.

35. The method of claim 26, wherein determining depth information includes receiving kinematic information for the imaging system.

36. The method of claim 35, wherein the kinematic information includes a kinematic location of a distal end portion of an endoscopic instrument of the imaging system.

37. The method of claim 35, wherein the kinematic information includes movement information for an endoscopic instrument of the imaging system.

38. The method of claim 26, wherein determining depth information includes receiving kinematic information for an instrument in the field of view.

39. The method of claim 26, wherein determining depth information includes receiving digital zoom status information for the imaging system.

40. The method of claim 26, wherein the target display distance is determined from a minimum depth for the field of view.

41. The method of claim 26, further comprising: adjusting the stereo separation distance in response to a system status indicating one or more of: a display system disengagement or an instrument exchange being performed.

42. The method of claim 26, wherein the target display distance is a first target display distance, the method further comprising: receiving stereoscopic image data for a second field of view of the imaging system; determining second depth information for the second field of view; determining a second target display distance for the graphical user interface element based on the second depth information; determining a second stereo separation distance for stereo presentation of the graphical user interface element based on the determined second target display distance; and modifying the first target display distance for the graphical user interface if a display difference between the first target display distance and the second target display distance is greater than a predetermined threshold.

43. The method of claim 42, further comprising: receiving a system control mode, wherein modifying the first target display distance includes adjusting a speed of the modification based on the system control mode.

44. The method of claim 42, further comprising: determining a stereo separation distance modification based on the display difference.

45. The method of claim 44, further comprising: displaying the graphical user interface element in stereo presentation at the stereo separation distance, adjusted by the stereo separation distance modification.

46. The method of claim 42, wherein a rate of adjustment between the first target display distance and the second target display distance is determined based on a system status.

47. The method of claim 26, further comprising: receiving a user input indicating a selected display parameter, wherein determining the target display distance for the graphical user interface element is based on the selected display parameter.

48. The method of claim 47, wherein the selected display parameter generates a target display distance that is at a minimum depth for the field of view.

49. The method of claim 47, wherein the selected display parameter generates a target display distance that is less than a minimum depth for the field of view.

50. A medical system comprising: a display system; and a control system, wherein the control system includes a processing unit including one or more processors, and wherein the processing unit is configured to: receive stereoscopic image data for a field of view of an imaging system; determine depth information for the field of view; determine a working distance for the field of view; determine a stereo separation distance for stereo presentation of the stereoscopic image data based on the working distance; and display, on the display system, the stereoscopic image data at the working distance.

51. The medical system of claim 50, wherein the working distance is a distance between a distal end portion of an endoscope of the imaging system and a structure of interest in the field of view.

52. The medical system of claim 51, wherein the structure of interest in the field of view includes an active instrument end effector.

53. The medical system of claim 51, wherein the structure of interest in the field of view includes a tissue determined to be in motion.

54. The medical system of claim 51, wherein the structure of interest in the field of view includes a tissue of interest.

55. The medical system of claim 1, stereo separation distance includes a measure of effective optical separation to achieve optical fusion at the working distance for the field of view.

Images