Systems and methods for guided tool change resilience

Abstract

The medical system may include a manipulator assembly and a control system. The control system includes a processing unit. The processing unit determines kinematic information associated with a first tool inserted into a work site. The first tool is coupled to the manipulator assembly. The processing unit also receives image data generated by an endoscopic imaging instrument having a field of view. The image data is generated by the endoscopic imaging instrument after the first tool is removed from the manipulator assembly. The processing unit also determines one or more guided tool change parameters for guiding a second tool into the work site. The second tool is received in connection with the manipulator assembly after the first tool is removed. The guided tool change parameters are based on the kinematic information associated with the first tool coupled to the manipulator assembly and on a depth map determined from the image data.

Description

[Technical Field]

[0001] Cross-reference to application This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 477,583, entitled "Systems and Methods for Guided Tool Change Resiliency," filed December 29, 2022, the contents of which are incorporated herein by reference.

[0002] Examples described herein relate to systems and methods for using endoscopic image data, including depth mapping or vision-based tool tracking, in robotic-assisted tool exchange procedures. [Background technology]

[0003] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during a medical procedure, thereby reducing patient recovery time, discomfort, and adverse side effects. Such minimally invasive techniques can be performed through natural orifices in the patient's anatomy or through one or more surgical incisions. Through these natural orifices or incisions, an operator can insert minimally invasive medical tools to reach a target tissue site. Minimally invasive medical tools include instruments such as therapeutic instruments, diagnostic instruments, biopsy instruments, and surgical instruments. Minimally invasive medical tools can also include imaging instruments, such as endoscopic instruments, that provide a user with a field of view within the patient's anatomy. Some minimally invasive medical tools and imaging instruments may be robotically or computer-assisted. A medical procedure may use a large number of different medical tools. When a different tool is needed during a medical procedure, the medical tool can be withdrawn from the surgical site, detached from its associated manipulator, and replaced with another tool. The new tool is then inserted into the surgical site. Improved systems and methods are needed to improve the reliability and availability of control-system-assisted tool exchange procedures. Summary of the Invention

[0004] The following presents a simplified summary of various examples described herein and is not intended to identify key or critical elements or to limit the scope of the claims.

[0005] According to some examples, a medical system may include a manipulator assembly and a control system. The control system includes a processing unit. The processing unit determines kinematic information associated with a first tool inserted into a work site. The first tool is coupled to the manipulator assembly. The processing unit also receives image data generated by an endoscopic imaging instrument having a field of view. The image data is generated by the endoscopic imaging instrument after the first tool is removed from the manipulator assembly. The processing unit also determines one or more guided tool exchange parameters for guiding a second tool to the work site. The second tool is coupled to and received by the manipulator assembly after the first tool is removed. The guided tool exchange parameters are based on the kinematic information associated with the first tool coupled to the manipulator assembly and based on a depth map determined from the image data.

[0006] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope thereof. In that regard, further aspects, features, and advantages of the present disclosure will become apparent to those skilled in the art from the following detailed description. [Brief explanation of the drawings]

[0007] [Figure 1A] 1 is a schematic diagram of a first medical tool and an imaging tool within a work site, according to some examples. [Figure 1B] 10 is a schematic diagram of a second medical tool and an imaging tool within a work site during or after a tool exchange, according to some examples. [Figure 2] 1 is a schematic diagram of a medical system, according to some examples. [Figure 3] 1 is a flowchart illustrating a method of performing a guided tool change procedure, according to some examples. [Figure 4A] 1 is a flowchart illustrating a method for generating guided tool change parameters for guiding a new tool, according to some examples. [Figure 4B] 10A-10C are schematic diagrams of target locations and insertion paths of secondary tools selected to correspond to the range of motion of a manipulator, according to some examples. [Figure 5] 1 is a flowchart illustrating a method for generating guided tool change parameters for guiding a new tool, according to some examples. [Figure 6] 1 is a flowchart illustrating a method for generating guided tool change parameters for guiding a new tool, according to some examples. [Figure 7A] FIG. 10 illustrates a graphical user interface displaying visual guidance for a guided tool change procedure. [Figure 7B] FIG. 10 illustrates a graphical user interface displaying visual guidance for a guided tool change procedure. [Figure 7C] FIG. 10 illustrates a graphical user interface displaying visual guidance for a guided tool change procedure. [Figure 7D] 1 illustrates a graphical user interface displaying visual guidance for a guided tool change procedure. Various examples described herein and their advantages are explained in the detailed description below. For purposes of describing the various examples described herein, like reference numerals may be used to identify like elements shown in one or more of the figures, but it should be understood that this is not intended to limit the various examples described herein. DETAILED DESCRIPTION OF THE INVENTION

[0008] A guided tool change procedure can kinematically determine the position of a tool attached to a robot-assisted manipulator and can help deliver a replacement tool to the position of the first tool. Various circumstances can cause the robot-assisted control system to disable the guided tool change or allow the guided tool change if an undetectable kinematic change occurs. Several systems and methods for guided tool change are described in U.S. Pat. No. 6,633,953, which is incorporated herein by reference in its entirety. The control system of a robot-assisted manipulator may determine that the replacement tool cannot be placed at the position of the first tool and disable or cancel the control system-guided tool change procedure, requiring the medical professional to manually introduce the replacement tool, leading to workflow inefficiencies and inaccurate tool positioning. Various reasons can contribute to the control system-guided tool change procedure being disabled. For example, tool tip inaccuracies can disable the control system-guided tool change procedure. Inaccuracies in the kinematic chain estimation of the position and / or orientation of the medical tool tip may result, for example, from a tool end effector with long jaws or from flexible joints with uncertain kinematic poses. Additionally or alternatively, the control system's guided tool change procedure may be disabled due to limitations in the manipulator's range of motion or if the manipulator encounters an obstacle during a reorientation attempt. Additionally or alternatively, the control system's guided tool change procedure may be disabled even if the control system senses manual intervention, such as when the manipulator is manually adjusted or when a clutch mechanism is engaged to initiate manual movement of the manipulator. Tissue movement, instrument deflection, and / or manipulator deflection may not be kinematically detected, and the control system uses incomplete information to determine the control system's guided tool change parameters.To provide a more robust control system guided tool change procedure that minimizes invalidations, improves workflow efficiency, and enables more robust availability, kinematic-based tool change inputs can be supplemented with image-based information to generate tool change parameters. A more robust technique for guided tool change can utilize endoscopic image data, including vision-based tool tracking and / or depth mapping, to supplement kinematic positioning and provide a modified insertion path, a modified insertion depth, and / or a modified tip position.

[0009] 1A and 1B illustrate a guided tool exchange operation of a medical instrument or tool 100 (e.g., a first tool) including an end effector 104 having a shaft 102, a clevis 103, and a tip 106. The tip 106 may be the tool tip of the end effector 104. The tool 100 may be inserted through an entry port 109, penetrating an anatomical wall 107, into a work site, which may be an anatomical region 108 within a patient's body cavity. An imaging instrument or tool 118 (e.g., an endoscope) having a field of view 119 may be inserted through the anatomical wall 107 into the anatomical region 108 through a port 111. The end effector 104 and the anatomical tissue 105 may be within the field of view 119 of the imaging tool 118. The medical tool 100 and the imaging tool 118 are manipulated from outside the anatomical region 108 and may have a particular configuration as shown in FIG. 1A. A robotic-assisted manipulator (eg, manipulator assembly 302) including one or more actuators can be used to manipulate the tools 100, 118.

[0010] As shown in FIG. 1B , before removing the tool 100 from the anatomical region 108 and introducing another tool 110 through the port 109, information can be collected from one or more sensor systems (e.g., sensor system 308), including an imaging system (e.g., imaging tool 118), to record the operational configuration (including position, orientation, and / or pose) of the medical tool 100. For example, the position of the tip 106 can be kinematically determined using sensors, such as potentiometers, encoders, or other types of position or velocity sensors, used to determine the kinematic configuration of the manipulator assembly and its associated tools. The sensed position can be used as a target point 122 to calculate parameters for controlling the manipulator arm to manipulate the second tool 110 and guide its movement into the operational configuration (including position, orientation, and / or pose) where the tip 116 of the second tool 110 is located at the target point 122. In some examples, the same manipulator arm used to move the first tool 100 is used to manipulate the second tool 110 during and after the tool exchange operation. In other examples, parameters for controlling the manipulator arm during a guided tool exchange procedure can be determined from other types of sensors, including imaging tools that can generate image data before or after removal of the tool 100. For example, image data generated before removal of the tool 100 can provide information about the position and orientation of the tip 106, including tool or manipulator deflections or obstructions that cannot be detected by kinematic sensors. Image data generated after removal of the tool 100 can provide information about the location of anatomical structures or other tools within the anatomical region that may move after removal of the tool 100. In some examples, the image data can be stereoscopic image data that can be used to generate a depth map that provides information about the distance from the tip of the imaging tool 118 to the surface of an object within the field of view 119. Depth mapping based on stereoscopic image correlation allows reconstruction of the topographical shape of the anatomical environment, enabling the control system to determine the distance from a point within the anatomical region (e.g., the target insertion point) to the nearest surface visible within the field of view.The depth map can represent near and far distances between objects in the field of view 119 and the plane of the imaging tool 118. In some examples, pixels in the depth map can have an associated quality metric (e.g., the vertical error of the nearest match from the epipolar plane) based on the output of a stereo matching process. A quality threshold can be used to evaluate the consistency of some or all of the depth map pixels sampled for projection or ray-casting. The generation of a depth map of a work site is not limited to the use of data from an endoscopic imaging instrument. The depth map can be generated using sensor data from other modalities (e.g., instead of or in combination with endoscopic imaging data). For example, the depth map can be generated using data from depth-sensing sensors (e.g., structured light sensors, time-of-flight (ToF) sensors, etc.), ultrasound sensors, computed tomography (CT), etc. In some implementations, the depth map can be generated by a combined or multimodal sensor (e.g., integrating endoscopic imaging and depth-sensing sensors).

[0011] When the replacement tool 110 is introduced into the anatomical region 108, the recorded operational configuration of the original tool 100 is used to guide the replacement tool 110 so that it is ready for operation, including being positioned in the same position, orientation, and / or posture as the original tool 100. The replacement tool 110 (e.g., the second tool) may include a shaft 112 and an end effector 114 having a tip 116. Using the recorded operational configuration information of the tool 100, the replacement tool 110 can be accurately positioned within the anatomical region in approximately the same position relative to the anatomical structure 105, the field of view 119 of the imaging tool 118, and other tools or structures that may be present within the anatomical region 108.

[0012] In some examples, the guided tool exchange procedure can record the operating position of the tip 106 of the first tool 100 and then use this as a target point 122 to determine the desired position of the tip 116 of the second tool 110. In this way, the insertion depth of the second tool 110 can be limited by the position of the tip 116, preventing the second tool 110 from overextending into the anatomical region 108 and causing the tip 116 to undesirably contact the anatomical region 108. In some examples, the second tool 110 can be introduced via a straight insertion path to position the tip 116 in the operating position. The insertion path is shown in FIG. 1B as a phantom line 120, which extends through a remote center or center of rotation 124 of the manipulator to which the tool 110 is coupled and intersects with a target point 122 associated with the tip 106 of the first tool 100 in the operating position. The remote center 124 can be the point about which the port 109 rotates relative to the anatomical wall 107. The location of the remote center 124 can be fixed relative to the patient wall 107 (e.g., in X, Y, and Z Cartesian coordinate space). In some examples, the remote center 124 is a hardware-constrained remote center of motion, where the location of the remote center 124 is set based on the configuration of the manipulator assembly. Additionally or alternatively, the remote center 124 can be a software-constrained remote center of motion. The insertion path 120 can be used as an insertion axis guide to guide the second tool 110 and is sometimes referred to as the in-out axis or IO axis. The IO axis can represent the tool's degrees of freedom of movement. For example, the tool can be mounted on a carriage driven to translate along a linear guide form of a robotic manipulator arm, which can move with additional degrees of freedom, including angular displacement, to position the tool. In other examples, the insertion path can be curved or any shape that allows the tip 116 to reach the target point 122.

[0013] In some examples, the tip 116 of the second tool 110 does not necessarily need to be positioned exactly at the target point 122. A target space 125 is defined near the target point 122 to provide an allowable area for positioning the tip 116 and may correspond to any location within the surgeon's field of view 119, for example. In some examples, the target space can encompass an offset or bias area of ​​approximately 0.5 cm from the target point. FIG. 1B illustrates a cylindrical target space 125 defined by specifying an allowable distance from the target point 122 for positioning the tip 116. In some examples, the target space 125 can be biased toward a generally distal center 124 to reduce the likelihood of the second tool 110 contacting tissue during insertion. In other examples, the target space can be spherical, rectangular, conical, frusto-conical, or any shape that increases the efficiency of tool exchange or prevents the tool from penetrating too deeply into the anatomical region. The insertion depth limit 126 can provide a depth limit along the insertion path 120 that can limit the insertion of the tool 110. The depth limit 126 can be configured as visual guidance on a display, tactile and / or force feedback to an operator control device, tactile and / or force feedback provided via the second tool 110 and / or manipulator assembly, a software insertion stop, an audio cue, or other type of guidance that limits the distance the end effector 116 of the second tool 110 can contact tissue. In some examples, the depth limit 126 can be biased away from obstacles based on uncertainty in the depth map and uncertainty in the remote center of the manipulator in the endoscope tip reference frame. In one example, tactile and / or force feedback regarding the depth limit 126 can be provided via the second tool 110 and / or manipulator assembly during a tool insertion operation that can be part of a guided tool exchange procedure to insert the second tool 116. The tool insertion operation can be performed tableside.For example, when the end effector 116 of the second tool 110 approaches the depth limit 126 during tool insertion, the manipulator assembly (e.g., a prismatic or rotary joint on the manipulator assembly) can increase force feedback to manually insert the tool and prevent the end effector 116 of the second tool 110 from being inserted beyond the depth limit 126.

[0014] In some examples, the components discussed above may be part of a computer-assisted or robotic-assisted system, as described in further detail below. Such computer-assisted or robotic-assisted systems may be suitable for use in, for example, surgery, robotic-assisted surgery, diagnostic, therapeutic, or biopsy procedures. While some examples are provided herein with respect to such procedures, reference to medical or surgical instruments and methods is non-limiting. The systems, instruments, and methods described herein may be used on animals, human cadavers, animal cadavers, parts of the human or animal anatomy, non-surgical diagnostics, and industrial and general-purpose robotic, general-purpose robotic-assisted, or robotic medical systems.

[0015] As shown in FIG. 2 , medical system 300 generally includes a manipulator assembly 302 that includes manipulator arms 303, 304. Tool 100 may be initially coupled to and manipulated by manipulator arm 303, and after a guided tool exchange procedure, tool 110 may be coupled to and manipulated by manipulator arm 303. Imaging tool 118 may be coupled to and manipulated by manipulator arm 304. Tools 100, 110, 118 may be used in performing various procedures on patient P positioned on table T. In some examples, manipulator assembly 302 may include more or less than the two arms shown. The manipulator assembly 302 may be robotic, non-robotic, or a hybrid robotic and non-robotic assembly, and may include selected degrees of freedom of movement (motorized and / or robotic), and selected degrees of freedom of movement (non-motorized and / or non-robotic). The medical system 300 may further include an input system 306, which typically includes one or more operator controls for controlling the manipulator assembly 302. The manipulator assembly 302 supports the tools 100, 110, 118 and may optionally include multiple actuators or motors that drive inputs to the medical tools in response to commands from a control system 312. The actuators may optionally include drive systems that, when coupled to the medical tools 100, 110, 118, may advance the medical tools through natural or surgically created anatomical orifices.

[0016] The medical system 300 also includes a display system 310 for displaying images or representations of the surgical site and medical tools generated by the imaging tool 118 and / or the sensor system 308. The display system 310 and the input system 306 may be oriented to allow the operator O to control the tools (e.g., medical tools, non-medical tools, imaging instruments, etc.) and manipulator assembly while experiencing telepresence. The display system 310 may include one or more display screens, including a display screen that provides visibility to a tableside operator tasked with interacting with the manipulator assembly and instruments during an instrument exchange. Additional information regarding the medical system 300 and medical tools is described in International Application Publication No. 2018 / 195216, filed April 18, 2018, and entitled "Graphical User Interface for Monitoring an Imaged-Guided Procedure," which is incorporated herein by reference in its entirety.

[0017] In some examples, the imaging tool 118 may include components of an imaging system including an endoscopic imaging instrument assembly that records concurrent or real-time images of the surgical site and provides the images to the operator or operators O via one or more displays of the medical system 300, such as one or more displays of the display system 310. The concurrent images may be, for example, two-dimensional or three-dimensional images captured by an imaging instrument positioned within the surgical site. The imaging system may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is executed by one or more computer processors, which may include a processor of the control system 312.

[0018] The sensor system 308 can include position / location sensors (e.g., actuator encoders or electromagnetic (EM) sensor systems) and / or shape sensors (e.g., fiber optic shape sensors) to determine the position, orientation, speed, velocity, attitude, and / or shape of the medical tool, the manipulator arms 303, 304, and / or components of the manipulator assembly 302. The sensor system 308 can also include pressure sensors, force sensors, contact sensors, or the like.

[0019] The control system 312 includes at least one computer processor or processing unit 314 for effecting control between the tools 100, 110, 118, the input system 306, the sensor system 308, and the display system 310, and at least one memory 316. The control system 312 also includes programmed instructions (e.g., a non-transitory machine-readable medium storing instructions) for implementing some or all of the methods described in accordance with aspects disclosed herein, including instructions for providing information to the display system 310.

[0020] 3 is a flowchart illustrating a method 400 for performing a guided tool change procedure, according to some examples. The methods disclosed herein may be depicted as a collection of operations or processes that may be performed in the same order as or a different order from the illustrated order. In some example methods, one or more of the illustrated processes may be omitted. Additionally, one or more processes not explicitly shown in the flowchart may be included before, after, between, or as part of an illustrated process. In some examples, one or more processes may be implemented, at least in part, by a control system executing code stored on a non-transitory, tangible, machine-readable medium, which, when executed by one or more processors (e.g., processors of the control system), may cause the one or more processors to perform one or more processes.

[0021] In process 402, an indication of an intended tool change may be received, for example, by a control system of a robotic-assisted medical system. This indication may be generated, for example, by an operator pressing a button on a master assembly, an operator interacting with a graphical user interface, or the control system identifying a condition corresponding to a tool change, such as a change in operating mode or the start of a new sequence in a medical procedure. In some examples, this indication may be an indication to change tool 100.

[0022] In process 404, kinematic information related to the position, orientation, and / or pose of the first tool may be determined and / or recorded. In some examples, the kinematic information may be used to determine the target point 122 based on the kinematic chain extending to the tip 106 of the tool 100. In some examples, process 404 may be performed before process 402. Process 404 may determine and / or record any information related to the state of the first tool and / or the manipulator assembly to which it is coupled that may be needed for subsequent calculations. Kinematic information may include the position, orientation, and / or pose of structures and joints in the kinematic chain including the manipulator assembly and the tool to be replaced. Kinematic information may be determined using sensors, such as potentiometers, encoders, or other types of position or velocity sensors, used to determine the kinematic configuration of the manipulator assembly and the tool coupled thereto. Kinematic information may also include the dimensions and shape of the links in the kinematic chain, including the components of the manipulator assembly and the tool. The kinematic information may be determined and / or recorded with reference to a frame of reference, such as the manipulator assembly, the patient, or the tip of an imaging device used to capture an image of the patient's anatomical field. In some examples, the kinematic information may be stored in a memory (e.g., memory 316) of the medical system.

[0023] In optional process 406, image data (e.g., first image data) of a tool within the field of view of an imaging instrument (e.g., an endoscopic imaging instrument) can be received and / or recorded from an imaging device. For example, image data of tool 100 within the field of view 119 of imaging tool 118 can be received and / or recorded. In some examples, the image data can be stereoscopic image data received from a stereoscopic endoscope. The stereoscopic image data can be used for vision-based tool tracking to triangulate the position of structures within the field of view, including tip 106, target point 122, clevis 103, shaft 102, and / or anatomical structure 105. The image data can enable a control system to determine the position of the structure relative to the tip of the endoscope, which can also be mapped to a common, fixed frame of reference between tool 100 and tool 118.

[0024] In process 408, guided tool exchange parameters for guiding the second tool may be determined. For example, guided tool exchange parameters may be determined for guiding tool 110 through port 109 to a work site (e.g., anatomical region 108) after removing tool 100. The guided tool exchange parameters may be determined from kinematic information, image data, or other inputs, as described in more detail in the method of FIGS. 4-6. The guided tool exchange parameters may include one or more of the target position and orientation of the second tool tip, the insertion path of the second tool between the port and the target position, the location of the remote center of the manipulator arm to which the second tool is coupled, the insertion depth limit of the second tool along the insertion path, or the configuration of the manipulator assembly that generates the insertion path.

[0025] In process 410, a guided tool exchange procedure can be performed using the determined guided tool exchange parameters to introduce a second tool (e.g., tool 110) into the anatomical region. For example, the manipulator assembly 302 and / or manipulator arm 303 can be repositioned to adjust the guide trajectory of the new tool 110. The new tool 110 can be recognized and engaged by the carriage of the manipulator assembly 302. The new tool 110 is typically different from the first tool 100, but in some cases may be the same tool. In some examples, data can be acquired from the second tool 110 via a readable memory chip. Alternatively, kinematic information, including dimensions and other data, of the second tool 110 can be input via an operator interface. A sensor system 308 can be used to detect engagement between the second tool 110 and the manipulator arm 303 to ensure proper engagement before performing the introduction process. The manipulator assembly 302 (or a portion thereof) can be repositioned prior to engagement with the second tool 110. This adjustment may adjust the IO axis and / or remote center. The tool 110 may be introduced along the insertion path 120 through the entry port 109 to the surgical site 108. The control system 312 provides the manipulator assembly 302 with floating degrees of freedom of movement along the insertion path 120 to allow the second tool 110 to be moved to the work site (e.g., the anatomical region 108), either by a surgeon's assistant or by the controller itself by moving the tool along the insertion path 120. The new tool 110 may be introduced into the anatomical region 108 until the tip reaches the target space 125 or target point 122. Introduction of the second tool 110 may be performed by direct manual manipulation by the operator, using a joystick or other interface, or remotely by the operator via the input system 306, but may alternatively be performed at least partially or completely by the control system 312.To guide the operator in inserting the second tool 110, illustrations of the insertion path, the target space 120, the target point 122, and / or images of the field of view 119 may be displayed on the display system 310. When the tip of the tool 110 reaches the target space 125 or the target point 122, the operator control of the input system 306 may encounter tactile resistance generated by the control system 312, causing the movement of the tool 110 to stop.

[0026] Typical tool introduction into a patient's anatomy may require one or more degrees of freedom of movement. Some degrees of freedom during tool introduction may be constrained. In some guided tool exchange procedures, a single degree of freedom is available, such as the degree of freedom corresponding to movement along the insertion axis, while all other degrees of freedom are constrained, particularly the two other proximal degrees of freedom used to position the tool mount before reinsertion and the distal degrees of freedom of movement associated with the instrument's clevis and end effector. These degrees of freedom may be released for movement by the operator after the operational connection between the master control, manipulator assembly, and tool is reestablished, preferably after the tool exchange is complete. The unconstrained degree of freedom may constitute a linear motion axis that precisely coincides with one joint of the tool carriage on the manipulator arm. Generally, to describe a guided tool exchange, several degrees of freedom of movement that are free to move are identified, and the remaining degrees of freedom of movement are absolutely or substantially constrained, via servo control or the like, to prevent or resist movement in that direction, respectively. The entry of the tip of the second tool into the target space 125, as sensed by the sensor system 308, can indicate the end of the guided tool change. Control is then returned to the operator, either automatically or by notification that the operator can take control by performing some action, such as actuating the end effector or pressing an input button. It should be understood that there can be other criteria that indicate the end of the guided tool change. These criteria include the position of another point of the tool in another target space, the position of a joint in a pre-defined or pre-calculated region, other sensor input (e.g., visual or image-based detection of the tool), or explicit user input.

[0027] FIG. 4A illustrates a method 500 for generating one or more guided tool exchange parameters for guiding a new tool. Method 500 can be used as part of process 408 for determining guided tool exchange parameters for guiding a second tool. Process 502 can generate guided tool exchange parameter inputs based on kinematic information associated with the configuration of a first tool, including its position, pose, and / or orientation. The guided tool exchange parameter inputs can include any kinematic information recorded in process 404, such as the position, pose, and / or orientation of structures and joints in the kinematic chain including the manipulator assembly and the tool being replaced. The guided tool exchange parameter inputs can also be derived from the kinematic information. Such inputs can include a target point 122 at the tip 106 of the tool 100, a target space 125, an insertion depth limit 126, and / or kinematically determined distances between the moving tool, the imaging tool, and the anatomical structure.

[0028] In process 504, guided tool exchange parameters for guiding the second tool may be generated from the guided tool exchange parameter input. The guided tool exchange parameters may include, for example, a target position and orientation of the second tool tip, an insertion path of the second tool between the port and the target position, a location of the remote center of the second tool, an insertion depth limit of the second tool along the insertion path, and a configuration of the manipulator assembly for generating the insertion path. For example, an insertion path 120 of the new tool 110 may be determined from the guided tool exchange parameter input. This determination may include calculating a position of a tool carriage on the manipulator arm 303 such that, upon engagement with the new tool 110, the new tool 110 begins to be introduced into the anatomical region 108 along the specified path 120 such that the new tool path intersects with the target point 122 or target space 125 previously occupied by the end effector of the first tool. The determined insertion path may be used to manipulate the manipulator assembly 302 to reposition the tool carriage on the arm 303. The parameters may also include, for example, coordinates of specific points and joint angles of specific joints between the linkages included in the manipulator assembly 302 and the second tool 110. In a specific example, the parameters used to reposition the tool mount include an external pitch angle and an external yaw angle relative to a reference frame for guiding the tool 110 along the insertion path 120. The external pitch and external yaw of the second tool 110 may generally be different from the external pitch and external yaw of the first tool 100 in its operating position prior to removal.

[0029] In some examples, the kinematically determined guided tool change parameters are initial parameters, such as the initial target position 122 or the initial insertion path 120, that may be modified, adjusted, compensated, or otherwise altered in light of additional parameter input obtained from an imaging system, as described in further examples below. In some examples, the guided tool change procedure may be disabled if the generated guided tool change parameters violate other conditions of the guided tool change procedure. For example, the guided tool change may be disabled if the guided tool change parameters include an insertion path that exceeds the range of motion of the manipulator assembly (e.g., if the movement exceeds a 30-degree cone angle) or if the manipulator arm encounters an obstacle while moving to align with the insertion path.

[0030] FIG. 4B illustrates an example of a secondary target point selected to avoid invalidating a guided tool change. The initially selected target point 122A and insertion path 120A determined for the new tool, which may correspond to target point 122 and insertion path 120 of FIG. 2B, may be unattainable due to limited range of motion of the manipulator arm. To avoid invalidating the guided tool change procedure, target point 122A may be moved to target point 122B at a location within the first tool volume 150 (e.g., a location along a clevis or shaft of first tool 100) along a new insertion path 120B that does not exceed the range of motion of the manipulator assembly. In some examples, target point 122B may be identified as the most distal position along the initial tool assembly that is still within the field of view of the endoscope and satisfies the kinematic constraints of the subsequent insertion trajectory.

[0031] In optional process 506, visual guidance may be displayed to assist an operator, such as a tableside operator, in performing a guided tool exchange. For example, visual guidance including a graphical representation of target point 122, target space 125, insertion path 120, arrows indicating direction of movement, and / or field of view 119 may be displayed on display system 310 to assist with guidance of the new tool 110. Examples of visual guidance may be provided via a graphical user interface such as that shown in FIGS. 7A-7D.

[0032] In optional process 508, haptic guidance can be provided to assist an operator, such as a tableside operator, in performing a guided tool exchange. For example, haptic guidance in the form of a resistive force, vibration, or other haptic sensation can be provided to the manipulator arm 303 to snap the arm into an insertion path when the arm's insertion axis aligns with the target location. For example, during tool insertion by the tableside operator, the manipulator assembly (e.g., a prismatic or revolute joint on the manipulator assembly) can provide haptic feedback and / or force feedback when the end effector of the second tool approaches or reaches the target point and / or depth limit. The operator can override the haptic force if desired.

[0033] In some examples, the guided tool change procedure may be invalidated if the operator releases the manipulator arm from a gripping or constrained position during an instrument exchange. Visual and / or tactile guidance may be provided to prevent invalidation of the guided tool change procedure and allow the operator to resume use of the guided tool change procedure. For example, when the control system detects a clutch or movement, tactile or visual guidance may be provided to the operator to reposition the manipulator arm. The visual and tactile guidance may allow the operator to optionally realign the arm with the guided tool change trajectory before resuming the instrument exchange procedure and before advancing the instrument into the body. The guidance may include a displayed position of the target location and insertion path. The guidance may also be updated to include a virtual extension of the insertion path based on the configuration of the moved manipulator. The operator can compare the target location, insertion path, and virtual extension. Tactile guidance may be provided as the user moves the manipulator assembly to align the insertion axis of the manipulator arm with the insertion path. To provide visual guidance as the manipulator assembly moves, accurately calibrated image and manipulator reference frames may be required.

[0034] FIG. 5 illustrates a method 600 for generating one or more guided tool exchange parameters for guiding a new tool. Method 600 can be used as part of process 408 for determining guided tool exchange parameters for guiding a second tool. Process 602 can generate guided tool exchange parameter inputs based on first image data recorded in process 406. In some examples, the first image data can be stereoscopic image data of the field of view 119 collected while the tool 100 is within the anatomical region 108. This image data can be used to supplement or correct the kinematic information recorded in process 404. For some instruments or instrument configurations, the kinematic information may not accurately represent the true position and orientation of the distal tip of the instrument end effector. For example, an end effector with long jaws may deflect due to interaction with tissue or other structures within the anatomical region, causing the tip of the end effector to deviate by several millimeters from its expected kinematic position. Additionally or alternatively, compliance or backlash issues in the drivetrain of the control cables articulating the end effector may cause the tip of the end effector to deviate from its expected, commanded kinematic position. Additionally or alternatively, kinematic uncertainties may be associated with the joints and links of the manipulator assembly. The position and orientation of the distal tip of the end effector, ascertained from the image data, can be used to correct the kinematic information, providing a more accurate determination of the target point 122, the target space 125, and the insertion path 120. The reference frame of the image data can be registered to a common reference frame with the manipulator assembly and tool 100, allowing for position and orientation corrections within the common reference frame. Image data from the field of view 119 can provide guided tool exchange parameter input, for example, in the form of tool and tissue position and orientation information that can be used to correct or modify kinematically derived parameter inputs.

[0035] In process 604, guided tool exchange parameters for guiding a second tool can be generated from the guided tool exchange parameter input. The guided tool exchange parameters can include, for example, a target position and orientation of the second tool tip, an insertion path of the second tool between the port and the target position, an insertion depth limit of the second tool along the insertion path, and a configuration of the manipulator assembly for generating the insertion path. For example, the guided tool exchange parameter input from the first image data can be combined with or used to modify the guided tool exchange parameter input from the kinematic information (process 502) to generate more accurate guided tool exchange parameters. For example, a more accurate target point 122 of the first tool can be determined to more accurately determine the target position and orientation of the second tool tip. More specifically, an image-based position of the instrument tip 106 in the image reference frame can be determined. The position of the tip 106 can be transformed from the image reference frame to the manipulator assembly reference frame. By comparing the transformed image-based position of the tip 106 with the kinematically determined target point 122 (corresponding to the kinematically determined position of the tip 106) in the manipulator assembly reference frame, inaccuracies in the kinematically determined position can be identified. A corrected target point 122 can be determined based on the identified inaccuracies, generating a corrected target position and orientation for the second tool tip. More specifically, in some examples, the uncertainty in the tip position associated with the kinematically-based solution can be compared with the uncertainty associated with the image-based tip position. The system can conditionally use the estimate with the lowest tolerable uncertainty (e.g., a threshold or a predetermined minimum uncertainty). If neither estimate has sufficiently low uncertainty, then the guided tool change process can be canceled. Other parameters, such as the insertion path and the configuration of the manipulator assembly, can be adjusted based on the corrected target point.

[0036] In optional process 606, visual guidance may be displayed to assist an operator, such as a tableside operator, in performing a guided tool exchange. For example, visual guidance including a graphical display of image adjustment parameters, including adjusted target point 122, adjusted target space 125, and / or adjusted insertion path 120, may be displayed on display system 310 along with an image of field of view 119 to assist in the guidance of a new tool 110. In some examples, indicators indicating hidden target points may be displayed on display system 310. For example, if the location of a determined target point is not visible in the image data (e.g., because tissue or a tool is obscuring the view or is outside the field of view), a graphic or text indicator may be displayed during a tool exchange procedure to indicate the location of the hidden or off-screen target point.

[0037] In optional process 608, tactile guidance may be provided to assist an operator, such as a tableside operator, in performing a guided tool change. For example, tactile guidance in the form of a resistive force, vibration, or other tactile sensation may be provided, as described in process 508.

[0038] FIG. 6 illustrates a method 700 for generating one or more guided tool exchange parameters for guiding a new tool. Method 700 may be used as part of process 408 for determining guided tool exchange parameters for guiding a second tool. In process 702, image data (e.g., second image data) of the field of view of the endoscopic instrument may be recorded after removing a first tool. For example, image data of the anatomical region 108, anatomical tissue 105, and other tools or structures within the field of view 119 of the imaging tool 118 may be recorded after removing tool 100 and before inserting another tool through port 109. The second image data may capture displacement or movement of tissue 105 or other tools and structures within the field of view 119 after removing tool 100 and provide information regarding obstructions in the insertion path that may impede the introduction of a subsequent tool. The second image data may be used to supplement or correct the kinematic information recorded in process 404 and / or the first image data recorded in process 406. Process 702 may be optional in method 700 .

[0039] In process 704, an image depth map may be generated from the second image data. In some examples, the image data is stereo image data received from a stereo endoscope and may be used to generate a depth map that provides information about the distance from the tip of the imaging tool 118 to the surface of an object within the field of view 119. The depth map may represent the near and far distance between the object within the field of view 119 and the plane of the imaging tool 118. In some examples, the mapping of points and / or vectors between the depth map image space and the imaging tool tip coordinates may depend on a calibrated camera model (i.e., intrinsic and extrinsic parameters).

[0040] The present invention also relies on the ability to map the remote center position and insertion axis of the instrument manipulators to endoscope tip coordinates using a kinematic linkage and common frame of reference between the manipulators.

[0041] In process 706, guided tool change parameter inputs may be generated based on the image depth map. The image depth map may be used to supplement or modify the kinematic information recorded in process 404 and / or the first image data recorded in process 406. After removing the first tool 100, the anatomical region 108 may change. For example, tissue in contact with the first tool 100 may move into the space vacated by the tool. Additionally or alternatively, deformable tissue may move or slide relative to other tissue, and some tissue may move in response to breathing, cardiac motion, or blood flow. Guided tool change parameter inputs determined from the depth map may include, for example, the distance between the kinematically identified target point 122 and the surface of the anatomical structure 105, or the distance between the surface of the anatomical structure 105 and the anatomical wall 107 adjacent to the imaging tool or adjacent to the port 109. In some examples, the guided tool change parameter inputs may include the location of the intersection of the kinematically derived insertion path 120 and the anatomical structure 105. The proximity of the target point 122 and / or insertion path 120 to the depth map corresponding to the anatomical tissue 105 can determine whether the target location and insertion path remain valid, become invalid, or are adjusted.

[0042] In process 708, if the target location remains valid and reachable by the second tool, the depth map can be used to generate an indicator of the hidden target location. For example, if the depth map may indicate that the kinematically determined target point 122 is hidden by the anatomy 105, a graphical marker, text, or other indicator can be displayed that indicates the location of the target location.

[0043] In process 710, guided tool change parameters for guiding the second tool can be generated. The guided tool change parameters can include, for example, a target position and orientation of the second tool tip, an insertion path of the second tool between the port and the target position, an insertion depth limit of the second tool along the insertion path, and a configuration of the manipulator assembly for generating the insertion path. For example, guided tool change parameter inputs based on the depth map can be combined with or used to modify guided tool change parameter inputs from the kinematic information (process 502) and / or the image information (process 602) to generate more accurate guided tool change parameters.

[0044] In some examples, an insertion depth limit, adjusted or limited by the intersection of the insertion path with the depth map, may be continuously calculated during insertion of the second tool to account for anatomical movement, such as respiratory or pulsatile movement. The insertion depth limit may be determined as the minimum depth along the insertion path observed over a period of time just before the instrument is finally advanced to the target location (e.g., within a threshold distance or within a predicted arrival time at the target location). The insertion depth map limit may be continuously adjusted until the tip of the new instrument is advanced to a predetermined position, such as a position where the tip of the instrument obstructs the endoscopic view of the target.

[0045] In some examples, the tool 110 may first be modeled as a cylinder extending along the insertion path 120 and tested for intersection with a depth map corresponding to the anatomy 105 within the anatomical region 108. If the modeled cylinder does not intersect with the depth map corresponding to the anatomy 105, the target point 122 may remain valid. The remote center 124 may also be maintained at a kinematically determined position. In some examples, the diameter of the modeled cylinder may be larger than the diameter of the tool shaft to account for uncertainty in the remote center of the manipulator relative to the frame of reference of the endoscope tip.

[0046] In some examples, if the kinematically determined target point 122 becomes invalid due to, for example, inaccuracies in the kinematic position of the tooltip, limitations in the manipulator's range of motion, or manual adjustments to the manipulator assembly, a revised target position may be determined using the depth map. For example, a ray casting procedure may cast a ray segment along the kinematically determined insertion path 120 and determine whether the cast ray intersects the depth map before reaching the kinematically determined target point 122. If the ray intersects the depth map, the target position may be considered occluded, and the target position may be adjusted to a revised target position at or near the intersection of the cast ray and the depth map.

[0047] In some examples, the target position can be determined or modified from the kinematically determined target position by determining a trajectory between the remote center of the manipulator arm and a center point in the field of view captured by the second image data. One or more test rays can be cast along the trajectory, and the insertion depth and target position can be adjusted based on the location of their interference with the depth map. Once an optimized target position is determined by the test rays, inverse arm kinematics can be used to determine a pose of the manipulator arm with an insertion axis that intersects with the optimized target position and determine the corresponding insertion depth to reach the optimized target position. In some examples, the ray cast should be emitted from an unobstructed point in space, such as the tip of the port 109, the distal tip of a cannula extending into the port, or the distal tip of an instrument. Such an emission point can avoid premature intersections with depth planes introduced by the components themselves (e.g., port, cannula, instrument). Furthermore, the ray cast calculation can be constrained to only consider one-sided depth map plane transitions. For example, if a ray originates from inside the depth map, the first transition from inside to outside may be ignored.

[0048] In some examples, the guided tool change procedure may be disabled if the manipulator arm is grasped or moved by the operator. To avoid disabling the guided tool change procedure, the depth map may be referenced to determine whether the kinematically determined target point 122 or insertion path 120 is obstructed by other structures within the anatomical tissue 105 or anatomical region 108. If no obstruction is identified based on the depth map, the tool change procedure may proceed without being disabled. In some examples, the guided tool change procedure may proceed only if the adjustment in the manipulator pose is sufficiently small (e.g., less than a threshold pose change from the initial pose) and the insertion path is within an acceptable range of the original insertion path.

[0049] In some examples, the depth map is used to configure the end effector of the second tool after inserting the second tool into the anatomical region. For example, evaluating the depth map can determine the location of surrounding tissue and determine the jaw opening and / or wrist or clevis orientation of the second tool to avoid impacting the surrounding tissue.

[0050] In optional process 712, visual guidance may be displayed to assist the operator in performing the guided tool change. For example, visual guidance including a graphical display of image adjustment parameters including new or modified target points, modified target space, and / or modified insertion path 120 may be displayed on display system 310 along with the image of field of view 119 to assist in the guidance of the new tool 110.

[0051] 7A-7D illustrate a graphical user interface 800 displaying visual guidance including image adjustment parameters. The graphical user interface 800 can be displayed, for example, on a display system (e.g., display system 310) of a medical system. FIG. 7A illustrates the graphical user interface 800 including an image of an imaging tool's field of view 802 (e.g., field of view 119) including an anatomical structure 804 and a first tool 806. The distal portion of the first tool 806 may be located at a position 808. As shown in FIG. 7B, after removing the first tool 806, a marker 810 is displayed at a target position that may be determined based on kinematic information associated with the first tool 806, as described herein. For example, the target position may correspond to or be determined based on the position 808 of the removed first tool 806. In some examples, the target position may be determined based on the distal jaw tip position of the first tool immediately prior to removal. A modeled or synthesized cylinder 812 may provide guidance in the form of a graphical insertion path for the second tool to advance toward the marker 810. In some examples, as shown in FIG. 7C , the target location may be adjusted based on structures within the field of view. For example, tissue 804 may move (including move, expand, bulge, or otherwise displace) to occlude the marked location 808. As described in method 700, depth map information may be used to identify the occluding tissue, generate guided tool exchange parameters including a modified target location, and display a modified marker 814 at the modified target location. The modified target location may be located near the tissue 804 without being obstructed by the tissue 804. A second tool 816 may be guided along the modeled cylinder 802 until its tip reaches the modified target location. By delivering the second tool 816 to the modified target location, direct or penetrating contact with the tissue 804 may be avoided. In some examples, as shown in FIG. 7D, the modification or change in target position may be displayed in a graphical user interface to make the operator more aware of the deviation in target position.For example, a marker 818 corresponding to the original target location and a marker 814 corresponding to the modified target location may be simultaneously displayed within the graphical user interface 800. A characteristic of the marker 818, such as color, texture, opacity, or shape, may indicate that the original location is occluded by tissue and thus cannot be reached by the second tool 816. Additionally or alternatively, the volume within the cylinder 812 bounded by the markers 818 and 814 may be visually distinct (e.g., by color, shading, transparency, texture, etc.) from the remainder of the cylinder 812 to provide additional visual cues regarding the change in target location. In some examples, the orientation of the modeled cylinder 802 may remain constant, and the modified target location 814 may be located at a location proximal to the target location 808 along the longitudinal axis of the modeled cylinder 802 (e.g., the location indicated by markers 810, 818). In other examples, the orientation of the modeled cylinder may change in response to the moved tissue, and the modified target position may lie along the modified longitudinal axis of the modeled cylinder. In some examples, the manipulator assembly to which the second tool is attached (e.g., manipulator assembly 302) may enforce the modified target position 814 and restrict movement of the second tool beyond the modified target position.

[0052] In optional process 714, tactile guidance may be provided to assist the operator in performing the guided tool change. For example, tactile guidance in the form of a resistive force, vibration, or other tactile sensation may be provided to the operator controls of the master assembly 306.

[0053] The singular forms "a," "an," and "the" include the plural forms unless the context indicates otherwise. Furthermore, the terms "comprises," "comprises," "includes," "has," and the like specify the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be directly coupled electrically or mechanically, or indirectly coupled through one or more intermediate components. The auxiliary verb "may" similarly means that a feature, step, operation, element, or component is optional.

[0054] Specific details are provided herein to explain some embodiments. Numerous specific details are provided to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not limiting. Those skilled in the art will recognize other elements not specifically described herein that are within the scope and spirit of the present disclosure.

[0055] Elements described in detail with reference to one embodiment, implementation, or application may, where practical, be included in other embodiments, implementations, or applications for which they are not specifically shown or described. For example, even if an element is described in detail with reference to one embodiment but not with reference to a second embodiment, the element may still be claimed as being included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in connection with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or applications unless specifically stated otherwise, or unless one or more elements do not enable the embodiment or implementation, or unless two or more elements provide conflicting functions. Not all illustrated processes are performed in all embodiments of the disclosed methods. Furthermore, one or more processes not explicitly illustrated may be included before, after, between, or as part of the illustrated processes. In some embodiments, one or more processes are executed by a control system or are implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors, causes the one or more processors to perform one or more processes.

[0056] Any changes and further modifications to the described devices, apparatus, methods, and further applications of the principles of the present disclosure are fully contemplated, as would normally occur to one skilled in the art to which the present disclosure pertains. Furthermore, the dimensions provided herein are for specific examples, and it is contemplated that different sizes, dimensions, and / or proportions may be utilized to realize the concepts of the present disclosure. To avoid unnecessary repetition of description, one or more components or operations described according to one illustrative embodiment may be used or omitted, as appropriate, from other illustrative embodiments. For brevity, multiple iterations of these combinations will not be described individually. For brevity, the same reference numbers may be used throughout the drawings to refer to the same or similar parts in some instances.

[0057] The systems and methods described herein may be suitable for procedures involving any of a variety of anatomical systems, such as the lungs, colon, intestines, stomach, liver, kidneys and calyces, brain, heart, and / or the circulatory system, including the vasculature. While some embodiments related to medical procedures are provided herein, references to medical or surgical instruments and methods are not intended to be limiting. For example, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial applications, general-purpose robotic applications, and sensing or manipulating non-tissue workpieces. Other example applications include cosmetic enhancement, imaging of the human or animal anatomical structure, data collection from the human or animal anatomical structure, and training of medical or non-medical personnel. Additional example applications include procedures on tissue removed from the human or animal anatomical structure (without replacing it back into the human or animal anatomical structure), and use in procedures on human or animal cadavers. Furthermore, these techniques can also be used in surgical and non-surgical medical or diagnostic procedures.

[0058] One or more elements of the embodiments of the present disclosure may be implemented in software for execution on a processor of a computer system, such as a control processing system. When implemented in software, the elements of the embodiments of the present disclosure may be code segments for performing various tasks. Programs or code segments may be stored in a processor-readable storage medium or device downloaded by a computer data signal embodied in a carrier wave via a transmission medium or a communication link. Processor-readable storage devices may include any medium capable of storing information, including optical, semiconductor, and / or magnetic media. Examples of processor-readable storage devices include electronic circuits, semiconductor devices, semiconductor memory devices, read-only memories (ROMs), flash memories, erasable programmable read-only memories (EPROMs), floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. Code segments may be downloaded over a computer network, such as the Internet or an intranet. Any of a variety of centralized or distributed data processing architectures may be employed. Programmed instructions may be implemented as multiple individual programs or subroutines, or may be integrated into multiple other aspects of the systems described herein. In some examples, the control system may support wireless communication protocols such as Bluetooth, Infrared Data Association (IrDA), HomeRF, IEEE 802.11, Digital Enhanced Cordless Telecommunications (DECT), Ultra Wideband (UWB), ZigBee, wireless telemetry, and the like.

[0059] It should be noted that the presented processes and displays may not be inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the described operations. The required structure for a variety of systems is set forth as elements of the claims. Additionally, embodiments of the present invention are not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​can be used to implement the teachings of the present invention as described herein.

[0060] This disclosure describes various instruments, portions of instruments, and anatomical structures in terms of their state in three-dimensional space. As used herein, the term "position" refers to the position of an object or portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational configuration of an object or portion of an object (e.g., one or more rotational degrees of freedom, such as roll, pitch, and / or yaw). As used herein, the term "pose" refers to the position of an object or portion of an object in at least one translational degree of freedom and the orientation of that object or portion of an object in at least one rotational degree of freedom (e.g., up to a total of six degrees of freedom). As used herein, the term "shape" refers to the collection of poses, positions, or orientations measured along an object.

[0061] While specific exemplary embodiments of the present invention have been shown and described in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of the broad invention and not limiting, and since various other modifications may occur to those skilled in the art, embodiments of the invention are not limited to the specific constructions and arrangements shown and described.

Claims

1. 1. A medical system, comprising: a manipulator assembly; a control system; The control system includes a processing unit including one or more processors, the processing unit comprising: determining kinematic information associated with a first tool inserted into a work site, said first tool being coupled to said manipulator assembly; receiving image data generated by an endoscopic imaging instrument having a field of view, the image data being generated by the endoscopic imaging instrument after removing the first tool from the manipulator assembly; determining one or more guided tool exchange parameters for guiding a second tool to the work site, the second tool being received in connection with the manipulator assembly after the first tool has been removed, the one or more guided tool exchange parameters being based on the kinematic information associated with the first tool coupled to the manipulator assembly and based on a depth map determined from the image data. Healthcare system.

2. The medical system of claim 1 , wherein the one or more guided tool exchange parameters include a target position for a tool tip of the second tool.

3. The medical system of claim 2 , wherein the one or more guided tool exchange parameters include an insertion path of the second tool through an entry port to the target location.

4. The medical system of claim 3 , wherein the one or more guided tool exchange parameters include an insertion depth limit for the second tool along the insertion path.

5. The medical system of claim 3 , wherein the one or more guided tool exchange parameters include a configuration of the manipulator assembly that directs the second tool along the insertion path.

6. The medical system of claim 1 , wherein the one or more guided tool exchange parameters include a position of a remote center of motion of a manipulator arm of the manipulator assembly.

7. 2. The medical system of claim 1, wherein determining one or more guided tool exchange parameters comprises determining an initial target position for a tool tip of the second tool based on the kinematic information associated with the first tool, and modifying the initial target position based on the depth map.

8. The medical system of claim 7 , wherein modifying the initial target location includes determining a distance between the initial target location and an anatomical surface based on the depth map.

9. 8. The medical system of claim 7, wherein modifying the initial target location includes determining whether an initial insertion path from the manipulator assembly to the initial target location is obstructed based on the depth map.

10. 10. The medical system of claim 9, wherein determining whether the initial insertion path is occluded comprises modeling the initial insertion path as a cylinder and testing whether the cylinder intersects with the depth map.

11. The medical system of claim 9 , wherein the modified target location is at or near an intersection of the initial insertion path and the depth map.

12. Modifying the initial target position comprises: determining a trajectory between a center of rotation of the manipulator assembly and a center point of a current endoscopic field of view; determining a revised target position at an insertion depth where the trajectory intersects the depth map; and determining a posture of the manipulator assembly based on the insertion depth, the manipulator assembly having an insertion axis that intersects with the modified target position.

13. 8. The medical system of claim 7, further comprising recording image data of the first tool within the field of view, wherein the initial target position of the tool tip of the second tool is further based on the image data of the first tool.

14. The medical system of claim 7 , further comprising determining a configuration of an end effector of the second tool based on the depth map.

15. The medical system of claim 1 , wherein the kinematic information includes a compensated remote center or in-out axis of the manipulator assembly.

16. The medical system of claim 1 , further comprising a display system configured to display a field of view of the endoscopic imaging instrument.

17. The medical system of claim 16 , wherein the processing unit is further configured to display visual guidance on the display system for performing a guided tool exchange operation.

18. The medical system of claim 17 , wherein the visual guidance includes a graphical indicator that indicates a target position for the second tool.

19. 20. The medical system of claim 18, wherein the target location is obscured within the displayed field of view.

20. The medical system of claim 18 , wherein the target location is adjusted based on structures within the field of view.

21. The medical system of claim 17 , wherein the visual guidance comprises a graphical depiction of an insertion path displayed on an image of the field of view.

22. 18. The medical system of claim 17, wherein displaying the visual guidance includes simultaneously displaying a first marker corresponding to the initial target position and a second marker corresponding to the modified target position.

23. The medical system of claim 1 , wherein the processing unit is further configured to provide tactile guidance to an operator control device, and the manipulator assembly is responsive to movements of the operator control device.

24. 1. A method, comprising: determining kinematic information associated with a first tool inserted into a work site, the first tool being coupled to a manipulator assembly; receiving image data generated by an endoscopic imaging instrument having a field of view, the image data being generated by the endoscopic imaging instrument after removing the first tool from the manipulator assembly; determining one or more guided tool exchange parameters for guiding a second tool into the work site, the second tool being received connected to the manipulator assembly after the first tool has been removed, the one or more guided tool exchange parameters being based on the kinematic information associated with the first tool coupled to the manipulator assembly and based on a depth map determined from the image data; method.

25. The method of claim 24 , wherein the one or more guided tool change parameters include a target position for a tool tip of the second tool.

26. 26. The method of claim 25, wherein the one or more guided tool exchange parameters include an insertion path of the second tool through an entry port to the target location.

27. 27. The method of claim 26, wherein the one or more guided tool exchange parameters include an insertion depth limit for the second tool along the insertion path.

28. 27. The method of claim 26, wherein the one or more guided tool exchange parameters include a configuration of the manipulator assembly that directs the second tool along the insertion path.

29. 25. The method of claim 24, wherein the one or more guided tool exchange parameters include a position of a remote center of motion of a manipulator arm of the manipulator assembly.

30. 25. The method of claim 24, wherein determining one or more guided tool exchange parameters comprises determining an initial target position for a tool tip of the second tool based on kinematic information associated with the first tool, and modifying the initial target position based on the depth map.

31. 31. The method of claim 30, wherein modifying the initial target location comprises determining a distance between the initial target location and an anatomical surface based on the depth map.

32. 31. The method of claim 30, wherein revising the initial target location includes determining, based on the depth map, whether an initial insertion path from the manipulator assembly to the initial target location is obstructed.

33. 33. The method of claim 32, wherein determining whether the initial insertion path is occluded comprises modeling the initial insertion path as a cylinder and testing whether the cylinder intersects the depth map.

34. 33. The method of claim 32, wherein a revised target location is at or near an intersection of the initial insertion path and the depth map.

35. The step of modifying the initial target position includes: determining a trajectory between a center of rotation of the manipulator assembly and a center point within a current endoscopic field of view; determining a revised target position at an insertion depth where the trajectory intersects the depth map; and determining a pose of the manipulator assembly based on the insertion depth, the manipulator assembly having an insertion axis that intersects the modified target position.

36. 31. The method of claim 30, further comprising recording image data of the first tool within the field of view, wherein the initial target position of the tool tip of the second tool is further based on the image data of the first tool.

37. The method of claim 30, further comprising determining a configuration of an end effector of the second tool based on the depth map.

38. 25. The method of claim 24, further comprising displaying, on a display system, a field of view of the endoscopic imaging instrument and visual guidance for performing a guided tool exchange operation.

39. 39. The method of claim 38, wherein the visual guidance includes a graphical indicator that indicates a target position for the second tool.

40. 40. The method of claim 39, wherein the target location is obscured within the displayed field of view.

41. 40. The method of claim 39, wherein the target location is adjusted based on structures within the field of view.

42. 39. The method of claim 38, wherein the visual guidance comprises a graphical depiction of an insertion path displayed on an image of the field of view.

43. 39. The method of claim 38, wherein displaying the visual guidance comprises simultaneously displaying a first marker corresponding to the initial target position and a second marker corresponding to the modified target position.

44. 25. The method of claim 24, further comprising providing tactile guidance to an operator control device, wherein the manipulator assembly is responsive to movement of the operator control device.

45. A non-transitory machine-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: determining kinematic information associated with a first tool inserted into a work site, the first tool being coupled to a manipulator assembly; receiving image data generated by an endoscopic imaging instrument having a field of view, the image data being generated by the endoscopic imaging instrument after removing the first tool from the manipulator assembly; determining one or more guided tool exchange parameters for guiding a second tool into the work site, the second tool being received in connection with the manipulator assembly after removal of the first tool, the one or more guided tool exchange parameters being based on the kinematic information associated with the first tool coupled to the manipulator assembly and based on a depth map determined from the image data. Non-transitory machine-readable media.

46. 46. ​​The non-transitory machine-readable medium of claim 45, wherein the one or more guided tool exchange parameters include a target position for a tool tip of the second tool.

47. 47. The non-transitory machine-readable medium of claim 46, wherein the one or more guided tool exchange parameters include an insertion path of the second tool through an entry port to the target location.

48. 48. The non-transitory machine-readable medium of claim 47, wherein the one or more guided tool exchange parameters include an insertion depth limit for the second tool along the insertion path.

49. 48. The non-transitory machine-readable medium of claim 47, wherein the one or more guided tool exchange parameters include a configuration of the manipulator assembly that directs the second tool along the insertion path.

50. 46. ​​The non-transitory machine-readable medium of claim 45, wherein the one or more guided tool exchange parameters include a position of a remote center of motion of a manipulator arm of the manipulator assembly.

51. 46. ​​The non-transitory machine-readable medium of claim 45, wherein determining one or more guided tool exchange parameters comprises determining an initial target position for a tool tip of the second tool based on the kinematic information associated with the first tool, and modifying the initial target position based on the depth map.

52. 52. The non-transitory machine-readable medium of claim 51, wherein refining the initial target location comprises determining a distance between the initial target location and an anatomical surface based on the depth map.

53. 52. The non-transitory machine-readable medium of claim 51 , wherein revising the initial target location comprises determining, based on the depth map, whether an initial insertion path from the manipulator assembly to the initial target location is obstructed.

54. 54. The non-transitory machine-readable medium of claim 53, wherein determining whether the initial insertion path is occluded comprises modeling the initial insertion path as a cylinder and testing whether the cylinder intersects with the depth map.

55. 54. The non-transitory machine-readable medium of claim 53, wherein a revised target location is at or near an intersection between the initial insertion path and the depth map.

56. Modifying the initial target position comprises: determining a trajectory between a center of rotation of the manipulator assembly and a center point within a current endoscopic field of view; determining a revised target position at an insertion depth where the trajectory intersects the depth map; and determining a pose of the manipulator assembly based on the insertion depth, the manipulator assembly having an insertion axis that intersects the modified target position.

57. storing instructions that, when executed by one or more processors, cause the one or more processors to:

52. The non-transitory machine-readable medium of claim 51 , further causing the system to: record image data of the first tool within the field of view, wherein the initial target position of the tool tip of the second tool is further based on the image data of the first tool.

58. storing instructions that, when executed by one or more processors, cause the one or more processors to:

52. The non-transitory machine-readable medium of claim 51, further comprising determining a configuration of an end effector of the second tool based on the depth map.

59. storing instructions that, when executed by one or more processors, cause the one or more processors to:

46. ​​The non-transitory machine-readable medium of claim 45, further causing displaying, on a display system, a field of view of the endoscopic imaging instrument and visual guidance for performing a guided tool exchange operation.

60. 60. The non-transitory machine-readable medium of claim 59, wherein the visual guidance comprises a graphical indicator that indicates a target position for the second tool.

61. 61. The non-transitory machine-readable medium of claim 60, wherein the target location is obscured within the displayed field of view.

62. 61. The non-transitory machine-readable medium of claim 60, wherein the target location is adjusted based on structures within the field of view.

63. 60. The non-transitory machine-readable medium of claim 59, wherein the visual guidance comprises a graphical depiction of an insertion path displayed on an image of the field of view.

64. 60. The non-transitory machine-readable medium of claim 59, wherein displaying visual guidance includes simultaneously displaying a first marker corresponding to the initial target position and a second marker corresponding to the modified target position.

65. 46. ​​The non-transitory machine-readable medium of claim 45, storing instructions that, when executed by one or more processors, further cause the one or more processors to provide tactile guidance to an operator control device, wherein the manipulator assembly is responsive to movement of the operator control device.

Images