Systems and methods for guided tools to alter resilience

Abstract

A medical system may include a manipulator assembly and a control system. The control system includes a processing unit. The processing unit determines kinematic information associated with a first tool inserted into the working site. The first tool is coupled to the manipulator assembly. The processing unit also receives image data generated by an endoscopic imaging instrument having a field of view. Image data is generated by the endoscopic imaging instrument after the first tool is removed from the manipulator assembly. The processing unit also determines one or more guided tool change parameters for guiding the second tool into the working site. The second tool is received for connection with the manipulator assembly after the first tool is removed. The guided tool change parameter is based on kinematic information associated with a first tool coupled to the manipulator assembly and based on a depth map determined from the image data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 477,583, filed on December 29, 2022, and entitled “Systems and Methods for Guided Tool Change Resiliency,” which is incorporated herein by reference in its entirety. Technical Field

[0003] Examples described herein relate to systems and methods for using endoscopic image data (including depth mapping or vision-based tool tracking) in robotic-assisted tool changing procedures. Background Art

[0004] Minimally invasive medical technology aims to reduce the amount of tissue damaged during a medical procedure, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques can be performed through natural orifices in the patient's anatomy or through one or more surgical incisions. Through these natural orifices or incisions, the operator can insert minimally invasive medical tools to reach the target tissue location. Minimally invasive medical tools include instruments such as treatment, diagnosis, biopsy, and surgical instruments. Minimally invasive medical tools may also include imaging instruments, such as endoscopic instruments that provide the user with a field of view within the patient's anatomy. Some minimally invasive medical tools and imaging instruments may be robot-assisted or otherwise computer-assisted. Medical procedures can employ many different medical tools. When a different tool is needed during a medical procedure, the medical tool can be withdrawn from the surgical site so that it can be removed from its associated manipulator and replaced with a different tool. The new tool is then inserted into the surgical site. Improved systems and methods are needed to improve the reliability and usability of control system-assisted tool change processes. Summary of the Invention

[0005] The following presents a simplified summary of various examples described herein and is not intended to identify key or critical elements or to delineate the scope of the claims.

[0006] Consistent with some examples, a medical system may include a manipulator assembly and a control system. The control system includes a processing unit. The processing unit determines kinematic information associated with a first tool inserted into a working site. The first tool is coupled to the manipulator assembly. The processing unit also receives image data generated by an endoscopic imaging instrument having a field of view. The image data is generated by the endoscopic imaging instrument after the first tool is removed from the manipulator assembly. The processing unit also determines one or more guided tool change parameters for guiding a second tool to the working site. The second tool is received and connected to the manipulator assembly after the first tool is removed. The guided tool change parameters are based on the kinematic information associated with the first tool coupled to the manipulator assembly and based on a depth map determined based on the image data.

[0007] It should be understood that both the foregoing general description and the following detailed description are exemplary and illustrative in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1A Schematic diagrams of a first medical tool and an imaging tool within a working site are provided according to some examples.

[0009] Figure 1B Schematic diagrams of a second medical tool and an imaging tool within a worksite during or after a tool change are provided, according to some examples.

[0010] Figure 2 is a schematic diagram of a medical system according to some examples.

[0011] Figure 3 is a flow chart illustrating a method for performing a guided tool change process according to some examples.

[0012] Figure 4A is a flow chart illustrating a method for generating guided tool change parameters for guiding a new tool, according to some examples.

[0013] Figure 4B Schematic diagrams of auxiliary tool target positions and insertion paths selected to accommodate the manipulator's range of motion are provided according to some examples.

[0014] Figures 5 and 6 is a flow chart illustrating a method for generating guided tool change parameters for guiding a new tool, according to some examples.

[0015] 7A to 7D A graphical user interface displaying a visual guide for a guided tool change process is shown.

[0016] Various examples described herein and their advantages are described in the following detailed description.It should be understood that for purposes of illustrating but not limiting the various examples described herein, like reference numerals are used to identify like elements shown in one or more of the drawings. DETAILED DESCRIPTION

[0017] A guided tool change process can kinematically determine the position of a tool attached to a robotic-assisted manipulator and can help deliver a replacement tool to the position of the first tool. When a kinematically undetectable replacement has occurred, various conditions can cause the robotic-assisted control system to invalidate or allow the guided tool change. Some systems and methods for guided tool change are described in U.S. Patent No. 6,645,196, which is incorporated herein by reference in its entirety. Sometimes, the control system of the robotic-assisted manipulator determines that the replacement tool cannot be located at the position of the first tool and invalidates or otherwise cancels the control system-guided tool change process, forcing the medical staff to manually introduce the replacement tool and resulting in inefficient workflow and inaccurate tool positioning. Various reasons may cause the control system-guided tool change process to be invalid. For example, tool tip inaccuracy may invalidate the control system-guided tool change process. Inaccuracies in the kinematic chain assessment of the position and / or orientation of the tip of the medical tool may be due to, for example, a tool end effector with long jaws or other compliant joints with uncertain kinematic posture. Additionally or alternatively, a control system-guided tool change process may be ineffective due to limitations on the manipulator's range of motion or if the manipulator encounters an obstacle during an attempt to reorient. Additionally or alternatively, a control system-guided tool change process may be ineffective if the manipulator is manually adjusted or even if the control system senses manual intervention, such as when a clutch mechanism is engaged to initiate manual movement of the manipulator. Sometimes, tissue movement, instrument deflection, and / or manipulator deflection may not be kinematically detected, resulting in the control system using incomplete information to determine parameters for control system-guided tool changes. In order to provide a more robust control system-guided tool change process that minimizes ineffectiveness, improves workflow efficiency, and allows for more robust usability, image-based information can be used to supplement kinematic tool change inputs to generate tool change parameters. More robust techniques for guided tool changes can utilize endoscopic image data (including vision-based tool tracking and / or depth mapping) to supplement kinematic position determination to provide a corrected insertion path, corrected insertion depth, and / or corrected tip position.

[0018] Figure 1A and Figure 1BA guided tool change operation is shown for a medical instrument or tool 100 (e.g., a first tool) that includes a shaft 102, a clevis 103, and an end effector 104 having a distal end 106. The distal end 106 can be a tool tip of the end effector 104. The tool 100 can be inserted through an anatomical wall 107 via an access port 109 into a working site, which can be an anatomical region 108 in a body cavity of a patient. An imaging instrument or tool 118 (e.g., an endoscope) having a field of view 119 can be inserted through the anatomical wall 107 via a port 111 into the anatomical region 108. The end effector 104 and anatomical tissue 105 can be within the field of view 119 of the imaging tool 118. The medical tool 100 and the imaging tool 118 can be manipulated from outside the anatomy 108 to have a Figure 1A The particular configuration shown. The tools 100, 118 may be manipulated using a robotic-assisted manipulator (eg, manipulator assembly 302) having one or more actuators.

[0019] like Figure 1BAs shown, before removing the tool 100 from the anatomical region 108 and introducing another tool 110 through the port 109, information can be collected from one or more sensor systems (e.g., sensor system 308) including an imaging system (e.g., imaging tool 118) to record the operational configuration of the medical tool 100, including the position, orientation, and / or pose. For example, the position of the distal end 106 can be kinematically determined using sensors such as potentiometers, encoders, or other types of position or velocity sensors for determining the kinematic configuration of the manipulator assembly and any tools coupled thereto. The sensed position can be used as a target point 122 to calculate parameters for controlling the manipulator arm to manipulate the second tool 110 and direct its movement to the operational configuration, including the position, orientation, and / or pose of the distal end 116 of the second tool 110 at the target point 122. In some examples, the same manipulator arm used to move the first tool 100 is used to manipulate the second tool 110 during and after the tool change operation. In other examples, parameters for controlling the manipulator arm during the guided tool change process can be determined from other types of sensors, including imaging tools, that can generate image data before or after the tool 100 is removed. For example, image data generated before the tool 100 is removed can provide information about the position and orientation of the distal end 106, including deflection or obstructions of the tool or manipulator that cannot be detected by kinematic sensors. Image data generated after the tool 100 is removed can provide information about the positioning of anatomical structures or other tools in the anatomical region that may have moved after the tool 100 is removed. In some examples, the image data can be stereo image data, which can be used to generate a depth map that provides information about the distance of the surface of an object in the field of view 119 from the distal end of the imaging tool 118. Depth mapping based on stereo image correlation can enable reconstruction of the topographic geometry of the anatomical environment and can enable the control system to determine the distance of a point in the anatomical region (e.g., the target insertion point) to the nearest surface visible in the field of view. The depth map can represent the perspective distance between the object in the field of view 119 and the plane of the imaging tool 118. In some examples, the depth map pixels may have an associated quality metric based on the output of the stereo matching process (e.g., the perpendicular error of the closest match relative to the epipolar plane). A quality threshold may be used to assess the integrity of any or all depth map pixels sampled for a projection or ray casting. The generation of a depth map of a working part is not limited to using data from an endoscopic imaging instrument. Other modalities of sensor data may be used (e.g., in place of or in combination with endoscopic imaging data) to generate a depth map. For example, a depth map may be generated using data from a depth sensing sensor (e.g., a structured light sensor, a time of flight (ToF) sensor, etc.), an ultrasound sensor, a computed tomography (CT), etc.In some implementations, the depth map can be generated by a composite or multimodal sensor (eg, an integrated endoscopic imaging and depth sensing sensor).

[0020] When the replacement tool 110 is introduced into the anatomical region 108, the recorded operational configuration information of the original tool 100 is used to provide guidance for the replacement tool 110 so that the replacement tool 110 can be prepared for operation, including being located in the same position, orientation, and / or pose as the original tool 100. The replacement tool 110 (e.g., a second tool) can include a shaft 112 and an end effector 114 having a distal end 116. Using the recorded operational configuration information of the tool 100, the replacement tool 110 can be accurately positioned in the anatomical region in substantially the same position relative to the anatomical tissue 105, the field of view 119 of the imaging tool 118, and any other tools or structures that may be in the anatomical region 108.

[0021] In some examples, the guided tool change process can record the operating position of the distal end 106 of the first tool 100 and then use this position as a target point 122 to determine the desired position of the distal end 116 of the second tool 110. In this way, the insertion depth of the second tool 110 can be limited by the position of the distal end 116 to prevent the second tool 110 from being extended too far into the anatomical region 108 and causing undesirable contact of the distal end 116 with the anatomical region 108. In some examples, the second tool 110 can be introduced via a straight insertion path to position the distal end 116 in the operating position. The insertion path is Figure 1B 10 is shown as an imaginary line 120 that extends through the remote center or rotation center 124 of the manipulator to which the tool 110 is coupled and intersects a target point 122 associated with the distal end 106 of the first tool 100 in the operating position. The remote center 124 can be the point about which the port 109 pivots relative to the anatomical wall 107. The remote center 124 position can be fixed relative to the patient's wall 107 (e.g., in X, Y, Z Cartesian space). In some examples, the remote center 124 is a remote center of motion constrained by hardware, wherein the positioning of the remote center 124 is set based on the configuration of the manipulator assembly. Additionally or alternatively, the remote center 124 can be a remote center of motion constrained by software. The insertion path 120 can serve as an insertion axis guide for guiding the second tool 110 and can be referred to as an in-and-out axis or IO axis. The IO axis can represent the freedom of movement of the tool. For example, the tool can be mounted to a carriage that is driven to translate along a linear guide structure of a robotic manipulator arm that can move with additional degrees of freedom, including angular displacement, to position the tool. In other examples, the insertion path can be curvilinear or have any shape that enables the distal end 116 to reach the target point 122.

[0022] In some examples, the distal end 116 of the second tool 110 does not need to be precisely positioned at the target point 122. A target space 125 can be defined about the target point 122 to provide an acceptable area for positioning the distal end 116, corresponding to, for example, any position within the surgeon's field of view 119. In some examples, the target space can define an offset or bias area of approximately 0.5 cm from the target point. Figure 1B A cylindrical target space 125 is shown, defined by specifying acceptable distances from a target point 122 at which the distal end 116 is positioned. In some examples, the target space 125 can be generally offset toward the distal center 124 to reduce the likelihood that the second tool 110 will contact tissue during insertion. In other examples, the target space can be spherical, rectangular prism-shaped, conical, frustum-shaped, or any shape that improves tool change efficiency or prevents extending the tool too far into the anatomical region. An insertion depth limit 126 can provide a depth limit along the insertion path 120 beyond which insertion of the tool 110 can be limited. The depth limit 126 can be applied as a visual guide on a display, tactile and / or force feedback to an operator control device, tactile and / or force feedback provided via the second tool 110 and / or manipulator assembly, a software-applied insertion stop, an audio prompt, or any other type of guidance that limits the distance beyond which the end effector 116 of the second tool 110 may contact tissue. In some examples, the depth limit 126 can be biased away from obstacles based on depth map uncertainty and manipulator remote center uncertainty in the endoscope tip reference frame. In one example, tactile and / or force feedback related to the depth limit 126 can be provided via the second tool 110 and / or manipulator assembly during a tool insertion operation to insert the second tool 116, which can be part of a guided tool change process. The tool insertion operation can be performed by table-side personnel. For example, when the end effector 116 of the second tool 110 approaches the depth limit 126 during tool insertion, the manipulator assembly (e.g., a prismatic or rotary joint on the manipulator assembly) can provide increased force feedback to prevent manual tool insertion from inserting the end effector 116 of the second tool 110 beyond the depth limit 126.

[0023] In some examples, the components discussed above can be part of a computer-assisted or robot-assisted system as described in further detail below. Such a computer-assisted or robot-assisted system can be suitable for use in, for example, surgery, robot-assisted surgery, diagnosis, treatment, or biopsy procedures. Although some examples of such procedures are provided herein, any reference to medical or surgical instruments and medical or surgical methods is non-limiting. The systems, instruments, and methods described herein can be used for animals, human cadavers, animal cadavers, parts of human or animal anatomical structures, non-surgical diagnostics, as well as for industrial systems and general-purpose robots, general-purpose robot-assisted or robotic medical systems.

[0024] like Figure 2 As shown, medical system 300 generally includes a manipulator assembly 302, which includes manipulator arms 303 and 304. Tool 100 can initially be coupled to and manipulated by manipulator arm 303, and after a guided tool change procedure, tool 110 can be coupled to and manipulated by manipulator arm 303. Imaging tool 118 can be coupled to and manipulated by manipulator arm 304. Tools 100, 110, 118 can be used when performing various procedures on a patient P positioned on a table T. In some examples, manipulator assembly 302 can include more or fewer than the two arms shown. Manipulator assembly 302 can be a robotically assisted, non-robotically assisted, or a hybrid robotically assisted and non-robotically assisted assembly, having selected degrees of freedom of motion that can be motorized and / or robotically assisted, and selected degrees of freedom of motion that can be non-motorized and / or non-robotically assisted. The medical system 300 may also include an input system 306, which generally includes one or more operator control devices for controlling the manipulator assembly 302. The manipulator assembly 302 supports the tools 100, 110, 118 and may optionally include a plurality of actuators or motors that drive inputs on the medical tools in response to commands from a control system 312. The actuators may optionally include a drive system that, when coupled to the medical tools 100, 110, 118, can advance the medical tools into natural or surgically created anatomical orifices.

[0025] The medical system 300 also includes a display system 310 for displaying images or representations of the surgical site and medical tools generated by the imaging tool 118 and / or the sensor system 308. The display system 310 and the input system 306 can be oriented so that the operator O can control the tools (e.g., medical tools, non-medical tools, imaging instruments, etc.) and manipulator assemblies using the perception of telepresence. The display system 310 can include one or more display screens, including a display screen visible to a table-side operator whose task is to interact with the manipulator assemblies and tools during instrument changes. Additional information about the medical system 300 and medical tools can be found in International Application Publication No. WO 2018 / 195216, filed on April 18, 2018, entitled “Graphical User Interface for Monitoring an Image-Guided Procedure,” which is incorporated herein by reference in its entirety.

[0026] In some examples, the imaging tool 118 may include components of an imaging system that includes an endoscopic imaging instrument assembly that records simultaneous or real-time images of the surgical site and provides the images to an operator or operator O via one or more displays of the medical system 300 (e.g., one or more displays of the display system 310). The simultaneous images may be, for example, two-dimensional or three-dimensional images captured by an imaging instrument positioned within the surgical site. The imaging system may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which may include a processor of the control system 312.

[0027] The sensor system 308 may include position / location sensors (e.g., actuator encoders or electromagnetic (EM) sensor systems) and / or shape sensors (e.g., fiber optic shape sensors) for determining the position, orientation, velocity, speed, pose, and / or shape of the medical tool, manipulator arms 303, 304, and / or components of the manipulator assembly 302. The sensor system 308 may also include pressure sensors, force sensors, contact sensors, and the like.

[0028] The control system 312 includes at least one memory 316 and at least one computer processor or processing unit 314 for implementing control between the tools 100, 110, 118, the input system 306, the sensor system 308, and the display system 310. The control system 312 also includes programming instructions (e.g., a non-transitory machine-readable medium having instructions stored thereon) for implementing some or all of the methods described in accordance with various aspects disclosed herein, including instructions for providing information to the display system 310.

[0029] Figure 3 is a flowchart illustrating a method 400 for performing a guided tool change process according to some examples. The methods disclosed herein can be illustrated as a set of operations or processes that can be performed in the same or different order than the order shown. In some examples of the method, one or more of the processes shown can be omitted. Additionally, one or more processes not explicitly shown in the flowchart may be included before, after, between, or as part of the processes shown. In some examples, one or more of the processes can be implemented at least in part by a control system executing code stored on a non-transitory tangible machine-readable medium, which, when executed by one or more processors (e.g., a processor of the control system), can cause the one or more processors to perform one or more of the processes.

[0030] At process 402, an indication of an intended tool change may be received, for example, by a control system of a robotic-assisted medical system. The indication may be generated, for example, by an operator pressing a button on a master assembly, an operator interacting with a graphical user interface, or the control system identifying a condition corresponding to a tool change (e.g., a change in operating mode or the start of a new sequence in a medical procedure). In some examples, the indication may be an indication to change tool 100.

[0031] At process 404, kinematic information associated with the position, orientation, and / or posture of the first tool can be determined and / or recorded. In some examples, the kinematic information can be used to determine the target point 122 based on a kinematic chain extending to the distal end 106 of the tool 100. In some examples, process 404 can be performed before process 402. Process 404 can determine and / or record any information about the state of the first tool and / or its coupled manipulator assembly, as may be required for later calculations. The kinematic information can include the position, orientation, and / or posture of the structures and joints in the kinematic chain, which includes the manipulator assembly and the tool to be replaced. The kinematic information can be determined using sensors such as potentiometers, encoders, or other types of position or velocity sensors for determining the kinematic configuration of the manipulator assembly and any tool coupled thereto. The kinematic information can also include the size and shape of the links in the kinematic chain (including components of the manipulator assembly and the tool). The kinematic information can be determined and / or recorded with reference to a reference frame, such as the distal end of the manipulator assembly, the patient, or an imaging device for capturing images of the patient's anatomical structure. In some examples, kinematic information may be stored in a memory of the medical system (eg, memory 316 ).

[0032] At optional process 406, image data (e.g., first image data) of a tool within the field of view of an imaging instrument (e.g., an endoscopic imaging instrument) can be received and / or recorded from an imaging instrument. For example, image data of tool 100 within the field of view 119 of imaging instrument 118 can be received and / or recorded. In some examples, the image data can be stereo image data received from a stereo endoscope. The stereo image data can be used for vision-based tool tracking to triangulate the position of structures in the field of view, including distal end 106, target point 122, clevis 103, shaft 102, and / or anatomical tissue 105. The image data can enable a control system to determine the position of the structure relative to the endoscope tip, which can also be mapped to a common stationary reference frame between tool 100 and tool 118.

[0033] At process 408, guided tool change parameters for guiding the second tool may be determined. For example, guided tool change parameters may be determined for guiding tool 110 to the work site (e.g., anatomical region 108) through port 109 after tool 100 is removed. Figure 6 As described in more detail in the method of , the guided tool change parameters can be determined based on kinematic information, image data, or other inputs. The guided tool change parameters can include one or more of: a target position and orientation of the second tool tip, an insertion path of the second tool between the port and the target position, a location of a remote center of a manipulator arm to which the second tool is coupled, an insertion depth limit of the second tool along the insertion path, or a configuration of a manipulator assembly used to generate the insertion path.

[0034] At process 410, the determined guided tool change parameters can be used to perform a guided tool change process to introduce a second tool (e.g., tool 110) into the anatomical region. For example, the manipulator assembly 302 and / or manipulator arm 303 can be repositioned to adjust the guide trajectory of the new tool 110. The new tool 110 can be identified and engaged at the bracket of the manipulator assembly 302. The new tool 110 will typically be different from the first tool 100, although in some cases they can be the same tool. In some examples, data can be retrieved from the second tool 110 via a readable memory chip. Alternatively, kinematic information including the dimensions and other data of the second tool 110 can be input via an operator interface. The sensor system 308 can be used to detect the engagement between the second tool 110 and the manipulator arm 303 to ensure proper engagement before performing the introduction process. The manipulator assembly 302 (or one or more parts thereof) can be repositioned before engaging with the second tool 110. This adjustment may result in adjustments to the I / O axis and / or remote center. The tool 110 can be introduced into the surgical site 108 along the insertion path 120 through the access port 109. The control system 312 can guide the manipulator assembly 302 to float the movement degrees of freedom along the insertion path 120 so that the second tool 110 can be moved into the working site (e.g., the anatomical region 108) by the surgeon's assistant or by the controller itself, causing the tool to move along the insertion path 120. The new tool 110 can be introduced into the anatomical region 108 until the tip reaches the target space 125 or target point 122. The introduction of the second tool 110 can be performed by the operator through direct manual operation or using a joystick or other interface, or by the operator remotely via the input system 306, but can alternatively be performed at least partially or completely by the control system 312. A graphical representation of the insertion path 120, the target space 125, the target point 122, and / or the image of the field of view 119 can be displayed on the display system 310 to guide the operator in inserting the second tool 110. When the distal end of the tool 110 reaches the target space 125 or target point 122 , the operator control at the input system 306 may encounter tactile resistance generated by the control system 312 to stop the movement of the tool 110 .

[0035] Typically, introducing a tool into the patient's anatomy may require movement of one or more degrees of freedom. Some degrees of freedom during tool introduction may be constrained. In some guided tool change processes, a single degree of freedom (e.g., corresponding to the degree of freedom of movement along the insertion axis) is available, while all other degrees of freedom are constrained, particularly the other two proximal degrees of freedom for positioning the tool mount before reinsertion and the distal degrees of freedom associated with the instrument's U-shaped clamp and end effector. After the operational connection between the main control, manipulator assembly, and tool is reestablished, preferably after the tool change is complete, these degrees of freedom can be released for operator movement. The unconstrained degrees of freedom can constitute a linear axis of movement that precisely coincides with a joint of the tool holder on the manipulator arm. Typically, to describe guided tool change, some degrees of freedom of movement are determined to be free to move, while the remaining degrees of freedom of movement are absolutely or substantially constrained via servo control, etc., so as to not move or resist movement in that direction, respectively. As sensed by the sensor system 308, the entry of the tip of the second tool into the target space 125 can indicate the end of the guided tool change. A transfer of control back to the operator is then provided, either automatically or by indicating that the operator can take control by taking some action (e.g., by actuating the end effector or by pressing an input button). It will be appreciated that other criteria may also indicate the end of the guided tool change. These include positioning of other points on the tool in other target spaces, positioning of the joint in some predefined or precalculated region, other sensor input (e.g., vision- or image-based tool detection), or explicit user input.

[0036] Figure 4A A method 500 for generating one or more guided tool change parameters for guiding a new tool is shown. The method 500 can be used as part of a process 408 for determining guided tool change parameters for guiding a second tool. At process 502, a guided tool change parameter input can be generated based on kinematic information associated with the configuration (including position, pose, and / or orientation) of the first tool. The guided tool change parameter input can include any kinematic information recorded at process 404, such as the position, orientation, and / or pose of structures and joints in the kinematic chain of the manipulator assembly and the tool to be replaced. The guided tool change parameter input can also be derived from the kinematic information. Such input can include a target point 122 at the distal end 106 of the tool 100, a target space 125, an insertion depth limit 126, and / or a kinematically determined distance between the operating tool, the imaging tool, and the anatomical structure.

[0037] At process 504, guided tool change parameters for guiding the second tool can be generated based on the guided tool change parameter input. The guided tool change parameters may include, for example, a target position and orientation of the tip of the second tool, an insertion path of the second tool between the port and the target position, a location of the remote center of the second tool, an insertion depth limit of the second tool along the insertion path, and a configuration of the manipulator assembly for generating the insertion path. For example, an insertion path 120 for the new tool 110 can be determined based on the guided tool change parameter input. This determination may include calculating a position of the tool holder on the manipulator arm 303 that, when engaged with the new tool 110, will enable the new tool 110 to begin to be introduced into the anatomical region 108 along the specified path 120 so that the new tool path will intersect the target point 122 or target space 125 previously occupied by the end effector of the first tool. The determined insertion path can be used to operate the manipulator assembly 302 to reposition the tool holder on the arm 303. The parameters may also include, for example, coordinates of specific points and joint angles of specific joints between the manipulator assembly 302 and a linkage included in the second tool 110. In a specific example, the parameters used to reposition the tool mount include an outer pitch angle and an outer yaw angle relative to a reference frame used to guide the tool 110 along the insertion path 120. The outer pitch and outer yaw of the second tool 110 may generally differ from the outer pitch and outer yaw of the first tool 100 in its operative position prior to its removal.

[0038] In some examples, the kinematically determined guided tool change parameters can be initial parameters, such as the initial target position 122 or the initial insertion path 120, which can be modified, adjusted, corrected, or otherwise changed based on additional parameter inputs derived from the imaging system, as described below in other examples. In some examples, the guided tool change process can be invalidated if the generated guided tool change parameters violate other conditions of the guided tool change process. For example, if the guided tool change parameters include an insertion path that exceeds the range of motion of the manipulator assembly (e.g., if the motion exceeds a 30-degree cone angle), or if the manipulator arm encounters an obstacle when moving to align with the insertion path, the guided tool change can be invalidated.

[0039] Figure 4B An example of an auxiliary target point selected to avoid invalidating a guided tool change is provided. The initially selected target point 122A and insertion path 120A (which may correspond to a target point determined for a new tool) are shown in FIG. Figure 2The target point 122A and insertion path 120B may not be achievable due to limitations on the range of motion of the manipulator arm. To avoid invalidating the guided tool change process, for example, the target point 122A can be moved along the new insertion path 120B that does not exceed the range of motion of the manipulator assembly to a target point 122B at a location within the volume 150 of the first tool (e.g., a clevis or along the axis of the first tool 100). In some examples, the target point 122B can be identified as the most distal location along the initial tool assembly that satisfies the kinematic constraints of the subsequent insertion trajectory while still being within the field of view of the endoscope.

[0040] At optional process 506, visual guidance may be displayed to assist an operator (e.g., a table-side operator) in performing a guided tool change. For example, a visual guidance including a target point 122, a target space 125, an insertion path 120, an arrow depicting a direction of motion, and / or a graphical representation of the field of view 119 may be displayed on the display system 310 to assist in guiding the new tool 110. This may be accomplished via a control panel such as a computer. 7A to 7D The graphical user interface shown in the figure provides an example of a visual guide.

[0041] At optional process 508, tactile guidance can be provided to assist an operator (e.g., a table-side operator) in performing a guided tool change. For example, tactile guidance in the form of resistance, vibration, or other tactile sensation can be provided to the manipulator arm 303 so that the arm can snap into the insertion path when the insertion axis of the arm becomes aligned with the target position. For example, when the end effector of the second tool approaches or reaches the target point and / or depth limit during tool insertion by the table-side operator, the manipulator assembly (e.g., a prismatic or rotational joint on the manipulator assembly) can provide tactile and / or force feedback. The operator can optionally override the tactile force.

[0042] In some examples, if the manipulator arm is clutched or moved out of a constrained position by the operator during an instrument change, the guided tool change process may be invalidated. To avoid invalidating the guided tool change process and enable the operator to resume using the guided tool change process, visual and / or tactile guidance can be provided. For example, when the control system becomes alerted to disengagement or movement, tactile or visual guidance can be provided to the operator to reposition the manipulator arm. Visual and tactile guidance can enable the operator to optionally realign the arm with the guided tool change trajectory before resuming the instrument change process and before advancing the instrument into the body. Guidance can include displayed positioning of the insertion path and target position. Guidance can also be updated to include a virtual extension of the insertion path based on the manipulator configuration of the movement. The operator can compare the target position, insertion path, and virtual extension. Tactile guidance can be provided when the user moves the manipulator assembly to align the insertion axis of the manipulator arm with the insertion path. An accurately calibrated image and manipulator reference system may be required to provide visual guidance as the manipulator assembly moves.

[0043] Figure 5A method 600 for generating one or more guided tool change parameters for guiding a new tool is shown. Method 600 can be used as part of process 408 for determining guided tool change parameters for guiding a second tool. At process 602, guided tool change parameter inputs can be generated based on first image data recorded at process 406. In some examples, the first image data can be stereoscopic endoscopic image data of field of view 119 collected when tool 100 is within anatomical region 108. The image data can be used to supplement or modify the kinematic information recorded at process 404. For some instruments or instrument configurations, kinematic information may not accurately indicate the true position and orientation of the distal tip of the instrument end effector. For example, a long-jawed end effector may be deflected by interaction with tissue or other structures in the anatomical region, causing the tip of the end effector to displace several millimeters from the intended kinematic position. Additionally or alternatively, backlash in the control cables that engage the end effector or drive train compliance issues may cause the tip of the end effector to displace from the intended and commanded kinematic position. Additionally or alternatively, kinematic uncertainty can be associated with the joints and linkages of the manipulator assembly. The position and orientation of the distal tip of the end effector visible in the image data can be used to correct the kinematic information to provide a more accurate determination of the target point 122, the target space 125, and the insertion path 120. The image data reference frame can be registered to a common reference frame with the manipulator assembly and the tool 100 to correct the position and orientation in the common reference frame. The image data from the field of view 119 can provide guided tool change parameter inputs, for example, in the form of tool and tissue position and orientation information, which can be used to correct or modify the kinematically derived parameter inputs.

[0044] At processing 604, guided tool change parameters for guiding the second tool can be generated based on the guided tool change parameter input. The guided tool change parameters may include, for example, the target position and orientation of the second tool tip, the insertion path of the second tool between the port and the target position, the insertion depth limit of the second tool along the insertion path, and the configuration of the manipulator assembly for generating the insertion path. For example, the guided tool change parameter input from the first image data can be combined with the guided tool change parameter input from the kinematic information (processing 502) or used to modify the guided tool change parameter input from the kinematic information (processing 502) to generate more accurate guided tool change parameters. For example, in order to more accurately determine the target position and orientation of the second tool tip, a more accurate target point 122 of the first tool can be determined. More specifically, the image-based position of the instrument distal end 106 can be determined in the image reference system. The position of the distal end 106 can be transformed from the image reference system to the manipulator assembly reference system. Comparison of the transformed image-based position of the distal end 106 with a kinematically determined target point 122 (corresponding to the kinematically determined position of the distal end 106) in the manipulator assembly reference frame can identify inaccuracies in the kinematically determined position. A modified target point 122 can be determined based on the identified inaccuracies to generate a modified target position and orientation of the second tool tip. More specifically, in some examples, the distal tip position uncertainty associated with the kinematically based solution and the uncertainty associated with the image-based distal tip position can be compared. The system can conditionally use the estimate with the lowest acceptable uncertainty (e.g., a threshold or a predetermined lowest uncertainty). If neither estimate has a sufficiently low uncertainty, the guided tool change process can be canceled. Other parameters including the insertion path and the configuration of the manipulator assembly can be adjusted based on the revised target point.

[0045] At optional process 606, visual guidance can be displayed to assist an operator (e.g., a table-side operator) in performing a guided tool change. For example, visual guidance including a graphical representation of image adjustment parameters, including an adjusted target point 122, an adjusted target space 125, and / or an adjusted insertion path 120, can be displayed on the display system 310 along with the image of the field of view 119 to assist in guiding the new tool 110. In some examples, an indicator can be displayed on the display system 310 to indicate an occluded target point. For example, if the determined target point location is not visible in the image data (e.g., because tissue or a tool blocks the field of view or because the determined target point location is outside the field of view), a graphical or textual indicator can be displayed during the tool change process to indicate the location of the occluded or off-screen target point.

[0046] At optional process 608 , tactile guidance can be provided to assist an operator (eg, a table-side operator) in performing a guided tool change. For example, tactile guidance in the form of resistance, vibration, or other tactile sensations as described at process 508 can be provided.

[0047] Figure 6 A method 700 for generating one or more guided tool change parameters for guiding a new tool is shown. Method 700 can be used as part of process 408 of determining guided tool change parameters for guiding a second tool. At process 702, image data (e.g., second image data) of the field of view of the endoscopic instrument can be recorded after the first tool is removed. For example, image data of the anatomical region 108, anatomical tissue 105, and any other tools or structures within the field of view 119 of the imaging tool 118 can be recorded after the tool 100 is removed and before other tools are inserted through port 109. After the tool 100 is removed, the second image data can capture the displacement or movement of the tissue 105 or other tools and structures in the field of view 119, thereby providing information about insertion path obstructions that may hinder the introduction of subsequent tools. The second image data can be used to supplement or modify the kinematic information recorded at process 404 and / or the first image data recorded at process 406. Process 702 may be optional to method 700.

[0048] At process 704, an image depth map can be generated based on the second image data. In some examples, the image data can be stereo image data received from a stereo endoscope and can be used to generate a depth map that provides information about the distance of the surface of an object in the field of view 119 from the distal end of the imaging tool 118. The depth map can represent the perspective distance between the object in the field of view 119 and the plane of the imaging tool 118. In some examples, the mapping points and / or vectors between the depth map image space and the imaging tool tip coordinates can depend on a calibrated camera model (i.e., intrinsic and extrinsic parameters).

[0049] The present invention also depends on being able to map the remote center position and insertion axis of the instrument manipulators into endoscope tip coordinates using a common reference frame and kinematic chain between the manipulators.

[0050] At process 706, a guided tool change parameter input can be generated based on the image depth map. The image depth map can be used to supplement or modify the kinematic information recorded at process 404 and / or the first image data recorded at process 406. Sometimes, the anatomical region 108 can change after the first tool 100 is removed. For example, tissue in contact with the first tool 100 can move into the space vacated by the tool. Additionally or alternatively, the deformable tissue can move or slide relative to other tissues, and some tissues can move in response to breathing, cardiac motion, or blood flow. The guided tool change parameter input determined based on the depth map can include, for example, the distance between the kinematically identified target point 122 and the surface of the anatomical tissue 105 or the distance between the surface of the anatomical tissue 105 and the anatomical wall 107 adjacent to the imaging tool or adjacent to the port 109. In some examples, the guided tool change parameter input can include the location of the intersection of the kinematically derived insertion path 120 and the anatomical tissue 105. The proximity of the target point 122 and / or the insertion path 120 to the depth map corresponding to the anatomy 105 may determine whether the target location and insertion path remain valid, become invalid, or may be adjusted.

[0051] If the target location remains valid and reachable by the second tool, the depth map can be used to generate an indicator of the occluded target location at process 708. For example, if the depth map indicates that the kinematically determined target point 122 is occluded by the anatomical tissue 105, a graphical marker, text, or other indicator of the location of the target location can be displayed.

[0052] At process 710, guided tool change parameters for guiding the second tool can be generated. The guided tool change parameters can include, for example, a target position and orientation of the second tool tip, an insertion path of the second tool between the port and the target position, an insertion depth limit of the second tool along the insertion path, and a configuration of a manipulator assembly for generating the insertion path. For example, the guided tool change parameter input based on the depth map can be combined with or used to modify the guided tool change parameter input from the kinematic information (process 502) and / or the image information (process 602) to generate more accurate guided tool change parameters.

[0053] In some examples, an insertion depth limit, adjusted or limited by the intersection of the insertion path and the depth map, can be continuously calculated during insertion of the second tool to account for anatomical motion, such as respiration or pulsation. The insertion depth limit can be determined as the minimum depth along the insertion path observed over a period of time just before the instrument is finally advanced to the target location (e.g., within a threshold distance or estimated time of arrival to the target location). The insertion depth map limit can be continuously refined until the new instrument tip has been advanced to a predetermined location, such as a location obstructed by the instrument tip or the endoscopic field of view of the target.

[0054] In some examples, the tool 110 can be initially modeled as a cylinder extending along the insertion path 120 and tested for intersection with a depth map corresponding to the anatomical tissue 105 in the anatomical region 108. If the modeled cylinder does not intersect the depth map corresponding to the anatomical tissue 105, the target point 122 can remain valid. The remote center 124 can also remain at the kinematically determined location. In some examples, the diameter of the modeled cylinder can be larger than the diameter of the tool shaft to account for uncertainty in the manipulator remote center relative to the endoscope tip reference frame.

[0055] In some examples, if the kinematically determined target point 122 is invalid due to, for example, tool tip kinematic positioning inaccuracies, limitations on the manipulator's range of motion, or manual adjustments to manipulator components, the depth map can be used to determine a modified target position. For example, a ray casting process can cast a ray segment along the kinematically determined insertion path 120 to determine whether the cast ray intersects the depth map before reaching the kinematically determined target point 122. If the ray intersects the depth map, the target position can be considered occluded, and the target position can be adjusted to a modified target position at or near the intersection of the cast ray and the depth map.

[0056] In some examples, the target position can be determined or modified based on the kinematically determined target position by determining a trajectory between the remote center of the manipulator arm and the center point in the field of view captured by the second image data. One or more test rays can be projected along the trajectory to adjust the insertion depth and target position based on the positioning of the interference with the depth map. Once the optimized target position is determined by the test rays, the inverse arm kinematics can be used to determine the manipulator arm pose with an insertion axis that intersects the optimized target position, and determine the corresponding insertion depth to reach the optimized target position. In some examples, the ray casting should be emitted from an unobstructed point in space, such as the distal end of port 109, the distal tip of the cannula extending in the port, or the distal tip of the instrument. Such a launch point can avoid premature intersection with the depth surface contributed by the component (e.g., port, cannula, instrument) itself. Additionally, the ray casting calculation can be constrained to consider only one-sided depth map surface transitions. For example, if the ray originates from the inside of the depth map, the initial transition from the inside to the outside can be ignored.

[0057] In some examples, the guided tool change process may be invalidated if the manipulator arm is disengaged or moved by the operator. To avoid invalidating the guided tool change process, a depth map can be referenced to determine whether the kinematically determined target point 122 or the insertion path 120 is blocked by the anatomical tissue 105 or other structures in the anatomical region 108. If no obstruction is identified based on the depth map, the tool change process can continue without being invalidated. In some examples, a guided tool change process can only be performed if the adjustment of the manipulator pose is small enough (e.g., below a threshold pose change relative to the initial pose) so that the insertion path is within a tolerance of the original insertion path.

[0058] In some examples, the depth map is used to configure the end effector of the second tool after the second tool is inserted into the anatomical region. For example, the depth map can be evaluated to determine the location of surrounding tissue and determine the jaw opening and / or wrist or clevis orientation of the second tool that can avoid impacting the surrounding tissue.

[0059] At optional process 712, visual guidance may be displayed to assist the operator in performing the guided tool change. For example, visual guidance including a graphical representation of image adjustment parameters including a new or modified target point, a modified target space, and / or a modified insertion path 120 may be displayed on the display system 310 along with the image of the field of view 119 to assist in guiding the new tool 110.

[0060] 7A to 7D A graphical user interface 800 is shown displaying a visual guide including image adjustment parameters. The graphical user interface 800 can be displayed on a display system (eg, display system 310) of a medical system, for example. Figure 7A A graphical user interface 800 is shown that includes an image of an imaging tool field of view 802 (e.g., field of view 119) that includes anatomical tissue 804 and a first tool 806. The distal portion of the first tool 806 can be located at position 808. Figure 7B As shown, after the first tool 806 is removed, a marker 810 can be displayed at a target location, which, as described herein, can be determined based on kinematic information associated with the first tool 806. For example, the target location can correspond to the position 808 of the removed first tool 806, or can be otherwise determined based on the position 808 of the removed first tool 806. In some examples, the target location can be based on the distal-most jaw tip position of the first tool just before removal. A modeled or synthesized cylinder 812 can provide guidance for the second tool to advance toward the marker 810 in the form of a graphical insertion path. In some examples, as Figure 7C As shown, the target position can be adjusted based on the structures in the field of view. For example, the tissue 804 can move, including shifting, expanding, bulging, or otherwise becoming displaced, to obstruct the marker position 808. As described in method 700, the depth map information can be used to identify blocking tissue and generate guided tool change parameters including a revised target position, and the revised marker 814 can be displayed at the revised target position. The revised target position can be located near the tissue 804 without being obstructed by the tissue. The second tool 816 can be guided along the modeled cylinder 802 until the distal portion reaches the revised target position. By delivering the second tool 816 to the revised target position, direct or penetrating contact with the tissue 804 can be avoided. In some examples, such as Figure 7DAs shown, the correction or change of the target location can be displayed in the graphical user interface to enhance the operator's perception of the displacement of the target location. For example, a mark 818 corresponding to the original target location and a mark 814 corresponding to the revised target location can be displayed simultaneously in the graphical user interface 800. The features (such as color, texture, opacity or shape) of the mark 818 can provide that the original location is blocked by tissue and therefore cannot be reached by the second tool 816. In addition or as an alternative, the volume defined by the marks 818 and 814 in the cylinder 812 can be displayed as visually distinguished from the rest of the cylinder 812 (for example, in terms of color, shadow, transparency level, texture, etc.) to provide additional visual cues of the change of the target location. In some examples, the orientation of the modeled cylinder 802 can remain constant, and the revised target location 814 can be located at a position (for example, as marked by marks 810, 818) proximal to the target location 808 along the longitudinal axis of the modeled cylinder 802. In other examples, the orientation of the modeled cylinder can change in response to the moving tissue, and the revised target position can be located along the revised longitudinal axis of the modeled cylinder. In some examples, a manipulator assembly (e.g., manipulator assembly 302) to which the second tool is attached can enforce the revised target position 814 and restrict the second tool from moving beyond the revised target position.

[0061] At optional process 714 , tactile guidance can be provided to assist the operator in performing guided tool changes. For example, tactile guidance in the form of resistance, vibration, or other tactile sensations can be provided to the operator control device at the main assembly 306 .

[0062] Unless the context indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms as well. And the terms "comprises", "comprising", "includes", "has", etc. specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be directly coupled electrically or mechanically, or they may be indirectly coupled via one or more intermediate components. The auxiliary verb "may" also implies that a feature, step, operation, element, or component is optional.

[0063] In the description, the specific details describing some embodiments have been set forth. Many specific details have been set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be exemplary and non-restrictive. Those skilled in the art may implement other elements that are within the scope and spirit of this disclosure, although not specifically described herein.

[0064] Where feasible, the elements described in detail with reference to an embodiment, implementation or application may optionally be included in other embodiments, implementations or applications in which they are not specifically shown or described. For example, if an element is described in detail with reference to an embodiment without describing the element with reference to a second embodiment, the element may still be considered to be included in the second embodiment. Therefore, in order to avoid unnecessary repetition in the following description, one or more elements shown and described in association with an embodiment, implementation or application may be incorporated into other embodiments, implementations or aspects, unless specifically described otherwise, unless the one or more elements will make the embodiment or implementation inoperative, or unless two or more elements in the element provide conflicting functions. Not all of the processes shown can be performed in all embodiments of the disclosed method. Additionally, one or more processes not clearly shown may be included before, after, between or as part of the processes shown. In some embodiments, one or more processes in the process may be performed by a control system, or may be implemented at least in part in the form of executable code stored on a non-transient tangible machine-readable medium, which may enable one or more processors to perform one or more processes in the process when being run by one or more processors.

[0065] Any changes and other modifications to the described equipment, apparatus, method and any other application of the principles of the present disclosure as those skilled in the art to which the present disclosure relates would normally be fully anticipated. In addition, the dimensions provided herein are for specific examples, and it is contemplated that different sizes, dimensions and / or ratios can be utilized to implement the conception of the present disclosure. In order to avoid unnecessary descriptive repetition, one or more components or actions described according to an exemplary embodiment can be used or omitted as appropriate in other exemplary embodiments. For the sake of brevity, many iterations of these combinations will not be described separately. For the sake of simplicity, in some cases, the same reference numerals are used throughout the accompanying drawings to refer to identical or similar components.

[0066] The systems and methods described herein can be applicable to processes involving any of a variety of anatomical systems, including the lungs, colon, intestines, stomach, liver, kidneys and calyces, brain, heart, circulatory system including vascular system, etc. Although some embodiments of medical procedures are provided herein, any reference to medical or surgical instruments and medical or surgical methods is non-restrictive. For example, the instruments, systems and methods described herein can be used for non-medical purposes, including industrial uses, general robotic uses, and sensing or manipulating non-tissue artifacts. Other example applications relate to cosmetic improvements, imaging of human or animal anatomical structures, collecting data from human or animal anatomical structures, and training medical or non-medical personnel. Additional example applications include performing surgery on tissue removed from human or animal anatomical structures (not returned to human or animal anatomical structures) and performing surgery on human or animal corpses. In addition, these technologies can also be used for surgical and non-surgical medical treatments or diagnostic processes.

[0067] One or more elements in the embodiments of the present disclosure may be implemented in software to be executed on a processor of a computer system such as a control processing system. When implemented in software, the elements of the embodiments of the present disclosure may be code segments for performing various tasks. The program or code segment may be stored in a processor-readable storage medium or device, which may be downloaded via a transmission medium or communication link via a computer data signal embodied in a carrier wave. The processor-readable storage device may include any medium that can store information, including optical media, semiconductor media, and / or magnetic media. Examples of processor-readable storage devices include: electronic circuits; semiconductor devices, semiconductor memory devices, read-only memories (ROMs), flash memories, erasable programmable read-only memories (EPROMs); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. The code segments may be downloaded via a computer network such as the Internet, an intranet, or the like. Any of a variety of centralized or distributed data processing architectures may be employed. The programming instructions may be implemented as many separate programs or subroutines, or they may be integrated into many other aspects of the system described herein. In some examples, the control system may support wireless communication protocols such as Bluetooth, Infrared Data Association (IrDA), HomeRF, IEEE 802.11, Digital Enhanced Cordless Telecommunications (DECT), Ultra-Wideband (UWB), ZigBee, and wireless telemetry.

[0068] Note that the processes and displays presented may not be inherently related to any particular computer or other device. Various general-purpose systems may be used with programs according to the teachings herein, or it may prove convenient to construct more specialized devices to perform the described operations. The structures required for various such systems will appear as elements in the claims. In addition, embodiments of the present invention are not described with reference to any particular programming language. It will be understood that various programming languages may be used to implement the teachings of the present invention described herein.

[0069] The present disclosure describes various instruments, parts of instruments and anatomical structures according to their states in three-dimensional space. As used herein, the term "position" refers to the positioning of an object or a part of an object in three-dimensional space (e.g., along three translational degrees of freedom of Cartesian x, y and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or a part of an object (e.g., in one or more rotational degrees of freedom such as roll, pitch and / or yaw). As used herein, the term "pose" refers to the position of an object or a part of an object in at least one translational degree of freedom, and refers to the orientation of the object or a part of an object in at least one rotational degree of freedom (e.g., up to six total degrees of freedom). As used herein, the term "shape" refers to a set of poses, positions or orientations measured along an object.

[0070] While certain exemplary embodiments of the invention have been described and shown in the drawings, it is to be understood that such embodiments are merely illustrative and not restrictive of the broader invention, and that the embodiments of the invention are not limited to the specific construction and arrangements shown and described, since various other modifications may occur to those skilled in the art.

Claims

1. A medical system comprising: Manipulator assembly; as well as A control system, wherein the control system comprises a processing unit comprising one or more processors, and wherein the processing unit is configured to: determining kinematic information associated with a first tool inserted into a work site, wherein the first tool is coupled to the manipulator assembly; receiving image data generated by an endoscopic imaging instrument having a field of view, the image data generated by the endoscopic imaging instrument after the first tool is removed from the manipulator assembly; and One or more guided tool change parameters are determined for guiding a second tool into the work site, wherein the second tool is received in connection with the manipulator assembly after the first tool is removed, and wherein the one or more guided tool change parameters are based on the kinematic information associated with the first tool coupled to the manipulator assembly and based on a depth map determined from the image data.

2. The medical system according to claim 1, wherein: The one or more guided tool change parameters include a target position of a tool tip of the second tool.

3. The medical system according to claim 2, wherein: The one or more guided tool change parameters include an insertion path of the second tool through an access port to the target location.

4. The medical system according to claim 3, wherein: The one or more guided tool change parameters include an insertion depth limit for the second tool along the insertion path.

5. The medical system according to claim 3, wherein: The one or more guided tool change parameters include a configuration of the manipulator assembly that guides the second tool along the insertion path.

6. The medical system according to claim 1, wherein: The one or more guided tool change parameters include a position of a remote center of motion of a manipulator arm of the manipulator assembly.

7. The medical system according to claim 1, wherein: Determining one or more guided tool change parameters includes determining an initial target position for a tool tip of the second tool based on the kinematic information associated with the first tool, and modifying the initial target position based on the depth map.

8. The medical system according to claim 7, wherein: Modifying the initial target position includes determining a distance between the initial target position and an anatomical surface based on the depth map.

9. The medical system according to claim 7, wherein: Modifying the initial target location includes determining, based on the depth map, whether an initial insertion path from the manipulator assembly to the initial target location is blocked.

10. The medical system according to claim 9, wherein: Determining whether the initial insertion path is blocked includes modeling the initial insertion path as a cylinder and testing an intersection of the cylinder with the depth map.

11. The medical system according to claim 9, wherein: The modified target position is at or near the intersection between the initial insertion path and the depth map.

12. The medical system according to claim 7, wherein: Modifying the initial target position includes: determining a trajectory between the center of rotation of the manipulator assembly and a center point in the current endoscope field of view, determining a modified target position at an insertion depth where the trajectory intersects the depth map, and A pose of the manipulator assembly is determined, the pose having an insertion axis intersecting the modified target position and based on the insertion depth.

13. The medical system of claim 7, further comprising: Image data of the first tool in the field of view is recorded, wherein the initial target position of the tool tip of the second tool is also based on the image data of the first tool.

14. The medical system of claim 7, further comprising determining a configuration of an end effector of the second tool based on the depth map.

15. The medical system of claim 1, wherein: The kinematic information includes a calibrated remote center or in-and-out axis of the manipulator assembly.

16. The medical system of claim 1 , further comprising: A display system is configured to display a field of view of the endoscopic imaging instrument.

17. The medical system according to claim 16, wherein: The processing unit is further configured to: A visual guide for performing a guided tool change operation is displayed on the display system.

18. The medical system according to claim 17, wherein: The visual guide includes a graphical indicator for indicating a target position of the second tool.

19. The medical system according to claim 18, wherein: The target location is occluded in the displayed field of view.

20. The medical system of claim 18, wherein: The target position is adjusted based on structures in the field of view.

21. The medical system of claim 17, wherein: The visual guide includes a graphical depiction of an insertion path displayed over an image of the field of view.

22. The medical system of claim 17, wherein: Displaying the visual guide includes simultaneously displaying a first marker corresponding to the initial target position and a second marker corresponding to the modified target position.

23. The medical system of claim 1, wherein: The processing unit is further configured to provide tactile guidance to an operator control device, wherein the manipulator assembly is responsive to movement of the operator control device.

24. A method comprising: determining kinematic information associated with a first tool inserted into a work site, wherein the first tool is coupled to a manipulator assembly; receiving image data generated by an endoscopic imaging instrument having a field of view, the image data generated by the endoscopic imaging instrument after the first tool is removed from the manipulator assembly; and One or more guided tool change parameters are determined for guiding a second tool into the work site, wherein the second tool is received in connection with the manipulator assembly after the first tool is removed, and wherein the one or more guided tool change parameters are based on the kinematic information associated with the first tool coupled to the manipulator assembly and based on a depth map determined from the image data.

25. The method according to claim 24, wherein The one or more guided tool change parameters include a target position of a tool tip of the second tool.

26. The method according to claim 25, wherein The one or more guided tool change parameters include an insertion path of the second tool through an access port to the target location.

27. The method according to claim 26, wherein The one or more guided tool change parameters include an insertion depth limit for the second tool along the insertion path.

28. The method according to claim 26, wherein The one or more guided tool change parameters include a configuration of the manipulator assembly that guides the second tool along the insertion path.

29. The method according to claim 24, wherein The one or more guided tool change parameters include a position of a remote center of motion of a manipulator arm of the manipulator assembly.

30. The method of claim 24, wherein: Determining one or more guided tool change parameters includes determining an initial target position for a tool tip of the second tool based on the kinematic information associated with the first tool, and modifying the initial target position based on the depth map.

31. The method according to claim 30, wherein Modifying the initial target position includes determining a distance between the initial target position and an anatomical surface based on the depth map.

32. The method according to claim 30, wherein Modifying the initial target location includes determining, based on the depth map, whether an initial insertion path from the manipulator assembly to the initial target location is blocked.

33. The method according to claim 32, wherein Determining whether the initial insertion path is blocked includes modeling the initial insertion path as a cylinder and testing an intersection of the cylinder with the depth map.

34. The method of claim 32, wherein: The modified target position is at or near the intersection between the initial insertion path and the depth map.

35. The method of claim 30, wherein: Modifying the initial target position includes: determining a trajectory between the center of rotation of the manipulator assembly and a center point in the current endoscope field of view, determining a modified target position at an insertion depth where the trajectory intersects the depth map, and A pose of the manipulator assembly is determined, the pose having an insertion axis intersecting the modified target position and based on the insertion depth.

36. The method of claim 30, further comprising: Image data of the first tool in the field of view is recorded, wherein the initial target position of the tool tip of the second tool is also based on the image data of the first tool.

37. The method of claim 30, further comprising determining a configuration of an end effector of the second tool based on the depth map.

38. The method of claim 24, further comprising: The field of view of the endoscopic imaging instrument and a visual guide for performing a guided tool change operation are displayed on a display system.

39. The method according to claim 38, wherein The visual guide includes a graphical indicator for indicating a target position of the second tool.

40. The method of claim 39, wherein The target location is occluded in the displayed field of view.

41. The method of claim 39, wherein: The target position is adjusted based on structures in the field of view.

42. The method of claim 38, wherein The visual guide includes a graphical depiction of an insertion path displayed over an image of the field of view.

43. The method of claim 38, wherein Displaying the visual guide includes simultaneously displaying a first marker corresponding to the initial target position and a second marker corresponding to the modified target position.

44. The method of claim 24, further comprising: A tactile guidance is provided to an operator control device, wherein the manipulator assembly is responsive to movement of the operator control device.

45. A non-transitory machine-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: Kinematic information associated with a first tool inserted into a work site is determined, wherein the first tool being coupled to a manipulator assembly; receiving image data generated by an endoscopic imaging instrument having a field of view, the image data generated by the endoscopic imaging instrument after the first tool is removed from the manipulator assembly; as well as One or more guided tool change parameters are determined for guiding a second tool into the work site, wherein the second tool is received in connection with the manipulator assembly after the first tool is removed, and wherein the one or more guided tool change parameters are based on the kinematic information associated with the first tool coupled to the manipulator assembly and based on a depth map determined from the image data.

46. The non-transitory machine-readable medium of claim 45, wherein: The one or more guided tool change parameters include a target position of a tool tip of the second tool.

47. The non-transitory machine-readable medium of claim 46, wherein: The one or more guided tool change parameters include an insertion path of the second tool through an access port to the target location.

48. The non-transitory machine-readable medium of claim 47, wherein: The one or more guided tool change parameters include an insertion depth limit for the second tool along the insertion path.

49. The non-transitory machine-readable medium of claim 47, wherein: The one or more guided tool change parameters include a configuration of the manipulator assembly that guides the second tool along the insertion path.

50. The non-transitory machine-readable medium of claim 45, wherein: The one or more guided tool change parameters include a position of a remote center of motion of a manipulator arm of the manipulator assembly.

51. The non-transitory machine-readable medium of claim 45, wherein: Determining one or more guided tool change parameters includes determining an initial target position for a tool tip of the second tool based on the kinematic information associated with the first tool, and modifying the initial target position based on the depth map.

52. The non-transitory machine-readable medium of claim 51, wherein: Modifying the initial target position includes determining a distance between the initial target position and an anatomical surface based on the depth map.

53. The non-transitory machine-readable medium of claim 51 , wherein: Modifying the initial target location includes determining, based on the depth map, whether an initial insertion path from the manipulator assembly to the initial target location is blocked.

54. The non-transitory machine-readable medium of claim 53, wherein: Determining whether the initial insertion path is blocked includes modeling the initial insertion path as a cylinder and testing an intersection of the cylinder with the depth map.

55. The non-transitory machine-readable medium of claim 53, wherein: The modified target position is at or near the intersection between the initial insertion path and the depth map.

56. The non-transitory machine-readable medium of claim 51, wherein: Modifying the initial target position includes: determining a trajectory between the center of rotation of the manipulator assembly and a center point in the current endoscope field of view, determining a modified target position at an insertion depth where the trajectory intersects the depth map, and A pose of the manipulator assembly is determined, the pose having an insertion axis intersecting the modified target position and based on the insertion depth.

57. The non-transitory machine-readable medium of claim 51 , storing instructions that, when executed by one or more processors, further cause the one or more processors to: Recording image data of the first tool in the field of view, wherein: The initial target position of the tool tip of the second tool is also based on the image data of the first tool.

58. The non-transitory machine-readable medium of claim 51 , storing instructions that, when executed by one or more processors, further cause the one or more processors to: A configuration of an end effector of the second tool is determined based on the depth map.

59. The non-transitory machine-readable medium of claim 45, storing instructions that, when executed by one or more processors, further cause the one or more processors to: The field of view of the endoscopic imaging instrument and a visual guide for performing a guided tool change operation are displayed on a display system.

60. The non-transitory machine-readable medium of claim 59, wherein: The visual guide includes a graphical indicator for indicating a target position of the second tool.

61. The non-transitory machine-readable medium of claim 60, wherein: The target location is occluded in the displayed field of view.

62. The non-transitory machine-readable medium of claim 60, wherein: The target position is adjusted based on structures in the field of view.

63. The non-transitory machine-readable medium of claim 59, wherein: The visual guide includes a graphical depiction of an insertion path displayed over an image of the field of view.

64. The non-transitory machine-readable medium of claim 59, wherein: Displaying the visual guide includes simultaneously displaying a first marker corresponding to the initial target position and a second marker corresponding to the modified target position.

65. The non-transitory machine-readable medium of claim 45, storing instructions that, when executed by one or more processors, further cause the one or more processors to: Providing tactile guidance to an operator control device, wherein The manipulator assembly is responsive to movement of the operator controlled device.

Images