Directional filter and / or translational constraints for coordinate registration

Abstract

A system receives a three-dimensional representation, in imaging coordinates, of an anatomical structure and a portion (optionally including a tip) of a flexible elongate device disposed in the anatomical structure. The system also receives, from a sensing unit, shape data, in sensing coordinates, indicative of a shape of the flexible elongate device, the shape data optionally including a position of the tip. Furthermore, the system computes, based at least in part on the received three-dimensional representation, one or more curve segments indicative of the shape of the flexible elongate device in imaging coordinates. To register the imaging and sensing coordinate systems, the system computes a transformation between the imaging coordinate system and the sensing coordinate system based at least in part on the shape data and the curve segments.

Description

DIRECTIONAL FILTER AND / OR TRANSLATIONAL CONSTRAINTS FOR COORDINATE REGISTRATIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the filing date of provisional U.S. Patent Application No. 63 / 552,083 entitled “DIRECTIONAL FILTER AND / OR TRANSLATIONAL CONSTRAINTS FOR COORDINATE REGISTRATION,” filed on February 9, 2024. The entire contents of the provisional application are hereby expressly incorporated herein by reference.FIELD

[0002] The present disclosure relates to planning and / or navigating minimally invasive medical procedures and, more specifically, to improving registration of imaging and sensing coordinate systems.BACKGROUND

[0003] Minimally invasive medical techniques are intended to reduce the amount of tissue that is damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques may be performed through natural orifices in a patient anatomy or through one or more surgical incisions. Through these natural orifices or incisions, physicians may insert minimally invasive medical instruments (including surgical, diagnostic, and / or therapeutic instruments) to reach a target tissue location. One such minimally invasive technique is to use a flexible and / or steerable elongate device, such as a flexible catheter, that can be inserted into anatomical passageways and navigated toward a region of interest (ROI) within the patient anatomy.

[0004] A combination of one or more localization sensors disposed at a flexible elongate device and intraoperative imaging can greatly aid in planning and navigating a minimally invasive procedure. Specifically, combining sensor data with intraoperative images can enable accurate determination of a position, orientation, and / or pose of the flexible elongate device within the patent anatomy. However, accurately registering sensor data with imaging data in space and time remains a challenge using current techniques.SUMMARY OF THE DISCLOSURE

[0005] The following presents a simplified summary of various examples described herein and is not intended to identify key or critical elements or to delineate the scope of the claims.

[0006] In some examples, a tangible, non-transitory, computer readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to obtain, based on image data received from an imaging system, a three-dimensional representation of an anatomical structure and a portion of a flexible elongate device disposed in the anatomical structure, the portion including a tip, wherein the three- dimensional representation is in an imaging coordinate system associated with the imaging system. The instructions further cause the one or more processors to receive, from a sensing unit, shape data indicative of a shape of the flexible elongate device including a position of the tip, wherein the shape data is in a sensing coordinate system associated with the sensing unit. The instructions further cause the one or more processors to obtain a position of the tip in the imaging coordinate system and compute a translational transformation between the imaging coordinate system and the sensing coordinate system based on the position of the tip in the sensing coordinate system and the position of the tip in the imaging coordinate system. The instructions further cause the one or more processors to compute, based at least in part on the obtained three-dimensional representation, one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system and compute a rotational transformation between the imaging coordinate system and the sensing coordinate system based at least in part on the shape data and the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system. Still further, the instructions cause the one or more processors to register the imaging coordinate system with the sensing coordinate system based on the translational transformation and the rotational transformations.

[0007] In some examples, a system comprises one or more processors and a tangible, non-transitory, computer readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to obtain, based on image data received from an imaging system, a three-dimensional representation of an anatomical structure and a portion of a flexible elongate device disposed in the anatomical structure, the portion including a tip, wherein the three-dimensional representation is in an imaging coordinate system associated with the imaging system. The instructions further cause the one or more processors to receive, from a sensing unit, shape data indicative of a shape of the flexible elongate device including a position of the tip, wherein the shape data is in a sensing coordinate system associated with the sensing unit. The instructions further cause the one or more processors to obtain a position of the tip in the imaging coordinate system and compute a translational transformation between the imaging coordinate system and the sensing coordinate system based on the position of the tip in the sensing coordinate system and the position of the tip in the imaging coordinate system. The instructions further cause the one or more processors to compute, based at least in part on the obtained three- dimensional representation, one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system and compute a rotationaltransformation between the imaging coordinate system and the sensing coordinate system based at least in part on the shape data and the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system. Still further, the instructions cause the one or more processors register the imaging coordinate system with the sensing coordinate system based on the translational transformation and the rotational transformations.

[0008] In some examples, a method comprises obtaining, by one or more processors and based on image data received from an imaging system, a three-dimensional representation of an anatomical structure and a portion of a flexible elongate device disposed in the anatomical structure, the portion including a tip, wherein the three-dimensional representation is in an imaging coordinate system associated with the imaging system. The method further comprises receiving, from a sensing unit, shape data indicative of a shape of the flexible elongate device including a position of the tip, wherein the shape data is in a sensing coordinate system associated with the sensing unit. The method further comprises obtaining a position of the tip in the imaging coordinate system and computing a translational transformation between the imaging coordinate system and the sensing coordinate system based on the position of the tip in the sensing coordinate system and the position of the tip in the imaging coordinate system. The method further comprises computing, based at least in part on the obtained three-dimensional representation, one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system and computing a rotational transformation between the imaging coordinate system and the sensing coordinate system based at least in part on the shape data and the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system. Still further, the method comprises registering, by the one or more processors, the imaging coordinate system with the sensing coordinate system based on the translational transformation and the rotational transformations.

[0009] In some examples, a tangible, non-transitory, computer readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to receive, from a sensing unit, shape data indicative of a shape of the flexible elongate device, wherein the shape data is in a sensing coordinate system associated with the sensing unit. The instructions further cause the one or more processors to obtain, based on image data received from an imaging system, a three-dimensional representation of an anatomical structure and a portion of a flexible elongate device disposed in the anatomical structure, wherein the three-dimensional representation is in an imaging coordinate system associated with the imaging system. The instructions further cause the one or more processors to compute, based at least in part on the obtained three-dimensionalrepresentation, one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system and compute a transformation between the imaging coordinate system and the sensing coordinate system based at least in part on the shape data and the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system. Computing the one or more curve segments includes: computing a first curve segment and a candidate second curve segment, computing a first direction vector representative of direction of the first curve segment at a terminal point of the first curve segment, and computing a filter region based at least in part on a maximum angular deviation from the first direction vector. Computing the one or more curve segments further includes: identifying a proximal point of the candidate second curve segment, determining that the proximal point of the candidate second curve segment is within the filter region, and designating the candidate second curve segment as an accepted candidate second curve segment based at least in part on determining that the proximal point of the candidate second curve segment lies within the filter region. Still further, the instructions cause the one or more processors to register the imaging coordinate system with the sensing coordinate system based on the transformation.

[0010] In some examples, a system comprises one or more processors and a tangible, non-transitory, computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to receive, from a sensing unit, shape data indicative of a shape of the flexible elongate device, wherein the shape data is in a sensing coordinate system associated with the sensing unit. The instructions further cause the one or more processors to obtain, based on image data received from an imaging system, a three-dimensional representation of an anatomical structure and a portion of a flexible elongate device disposed in the anatomical structure, wherein the three-dimensional representation is in an imaging coordinate system associated with the imaging system. The instructions further cause the one or more processors to compute, based at least in part on the received three-dimensional representation, one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system and compute a transformation between the imaging coordinate system and the sensing coordinate system based at least in part on the shape data and the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system. Computing the one or more curve segments includes: computing a first curve segment and a candidate second curve segment, computing a first direction vector representative of direction of the first curve segment at a terminal point of the first curve segment, and computing a filter region based at least in part on a maximum angular deviation from the first direction vector. Computing the one or more curve segments further includes: identifying a proximal point ofthe candidate second curve segment, determining that the proximal point of the candidate second curve segment is within the filter region, and designating the candidate second curve segment as an accepted candidate second curve segment based at least in part on determining that the proximal point of the candidate second curve segment lies within the filter region. Still further, the instructions cause the one or more processors to register the imaging coordinate system with the sensing coordinate system based on the transformation.

[0011] Still in some examples, a method comprises receiving, by one or more processors and from a sensing unit, shape data indicative of a shape of the flexible elongate device, wherein the shape data is in a sensing coordinate system associated with the sensing unit. The method further comprises obtaining, by the one or more processors and based on image data received from an imaging system, a three-dimensional representation of an anatomical structure and a portion of a flexible elongate device disposed in the anatomical structure, wherein the three-dimensional representation is in an imaging coordinate system associated with the imaging system. The method further comprises computing, by the one or more processors and based at least in part on the received three-dimensional representation, one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system, and computing a transformation between the imaging coordinate system and the sensing coordinate system based at least in part on the shape data and the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system. Computing the one or more curve segments includes: computing a first curve segment and a candidate second curve segment, computing a first direction vector representative of direction of the first curve segment at a terminal point of the first curve segment, and computing a filter region based at least in part on a maximum angular deviation from the first direction vector. Computing the one or more curve segments further includes: identifying a proximal point of the candidate second curve segment, determining that the proximal point of the candidate second curve segment is within the filter region, and designating the candidate second curve segment as an accepted candidate second curve segment based at least in part on determining that the proximal point of the candidate second curve segment lies within the filter region. Still further, the method comprises registering, by the one or more processors, the imaging coordinate system with the sensing coordinate system based on the transformation.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG. 1 A depicts an example system for navigating during a medical procedure within an operating environment.

[0013] FIG. 1 B is a simplified diagram of a flexible elongate device disposed within an anatomical structure.

[0014] FIG. 2 is a schematic representation of an example operating environment including a robotic-assisted system for controlling a flexible elongate and an intraoperative imaging system.

[0015] FIG. 3 is a schematic representation of an example timeline for obtaining intraoperative images and / or shape sensor data.

[0016] FIG. 4A depicts an example flexible elongate device representation extracted from imaging data in a joint coordinate system with shape sensor data prior to registration.

[0017] FIG. 4B depicts a rigid body in two example coordinate systems.

[0018] FIG. 4C schematically illustrates an example coordinate transformation process.

[0019] FIGS. 5A, B schematically illustrate example sources of error in coordinate registration.

[0020] FIGS. 5C, D schematically illustrate an example of using a translational constraint for improving coordinate registration accuracy.

[0021] FIG. 6A schematically illustrates an example image of a flexible elongate device disposed within an anatomical region.

[0022] FIG. 6B schematically illustrates example candidate segments of a flexible elongate device extracted from the image in FIG. 6A.

[0023] FIG. 6C schematically illustrates representing one of the segments in FIG. 6B by a centerline curve segment.

[0024] FIG. 6D schematically illustrates example techniques for selecting between alternative candidate centerline curve segments based on candidate segments of a flexible elongate device of FIG. 6B.

[0025] FIG. 7 is a simplified diagram of a medical system according to some examples.

[0026] FIG. 8A is a simplified diagram of a medical instrument system according to some examples.

[0027] FIG. 8B is a simplified diagram of a medical instrument including a medical tool within an elongate device according to some examples.

[0028] FIGS. 9A and 9B are simplified diagrams of side views of a patient coordinate space including a medical instrument mounted on an insertion assembly according to some examples.

[0029] FIG. 10 depicts a flow chart for an example method for visualizing a flexible elongate device disposed within patient anatomy during a medical procedure.

[0030] FIGS. 11 A and B depict flow charts for another example method for visualizing a flexible elongate device disposed within patient anatomy during a medical procedure.

[0031] Examples of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, wherein showings therein are for purposes of illustrating examples of the present disclosure and not for purposes of limiting the same.DETAILED DESCRIPTION

[0032] In the following description, specific details are set forth describing some examples consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the examples. It will be apparent to one skilled in the art that some examples may be practiced without some or all of these specific details. The specific examples disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one example may be incorporated into other examples unless specifically described otherwise or if the one or more features would make an example non-functional. In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the examples.

[0033] This disclosure describes various instruments and portions of instruments in terms of their state in three-dimensional (3D) space. As used herein, the term “position” refers to the location of an object or a portion of an object in a 3D space (e.g., three degrees of translational freedom along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to the rotational placement of an object or a portion of an object (e.g., one or more degrees of rotational freedom such as roll, pitch, and yaw). As used herein, the term “pose” refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of the object in at least one degree of rotational freedom (e.g., up to six total degrees of freedom). As used herein, the term “shape” refers to a set of poses, positions, and / or orientations measured along an object. As used herein, the term “distal” refers to a position that is closer to a procedural site and the term “proximal” refers to a position that is further from the procedural site.Accordingly, the distal portion or distal end of an instrument is closer to a procedural sitethan a proximal portion or proximal end of the instrument when the instrument is being used as designed to perform a procedure.

[0034] The disclosure generally relates to systems and methods that facilitate user (e.g., physician) planning of, and / or user navigation during, a minimally invasive (e.g., endoluminal) medical procedure. The medical procedure may include operating, using a robotic-assisted system, a flexible elongate device (or another suitable instrument) disposed within patient anatomy. A sensor system may sense the shape of the flexible elongate device at different times (e.g., with a set periodicity, on demand, etc.) in a sensing coordinate system. The sensing coordinate system may be referred to as an instrument coordinate system or a robotic coordinate system. An intraoperative imaging system may image a portion of patient anatomy and a portion of the flexible elongate device disposed therein to generate imaging data referenced to an imaging coordinate system. The systems and methods described in the disclosure facilitate registration between the imaging coordinate system and the sensing coordinate system. The registration techniques described in the disclosure can improve the medical procedure accuracy and / or speed by facilitating at least in part translating a target location from intraoperative images into the sensing coordinate system. Additionally or alternatively, target registration may include translating position, orientation, and / or pose of at least a portion of the flexible elongate device (e.g., the tip, a length proximal to the tip, etc.) into imaging coordinates within which the target location is estimated.

[0035] The registration of the sensing coordinate system to the imaging coordinate system entails computing a 6-DoF transformation between the two coordinate systems, based at least in part on an overlap region between the shape data and one or more centerline curve segments that represent the flexible elongate device and are computed from the imaging data. The registration can be based on optimization in 6-DoF, using, for example, an iterative closest point (ICP) algorithm. The optimizations in 6-DoF, however, may settle on a local minimum, particularly when the shape of the flexible elongate device has a nearly constant curvature (e.g., straight or circular shape) in the overlap region. That is, the centerline curves transformed into the sensing coordinate system may substantially overlap the shape data, but “slide along” the constant curvature (e.g., as discussed with reference to FIG. 5B), leading to an error (which may be referred to as “sliding error”) in position of a tip of the flexible elongate device.

[0036] The systems and methods discussed herein reduce the sliding error by separating computation of the 6-DoF transformation into two parts. One part includes computing a translational transformation between tip position in the imaging data and tip position in shape data. In some examples, the tip position in the imaging data is obtained from an operator viaa graphical user interface. Another part includes computing a rotational transformation between the shape data and the centerline curves after registering the tip position between the two coordinate systems. The rotational transformation may be computed using optimization over the three-dimensional rotational angle space to minimize a cost function indicative of quality of overlap (and, thus, registration quality). In some examples, the cost function may place higher weight to aligning points closer to the tip than to aligning points farther from the tip. Furthermore, an additional 6-DoF transformation may be computed to fine-tune registration using constrained optimization.

[0037] An example coordinate registration problem is discussed below with reference to FIG. 4A. A general discussion of coordinate transformations is below with reference to FIGS. 4b and C. The sliding error and an example technique to reduce it by separating translational and rotational transformations are further discussed below, e.g., with reference to FIG. 5A-D.

[0038] In some examples, a graphical user interface on a display device may render the flexible elongate device from the shape data transformed onto the imaging coordinate system such that the transformed shape data is overlaid over the image of the flexible elongate device. An operator may then analyze the image to provide an indication of registration quality. If the registration quality is poor, the graphical user interface may prompt the operator to correct the indication of tip position imaging data (such as by providing a new indication of the position of the tip in the imaging data).

[0039] In some examples, the systems and methods discussed herein include synchronizing shape data and imaging data by selecting shape data (e.g., from a time sequence of shape data) corresponding to an imaging time at which the imaging data was captured. To that end, shape data corresponding to a time sequence may be stored in a shape data buffer. At least a subset of the shape data buffer may be selected for processing based on an indication of imaging time. The indication of imaging time may be obtained from the imaging system or from an operator. Shape data may be selected from the shape data buffer based at least in part on finding the time when the shape data remains substantially static, which may be indicative of a subject holding their breath during the imaging time. Synchronization of shape data and imaging data is discussed in more detail below, e.g., with reference to FIGS. 2 and 3.

[0040] Additionally, the present disclosure outlines a method for selecting a set of centerline curve segments from a plurality of candidate centerline curve segments. Using a plurality of centerline curve segments may result in a higher quality registration than using only the centerline curve segment including the tip. However, including a candidatecenterline curve segment that does not belong to the flexible elongate device may damage the quality of registration. Candidate centerline curve segments to add to a selected centerline curve segment may be rejected when they do not fall within a filter region determined by the geometry of the shape of the selected centerline segment. More specifically, a reference direction vector tangent to the selected centerline curve segment at the segment’s terminal point may define a filter region. For example, the filter region may encompass a region defined by a set of directions within a range of the reference direction vector. Proximal points (e.g., nearest to the terminal point of the selected centerline curve segment) of respective candidate centerline curve segments may determine whether a given candidate centerline curve segment is accepted. Aggregation of candidate centerline curve segments into the one or more centerline curve segments that represent the flexible elongate device in imaging coordinates is discussed in more detail with below, e.g., with reference to FIGS. 6A-D.

[0041] FIG. 1 A depicts an example system 100 for navigating during a medical procedure within an operating environment 101. The system 100 may obtain images from a portion of the operating environment 101 disposed within a field of view F (approximately demarcated by dashed lines) of an imaging unit 110. To that end, the system 100 may be in communicative connection with the imaging unit 110. Additionally, the system 100 may obtain data from one or more sensors disposed within the operating environment. To that end, the system 100 may be in communicative connection with the one or more sensors via a communicative connection with a sensing unit 120.

[0042] The system 100 includes a processing unit 125 and a display unit 130 in communicative connection with each other. Although in FIG. 1 A the imaging unit 110 and the sensing unit 120 are depicted as being distinct from the system 100, in other examples, the system 100 may include the imaging unit 110 and / or the sensing unit 120. In any case, one or more processors of the processing unit 125 of the system 100 may be configured to receive images and / or processed image information from the imaging unit 110 and to receive data from the one or more sensors by way of the sensing unit 120.

[0043] Throughout the disclosure, the descriptions of example operations performed by the processing unit 125 below are to be understood to be executed by the one or more processors of the processing unit 125. In some examples, the one or more processors may include hardware specifically configured (e.g., hardwired or programmable) to carry out at least a portion of the example operations described in this disclosure. Additionally or alternatively, the one or more processors may be configured to carry out at least a portion of the example operations described in this disclosure by carrying out a set of software instructions. To that end, the system 100 may include or be communicatively connected to atangible, non-transitory, computer readable medium. The medium may store instructions which, when executed by the processing unit 125, perform any one or more of the example operations described below. For example, the instructions may cause the processing unit 125 to perform image processing operations on the images received from the imaging unit 110 and / or to perform computations (e.g., for coordinate registration) based on the data received by way of the sensing unit 120. Furthermore, the instructions may cause the processing unit 125 to cause the display unit 130 to display, via a GUI, information based on the processing of images received from the imaging unit 110 and the processing of data received by way of the sensing unit 120. For example, the processing unit 125 may send the information, or send data representing the entire GUI including the information, to the display unit 130.

[0044] An operator (e.g., a physician, another medical practitioner, or a fully-automated robotic surgery system) of a medical system may use the information displayed at the display unit 130 to perform a medical procedure (e.g., endoscopy, biopsy, pharmacological treatment, and / or treatment, such as ablation). During a medical procedure, the operator may control a flexible elongate device 140 inserted through an orifice O (or through a suitable surgical incision) into an anatomical structure A of a patient P disposed at a table T. For example, the medical procedure may include navigating the flexible elongate device 140 (indicated with solid lines outside and dashed lines inside the patient P) toward a region of interest, or ROI, R within the anatomical structure A with the aid of information displayed at the display unit 130. The ROI R, for example, may be a designated procedure site for visual examination, biopsy, treatment, or any other medical procedure. Throughout the disclosure, the ROI R may be referred to as a region or a target.

[0045] One or more fiducial may be disposed at (e.g., physically contacting, integrated within, fixedly attached to, or removably attached to in a manner that, during operation / use, forms a rigid relationship with) the flexible elongate device 140. The fiducials are configured to be visible in images obtained by the imaging unit 110 to thereby enhance visibility of the flexible elongate device 140 and / or to aid in identifying certain points along the device 140. To that end the fiducials may include elements of a variety of materials and / or structures such as metals, plastics, etched glass, dyes, radioactive or fluorescent markings, confined fluids (e.g., bubbles), etc. At least some of the elements of the fiducial 142 may be integrated (e.g., etched, deposited, painted, or otherwise fixedly attached) onto the flexible elongate device 140. Additionally or alternatively, the fiducials may include elements removably disposed at the flexible elongate device 140. For example, the fiducials may be integrated onto a removable structure, such as a sleeve or a stylet, which in turn may be removably attached to the flexible elongate device 140.

[0046] One or more sensors may also be disposed at the flexible elongate device. The sensors may be mechanical sensors, optical sensors, electromagnetic (EM) sensors, or any other suitable sensors. The sensors may be integrated into the flexible elongate device 140, or removably attached to the flexible elongate device 140. The sensors may be configured to communicate with the sensing unit 120.

[0047] In some examples, the sensors are active sensors, configured to transmit electromagnetic (e.g., optical, radio, low-frequency magnetic) or radioactive radiation. The sensing unit 120 may include components to receive the radiation emitted by the sensors and triangulate or trilaterate locations of the sensors in the sensor system coordinates. The sensor system coordinates may be the coordinates of the robotic-assisted system configured to manipulate, control, or guide the flexible elongate device 140. As such, the sensor system coordinates may be referred to as instrument coordinates.

[0048] In other examples, the sensors are passive and emit no radiation. The passive sensors may sense radiation emitted by components of the sensing unit 120 disposed within the operating environment 101. For example, in an EM sensing system, one or more electrified coils may be disposed within the operating environment 101 to generate static or dynamic magnetic fields. The sensors may be configured to pick up variations in the dynamic magnetic fields or, as the sensors move, variations in the sensed static or quasistatic magnetic field and convert the sensed variations into electrical currents that are received by the sensing unit 120. In turn, the sensing unit 120 may compute indications of positions of the sensors from the received electrical currents.

[0049] The sensors may be fiberoptic sensors disposed along the length of the flexible elongate device. The fiberoptic sensors may include Bragg gratings and / or materials to enhance non-linear scattering. The fiberoptic sensors may be configured to change spectral reflectivity based on material strain. Such sensors may scatter light emitted, for example, by the sensing unit 120 in a manner that indicates locations and degrees of bends in the flexible elongate device 140.

[0050] In some examples, sensors may include accelerometers, gyroscopes, and / or magnetometers. Furthermore, the point sensors may include inertial measurement units (IMUs) that combine multiple sensors (e.g., accelerometers, gyroscopes) and / or inertial and magnetic measurement units (IMMUs) that combine multiple sensors (e.g., accelerometers, gyroscopes, magnetometers). IMUs and / or IMMUs may generate signals indicative of orientation of the flexible elongate device at a given point with respect to gravitational field and / or magnetic field (e.g., of Earth). In some examples, additional magnetic fields may be introduced to the sensor environment. Additionally or alternatively, IMUs and / or IMMUs maygenerate signals indicative of motion of the flexible elongate device (e.g., caused by motion of an anatomical structure due to breathing and / or other factors, and / or independent motion of the flexible elongate device within the anatomical structure). In some examples, the sensing unit 120 may combine indications of orientation (e.g., up to three degrees of freedom) from IMlls with indications of position from other (e.g., EM) sensors to generate more complete data indicative of pose of the flexible elongate device. Furthermore, in some examples, the sensing unit 120 may combine data from sensors in multiple sensor coordinate systems. The sensing unit 120 and / or the processing unit 125 may register multiple sensor coordinate systems with each other.

[0051] Generally, the system 100 may use any suitable combinations of the sensors described above to obtain shape data indicative of shape of the flexible elongate device 140 (e.g., pose, including location and orientation, at different points along the length of the flexible elongate device 140).

[0052] FIG. 1 B is a simplified diagram of the flexible elongate device 140 disposed within the anatomical structure A. FIG. 1 B is included to give an expanded and more detailed view of a portion of the operating environment 101 disposed within the field of view F. The anatomical structure A may be a lung of the patient P. The flexible elongate device 140 may be inserted into and navigated by the operator toward the region R (e.g., target of the medical procedure), for example, for the purpose of investigating or treating a pathology in the region R. The techniques described in the present disclosure can facilitate the navigation process by generating and displaying timely and accurate imaging and sensing of the flexible elongate device 140 and imaging of the anatomical structure A. These techniques incorporate location, pose, and / or shape data in one coordinate system obtained using the sensors and location, pose, and / or shape data obtained using imaging in another coordinate system. The combined data may improve speed, accuracy, reliability and / or safety of a medical procedure. For example, the processing unit 125 may combine the data from the imaging unit 110 and / or the sensing unit 120 to determine the position, orientation, and / or pose of at least a portion of the flexible elongate device 140 within reference to the anatomical structure A of the patient P and, particularly, the ROI or target R. The techniques of this disclosure can improve speed and / or accuracy of the determination.

[0053] Furthermore, the processing unit 125 may generate a graphical user interface (GUI) or update GUI data for display on the display device 130 to aid an operator with the medical procedure. In some examples, the processing unit 125 may generate data and / or control signals for a control unit of a robotic system configured to manipulate and / or navigate the flexible elongate device 140. Additionally or alternatively, the processing unit 125 may be configured to generate, based on the combined imaging and sensing data, one or morealerts. The alerts may include, for example, an alert indicating proximity to the region R, an alert indicating a potential navigation error, and alert indicating that confidence in location of the tip of the flexible elongate device 140 fell below a threshold level, etc. It should be noted that combining intra-operative data from the imaging unit 110 and the sensing unit 120 entails registering two coordinate systems with respect to each other, e.g., as described with reference to FIGS. 4A-C and 5A-D below.

[0054] Generally, a suitably configured imaging unit (e.g., imaging unit 110) may generate the intra-operative imaging data. The intra-operative imaging data may include, for example, computed tomography (CT), particularly cone-bean computed tomography (CBCT) data. To that end, the imaging unit may include a C-arm CBCT imaging system, and / or magnetic- resonance imaging (MRI) data. Additionally or alternatively, the intraoperative imaging data may be obtained using thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging or any other suitable imaging technique. Intra-operative imaging data may include fluoroscopy X-ray data (e.g., generated by a C-arm X-ray device), and, particularly, tomosynthesis data (reconstructed from 2D X-ray images, such as fluoroscopy images, into 3D volume).

[0055] FIGS. 2 is a schematic representation of an example operating environment 201 (which may be operating environment 101 ) including a robotic system 205 (which may be system 100) for performing minimally invasive operating procedures. The operating environment 201 includes an X-ray imaging system 210 (which may be the imaging unit 110). The X-ray imaging system 210 can be referred to as the imaging system 210. The imaging system 210 includes a motion actuation unit 211 in mechanical connection with a C- arm 212. The C-arm 212 is in mechanical connection with an X-ray source 213 and an X- ray imaging detector 214. Additionally, the imaging system 210 includes an operator interface unit 215, a communication interface 216, and a processing unit 218. The processing unit 218 is in communicative connection with the motion actuation unit 211 , the X-ray source 213, the X-ray imaging detector 214, the operator interface unit 215, and the communication interface 216.

[0056] The robotic system 205 includes a display unit 230, a manipulator assembly 234, a communication interface 236, and a processing unit 238 (which may be the processing unit 125). The manipulator assembly 234 is mechanically connected to and configured to manipulate a flexible elongate device 240 (which may be the flexible elongate device 140). The robotic system 205 includes a sensing unit 235 (which may be the sensing unit 120) disposed at the manipulator assembly 234. The processing unit 238 is in communicative connection with the display unit 230, the manipulator assembly 234, the sensing unit 235, and the communication interface 236.

[0057] In some examples (not illustrated in FIG. 2), a sensing unit (e.g., sensing unit 120) included in the robotic system 205 may be detached from the manipulator assembly 234. Still in other examples, a sensing unit (e.g., sensing unit 120) may be separate from the robotic system 205 but in communicative connection with the processing unit 238.

[0058] In operation of the robotic system 205, the sensing unit (e.g., the sensing unit 235 disposed at the manipulator assembly 234) is configured to sense the shape of the flexible elongate device at different points in time (e.g., periodically and / or on demand) and send shape data to the processing unit 238. The processing unit 238 may include buffer memory or, simply, a buffer to store shape data over a limited time interval.

[0059] The operating environment 201 includes, disposed on top of the table T, the patient P within whose anatomical structure A a portion of the flexible elongate device 240 is disposed during the minimally invasive procedure. The operating environment 201 further includes a medical professional M who may operate the robotic system 205 and / or the imaging system 210 to conduct a minimally invasive medical procedure. The medical professional M may be referred to as the operator M of the robotic system 205 and / or imaging system 210.

[0060] In operation of the imaging system 210, the motion actuation unit 211 rotates the C-arm 212 while the X-ray source emits X-rays that pass through the anatomical region A and are detected by the X-ray imaging detector 214, generating projection data at a set of angles. The processing unit 218 reconstructs projection data to generate image data. The image data includes a 3D representation of the anatomical structure A and a portion of the flexible elongate device 240 disposed in the anatomical structure A.

[0061] The processing unit 238 of the robotic system 205 may receive the imaging data from the imaging system 210 by way of a communication link 250 connecting the communication interfaces 216 and 236. The communication link may be wired (e.g., electrical or fiberoptic) or wireless (e.g., radio) and use any suitable standard protocol (e.g., ethernet, WiFi, Bluetooth, etc.) or a custom protocol.

[0062] During the minimally invasive medical procedure, the operator M may use the robotic system 205 to navigate the flexible elongate device 240 towards an ROI (e.g., ROI R) within the anatomical structure A. To aid in navigation, the operator M may, at certain times, engage (e.g., via a GUI the imaging system 210 by initiating an imaging cycle to image the anatomical region A along with the portion of the flexible elongate device 240 disposed within the anatomical region A. At the start of the imaging cycle, the imaging system 210 may conduct a collision check to ensure that no obstacles impede the rotation of the C-arm 212. Subsequently, the imaging system 210 may prepare for a spin, conduct the spin duringwhich projection data is collected, reconstruct the projection data to generate image data in imaging coordinates, and transfer the image data (e.g., to the robotic system 205).

[0063] FIG. 3 is a schematic representation of an example timeline 300 for obtaining intraoperative images. The example timeline 300 may represent the imaging cycle and the transfer of image data in the example operating environment 201 above. The timeline 300 includes a time axis 310 depicting the passage of time, a shape sensor recording process 320 (e.g., performed by the robotic system 205) and steps 331 -335 of obtaining intraoperative images by an imaging system (e.g., the imaging system 210).

[0064] The shape sensor recording process 320 is ongoing, continually generating shape data with a suitable sampling interval of 0.2, 0.5, 1 , 2, 5, 10, 20 sec or any other suitable periodicity. A processing unit (e.g., processing unit 238) of a robotic system (e.g., the robotic system 205) may store at least some of the shape data (e.g., received from the sensing unit 235) in a buffer of a memory device within or communicatively connected to the processing unit. From the shape data stored in the buffer, the processing unit may select a portion that corresponds to imaging data (e.g., time when projection data is acquired during a spin of the C-arm 212). To that end, an operator (e.g., operator M) may generate a starting trigger at time to to mark the starting time from which the shape data may be analyzed to select a portion corresponding to the imaging data. The operator may generate the starting trigger using an interface of the robotic system (e.g., a GUI on the display unit 230) just before initiating an imaging cycle of the imaging system. Additionally or alternatively, the imaging system may generate the starting trigger or an additional trigger and send the generated trigger to the robotic system by way of a suitable communication link (e.g., communication link 250). In some examples, the operator or the imaging system may generate the starting trigger during a collision check step 331 , or a spin prep step 332. The imaging system may start a spin step 333 at time ti and end the spin step at time t2. The time interval between the time ti and the time t2 may be referred to as an imaging time window. The imaging system may continue with a reconstruction step 334 to generate image data from projection data and a transfer step 335 to send the image data to the robotic system. The length of each of the steps 331-335 may be variable and depend on settings of the imaging system. Time ts marks the end of the shape data in the buffer designated for processing to select the portion corresponding to the imaging data. The time ts may be based on an ending trigger generated by the operator at the robotic system or generated and sent to the robotic system by the imaging system. The ending trigger is generated after a reconstruction step 334 during a transfer step 335. In some examples, the ending trigger may be generated during the reconstruction step 334 or after the transfer step 335. In some examples, time ts may be offset by a pre-programmed amount of time from time t0. In any case, the robotic systemmay process shape data stored between times to and ts to select or identify the portion of the shape data corresponding to the imaging data from the imaging time window. To that end, a processing unit of the robotic system may identify a portion of the shape data indicative of a low-motion period. That is, the processing unit may select a sequence of samples of the shape data during which the shape data exhibit limited change. The low-motion period may correspond to the patient P holding their breath during the spin, as, for example, instructed by the operator M.

[0065] FIG. 4A depicts, a joint coordinate system 401 , an example flexible elongate device representation 402 extracted from imaging data and shape data 403 prior to registration. The joint coordinate system 410 may use imaging system coordinates for the representation 402 and sensing coordinates for the shape data 403. Under the condition that a transformation mapping the imaging coordinate system to the sensor coordinate system is isometric, there is an isometric mapping that can overlay the representation 402 onto the shape data 403 and enable coordinate registration between the imaging coordinate system and the sensing coordinate system.

[0066] The shape data 403 may take the form of an ordered set of poses corresponding to points along the flexible elongate device, an ordered set of points in 3D, or another suitable form. For example, the shape data 403 may be represented by a set of splines that allow generating an ordered set of points at any spacing. The representation 402 extracted from the imaging data may be a point cloud a subset of points in a 3D grid, with optional weights at the points, or another suitable representation. The representation 402 may also be a set of parametric polynomial curves and / or splines. The parametric curves or splines may, for example, enable an ordered point form of at least portions of the representation 402.

[0067] The representation 402 may be disjoined or segmented. Regions 404a and b may exist in an image where identifying and segmenting points corresponding to the flexible elongate device may be difficult. For example, computed probabilities of belonging to the representation 402 for points in the regions 404a and b may be below a threshold probability. Ordering segments of the representation 402 is discussed in more detail with reference to FIGS. 6B-D.

[0068] In one example, the representation 402 may be a set of parametric splines tracing a centerline curve in 3D. That is, an ordered set of points (x[n],y[n],z[n]) form of the representation 402 may be generated by sampling spline curves x(s), y(s), and z(s). The domain of the parameter s may be restricted to intervals that define the segments of the representation 402. The shape data 403 may be represented by parametric spline curvesx'(s), y'(s), and z'(s). A mapping by a rigid transformation may exist that maps any point (x(s),y(s),z(s)) onto a corresponding point (x'(s),y'(s),z'(s)) for any s within the domain of s. Such mapping, wich enables coordinate registration, is discussed in more detail with reference to FIGS. 4B and C.

[0069] FIG. 4B depicts a rigid body 412 in two example coordinate systems to illustrate the concept of registration between two coordinate systems. A first coordinate system 414 may correspond to coordinates of shape data obtained or generated, for example, by a sensing unit (e.g., the sensing unit 120 or the sensing unit 235). The sensing unit may be a part of a manipulator assembly (e.g., manipulator assembly 234) configured to actuate and / or manipulate a flexible elongate device (e.g., the flexible elongate device 140 or 240). The first coordinate system 414 may therefore represent the coordinate system of the sensing unit. A second coordinate system 416 may correspond to coordinates of images obtained or generated by an imaging system (e.g., the imaging unit 110 or the imaging system 210). The first coordinate system 414 may be referred to as the sensing coordinate system. The second coordinate system 416 may be referred to as the imaging coordinate system.

[0070] The rigid body 412 may represent a portion (e.g., a segment of length) of the flexible elongate device (e.g., flexible elongate device 140 or 240). Although the flexible elongate device is flexible, a short portion (e.g., an infinitesimal segment) may be approximated as a rigid body. Furthermore, for the purpose of registration between the sensing coordinate system 414 and the imaging coordinate system 416, the flexible elongate device may be considered a rigid body if the coordinate systems 414 and 416 are isometric with each other.

[0071] A state (e.g., position and orientation) of the rigid body 412 in 3D space can be described using the first coordinate system 414 and / or the second coordinate system 416. A rigid body without symmetries (e.g., body 412) has six degrees of freedom (6 DOFs) and its position and orientation can be described with six coordinates. In the first coordinate system 414, the rigid body 412 may have coordinates (x, y, z, 0, <|), a), where x, y and z are position coordinates of a center 418 of the rigid body 412 along the axes of the first coordinate system 414 with respect to origin O. In other examples, the position coordinates may be designated for any point within the rigid body 412, or, in fact, any point in a rigidly defined geometric relationship to the body 412. Coordinates 0, <|) and a can describe orientation of the body 412 with the aid of an orientation vector 419, which, in the example of FIG. 4A, originates in the center 418 and goes through the middle of one of the facets of the rigid body 412. For example, 0 may be an elevation angle with respect to z-axis, <|) may be an azimuthal angle parallel to the xy-plane, and a may be the angle of rotation of the rigid body412 around the orientation vector 419. Alternatively, the three orientation coordinates may be roll, pitch and yaw of the rigid body 412 with respect to any suitable reference direction. Analogously to the coordinates (x, y, z, 0, <|), a) of first coordinate system 414, coordinates (x’, y’, z’, 0’, <()’, o’) of the second coordinate system 416 describe position (x’, y’, z’) with respect to origin O’ and orientation (0’, <|>’, o’) with respect to, for example, z’-axis of the rigid body 412. Registering the first coordinate system 414 with the second coordinate system 416, at least in the vicinity of the rigid body 412, includes finding a mapping (e.g., a transformation, a mathematical relationship, etc.) at least between the coordinates (x, y, z, 0, <t>, a) and the coordinates (x’, y’, z’, 0’, <|>’, o’). Another way of looking at the registration, is that the mapping defines a corresponding point (u, v, w) in the vicinity of position (x, y, z) for any point (u’, v’, w’) in the vicinity of position (x’, y’, z’).

[0072] In the discussion above, it is assumed that the first coordinate system 414 and the second coordinate system 416 have the same scale and the transformation from the first coordinate system 414 to the second coordinate system 416 is rigid. In some examples, the mapping from the first coordinate system 414 to the second coordinate system 416 may include one or more scaling factors for the axes. Thus, the mapping may include three translation variables, three rotation variables, and / or three scaling variables. Furthermore, each of the variables may depend on position and the mapping may include deformations.

[0073] In some examples, each of the coordinate systems 414 and 416 are independently calibrated to have accurate and consistent scaling within a shared operating volume. Coordinate registration process may then be defined in terms of three translation constants and three rotation constants for the shared operating volume. In other examples, gradual variations in scaling within at least one of the coordinate systems 414 and / or 416 may necessitate use of up to three translation variables and up to three rotation variables, each a function of location within the shared operating volume.

[0074] FIG. 4C schematically illustrates an example coordinate transformation process. In one example, a processing unit (e.g., the processing units 125, 238) may obtain or determine a position and an orientation of a rigid body (e.g., rigid body 412) within the first coordinate system 414 as (x, y, z, 0, <|), a) and the second coordinate system 416 as (x’, y’, z’, 0’, <])’, o’). The processing unit may then generate a mapping, M, between the two coordinate systems 414 and 416. The processing unit may be configured to map a new position (u’, v’, w’) within the second coordinate system 416 on to a corresponding position (u, v, w) within the first coordinate system 414. Additionally or alternatively, the processing unit may be configured to map coordinates from the first coordinate system 414 to the second coordinate system 416. The mapping may be only valid in a region around (x, y, z). By collecting rigid body coordinates in the two coordinate systems 414 and 416, theprocessing unit may extend the validity of mapping over any portion of a shared operating MH C CO ' volume of the two coordinate systems 414 and 416. For example, the processing unit mJ ti3 •ay implement the coordinate registration as a linear mapping: 0 sin / ?' 1 0 O ' u'' 1 0 0 cos a — sin a v' +0 cos / 3. .0 sin a cos a . .w'. where a, p and y are rotation parameters (e.g., roll, pitch and yaw), Sn, S22 and S33 are scaling parameters (which may be unity, as discussed above), and di, d2 and ds are displacement factors. Thus, with scaling factors, there may be nine mapping parameters, and six mapping parameters when scaling can be ignored. As discussed above, the linear mapping may be a function of the input position coordinates (u’, v’, w’). For example, the processing unit may store and / or access a lookup table find an entry for mapping parameters corresponding to the input position coordinates. Because the lookup table can only have a limited number of recorded input coordinates (herein, recorded coordinates), the system may use the entry corresponding to the recorded coordinates nearest to the input coordinates. Alternatively, the system may interpolate mapping parameters corresponding to a set of recorded coordinates near the input coordinates. In other examples, the processing unit may store and / or access one or more polynomial, spline, or another suitable fit function relating input coordinates to the mapping parameters. The mapping may relate the shape data 403 for the flexible elongate device obtained from a sensing system and the flexible elongate device representation 402 in the imaging coordinate system as discussed with reference to FIG. 4A. Furthermore, the system may use the mapping to update a target (e.g., target R) position within the instrument / robotic coordinate system, based on an identified target location within the imaging coordinate system.

[0075] FIGS. 5A, B schematically illustrate example sources of error in coordinate registration. In FIG. 5A, a coordinate system 501 (e.g., the coordinate system 401 ) is an arbitrary coordinate system. A portion of imaging data corresponding to a flexible elongate device (e.g., flexible elongate device 140 and / or 240) is represented by centerline curve segments 502a-c. The coordinates of the centerline curve segments 502a-c in the coordinate system 501 may be imaging coordinates or coordinates obtained by any well- defined rigid transformation. Shape data 503, on the other hand, may be in sensing system coordinates or coordinates obtained by any well-defined rigid transformation. In this context, a well-defined rigid transformation simply means that the transformation is rigid, know and, therefore reversible. Finding a rigid transformation that overlays the centerline curve segments 502a-c with the shape data 503 can therefore be used to register the imaging coordinate system with the sensing (e.g., robotic) coordinate system.

[0076] For the sake of the present discussion, it can be assumed that aggregated centerline curve segments 502a-c and the shape data 503 are each represented by a respective collection of points (e.g., point clouds, ordered lists of points, etc.). Additionally, it can be assumed that for each point of the aggregated centerline curve segments 502a-c there is a corresponding point in the shape data 503. However, for some points in the shape data 503, there may not be corresponding points in the aggregated centerline curve segments 502a-c because, as illustrated in FIG. 4A, there may be regions (e.g., regions 404a and b) in imaging data from which the system cannot segment points corresponding to the flexible elongate device. The point correspondence described above may aid in computing a metric overlap between transformed point sets using L2-norms, weighted L2 norms of corresponding point separations or any other suitable method.

[0077] More generally, the centerline curve segments 502a-c need not have unique corresponding points in the shape data 503 or vice-versa. The system can compute the point-to-point correspondence using any suitable version of ICP or modified ICP algorithm, curve fitting of sets of points and subsequent sampling, and / or interpolation between points in point sets.

[0078] FIG. 5B illustrates potential sources of error in registration. For example, in FIG. 5B, the centerline curve segment 502a is overlayed with a corresponding portion of the shape data 503, forming an overlap region. A transformation leading to the overlap region may be a result of a 3D 6-DoF optimization. Because the overlapping region has nearly constant curvature, the 6-DoF optimization may be insensitive to a rotational shift 532 along the curvature. The optimization may settle on a local minimum of a cost function or the global minimum of the cost function may be incorrect due to noisy data. In any case, the optimized 6-DoF transformation may settle on an incorrect point correspondence between the centerline curve segment 502a and the corresponding portion of the shape data 503. That is, the point correspondence may shift or slip along the nearly constant curvature, leading to a registration error. In turn, the registration error may result in an incorrect estimate of location of a tip (e.g., distal end, working end) of the flexible elongate device in the imaging coordinate system and, potentially, an error in estimating the relative position between the tip and a target (e.g., ROI R).

[0079] As further illustrated in FIG. 6B, a rigid transformation generating an overlap between the centerline curve segment 502c and a corresponding portion of the shape data 503 may be subject to another error. The straight overlap region may result in an optimization being insensitive to a linear shift 534. Therefore, in the presence of noise or errors in data, the optimization may settle on an incorrect position along the shift 534. In some examples, using multiple centerline curve segments (e.g., at least two of the centerlinecurve segments 502a-c) may alleviate the errors described above. In other examples, particularly if the transformation between imaging and sensing coordinate system is slightly non-rigid (e.g., includes scaling or spatial nonuniformity) using multiple centerline curve segments may not improve registration. For example, an interplay between the shifts 532 and 534 may result in a registration error.

[0080] FIGS. 5C, D schematically illustrate an example of using a translational constraint for improving coordinate registration accuracy. By computing a translational portion of the rigid transformation between the centerline curve segments 502a-c and the shape data 503, a system (e.g., system 100) may establish a correspondence between points of the centerline curve segments 502a-c and the corresponding points in the shape data 503. Thereby, the system may prevent the shifts described in FIG. 5B.

[0081] To compute the translational transformation between the centerline curve segments 502a-c and the shape data 503 (and, thereby, a translational transformation between imaging and sensing coordinates), the system may obtain a position of a tip 505 of the centerline curve segment 502a, as illustrated in FIG. 5C. Position of the tip 505 in imaging coordinates may be based on reconstructed imaging data, for example, by way of an operator (e.g., operator M in FIG. 2) selecting the position in imaging data displayed in a GUI on a display unit (e.g., display unit 130 or 230). Position of a tip 506 in shape data illustrated in FIG. 5C may be read directly from a sensing unit (e.g., sensing unit 120 or 235) in sensing coordinates.

[0082] Subtracting the position of the tip 505 from the centerline curve segments 502a-c and subtracting the position of the tip 506 from the shape data 503, translates both, the aggregate centerline curve segments 502a-c and the shape data 503, to the origin of the coordinate system 501 , as illustrated in FIG. 5D. When the position of the tip 505 and the position of the tip 506 correspond to the tip (or, at least, the same point) of the flexible elongate device, the translation transformations between FIG. 5C and D generate an accurate correspondence between points of the aggregate centerline curve segments 502a- c and the shape data 503. A rotational transformation 550 may then map of the aggregate centerline curve segments 502a-c to the shape data 503. The translational and rotational transformations in FIGS. 5C and D can be combined to create a mapping between the imaging and the sensing coordinate system, as discussed, for example, with reference to FIGS. 4B and C.

[0083] It should be noted that while the discussion with reference to FIGS. 5A-D focused on centerline curve segments 502a-c associated with a flexible elongate device, other representation of the system may represent the image data corresponding to the flexibleelongate device in other ways. For example, the system may represent the flexible elongate device as segments of point sets segmented from image data. The points, for example, may be associated with edges of the flexible elongate device, as discussed, for example with reference to FIGS. 6B and C.

[0084] FIG. 6A schematically illustrates an example image 600 of a flexible elongate device 640 (e.g., flexible elongate device 140, 240), disposed within an anatomical region A. The image 600 may be a view of a 3D reconstruction of projection data, as discussed, for example, with reference to FIGS. 2 and 3. The image 600 may be displayed via a GUI on a display unit (e.g., display units 130, 230). In some examples, as discussed with reference to FIGS. 5C and D, an operator may be prompted to identify a tip 645 of the flexible elongate device 640 in the image 600. Furthermore, a system (e.g., system 100) may segment the image 600 and / or 3D imaging data associated with the image 600 to identify pixels associated with the flexible elongate device 640.

[0085] FIG. 6B schematically illustrates example candidate segments 650a-g of the flexible elongate device 640 segmented from the image 600 or the 3D image data associated with the image 600. The candidate segments 650a-g may be referred to, simply, as segments 650a-g. It should be noted that segments 650a-g are distinct from the centerline curve segments discussed with reference to FIG. 6D in that the segments 650a-g are volumetric features (e.g., point clouds representing volumetric features). The system may be configured to identify the candidate segments 650a-c that belong to the flexible elongate device 640 and reject the spurious candidate segments 650d-g. In some examples, the system may represent each of the candidate segments 650a-c with a respective candidate centerline curve segment, as discussed with reference to FIGS. 6C, D. On the other hand, the system may be configured to identify the first segment 650a and to reject at least some of the other candidate segments 650b-g without computing at least some of the respective centerline curve segments. The system may identify the first segment 650a based on receiving an indication of tip position in imaging coordinates. For example, the cloud point associated with the segment 650a may include the tip position or have points nearest to the tip position.

[0086] Based on the first segment 650a, the system may compute a filter region F1 to implement what can be referred to as a direction filter. To compute the filter region F1 , the system may compute a direction vector at an end of the first segment 650a opposite from the tip 645. For example, the direction vector 655a may be computed based on fitting the segment 650a with a centerline curve (e.g., centerline curve segment 660a in FIG. 6C) and taking the orientation in 3D of the centerline curve at the suitable terminal point. The filter region F1 may then be limited by an angular extent from the direction vector 655a. In someexamples, the filter region F1 may have a conical shell, isotropic around the direction vector 655a. In other examples, the region F1 may be anisotropic, with the shell surface of an elliptical cone. For example, the centerline of the segment 650a may lie predominantly in a plane. The system may be configured to give the filter region F1 a larger in-plane angle than an orthogonal out-of-plane angle. In addition to the angular or directional bound, the filter region may be bounded by a maximum distance from the terminal point of the segment 650a. The system may reject segments 650c, e-g as continuations of the first segment 650a because they lie entirely outside of the filter region F1 . On the other hand, the system may designate segments 650b and d as suitable candidate second segments. The system may proceed to select candidate segment 650b as the second segment using suitable criteria (e.g., as discussed with reference to FIG. 6D) and proceed to find additional segments that represent the flexible elongate device 640. The system may repeat the filtering and selection process by treating segment 650b, in a sense, as the new “first segment,” computing the direction vector 655b, computing filter region F2, and identifying candidate segments 650c, f for expanding the segment-by-segment representation of the flexible elongate device 640. Selecting among candidate segments and identifying one or more centerline curve segments representing a flexible elongate device in image data, is discussed in more detail with reference to FIG. 6D.

[0087] FIG. 6C schematically illustrates representing one of the segments in FIG. 6B by a centerline curve segment 660a. The system may compute the centerline curve segment 660a as a set of parametric polynomial equations, splines, and / or any other suitable method. For example, the centerline curve segment 660a may be computed to minimize mean square distance to nearest points in the point cloud associated with the segment 650a.

[0088] FIG. 6D schematically illustrates example techniques for selecting between alternative candidate centerline curve segments based on candidate segments of a flexible elongate device of FIG. 6B. A system (e.g., system 100) may compute candidate centerline curve segments 660a-e based on the respective candidate segments 650a-d, f as discussed above for the case of the centerline curve segment 660a with reference to FIG. 6C. The centerline curve segment 660a may be referred to as the first centerline curve segment 660a with reference to FIG. 6D. Generally, centerline curve segments may be referred to, more simply, as curve segments. Consequently, candidate centerline curve segments may be referred to as candidate curve segments, and, similarly, for other descriptors. Terminal points 670a and 670b of the first curve segment 660a are, respectively, the tip or the tip- proximal terminal point 670a and the tip-distal terminal point 670b. The system may identify the tip-proximal terminal point 670a based on receiving an indication of tip position from imaging data. Other terminal points 670c-g play various roles in assembling a set of curvesegments to represent the flexible elongate device 640. After identifying the first curve segment 660a and the distal terminal point 670b, the system may compute a first direction vector 680a and a respective filter region F3. The first direction vector 680a may be the same as the direction vector 655, while the filter region F3 may be the filter region F1 . The system may compute or identify candidate second curve segments 660b and d by considering proximal points 670c, d, f and g of the remaining candidate curve segments 660d, b, c and e, respectively. With respect to the first curve segment 660a, the system may designate the remaining candidate curve segments 660b-e as candidate second curve segments. In the illustrated example, the proximal points 670c, d, f and g are closest, in their respective candidate curve segments 660d, b, c and e to the distal terminal point 670b of the first curve segment 660a. Only two proximal points - 670c and d - are within the directional filter region F1 . Therefore, the system accepts only two respective candidate second curve segments - 660d and b - as accepted candidate second curve segments. It should be noted that, generally, proximal points (e.g., to the terminal point 670b) need not be segment end points. In some examples, the system may identify a proximal point to a reference point as a point at least as close to the reference point as all other points of the respective candidate curve segment. In other examples, the system may identify a proximal point to a reference point as a terminal point at least as close to the reference point as an alternative terminal point of the respective candidate curve segment. Without loss of generality in the discussion, the proximal points 670c, d, f and g with reference to the reference terminal point 670b fit both of the above descriptions of terminal points.

[0089] Having designated the accepted candidate second curve segments 660b and d, the system may choose no more than one as the second curve segment. To that end, the system may compute direction vectors 685a and c at the respective proximal points 670c and d. Without loss of generality, we can refer to the curve segment 660d as the accepted candidate second curve segment 660d and the curve segment 660b as an alternative accepted second curve segment 660b. Respectively, we can refer to the direction vector 685c as a second direction vector and the direction vector 685a as an alternative second direction vector. The direction vectors 685a and c are vectors tangent to the respective curve segments 660b and d at the respective proximal points 670d and c. In other examples, the direction vectors 685a and c of the respective curve segments 660b and d may be vectors pointing to the reference terminal point 670b from the respective proximal points 670d and c. To select which, if any, of the two candidate second curve segments 660b and d is designated as the second curve segment, the system may compute collinearity factors between the first direction vector 680a and the second direction vector 685c and the second direction vector between the first direction vector 680a and thealternative second direction vector 685a. The collinearity factor between the first direction vector 680a and the alternative second direction vector 685a may be referred to as an alternative collinearity factor. As illustrated with the help of insert 690, a collinearity factor depends on the magnitude of an acute angle formed between two vectors - the smaller the angle, the higher the collinearity factor, regardless of the chosen formula. Because the angle between the first direction vector 680a and the second direction vector 685c is larger than the angle between the first direction vector 680a and the alternative second direction vector 685a, the system can reject the candidate second curve segment 660d. In cases where the collinearity threshold is lower than the collinearity factor between the first direction vector 680a and the alternative second direction vector 685a, the system can designate the alternative candidate second curve segment 660b as the second curve segment 660b.

[0090] It should be noted that, in some examples, the system may compute direction vectors and respective collinearity vectors for selecting candidate second curve segments. That is, the system may outright reject a vector (avoid accepting as a candidate second vector) in the case that the respective collinearity factor exceeds a threshold. In a sense, adding a threshold to a collinearity factor may be thought of as an alternative or additional filter criterion (along with directions and / or distances to proximal points). Furthermore, the system may select among alternative candidate second curve segments (e.g., curve segments 660b and d) based on the positions of respective proximal points (e.g., proximal points 670d and c of curve segments 660b and d, respectively) within a filter region (e.g., region F3). For example, a candidate second curve segment with a proximal point closer to the directional center of the filter region (than a proximal point of another candidate second curve segment) may be accepted as the second curve segment. In the case of Fig. 6D, the curve segment 660d may be selected as the second curve segment over the curve segment 660b because the proximal point 670c is closer to the direction center of the filter region F3 than the proximal point 670d. In other words, a vector pointing from the terminal point 670b to proximal point 670c is more collinear with the direction vector 680d than a vector pointing from the terminal point 670b to proximal point 670d, and, therefore, the system may designate the candidate second curve segment 660d, rather than the candidate second curve segment 660b, as the second curve segment. In some examples, the system may reject the curve segment 660d (e.g., not accept the curve segment 660d as a candidate second curve segment) based on a low collinearity factor between the first direction vector 680a and the second direction vector 685b. As discussed above, the collinearity factor may be one of filter criteria.

[0091] Having designated the second curve segment 660b, the system may repurpose the procedure described above to select a third curve segment from the candidate curvesegments 660c and e. To that end, the system may compare the collinearity factor between direction vectors 680b and 685b to the collinearity factor between direction vectors 680b and 685d. The system can designate the candidate curve segment 660c, corresponding to the larger of the two collinearity factors, as the third curve segment, provided that the collinearity factor is above the threshold value.

[0092] FIGS. 7-9B depict diagrams of a medical system that may be used for manipulating a medical instrument that includes a flexible elongate device according to any of the methods and systems described above, in some examples. For example, each reference above to the “system” may refer to a system (e.g., system 700) discussed below, or to a subsystem thereof.

[0093] FIG. 7 is a simplified diagram of a medical system 700 according to some examples. The medical system 700 may include at least portions of the system 100 described with reference to FIG. 1 . The medical system 700 may be suitable for use in, for example, surgical, diagnostic (e.g., biopsy), or therapeutic (e.g., ablation, electroporation, etc.) procedures. While some examples are provided herein with respect to such procedures, any reference to medical or surgical instruments and medical or surgical methods is non-limiting. The systems, instruments, and methods described herein may be used for animals, human cadavers, animal cadavers, portions of human or animal anatomy, non-surgical diagnosis, as well as for industrial systems, general or special purpose robotic systems, general or special purpose teleoperational systems, or robotic medical systems.

[0094] As shown in FIG. 7, medical system 700 may include a manipulator assembly 702 that controls the operation of a medical instrument 704 in performing various procedures on a patient (e.g., patient P on table T, as in FIG. 1 ). The medical instrument 704 may include the flexible elongated device 140 of FIG. 1 . Medical instrument 704 may extend into an internal site within the body of patient P via an opening in the body of patient P. The manipulator assembly 702 may be teleoperated, non-teleoperated, or a hybrid teleoperated and non-teleoperated assembly with one or more degrees of freedom of motion that may be motorized and / or one or more degrees of freedom of motion that may be non-motorized (e.g., manually operated). The manipulator assembly 702 may be mounted to and / or positioned near patient table T. A master assembly 706 allows an operator O (e.g., a surgeon, a clinician, a physician, or other user, as described above) to control the manipulator assembly 702. In some examples, the master assembly 706 allows the operator O to view the procedural site or other graphical or informational displays. In some examples, the manipulator assembly 702 may be excluded from the medical system 700 and the instrument 704 may be controlled directly by the operator O. In some examples, the manipulator assembly 702 may be manually controlled by the operator O. Direct operatorcontrol may include various handles and operator interfaces for hand-held operation of the instrument 704.

[0095] The master assembly 706 may be located at a surgeon’s console which is in proximity to (e.g., in the same room as) the patient table T on which patient P is located, such as at the side of the patient table T. In some examples, the master assembly 706 is remote from the patient table T, such as in in a different room or a different building from the patient table T. The master assembly 706 may include one or more control devices for controlling the manipulator assembly 702. The control devices may include any number of a variety of input devices, such as joysticks, trackballs, scroll wheels, directional pads, buttons, data gloves, trigger-guns, hand-operated controllers, voice recognition devices, motion or presence sensors, and / or the like.

[0096] The manipulator assembly 702 supports the medical instrument 704 and may include a kinematic structure of links that provide a set-up structure. The links may include one or more non-servo controlled links (e.g., one or more links that may be manually positioned and locked in place) and / or one or more servo controlled links (e.g., one or more links that may be controlled in response to commands, such as from a control system 712). The manipulator assembly 702 may include a plurality of actuators (e.g., motors) that drive inputs on the medical instrument 704 in response to commands, such as from the control system 712. The actuators may include drive systems that move the medical instrument 704 in various ways when coupled to the medical instrument 704. For example, one or more actuators may advance medical instrument 704 into a naturally or surgically created anatomical orifice. Actuators may control articulation of the medical instrument 704, such as by moving the distal end (or any other portion) of medical instrument 704 in multiple degrees of freedom. These degrees of freedom may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). One or more actuators may control rotation of the medical instrument about a longitudinal axis. Actuators can also be used to move an articulable end effector of medical instrument 704, such as for grasping tissue in the jaws of a biopsy device and / or the like, or may be used to move or otherwise control tools (e.g., imaging tools, ablation tools, biopsy tools, electroporation tools, etc.) that are inserted within the medical instrument 704.

[0097] The control system 712 may include at least portions of the processing unit 125. Additionally or alternatively, the control system 712 may be in communicative connection with the processing unit 125. In some examples, the output of the processing unit 125 according to the techniques described above may cause the control system 712 toautonomously (without input from the operator O) control certain movements of the flexible elongate device 140.

[0098] The medical system 700 may include a sensor system 708 (which may include at least a portion of the sensing unit 120) with one or more sub-systems for receiving information about the manipulator assembly 702 and / or the medical instrument 704. Such sub-systems may include a position sensor system (e.g., that uses electromagnetic (EM) sensors or other types of sensors that detect position or location); a shape sensor system for determining the position, orientation, speed, velocity, pose, and / or shape of a distal end and / or of one or more segments along a flexible body of the medical instrument 704; a visualization system for capturing images, such as from the distal end of medical instrument 704 or from some other location; and / or actuator position sensors such as resolvers, encoders, potentiometers, and the like that describe the rotation and / or orientation of the actuators controlling the medical instrument 704. The subsystems may include an imaging sub-system (e.g., using a color imaging device, an infrared imaging device, an ultrasound imaging device, an x-ray imaging device, a fluoroscopic imaging device, a computed tomography (CT) imaging device, a magnetic resonance imaging (MRI) imaging device, or some other type of imaging device), such as the imaging unit 110.

[0099] It should be noted that the positions and orientations of sensors in the sensor system 708 may be determined in the sensor coordinate system. In some examples, the sensor coordinate system is integrated with or identical to the coordinate system of the manipulator assembly 702.

[0100] The medical system 700 may include a display system 710 (e.g., display device 130) for displaying an image or representation of the procedural site and the medical instrument 704. Display system 710 and master assembly 706 may be oriented so physician O can control medical instrument 704 and master assembly 706 with the perception of telepresence. The display system 710 may include at least portions of the display unit 130.

[0101] In some examples, the medical instrument 704 may include a visualization system, which may include an image capture assembly that records a concurrent or real-time image of a procedural site and provides the image to the operator O through one or more displays of display system 710. The image capture assembly may include various types of imaging devices. The concurrent image may be, for example, a two-dimensional image or a 3D image captured by an endoscope positioned within the anatomical procedural site. The visualization system may obtain intra-operative images in image system coordinates, distinct from the sensor system coordinates. In some examples, the visualization system mayinclude endoscopic components that may be integrally or removably coupled to medical instrument 704. Additionally or alternatively, a separate endoscope, attached to a separate manipulator assembly, may be used with medical instrument 704 to image the procedural site. The visualization system may be implemented as hardware, firmware, software or a combination thereof which interact with or are otherwise executed by one or more computer processors, such as of the control system 712.

[0102] Display system 710 may also display an image of the procedural site and medical instruments, which may be captured by the visualization system. In some examples, the medical system 700 provides a perception of telepresence to the operator O. For example, images captured by an imaging device at a distal portion of the medical instrument 704 may be presented by the display system 710 to provide the perception of being at the distal portion of the medical instrument 704 to the operator O. The input to the master assembly 706 provided by the operator O may move the distal portion of the medical instrument 704 in a manner that corresponds with the nature of the input (e.g., distal tip turns right when a trackball is rolled to the right) and results in corresponding change to the perspective of the images captured by the imaging device at the distal portion of the medical instrument 704. As such, the perception of telepresence for the operator O is maintained as the medical instrument 704 is moved using the master assembly 706. The operator O can manipulate the medical instrument 704 and hand controls of the master assembly 706 as if viewing the workspace in substantially true presence, simulating the experience of an operator that is physically manipulating the medical instrument 704 from within the patient anatomy.

[0103] In some examples, the display system 710 may present virtual images of a procedural site that are created using image data recorded pre-operatively (e.g., prior to the procedure performed by the medical instrument system 200) or intra-operatively (e.g., concurrent with the procedure performed by the medical instrument system 200), such as image data created using computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or the like. The virtual images may include two-dimensional, 3D, or higher-dimensional (e.g., including, for example, time based or velocity-based information) images. In some examples, one or more models are created from pre-operative or intra-operative image data sets and the virtual images are generated using the one or more models.

[0104] In some examples, for purposes of imaged guided medical procedures, display system 710 may display a virtual image that is generated based on tracking the location of medical instrument 704. For example, the tracked location of the medical instrument 704may be registered (e.g., dynamically referenced) with the model generated using the preoperative or intra-operative images, with different portions of the model correspond with different locations of the patient anatomy. As the medical instrument 704 moves through the patient anatomy, the registration is used to determine portions of the model corresponding with the location and / or perspective of the medical instrument 704 and virtual images are generated using the determined portions of the model. This may be done to present the operator O with virtual images of the internal procedural site from viewpoints of medical instrument 704 that correspond with the tracked locations of the medical instrument 704.

[0105] The display system 710 may include the display unit 130 and may display images including the position, orientation, and / or pose of the medical instrument 704 in a joint coordinate system based on registering the sensor coordinate system with the imaging coordinate system according to the techniques described above with reference to FIGS. 2- 6D.

[0106] The medical system 700 may also include the control system 712, which may include processing circuitry (e.g., the processing unit 125) that implements the some or all of the methods or functionality discussed herein. The control system 712 may include at least one memory and at least one processor for controlling the operations of the manipulator assembly 702, the medical instrument 704, the master assembly 706, the sensor system 708, and / or the display system 710. Control system 712 may include instructions (e.g., a non-transitory machine-readable medium storing the instructions) that when executed by the at least one processor, configures the one or more processors to implement some or all of the methods or functionality discussed herein. While the control system 712 is shown as a single block in FIG. 7, the control system 712 may include two or more separate data processing circuits with one portion of the processing being performed at the manipulator assembly 702, another portion of the processing being performed at the master assembly 706, and / or the like. In some examples, the control system 712 may include other types of processing circuitry, such as application-specific integrated circuits (ASICs) and / or field- programmable gate array (FPGAs). The control system 712 may be implemented using hardware, firmware, software, or a combination thereof.

[0107] In some examples, the control system 712 may receive feedback from the medical instrument 704, such as force and / or torque feedback. Responsive to the feedback, the control system 712 may transmit signals to the master assembly 706. In some examples, the control system 712 may transmit signals instructing one or more actuators of the manipulator assembly 702 to move the medical instrument 704. In some examples, the control system 712 may transmit informational displays regarding the feedback to the display system 710 for presentation or perform other types of actions based on the feedback.

[0108] The control system 712 may include a virtual visualization system to provide navigation assistance to operator O when controlling the medical instrument 704 during an image-guided medical procedure. Virtual navigation using the virtual visualization system may be based upon an acquired pre-operative or intra-operative dataset of anatomical passageways of the patient P. The control system 712 or a separate computing device may convert the recorded images, using programmed instructions alone or in combination with operator inputs, into a model of the patient anatomy. The model may include a segmented two-dimensional or 3D composite representation of a partial or an entire anatomical organ or anatomical region. An image data set may be associated with the composite representation. The virtual visualization system may obtain sensor data from the sensor system 708 that is used to compute an (e.g., approximate) location of the medical instrument 704 with respect to the anatomy of patient P. The sensor system 708 may be used to register and display the medical instrument 704 together with the pre-operatively or intra-operatively recorded images. For example, PCT Publication WO 2016 / 191298 (published December 1 , 2016 and titled “Systems and Methods of Registration for Image Guided Surgery”), which is incorporated by reference herein in its entirety, discloses example systems. Additionally or alternatively, the registration may be based on the techniques discussed above with reference to FIGS. 2-6D.

[0109] During a virtual navigation procedure, the sensor system 708 may be used to compute the (e.g., approximate) location of the medical instrument 704 with respect to the anatomy of patient P. The location can be used to produce both macro-level (e.g., external) tracking images of the anatomy of patient P and virtual internal images of the anatomy of patient P. The system may include one or more electromagnetic (EM) sensors, fiber optic sensors, and / or other sensors to register and display a medical instrument together with pre- operatively recorded medical images. For example, U.S. Patent No. 8,900,131 (filed May 13, 2011 and titled “Medical System Providing Dynamic Registration of a Model of an Anatomical Structure for Image-Guided Surgery”), which is incorporated by reference herein in its entirety, discloses example systems.

[0110] Medical system 700 may further include operations and support systems (not shown) such as illumination systems, steering control systems, irrigation systems, and / or suction systems. In some examples, the medical system 700 may include more than one manipulator assembly and / or more than one master assembly. The exact number of manipulator assemblies may depend on the medical procedure and space constraints within the procedural room, among other factors. Multiple master assemblies may be co-located or they may be positioned in separate locations. Multiple master assemblies may allow more than one operator to control one or more manipulator assemblies in various combinations.

[0111] FIG. 8A is a simplified diagram of a medical instrument system °0 according to some examples. The medical instrument system 800 includes a flexible elongate device 802 (e.g., device 140), also referred to as elongate device 802, a drive unit 804, and a medical tool 826 that collectively is an example of a medical instrument 704 of a medical system 700. The medical system 700 may be a teleoperated system, a non-teleoperated system, or a hybrid teleoperated and non-teleoperated system, as explained with reference to FIG. 7. A visualization system 831 , tracking system 830, and navigation system 832 are also shown in FIG. 8A and are example components of the control system 712 of the medical system 700. In some examples, the medical instrument system 800 may be used for non-teleoperational exploratory procedures or in procedures involving traditional manually operated medical instruments, such as endoscopy. The medical instrument system 800 may be used to gather (e.g., measure) a set of data points corresponding to locations within anatomical passageways of a patient, such as patient P.

[0112] The elongate device 802 is coupled to the drive unit 804. The elongate device 802 includes a channel 821 through which the medical tool 826 may be inserted. The elongate device 802 navigates within patient anatomy to deliver the medical tool 826 to a procedural site. The elongate device 802 includes a flexible body 816 having a proximal end 817 and a distal end 818. In some examples, the flexible body 816 may have an approximately 3 mm outer diameter. Other flexible body outer diameters may be larger or smaller.

[0113] Medical instrument system 800 may include the tracking system 830 for determining the position, orientation, speed, velocity, pose, and / or shape of the flexible body 816 at the distal end 818 and / or of one or more segments 824 along flexible body 816, as will be described in further detail below. The tracking system 830 may include one or more sensors and / or imaging devices. The flexible body 816, such as the length between the distal end 818 and the proximal end 817, may include multiple segments 824. The tracking system 830 may be implemented using hardware, firmware, software, or a combination thereof. In some examples, the tracking system 830 is part of control system 712 shown in FIG. 7. The tracking system 830 may implement at least some of the techniques described with reference to FIGS. 1 A-6, and, to that end, may include at least portions of or be in communicative connection with the processing unit 125 of FIG. 1A.

[0114] T racking system 830 may track the distal end 818 and / or one or more of the segments 824 of the flexible body 816 using a shape sensor 822. It should be noted that, with the techniques of the instant disclosure, the shape sensor 822 may be omitted. The shape sensor 822 may include an optical fiber aligned with the flexible body 816 (e.g., provided within an interior channel of the flexibly body 816 or mounted externally along the flexible body 816). In some examples, the optical fiber may have a diameter ofapproximately 800 pm. In other examples, the diameter may be larger or smaller. The optical fiber of the shape sensor 822 may form a fiber optic bend sensor for determining the shape of flexible body 816. Optical fibers including Fiber Bragg Gratings (FBGs) may be used to provide strain measurements in structures in one or more dimensions. Various systems and methods for monitoring the shape and relative position of an optical fiber in three dimensions, which may be applicable in some examples, are described in U.S. Patent Application Publication No. 2006 / 0013523 (filed July 13, 2005 and titled “Fiber optic position and shape sensing device and method relating thereto”); U.S. Patent No. 7,772,541 (filed on March 12, 2008 and titled “Fiber Optic Position and / or Shape Sensing Based on Rayleigh Scatter”); and U.S. Patent No. 8,773,650 (filed on Sept. 2, 2010 and titled “Optical Position and / or Shape Sensing”), which are all incorporated by reference herein in their entireties. Sensors in some examples may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and Fluorescence scattering.

[0115] In some examples, the shape of the flexible body 816 may be determined using other techniques. For example, a history of the position and / or pose of the distal end 818 of the flexible body 816 can be used to reconstruct the shape of flexible body 816 over an interval of time (e.g., as the flexible body 816 is advanced or retracted within a patient anatomy). In some examples, the tracking system 830 may alternatively and / or additionally track the distal end 818 of the flexible body 816 using a position sensor system 820.Position sensor system 820 may be a component of an EM sensor system with the position sensor system 820 including one or more position sensors. Although the position sensor system 820 is shown as being near the distal end 818 of the flexible body 816 to track the distal end 818, the number and location of the position sensors of the position sensor system 820 may vary to track different regions along the flexible body 816. In one example, the position sensors include conductive coils that may be subjected to an externally generated electromagnetic field. Each coil of position sensor system 820 may produce an induced electrical signal having characteristics that depend on the position and orientation of the coil relative to the externally generated electromagnetic field. The position sensor system 820 may measure one or more position coordinates and / or one or more orientation angles associated with one or more portions of flexible body 816. In some examples, the position sensor system 820 may be configured and positioned to measure six degrees of freedom, e.g., three position coordinates X, Y, Z and three orientation angles indicating pitch, yaw, and roll of a base point. In some examples, the position sensor system 820 may be configured and positioned to measure five degrees of freedom, e.g., three position coordinates X, Y, Z and two orientation angles indicating pitch and yaw of a base point. Further description of a position sensor system, which may be applicable in some examples,is provided in U.S. Patent No. 6,380,732 (filed August 11 , 1999 and titled “Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked”), which is incorporated by reference herein in its entirety.

[0116] A processing unit (e.g., processing unit 125) may enhance the accuracy of positions obtained by the sensor system 820 by combing data obtained by the sensor system 820 with data obtained by an external imaging system (e.g., by way of the imaging unit 110) according to the techniques of this disclosure described above with reference to FIGS. 2-6D.

[0117] In some examples, the tracking system 830 may alternately and / or additionally rely on a collection of pose, position, and / or orientation data stored for a point of an elongate device 802 and / or medical tool 826 captured during one or more cycles of alternating motion, such as breathing. This stored data may be used to develop shape information about the flexible body 816. In some examples, a series of position sensors (not shown), such as EM sensors like the sensors in position sensor system 820 or some other type of position sensors may be positioned along the flexible body 816 and used for shape sensing. In some examples, a history of data from one or more of these position sensors taken during a procedure may be used to represent the shape of elongate device 802, particularly if an anatomical passageway is generally static.

[0118] FIG. 8B is a simplified diagram of the medical tool 826 within the elongate device 802 according to some examples. The flexible body 816 of the elongate device 802 may include the channel 821 sized and shaped to receive the medical tool 826. In some examples, the medical tool 826 may be used for procedures such as imaging, surgery, biopsy, ablation, illumination, irrigation, suction, electroporation, etc. Medical tool 826 can be deployed through channel 821 of flexible body 816 and operated at a procedural site within the anatomy. Medical tool 826 may be, for example, an image capture probe, a biopsy tool (e.g., a needle, grasper, brush, etc.), an ablation tool (e.g., a laser ablation tool, radio frequency (RF) ablation tool, cryoablation tool, thermal ablation tool, heated liquid ablation tool, etc.), an electroporation tool, and / or another surgical, diagnostic, or therapeutic tool. In some examples, the medical tool 826 may include an end effector having a single working member such as a scalpel, a blunt blade, an optical fiber, an electrode, and / or the like. Other end types of end effectors may include, for example, forceps, graspers, scissors, staplers, clip appliers, and / or the like. Other end effectors may further include electrically activated end effectors such as electrosurgical electrodes, transducers, sensors, and / or the like.

[0119] The medical tool 826 may be a biopsy tool used to remove sample tissue or a sampling of cells from a target anatomical location. In some examples, the biopsy tool is a flexible needle. The biopsy tool may further include a sheath that can surround the flexible needle to protect the needle and interior surface of the channel 821 when the biopsy tool is within the channel 821 . The medical tool 826 may be an image capture probe that includes a distal portion with a stereoscopic or monoscopic camera that may be placed at or near the distal end 818 of flexible body 816 for capturing images (e.g., still or video images). The captured images may be processed by the visualization system 831 for display and / or provided to the tracking system 830 to support tracking of the distal end 818 of the flexible body 816 and / or one or more of the segments 824 of the flexible body 816. The image capture probe may include a cable for transmitting the captured image data that is coupled to an imaging device at the distal portion of the image capture probe. In some examples, the image capture probe may include a fiber-optic bundle, such as a fiberscope, that couples to a more proximal imaging device of the visualization system 831 . The image capture probe may be single-spectral or multi-spectral, for example, capturing image data in one or more of the visible, near-infrared, infrared, and / or ultraviolet spectrums. The image capture probe may also include one or more light emitters that provide illumination to facilitate image capture. In some examples, the image capture probe may use ultrasound, x-ray, fluoroscopy, CT, MRI, or other types of imaging technology.

[0120] In some examples, the image capture probe is inserted within the flexible body 816 of the elongate device 802 to facilitate visual navigation of the elongate device 802 to a procedural site and then is replaced within the flexible body 816 with another type of medical tool 826 that performs the procedure. In some examples, the image capture probe may be within the flexible body 816 of the elongate device 802 along with another type of medical tool 826 to facilitate simultaneous image capture and tissue intervention, such as within the same channel or in separate channels. A medical tool 826 may be advanced from the opening of the channel 821 to perform the procedure (or some other functionality) and then retracted back into the channel 821 when the procedure is complete. The medical tool 826 may be removed from the proximal end 817 of the flexible body 816 or from another optional instrument port (not shown) along flexible body 816.

[0121] In some examples, the elongate device 802 may include integrated imaging capability rather than utilize a removable image capture probe. For example, the imaging device (or fiber-optic bundle) and the light emitters may be located at the distal end 818 of the elongate device 802. The flexible body 815 may include one or more dedicated channels that carry the cable(s) and / or optical fiber(s) between the distal end 818 and thevisualization system 831 . Here, the medical instrument system 800 can perform simultaneous imaging and tool operations.

[0122] In some examples, the medical tool 826 is capable of controllable articulation. The medical tool 826 may house cables (which may also be referred to as pull wires), linkages, or other actuation controls (not shown) that extend between its proximal and distal ends to controllably bend the distal end of medical tool 826, such as discussed herein for the flexible elongate device 802. The medical tool 826 may be coupled to a drive unit 804 and the manipulator assembly 702. In these examples, the elongate device 802 may be excluded from the medical instrument system 800 or may be a flexible device that does not have controllable articulation. Steerable instruments or tools, applicable in some examples, are further described in detail in U.S. Patent No. 7,316,681 (filed on Oct. 4, 2005 and titled “Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity”) and U.S. Patent No. 9,259,274 (filed Sept. 30, 2008 and titled “Passive Preload and Capstan Drive for Surgical Instruments”), which are incorporated by reference herein in their entireties.

[0123] The flexible body 816 of the elongate device 802 may also or alternatively house cables, linkages, or other steering controls (not shown) that extend between the drive unit 804 and the distal end 818 to controllably bend the distal end 818 as shown, for example, by broken dashed line depictions 819 of the distal end 818 in FIG. 2A. In some examples, at least four cables are used to provide independent up-down steering to control a pitch of the distal end 818 and left-right steering to control a yaw of the distal end 881 . In these examples, the flexible elongate device 802 may be a steerable catheter. Examples of steerable catheters, applicable in some examples, are described in detail in PCT Publication WO 2019 / 018736 (published Jan. 24, 2019 and titled “Flexible Elongate Device Systems and Methods”), which is incorporated by reference herein in its entirety.

[0124] In examples where the elongate device 802 and / or medical tool 826 are actuated by a teleoperational assembly (e.g., the manipulator assembly 702), the drive unit 804 may include drive inputs that removably couple to and receive power from drive elements, such as actuators, of the teleoperational assembly. In some examples, the elongate device 802 and / or medical tool 826 may include gripping features, manual actuators, or other components for manually controlling the motion of the elongate device 802 and / or medical tool 826. The elongate device 802 may be steerable or, alternatively, the elongate device 802 may be non-steerable with no integrated mechanism for operator control of the bending of distal end 818. In some examples, one or more channels 821 (which may also be referred to as lumens), through which medical tools 826 can be deployed and used at atarget anatomical location, may be defined by the interior walls of the flexible body 816 of the elongate device 802.

[0125] In some examples, the medical instrument system 800 (e.g., the elongate device 802 or medical tool 826) may include a flexible bronchial instrument, such as a bronchoscope or bronchial catheter, e.g., for use in visual examination and diagnosis, biopsy, and / or treatment of a lung. The medical instrument system 800 may also be suited for navigation and treatment of other tissues, via natural or surgically created connected passageways, in any of a variety of anatomical systems, including the colon, the intestines, the kidneys and kidney calices, the brain, the heart, the circulatory system including vasculature, and / or the like.

[0126] The information from the tracking system 830 may be sent to the navigation system 832, where the information may be combined with information from the visualization system 831 and / or pre-operatively obtained models to provide the physician, clinician, surgeon, or other operator with real-time position information. The tracking system 830, the navigation system 832, and the visualization system 831 may cooperatively implement, at least partially, the functionality of the system 100 in implementing the techniques described with reference to FIGS. 1 A-6. In some examples, the real-time position information may be displayed on the display system 710 for use in the control of the medical instrument system 800. In some examples, the navigation system 832 may utilize the position information as feedback for positioning medical instrument system 800.

[0127] FIGS. 9A and 9B are simplified diagrams of side views of a patient coordinate space including a medical instrument mounted on an insertion assembly according to some examples. As shown in FIGS. 9A and 9B, a surgical environment 900 may include the patient P positioned on the patient table T. Patient P may be stationary within the surgical environment 900 in the sense that gross patient movement is limited by sedation, restraint, and / or other means. Cyclic anatomical motion, including respiration and cardiac motion, of patient P may continue. Within surgical environment 900, a medical instrument 904 is used to perform a medical procedure which may include, for example, surgery, biopsy, ablation, illumination, irrigation, suction, or electroporation. The medical instrument 904 may also be used to perform other types of procedures, such as a registration procedure to associate the position, orientation, and / or pose data captured by the sensor system 708 to a desired (e.g., anatomical or system) reference frame. The medical instrument 904 may be, for example, the medical instrument 704. In some examples, the medical instrument 904 may include an elongate device 910 (e.g., a catheter) coupled to an instrument body 912. Elongate device 910 may be the elongate device 140 of FIG. 1. Elongate device 910 includes one or more channels sized and shaped to receive a medical tool.

[0128] Elongate device 910 may also include one or more sensors (e.g., components of the sensor system 708). In some examples, a shape sensor 914 may be fixed at a proximal point 916 on the instrument body 912. The proximal point 916 of the shape sensor 914 may be movable with the instrument body 912, and the location of the proximal point 916 with respect to a desired reference frame may be known (e.g., via a tracking sensor or other tracking device). The shape sensor 914 may measure a shape from the proximal point 916 to another point, such as a distal end 918 of the elongate device 910. The shape sensor 914 may be aligned with the elongate device 910 (e.g., provided within an interior channel or mounted externally). In some examples, the shape sensor 914 may optical fibers used to generate shape information for the elongate device 910.

[0129] In some examples, position sensors (e.g., EM sensors) may be incorporated into the medical instrument 904. A series of position sensors may be positioned along the flexible elongate device 910 and used for shape sensing. Position sensors may be used alternatively to the shape sensor 914 or with the shape sensor 914, such as to improve the accuracy of shape sensing or to verify shape information.

[0130] Elongate device 910 may house cables, linkages, or other steering controls that extend between the instrument body 912 and the distal end 918 to controllably bend the distal end 918. In some examples, at least four cables are used to provide independent up- down steering to control a pitch of distal end 918 and left-right steering to control a yaw of distal end 918. The instrument body 912 may include drive inputs that removably couple to and receive power from drive elements, such as actuators, of a manipulator assembly.

[0131] The instrument body 912 may be coupled to an instrument carriage 906. The instrument carriage 906 may be mounted to an insertion stage 908 that is fixed within the surgical environment 900. Alternatively, the insertion stage 908 may be movable but have a known location (e.g., via a tracking sensor or other tracking device) within surgical environment 900. Instrument carriage 906 may be a component of a manipulator assembly (e.g., manipulator assembly 702) that couples to the medical instrument 904 to control insertion motion (e.g., motion along an insertion axis A) and / or motion of the distal end 918 of the elongate device 910 in multiple directions, such as yaw, pitch, and / or roll. The instrument carriage 906 or insertion stage 908 may include actuators, such as servomotors, that control motion of instrument carriage 906 along the insertion stage 908.

[0132] A sensor device 920, which may be a component of the sensor system 708, may provide information about the position of the instrument body 912 as it moves relative to the insertion stage 908 along the insertion axis A. The sensor device 920 may include one or more resolvers, encoders, potentiometers, and / or other sensors that measure the rotationand / or orientation of the actuators controlling the motion of the instrument carriage 906, thus indicating the motion of the instrument body 912. In some examples, the insertion stage 908 has a linear track as shown in FIGS. 9A and 9B. In some examples, the insertion stage 908 may have curved track or have a combination of curved and linear track sections.

[0133] FIG. 9A shows the instrument body 912 and the instrument carriage 906 in a retracted position along the insertion stage 908. In this retracted position, the proximal point 916 is at a position L0 on the insertion axis A. The location of the proximal point 916 may be set to a zero value and / or other reference value to provide a base reference (e.g., corresponding to the origin of a desired reference frame) to describe the position of the instrument carriage 906 along the insertion stage 908. In the retracted position, the distal end 918 of the elongate device 910 may be positioned just inside an entry orifice of patient P. Also in the retracted position, the data captured by the sensor device 920 may be set to a zero value and / or other reference value (e.g., l=0). In FIG. 9B, the instrument body 912 and the instrument carriage 906 have advanced along the linear track of insertion stage 908, and the distal end 918 of the elongate device 910 has advanced into patient P. In this advanced position, the proximal point 916 is at a position L1 on the insertion axis A. In some examples, the rotation and / or orientation of the actuators measured by the sensor device 920 indicating movement of the instrument carriage 906 along the insertion stage 908 and / or one or more position sensors associated with instrument carriage 906 and / or the insertion stage 908 may be used to determine the position L1 of the proximal point 916 relative to the position L0. In some examples, the position L1 may further be used as an indicator of the distance or insertion depth to which the distal end 918 of the elongate device 910 is inserted into the passageway(s) of the anatomy of patient P.

[0134] FIG. 10 depicts a flow chart for an example method 1000 for visualizing patient anatomy during a medical procedure. The method 1000 may be implemented by a system (e.g., system 100) which may include one or more processors (e.g., disposed at the processing unit 125). Furthermore, instructions for executing the method 1000 on the one or more processors may be stored on a tangible, non-transitory, computer readable medium.

[0135] At block 1010, the method 1000 includes obtaining, based on image data received from an imaging system (e.g., imaging unit 110, imaging system 210, etc.), a three- dimensional (3D) representation (e.g., image 600) of an anatomical structure (e.g., anatomical structure A) and a portion of a flexible elongate device (e.g., flexible elongate device 640) disposed in the anatomical structure, the portion including a tip (e.g., tip 505 or 645). In some examples, obtaining the 3D representation may include receiving the 3D representation from the imaging system. In other examples, obtaining the 3D representation may include reconstructing, using one or more processors, the 3D representation based onimage data (e.g., projection data) received from the imaging system. The 3D representation is in an imaging coordinate system associated with the imaging system. The system may receive the 3D representation from an imaging system (e.g., imaging unit 110, imaging system 210) via any suitable communication link (e.g., link 250). In some examples, the system receives the 3D representation by way of digital storage that is outside of the imaging system. The 3D representation may be a 3D image or 2D slices that are reconstructed from projections (e.g., by the imaging system). In some examples, the system may receive projection data directly and / or perform at least some of the reconstruction.

[0136] At block 1020, the method 1000 includes receiving, from a sensing unit (e.g., sensing unit 120 or 235), shape data (e.g., shape data 403 or 503) indicative of a shape of the flexible elongate device including a position of the tip (e.g., tip 506). The shape data is in a sensing coordinate system associated with the sensing unit.In some examples, the method 1000 may include determining, by the one or more processors, that the shape data is received in an imaging time window as discussed, for example, with reference to FIG. 3. The imaging time window is a time period that best reflects when imaging data was acquired by the imaging system. Determining that the shape data is received in the imaging time window may include recording at least some of the received shape data in a buffer of a memory device and identifying, within the shape data recorded in the buffer, a portion of the shape data indicative of a low-motion period. The low-motion period may be a period during which a patient holds their breath, per operator instructions, and the imaging data is being acquired.

[0137] At block 1030, the method 1000 includes obtaining a position of the tip in the imaging coordinate system. Obtaining the position of the tip may be based on receiving the position based on a selection (e.g., by an operator) at a GUI of a display device. Additionally or alternatively, suitable image processing techniques may aid in the selection of the tip position.

[0138] At block 1040, the method 1000 includes computing a translational transformation between the imaging coordinate system and the sensing coordinate system based on the position of the tip in the sensing coordinate system and the position of the tip in the imaging coordinate system (e.g., as discussed with reference to FIGS. 5C, D).

[0139] At block 1050, the method 1000 includes computing (e.g., as discussed with reference to FIGS. 6C, D), based on the received 3D representation, one or more curve segments (e.g., segments 502a-c, 660a-c, etc.) indicative of the shape of the flexible elongate device in the imaging coordinate system. As discussed above, computing the one or more curve segments may include a set of operations. The system may compute a firstcurve segment and one or more candidate second curve segments. Furthermore, the system may compute a first direction vector representative of direction of the first curve segment (e.g., pointing tangentially to the first curve segment) at a terminal point of the first curve segment. The system may computing a filter region based at least in part on selecting a threshold angular deviation from the first direction vector. The filter region may also be bound by a maximum distance from the terminal point of the first curve segment. The system may use the filter region to accept and / or reject candidate second segments. To do so, the system may identify, for each candidate second curve segment, a proximal point (of the candidate second curve segment with respect to the first curve segment). The proximal point may be a point that is closest (or at least as close as all other points of the candidate second curve) to a terminal point of the first curve segment. Additionally or alternatively, the proximal point may be a terminal point of the candidate second curve segment closest (or at least as close as the alternative terminal point) to the terminal point of the first curve segment. The system may accept candidate second curve segments (designate them as accepted candidate second curve segments) when the system determines that the proximal point of the candidate second curve is within the filter region. When more than one candidate second curve segment are accepted, the system may choose one as the accepted second curve segment as discussed with reference to FIG. 6D. In other words, candidate second curve segments may be accepted based on the filter region and the accepted second curve segment is chosen from the accepted candidate second curve segments based on additional selection criteria. In some examples, even with multiple accepted candidate second curve segments, no accepted second curve segment is designated. That is, the system may use only the first curve segment including the tip and the shape data to compute the rotational coordinate transformation.

[0140] Selecting between one accepted candidate second curve segment and another accepted alternative candidate second curve segment can include several computations. For example, the system may compute, for each of the candidate second curve segments a respective direction vector. In one example, the computed direction vectors are vectors tangent to the candidate second curve vectors at the proximal (to the first curve segment, as discussed above) points. The system may then compute collinearity factors (as discussed with reference to FIG. 6D) between each of the direction vectors at the candidate second curve segments and the first direction vector at the first curve segment. The system may select the candidate second curve segment that has a direction vector most collinear with the first direction vector. It should be noted that the system may select among three, four, or any other suitable number of candidate second curve segments using the presently discussed techniques. When the collinearity factor for the selected candidate second curvesegments is lower (less collinear) than a threshold value, the system may reject the selected candidate second curve segment. Conversely, when the collinearity factor for the selected candidate second curve segments is at or above the threshold value, the system may designate the selected candidate second curve segment as the accepted second curve segment.

[0141] After designating the accepted second curve segment, the system may repeat the process discussed above to find a third curve segment. That is, candidate third curve segments may be filtered with a direction filter at a distal terminal point of the accepted second curve segment and an accepted third curve segment may be selected from the filtered option as discussed above. In a sense, the second curve segment becomes, for the selection of the third segment, what the first curve segment was for the selection of the second curve segment. The process may continue to string together any suitable number of curve segments.

[0142] At block 1060, the method 1000 includes computing a rotational transformation (e.g., transformation 550) between the imaging coordinate system and the sensing coordinate system based on the shape data and the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system. Computing the rotational transformation may be based at least in part on minimizing a cost function indicative of overlap of the shape data and the one or more curve segments in a shared coordinate system (e.g., as discussed with reference to FIG. 5D). The cost function may be based at least in part on a weighted L2 norm of distances between a set of points in the shape data and a respective set of points in the one or more curve segments in the shared coordinate system. For example, the weights of the L2 norm may decrease away from the tip. In a sense, performing translation to align tips prior to performing the rotational transformation can be thought of as assigning the discrepancy at the tip a weight that approaches infinity. In some examples, the system may compute the rotational transformation based at least in part on an iterative closest point algorithm or a modified iterative closest point algorithm. Additionally or alternatively, the system may use a coherent point drift algorithm to compute the rotational transformation.

[0143] In some examples, segments added to the one or more curve segments, selected as discussed above, may have deleterious effects of computing the transformation. For example, a curve segment may include an imaging artifact that results in a lower quality transformation than could be obtained without the segment. In some examples, the system may discard one or more curve segment to obtain a better transformation quality.

[0144] At block 1070, the method 1000 includes registering the imaging coordinate system with the sensing coordinate system based on the translational transformation and the rotational transformations. Registering the imaging coordinate system with the sensing coordinate system may include assigning to at least some of the shape data coordinates in the imaging system. Additionally or alternatively, registering the imaging coordinate system with the sensing coordinate system may include assigning to at least some of the points in the imaging coordinate system respective coordinates in the sensing coordinate system. For example, the system may segment a target (e.g., target R), find one or more positions associated with the target, and assign sensing (and, thereby, robotic) coordinates to the positions associated with the target. In other words, the method 1000 may include identifying a position of a target in the imaging coordinate system and computing the position of the target in the sensing coordinate system based at least in part on the computed translational and rotational transformations. Additionally or alternatively, registration may generate coordinates in a registration coordinate system that is distinct from both, the imaging and the sensing coordinates. That is, the system may perform registration by assigning registration coordinates to one set of points from the sensing coordinates and another set of points from the imaging coordinates so as to reflect the relative positions among the two sets. For example, the registration coordinate system may be chosen to have a coordinate origin located at the tip of the flexible elongate device and a z-axis along the pointing direction of the flexible elongate device at the tip. In another example, the origin of the registration coordinate system may be placed at the target.

[0145] In some examples, the method 1000 further includes displaying on a display device (e.g., of display units 130 or 230) overlayed images of the shape data and the one or more curve segments. The overlayed images may aid in operating a robotic system (e.g., robotic system 205) to control the flexible elongate device, evaluating the quality of registration, and / or making other medical procedure decisions.

[0146] In some examples, the method 1000 further includes obtaining an indication of low registration quality. For example, the system may compare the value of the minimized cost function to a threshold and find that the optimization failed. Additionally or alternatively, the system may receive the indication of low registration quality (e.g., from an operator) by way of a GUI. In response to obtaining the indication of low registration quality, the system may generate (e.g., on a GUI) a prompt to select a new position of the tip. Additionally or alternatively, the system may generate a suggestion (e.g., on a GUI) to acquire new imaging data using the imaging device. The method 1000 may include recomputing the translational and rotational transformations as described above (e.g., computing new translational and / or rotational transformations), resulting in a new registration. The new registration may bebased, for example, on new position of the tip and / or new imaging data. In some examples, the registration may be based at least in part on averaging (e.g., with weights selected by the system and / or the user) multiple registration.

[0147] FIGS. 11 A and B depicts a flow chart for an example method 1100 for visualizing patient anatomy during a medical procedure. The method 1100 may be implemented by a system (e.g., system 100) which may include one or more processors (e.g., disposed at the processing unit 125). Furthermore, instructions for executing the method 1100 on the one or more processors may be stored on a tangible, non-transitory, computer readable medium. FIG. 11 A includes blocks 1110-1160 associated with higher level elements of the method 1100 insofar as the block 1130 includes additional elements discussed in FIG. 11 B with reference to a sub-method 1131 comprising blocks 1132-1137.

[0148] At block 1110, the method 1100 includes receiving, by one or more processors and from a sensing unit, shape data indicative of a shape of the flexible elongate, wherein the shape data is in a sensing coordinate system associated with the sensing unit. Block 1110 may include the same options as block 1010 in FIG. 10. At block 1120, the method 1100 includes obtaining, by the one or more processors and based on image data received from an imaging system, a three-dimensional (3D) representation of an anatomical structure and a portion of a flexible elongate device disposed in the anatomical structure, wherein the three-dimensional representation is in an imaging coordinate system associated with the imaging system. In some examples, obtaining the 3D representation may include receiving the 3D representation from the imaging system. In other examples, obtaining the 3D representation may include reconstructing, using one or more processors, the 3D representation based on image data (e.g., projection data) received from the imaging system. Block 1120 may include the same options as block 1020 in FIG. 10. At block 1130, the method 1100 includes computing based at least in part on the received three- dimensional representation, one or more curve segments indicative of the shape of the flexible elongate device.

[0149] At block 1130, to compute one or more curve segments indicative of the shape of the flexible elongate device, the method 1100 includes an example sub-method 1131 , which is illustrated in FIG. 11 B. The sub-method 1131 includes the blocks 1132-1137. At block 1132, the sub-method 1131 includes computing a first curve segment and a candidate second curve segment. At block 1133, the sub-method 1131 includes computing a first direction vector representative of direction of the first curve segment (e.g., pointing tangentially to the first curve segment) at a terminal point of the first curve segment. At block 1134, the sub-method 1131 includes computing a filter region based on the first curve segment, wherein computing the filter region is based at least in part on selecting amaximum angular deviation from the first direction vector. At block 1135, the sub-method 1131 includes identifying a proximal point of the candidate second curve segment. For example, the proximal point can be identified as either i) at least as close to a terminal point of the first curve segment as all other points of the candidate second curve segment; or ii) is a terminal point of the candidate second curve segment, where the terminal point of the candidate second curve is at least as close to the terminal point of the first curve segment as an alternative terminal point of the candidate second curve segment. At block 1136, the sub-method 1131 includes determining that the proximal point of the candidate second curve is within the filter region. At block 1137, the sub-method 1131 includes designating the candidate second curve segment as an accepted candidate second curve segment based at least in part on determining that the proximal point of the candidate second curve segment lies within the filter region. It should be noted that the blocks of the sub-method 1131 may include the elements discussed above with reference to block 1050 in FIG. 10.

[0150] In some examples, the method 1100 may use data indicative of position of a tip of the flexible elongate device in the shape data and / or in the 3D representation based on image data. For example, the position of the tip in the 3D representation may aid in ordering points of the first segment. In other examples, the method 1100 may include identifying one or more points as proximal to the tip to aid, for example, in orienting the first curve segment with respect to the tip of the elongate device. The orientation of the first curve segment may aid in identifying the terminal point of the first curve segment. Additionally or alternatively, the method 1100 may identify the terminal point of the first segment based on the relative position of points of other curve segments. Still in some example, the method 1100 may use input from an operator (e.g., via a user interface) to identify the first curve segment and / or the orientation of the first curve segment with respect to the tip of the flexible elongate device.

[0151] Returning to FIG. 11 A, at block 1140, the method 1100 includes computing a transformation between the imaging coordinate system and the sensing coordinate system. Computing the transformation is based on the shape data and the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system. Computing the transformation may be based at least in part on minimizing a cost function indicative of overlap of the shape data and the one or more curve segments in a shared coordinate system (e.g., as discussed with reference to FIG. 5D). The cost function may be based at least in part on a weighted L2 norm of distances between a set of points in the shape data and a respective set of points in the one or more curve segments in the shared coordinate system. For example, the weights of the L2 norm may decrease away from the tip (whether or not the position of the tip is known). In some examples, the systemmay compute the transformation based at least in part on an iterative closest point algorithm or a modified iterative closest point algorithm. Additionally or alternatively, the system may use a coherent point drift algorithm to compute the transformation. In some examples, segments added to the one or more curve segments, selected as discussed above, may have deleterious effects of computing the transformation. For example, a curve segment may include an imagine artifact that results in a lower quality transformation than could be obtained without the segment. In some examples, the system may discard one or more curve segment to obtain a better transformation quality.

[0152] At block 1150, the method 1100 includes registering the imaging coordinate system with the sensing coordinate system based on the transformation. Block 1130 may include the same options as block 1070 in FIG. 10. In a sense, the method 1100 includes similar techniques as method 1000 with the difference that computing the transformation need not be separated into translational and rotational transformations. That is, the optimization of the cost function may include translation. As discussed above, the separation of the tip obtained from the imaging data and the tip obtained from the shape data may be given a suitably high weight, which may result in the registration result of method 1100 approaching the registration of method 1000. On the other hand, the potentially weaker constraint on the tip of the method 1000 may yield a better registration when the estimate of the tip in the imaging data has an uncertainty above a certain value.

[0153] One or more components of the examples discussed in this disclosure, such as control system 712, may be implemented in software for execution on one or more processors of a computer system. The software may include code that when executed by the one or more processors, configures the one or more processors to perform various functionalities as discussed herein. The code may be stored in a non-transitory computer readable storage medium (e.g., a memory, magnetic storage, optical storage, solid-state storage, etc.). The computer readable storage medium may be part of a computer readable storage device, such as an electronic circuit, a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device. The code may be downloaded via computer networks such as the Internet, Intranet, etc. for storage on the computer readable storage medium. The code may be executed by any of a wide variety of centralized or distributed data processing architectures. The programmed instructions of the code may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the systems described herein. The components of the computing systems discussed herein may be connected using wired and / or wireless connections. In someexamples, the wireless connections may use wireless communication protocols such as Bluetooth, near-field communication (NFC), Infrared Data Association (IrDA), home radio frequency (HomeRF), IEEE 802.11 , Digital Enhanced Cordless Telecommunications (DECT), and wireless medical telemetry service (WMTS).

[0154] Various general-purpose computer systems may be used to perform one or more processes, methods, or functionalities described herein. Additionally or alternatively, various specialized computer systems may be used to perform one or more processes, methods, or functionalities described herein. In addition, a variety of programming languages may be used to implement one or more of the processes, methods, or functionalities described herein.

[0155] While certain examples and examples have been described above and shown in the accompanying drawings, it is to be understood that such examples and examples are merely illustrative and are not limited to the specific constructions and arrangements shown and described, since various other alternatives, modifications, and equivalents will be appreciated by those with ordinary skill in the art.

Claims

WHAT IS CLAIMED IS:1 . A tangible, non-transitory, computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: obtain, based on image data received from an imaging system, a three-dimensional representation of an anatomical structure and a portion of a flexible elongate device disposed in the anatomical structure, the portion including a tip, wherein the three- dimensional representation is in an imaging coordinate system associated with the imaging system; receive, from a sensing unit, shape data indicative of a shape of the flexible elongate device, the shape data including a position of the tip, wherein the shape data is in a sensing coordinate system associated with the sensing unit; obtain a position of the tip in the imaging coordinate system; compute a translational transformation between the imaging coordinate system and the sensing coordinate system based on the position of the tip in the sensing coordinate system and the position of the tip in the imaging coordinate system; compute, based at least in part on the obtained three-dimensional representation, one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system; compute a rotational transformation between the imaging coordinate system and the sensing coordinate system based at least in part on the shape data and the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system; and register the imaging coordinate system with the sensing coordinate system based on the translational transformation and the rotational transformation.

2. The tangible, non-transitory, computer readable medium of claim 1 , wherein the instructions further cause the one or more processors to: identify a position of a target in the imaging coordinate system; and compute the position of the target in the sensing coordinate system based at least in part on the translational transformation and the rotational transformation.

3. The tangible, non-transitory, computer readable medium of claim 1 , wherein the instructions further cause the one or more processors to determine that the shape data is received in an imaging time window.

4. The tangible, non-transitory, computer readable medium of claim 3, wherein determining that the shape data is received in the imaging time window includes: recording the received shape data in a buffer of a memory device; and identifying, within the shape data recorded in the buffer of the memory device, a portion of the shape data indicative of a low-motion period.

5. The tangible, non-transitory, computer readable medium of claim 1 , wherein computing the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system includes: computing a first curve segment and a candidate second curve segment; computing a first direction vector representative of direction of the first curve segment at a terminal point of the first curve segment; computing a filter region based at least in part on the first curve segment and on a maximum angular deviation from the first direction vector; identifying a proximal point of the candidate second curve segment; determining that the proximal point of the candidate second curve segment is within the filter region; and designating the candidate second curve segment as an accepted candidate second curve segment based at least in part on determining that the proximal point of the candidate second curve segment is within the filter region.

6. The tangible, non-transitory, computer readable medium of claim 5, wherein computing the filter region is based at least in part on a distance from the terminal point of the first curve segment.

7. The tangible, non-transitory, computer readable medium of claim 5, wherein computing the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system further includes: computing an alternative candidate second curve segment; identifying a proximal point of the alternative candidate second curve segment; determining that the proximal point of the alternative candidate second curve segment is within the filter region; designating the alternative candidate second curve segment as an accepted alternative candidate second curve segment based at least in part on determining that the proximal point of the alternative candidate second curve segment is within the filter region; anddesignating an accepted second curve segment by selecting between the accepted candidate second curve segment and the accepted alternative candidate second curve segment.

8. The tangible, non-transitory, computer readable medium of claim 7, wherein: selecting between the accepted candidate second curve segment and the accepted alternative candidate second curve segment includes: computing a second direction vector at the proximal point of the accepted candidate second curve segment; computing an alternative second direction vector at the proximal point of the accepted alternative candidate second curve segment; and computing a collinearity factor between the first direction vector and the second direction vector; and computing an alternative collinearity factor between the first direction vector and the alternative second direction vector; and selecting between the accepted candidate second curve segment and the accepted alternative candidate second curve segment is based at least in part on comparing the collinearity factor and the alternative collinearity factor.

9. The tangible, non-transitory, computer readable medium of any of one of claims 1-8, wherein computing the rotational transformation is based at least in part on minimizing a cost function indicative of overlap of the shape data and the one or more curve segments in a shared coordinate system.

10. The tangible, non-transitory, computer readable medium of claim 9, wherein the cost function is based at least in part on a weighted L2 norm of distances between a set of points in the shape data and a respective set of points in the one or more curve segments in the shared coordinate system.11 . The tangible, non-transitory, computer readable medium of any of one of claims 1-8, wherein computing the rotational transformation is based at least in part on at least one of: i) an iterative closest point algorithm, ii) a modified iterative closest point algorithm, or ill) a coherent point drift algorithm.

12. The tangible, non-transitory, computer readable medium of any of one of claims 1-8 wherein the instructions further cause the one or more processors to: obtain an indication of low registration quality;generate a prompt to select a new position of the tip on a graphical user interface displayed at a display device; obtain a new position of the tip in the imaging coordinate system; compute a new translational transformation between the imaging coordinate system and the sensing coordinate system based on the position of the tip in the sensing coordinate system and the new position of the tip in the imaging coordinate system; and register the imaging coordinate system with the sensing coordinate system based on the new translational transformation and the rotational transformation.

13. The tangible, non-transitory, computer readable medium of claim 12, wherein: computing the rotational transformation is based at least in part on minimizing a cost function indicative of overlap of the shape data and the one or more curve segments in a shared coordinate system; and obtaining the indication of low registration quality includes comparing a value of the minimized cost function to a threshold.

14. The tangible, non-transitory, computer readable medium of claim 12, wherein: obtaining the indication of low registration quality includes receiving the indication of low registration quality via the graphical user interface.

15. The tangible, non-transitory, computer readable medium of any of one of claims 1-8, wherein the instructions further cause the one or more processors to display, on a display device, overlayed images of the shape data and the one or more curve segments.

16. A system comprising: one or more processors; and tangible, non-transitory, computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: obtain, based on image data received from an imaging system, a three- dimensional representation of an anatomical structure and a portion of a flexible elongate device disposed in the anatomical structure, the portion including a tip, wherein the three-dimensional representation is in an imaging coordinate system associated with the imaging system; receive, from a sensing unit, shape data indicative of a shape of the flexible elongate device, the shape data including a position of the tip, wherein the shape data is in a sensing coordinate system associated with the sensing unit; obtain a position of the tip in the imaging coordinate system;compute a translational transformation between the imaging coordinate system and the sensing coordinate system based on the position of the tip in the sensing coordinate system and the position of the tip in the imaging coordinate system; compute, based at least in part on the obtained three-dimensional representation, one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system; compute a rotational transformation between the imaging coordinate system and the sensing coordinate system based at least in part on the shape data and the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system; and register the imaging coordinate system with the sensing coordinate system based on the translational transformation and the rotational transformations.

17. The system of claim 16, wherein the instructions further cause the one or more processors to: identify a position of a target in the imaging coordinate system; and compute the position of the target in the sensing coordinate system based at least in part on the computed translational transformation and the computed rotational transformation.

18. The system of claim 16, wherein the instructions further cause the one or more processors to determine that the shape data is received in an imaging time window.

19. The system of claim 18, wherein determining that the shape data is received in the imaging time window includes: recording at least some of the received shape data in a buffer of a memory device; and identifying, within the at least some of the shape data recorded in the buffer of the memory device, a portion of the shape data indicative of a low-motion period.

20. The system of claim 16, wherein computing the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system includes: computing a first curve segment and a candidate second curve segment; computing a first direction vector representative of direction of the first curve segment at a terminal point of the first curve segment;computing a filter region based at least in part on the first curve segment and on a maximum angular deviation from the first direction vector; identifying a proximal point of the candidate second curve segment; determining that the proximal point of the candidate second curve segment is within the filter region; and designating the candidate second curve segment as an accepted candidate second curve segment based at least in part on determining that the proximal point of the candidate second curve segment lies within the filter region.21 . The system of claim 20, wherein computing the filter region is based at least in part on a distance from the terminal point of the first curve segment.

22. The system of claim 20, wherein computing the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system further includes: computing an alternative candidate second curve segment; identifying a proximal point of the alternative candidate second curve segment; determining that the proximal point of the alternative candidate second curve segment lies within the filter region; designating the alternative candidate second curve segment as an accepted alternative candidate second curve segment based at least in part on determining that the proximal point of the alternative candidate second curve segment lies within the filter region; and designating an accepted second curve segment by selecting between the accepted candidate second curve segment and the accepted alternative candidate second curve segment.

23. The system of claim 22, wherein: selecting between the accepted candidate second curve segment and the accepted alternative candidate second curve segment includes: computing a second direction vector at the proximal point of the accepted candidate second curve segment; computing an alternative second direction vector at the proximal point of the accepted alternative candidate second curve segment; and computing a collinearity factor between the first direction vector and the second direction vector; andcomputing an alternative collinearity factor between the first direction vector and the alternative second direction vector; and is based at least in part on comparing the collinearity factor and the alternative collinearity factor.

24. The system of any of one of claims 16-23, wherein computing the rotational transformation is based at least in part on minimizing a cost function indicative of overlap of the shape data and the one or more curve segments in a shared coordinate system.

25. The system of claim 24, wherein the cost function is based at least in part on a weighted L2 norm of distances between a set of points in the shape data and a respective set of points in the one or more curve segments in the shared coordinate system.

26. The system of any of one of claims 16-23, wherein computing the rotational transformation is based at least in part on at least one of: i) an iterative closest point algorithm, ii) a modified iterative closest point algorithm, and / or ill) a coherent point drift algorithm.

27. The system of any of one of claims 16-23 wherein the instructions further cause the one or more processors to: obtain an indication of low registration quality, generate a prompt to select a new position of the tip on a graphical user interface displayed at a display device; obtain a new position of the tip in the imaging coordinate system; compute a new translational transformation between the imaging coordinate system and the sensing coordinate system based on the position of the tip in the sensing coordinate system and the new position of the tip in the imaging coordinate system; and register the imaging coordinate system with the sensing coordinate system based on the new translational transformation and the rotational transformation.

28. The system of claim 27, wherein: computing the rotational transformation is based at least in part on minimizing a cost function indicative of overlap of the shape data and the one or more curve segments in a shared coordinate system; and obtaining the indication of low registration quality includes comparing a value of the minimized cost function to a threshold.

29. The system of claim 27, wherein: obtaining the indication of low registration quality includes receiving the indication of low registration quality by way of the graphical user interface.

30. The system of any of one of claims 16-23, wherein the instructions further cause the one or more processors to display, on a display device, overlayed images of the shape data and the one or more curve segments.

31. A method comprising: obtaining, by one or more processors and based on image data from an imaging system, a three-dimensional representation of an anatomical structure and a portion of a flexible elongate device disposed in the anatomical structure, the portion including a tip, wherein the three-dimensional representation is in an imaging coordinate system associated with the imaging system; receiving, by the one or more processors and from a sensing unit, shape data indicative of a shape of the flexible elongate device, the shape data including a position of the tip, wherein the shape data is based on a sensing coordinate system associated with the sensing unit; obtaining, by the one or more processors, a position of the tip in the imaging coordinate system; computing, by the one or more processors, a translational transformation between the imaging coordinate system and the sensing coordinate system based on the position of the tip in the sensing coordinate system and the position of the tip in the imaging coordinate system; computing, by the one or more processors and based at least in part on the obtained three-dimensional representation, one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system; computing, by the one or more processors, a rotational transformation between the imaging coordinate system and the sensing coordinate system based at least in part on the shape data and the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system; and registering, by the one or more processors, the imaging coordinate system with the sensing coordinate system based on the translational transformation and the rotational transformations.

32. The method of claim 31 , further comprising:identifying, by the one or more processors, a position of a target in the imaging coordinate system; and computing, by the one or more processors, the position of the target in the sensing coordinate system based at least in part on the computed translational transformation and the computed rotational transformation.

33. The method of claim 31 , further comprising: determining, by the one or more processors, that the shape data is received in an imaging time window.

34. The method of claim 33, wherein determining that the shape data is received in the imaging time window includes: recording, by the one or more processors, at least some of the received shape data in a buffer of a memory device; and identifying, by the one or more processors, within the at least some of the shape data recorded in the buffer of the memory device, a portion of the shape data indicative of a low- motion period.

35. The method of claim 31 , wherein computing the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system includes: computing, by the one or more processors, a first curve segment and a candidate second curve segment; computing, by the one or more processors, a first direction vector representative of direction of the first curve segment at a terminal point of the first curve segment; computing, by the one or more processors, a filter region based at least in part on the first curve segment and on a maximum angular deviation from the first direction vector; identifying, by the one or more processors, a proximal point of the candidate second curve segment; determining, by the one or more processors, that the proximal point of the candidate second curve segment is within the filter region; and designating, by the one or more processors, the candidate second curve segment as an accepted candidate second curve segment based at least in part on determining that the proximal point of the candidate second curve segment lies within the filter region.

36. The method of claim 35, wherein computing the filter region is based at least in part on a distance from the terminal point of the first curve segment.

37. The method of claim 35, wherein computing the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system further includes: computing, by the one or more processors, an alternative candidate second curve segment; identifying, by the one or more processors, a proximal point of the alternative candidate second curve segment; determining, by the one or more processors, that the proximal point of the alternative candidate second curve segment lies within the filter region; designating, by the one or more processors, the alternative candidate second curve segment as an accepted alternative candidate second curve segment based at least in part on determining that the proximal point of the alternative candidate second curve segment lies within the filter region; and designating, by the one or more processors, an accepted second curve segment by selecting between the accepted candidate second curve segment and the accepted alternative candidate second curve segment.

38. The method of claim 37, wherein: selecting, by the one or more processors, between the accepted candidate second curve segment and the accepted alternative candidate second curve segment includes: computing, by the one or more processors, a second direction vector at the proximal point of the accepted candidate second curve segment; computing, by the one or more processors, an alternative second direction vector at the proximal point of the accepted alternative candidate second curve segment; and computing, by the one or more processors, a collinearity factor between the first direction vector and the second direction vector; and computing, by the one or more processors, an alternative collinearity factor between the first direction vector and the alternative second direction vector; and is based at least in part on comparing the collinearity factor and the alternative collinearity factor.

39. The method of any of one of claims 31-38, wherein computing the rotational transformation is based at least in part on minimizing a cost function indicative of overlap of the shape data and the one or more curve segments in a shared coordinate system.

40. The method of claim 39, wherein the cost function is based at least in part on a weighted L2 norm of distances between a set of points in the shape data and a respective set of points in the one or more curve segments in the shared coordinate system.41 . The method of any of one of claims 31-38, wherein computing the rotational transformation is based at least in part on at least one of: i) an iterative closest point algorithm, ii) a modified iterative closest point algorithm, and / or ill) a coherent point drift algorithm.

42. The method of any of one of claims 31-38 wherein: obtaining, by the one or more processors, an indication of low registration quality, generating, by the one or more processors, a prompt to select a new position of the tip on a graphical user interface displayed at a display device; obtaining, by the one or more processors, a new position of the tip in the imaging coordinate system; computing, by the one or more processors, a new translational transformation between the imaging coordinate system and the sensing coordinate system based on the position of the tip in the sensing coordinate system and the new position of the tip in the imaging coordinate system; and registering, by the one or more processors, the imaging coordinate system with the sensing coordinate system based on the new translational transformation and the rotational transformation.

43. The method of claim 42, wherein: computing the rotational transformation is based at least in part on minimizing a cost function indicative of overlap of the shape data and the one or more curve segments in a shared coordinate system; and obtaining the indication of low registration quality includes comparing a value of the minimized cost function to a threshold.

44. The method of claim 42, wherein: obtaining the indication of low registration quality includes receiving the indication of low registration quality by way of a graphical user interface.

45. The method of any of one of claims 31-38, further comprising: displaying, by the one or more processors on a display device, overlayed images of the shape data and the one or more curve segments.

46. A tangible, non-transitory, computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: receive, from a sensing unit, shape data indicative of a shape of a flexible elongate device, wherein the shape data is in a sensing coordinate system associated with the sensing unit; obtain, based on image data received from an imaging system, a three-dimensional representation of an anatomical structure and a portion of the flexible elongate device disposed in the anatomical structure, wherein the three-dimensional representation is in an imaging coordinate system associated with the imaging system; compute, based at least in part on the obtained three-dimensional representation, one or more curve segments indicative of the shape of the flexible elongate device, wherein computing the one or more curve segments includes: computing a first curve segment and a candidate second curve segment; computing a first direction vector representative of direction of the first curve segment at a terminal point of the first curve segment; computing a filter region based at least in part on a maximum angular deviation from the first direction vector; identifying a proximal point of the candidate second curve segment; determining that the proximal point of the candidate second curve segment is within the filter region; and designating the candidate second curve segment as an accepted candidate second curve segment based at least in part on determining that the proximal point of the candidate second curve segment lies within the filter region; and compute a transformation between the imaging coordinate system and the sensing coordinate system based at least in part on the shape data and the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system; and register the imaging coordinate system with the sensing coordinate system based on the transformation.

47. The tangible, non-transitory, computer readable medium of claim 46, wherein computing the filter region is based at least in part on a distance from the terminal point of the first curve segment.

48. The tangible, non-transitory, computer readable medium of claim 46, wherein computing the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system further includes: computing an alternative candidate second curve segment; identifying a proximal point of the alternative candidate second curve segment; determining that the proximal point of the alternative candidate second curve segment lies within the filter region; designating the alternative candidate second curve segment as an accepted alternative candidate second curve segment based at least in part on determining that the proximal point of the alternative candidate second curve segment is within the filter region; and designating an accepted second curve segment by selecting between the accepted candidate second curve segment and the accepted alternative candidate second curve segment.

49. The tangible, non-transitory, computer readable medium of claim 48, wherein: selecting between the accepted candidate second curve segment and the accepted alternative candidate second curve segment includes: computing a second direction vector at the proximal point of the accepted candidate second curve segment; computing an alternative second direction vector at the proximal point of the accepted alternative candidate second curve segment; and computing a collinearity factor between the first direction vector and the second direction vector; and computing an alternative collinearity factor between the first direction vector and the alternative second direction vector; and is based at least in part on comparing the collinearity factor and the alternative collinearity factor.

50. The tangible, non-transitory, computer readable medium of claim 46, wherein computing the transformation is based at least in part on minimizing a cost function indicative of overlap of the shape data and the one or more curve segments in a shared coordinate system.51 . The tangible, non-transitory, computer readable medium of claim 50, wherein the cost function is based at least in part on a weighted L2 norm of distances between a set of points in the shape data and a respective set of points in the one or more curve segmentsin the shared coordinate system, wherein weights of the distances decrease away from a tip of the flexible elongate device.

52. The tangible, non-transitory, computer readable medium of claim 46, wherein the instructions further cause the one or more processors to: identify a position of a target in the imaging coordinate system; and compute the position of the target in the sensing coordinate system based at least in part on the transformation.

53. The tangible, non-transitory, computer readable medium of claim 46, wherein the instructions further cause the one or more processors to determine that the shape data is received in an imaging time window.

54. The tangible, non-transitory, computer readable medium of claim 53, wherein determining that the shape data is received in the imaging time window includes: recording the received shape data in a buffer of a memory device; and identifying, within the shape data recorded in the buffer of the memory device, a portion of the shape data indicative of a low-motion period.

55. The tangible, non-transitory, computer readable medium of any of one of claims 46-54, wherein the instructions further cause the one or more processors to display, on a display device, overlayed images of the shape data and the one or more curve segments.

56. A system comprising: one or more processors; and tangible, non-transitory, computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: receive, from a sensing unit, shape data indicative of a shape of a flexible elongate device, wherein the shape data is in a sensing coordinate system associated with the sensing unit; obtain, based on image data received from an imaging system, a three- dimensional representation of an anatomical structure and a portion of the flexible elongate device disposed in the anatomical structure, wherein the three-dimensional representation is in an imaging coordinate system associated with the imaging system;compute, based at least in part on the obtained three-dimensional representation, one or more curve segments indicative of the shape of the flexible elongate device, wherein computing the one or more curve segments includes: computing a first curve segment and a candidate second curve segment; computing a first direction vector representative of direction of the first curve segment at a terminal point of the first curve segment; computing a filter region based at least in part on a maximum angular deviation from the first direction vector; identifying a proximal point of the candidate second curve segment; determining that the proximal point of the candidate second curve segment is within the filter region; and designating the candidate second curve segment as an accepted candidate second curve segment based at least in part on determining that the proximal point of the candidate second curve segment lies within the filter region; and compute a transformation between the imaging coordinate system and the sensing coordinate system based at least in part on the shape data and the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system; and register the imaging coordinate system with the sensing coordinate system based on the transformation.

57. The system of claim 56, wherein computing the filter region is based at least in part on a distance from the terminal point of the first curve segment.

58. The system of claim 56, wherein computing the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system further includes: computing an alternative candidate second curve segment; identifying a proximal point of the alternative candidate second curve segment; determining that the proximal point of the alternative candidate second curve segment lies within the filter region; designating the alternative candidate second curve segment as an accepted alternative candidate second curve segment based at least in part on determining that the proximal point of the alternative candidate second curve segment is within the filter region; anddesignating an accepted second curve segment by selecting between the accepted candidate second curve segment and the accepted alternative candidate second curve segment.

59. The system of claim 58, wherein: selecting between the accepted candidate second curve segment and the accepted alternative candidate second curve segment includes: computing a second direction vector at the proximal point of the accepted candidate second curve segment; computing an alternative second direction vector at the proximal point of the accepted alternative candidate second curve segment; and computing a collinearity factor between the first direction vector and the second direction vector; and computing an alternative collinearity factor between the first direction vector and the alternative second direction vector; and is based at least in part on comparing the collinearity factor and the alternative collinearity factor.

60. The system of claim 56, wherein computing the transformation is based at least in part on minimizing a cost function indicative of overlap of the shape data and the one or more curve segments in a shared coordinate system.61 . The system of claim 60, wherein the cost function is based at least in part on a weighted L2 norm of distances between a set of points in the shape data and a respective set of points in the one or more curve segments in the shared coordinate system, wherein weights of the distances decrease away from a tip of the flexible elongate device.

62. The system of claim 56, wherein the instructions further cause the one or more processors to: identify a position of a target in the imaging coordinate system; and compute the position of the target in the sensing coordinate system based at least in part on the computed transformation.

63. The system of claim 56, wherein the instructions further cause the one or more processors to determine that the shape data is received in an imaging time window.

64. The system of claim 63, wherein determining that the shape data is received in the imaging time window includes: recording the received shape data in a buffer of a memory device; and identifying, within the shape data recorded in the buffer of the memory device, a portion of the shape data indicative of a low-motion period.

65. The system of any of one of claims 56-64, wherein the instructions further cause the one or more processors to display, on a display device, overlayed images of the shape data and the one or more curve segments.

66. A method comprising: receiving, by one or more processors and from a sensing unit, shape data indicative of a shape of a flexible elongate device, wherein the shape data is in a sensing coordinate system associated with the sensing unit; obtaining, by the one or more processors and based on image data received from an imaging system, a three-dimensional representation of an anatomical structure and a portion of the flexible elongate device disposed in the anatomical structure, wherein the three- dimensional representation is in an imaging coordinate system associated with the imaging system; computing, by the one or more processors, based at least in part on the obtained three-dimensional representation, one or more curve segments indicative of the shape of the flexible elongate device, wherein computing the one or more curve segments includes: computing a first curve segment and a candidate second curve segment; computing a first direction vector representative of direction of the first curve segment at a terminal point of the first curve segment; computing a filter region based at least in part on a maximum angular deviation from the first direction vector; identifying a proximal point of the candidate second curve segment; determining that the proximal point of the candidate second curve segment is within the filter region; and designating the candidate second curve segment as an accepted candidate second curve segment based at least in part on determining that the proximal point of the candidate second curve segment lies within the filter region; computing, by the one or more processors, a transformation between the imaging coordinate system and the sensing coordinate system based at least in part on the shape data and the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system; andregistering, by the one or more processors, the imaging coordinate system with the sensing coordinate system based on the transformation.

67. The method of claim 66, wherein computing the filter region is based at least in part on a distance from the terminal point of the first curve segment.

68. The method of claim 66, wherein computing the one or more curve segments indicative of the shape of the flexible elongate device in the imaging coordinate system further includes: computing an alternative candidate second curve segment; identifying a proximal point of the alternative candidate second curve segment; determining that the proximal point of the alternative candidate second curve segment lies within the filter region; designating the alternative candidate second curve segment as an accepted alternative candidate second curve segment based at least in part on determining that the proximal point of the alternative candidate second curve segment lies within the filter region; and designating an accepted second curve segment by selecting between the accepted candidate second curve segment and the accepted alternative candidate second curve segment.

69. The method of claim 68, wherein: selecting between the accepted candidate second curve segment and the accepted candidate alternative second curve segment includes: computing a second direction vector at the proximal point of the accepted candidate second curve segment; computing an alternative second direction vector at the proximal point of the accepted alternative candidate second curve segment; and computing a collinearity factor between the first direction vector and the second direction vector; and computing an alternative collinearity factor between the first direction vector and the alternative second direction vector; and is based at least in part on comparing the collinearity factor and the alternative collinearity factor.

70. The method of claim 66, wherein computing the transformation is based at least in part on minimizing a cost function indicative of overlap of the shape data and the one or more curve segments in a shared coordinate system.71 . The method of claim 70, wherein the cost function is based at least in part on a weighted L2 norm of distances between a set of points in the shape data and a respective set of points in the one or more curve segments in the shared coordinate system, wherein weights of the distances decrease away from a tip of the flexible elongate device.

72. The method of claim 66, further comprising: identifying, by the one or more processors, a position of a target in the imaging coordinate system; and computing, by the one or more processors, the position of the target in the sensing coordinate system based at least in part on the computed transformation.

73. The method of claim 66, further comprising: determining, by the one or more processors, that the shape data is received in an imaging time window.

74. The method of claim 73, wherein determining that the shape data is received in the imaging time window includes: recording the received shape data in a buffer of a memory device; and identifying, within the shape data recorded in the buffer of the memory device, a portion of the shape data indicative of a low-motion period.

75. The method of any of one of claims 66-74, further comprising: displaying, by the one or more processors on a display device, overlayed images of the shape data and the one or more curve segments.

Images