In-painting and target update for intraoperative imaging

Abstract

A system receives from an imaging device a set of two-dimensional projection images associated with respective projection angles. The projection images contain projections of an anatomical structure with a flexible elongate device disposed therein, an anatomical target, and, possibly, fiducial markings. The system may reconstruct from the set of two-dimensional projection images a three-dimensional image. In some examples, prior to the reconstruction, the system may detect and in-paint pixels (within the projection images) associated with the fiducial markings, the flexible elongate device, and / or other objects. The system may enhance the reconstruction using a style transfer model. Furthermore, the system may receive shape data, in the sensor / instrument coordinate system, associated with the flexible elongate device. The system may compute a transformation between the imaging coordinate system and the instrument coordinate system, identify a target within the reconstructed image, and, using the computed transformation, update target location in the instrument coordinate system.

Description

IN-PAINTING AND TARGET UPDATE FOR INTRAOPERATIVE IMAGINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 581 ,526 entitled “IN-PAINTING AND TARGET UPDATE FOR INTRAOPERATIVE IMAGING,” filed on September 8, 2023. The entire contents of which are hereby expressly incorporated herein by reference.FIELD

[0001] The present disclosure relates to planning and / or navigating minimally invasive medical procedures and, more specifically, to in-painting and other enhancements for intraoperative imaging.BACKGROUND

[0002] Minimally invasive medical techniques are intended to reduce the amount of tissue that is damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques may be performed through natural orifices in a patient anatomy or through one or more surgical incisions. Through these natural orifices or incisions, physicians may insert minimally invasive medical instruments (including surgical, diagnostic, and / or therapeutic instruments) to reach a target tissue location. One such minimally invasive technique is to use a flexible and / or steerable elongate device, such as a flexible catheter, that can be inserted into anatomic passageways and navigated toward a region of interest within the patient anatomy.

[0003] A combination of localization sensors disposed at a flexible elongate device and intraoperative imaging can greatly aid in planning and navigating a minimally invasive procedure. Specifically, combining sensor data with intraoperative images can enable accurate determination of a position, orientation, and / or pose of the flexible elongate device within the patent anatomy. However, accessible intraoperative techniques, such as tomosynthesis based on a limited set of projections, can suffer from low image quality and reconstruction artifacts. Improving the quality of intraoperative tomosynthesis images and extracting accurate intra-operative anatomical data remains a challenge using current techniques.SUMMARY OF THE DISCLOSURE

[0004] The following presents a simplified summary of various examples described herein and is not intended to identify key or critical elements or to delineate the scope of the claims.

[0005] In some examples, a tangible, non-transitory, computer readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to obtain a first plurality of two-dimensional images depicting an anatomical structure and one or more objects. The instructions may further cause the one or more processors to compute, based at least in part on the first plurality of two-dimensional images, a second plurality of two-dimensional images by in-painting at least a portion of pixels associated with at least one of the one or more objects. Still further, the instructions may cause the one or more processors to cause a display device to display a graphical user interface depicting a visualization based at least in part on at least one of the second plurality of two-dimensional images.

[0006] In other examples, a system for visualizing patient anatomy during a medical procedure comprises a display device, one or more processors, and one or more non- transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to obtain a first plurality of two- dimensional images depicting an anatomical structure and one or more objects. The instructions may further cause the one or more processors to compute, based at least in part on the first plurality of two-dimensional images, a second plurality of two-dimensional images by in-painting at least a portion of pixels associated with at least one of the one or more objects. Still further, the instructions may cause the one or more processors to cause a display device to display a graphical user interface depicting a visualization based at least in part on at least one of the second plurality of two-dimensional images.

[0007] In other examples, a method for visualizing patient anatomy during a medical procedure comprises obtaining, by one or more processors, a first plurality of two- dimensional images depicting an anatomical structure and one or more objects. The method may further comprise computing, by the one or more processors, based at least in part on the first plurality of two-dimensional images, a second plurality of two-dimensional images by in-painting at least a portion of pixels associated with at least one of the one or more objects. Still further, the method may comprise causing, by the one or more processors, a display device to display a graphical user interface depicting a visualization based at least in part on at least one of the second plurality of two-dimensional images.

[0008] In other examples, a tangible, non-transitory, computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to obtain a plurality of two-dimensional projection images corresponding to a plurality of respective projection angles and depicting an anatomical structure and a flexible elongate device, and to identify, within each of at least two of the plurality of two-dimensional projection images, respective two-dimensional coordinates of a reference point disposed atthe flexible elongate device. The instructions may further cause the one or more processors to compute, based at least in part on the respective two-dimensional coordinates and two respective projection angles, three dimensional coordinates, in an imaging coordinate system, of the reference point, and to identify, within at least one of the at least two of the plurality of two-dimensional projection images, a curve corresponding to at least a portion of the flexible elongate device including the reference point. Still further, the instructions may cause the one or more processors to receive, from a sensing unit, shape data in a sensing coordinate system for the at least a portion of the flexible elongate device including the reference point, and to register the sensing coordinate system to the imaging coordinate system based at least in part on the received shape data and the identified curved within the at least one of the at least two of the plurality of two-dimensional projection images. Still further, the instructions may cause the one or more processors to reconstruct, based at least in part on the plurality of two-dimensional projection images corresponding to the plurality of respective projection angles, a three-dimensional image of the anatomical structure, and to identify, within the reconstructed three-dimensional image of the anatomical structure, a target. Still further, the instructions may cause the one or more processors to compute, in an instrument coordinate system and based at least in part on identifying the target and registering the sensing coordinate system to the imaging coordinate system, a location of the target.

[0009] In other examples, a system for visualizing patient anatomy during a medical procedure comprises a display device, one or more processors, and one or more non- transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to obtain a plurality of two-dimensional projection images corresponding to a plurality of respective projection angles and depicting an anatomical structure and a flexible elongate device. The instruction may further cause the one or more processors to identify, within each of at least two of the plurality of two- dimensional projection images, respective two-dimensional coordinates of a reference point disposed at the flexible elongate device, and to compute, based at least in part on the respective two-dimensional coordinates and two respective projection angles, three dimensional coordinates in an imaging coordinate system of the reference point. Still further, the instructions may cause the one or more processors to identify, within at least one of the at least two of the plurality of two-dimensional projection images, a curve corresponding to at least a portion of the flexible elongate device including the reference point. Still further, the instructions may cause the one or more processors to receive, from a sensing unit, shape data in a sensing coordinate system for the at least a portion of the flexible elongate device including the reference point, and to register the sensing coordinate system to the imagingcoordinate system based at least in part on the received shape data and the identified curved within the at least one of the at least two of the plurality of two-dimensional projection images. Still further, the instructions may cause the one or more processors to reconstruct, based at least in part on the plurality of two-dimensional projection images corresponding to the plurality of respective projection angles, a three-dimensional image of the anatomical structure, and to identify, within the reconstructed three-dimensional image of the anatomical structure, a target. Still further, the instructions may cause the one or more processors to compute, in an instrument coordinate system and based at least in part on identifying the target and registering the sensing coordinate system to the imaging coordinate system, a location of the target, and to cause the display device to display a graphical user interface depicting the computed location of the target in the instrument coordinate system.

[0010] In other examples, a method for visualizing patient anatomy during a medical procedure comprises obtaining, by one or more processors, a plurality of two-dimensional projection images corresponding to a plurality of respective projection angles and depicting an anatomical structure and a flexible elongate device. The method may further comprise identifying, by the one or more processors, within each of at least two of the plurality of two- dimensional projection images, respective two-dimensional coordinates of a reference point disposed at the flexible elongate device, and computing, by the one or more processors, based at least in part on the respective two-dimensional coordinates and two respective projection angles, three dimensional coordinates, in an imaging coordinate system, of the reference point. Still further, the method may comprise identifying, by the one or more processors, within at least one of the at least two of the plurality of two-dimensional projection images, a curve corresponding to at least a portion of the flexible elongate device including the reference point. Still further, the method may comprise receiving, by the one or more processors, from a sensing unit, shape data in a sensing coordinate system for the at least a portion of the flexible elongate device including the reference point, and registering, by the one or more processors, the sensing coordinate system to the imaging coordinate system based at least in part on the received shape data and the identified curved within the at least one of the at least two of the plurality of two-dimensional projection images. Still further, the method may comprise reconstructing, by the one or more processors, based at least in part on the plurality of two-dimensional projection images corresponding to the plurality of respective projection angles, a three-dimensional image of the anatomical structure, and identifying, by the one or more processors, within the reconstructed three- dimensional image of the anatomical structure, a target. Still further, the method may comprise computing, by the one or more processors, in an instrument coordinate system and based at least in part on identifying the target and registering the sensing coordinatesystem to the imaging coordinate system, a location of the target; and causing, by the one or more processors, a display device to display a graphical user interface depicting the identified or updated location of the target in the instrument coordinate system.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1 A depicts an example system for navigating during a medical procedure within an operating environment.

[0012] FIG. 1 B is a simplified diagram of a flexible elongate device disposed within an anatomical structure.

[0013] FIGS. 1 C, 1 D are simplified diagrams depicting an example intraoperative imaging geometry for tomosynthesis.

[0014] FIG. 1 E is a simplified three-dimensional representation of a flexible elongate device disposed within an anatomical structure within the imaging geometry of FIGS. 1C, D.

[0015] FIG. 1 F depicts a portion of a projection image of a flexible elongate device disposed within an anatomical structure overlayed with fiducial markings.

[0016] FIGS. 1G, 1 H schematically illustrates projections of an example fiducial plate at two projection angles.

[0017] FIGS. 2A-D schematically illustrate an example procedure for in-painting fiducial markings in a projection image of a flexible elongate device disposed within an anatomical structure.

[0018] FIGS. 2E-H schematically illustrate an example procedure for in-painting a flexible elongate device in a projection image of an anatomical structure.

[0019] FIG. 3A schematically illustrates an example multi-projection image acquisition geometry for a flexible elongate device disposed within an anatomical structure.

[0020] FIG. 3B, 3C are example projection images for the image acquisition geometry of FIG. 3A.

[0021] FIG. 4A depicts a rigid body in two example coordinate systems.

[0022] FIG. 4B schematically illustrates an example coordinate transformation process.

[0023] FIG. 4C depicts an example geometry for flexible elongate device extraction and coordinate registration using shape data and projection images.

[0024] FIGS. 5A, 5B schematically illustrate example processes for flexible elongate device extraction, coordinate registration and target update using shape data and projection images.

[0025] FIG. 6A schematically illustrates an example style transfer for three-dimensional images.

[0026] FIG. 6B schematically illustrates an example training process for a style-transfer machine learning model.

[0027] FIG. 7 is a simplified diagram of a medical system according to some examples.

[0028] FIG. 8A is a simplified diagram of a medical instrument system according to some examples.

[0029] FIG. 8B is a simplified diagram of a medical instrument including a medical tool within an elongate device according to some examples.

[0030] FIGS. 9A and 9B are simplified diagrams of side views of a patient coordinate space including a medical instrument mounted on an insertion assembly according to some examples.

[0031] FIG. 10 depicts a flow chart for an example method for visualizing patient anatomy during a medical procedure.

[0032] FIGS. 11 A, B depict a flow chart for another example method for visualizing patient anatomy during a medical procedure.

[0033] FIG. 12A depicts a flow chart for an example method for training a machine learning and / or deep learning model for style transfer.

[0034] FIG. 12B depicts a flow chart for an example method for visualizing patient anatomy using style transfer.

[0035] FIGS. 13A, 13B depict, respectively, outputs of tomosynthesis imaging and conebeam computed tomography (CBCT) imaging of patient anatomy.

[0036] FIGS. 14A, 14B illustrate performance of style-transfer models transferring tomosynthesis images into, respectively, a computed-tomography-style image and a CBCT- style image.

[0037] Examples of the present disclosure and their advantages are best understood by referring to the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, wherein showings therein are for purposes of illustrating examples of the present disclosure and not for purposes of limiting the same.DETAILED DESCRIPTION

[0038] In the following description, specific details are set forth describing some examples consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the examples. It will be apparent, however, to one skilled in the art that some examples may be practiced without some or all of these specific details. The specific examples disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one example may be incorporated into other examples unless specifically described otherwise or if the one or more features would make an example non-functional. In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the examples.

[0039] This disclosure describes various instruments and portions of instruments in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an object or a portion of an object in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to the rotational placement of an object or a portion of an object (e.g., one or more degrees of rotational freedom such as roll, pitch, and yaw). As used herein, the term “pose” refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of the object in at least one degree of rotational freedom (e.g., up to six total degrees of freedom). As used herein, the term “shape” refers to a set of poses, positions, and / or orientations measured along an object. As used herein, the term “distal” refers to a position that is closer to a procedural site and the term “proximal” refers to a position that is further from the procedural site. Accordingly, the distal portion or distal end of an instrument is closer to a procedural site than a proximal portion or proximal end of the instrument when the instrument is being used as designed to perform a procedure.

[0040] The disclosure generally relates to systems and methods that facilitate user (e.g., physician) planning of, and / or user navigation during, an endoluminal medical procedure. These systems and methods can provide an improvement in visualization of patient anatomy based on tomosynthesis of two-dimensional projection images from a limited set of angles. Furthermore, the improvement in tomosynthesis output can lead to improved detectability of lesions and / or other targets within the patient anatomy. Additionally, the systems and methods described in the disclosure can facilitate registration between an imaging coordinate system and a coordinate system of an instrument (e.g., a flexible elongatedevice) disposed within patient anatomy. The registration techniques described in the disclosure can improve the medical procedure accuracy and / or speed by facilitating, in turn, translating a target location from intraoperative images into the instrument coordinate system.

[0041] The systems and methods described herein improve the three-dimensional output image of tomosynthesis by, for example, in-painting fiducial markings (and / or other image elements) from two dimensional images prior to tomographic reconstruction. Fiducial markings, which may originate from a fiducial plate placed beneath, above, or around the body of the patient, can aid in determining projection angles (e.g., the angular position of the arm of an x-ray imaging apparatus) but tend to interfere with tomographic reconstruction once the angles are determined.

[0042] In-painting the fiducial markings may comprise several steps. First, for each of the two-dimensional images, the system may generate a binary value mask that zeros out pixel values associated with the fiducial markings. In some examples, the system may generate the mask for a given image solely based on the image itself. In other examples, the system may use a sequence of images from different angles to aid in determining the mask for each of the angles. To generate the masks, the system may detect fiducials based on fiducial size, fiducial shape, and / or spacings between the fiducials. The system may use a variety of signal processing techniques to compensate for partial detections of fiducial markings. For example, the system may “dilate” a mask to ensure capturing all the pixels of the fiducial markings. After removing the fiducial markings, the system may fill in the removed pixels using one or more of a variety of in-painting techniques. For example, the system may use computer vision or image processing algorithms such as a Navier-Stokes gradient-based method or a fast-marching-based method. Additionally or alternatively, the system may use machine learning (ML) for in-painting. For example, the system may use progressive inpainting, attention-based in-painting, and / or pluralistic in-painting. The system may use a variety of possible neural network structures such as an auto-encoder, a generative adversarial network (GAN), and / or a diffusion model to implement in-painting.

[0043] In some examples, systems and methods of the disclosure may apply in-painting techniques to remove an instrument (e.g., a flexible elongate device) disposed within patient anatomy. To that end, the systems and methods of the disclosure may need to identify pixels associated with the instrument within the two-dimensional projections. In some examples, a system may identify pixels associated with the instrument based at least in part on user input. For example, the system may generate on a display device a graphical user interface (GUI) displaying a plurality of projections, and the user may select in at least two of the displayed projections pixels corresponding to the same reference point disposed at theinstrument. The system may compute the coordinates of the reference point in three dimensions by tracing back from the projected reference point along the projection rays and finding the intersection of the projection rays from different projections. Furthermore, the system may identify pixels forming a continuous curve associated with a section of the instrument in at least one of the projections. In some examples, the system may generate a GUI display and a projection and prompt a user to select a plurality of points along the curve associated with the section of the instrument. In other examples, the system may automatically determine the pixels forming a curve associated with the section of the instrument using one or more image processing techniques (e.g., thresholding, segmentation, contour detection, etc.). Based on the identified pixels forming a continuous curve (e.g., a center line) associated with a section of the instrument, the system may back- project the two-dimensional curve into three dimensions of the imaging coordinate system to compute a three-dimensional curve associated with the section of the instrument. In some examples, the system may perform a reconstruction of the three-dimensional image based on projections and trace rays along the projection direction from the projection in which the pixels forming the curve are identified to find intensity maxima and / or other indications of the instrument. In other examples, the system may trace along projections of rays from the first projection image (where the pixels associated with the curve are identified) in a second projection image and identify pixels associated with the projection of the instrument onto the second projection image. In this manner, the system may reconstruct the curve associated with a section of the instrument in three dimensions based on at least two projection images, as described in more detail with reference to FIG. 4C.

[0044] In some examples, the system may compute the three-dimensional curve associated with the instrument (e.g., at least a section of the instrument) disposed within an anatomical structure based at least in part on shape data obtained from a sensing system. Because the shape data from a sensing system may be in a different coordinate system from the imaging coordinate system, a method of using the shape data may include a registration process between the two coordinate systems. Registration of the coordinate systems along with using the shape data to identify the three-dimensional curve (associated with the instrument) in imaging coordinates is discussed in more detail with reference to FIGS. 4A-C.

[0045] The three-dimensional curve associated with the instrument may serve to generate in-painting masks for the instrument. For example, the system may project the curve onto each a set of two-dimensional projection images obtained from an imaging system to generate in-painting masks. The in-painting masks may be dilated, and the in-paintingprocess may be analogous to the in-painting process for fiducial markings, as described below with reference to FIGS. 2A-H.

[0046] After obtaining a set of two-dimensional images with fiducial markings and / or the instrument removed and painted in, the system may proceed to reconstruct a three- dimensional section of patient anatomy based on the set of two-dimensional images and respective angles. The system may use back projection with suitable regularization and / or any other suitable algorithm for three-dimensional image reconstruction. The system may be configured to enhance the reconstructed image using a style transfer technique. That is, the system may be configured to generate from a tomosynthesis image an enhanced image that has the style of a computed tomography (CT) or cone beam computed tomography (CBCT) image. To that end, a method of operating the system may include training an ML model to transfer a CT / CBCT style to the tomosynthesis image. The ML model may include a convolutional neural network (CNN) (e.g., U-net), a transformer network, a GAN, and / or a diffusion model. Training may include generating simulated tomosynthesis input by taking a subset of angles from CT / CBCT data, adding suitable noise to projections and respective angle uncertainties, generating tomosynthesis data and training based on the simulated tomosynthesis data to reduce loss with respect to the ground truth based on CT or CBCT data. The system may then apply the trained style transfer model to real tomosynthesis data to reduce tomosynthesis artifacts and generate volumetric images or slices that resemble CT and / or CBCT outputs. More generally, the techniques of this disclosure may enhance an inverse image obtained from a sparse set of measurements using a style transfer model trained on a less sparse set of measurements.

[0047] The system may render on a display device a graphical user interface depicting visualization data at any stage of the process: two-dimensional images with fiducials, the images with fiducials removed, a tomosynthesis image before style transfer, and / or a tomosynthesis image after style transfer. The system may allow a user to select, e.g., via the graphical user interface, parameters and / or techniques (e.g., as briefly described above) for any stage of image processing.

[0048] In other examples, a system may enhance one or more two-dimensional images which are not associated with respective viewing angles by in-painting one or more fiducial markings within the one or more two-dimensional images. Still in other examples, the system may enhance a tomosynthesis image using style transfer (e.g., transferring a CT / CBCT style to the tomosynthesis image). Broadly speaking, fiducial in-painting need not be associated with tomosynthesis or style transfer, and style transfer enhancement of tomosynthesis or other imaging modalities need not be associated with fiducial in-painting(e.g., can apply to methods where the system determines viewing angles without the use of fiducial plates).

[0049] The techniques for coordinate registration between the instrument coordinate system and the imaging coordinate system, by themselves or in combination with the enhancements of tomosynthesis images, may aid (as described, for example, with reference to FIGS. 5A, B) in accurately determining a location of a target within patient anatomy and, thereby, increase speed and / or accuracy of a medical procedure.

[0050] FIG. 1 A depicts an example system 100 for navigating during a medical procedure within an operating environment 101. The system 100 may obtain images from a portion of the operating environment 101 disposed within a field of view F (approximately demarcated by dashed lines) of an imaging unit 110. To that end, the system 100 may be in communicative connection with the imaging unit 110. Additionally, the system 100 may obtain data from one or more sensors disposed within the operating environment. To that end, the system 100 may be in communicative connection with the one or more sensors via a communicative connection with a sensing unit 115.

[0051] The system 100 includes a processing unit 120 and a display unit 130 in communicative connection with each other. Although in FIG. 1 A the imaging unit 110 and the sensing unit 115 are depicted as being distinct from the system 100, in other examples, the system 100 may include the imaging unit 110 and / or the sensing unit 115. In any case, one or more processors of the processing unit 120 of the system 100 may be configured to receive images and / or processed image information from the imaging unit 110 and to receive data from the one or more sensors by way of the sensing unit 115.

[0052] Throughout the disclosure, the descriptions of example operations performed by the processing unit 120 below are to be understood to be executed by the one or more processors of the processing unit 120. In some examples, the one or more processors may include hardware specifically configured (e.g., hardwired or programmable) to carry out at least a portion of the example operations described in this disclosure. Additionally or alternatively, the one or more processors may be configured to carry out at least a portion of the example operations described in this disclosure by carrying out a set of software instructions. To that end, the system 100 may include or be communicatively connected to a tangible, non-transitory, computer readable medium. The medium may store instructions which, when executed by the processing unit 120, perform any one or more of the example operations described below. For example, the instructions may cause the processing unit 120 to perform image processing operations on the images received from the imaging unit 110 and / or to perform computations (e.g., for coordinate registration) based on the datareceived by way of the sensing unit 115. Furthermore, the instructions may cause the processing unit 120 to cause the display unit 130 to display, via a GUI, information based on the processing of images received from the imaging unit 110 and the processing of data received by way of the sensing unit 115. For example, the processing unit 120 may send the information, or send data representing the entire GUI including the information, to the display unit 130.

[0053] An operator (e.g., a physician, another medical practitioner, or a fully-automated robotic surgery system) of a medical system may use the information displayed at the display unit 130 to perform a medical procedure (e.g., endoscopy, biopsy, pharmacological treatment, and / or treatment, such as ablation). During a medical procedure, the operator may control a flexible elongate device 140 inserted through an orifice O (or through a suitable surgical incision) into an anatomical structure A of a patient P disposed at a table T. For example, the medical procedure may include navigating the flexible elongate device 140 (indicated with solid lines outside and dashed lines inside the patient P) toward an ROI R within the anatomical structure A with the aid of information displayed at the display unit 130. The ROI R, for example, may be a designated procedure site for visual examination, biopsy, treatment, or any other medical procedure. Throughout the disclosure, the ROI R may be referred to as a region or a target.

[0054] One or more fiducial may be disposed at (e.g., physically contacting, integrated within, fixedly attached to, or removably attached to in a manner that, during operation / use, forms a rigid relationship with) the flexible elongate device 140. The fiducials are configured to be visible in images obtained by the imaging unit 110 to thereby enhance visibility of the flexible elongate device 140 and / or to aid in identifying certain points along the device 140. To that end the fiducials may include elements of a variety of materials and / or structures such as metals, plastics, etched glass, dyes, radioactive or fluorescent markings, confined fluids (e.g., bubbles), etc. At least some of the elements of the fiducial 142 may be integrated (e.g., etched, deposited, painted, or otherwise fixedly attached) onto the flexible elongate device 140. Additionally or alternatively, the fiducials may include elements removably disposed at the flexible elongate device 140. For example, the fiducials may be integrated onto a removable structure, such as a sleeve or a stylet, which in turn may be removably attached to the flexible elongate device 140.

[0055] One or more sensors may also be disposed at the flexible elongate device. The sensors may be mechanical sensors, optical sensors, electromagnetic (EM) sensors, or any other suitable sensors. The sensors may be integrated into the flexible elongate device 140, or removably attached to the flexible elongate device 140. The sensors may be configured to communicate with the sensing unit 115.

[0056] In some examples, the sensors are active sensors, configured to transmit electromagnetic (e.g., optical, radio, low-frequency magnetic) or radioactive radiation. The sensing unit 115 may include components to receive the radiation emitted by the sensors and triangulate or trilaterate locations of the sensors in the sensor system coordinates. The sensor system coordinates may be the coordinates of the robotic-assisted system configured to manipulate, control, or guide the flexible elongate device 140. As such, the sensor system coordinates may be referred to as instrument coordinates.

[0057] In other examples, the sensors are passive and emit no radiation. The passive sensors may sense radiation emitted by components of the sensing unit 115 disposed within the operating environment 101. For example, in an EM sensing system, one or more electrified coils may be disposed within the operating environment 101 to generate static or dynamic magnetic fields. The sensors may be configured to pick up variations in the dynamic magnetic fields or, as the sensors move, variations in the sensed static or quasistatic magnetic field and convert the sensed variations into electrical currents that are received by the sensing unit 115. In turn, the sensing unit 115 may compute indications of positions of the sensors from the received electrical currents.

[0058] The sensors may be fiberoptic sensors disposed along the length of the flexible elongate device. The fiberoptic sensors may include Bragg gratings and / or materials to enhance non-linear scattering. The fiberoptic sensors may be configured to change spectral reflectivity based on material strain. Such sensors may scatter light emitted, for example, by the sensing unit 115 in a manner that indicates locations and degrees of bends in the flexible elongate device 140.

[0059] In some examples, sensors may include accelerometers, gyroscopes, and / or magnetometers. Furthermore, the point sensors may include inertial measurement units (IMUs) that combine multiple sensors (e.g., accelerometers, gyroscopes) and / or inertial and magnetic measurement units (IMMUs) that combine multiple sensors (e.g., accelerometers, gyroscopes, magnetometers). IMUs and / or IMMUs may generate signals indicative of orientation of the flexible elongate device at a given point with respect to gravitational field and / or magnetic field (e.g., of Earth). In some examples, additional magnetic fields may be introduced to the sensor environment. Additionally or alternatively, IMUs and / or IMMUs may generate signals indicative of motion of the flexible elongate device (e.g., caused by motion of an anatomical structure due to breathing and / or other factors, and / or independent motion of the flexible elongate device within the anatomical structure). In some examples, the sensing unit 115 may combine indications of orientation (e.g., up to three degrees of freedom) from IMUs with indications of position from other (e.g., EM) sensors to generate more complete data indicative of pose of the flexible elongate device. Furthermore, in someexamples, the sensing unit 115 may combine data from sensors in multiple sensor coordinate systems. The sensing unit 115 and / or the processing unit 120 may register multiple sensor coordinate systems with each other.

[0060] Generally, the system 100 may use any suitable combinations of the sensors described above to obtain shape data indicative of shape of the flexible elongate device 140 (e.g., pose, including location and orientation, at different points along the length of the flexible elongate device 140).

[0061] FIG. 1 B is a simplified diagram of the flexible elongate device 140 disposed within the anatomical structure A. FIG. 1 B is included to give an expanded and more detailed view of a portion of the operating environment 101 disposed within the field of view F. The anatomical structure A may be a lung of the patient P. The flexible elongate device 140 may be inserted into and navigated by the operator toward the region R (e.g., target of the medical procedure), for example, for the purpose of investigating or treating a pathology in the region R. The techniques described in the present disclosure can facilitate the navigation process by generating and displaying timely and accurate imaging and sensing of the flexible elongate device 140 and imaging of the anatomical structure A. These techniques incorporate location, pose, and / or shape data in one coordinate system obtained using the sensors and location, pose, and / or shape data obtained using imaging in another coordinate system. The combined data may improve speed, accuracy, reliability and / or safety of a medical procedure. For example, the processing unit 120 may combine the data from the imaging unit 110 and / or the sensing unit 115 to quickly and accurately determine the position, orientation, and / or pose of at least a portion of the flexible elongate device 140 within reference to the anatomical structure A of the patient P and, particularly, the ROI or target R.

[0062] Furthermore, the processing unit 120 may generate a graphical user interface (GUI) or update GUI data for display on the display unit 130 to aid an operator with the medical procedure. In some examples, the processing unit 120 may generate data and / or control signals for a control unit of a robotic system configured to manipulate and / or navigate the flexible elongate device 140. Additionally or alternatively, the processing unit 120 may be configured to generate, based on the combined imaging and sensing data, one or more alerts. The alerts may include, for example, an alert indicating proximity to the region R, an alert indicating a potential navigation error, and alert indicating that confidence in location of the tip of the flexible elongate device 140 fell below a threshold level, etc. It should be noted that combining intra-operative data from the imaging unit 110 and the sensing unit 115 entails registering two coordinate systems with respect to each other, as described with reference to FIGS. 4A-B below.

[0063] Generally, a suitably configured imaging unit (e.g., imaging unit 110) may generate the intra-operative imaging data. The intra-operative imaging data may include, for example, computed tomography (CT), particularly cone-bean computed tomography (CBCT) data. To that end, the imaging unit may include a C-arm CBCT imaging system, and / or magnetic- resonance imaging (MRI) data. Additionally or alternatively, the intraoperative imaging data may be obtained using thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging or any other suitable imaging technique.

[0064] In the present disclosure, particular attention is given to three-dimensional (3D) image reconstruction from two-dimensional (2D) projection images. Such intra-operative imaging data may include fluoroscopy X-ray data (e.g., generated by a C-arm X-ray device), and, particularly, tomosynthesis data (reconstructed from 2D X-ray images, such as fluoroscopy images, into a 3D volume),

[0065] FIGS. 1C, D are simplified diagrams depicting an example intraoperative imaging geometry for tomosynthesis. As in FIG. 1C, the patient P is disposed within the operating environment 101. An X-ray source 150 (which may be included in the imaging unit 110) may be disposed at a location (e.g., C1 , C2, C3, etc.) along a circular arc C and directed toward a diametrically disposed detector 155 (which, likewise, may be included in the imaging unit 110) on the opposite side of the patient P. The circular arc C may lie in a plane orthogonal to an axis A disposed lengthwise through the patient P, and the center of the arc C may be disposed at the axis A. More generally, the axis A need not run lengthwise through the patient P. Generally, the axis of rotation for the arc C need not run through the patient, but may be below or above the patient P and at any suitable angle with respect to the body of the patient P. Furthermore, the arc C need not be circular to apply the techniques of the disclosure. In some examples, the arc C may be determined by the movement of a C-arm, e.g., of a fluoroscopy imaging device. The system, based on the geometry of FIG. 1 C may generate projections from a limited set of projection angles (e.g., 120°, 100°, 110°, 90°, 80°, 70°, 60°, or any other suitable span).

[0066] Besides passing through the patient P, X-rays emanating from the X-ray source 150 may pass through a fiducial plate 160 disposed, for example, under the patient P (or any other suitable location at or near the body of the patient P). In FIG. 1C, the fiducial plate 160 is shown separately to illustrate example fiducial markings disposed, as an example, as a grid throughout the plate 160. The fiducial markings may be high X-ray density spheres and / or cylinders (with regular or varying diameters of 1 , 2, 5, 10 mm or any other suitable size) placed throughout the plate 160 at regular or varying spacing of 10, 20, 30, 40 mm or any other suitable spacing. Other fiducial markings, such as lines squares, or other suitablemarkings that have high X-ray attenuation or scattering may be included in the plate 160. As described in more detail below, the system 100 may use the fiducial plate 160 to determine projection angles for 2D images (e.g., fluoroscopy images) detected by the detector 155 of the imaging unit 110.

[0067] FIG. 1 D defines an example imaging coordinate system. The axis A may be referred to as the x-axis. The orthogonal horizontal axis may be referred to as the y-axis. Finally, the vertical axis may be referred to as the z-axis. Planes Sx, Sy, and Sz are, respectively, orthogonal to the x-axis, the y-axis, and the z-axis, and may be referred to, respectively, as axial, sagittal, and coronal planes. It should be noted that the axial plane Sx is parallel to the plane of the arc C and does not correspond to any possible projection angle. In contrast, the coronal plane Sz corresponds to the projection acquired with the X- ray source 150 at the apex of the curve C. The range of angles with respect to the apex of the curve C may not permit a projection corresponding to the sagittal plane, which would require an angle of ±90° with respect to vertical. The fluoroscopy acquisition angles may lie, for example, between ±60 “with respect to vertical, subtending an arc that may be less than 120°. A reconstructed tomosynthesis image, on the other hand, may be presented in slices parallel to any one or more of the three planes Sx, Sy, and / or Sz.

[0068] FIG. 1 E is a simplified three-dimensional representation of a flexible elongate device 140 disposed within the anatomical structure A within the imaging geometry of FIGS. 1C, D. It is included to help visualize projections acquired by the tomosynthesis imaging system illustrated in FIG. 1 C with respect to the coordinate system of FIG. 1 D. The flexible elongate device 140 may have higher X-ray density than the surrounding anatomical structure as may be seen in an example projection in FIG. 1 F.

[0069] FIG. 1 F depicts a portion of a projection image 180 of a flexible elongate device 141 (which may be device 140) disposed within an anatomical structure (e.g., anatomical structure A) overlayed with fiducial markings (e.g., from the fiducial board 160). The flexible elongate device 141 appears darker than the surrounding tissue, but may be converted, in the negative, to the high-intensity portion of the image. The high X-ray density fiducial marks (e.g., mark 161 ) are highly visible in the image 180.

[0070] FIGS. 1G, H schematically illustrates projections (i.e., projected images) of the example fiducial plate 160 at two projection angles. One can assume, for example, that the plate 160 has fiducial markings arranged in a regular square grid, with identical spacing along two directions. In FIG. 1G, the projection angle may then be 0° with respect to the z- axis, i.e., normal to the plate, resulting in a regular grid of fiducial markings within the projected image of the plate 160. In FIG. 1 H, on the other hand, the projection angledeviates from the normal, resulting in the columns of fiducial markings being closer that the rows, as well as other distortions. Certain distortions may arise from the projection angle deviating from the normal with respect to the two symmetry axes (represented by dashed lines in FIGS. 1 G, H) of the fiducial plate 160. The projection in FIG. 1 H may additionally have a perspective distortion resulting, for example, from slightly nonparallel X-rays admitted by the X-ray source 150.

[0071] The system 100 may use the geometric distortions in projections of the fiducial plate 160 to determine the X-ray incidence angle and, therefore, the projection angle and the position of the X-ray source 150 along the arc C. In some examples, the system 100 may rely on geometric formulas, such as the spacing of fiducial elements in a given direction being proportional to the cosine of the respective angle, to determine the projection angle. Additionally or alternatively, the system 100 may obtain and use calibration data from known projection angles.

[0072] FIGS. 2A-D schematically illustrate an example procedure for in-painting fiducial markings in a projection image of a flexible elongate device disposed within an anatomical structure. FIG. 2A schematically illustrates a projection image with the anatomical structure A, having an ROI or target R, a flexible elongate device 140, and a grid of fiducial markings. FIG. 2A can be thought of as a schematic example of the projection image 180 in FIG. 1 F.

[0073] One part of the in-painting process performed by one or more processors of a system (e.g., by the processing unit 120 of the system 100) may include identifying the fiducial markings in a projection image. To that end, the system may use blob detection, spatial frequency analysis, circular Hough transforms, or any other suitable image processing algorithms. In some examples, the system may use machine learning models to detect fiducial markings. The system may identify fiducial markings for each individual projection or, in some examples, may use correlations within a sequence of projections (e.g., within a fluoroscopy video) to identify the fiducial markings. Additionally or alternatively, the system may use prior information about the markings, such as size, shape, spacing to aid in detection and identification of the fiducial markings.

[0074] Another part of the in-painting process performed by the one or more processors of the system includes generating a mask for masking and removing fiducial markings from projection images. An example mask is illustrated in FIG. 2B. In some examples, the mask may be based on segmented fiducial markings in a projection image. For example, the segmented markings may be dilated to include pixels around the segmented regions, compensating for possible segmentation errors that resulted in missed pixels. In other examples, detecting fiducial markings may result in identifying the center points of themarkings and, subsequently, using prior knowledge about the markings to generate the mask. That is, if the system identifies the projection angle and the center points of the markings in FIG. 2A, the system may generate the mask in FIG. 2B based on prior knowledge about size, shape, and / or spacing of the markings.

[0075] Another part of the in-painting process performed by the one or more processors of the system includes removing or zeroing out pixels from a projection image based on the generated mask, as illustrated in FIG. 2C. The removal of the pixels leaves the values of the pixels unknown, zero, or any other suitable value, depending on algorithms and data structures used in in-painting of projection images. In any case, the resulting gaps of information within a projection image are schematically illustrated in FIG. 2C. The system may fill the gaps using a suitable in-painting technique, resulting in an in painted image schematically illustrated in FIG. 2D. The result of in-painting the fiducial markings may preserve, as much as possible, the information pertaining to the projection of the anatomical structure A including the ROI or target R as well as the projection of the flexible elongate device 140.

[0076] In-painting techniques may include explicitly-defined image processing algorithms and / or be based on machine learning models (e.g., trained on projection image data). The algorithms may include fluid-dynamics-based algorithms, such as a Navier-Stokes gradientbased method and / or diffusion-based methods. Additionally or alternatively, the in-painting algorithm may be based on a fast marching method. ML-based techniques may include progressive in-painting, attention-based in-painting, and / or pluralistic-based in-painting. Progressive in-painting fills images in a step-wise manner based on surrounding pixels or structures. Attention-based in-painting considers the information from distant spatial locations. Pluralistic in-painting generates multiple results for a single image. Additional information, such as correlation among projection images in a sequence (or other suitable considerations) may guide the system’s decision regarding the choice of results generated by pluralistic in-painting. From the network structure perspective, and ML model for inpainting may be based on an auto-encoder, a variational water-encoder (VAE), a generative adversarial network (GAE), or a diffusion model.

[0077] In some examples, the system may be configured to in-paint other aspects and / or elements of projection images, such as spurious images of objects in the operative environment and / or an instrument (e.g., a flexible elongate device) disposed within patient anatomy.

[0078] FIGS. 2E-H schematically illustrate an example procedure for in-painting a flexible elongate device in a projection image of an anatomical structure. Generally, the system mayimplement the in-painting process for a flexible elongate device 240 (which may be the flexible elongate device 140) disposed within the anatomical structure A which includes the ROI or target R in a similar sequence to in painting fiducial markings discussed above. FIG. 2E illustrates the original projection image, while FIG. 2F illustrates a mask 241 corresponding to the flexible elongate device 240 within the projection image. Example processes for generating the mask 241 are discussed in more detail with reference to FIG. 4C. FIG. 2G illustrates the projection image of FIG. 2E with the pixels associated with the mask 241 removed from the image (as discussed with reference to FIG. 2C), resulting in gaps G within the projection image of the anatomical structure A. FIG. 2H illustrates the projection image of FIG. 2E with the flexible elongate device 240 removed and the gaps G filled in using, for example, the in-painting techniques discussed above.

[0079] FIG. 3A schematically illustrates an example multi-projection image acquisition geometry for a flexible elongate device 340 (which may be the flexible elongate device 140 or 240) disposed within the anatomical structure A. The geometry is analogous to the one displayed in FIG. 1C above, with the anatomical structure A replacing the patient P to more clearly illustrate the generation of projection images of the anatomical structure A. For example, one projection image of the anatomical structure A and the flexible elongate device 340 may correspond to the X-ray source 150 at the location C1 and the detector 155 at the corresponding location D1 . Another projection image of the anatomical structure A and the flexible elongate device 340 may correspond to the X-ray source 150 at the location C2 and the detector 155 at the corresponding location D2.

[0080] FIG. 3B, C are example projection images for the image acquisition geometry of FIG. 3A. For example, the image 301 may correspond to the X-ray source at the location C1 , while the image 302 may correspond to the X-ray source at the location C2. A processing unit (e.g., processing unit 120) of a system (e.g., system 100) may cause a display device (e.g., display unit 130) to generate a GUI and to display within the GUI the projection image 301 and / or the projection image 302.

[0081] The system may identify within the projection image 301 a reference point 310. The reference point 310 may be the tip of the flexible elongate device 340 or another suitable reference point (e.g., identifiable by a fiducial disposed at the flexible elongate device 340, as described above with reference to FIG. 1 A). In some examples, the system may identify the reference point 310 automatically. In other examples, the system may prompt a user to identify the reference point 310 within the displayed projection image 301 , using for example, the GUI generated at the display. Furthermore, the system may identify a reference point 320 within the projection image 302, which may correspond to the same physical point at the flexible elongate device 340 as the reference point 310. The systemmay identify the point 320 automatically or based on user input, as described above. Furthermore, the system may identify within the projection image 302 additional points 322, 324 disposed at the flexible elongate device 340. Identifying the points 310-324 may include identifying two-dimensional coordinates within their respective projection images (e.g., images 301 , 302). The system may use the identified points 310-320 (e.g., the identified two- dimensional coordinates) to identify a curve corresponding to at least a portion of the flexible elongate device 340 within at least one of the projection images (e.g., image 302). Furthermore, the system may use the identified points 310-320 and / or the identified two- dimensional curve to reconstruct a corresponding portion of the flexible elongate device 340 in three dimensions. Additionally or alternatively, the system may use at least two projection images (e.g., the projection images 301 , 302) to register, using shape data received from a sensing system (e.g., the sensing unit 1 15), an imaging coordinate system with a sensing or instrument coordinate system as described with reference to FIGS. 4A-C.

[0082] FIG. 4A depicts a rigid body 402 in two example coordinate systems to illustrate the concept of registration between two coordinate systems. A first coordinate system 404 may correspond to coordinates of shape data obtained or generated, for example, by the sensing unit 1 15. The sensing unit 115 may be a part of a robotic unit configured to actuate and / or manipulate the flexible elongate device 140. The first coordinate system 404 may therefore represent the coordinate system of the robotic unit. A second coordinate system 406 may correspond to coordinates of images obtained or generated by an imaging system, e.g., the imaging unit 110.

[0083] The rigid body 402 may represent a portion (e.g., a segment of length) of the flexible elongate device (e.g., flexible elongate device 140, 240, or 340). Although the flexible elongate device 140 is flexible, a short portion (e.g., an infinitesimal segment) may be considered rigid for all practical purposes.

[0084] A state (e.g., position and orientation) of the rigid body 402 in three-dimensional space can be described using the first coordinate system 404 and / or the second coordinate system 406. A rigid body without symmetries (e.g., body 402) has six degrees of freedom (6 DOFs) and its position and orientation can be described with six coordinates. In the first coordinate system 404, the rigid body 402 may have coordinates (x, y, z, 0, <j>, a), where x, y and z are position coordinates of a center 408 of the rigid body 402 along the axes of the first coordinate system 404 with respect to origin O. In other examples, the position coordinates may be designated for any point within the rigid body 402, or, in fact, any point in a rigidly defined geometric relationship to the body 402. Coordinates 0, <j> and a can describe orientation of the body 402 with the aid of an orientation vector 409, which, in the example of FIG. 4A, originates in the center 408 and goes through the middle of one of thefacets of the rigid body 402. For example, 0 may be an elevation angle with respect to z- axis, <|> may be an azimuthal angle parallel to the xy-plane, and a may be the angle of rotation of the rigid body 402 around the orientation vector 409. Alternatively, the three orientation coordinates may be roll, pitch and yaw of the rigid body 402 with respect to any suitable reference direction. Analogously to the coordinates (x, y, z, 0, <|), a) of first coordinate system 404, coordinates (x’, y’, z’, 0’, <|>’, o’) of the second coordinate system 406 describe position (x’, y’, z’) with respect to origin O’ and orientation (0’, <|>’, o’) with respect to, for example, z’-axis of the rigid body 402. Registering the first coordinate system 404 with the second coordinate system 406, at least in the vicinity of the rigid body 402, includes finding a mapping (e.g., a transformation, a mathematical relationship, etc.) at least between the coordinates (x, y, z, 0, <|), a) and the coordinates (x’, y’, z’, 0’, <()’, o’). Another way of looking at the registration, is that the mapping defines a corresponding point (u, v, w) in the vicinity of position (x, y, z) for any point (u’, v’, w’) in the vicinity of position (x’, y’, z’).

[0085] In the discussion above, it is assumed that the first coordinate system 404 and the second coordinate system 406 have the same scale and the transformation from the first coordinate system 404 to the second coordinate system 406 is rigid. In some examples, however, the mapping from the first coordinate system 404 to the second coordinate system 406 may include one or more scaling factors for the axes. Thus, the mapping may include three translation variables, three rotation variables, and / or three scaling variables. Furthermore, each of the variables may depend on position and the mapping may include deformations.

[0086] In some examples, each of the coordinate systems 404 and 406 are independently calibrated to have accurate and consistent scaling within a shared operating volume. Coordinate registration process may then be defined in terms of three translation constants and three rotation constants for the shared operating volume. In other examples, gradual variations in scaling within at least one of the coordinate systems 404 and / or 406 may necessitate use of up to three translation variables and up to three rotation variables, each a function of location within the shared operating volume.

[0087] FIG. 4B schematically illustrates an example coordinate transformation process. In one example, a processing unit (e.g., the processing unit 120) may obtain or determine a position and an orientation of a rigid body (e.g., rigid body 402) within the first coordinate system 404 as (x, y, z, 0, <|), a) and the second coordinate system 406 as (x’, y’, z’, 0’, <|>’, o’). The processing unit may then generate a mapping, M, between the two coordinate systems 404 and 406. The processing unit may be configured to map a new position (u’, v’, w’) within the second coordinate system 406 on to a corresponding position (u, v, w) within the first coordinate system 404. Additionally or alternatively, the processing unit may be configuredto map coordinates from the first coordinate system 404 to the second coordinate system 406. The mapping may be only valid in a region around (x, y, z). By collecting rigid bo MH C CO '7d3y coordinates in the two coordinate systems 404 and 406, the processing unit may extend the validity of mapping over any portion of a shared operating volume of the two coordinate systems 404 and 406. For example, the processing unit may implement the coordinate registration as a linear mapping: 0 sin / ?' 1 0 O ' u'' 1 0 0 cos a — sin a v' +0 cos / 3. .0 sin a cos a . .w'. where a, p and y are rotation parameters (e.g., roll, pitch and yaw), Sn, S22 and S33 are scaling parameters (which may be unity, as discussed above), and di, d2 and ds are displacement factors. Thus, with scaling factors, there may be nine mapping parameters, and six mapping parameters when scaling can be ignored. As discussed above, the linear mapping may be a function of the input position coordinates (u’, v’, w’). For example, the processing unit may store and / or access a lookup table find an entry for mapping parameters corresponding to the input position coordinates. Because the lookup table can only have a limited number of recorded input coordinates (herein, recorded coordinates), the system may use the entry corresponding to the recorded coordinates nearest to the input coordinates. Alternatively, the system may interpolate mapping parameters corresponding to a set of recorded coordinates near the input coordinates. In other examples, the processing unit may store and / or access a polynomial, spline, or another suitable fit function relating input coordinates to the mapping parameters. The mapping may relate the shape data for the flexible elongate device obtained from a sensing system and the flexible elongate device shape data in the imaging coordinate system as discussed with reference to FIG. 4C. Furthermore, the system may use the mapping to update a target (e.g., target R) position within the instrument coordinate system, as described with reference to FIG. 5A.

[0088] FIG. 4C depicts an example geometry for flexible elongate device extraction and coordinate registration using shape data and projection images. Two-dimensional projections 410, 420 of a three-dimensional curve 430 representative of at least a portion of a flexible elongate device (e.g., flexible elongate device 140, 240, or 340). The projections 410, 420 (which may be referred to as projected curves 410, 420) may be analogous to and / or representative of the two-dimensional projections of the flexible elongate device 340 in projection images 301 , 302. The projections 410, 420 are depicted on orthogonal planes - the xz-plane and the xy-plane, respectively. Although orthogonal projections may be obtained (e.g., at ±45°) in the imaging geometries described above, the discussion with reference to FIG. 4C can be generalized to non-orthogonal projections.

[0089] The three-dimensional image and coordinate system, defined by the x-, y-, and z- axes of FIG. 4C may be defined with respect to the origin located at a reference point disposed at the flexible elongate device. For example, the reference point 310 (which may be a tip of the flexible elongate device 340) in the projection image 301 and the reference point 320 in the projection image 302 may be projections of the same reference point along the flexible elongate device 340 and may define the origin, without any loss of generality, of the three-dimensional imaging coordinate system. Thus, as depicted in FIG. 4C, the projections 410, 420 and the three-dimensional curve 430 representation of the section of the flexible elongate device all intersect at the origin of the coordinate system.

[0090] A system (e.g., system 100) may be configured to reconstruct the curve 430 based on two-dimensional projection images which include projected curves 410, 420. To that end, the system may first identify within at least one of the projection images (e.g., corresponding to the projection onto the xy-plane) a projected curve (e.g., the projected curve 420). When identified, the two-dimensional curve 420 may be represented as a centerline of the projection of the flexible elongate device. Identification of the centerline curve within a projection image may, in some examples be based on at least in part identifying several points (e.g., points 320, 322, and 324) along the curve. The system may identify the points along the curve based on user input via a GUI displaying the respective projection or based on automatic extraction of the two-dimensional curve corresponding to the flexible elongate device from the respective projection image using suitable image processing algorithms.

[0091] In some examples the system may extract a two-dimensional curve (e.g., curve 410) in a second projection image based on the extracted two-dimensional curve (e.g., curve 420) in the first projection image. The curves in the two projection images necessarily share the same coordinate along the axis defined by the intersection of the two projection planes (e.g., the x-axis with respect to the curves 410, 420). Consequently for a given point along one two-dimensional projected curve one can find a corresponding point on the second two- dimensional projected curve. For example, a point on the curve 420 may be projected along a projection ray 462 to the x-axis. The system may find the corresponding point on the curve 410 by tracing along a projection ray 464 to find the intersection with the curve 410. The system may identify the intersection as a high or a low (depending on the format of the projection image) intensity point in the projection in the projection image that intersects the projection ray 464. In an example of another point along the curve 420 projected onto the x- axis by ray 466, the corresponding gray 468 intersects the curve 410 at at least two points. The system may resolve the ambiguity by selecting the point that would be contiguous with previous points detected along the curve 410.

[0092] Once the system identifies two corresponding points in the projection images, the corresponding point in three dimensions may be found, e.g., at an intersection 470 of back- projected rays emanating from the two corresponding points in two-dimensional projection images along the respective projection angles. In some examples, the system may use more than two projections to identify corresponding points in each of the projections and back project the identified points to find the correspondent point in three dimensions. In the presence of measurement uncertainties, the back projected rays may not intersect. The system may then find a point in three dimensions that minimizes a metric (e.g., root mean square or L2-norm) of distances between the three-dimensional point in the back projection rays. In the manner described above, the curve 430 in three dimensions may be reconstructed from the curves 410, 420 in two-dimensional projections.

[0093] In other example, the system may reconstruct the curve 430 in three dimensions based on the curve 420 (or curve 410) by first performing a reconstruction of the imaging volume based on the full set of two-dimensional projections. The curve 430 in the resulting three-dimensional reconstructed image may be computed by back-tracing or back-projecting along rays corresponding to the projection angle from points along the curve 420. The system may identify corresponding points along the curve 430 based on suitable intensity thresholds and continuity requirements in a manner similar to identifying corresponding points on the two-dimensional curve 410, as discussed above.

[0094] In some examples, the system may use the reconstructed curve 430 to register the imaging coordinate system with a sensing coordinate system. To that end the system may receive shape data from a sensing unit (e.g., sensing unit 115) and identify, within the shape data, sensing system coordinates corresponding to the reference point (e.g., tip of the flexible elongate device) defining the origin of the imaging coordinate system in FIG. 4C. The first step of registration may be defining the translation portion of the transformation (e.g., transformation M described with reference to FIG. 4B), placing the reference point in at the common origin of the coordinate systems registered by the translation. The shape data from the sensing system may then be represented by the dashed-line curve 480, assuming the same scale in the two coordinate systems, as discussed above. The remaining portion of the registration may include finding roll, pitch, and yaw angles that align the shape data curve 480 to the curve 430. Although perfect alignment may not be possible, due to noise and distortions in shape data received from the sensing unit and computed based on the imaging data from the imaging unit, an optimization algorithm may minimize a cost function indicative of misalignment or misregistration. For example, the system may compute the registration transformation between the shape data 480 and the curve 430 in the leastsquare sense with or without regularization accounting for prior transformation data and / orother suitable constraints. Additionally or alternatively, the system may minimize a weighted least-square difference, valuing the alignment near the origin more heavily than the alignment away from the origin. Valuing the alignment near the origin more heavily may be particularly applicable to examples where the reference point at the origin of the registered coordinate systems in the tip of the flexible elongate device when the tip is in proximity to an anatomical target (e.g., target R).

[0095] In some examples, the registration between the imaging and sensing coordinate systems need not follow the extraction of the curve 430 in three-dimensions. Instead, the system may use the shape data 480 to simultaneously extract the curve 430 and register the coordinate systems. To that end, the system may compute coordinate transformations to align projections of the shape data 480 with at least one of the projected curves 410, 420. In this manner, the shape data 480 from the sensing unit may constrain the reconstruction of the curve 430.

[0096] Regardless of the processes by which the system computes coordinate registration and extraction of the three-dimensional curve 430, the system may use the registration transformation and curve extraction results in at least two ways. First, re-projecting the three-dimensional curve 430 onto the set of two-dimensional projections may facilitate generating, for each one of the projections, an in-painting mask. For example, the projected curves may be dilated and used as masks, as discussed with reference to FIGS. 2E-H. Second, the registration between coordinate systems may facilitate updating anatomical target location in the sensing (instrument) coordinates, as discussed in more detail with reference to FIGS. 5A, B. Furthermore, the in-painting may aid in updating target location by enabling an image reconstruction that minimizes artifacts due the flexible elongate device in the vicinity of the target. The improved image quality may enable an operator to identify target location more accurately in reconstructed images.

[0097] FIGS. 5A, B schematically illustrate example processes 510, 520, 530, 540, 550, 554, 556, 558, and 560 for flexible elongate device extraction, coordinate registration and target update using shape data and projection images. A system (e.g., system 100) may execute the processes 510, 520, 530, 540, 550, 554, 556, 558, and 560 using one or more processors of a processing unit (e.g., processing unit 120) and a display unit (e.g., display unit 130).

[0098] The system may obtain (e.g., from imaging unit 110) a set of two-dimensional projection images 502 corresponding to a respective set of projection angles obtained, for example, using the imaging geometry discussed with reference to FIG. 1C. The system may select from the set 502 projection images 504, 506, which may be, for example, projectionimages 301 , 302. Within each of the projection images 504, 506, the system may identify a corresponding pair of reference points (e.g., corresponding to a tip or a distal end of a flexible elongate device), such as, for example, points 310 and 320. To that end, the system may generate on the display unit a GUI and prompt an operator to select the corresponding points. In other examples, the system may select the corresponding points automatically.

[0099] With a pair of corresponding points in the projection images 504, 506 as input, the system may execute a triangulation process 510 to compute coordinates in three dimensions of the imaging coordinate system of the reference point. The output of the triangulation process 510 may be an imaging coordinate system 512 with the reference point as the origin, without any loss of generality, as described with reference to FIG. 4C.

[0100] With the projection image 504 as input, the system may execute a 2D centerline extraction process 520 to identify within the projection 504 a curve 522 (e.g., curve 420) corresponding to at least a portion of the flexible elongate device including the reference point. To that end, the system may generate on the display unit a GUI and prompt an operator to select a plurality of points on the curve 522. Additionally or alternatively, the system may use a one or more image processing algorithms to identify the curve 522.

[0101] With the coordinate system 512, the curve 522 and the projection 506 as inputs, the system may execute a back tracing process to generate, in the imaging system coordinates 512, a three-dimensional curve 532 (e.g., curve 430). In some examples, the back tracing process 530 may use the set of projection images 502, as described with reference to FIG. 4C.

[0102] The system may use the generated curve 532 along with shape data 535 obtained from a sensor unit (e.g., sensing unit 115) to compute a transformation 542 between the image coordinate system 512 and an instrument coordinate system, as discussed, for example, with reference to FIG. 4C. Additionally or alternatively, the system may use the registration process 540 in generating the curve 532.

[0103] In any case, the system may use the curve 532 along with the set of projections 502 as inputs into the reconstruction with in-painting process 550. The process 550, as discussed above, with reference to FIG. 4C and FIGS. 2E-H, may include projecting the curve 532 back to the set of projection images 502 to generate in-painting masks. The reconstruction with in-painting process 550 may generate a reconstructed three-dimensional image 552 of patient anatomy with a target 553. The three-dimensional image 552 may remove artifacts due to the presence of the flexible elongate device disposed within patient anatomy, providing a clear image of the target 553. It should be noted, that in some examples the tomosynthesis reconstruction from the set of images 502 may be performedwithout in-painting. That is the process 550 may be replaced with a direct tomosynthesis reconstruction process, instead of a sequence of sub-processes 554, 556, and 558 discussed with reference to FIG. 5B.

[0104] The system may use the reconstructed image 552 along with the transformation 542 as inputs for a target update process 560. In some examples, the target update process 560 may include generating the three-dimensional image 552 and / or corresponding slices at a GUI and prompting an operator to identify the center of the target 563. In other examples, the system may automatically identify and / or segment the target 563 within the image 552. The center of the target 553 in the image and coordinate system identified based on the image 552 may be mapped using the transformation 542 onto the instrument coordinate system 562. In some examples, the system may position the target 563 solely based on the transformation 542 and the target 553 from the image 552. In other examples, the system may use the target update process 560 along with prior information about the target 553 (e.g., from prior update based on intraoperative images and / or pre-operative images) to update coordinates of the target 563 within the instrument coordinate system 562.

[0105] FIG. 5B schematically illustrates sub-processes 554, 556, and 558 comprising the process 550, as discussed for example with reference to FIGS. 2E-H and 4C. The mask generation process 554 may generate masks 555 based on the curve 532, by projecting a dilated version of the curve 532 onto the set of projections 502. The in-painting process 556 may perform in painting as discussed above with reference to FIGS. 2A-H, outputting a set of projections 557 with the flexible elongate device removed. In some examples, the inpainting process may also remove fiducial markings from the set of projections 557. The system may then reconstruct the in-painted projections 557 using any suitable tomosynthesis process 558 (e.g., using FDK reconstruction or any other suitable reconstruction technique), generating the three-dimensional image 552.

[0106] It should be noted, that the system may additionally enhance the image 552 prior to identifying the target 553. For example, the system may use a style transfer process as discussed below with reference to FIGS. 6A, B.

[0107] FIG. 6A schematically illustrates an example style transfer for three-dimensional images. A scene 600 (which may include, as discussed above, patient anatomy with an anatomical target, a device disposed in the patient anatomy, one or more fiducial to aid, for example, in image analysis and / or reconstruction) may be imaged using a choice of imaging modalities. An example image 601 may be a result of imaging with a first imaging modality (e.g., CT or CBCT) and image 602 may be a result of a second imaging modality (e.g., tomosynthesis of fluoroscopy images). Generally, the first modality may be a higher-fidelitymodality than the second modality and, consequently, the image 601 may be a higher fidelity image than image 602. In some examples, a system (e.g., system 100) may use a machine learning and / or deep learning (ML / DL) model 610 to generate, from the image 602, an image with characteristics similar to the image 601 . That is, the image generated by the model 610 may not only look like image 601 , but may also have higher fidelity than the image 602. In a sense, the model 610, when effectively trained, may be seen as implementing denoising, deconvolution, image restoration and other image-enhancing processes.

[0108] In some examples, the model 610 may use or be based on a convolutional neural network (CNN) (e.g., ll-net) and / or a transformer network. Additionally or alternatively, the model 610 may be a generative-adversarial network (GAN) model and / or a diffusion model. The model 610 may be based on a combination of models, generating multiple outputs from a single image input and selecting one of the output based on a choice of an operator or one or more computed metrics of the outputs.

[0109] In some examples, the system may render one or more outputs of the model on a display unit (e.g., display unit 130) of the system.

[0110] FIG. 6B schematically illustrates an example training process 620 for a styletransfer ML / DL model 630 (e.g., model 610). The training process 620 may, but need not be implemented by the system that uses the model (e.g., system 100). The training process may be performed by another suitable system with appropriate computing resources and the trained model may be transferred to a system for intraoperative imaging.

[0111] The training process 630 may start with selecting example high-fidelity volumetric data 630, for example from high-fidelity first-modality images (e.g., images such as image 1). The high-fidelity volumetric data 630 may serve as the ground truth for the training process 620. The volumetric data 630 may be, for example, reconstructed CT or CBCT data and may include associated projection data. Either from reconstructed data or from the original projection data associated with the high-fidelity volumetric model 630, the process may generate a simulated second modality sinogram (e.g., a simulated set of fluoroscopy images for tomosynthesis). In some examples, generating the simulated sinogram may include selecting projections from a limited set of projection angles (e.g., 120°, 100°, 110°, 90°, 80°, 70°, 60°, or any other suitable span). In other examples, generating the simulated sinogram may include simulating projections from interpolated volumetric data using, for example, and X-ray propagation model. Generating the simulated sinogram may include adding noise and / or distortions to the simulated projections and / or adding noise and / or distortions to the projection angles associated with the projections. From the simulated second modality sinogram 640, the process 620 may generated reconstructed second modality volumetricdata 650 (e.g., tomosyntheses reconstruction) based on any suitable reconstruction algorithm (e.g., FDK). The reconstructed data 650 serves as the training input to the model 660, and the model 660 is trained according to any suitable training techniques, to minimize loss with respect to the ground truth of the high-fidelity volumetric data 630. The model 660 thus trained with the suitable volume of training data my then be used as the style-transfer model for enhancing images generated by the lower-fidelity modality (e.g., tomosynthesis) to generate higher-fidelity output that may look like higher-fidelity (e.g., CT or CBCT) reconstructions. The higher-fidelity image output may facilitate, for example, identification of anatomical targets as discussed with reference to FIGS. 5A, B.

[0112] FIGS. 7-9B depict diagrams of a medical system that may be used for manipulating a medical instrument that includes a flexible elongate device according to any of the methods and systems described above, in some examples. For example, each reference above to the “system” may refer to a system (e.g., system 700) discussed below, or to a subsystem thereof.

[0113] FIG. 7 is a simplified diagram of a medical system 700 according to some examples. The medical system 700 may include at least portions of the system 100 described with reference to FIG. 1 . The medical system 700 may be suitable for use in, for example, surgical, diagnostic (e.g., biopsy), or therapeutic (e.g., ablation, electroporation, etc.) procedures. While some examples are provided herein with respect to such procedures, any reference to medical or surgical instruments and medical or surgical methods is non-limiting. The systems, instruments, and methods described herein may be used for animals, human cadavers, animal cadavers, portions of human or animal anatomy, non-surgical diagnosis, as well as for industrial systems, general or special purpose robotic systems, general or special purpose teleoperational systems, or robotic medical systems.

[0114] As shown in FIG. 7, medical system 700 may include a manipulator assembly 702 that controls the operation of a medical instrument 704 in performing various procedures on a patient (e.g., patient P on table T, as in FIG. 1 ). The medical instrument 704 may include the flexible elongated device 140 of FIG. 1 . Medical instrument 704 may extend into an internal site within the body of patient P via an opening in the body of patient P. The manipulator assembly 702 may be teleoperated, non-teleoperated, or a hybrid teleoperated and non-teleoperated assembly with one or more degrees of freedom of motion that may be motorized and / or one or more degrees of freedom of motion that may be non-motorized (e.g., manually operated). The manipulator assembly 702 may be mounted to and / or positioned near patient table T. A master assembly 706 allows an operator O (e.g., a surgeon, a clinician, a physician, or other user, as described above) to control the manipulator assembly 702. In some examples, the master assembly 706 allows theoperator O to view the procedural site or other graphical or informational displays. In some examples, the manipulator assembly 702 may be excluded from the medical system 700 and the medical instrument 704 may be controlled directly by the operator O. In some examples, the manipulator assembly 702 may be manually controlled by the operator O. Direct operator control may include various handles and operator interfaces for hand-held operation of the medical instrument 704.

[0115] The master assembly 706 may be located at a surgeon’s console which is in proximity to (e.g., in the same room as) the patient table T on which patient P is located, such as at the side of the patient table T. In some examples, the master assembly 706 is remote from the patient table T, such as in in a different room or a different building from the patient table T. The master assembly 706 may include one or more control devices for controlling the manipulator assembly 702. The control devices may include any number of a variety of input devices, such as joysticks, trackballs, scroll wheels, directional pads, buttons, data gloves, trigger-guns, hand-operated controllers, voice recognition devices, motion or presence sensors, and / or the like.

[0116] The manipulator assembly 702 supports the medical instrument 704 and may include a kinematic structure of links that provide a set-up structure. The links may include one or more non-servo controlled links (e.g., one or more links that may be manually positioned and locked in place) and / or one or more servo controlled links (e.g., one or more links that may be controlled in response to commands, such as from a control system 712). The manipulator assembly 702 may include a plurality of actuators (e.g., motors) that drive inputs on the medical instrument 704 in response to commands, such as from the control system 712. The actuators may include drive systems that move the medical instrument 704 in various ways when coupled to the medical instrument 704. For example, one or more actuators may advance medical instrument 704 into a naturally or surgically created anatomic orifice. Actuators may control articulation of the medical instrument 704, such as by moving the distal end (or any other portion) of medical instrument 704 in multiple degrees of freedom. These degrees of freedom may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). One or more actuators may control rotation of the medical instrument about a longitudinal axis. Actuators can also be used to move an articulable end effector of medical instrument 704, such as for grasping tissue in the jaws of a biopsy device and / or the like, or may be used to move or otherwise control tools (e.g., imaging tools, ablation tools, biopsy tools, electroporation tools, etc.) that are inserted within the medical instrument 704.

[0117] The control system 712 may include at least portions of the processing unit 120. Additionally or alternatively, the control system 712 may be in communicative connection with the processing unit 120. In some examples, the output of the processing unit 120 according to the techniques described above may cause the control system 712 to autonomously (without input from the operator O) control certain movements of the medical instrument 704.

[0118] The medical system 700 may include a sensor system 708 (which may include at least a portion of the sensing unit 115) with one or more sub-systems for receiving information about the manipulator assembly 702 and / or the medical instrument 704. Such sub-systems may include a position sensor system (e.g., that uses electromagnetic (EM) sensors or other types of sensors that detect position or location); a shape sensor system for determining the position, orientation, speed, velocity, pose, and / or shape of a distal end and / or of one or more segments along a flexible body of the medical instrument 704; a visualization system for capturing images, such as from the distal end of medical instrument 704 or from some other location; and / or actuator position sensors such as resolvers, encoders, potentiometers, and the like that describe the rotation and / or orientation of the actuators controlling the medical instrument 704. The subsystems may include an imaging sub-system (e.g., using a color imaging device, an infrared imaging device, an ultrasound imaging device, an x-ray imaging device, a fluoroscopic imaging device, a computed tomography (CT) imaging device, a magnetic resonance imaging (MRI) imaging device, or some other type of imaging device), such as the imaging unit 110.

[0119] It should be noted that the positions and orientations of sensors in the sensor system 708 may be determined in the sensor coordinate system. In some examples, the sensor coordinate system is integrated with or identical to the coordinate system of the manipulator assembly 702.

[0120] The medical system 700 may include a display system 710 (e.g., display unit 130) for displaying an image or representation of the procedural site and the medical instrument 704. Display system 710 and master assembly 706 may be oriented so physician O can control medical instrument 704 and master assembly 706 with the perception of telepresence. The display system 710 may include at least portions of the display unit 130.

[0121] In some examples, the medical instrument 704 may include a visualization system, which may include an image capture assembly that records a concurrent or real-time image of a procedural site and provides the image to the operator O through one or more displays of display system 710. The image capture assembly may include various types of imaging devices. The concurrent image may be, for example, a two-dimensional image or a three-dimensional image captured by an endoscope positioned within the anatomical procedural site. The visualization system may obtain intra-operative images in image system coordinates, distinct from the sensor system coordinates. In some examples, the visualization system may include endoscopic components that may be integrally or removably coupled to medical instrument 704. Additionally or alternatively, a separate endoscope, attached to a separate manipulator assembly, may be used with medical instrument 704 to image the procedural site. The visualization system may be implemented as hardware, firmware, software or a combination thereof which interact with or are otherwise executed by one or more computer processors, such as of the control system 712.

[0122] Display system 710 may also display an image of the procedural site and medical instruments, which may be captured by the visualization system. In some examples, the medical system 700 provides a perception of telepresence to the operator O. For example, images captured by an imaging device at a distal portion of the medical instrument 704 may be presented by the display system 710 to provide the perception of being at the distal portion of the medical instrument 704 to the operator O. The input to the master assembly 706 provided by the operator O may move the distal portion of the medical instrument 704 in a manner that corresponds with the nature of the input (e.g., distal tip turns right when a trackball is rolled to the right) and results in corresponding change to the perspective of the images captured by the imaging device at the distal portion of the medical instrument 704. As such, the perception of telepresence for the operator O is maintained as the medical instrument 704 is moved using the master assembly 706. The operator O can manipulate the medical instrument 704 and hand controls of the master assembly 706 as if viewing the workspace in substantially true presence, simulating the experience of an operator that is physically manipulating the medical instrument 704 from within the patient anatomy.

[0123] In some examples, the display system 710 may present virtual images of a procedural site that are created using image data recorded pre-operatively (e.g., prior to the procedure performed by the medical instrument system 200) or intra-operatively (e.g., concurrent with the procedure performed by the medical instrument system 200), such as image data created using computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or the like. The virtual images may include two-dimensional, three-dimensional, or higher-dimensional (e.g., including, for example, time based or velocity-based information) images. In some examples, one or more models are created from pre-operative or intra-operative image data sets and the virtual images are generated using the one or more models.

[0124] In some examples, for purposes of imaged guided medical procedures, display system 710 may display a virtual image that is generated based on tracking the location of medical instrument 704. For example, the tracked location of the medical instrument 704 may be registered (e.g., dynamically referenced) with the model generated using the preoperative or intra-operative images, with different portions of the model correspond with different locations of the patient anatomy. As the medical instrument 704 moves through the patient anatomy, the registration is used to determine portions of the model corresponding with the location and / or perspective of the medical instrument 704 and virtual images are generated using the determined portions of the model. This may be done to present the operator O with virtual images of the internal procedural site from viewpoints of medical instrument 704 that correspond with the tracked locations of the medical instrument 704.

[0125] The display system 710 may include the display unit 130 and may display images including the position, orientation, and / or pose of the medical instrument 704 in a joint coordinate system based on registering the sensor coordinate system with the imaging coordinate system according to the techniques described above with reference to FIGS. 2A- 6.

[0126] The medical system 700 may also include the control system 712, which may include processing circuitry (e.g., the processing unit 120) that implements the some or all of the methods or functionality discussed herein. The control system 712 may include at least one memory and at least one processor for controlling the operations of the manipulator assembly 702, the medical instrument 704, the master assembly 706, the sensor system 708, and / or the display system 710. Control system 712 may include instructions (e.g., a non-transitory machine-readable medium storing the instructions) that when executed by the at least one processor, configures the one or more processors to implement some or all of the methods or functionality discussed herein. While the control system 712 is shown as a single block in FIG. 7, the control system 712 may include two or more separate data processing circuits with one portion of the processing being performed at the manipulator assembly 702, another portion of the processing being performed at the master assembly 706, and / or the like. In some examples, the control system 712 may include other types of processing circuitry, such as application-specific integrated circuits (ASICs) and / or field- programmable gate array (FPGAs). The control system 712 may be implemented using hardware, firmware, software, or a combination thereof.

[0127] In some examples, the control system 712 may receive feedback from the medical instrument 704, such as force and / or torque feedback. Responsive to the feedback, the control system 712 may transmit signals to the master assembly 706. In some examples, the control system 712 may transmit signals instructing one or more actuators of themanipulator assembly 702 to move the medical instrument 704. In some examples, the control system 712 may transmit informational displays regarding the feedback to the display system 710 for presentation or perform other types of actions based on the feedback.

[0128] The control system 712 may include a virtual visualization system to provide navigation assistance to operator O when controlling the medical instrument 704 during an image-guided medical procedure. Virtual navigation using the virtual visualization system may be based upon an acquired pre-operative or intra-operative dataset of anatomic passageways of the patient P. The control system 712 or a separate computing device may convert the recorded images, using programmed instructions alone or in combination with operator inputs, into a model of the patient anatomy. The model may include a segmented two-dimensional or three-dimensional composite representation of a partial or an entire anatomic organ or anatomic region. An image data set may be associated with the composite representation. The virtual visualization system may obtain sensor data from the sensor system 708 that is used to compute an (e.g., approximate) location of the medical instrument 704 with respect to the anatomy of patient P. The sensor system 708 may be used to register and display the medical instrument 704 together with the pre-operatively or intra-operatively recorded images. For example, PCT Publication WO 2016 / 191298 (published December 1 , 2016 and titled “Systems and Methods of Registration for Image Guided Surgery”), which is incorporated by reference herein in its entirety, discloses example systems. Additionally or alternatively, the registration may be based on the techniques discussed above with reference to FIGS. 2A-6.

[0129] During a virtual navigation procedure, the sensor system 708 may be used to compute the (e.g., approximate) location of the medical instrument 704 with respect to the anatomy of patient P. The location can be used to produce both macro-level (e.g., external) tracking images of the anatomy of patient P and virtual internal images of the anatomy of patient P. The system may include one or more electromagnetic (EM) sensors, fiber optic sensors, and / or other sensors to register and display a medical instrument together with pre- operatively recorded medical images. For example, U.S. Patent No. 8,900,131 (filed May 13, 2011 and titled “Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery”), which is incorporated by reference herein in its entirety, discloses example systems.

[0130] Medical system 700 may further include operations and support systems (not shown) such as illumination systems, steering control systems, irrigation systems, and / or suction systems. In some examples, the medical system 700 may include more than one manipulator assembly and / or more than one master assembly. The exact number of manipulator assemblies may depend on the medical procedure and space constraints withinthe procedural room, among other factors. Multiple master assemblies may be co-located or they may be positioned in separate locations. Multiple master assemblies may allow more than one operator to control one or more manipulator assemblies in various combinations.

[0131] FIG. 8A is a simplified diagram of a medical instrument system °0 according to some examples. The medical instrument system 800 includes a flexible elongate device 802 (e.g., device 140), also referred to as elongate device 802, a drive unit 804, and a medical tool 826 that collectively is an example of a medical instrument 704 of a medical system 700. The medical system 700 may be a teleoperated system, a non-teleoperated system, or a hybrid teleoperated and non-teleoperated system, as explained with reference to FIG. 7. A visualization system 831 , tracking system 830, and navigation system 832 are also shown in FIG. 8A and are example components of the control system 712 of the medical system 700. In some examples, the medical instrument system 800 may be used for non-teleoperational exploratory procedures or in procedures involving traditional manually operated medical instruments, such as endoscopy. The medical instrument system 800 may be used to gather (e.g., measure) a set of data points corresponding to locations within anatomic passageways of a patient, such as patient P.

[0132] The elongate device 802 is coupled to the drive unit 804. The elongate device 802 includes a channel 821 through which the medical tool 826 may be inserted. The elongate device 802 navigates within patient anatomy to deliver the medical tool 826 to a procedural site. The elongate device 802 includes a flexible body 816 having a proximal end 817 and a distal end 818. In some examples, the flexible body 816 may have an approximately 3 mm outer diameter. Other flexible body outer diameters may be larger or smaller.

[0133] Medical instrument system 800 may include the tracking system 830 for determining the position, orientation, speed, velocity, pose, and / or shape of the flexible body 816 at the distal end 818 and / or of one or more segments 824 along flexible body 816, as will be described in further detail below. The tracking system 830 may include one or more sensors and / or imaging devices. The flexible body 816, such as the length between the distal end 818 and the proximal end 817, may include multiple segments 824. The tracking system 830 may be implemented using hardware, firmware, software, or a combination thereof. In some examples, the tracking system 830 is part of control system 712 shown in FIG. 7. The tracking system 830 may implement at least some of the techniques described with reference to FIGS. 1 A-6, and, to that end, may include at least portions of or be in communicative connection with the processing unit 120 of FIG. 1A.

[0134] Tracking system 830 may track the distal end 818 and / or one or more of the segments 824 of the flexible body 816 using a shape sensor 822. It should be noted that,with the techniques of the instant disclosure, the shape sensor 822 may be omitted. The shape sensor 822 may include an optical fiber aligned with the flexible body 816 (e.g., provided within an interior channel of the flexibly body 816 or mounted externally along the flexible body 816). In some examples, the optical fiber may have a diameter of approximately 800 pm. In other examples, the diameter may be larger or smaller. The optical fiber of the shape sensor 822 may form a fiber optic bend sensor for determining the shape of flexible body 816. Optical fibers including Fiber Bragg Gratings (FBGs) may be used to provide strain measurements in structures in one or more dimensions. Various systems and methods for monitoring the shape and relative position of an optical fiber in three dimensions, which may be applicable in some examples, are described in U.S. Patent Application Publication No. 2006 / 0013523 (filed July 13, 2005 and titled “Fiber optic position and shape sensing device and method relating thereto”); U.S. Patent No. 7,772,541 (filed on March 12, 2008 and titled “Fiber Optic Position and / or Shape Sensing Based on Rayleigh Scatter”); and U.S. Patent No. 8,773,650 (filed on Sept. 2, 2010 and titled “Optical Position and / or Shape Sensing”), which are all incorporated by reference herein in their entireties. Sensors in some examples may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and Fluorescence scattering.

[0135] In some examples, the shape of the flexible body 816 may be determined using other techniques. For example, a history of the position and / or pose of the distal end 818 of the flexible body 816 can be used to reconstruct the shape of flexible body 816 over an interval of time (e.g., as the flexible body 816 is advanced or retracted within a patient anatomy). In some examples, the tracking system 830 may alternatively and / or additionally track the distal end 818 of the flexible body 816 using a position sensor system 820.Position sensor system 820 may be a component of an EM sensor system with the position sensor system 820 including one or more position sensors. Although the position sensor system 820 is shown as being near the distal end 818 of the flexible body 816 to track the distal end 818, the number and location of the position sensors of the position sensor system 820 may vary to track different regions along the flexible body 816. In one example, the position sensors include conductive coils that may be subjected to an externally generated electromagnetic field. Each coil of position sensor system 820 may produce an induced electrical signal having characteristics that depend on the position and orientation of the coil relative to the externally generated electromagnetic field. The position sensor system 820 may measure one or more position coordinates and / or one or more orientation angles associated with one or more portions of flexible body 816. In some examples, the position sensor system 820 may be configured and positioned to measure six degrees of freedom, e.g., three position coordinates X, Y, Z and three orientation angles indicating pitch, yaw,and roll of a base point. In some examples, the position sensor system 820 may be configured and positioned to measure five degrees of freedom, e.g., three position coordinates X, Y, Z and two orientation angles indicating pitch and yaw of a base point. Further description of a position sensor system, which may be applicable in some examples, is provided in U.S. Patent No. 6,380,732 (filed August 11 , 1999 and titled “Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked”), which is incorporated by reference herein in its entirety.

[0136] A processing unit (e.g., processing unit 120) may enhance the accuracy of positions obtained by the position sensor system 820 by combing data obtained by the position sensor system 820 with data obtained by an external imaging system (e.g., by way of the imaging unit 110) according to the techniques of this disclosure described above with reference to FIGS. 2A-6.

[0137] In some examples, the tracking system 830 may alternately and / or additionally rely on a collection of pose, position, and / or orientation data stored for a point of an elongate device 802 and / or medical tool 826 captured during one or more cycles of alternating motion, such as breathing. This stored data may be used to develop shape information about the flexible body 816. In some examples, a series of position sensors (not shown), such as EM sensors like the sensors in position sensor system 820 or some other type of position sensors may be positioned along the flexible body 816 and used for shape sensing. In some examples, a history of data from one or more of these position sensors taken during a procedure may be used to represent the shape of elongate device 802, particularly if an anatomic passageway is generally static.

[0138] FIG. 8B is a simplified diagram of the medical tool 826 within the elongate device 802 according to some examples. The flexible body 816 of the elongate device 802 may include the channel 821 sized and shaped to receive the medical tool 826. In some examples, the medical tool 826 may be used for procedures such as imaging, surgery, biopsy, ablation, illumination, irrigation, suction, electroporation, etc. Medical tool 826 can be deployed through channel 821 of flexible body 816 and operated at a procedural site within the anatomy. Medical tool 826 may be, for example, an image capture probe, a biopsy tool (e.g., a needle, grasper, brush, etc.), an ablation tool (e.g., a laser ablation tool, radio frequency (RF) ablation tool, cryoablation tool, thermal ablation tool, heated liquid ablation tool, etc.), an electroporation tool, and / or another surgical, diagnostic, or therapeutic tool. In some examples, the medical tool 826 may include an end effector having a single working member such as a scalpel, a blunt blade, an optical fiber, an electrode, and / or the like. Other end types of end effectors may include, for example, forceps, graspers, scissors, staplers, clip appliers, and / or the like. Other end effectors may further include electricallyactivated end effectors such as electrosurgical electrodes, transducers, sensors, and / or the like.

[0139] The medical tool 826 may be a biopsy tool used to remove sample tissue or a sampling of cells from a target anatomic location. In some examples, the biopsy tool is a flexible needle. The biopsy tool may further include a sheath that can surround the flexible needle to protect the needle and interior surface of the channel 821 when the biopsy tool is within the channel 821 . The medical tool 826 may be an image capture probe that includes a distal portion with a stereoscopic or monoscopic camera that may be placed at or near the distal end 818 of flexible body 816 for capturing images (e.g., still or video images). The captured images may be processed by the visualization system 831 for display and / or provided to the tracking system 830 to support tracking of the distal end 818 of the flexible body 816 and / or one or more of the segments 824 of the flexible body 816. The image capture probe may include a cable for transmitting the captured image data that is coupled to an imaging device at the distal portion of the image capture probe. In some examples, the image capture probe may include a fiber-optic bundle, such as a fiberscope, that couples to a more proximal imaging device of the visualization system 831 . The image capture probe may be single-spectral or multi-spectral, for example, capturing image data in one or more of the visible, near-infrared, infrared, and / or ultraviolet spectrums. The image capture probe may also include one or more light emitters that provide illumination to facilitate image capture. In some examples, the image capture probe may use ultrasound, x-ray, fluoroscopy, CT, MRI, or other types of imaging technology.

[0140] In some examples, the image capture probe is inserted within the flexible body 816 of the elongate device 802 to facilitate visual navigation of the elongate device 802 to a procedural site and then is replaced within the flexible body 816 with another type of medical tool 826 that performs the procedure. In some examples, the image capture probe may be within the flexible body 816 of the elongate device 802 along with another type of medical tool 826 to facilitate simultaneous image capture and tissue intervention, such as within the same channel 821 or in separate channels. A medical tool 826 may be advanced from the opening of the channel 821 to perform the procedure (or some other functionality) and then retracted back into the channel 821 when the procedure is complete. The medical tool 826 may be removed from the proximal end 817 of the flexible body 816 or from another optional instrument port (not shown) along flexible body 816.

[0141] In some examples, the elongate device 802 may include integrated imaging capability rather than utilize a removable image capture probe. For example, the imaging device (or fiber-optic bundle) and the light emitters may be located at the distal end 818 of the elongate device 802. The flexible body 815 may include one or more dedicatedchannels that carry the cable(s) and / or optical fiber(s) between the distal end 818 and the visualization system 831 . Here, the medical instrument system 800 can perform simultaneous imaging and tool operations.

[0142] In some examples, the medical tool 826 is capable of controllable articulation. The medical tool 826 may house cables (which may also be referred to as pull wires), linkages, or other actuation controls (not shown) that extend between its proximal and distal ends to controllably bend the distal end of medical tool 826, such as discussed herein for the flexible elongate device 802. The medical tool 826 may be coupled to a drive unit 804 and the manipulator assembly 702. In these examples, the elongate device 802 may be excluded from the medical instrument system 800 or may be a flexible device that does not have controllable articulation. Steerable instruments or tools, applicable in some examples, are further described in detail in U.S. Patent No. 7,316,681 (filed on Oct. 4, 2005 and titled “Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity”) and U.S. Patent No. 9,259,274 (filed Sept. 30, 2008 and titled “Passive Preload and Capstan Drive for Surgical Instruments”), which are incorporated by reference herein in their entireties.

[0143] The flexible body 816 of the elongate device 802 may also or alternatively house cables, linkages, or other steering controls (not shown) that extend between the drive unit 804 and the distal end 818 to controllably bend the distal end 818 as shown, for example, by broken dashed line depictions 819 of the distal end 818 in FIG. 2A. In some examples, at least four cables are used to provide independent up-down steering to control a pitch of the distal end 818 and left-right steering to control a yaw of the distal end 881 . In these examples, the flexible elongate device 802 may be a steerable catheter. Examples of steerable catheters, applicable in some examples, are described in detail in PCT Publication WO 2019 / 018736 (published Jan. 24, 2019 and titled “Flexible Elongate Device Systems and Methods”), which is incorporated by reference herein in its entirety.

[0144] In examples where the elongate device 802 and / or medical tool 826 are actuated by a teleoperational assembly (e.g., the manipulator assembly 702), the drive unit 804 may include drive inputs that removably couple to and receive power from drive elements, such as actuators, of the teleoperational assembly. In some examples, the elongate device 802 and / or medical tool 826 may include gripping features, manual actuators, or other components for manually controlling the motion of the elongate device 802 and / or medical tool 826. The elongate device 802 may be steerable or, alternatively, the elongate device 802 may be non-steerable with no integrated mechanism for operator control of the bending of distal end 818. In some examples, one or more channels 821 (which may also be referred to as lumens), through which medical tools 826 can be deployed and used at atarget anatomical location, may be defined by the interior walls of the flexible body 816 of the elongate device 802.

[0145] In some examples, the medical instrument system 800 (e.g., the elongate device 802 or medical tool 826) may include a flexible bronchial instrument, such as a bronchoscope or bronchial catheter, e.g., for use in visual examination and diagnosis, biopsy, and / or treatment of a lung. The medical instrument system 800 may also be suited for navigation and treatment of other tissues, via natural or surgically created connected passageways, in any of a variety of anatomic systems, including the colon, the intestines, the kidneys and kidney calices, the brain, the heart, the circulatory system including vasculature, and / or the like.

[0146] The information from the tracking system 830 may be sent to the navigation system 832, where the information may be combined with information from the visualization system 831 and / or pre-operatively obtained models to provide the physician, clinician, surgeon, or other operator with real-time position information. The tracking system 830, the navigation system 832, and the visualization system 831 may cooperatively implement, at least partially, the functionality of the system 100 in implementing the techniques described with reference to FIGS. 1 A-6. In some examples, the real-time position information may be displayed on the display system 710 for use in the control of the medical instrument system 800. In some examples, the navigation system 832 may utilize the position information as feedback for positioning medical instrument system 800.

[0147] FIGS. 9A and 9B are simplified diagrams of side views of a patient coordinate space including a medical instrument mounted on an insertion assembly according to some examples. As shown in FIGS. 9A and 9B, a surgical environment 900 may include the patient P positioned on the patient table T. Patient P may be stationary within the surgical environment 900 in the sense that gross patient movement is limited by sedation, restraint, and / or other means. Cyclic anatomic motion, including respiration and cardiac motion, of patient P may continue. Within surgical environment 900, a medical instrument 904 is used to perform a medical procedure which may include, for example, surgery, biopsy, ablation, illumination, irrigation, suction, or electroporation. The medical instrument 904 may also be used to perform other types of procedures, such as a registration procedure to associate the position, orientation, and / or pose data captured by the sensor system 708 to a desired (e.g., anatomical or system) reference frame. The medical instrument 904 may be, for example, the medical instrument 704. In some examples, the medical instrument 904 may include an elongate device 910 (e.g., a catheter) coupled to an instrument body 912. Elongate device 910 may be the elongate device 140 of FIG. 1. Elongate device 910 includes one or more channels sized and shaped to receive a medical tool.

[0148] Elongate device 910 may also include one or more sensors (e.g., components of the sensor system 708). In some examples, a shape sensor 914 may be fixed at a proximal point 916 on the instrument body 912. The proximal point 916 of the shape sensor 914 may be movable with the instrument body 912, and the location of the proximal point 916 with respect to a desired reference frame may be known (e.g., via a tracking sensor or other tracking device). The shape sensor 914 may measure a shape from the proximal point 916 to another point, such as a distal end 918 of the elongate device 910. The shape sensor 914 may be aligned with the elongate device 910 (e.g., provided within an interior channel or mounted externally). In some examples, the shape sensor 914 may optical fibers used to generate shape information for the elongate device 910.

[0149] In some examples, position sensors (e.g., EM sensors) may be incorporated into the medical instrument 904. A series of position sensors may be positioned along the flexible elongate device 910 and used for shape sensing. Position sensors may be used alternatively to the shape sensor 914 or with the shape sensor 914, such as to improve the accuracy of shape sensing or to verify shape information.

[0150] Elongate device 910 may house cables, linkages, or other steering controls that extend between the instrument body 912 and the distal end 918 to controllably bend the distal end 918. In some examples, at least four cables are used to provide independent up- down steering to control a pitch of distal end 918 and left-right steering to control a yaw of distal end 918. The instrument body 912 may include drive inputs that removably couple to and receive power from drive elements, such as actuators, of a manipulator assembly.

[0151] The instrument body 912 may be coupled to an instrument carriage 906. The instrument carriage 906 may be mounted to an insertion stage 908 that is fixed within the surgical environment 900. Alternatively, the insertion stage 908 may be movable but have a known location (e.g., via a tracking sensor or other tracking device) within surgical environment 900. Instrument carriage 906 may be a component of a manipulator assembly (e.g., manipulator assembly 702) that couples to the medical instrument 904 to control insertion motion (e.g., motion along an insertion axis A) and / or motion of the distal end 918 of the elongate device 910 in multiple directions, such as yaw, pitch, and / or roll. The instrument carriage 906 or insertion stage 908 may include actuators, such as servomotors, that control motion of instrument carriage 906 along the insertion stage 908.

[0152] A sensor device 920, which may be a component of the sensor system 708, may provide information about the position of the instrument body 912 as it moves relative to the insertion stage 908 along the insertion axis A. The sensor device 920 may include one or more resolvers, encoders, potentiometers, and / or other sensors that measure the rotationand / or orientation of the actuators controlling the motion of the instrument carriage 906, thus indicating the motion of the instrument body 912. In some examples, the insertion stage 908 has a linear track as shown in FIGS. 9A and 9B. In some examples, the insertion stage 908 may have curved track or have a combination of curved and linear track sections.

[0153] FIG. 9A shows the instrument body 912 and the instrument carriage 906 in a retracted position along the insertion stage 908. In this retracted position, the proximal point 916 is at a position L0 on the insertion axis A. The location of the proximal point 916 may be set to a zero value and / or other reference value to provide a base reference (e.g., corresponding to the origin of a desired reference frame) to describe the position of the instrument carriage 906 along the insertion stage 908. In the retracted position, the distal end 918 of the elongate device 910 may be positioned just inside an entry orifice of patient P. Also in the retracted position, the data captured by the sensor device 920 may be set to a zero value and / or other reference value (e.g., l=0). In FIG. 9B, the instrument body 912 and the instrument carriage 906 have advanced along the linear track of insertion stage 908, and the distal end 918 of the elongate device 910 has advanced into patient P. In this advanced position, the proximal point 916 is at a position L1 on the insertion axis A. In some examples, the rotation and / or orientation of the actuators measured by the sensor device 920 indicating movement of the instrument carriage 906 along the insertion stage 908 and / or one or more position sensors associated with instrument carriage 906 and / or the insertion stage 908 may be used to determine the position L1 of the proximal point 916 relative to the position L0. In some examples, the position L1 may further be used as an indicator of the distance or insertion depth to which the distal end 918 of the elongate device 910 is inserted into the passageway(s) of the anatomy of patient P.

[0154] FIG. 10 depicts a flow chart for an example method 1000 for visualizing patient anatomy during a medical procedure. The method 1000 may be implemented by a system (e.g., system 100) which may include one or more processors (e.g., disposed at the processing unit 120) and a display device (e.g., the display unit 130). Furthermore, instructions for executing the method 1000 on the one or more processors may be stored on a tangible, non-transitory, computer readable medium.

[0155] At block 1010, the method 1000 includes obtaining a first plurality of two- dimensional images depicting an anatomical structure and one or more objects. An example system may obtain the images from an imaging unit, such as, for example a fluoroscopy imaging unit. The two-dimensional images may correspond to different projection angles, as discussed throughout the disclosure. To better identify the projection angles associated with the projection images, as discussed above, an example imaging setup may include a fiducial plate.

[0156] At block 1020, the method 1000 includes computing, based at least in part on the first plurality of two-dimensional images, a second plurality of two-dimensional images by inpainting at least a portion of pixels associated with at least one of the one or more objects. The objects may include fiducial elements manifesting as fiducial markings in the set of projection images. Additionally or alternatively, the objects may include an instrument (e.g., a flexible elongate device disposed within patient anatomy. Pixels associated with the non- anatomical objects within the projections may cause artifacts when the two-dimensional projection images are used to reconstruct a three-dimension volumetric image. Thus, inpainting, as discussed in detail above with reference to images 2A-H, 4C and 5B may improve reconstructed image quality. In some examples, the system may in-paint fiducial markings to better identify the pixels associated with a flexible elongate device and, subsequently in-paint the pixels associated with at least a portion of the flexible elongate device. A reconstruction of three-dimensional anatomical structure after in-painting may facilitate identifying an anatomical target for the medical procedure, improving speed and accuracy of the procedure.

[0157] At block 1030, the method 1000 includes causing, a display device to display a graphical user interface depicting a visualization based at least in part on at least one of the second plurality of two-dimensional images. In some examples, the visualization may be for the purpose of evaluating a stage of the medical procedure and / or aiding with navigation. In other examples, the visualization may be a part of an interactive display, prompting an operator for an input, such as, for example, identifying an anatomical target. Additionally or alternatively, the visualization may be a part of a GUI for identifying pixels associated with the flexible elongate device within the projection images, as discussed, for example with reference to FIGS. 3B, C and 4C.

[0158] At an optional block 1040, the method 1000 includes computing, based at least in part on the second plurality of two-dimensional images, a first three-dimensional image of at least a portion of the anatomical structure. As discussed above, the three-dimensional image may be based on reconstruction from projections (e.g., image 552 in FIG. 5A).

[0159] At an optional block 1050, the method 1000 includes computing a second three- dimensional image based on the first three-dimensional image using a style-transfer machine learning model, and wherein the visualization is based at least in part on the computed second three-dimensional image. As discussed above (e.g., with reference to FIGS. 6A, C, as well as below, with reference to FIGS. 13A, B and 14A, B, the first reconstructed image may have low fidelity due to, for example, a limited set of projection angles. Style transfer models, taking the first three-dimensional image as an input and computing the second three-dimensional image as an output may significantly enhanceimage quality (e.g., as in FIGS. 14A, B below). Training a style transfer model is discussed above, particularly with reference to FIG. 6B, as well as below, with reference to FIG. 12.

[0160] FIGS. 11 A, B depict a flow chart for an example method 1100 for visualizing patient anatomy during a medical procedure. The method 1100 may be implemented by a system (e.g., system 100) which may include one or more processors (e.g., disposed at the processing unit 120) and a display device (e.g., the display unit 130). Furthermore, instructions for executing the method 1100 on the one or more processors may be stored on a tangible, non-transitory, computer readable medium.

[0161] At block 1110, the method 1100 includes obtaining a plurality of two-dimensional projection images corresponding to a plurality of respective projection angles and depicting an anatomical structure and a flexible elongate device. At block 1120, the method 1100 includes identifying, within each of at least two of the plurality of two-dimensional projection images, respective two-dimensional coordinates of a reference point disposed at the flexible elongate device. As discussed above, identifying the two-dimensional coordinated within projection images may include obtaining, by the one or more processors, input from an operator via a GUI. Additionally or alternatively, image processing algorithms and / or ML models may aid in autonomously identifying, by a processing unit, the reference point, or, at least, candidate reference points within projection images. The operator and / or the system may identify the reference point as the distal end (e.g., tip) of the flexible elongate device, and / or by way of a fiducial disposed at the flexible elongate device to indicate the reference point.

[0162] At block 1130, the method 1100 includes computing, based at least in part on the respective two-dimensional coordinates and two respective projection angles, three dimensional coordinates, in an imaging coordinate system, of the reference point. The system may compute the three-dimensional coordinates by back-tracing rays from the two projections of the reference point, along the respective projection angles, to find an intersection of or a closest approach point of the back-traced rays. The system may use more than two projection images with identified projected reference point coordinates to compute the coordinates of the reference point in three dimensions to make the computation more robust to noise and / or errors in identifying reference point projections. For example, an erroneously identified projected reference point may lead to a back-traced ray that passes far from the closest approach point of other back-traced rays.

[0163] At block 1140, the method 1100 includes identifying, within at least one of the at least two of the plurality of two-dimensional projection images, a curve (e.g., a centerline) corresponding to at least a portion of the flexible elongate device including the referencepoint. Identifying the curve may include obtaining, by the one or more processors, input from an operator via a GUI. For example, as discussed above (e.g., with reference to FIG. 3C) the GUI may render a projection image depicting a projection of the flexible elongate device, wherein the user may select several points corresponding to the device. The one or more processors may segment the curve corresponding to the device using suitable image processing algorithms and user input. In other examples, the system may autonomously identify the curve.

[0164] At block 1150, the method 1100 includes receiving, from a sensing unit, shape data in a sensing coordinate system for the at least a portion of the flexible elongate device including the reference point. The shape data may be generated by one or more sensors as discussed with reference to FIG. 1 A. At block 1160, the method 1100 includes registering the sensing coordinate system to the imaging coordinate system based at least in part on the received shape data and the identified curved within the at least one of the at least two of the plurality of two-dimensional projection images. The registration process is discussed in detail with reference to FIGS. 4A-C. In some examples, the registration process may include first computing in three dimensions, based on projection images, the curve corresponding to the flexible elongate device and subsequently aligning the computed curve with the received shape data. In other examples, the shape data may serve as the input for the registration process, as discussed with reference to FIGS. 4C and 5A.

[0165] At block 1170, the method 1100 includes reconstructing, based at least in part on the plurality of two-dimensional projection images corresponding to the plurality of respective projection angles, a three-dimensional image of the anatomical structure. The reconstruction may be, for example, tomosynthesis based on a set of fluoroscopy images. As discussed throughout the disclosure, the reconstruction may include or be preceded by an in-painting step. Additionally or alternatively, the reconstruction may be enhance by using a style transfer model.

[0166] At block 1180, the method 1100 includes identifying, within the reconstructed three-dimensional image of the anatomical structure, a target. In some examples, identifying the target may include obtaining, by the one or more processors, input from an operator via a GUI, as discussed above, for example, with reference to FIG. 5A. In other examples, the system may autonomously identify the target within the reconstructed image using suitable image processing algorithms and / or an ML model. Additionally or alternatively, the system may use prior information (e.g., from pre-operative imaging) to identify the target within the reconstructed inter-operative image.

[0167] At block 1190, the method 1100 includes computing, in the instrument coordinate system and based at least in part on identifying the target and registering the sensing coordinate system to the imaging coordinate system, a location of the target. The location of the target may be computed based on user input identifying target center within the reconstructed image, based on identifying a center of a region (e.g., a collection of voxels) associated with the target, or by other suitable means. In some examples, computing the target location may be based on applying a registration transformation to the target location in the reconstructed image to compute the target location in instrument coordinates. Additionally or alternatively, computing the target location may include updating the target location from the previously identified target location in the instrument coordinate system. Updating the target may involve integrating the prior information, filtering time-evolution of the target location, and / or any other suitable signal processing techniques. At block 1195, the method 1100 includes causing the display device to display a graphical user interface depicting the computed location of the target in the instrument coordinate system. Depicting the target in the instrument coordinate system may aid the operator in conducting the medical procedure. Consequently, the techniques of the present disclosure to compute accurately and in a timely manner target location (which may move, for example, under the anatomical deformations caused by pressure from the operating instrument, breathing, and / or other physiological processes) may improve procedure speed and accuracy.

[0168] FIG. 12A depicts a flow chart for an example method 1200 for training a machine learning and / or deep learning model for style transfer as discussed, for example, with reference to FIGS. 6A, B. The method may be implemented on one or more processors. At least some of the processors may be included in the processing unit 120. Additionally or alternatively, the one or more processors may be distributed (e.g., locally and / or in the cloud). At least one of the one or more processors may be a GPU.

[0169] At block 1210, the method 1200 includes computing, based on a three-dimensional ground-truth image, a set of two-dimensional projection images corresponding to a span of projection angles. For example, the span of projection angles may be no more than 120 degrees. In some examples, the method may include computing a set of two-dimensional images with a span of greater than 120 degrees. Though tomosynthesis based on fluoroscopy commonly uses less than 120 degrees, larger spans are possible. Furthermore, the method 1200 is applicable to enhancing images of other modalities. For example, the method 1200 may enhance a tomographic image obtained from sparsely-spaced angles covering a full range of 180 degrees.

[0170] The three-dimensional ground-truth images may be, for example, CT and / or CBCT images reconstructed, respectively, from CT and CBCT data. Other possible ground truthimages may include MRI, 3D ultrasound, or any other suitable 3D modality or combination of modalities. Additionally or alternatively, ground truth images may be at least in part computer-generated. Sets of two-dimensional projection images may be computed, for example, as projections of the ground-truth 3D images using suitable projection physics models.

[0171] A system for implementing the method 1200 may compute multiple sets of two- dimensional projection images based on a single 3D ground truth image. For example, one set may be computed for a set of projection angles spanning 120 degrees. Another set may be computed for a set of projection angles spanning 90 degrees. In any case, the angles may be distributed in any suitable way with any suitable angle spacing (e.g., 10°, 5°, 3°, 2°, 1 °, etc.). In some examples, a system may compute one set of projection images and use the one set to generate one or more additional sets. For example, the system may generate an additional set (or multiple) by adding noise and / or artifacts to the projection images. In other examples, an additional set may be generated from the first set by removing images associated with some of the projection images, e.g., on either side of the span of angles, every second angle, etc. For example, the additional set may have no more than half of the projections. That is, sparse projection sets may be generated from a more dense one.

[0172] At block 1220, the method 1200 includes computing, based on the set of two- dimensional projection images, a three-dimensional tomosynthesis training image. A system may generate the tomosynthesis image using back-projection, filtered back-projection, iterative reconstruction, and / or any other suitable reconstruction method. More generally, a training image need not be a tomosynthesis image. It may be an MRI image (e.g., degraded by adding noise or k-space errors with respect to the ground truth image), or any other suitable modality image for generating a style-transfer model.

[0173] A tomosynthesis image may be generated from a set of projection images based on a respective set angles. In some examples, a training system may add noise to at least some projection angles within the span of projection angles. That is, the projection images may be computed using one set of angles, but reconstructed using a slightly different set of angles, with systemic and / or random shifts with respect to the correct set on angles. In this manner, a training system may generate a model that is robust with respect to angle uncertainties.

[0174] At block 1230, the method 1200 includes using the three-dimensional tomosynthesis training image as an input into a style-transfer machine learning model to generate a three-dimensional output image. The style-transfer machine learning model mayinclude a CNN, a recursive neural network, a transformer network, a GAN, a diffusion model, or any other suitable model architecture and / or combination of models.

[0175] The style transfer model may generate at least one output for each of the input training images. In some examples, the model may be a combination of component models, and each of the component models may generate a separate output. The separate outputs may then be combined using a deterministic algorithm, for example.

[0176] At block 1240, the method 1200 includes adjusting the style-transfer machine learning model to reduce an indication of loss between the three-dimensional output image and the three-dimensional ground-truth image. In some examples, the training system may adjust model parameters using, for example, a gradient descent method. Additionally or alternatively, the system may adjust model architecture and / or other suitable hyperparameters. The method 1200 may be used to train a stye-transfer ML / DL model that may, as a part of method 1000 or 1100, enhance visualization of patient anatomy and, thereby, aid in visualization of an anatomical target, as described below with reference to FIG. 12B.

[0177] FIG. 12B depicts a flow chart for an example method 1250 for visualizing patient anatomy using style transfer. The method 1250 may be implemented using the system 100 or another suitable system. In some examples, the method 1250 may enhance an image reconstruction after in-painting and reconstruction techniques described throughout this disclosure. In other examples, the method 1250 may enhance reconstructed images independently of whether in-painting techniques are used.

[0178] At block 1260, the method 1250 includes obtaining a plurality of two-dimensional images associated with a first range of viewing angles, the two-dimensional images depicting an anatomical structure. For example, the two-dimensional images may be fluoroscopy images obtained over a span of angles of 60°, 90°, 120° or any other suitable range and with any suitable regular or uneven angle spacing (e.g., 10°, 5°, 3°, 2°, 1 °, etc.).

[0179] At block 1270, the method 1250 includes computing (e.g., by tomosynthesis), based at least in part on the plurality of two-dimensional images, a first three-dimensional image of at least a portion of the anatomical structure. The computation may include inpainting of fiducial markings and / or other objects disposed within the anatomy, as discussed above.

[0180] At block 1280, the method 1250 includes computing a second three-dimensional image, based on at least a portion of the first three-dimensional image and using one or more style-transfer models. The style transfer models may be any one of the style transfer models described above (e.g., with reference to FIG. 12A). Furthermore, the one or moremodels may be trained as described above with reference to FIG. 12A. For example, source data for the ground-truth used in training the one or more models may be CT and / or CBCT data and may be associated with a range of viewing angles larger than the first range of viewing angles. The one or more models may include models trained for inputs (e.g., training images) with different ranges of viewing angles, different angle sparsity, different noise levels, etc.

[0181] At block 1290, the method 1250 includes causing a display device to display a GUI depicting the second three-dimensional image. In some examples, the method 1250 may include causing a display device to display a GUI depicting style-transfer model options.The method may further include obtaining user input with respect to option choice. For example, the GUI may present a choice and collect input for choosing among models trained on input images with different angle ranges, different angle sparsity, etc. Additionally or alternatively, style-transfer model options may include target style for the second three- dimensional image. For a given input image, the target style may be CT, CBCT, MRI, etc. That is, the output image may resemble an image obtained with the respective modality. As discussed with respect to FIGS. 13A, B and 14A, B the style transfer may improve visualization of patient anatomy by improving the appearance of artifacts associated with tomosynthesis of limited angle range images.

[0182] FIGS. 13A, B depict, respectively, outputs of tomosynthesis imaging and CBCT imaging of patient anatomy. In FIG. 13A, panel 1310 depicts an axial slice, panel 1320 depicts a sagittal slice, and panel 1330 depicts a coronal slice through the reconstructed tomosynthesis image. In FIG. 13B, panel 1340 depicts an axial slice, panel 1350 depicts a sagittal slice, and panel 1360 depicts a coronal slice through the reconstructed CBCT image. The axial and sagittal slices of the tomosynthesis image exhibit significant imaging artifacts (e.g., smearing) and have considerably lower fidelity than the axial and sagittal slices of the CBCT image. The style-transfer techniques of the present disclosure may at least partially remedy the defects of tomosynthesis images as illustrated, for example, in FIGS. 14A, B.

[0183] FIGS. 14A, B illustrate performance of style-transfer models transferring tomosynthesis images into, respectively, a CT-style image and a CBCT -style image. The style transfer models may be trained as discussed with reference to FIGS. 6A, B and 12. FIGS. 14A, B illustrate that a style transfer example model based on U-net architecture may perform quite well (exhibit fidelity comparable to ground-truth images) with a moderate amount of reconstruction noise. With larger amounts of reconstruction noise, U-net architecture models develop blurring and artifacts that appear as holes within output images. On the other hand, example models based on GAN architecture exhibit better performancethan U-net architecture example models for image reconstruction with larger amounts of reconstruction noise.

[0184] One or more components of the examples discussed in this disclosure, such as control system 712, may be implemented in software for execution on one or more processors of a computer system. The software may include code that when executed by the one or more processors, configures the one or more processors to perform various functionalities as discussed herein. The code may be stored in a non-transitory computer readable storage medium (e.g., a memory, magnetic storage, optical storage, solid-state storage, etc.). The computer readable storage medium may be part of a computer readable storage device, such as an electronic circuit, a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device. The code may be downloaded via computer networks such as the Internet, Intranet, etc. for storage on the computer readable storage medium. The code may be executed by any of a wide variety of centralized or distributed data processing architectures. The programmed instructions of the code may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the systems described herein. The components of the computing systems discussed herein may be connected using wired and / or wireless connections. In some examples, the wireless connections may use wireless communication protocols such as Bluetooth, near-field communication (NFC), Infrared Data Association (IrDA), home radio frequency (HomeRF), IEEE 802.11 , Digital Enhanced Cordless Telecommunications (DECT), and wireless medical telemetry service (WMTS).

[0185] Various general-purpose computer systems may be used to perform one or more processes, methods, or functionalities described herein. Additionally or alternatively, various specialized computer systems may be used to perform one or more processes, methods, or functionalities described herein. In addition, a variety of programming languages may be used to implement one or more of the processes, methods, or functionalities described herein.

[0186] While certain examples and examples have been described above and shown in the accompanying drawings, it is to be understood that such examples and examples are merely illustrative and are not limited to the specific constructions and arrangements shown and described, since various other alternatives, modifications, and equivalents will be appreciated by those with ordinary skill in the art.

Claims

WHAT IS CLAIMED IS:1 . A tangible, non-transitory, computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: obtain a first plurality of two-dimensional images depicting an anatomical structure and one or more objects; compute, based at least in part on the first plurality of two-dimensional images, a second plurality of two-dimensional images by in-painting at least a portion of pixels associated with at least one of the one or more objects; and cause a display device to display a graphical user interface depicting a visualization based at least in part on at least one of the second plurality of two-dimensional images.

2. The tangible, non-transitory, computer readable medium of claim 1 , wherein the first plurality of two-dimensional images is associated with a plurality of respective viewing angles.

3. The tangible, non-transitory, computer readable medium of claim 2, wherein the one or more objects comprise one or more fiducial markings, and wherein the instructions, when executed by the one or more processors, further cause the one or more processors to compute the plurality of respective viewing angles based at least in part on the one or more fiducial markings.

4. The tangible, non-transitory, computer readable medium of claim 2 or 3, wherein: the instructions, when executed by the one or more processors, further cause the one or more processors to compute, based at least in part on the second plurality of two- dimensional images, a first three-dimensional image of at least a portion of the anatomical structure; and the visualization is based at least in part on the computed first three-dimensional image.

5. The tangible, non-transitory, computer readable medium of claim 4, wherein: the instructions, when executed by the one or more processors, further cause the one or more processors to compute a second three-dimensional image based on the first three-dimensional image using a style-transfer machine learning model; andthe visualization is based at least in part on the computed second three-dimensional image.

6. The tangible, non-transitory, computer readable medium of claim 5, wherein the style-transfer machine learning model comprises at least one of: i) a convolutional neural network model, ii) a transformer network model, iii) a generative network model, and / or iv) a diffusion model.

7. The tangible, non-transitory, computer readable medium of any one of claims 1 -6, wherein in-painting the at least a portion of pixels includes computing, by the one or more processors, at least one of the following: i) a Navier-Stokes gradient-based algorithm, and / or ii) a fast-marching method.

8. The tangible, non-transitory, computer readable medium of any one of claims 1 -7, wherein in-painting the at least a portion of pixels is i) progressive; ii) attention-based, and / or iii) pluralistic.

9. The tangible, non-transitory, computer readable medium of any one of claims 1 -8, wherein in-painting the at least a portion of pixels is based at least in part on machine learning and uses: i) an auto-encoder; ii) a generative-adversarial network, and / or iii) a diffusion model.

10. The tangible, non-transitory, computer readable medium of any one of claims 1 -9, wherein computing the second plurality of two-dimensional images by in-painting the at least a portion of pixels associated with the at least one of the one or more objects includes: generating a mask by including at least some of the portion of pixels associated with the at least one of the one or more objects; expanding the mask to include pixels adjacent to the included at least some of the portion of pixels associated with the at least one of the one or more objects; and inpainting pixels associated with the mask.11 . The tangible, non-transitory, computer readable medium of claim 10, wherein the one or more objects include a flexible elongate device, and generating a mask by including at least some of the portion of pixels associated with the at least one of the one or more objects includes: identifying, within at least one of the first plurality of two-dimensional images, a first set of pixels associated with the flexible elongate device;generating a three-dimensional representation of the flexible elongate device based at least in part on the identified first set of pixels associated with the flexible elongate device; and generating, within at least another one of the first plurality of two-dimensional images, a second set of pixels associated with the flexible elongate device based at least in part on the three-dimensional representation of the flexible elongate device.

12. The tangible, non-transitory, computer readable medium of any one of claims 1 -10, wherein the one or more objects include a flexible elongate device.

13. A system for visualizing patient anatomy during a medical procedure, the system comprising: a display device; one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to: obtain a first plurality of two-dimensional images depicting an anatomical structure and one or more objects; compute, based at least in part on the first plurality of two-dimensional images, a second plurality of two-dimensional images by in-painting at least a portion of pixels associated with at least one of the one or more objects; and cause the display device to display a graphical user interface depicting a visualization based at least in part on at least one of the second plurality of two- dimensional images.

14. The system of claim 13, wherein the first plurality of two-dimensional images is associated with a plurality of respective viewing angles.

15. The system of claim 14, wherein the one or more objects comprise one or more fiducial markings, and wherein the instructions, when executed by the one or more processors, further cause the one or more processors to compute the plurality of respective viewing angles based at least in part on the one or more fiducial markings.

16. The system of claim 14 or 15, wherein: the instructions, when executed by the one or more processors, further cause the one or more processors to compute, based at least in part on the second plurality of two-dimensional images, a first three-dimensional image of at least a portion of the anatomical structure; and the visualization is based at least in part on the computed first three-dimensional image.

17. The system of claim 16, wherein: the instructions, when executed by the one or more processors, further cause the one or more processors to compute a second three-dimensional image based on the first three-dimensional image using a style-transfer machine learning model; and the visualization is based at least in part on the computed second three-dimensional image.

18. The system of claim 17, wherein the style-transfer machine learning model comprises at least one of: i) a convolutional neural network model, ii) a transformer network model, iii) a generative network model, and / or iv) a diffusion model.

19. The system of any one of claims 13-18, wherein in-painting the at least a portion of pixels includes computing, by the one or more processors, at least one of the following: i) a Navier-Stokes gradient-based algorithm, and / or ii) a fast-marching method.

20. The system of any one of claims 13-19, wherein in-painting the at least a portion of pixels is i) progressive; ii) attention-based, and / or iii) pluralistic.21 . The system of any one of claims 13-19, wherein in-painting the at least a portion of pixels is based at least in part on machine learning and uses: i) an auto-encoder; ii) a generative-adversarial network, and / or iii) a diffusion model.

22. The system of any one of claims 13-21 , wherein computing the second plurality of two-dimensional images by in-painting the at least a portion of pixels associated with the at least one of the one or more objects includes: generating a mask by including at least some of the portion of pixels associated with the at least one of the one or more objects; expanding the mask to include pixels adjacent to at least some of the portion of pixels associated with the at least one of the one or more objects; and inpainting pixels associated with the mask.

23. The system of claim 22, wherein the one or more objects include a flexible elongate device, and generating a mask by including at least some of the portion of pixels associated with the at least one of the one or more objects includes: identifying, within at least one of the first plurality of two-dimensional images, a first set of pixels associated with the flexible elongate device; generating a three-dimensional representation of the flexible elongate device based at least in part on the identified first set of pixels associated with the flexible elongate device; and generating, within at least another one of the first plurality of two-dimensional images, a second set of pixels associated with the flexible elongate device based at least in part on the three-dimensional representation of the flexible elongate device.

24. The system of any one of claims 13-23, wherein the one or more objects include a flexible elongate device.

25. The system of claim 24, wherein the flexible elongate device comprises a catheter.

26. A method for visualizing patient anatomy during a medical procedure, the method comprising: obtaining, by one or more processors, a first plurality of two-dimensional images depicting an anatomical structure and one or more objects; computing, by the one or more processors, based at least in part on the first plurality of two-dimensional images, a second plurality of two-dimensional images by in-painting at least a portion of pixels associated with at least one of the one or more objects; and causing, by the one or more processors, a display device to display a graphical user interface depicting a visualization based at least in part on at least one of the second plurality of two-dimensional images.

27. The method of claim 26, wherein the first plurality of two-dimensional images is associated with a plurality of respective viewing angles.

28. The method of claim 27 further comprising computing, by the one or more processors, the plurality of respective viewing angles based at least in part on one or more fiducial markings.

29. The method of claim 27 or 28 further comprising computing, based at least in part on the second plurality of two-dimensional images, a first three-dimensional image of at least a portion of the anatomical structure; and wherein the visualization is based at least in part on the computed first three-dimensional image.

30. The method of claim 29, further comprising computing a second three- dimensional image based on the first three-dimensional image using a style-transfer machine learning model; and wherein the visualization is based at least in part on the computed second three-dimensional image.31 . The method of claim 30, wherein: the plurality of viewing angles has a span of no more than 120 degrees; and training the style-transfer machine learning model includes: computing, based on a three-dimensional ground-truth image, a set of two- dimensional projection images corresponding to a span of no more than 120 degrees; computing, based on the set of two-dimensional projection images, a three- dimensional tomosynthesis training image; using the three-dimensional tomosynthesis training image as an input into the style-transfer machine learning model to generate a three-dimensional output image; and adjusting the style-transfer machine learning model to reduce an indication of loss between the three-dimensional output image and the three-dimensional groundtruth image.

32. The method of claim 31 , wherein the three-dimensional ground-truth image is a computed tomography image based on a span of more than 120 degrees.

33. The method of claim 31 or 32, wherein computing the three-dimensional tomosynthesis training image includes adding noise to (i) the set of two-dimensional projection images, and / or (ii) a set of angles associated with the set of two-dimensional projection images.

34. The method of any one of claims 30-33, wherein the style-transfer machine learning model comprises at least one of: i) a convolutional neural network model, ii) a transformer network model, iii) a generative network model, and / or iv) a diffusion model.

35. The method of any one of claims 26-34, wherein in-painting the at least a portion of pixels includes computing, by the one or more processors, at least one of the following: i) a Navier-Stokes gradient-based algorithm, and / or ii) a fast-marching method.

36. The method of any one of claims 26-35, wherein in-painting the at least a portion of pixels is i) progressive; ii) attention-based, and / or ill) pluralistic.

37. The method of any one of claims 26-35, wherein in-painting the at least a portion of pixels is based at least in part on machine learning and uses: i) an auto-encoder; ii) a generative-adversarial network, and / or ill) a diffusion model.

38. The method of any one of claims 26-37, wherein computing the second plurality of two-dimensional images by in-painting the at least a portion of pixels associated with the at least one of the one or more objects includes: generating a mask by including at least some of the portion of pixels associated with the at least one of the one or more objects; dilating the mask to include pixels adjacent to at least some of the portion of pixels associated with the at least one of the one or more objects; and inpainting pixels associated with the mask.

39. The method of claim 38, wherein the one or more objects include a flexible elongate device, and generating a mask by including at least some of the portion of pixels associated with the at least one of the one or more objects includes: identifying, within at least one of the first plurality of two-dimensional images, a first set of pixels associated with the flexible elongate device; generating a three-dimensional representation of the flexible elongate device based at least in part on the identified first set of pixels associated with the flexible elongate device; and generating, within at least another one of the first plurality of two-dimensional images, a second set of pixels associated with the flexible elongate device based at least in part on the three-dimensional representation of the flexible elongate device.

40. The method of any one of claims 26-38, wherein the one or more objects include a flexible elongate device.41 . A tangible, non-transitory, computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to:obtain a plurality of two-dimensional projection images corresponding to a plurality of respective projection angles and depicting an anatomical structure and a flexible elongate device; identify, within each of at least two of the plurality of two-dimensional projection images, respective two-dimensional coordinates of a reference point disposed at the flexible elongate device; compute, based at least in part on the respective two-dimensional coordinates and two respective projection angles, three dimensional coordinates, in an imaging coordinate system, of the reference point; identify, within at least one of the at least two of the plurality of two-dimensional projection images, a curve corresponding to at least a portion of the flexible elongate device including the reference point; receive, from a sensing unit, shape data in a sensing coordinate system for the at least a portion of the flexible elongate device including the reference point; register the sensing coordinate system to the imaging coordinate system based at least in part on the received shape data and the identified curved within the at least one of the at least two of the plurality of two-dimensional projection images; reconstruct, based at least in part on the plurality of two-dimensional projection images corresponding to the plurality of respective projection angles, a three-dimensional image of the anatomical structure; identify, within the reconstructed three-dimensional image of the anatomical structure, a target; and compute, in an instrument coordinate system and based at least in part on identifying the target and registering the sensing coordinate system to the imaging coordinate system, a location of the target.

42. The tangible, non-transitory, computer readable medium of claim 41 , wherein causing the one or more processors to identify, within each of the at least two of the plurality of two-dimensional projection images, the respective two-dimensional coordinates of the reference point disposed at the flexible elongate device, includes causing the one or more processors to: generate, on a display unit, a graphical user interface depicting the at least two of the plurality of two-dimensional projection images; and record selections, made within the graphical user interface, of the respective two- dimensional coordinates of the reference point.

43. The tangible, non-transitory, computer readable medium of claim 41 , wherein causing the one or more processors to identify, within the at least one of the at least two of the plurality of two-dimensional projection images, the curve corresponding to the at least a portion of the flexible elongate device including the reference point, includes causing the one or more processors to: generate, on a display unit, a graphical user interface depicting the at least one of the at least two of the plurality of two-dimensional projection images; and record selections, made within the graphical user interface, of respective two- dimensional coordinates of at least two points along the curve.

44. The tangible, non-transitory, computer readable medium of claim 41 , wherein causing the one or more processors to identify, within the at least one of the at least two of the plurality of two-dimensional projection images, the curve corresponding to the at least a portion of the flexible elongate device including the reference point, includes causing the one or more processors to: generate, on a display unit, a graphical user interface depicting a visualization of the reconstructed three-dimensional image of the anatomical structure; and record a selection, made within the graphical user interface, of three-dimensional coordinates of an estimated center of the target.

45. The tangible, non-transitory, computer readable medium of any one of claims 41 -44, wherein causing the one or more processors to register the sensing coordinate system to the imaging coordinate system, includes causing the one or more processors to: select a plurality of points along the curve within the at least one of the at least two of the plurality of two-dimensional projection images corresponding to the at least a portion of the flexible elongate device including the reference point; identify, for each of the selected plurality of points along the curve, two-dimensional coordinates of a respective point in another one of the at least two of the plurality of two- dimensional projection images; and compute, for each of the selected plurality of points along the curve, three- dimensional coordinates in the image coordinate systems based at least in part on the identified two-dimensional coordinates of the respective point in the another one of the at least two of the plurality of two-dimensional projection images.

46. The tangible, non-transitory, computer readable medium of any one of claims 41 -44, wherein causing the one or more processors to register the sensing coordinate system to the imaging coordinate system, includes causing the one or more processors to:select a plurality of points along the curve within the at least one of the at least two of the plurality of two-dimensional projection images corresponding to the at least a portion of the flexible elongate device including the reference point; and compute, for each of the selected plurality of points along the curve, three- dimensional coordinates in the image coordinate systems based at least in part on back- tracing along the respective projection angle to find an intensity peak within the reconstructed three-dimensional image of the anatomical structure.

47. The tangible, non-transitory, computer readable medium of any one of claims 41-46, wherein causing the one or more processors to register the sensing coordinate system to the imaging coordinate system, includes causing the one or more processors to: compute a two-dimensional projection based on the received shape data; and compute a coordinate transformation based at least in part on minimizing a difference between the two-dimensional projection based on the received shape data and the curve corresponding to the at least a portion of the flexible elongate device including the reference point.

48. The tangible, non-transitory, computer readable medium of any one of claims 41-47, wherein: the instructions further cause the one or more processors to detect a registration error; and computing the location of the target in the instrument coordinate system includes, in response to detecting the registration error, generating a translational shift with respect to a prior location of the target.

49. The tangible, non-transitory, computer readable medium of any one of claims 41-48, wherein the instructions further cause the one or more processors to: generate, based at least in part on registering the sensing coordinate system to the imaging coordinate system and the received shape data, within the plurality of two- dimensional projection images a respective plurality of in-painting masks; and generate a plurality of in-painted two-dimensional projection images at least in part by in-painting pixels corresponding to the in-painting masks within the plurality of two- dimensional projection images; and wherein: reconstructing the three-dimensional image of the anatomical structure is based at least in part on the plurality of in-painted two-dimensional projection images.

50. The tangible, non-transitory, computer readable medium of any one of claims 41-49, wherein the plurality of two-dimensional projection images are fluoroscopy images, and the plurality of respective projection angles spans no more than 120 degrees.51 . A system for visualizing patient anatomy during a medical procedure, the system comprising: a display device; one or more processors; and one or more non-transitory, computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to: obtain a plurality of two-dimensional projection images corresponding to a plurality of respective projection angles and depicting an anatomical structure and a flexible elongate device; identify, within each of at least two of the plurality of two-dimensional projection images, respective two-dimensional coordinates of a reference point disposed at the flexible elongate device; compute, based at least in part on the respective two-dimensional coordinates and two respective projection angles, three dimensional coordinates, in an imaging coordinate system, of the reference point; identify, within at least one of the at least two of the plurality of two- dimensional projection images, a curve corresponding to at least a portion of the flexible elongate device including the reference point; receive, from a sensing unit, shape data in a sensing coordinate system for the at least a portion of the flexible elongate device including the reference point; register the sensing coordinate system to the imaging coordinate system based at least in part on the received shape data and the identified curved within the at least one of the at least two of the plurality of two-dimensional projection images; reconstruct, based at least in part on the plurality of two-dimensional projection images corresponding to the plurality of respective projection angles, a three-dimensional image of the anatomical structure; identify, within the reconstructed three-dimensional image of the anatomical structure, a target; compute, in an instrument coordinate system and based at least in part on identifying the target and registering the sensing coordinate system to the imaging coordinate system, a location of the target; and cause the display device to display a graphical user interface depicting the computed location of the target in the instrument coordinate system.

52. The system of claim 51 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to: update the graphical user interface to depict the at least two of the plurality of two- dimensional projection images; and record selections, made within the graphical user interface, of the respective two- dimensional coordinates of the reference point.

53. The system of claims 51 or 52, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to: update the graphical user interface to depict the at least one of the at least two of the plurality of two-dimensional projection images; and record selections, made within the graphical user interface, of respective two- dimensional coordinates of at least two points along the curve.

54. The system of any one of claims 51-53, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to: update the graphical user interface to depict a visualization of the reconstructed three-dimensional image of the anatomical structure; and record a selection, made within the graphical user interface, of three-dimensional coordinates of an estimated center of the target.

55. The system of any one of claims 51-54, wherein causing the one or more processors to register the sensing coordinate system to the imaging coordinate system, includes causing the one or more processors to: select a plurality of points along the curve within the at least one of the at least two of the plurality of two-dimensional projection images corresponding to the at least a portion of the flexible elongate device including the reference point; identify, for each of the selected plurality of points along the curve, two-dimensional coordinates of a respective point in another one of the at least two of the plurality of two- dimensional projection images; and compute, for each of the selected plurality of points along the curve, three- dimensional coordinates in the image coordinate systems based at least in part on the identified two-dimensional coordinates of the respective point in the another one of the at least two of the plurality of two-dimensional projection images.

56. The system of any one of claims 51 -54, wherein causing the one or more processors to register the sensing coordinate system to the imaging coordinate system, includes causing the one or more processors to: select a plurality of points along the curve within the at least one of the at least two of the plurality of two-dimensional projection images corresponding to the at least a portion of the flexible elongate device including the reference point; and compute, for each of the selected plurality of points along the curve, three- dimensional coordinates in the image coordinate systems based at least in part on back- tracing along the respective projection angle to find an intensity peak within the reconstructed three-dimensional image of the anatomical structure.

57. The system of any one of claims 51-56, wherein causing the one or more processors to register the sensing coordinate system to the imaging coordinate system, includes causing the one or more processors to: compute a two-dimensional projection based on the received shape data; and compute a coordinate transformation based at least in part on minimizing a difference between the two-dimensional projection based on the received shape data and the curve corresponding to the at least a portion of the flexible elongate device including the reference point.

58. The system of any one of claims 51 -57, wherein: the instructions further cause the one or more processors to detect a registration error; and computing the location of the target in the instrument coordinate system includes, in response to detecting the registration error, generating a translational shift with respect to a prior location of the target.

59. The system of any one of claims 51-58, wherein the instructions further cause the one or more processors to: generate, based at least in part on registering the sensing coordinate system to the imaging coordinate system and the received shape data, within the plurality of two- dimensional projection images a respective plurality of in-painting masks; and generate a plurality of in-painted two-dimensional projection images at least in part by in-painting pixels corresponding to the in-painting masks within the plurality of two- dimensional projection images; and wherein:reconstructing the three-dimensional image of the anatomical structure is based at least in part on the plurality of in-painted two-dimensional projection images.

60. The system of any one of claims 51 -59, wherein the plurality of two- dimensional projection images are fluoroscopy images, and the plurality of respective projection angles spans no more than 120 degrees.61 . A method for visualizing patient anatomy during a medical procedure, the method comprising: obtaining, by one or more processors, a plurality of two-dimensional projection images corresponding to a plurality of respective projection angles and depicting an anatomical structure and a flexible elongate device; identifying, by the one or more processors, within each of at least two of the plurality of two-dimensional projection images, respective two-dimensional coordinates of a reference point disposed at the flexible elongate device; computing, by the one or more processors, based at least in part on the respective two-dimensional coordinates and two respective projection angles, three dimensional coordinates, in an imaging coordinate system, of the reference point; identifying, by the one or more processors, within at least one of the at least two of the plurality of two-dimensional projection images, a curve corresponding to at least a portion of the flexible elongate device including the reference point; receiving, by the one or more processors, from a sensing unit, shape data in a sensing coordinate system for the at least a portion of the flexible elongate device including the reference point; registering, by the one or more processors, the sensing coordinate system to the imaging coordinate system based at least in part on the received shape data and the identified curved within the at least one of the at least two of the plurality of two-dimensional projection images; reconstructing, by the one or more processors, based at least in part on the plurality of two-dimensional projection images corresponding to the plurality of respective projection angles, a three-dimensional image of the anatomical structure; identifying, by the one or more processors, within the reconstructed three- dimensional image of the anatomical structure, a target; computing, by the one or more processors, in an instrument coordinate system and based at least in part on identifying the target and registering the sensing coordinate system to the imaging coordinate system, a location of the target; andcausing, by the one or more processors, a display device to display a graphical user interface depicting the computed location of the target in the instrument coordinate system.

62. The method of claim 61 , further comprising: updating, by the one or more processors, the graphical user interface to depict the at least two of the plurality of two-dimensional projection images; and recording, by the one or more processors, selections, made within the graphical user interface, of the respective two-dimensional coordinates of the reference point.

63. The method of claims 61 or 62, further comprising: updating, by the one or more processors, the graphical user interface to depict the at least one of the at least two of the plurality of two-dimensional projection images; and recording, by the one or more processors, selections, made within the graphical user interface, of respective two-dimensional coordinates of at least two points along the curve.

64. The method of any one of claims 61 -63, further comprising: updating, by the one or more processors, the graphical user interface to depict a visualization of the reconstructed three-dimensional image of the anatomical structure; and recording, by the one or more processors, a selection, made within the graphical user interface, of three-dimensional coordinates of an estimated center of the target.

65. The method of any one of claims 61 -64, wherein registering the sensing coordinate system to the imaging coordinate system includes: selecting, by the one or more processors, a plurality of points along the curve within the at least one of the at least two of the plurality of two-dimensional projection images corresponding to the at least a portion of the flexible elongate device including the reference point; identifying, by the one or more processors, for each of the selected plurality of points along the curve, two-dimensional coordinates of a respective point in another one of the at least two of the plurality of two-dimensional projection images; and computing, by the one or more processors, for each of the selected plurality of points along the curve, three-dimensional coordinates in the image coordinate systems based at least in part on the identified two-dimensional coordinates of the respective point in the another one of the at least two of the plurality of two-dimensional projection images.

66. The method of any one of claims 61 -64, wherein registering the sensing coordinate system to the imaging coordinate system includes:selecting, by the one or more processors, a plurality of points along the curve within the at least one of the at least two of the plurality of two-dimensional projection images corresponding to the at least a portion of the flexible elongate device including the reference point; and computing, by the one or more processors, for each of the selected plurality of points along the curve, three-dimensional coordinates in the image coordinate systems based at least in part on back-tracing along the respective projection angle to find an intensity peak within the reconstructed three-dimensional image of the anatomical structure.

67. The method of any one of claims 61-66, wherein registering the sensing coordinate system to the imaging coordinate system includes: computing, by the one or more processors, a two-dimensional projection based on the received shape data; and computing, by the one or more processors, a coordinate transformation based at least in part on minimizing a difference between the two-dimensional projection based on the received shape data and the curve corresponding to the at least a portion of the flexible elongate device including the reference point.

68. The method of any one of claims 61 -67, further comprising detecting, by the one or more processors, a registration error; and wherein: identifying or updating, by the one or more processors, the location of the target in the instrument coordinate system includes, in response to detecting the registration error, generating a translational shift with respect to a previously computed location of the target.

69. The method of any one of claims 61 -68, further comprising: generating, by the one or more processors, based at least in part on registering the sensing coordinate system to the imaging coordinate system and the received shape data, within the plurality of two-dimensional projection images a respective plurality of in-painting masks; and generating, by the one or more processors, a plurality of in-painted two-dimensional projection images at least in part by in-painting pixels corresponding to the in-painting masks within the plurality of two-dimensional projection images; and wherein: reconstructing the three-dimensional image of the anatomical structure is based at least in part on the plurality of in-painted two-dimensional projection images.

70. The method of any one of claims 61 -69, wherein the plurality of two- dimensional projection images are fluoroscopy images, and the plurality of respective projection angles spans no more than 120 degrees.

Images