System and method for pose estimation for fluoroscopy imaging devices and for three-dimensional imaging of body structures

Abstract

The present disclosure relates to systems and methods for pose estimation of a fluoroscopy imaging device and for three-dimensional imaging of a body structure. The imaging systems and methods estimate the pose of a fluoroscopy imaging device that can be used to reconstruct 3D volume data of a target region based on a sequence of fluoroscopy images of a medical device or points on a medical device (e.g., radiopaque markers) captured by performing a fluoroscopy sweep. The systems and methods can identify and track points along the length of the medical device that appear in the captured fluoroscopy images. The 3D coordinates of the points can be obtained, for example, from electromagnetic sensors, or by performing a motion-deriving structure method on the captured fluoroscopy images. In other aspects, the 3D shape of the catheter is determined, and then the angle of the 3D catheter projection onto the 2D catheter in each captured fluoroscopy image is found.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Patent Application No. 62 / 897,760, filed September 9, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of imaging, and more particularly to attitude estimation of imaging devices and three-dimensional imaging of body structures. Background Technology

[0004] During surgery, fluoroscopic imaging equipment is typically located in the operating room to guide medical devices to their target locations within the patient's body. For example, clinicians can use fluoroscopic imaging to visualize and confirm the placement of medical devices, either while or after they have been guided to the desired location. While standard fluoroscopic images can visualize high-density objects such as metal tools and bones, as well as large soft tissue objects like the heart, they struggle to distinguish small soft tissue objects of interest, such as lesions to be ablated. Furthermore, fluoroscopic images are only two-dimensional projections, while 3D imaging is required for accurate and safe navigation within the body.

[0005] Therefore, there is a need for rapid, accurate, and robust three-dimensional reconstruction of structures based on fluorescence imaging during medical procedures. Summary of the Invention

[0006] In one aspect, this disclosure features a method for estimating the orientation of a fluoroscopic imaging device. The method includes capturing a fluoroscopic image of a catheter by scanning with the fluoroscopic imaging device. The method further includes identifying and tracking radiopaque markers along the length of the catheter in the fluoroscopic image of the catheter. The method also includes determining the three-dimensional (3D) coordinates of the catheter based on the tracked radiopaque markers. The method further includes estimating the orientation of the fluoroscopic imaging device based on the 3D coordinates of the catheter.

[0007] In various aspects, embodiments of this disclosure may include one or more of the following features: The radiopaque marker may be a tracking sensor. The tracking sensor may be a coil. The sweep may be a wide sweep, which includes a field of view of the fluorescence fluoroscopic imaging device around the longitudinal axis relative to a front-to-back position of greater than 30 degrees.

[0008] In another aspect, this disclosure features a method comprising capturing fluoroscopic images of a patient's body by scanning with a fluoroscopic imaging device. The method further comprises identifying and tracking radiopaque markers along the length of a catheter traveling through the patient's body in the fluoroscopic images. The method also includes a 3D motion-based structural estimation method on the tracked radiopaque markers to estimate the three-dimensional (3D) structure of the catheter and the orientation of the fluoroscopic imaging device. In some aspects, the method further includes constructing 3D volumetric data of a region based on the estimated orientation of the fluoroscopic imaging device.

[0009] In another aspect, this disclosure features a method for estimating the orientation of a fluoroscopic imaging device. The method includes determining the three-dimensional (3D) shape of a catheter. The method also includes capturing fluoroscopic images of the catheter within a patient body by scanning with the fluoroscopic imaging device. Furthermore, the method includes estimating, for each fluoroscopic image, the orientation of the catheter projected onto each fluoroscopic image, based on the 3D shape of the catheter.

[0010] In various aspects, embodiments of this disclosure may include one or more of the following features. The scan may be a wide scan, including a view of the fluoroscopic imaging device around the longitudinal axis of the patient's body at an anteroposterior position greater than 50 degrees. The catheter may include at least one fiber optic sensor disposed along the length of the catheter, and the 3D shape of the catheter may be determined by a 3D shape sensing method based on fiber optic sensor signals obtained from the at least one fiber optic sensor. Determining the 3D shape of the catheter may include a motion-based structural process of the fluoroscopic image to estimate a first 3D shape of the catheter, determining the body structure on which the catheter is disposed, determining the 3D shape of the body structure on which the catheter is disposed, determining a second 3D shape of the catheter based on the 3D shape of the body structure on which the catheter is disposed, and estimating the 3D shape of the catheter based on the first and second 3D shapes. The 3D shape of the body structure on which the catheter is disposed may be determined based on computed tomography (CT) images of the body structure on which the catheter is disposed. The body structure may be the airways of the lungs.

[0011] In various aspects, the fluoroscopic images may include a first fluoroscopic image of the catheter and a second fluoroscopic image of the catheter and marker structures, and determining the 3D shape of the catheter may include projection of the marker structures onto each image of the second fluoroscopic image, estimating the orientation of the fluoroscopic imaging device for each image of the second fluoroscopic image, and reconstructing the 3D shape of the catheter based on the estimated orientation of the fluoroscopic imaging device. The marker structures may be marker grids. The second fluoroscopic image may correspond to the viewpoint of the fluoroscopic imaging device around the longitudinal axis of the patient's body at an anteroposterior position of less than 30 degrees.

[0012] In another aspect, this disclosure features a method for constructing fluorescence-based 3D volumetric data of a target region within a patient's body. The method includes acquiring a sequence of fluorescence-based fluoroscopic images of the target region and radiopaque markers of a medical device by sweeping with a fluorescence-based fluoroscopic imaging apparatus. The method further includes identifying and tracking radiopaque markers along the length of the medical device in the sequence of fluorescence-based fluoroscopic images of the medical device traveling through the patient's body. The method also includes determining the three-dimensional (3D) coordinates of the medical device based on the tracked radiopaque markers. The method further includes estimating the angle of the fluorescence-based fluoroscopic imaging apparatus based on the 3D coordinates of the medical device. Finally, the method includes constructing fluorescence-based 3D volumetric data of the target region based on the estimated angle of the fluorescence-based fluoroscopic imaging apparatus.

[0013] In various aspects, embodiments of this disclosure may include one or more of the following features. The method further includes determining an offset between a medical device and a target region based on three-dimensional volumetric data obtained from fluorescence fluoroscopy. The method may also include using a positioning system indicating the position of the medical device on a display to facilitate navigation of the medical device to the target region. The method may further include correcting the displayed position of the medical device relative to the target region based on the determined offset between the medical device and the target region. The positioning system may be an electromagnetic positioning system. The method may further include displaying a 3D rendering of the target region on the display. The method may further include registering the positioning system to the 3D rendering. Correcting the position of the medical device relative to the target region includes updating the registration of the positioning system to the 3D rendering. The method may further include generating a 3D rendering of the target region based on previously acquired CT volumetric data of the target region. The target region may include at least a portion of a lung, and the medical device may navigate to the target region via an airway network. The target region may include at least a portion of a lung. The target region may include a soft tissue target. The target region may include a target to be ablated. Attached Figure Description

[0014] Various exemplary aspects are shown in the accompanying drawings, and these examples are not intended to be limiting. It should be understood that, for the sake of simplicity and clarity, the elements shown in the figures mentioned below are not necessarily drawn to scale. Furthermore, where deemed appropriate, reference numerals may be repeated in the figures to indicate similar, corresponding, or analogous elements. The figures are listed below.

[0015] Figure 1A This is a schematic diagram of an exemplary system for constructing three-dimensional volume data based on fluorescence perspective, according to various aspects of this disclosure.

[0016] Figure 1B This is a schematic diagram of a system configured for use with various aspects of this disclosure;

[0017] Figure 2A This is a schematic diagram of a conduit configured for use with various aspects of this disclosure;

[0018] Figure 2B This is a schematic diagram of another conduit configured for use in conjunction with various aspects of this disclosure;

[0019] Figure 3 This is a flowchart of a method for estimating the attitude of a fluorescence fluoroscopic imaging apparatus according to various aspects of this disclosure;

[0020] Figure 4 This is a flowchart of another method for estimating the attitude of a fluorescence fluoroscopic imaging device according to various aspects of this disclosure;

[0021] Figure 5A and 5B This is a flowchart illustrating other methods for estimating the attitude of a fluorescence fluoroscopic imaging apparatus according to various aspects of this disclosure; and

[0022] Figure 6 This is a flowchart of a method for generating 3D volumetric data according to various aspects of this disclosure. Detailed Implementation

[0023] This disclosure relates to an improved fluoroscopic navigation system and method, sufficient for procedures requiring accurate and robust three-dimensional (3D) imaging, such as biopsies and ablation procedures. In some cases, lesions may appear tailed in fluoroscopic images, and the shape of the lesion may not be accurate in the fluoroscopic images. For example, fluoroscopic images may not be sufficient to determine that the ablation zone will completely cover the lesion. If a fluoroscopic imaging scan of greater than 50 degrees is performed around the anterior-posterior (AP) position, the image quality is improved, making it sufficient for accurate and safe ablation procedures. When the fluoroscopic imaging scan is close to 180 degrees around the AP position, the image quality may approach that achievable using a cone-beam computed tomography (CBCT) system.

[0024] For certain beds or operating tables, such as those without crossbars or those with adjustable crossbars, the fluoroscopy device can be mechanically rotated more than 50 degrees around the AP position. However, when the fluoroscopy device is positioned too laterally relative to the patient's body, the projection of the marker grid placed below the patient to estimate the orientation of the fluoroscopy device may disappear from the fluoroscopic image; for example, when the angle of the fluoroscopy device view relative to the AP position is greater than 75 degrees. For instance, when the patient has an anterior lesion and the fluoroscopy device is rotated toward a lateral position, the marker grid begins to disappear from the fluoroscopic image, and only the patient's body appears in the fluoroscopic image.

[0025] According to various aspects of this disclosure, 3D reconstruction of a structure within a target region requires measuring the angle of a fluoroscopic imaging device relative to the target region. In some aspects, fluoroscopic-based systems and methods use medical devices such as catheters or markers placed thereon to estimate the orientation of the fluoroscopic imaging device while capturing fluoroscopic images of the patient's body. The estimated orientation can then be used to reconstruct 3D volumetric data of the target region. The systems and methods can track points, for example, radiopaque markers, along the length of the medical device appearing in the captured fluoroscopic images. The 3D coordinates of these points can be obtained, for example, from an electromagnetic sensor or by a motion-based structure-finding method on the captured fluoroscopic images. In various aspects, the shape of the catheter in 3D is determined, and then, for each captured fluoroscopic image, the angle at which the 3D catheter is projected onto the 2D catheter is found. The 3D shape of the catheter can be determined based on the shape of the body structure on which the medical device or catheter is placed.

[0026] In some aspects, the systems and methods of this disclosure use the shape of a catheter from a portion of a fluorescence fluoroscopy video image, in which a marker grid is visible. Based on a 3D reconstruction of those fluorescence fluoroscopy video images, the 3D shape of the catheter can be extracted. For all other frames in the fluorescence fluoroscopy video where the marker grid is too oblique to be seen, the shape of the catheter is still visible in the image. Therefore, based on the shape of the catheter in the projected image and the 3D shape of the catheter, the viewing angle of the fluorescence fluoroscopy imaging device relative to the catheter can be determined based on the fluorescence fluoroscopy video frames where the marker grid is too oblique to be seen.

[0027] Reference points (e.g., markers) can be placed on the catheter, or the entire shape of the catheter can be used. Reference points can be radiopaque loops spaced at predetermined distances along the length of the catheter, such as 1 cm, 1.5 cm, 2 cm, etc. The 3D shape of the catheter can be reconstructed using a narrow sweep (e.g., 50 degrees) around the AP view, and the 3D position of the radiopaque loops can then be detected along the catheter. Therefore, the system of this disclosure can address other angles where the marker grid is not visible from a wide sweep (e.g., a sweep between 160 and 180 degrees around the AP position) based on the 3D position of the radiopaque loops or other suitable markers.

[0028] Figure 1AAn aspect of an electromagnetic navigation (EMN) system 100 that can be used in various aspects of the systems and methods disclosed herein is depicted. The EMN system 100 is configured to view CT image data to identify one or more targets, plan a path to the identified target (planning phase), navigate the catheter 12 of the catheter guide assembly 40 to the target via a user interface (navigation phase), and confirm the placement of the catheter 12 (or any part of the catheter guide assembly 40 or any instrument inserted therein) relative to the target. One such electromagnetic navigation system is the ELECTROMAGNETIC NAVIGATION currently sold by Medtronic PLC. The system. The target can be tissue of interest, such as tissue to be ablated, or a relevant area identified during the review of CT image data during the planning phase. After navigation, a medical device, such as a biopsy tool, delivery device, or treatment device, can be inserted into catheter 12 to obtain a tissue sample from tissue located at or near the target location, deliver articles or therapies to the area, or treat the region.

[0029] like Figure 1A As shown, catheter 12 is part of catheter guiding assembly 40, which extends distally from handle 41 of catheter guiding assembly 40. In practice, catheter 12 can be inserted into bronchoscope 30 to access the lumen network of a patient “P”. Specifically, catheter 12 of catheter guiding assembly 40 can be inserted into the working channel of bronchoscope 30 for navigation through the patient’s lumen network. Positionable guide (LG) 32 (including sensor 44 disposed thereon) is inserted into catheter 12 and locked in place such that sensor 44 extends beyond the desired distance of the distal tip of catheter 12. The position and orientation of sensor 44 relative to a reference coordinate system in an electromagnetic field can be derived, and therefore the position and orientation of the distal end of catheter 12 can be derived.

[0030] The catheter guidance assembly 40 is currently manufactured by Medtronic Ltd. under the trade name... The surgical kit or EDGETM surgical kit is marketed and sold and is considered to be usable in conjunction with this disclosure.

[0031] EMN system 100 typically includes an operating table 20 configured to support a patient “P”; a bronchoscope 30 configured to be inserted into the airway of the patient “P” through the patient’s mouth; a monitoring device 120 coupled to the bronchoscope 30 (e.g., a video display for displaying video images received from a video imaging system of the bronchoscope 30); a tracking system 50 including a tracking module 52, multiple reference sensors 54 and a transmitter pad 56; and a computing device 125 including software and / or hardware for facilitating target identification, path planning to the target, navigation of medical devices or instruments to the target, and confirmation of placement of the catheter 12 or a suitable device passing through it relative to the target.

[0032] System 100 also includes a fluoroscopic imaging device 110 capable of acquiring fluoroscopic or X-ray images or videos of patient “P”. Fluoroscopic images, image sequences, or videos captured by the fluoroscopic imaging device 110 can be stored within the fluoroscopic imaging device 110 or transmitted to a computing device 125 for storage, processing, and display, as described in more detail herein. Additionally, the fluoroscopic imaging device 110 can be moved relative to patient “P”, allowing images to be acquired from different angles or viewpoints relative to patient “P” to create fluoroscopic videos. In one aspect of this disclosure, the fluoroscopic imaging device 110 includes an angle measuring device 111 configured to measure the angle of the fluoroscopic imaging device 110 relative to patient “P”. The angle measuring device 111 may be an accelerometer.

[0033] Fluorescence fluoroscopy imaging apparatus 110 may include one or more imaging devices. In aspects including multiple imaging devices, each imaging device may be of a different type or the same type. In various aspects, fluorescence fluoroscopy imaging apparatus 110 is based on a C-mounted fluoroscope with a C-arm 60. At one end of the C-arm 60 is an X-ray source 62 including an X-ray tube and a collimator (not shown). At the other end of the C-arm 60 is an X-ray detector 66 including an anti-scatter grid, an image intensifier, and a CCD camera (not shown). The collimator blocks X-rays emitted from the X-ray tube except at an aperture (not shown). The X-ray cone exits from the aperture and illuminates the anti-scatter grid and image intensifier of the X-ray detector 66. The image created in the image intensifier is captured by the CCD camera. Depending on the spatial density distribution in the object (e.g., a patient) through which the cone passes, each element of the CCD array of the CCD camera receives more or less light from the image intensifier, and the corresponding pixels of the image produced by the C-mounted fluoroscope are correspondingly darker or brighter.

[0034] The computing device 125 may be any suitable computing device including a processor and storage medium, wherein the processor is capable of executing instructions stored on the storage medium. The computing device 125 is operatively coupled to some or all components of the system 100, including the bronchoscope 30, the catheter guidance assembly 40, the positioning guide 32, and the tracking system 50. The computing device 125 may further include a database configured to store patient data, a CT dataset including CT images and volumetric renderings, a fluoroscopy dataset including fluoroscopy images and videos, navigation planning, and any other such data. Although not explicitly shown, the computing device 125 may include inputs, or may be otherwise configured to receive CT datasets, fluoroscopy images or videos, and other data as described herein. Additionally, the computing device 125 includes a display configured to display a graphical user interface. The computing device 125 may be connected to one or more networks through which one or more databases can be accessed.

[0035] Regarding the planning phase, computing device 125 uses previously acquired CT image data to generate and view a three-dimensional model of the patient "P's" airway, enabling target identification on the three-dimensional model (automatic, semi-automatic, or manual), and allowing the determination of the path through the patient "P's" airway to the tissues located at and around the target. More specifically, CT images acquired from previous CT scans are processed and compiled into a three-dimensional CT volume, which is then used to generate a three-dimensional model of the patient "P's" airway.

[0036] The 3D model can be displayed on a monitor associated with the computing device 125, or in any other suitable manner. Various views of the 3D model or a 2D image generated from the 3D model are presented using the computing device 125. The 3D model can be manipulated to facilitate target identification on the 3D model or 2D image, and to select an appropriate path through the patient's "P" airway to reach the tissue located at the target. Once selected, the path planning, the 3D model, and the resulting image can be saved and exported to the navigation system for use during one or more navigation phases. One such planning software is currently sold by Medtronic Ltd. Planning kit.

[0037] Regarding the navigation phase, image and navigation path registration can be performed using a six-degree-of-freedom electromagnetic tracking system 50 or other suitable positioning measurement systems, but other configurations are also anticipated. The tracking system 50 includes a tracking module 52, a reference sensor 54, and a transmitter pad 56. The tracking system 50 is configured for use with a positionable guide 32 and a sensor 44. As described above, the positionable guide 32 and the sensor 44 are configured for insertion into the airway of the patient “P” (with or without a bronchoscope 30) via the catheter 12 and can be selectively locked relative to each other via a locking mechanism.

[0038] A transmitter pad 56 is positioned below the patient “P”. The transmitter pad 56 generates an electromagnetic field around at least a portion of the patient “P”, within which a tracking module 52 can be used to determine the positions of a plurality of reference sensors 54 and sensor elements 44. The transmitter pad 56 may include a structure or grid of at least partially transmissive markers, which, in some aspects of this disclosure, are used to determine the 3D shape of a medical device or catheter being guided toward a target. In some aspects, in addition to the transmitter pad 56, one or more of the reference sensors 54 are also attached to the chest of the patient “P”. In other aspects, only the transmitter pad is used. The six-DOF coordinates of the reference sensors 54 are sent to a computing device 125 (which includes appropriate software), in which these coordinates are used to calculate a patient reference coordinate system.

[0039] Registration is typically performed to align the positions of the 3D model and 2D images from the planning phase with the airway of the patient “P” as observed through the bronchoscope 30, and to allow for accurate identification of the sensor 44’s position during the navigation phase, even in portions of the airway inaccessible to the bronchoscope 30. This disclosure also anticipates other suitable registration techniques and their implementation in luminal navigation.

[0040] Registration of the patient's "P" position on the transmitter pad 56 is performed by moving the positionable guide 32 through the patient's "P" airway. More specifically, as the positionable guide 32 moves through the airway, data relating to the position of the sensor 44 is recorded using the transmitter pad 56, reference sensor 54, and tracking module 52. The shape generated from this position data is compared with the internal geometry of the passage in a three-dimensional (3D) model generated during the planning phase, and the positional correlation between the shape and the 3D model based on the comparison is determined, for example, using software on computing device 125. Other registration methods are contemplated in this disclosure, including, for example, fluoroscopic registration with a 3D model, shape matching, and other suitable techniques for registering surgical images of anatomical features to preoperative images of those same anatomical features. In various aspects, these other registration methods may or may not utilize sensor 44 for registration.

[0041] Additionally, the software identifies non-tissue spaces (e.g., air cavities) within the 3D model. The software aligns or registers images of the sensor 44's location with the 3D model and 2D images generated from it, based on the recorded location data and the assumption that the locatable guide 32 remains within the non-tissue spaces of the patient "P's" airway. Alternatively, manual registration can be employed by navigating the bronchoscope 30 with sensor 44 to a pre-designated location in the patient "P's" lung and manually associating images from the bronchoscope with model data from the 3D model.

[0042] After registering the patient "P" to the image data and route planning, the user interface is displayed in the navigation software of System 100, which sets the path that clinicians should follow to reach the target. One such navigation software is currently sold by Medtronic Ltd. Navigation kit.

[0043] Once catheter 12 has successfully navigated to a position close to its target (as shown on the user interface), the locatable guide 32 can be unlocked and removed from catheter 12, leaving catheter 12 in place as a guide channel for guiding medical devices. Such medical devices may include, but are not limited to, optical systems, ultrasound probes, marker placement tools, biopsy tools, ablation tools (i.e., microwave ablation devices), laser probes, cryoprobes, sensor probes, and aspiration needles.

[0044] A three-dimensional model of the patient's lungs generated from a previously acquired CT scan may not provide a sufficient basis for accurately guiding the catheter 12 of the catheter guidance assembly 40 to the target during surgery. As described above, inaccuracies may be due to CT deviations relative to the body (deformations of the patient's lungs during surgery relative to the lungs at the time of previously acquired CT data). Therefore, an alternative imaging modality is needed to visualize the target and / or terminal bronchial branches and to enhance electromagnetic navigation surgery by correcting navigation during surgery, achieving target visualization, and confirming the placement of medical or surgical devices during surgery. To this end, the system described herein processes and converts image data captured by the fluorescence fluoroscopy imaging device 110 into a 3D reconstruction of the target area as described herein. This fluorescence fluoroscopy image data can be used to identify such targets and terminal bronchial branches, or incorporated and used to update data from CT scans in an effort to provide a more accurate navigation procedure. Furthermore, fluorescence fluoroscopy images can be captured after navigation and therefore include the visual effects of the catheter 12 and any medical devices passing through it relative to the target positioning.

[0045] Now for reference Figure 1B It is configured to be compatible with Figures 3 to 6 A schematic diagram of system 150 used in conjunction with the methods described above. System 150 may include... Figure 1AWorkstation 125 and Figure 1A The fluorescent imaging device or fluoroscope 110 is described. In some aspects, workstation 125 may be connected directly or indirectly to fluoroscope 110, for example, via wireless communication. Workstation 125 may include memory 102 (e.g., storage device), processor 104, display 106, and input device 10. Processor 104 may include one or more hardware processors. Workstation 125 may optionally include output module 112 and network interface 108.

[0046] Memory 102 may store application 81 and image data 114, including fluorescence fluoroscopic imaging data. Application 81 may include instructions executable by processor 104 to perform actions as described herein. Figures 3 to 6 The methods of this disclosure (and other functions) include the methods described herein. Application 81 may further include a user interface 116. Image data 114 may include 3D imaging data, such as preoperative CT scans, fluoroscopic three-dimensional reconstruction (F3DR) of the target area, and / or any other fluoroscopic image data and / or one or more virtual fluoroscopic images. Processor 104 may be coupled to memory 102, display 106, input device 10, output module 112, network interface 108, and fluoroscopic imaging device 110. Workstation 125 may be a fixed computing device (such as a personal computer) or a portable computing device (such as a tablet computer). Workstation 125 may embed multiple computing devices.

[0047] Memory 102 may include any non-transitory computer-readable storage medium for storing data and / or software, including instructions executable by processor 204 that control the operation of workstation 80 and, in some respects, the operation of fluoroscope 110. Fluoroscope 110 is used to capture a sequence of fluorescence fluoroscopic images based on which it generates F3DR. The two-dimensional fluorescence fluoroscopic images selected from the medical device may be selected from the captured sequence of fluorescence fluoroscopic images. In one aspect, the storage device or memory 102 may include one or more storage devices, such as solid-state storage devices (e.g., flash memory chips). As an alternative to or complement to one or more solid-state storage devices, memory 102 may include one or more mass storage devices connected to processor 104 via a mass storage controller (not shown) and a communication bus (not shown).

[0048] Although the description of computer-readable media herein refers to solid-state memory, those skilled in the art will understand that computer-readable storage media can be any available medium accessible by processor 104. That is, computer-readable storage media can include non-transitory, volatile and non-volatile, removable and non-removable media implemented in any way or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, computer-readable storage media can include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technologies, CD-ROM, DVD, Blu-ray disc or other optical storage devices, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store the required information and is accessible by workstation 125.

[0049] When processor 104 executes application 81, display 106 may display user interface 116. User interface 116 may be configured to present to the user F3DR, two-dimensional fluorescence perspective images, 3D imaging images, and virtual fluorescence perspective views. According to this disclosure, user interface 116 may be further configured to guide the user in selecting targets by (in particular) identifying and marking targets in the displayed F3DR or any other fluorescence perspective image data.

[0050] Network interface 108 can be configured to connect to a network, such as a local area network (LAN), wide area network (WAN), wireless mobile network, Bluetooth network, and / or the Internet, consisting of wired and / or wireless networks. Network interface 108 can be used to establish a connection between workstation 125 and fluoroscope 110. Network interface 108 can also be used to receive image data 114. Input device 10 can be any device that a user can use to interact with workstation 125, such as a mouse, keyboard, foot pedal, touchscreen, and / or voice interface. Output module 112 can include any connection port or bus, such as a parallel port, serial port, universal serial bus (USB), or any other similar connection port known to those skilled in the art.

[0051] Reference Figure 2A and 2B The diagram illustrates flexible catheters 200 and 210 configured for use with the systems and methods of this disclosure. The flexible catheter 200 includes a plurality of radiopaque markers 202 that can be captured in fluorescence imaging and used to reconstruct the 3D shape of the catheter 200, which can then be used to estimate... Figure 1A and 1BThe orientation of the fluorescence fluoroscopic imaging device 110. In other aspects, the flexible catheter 210 includes multiple radiopaque coils 212, which can be used to determine the position of the radiopaque coils in 3D space. This can be used to reconstruct the 3D shape of the catheter 200, and further to estimate... Figure 1A and 1B The orientation of the fluorescence fluoroscopic imaging device 110. In all aspects, although... Figure 2A and 2B The flexible catheters 200 and 210 shown include five radiopaque markers 202 and coils 212, respectively. However, the flexible catheters 200 and 210 may include any number of radiopaque markers 202 and coils 212 (e.g., three, four, or six radiopaque markers 202 or coils 212) suitable for reconstructing the 3D shape of the catheters 200 and 210.

[0052] refer to Figure 3 A flowchart of method 300 is provided for estimating the pose of a fluoroscopic imaging device while capturing fluoroscopic images of a patient's body. In various aspects, as described herein, the estimated pose can be used to reconstruct 3D volumetric data of a target region of the patient's body. At box 302, a fluoroscopic imaging device is used to capture fluoroscopic images of the patient's body by sweeping. In some aspects, the sweep includes a view of the fluoroscopic imaging device around the longitudinal axis of the patient's body relative to an anteroposterior position of greater than 30 degrees. At box 304, radiopaque markers positioned along the length of a catheter in the fluoroscopic image are identified and tracked. In various aspects, radiopaque markers can be identified and tracked when a fluoroscopic sweep is performed at box 302. In some aspects, such as Figure 2A As shown, the radiopaque markers include those placed along the length of the conduit.

[0053] Radiopaque markers can take any form suitable for identification and tracking in fluoroscopic images. For example, radiopaque markers can be rings, spirals, squares, dots, or other suitable symbols or shapes placed around the catheter. For instance, rings can be equidistant from each other at a predetermined distance. This predetermined distance can be a suitable distance for accurately determining the 3D position of the catheter. Alternatively, the rings may not be equidistant from each other. In some aspects, radiopaque markers can be placed at different axial locations around the catheter to facilitate identification of the catheter's shape. In other aspects, such as... Figure 2B As shown, the radiopaque markers include coils positioned along the length of the catheter. In other aspects, the catheter may be at least partially made of or coated with a radiopaque material, and image processing techniques may be used to track points on the radiopaque catheter.

[0054] In other aspects, the catheter may include both radiopaque markers or materials and tracking or electromagnetic (EM) sensors positioned along a portion of the catheter's length. In various aspects, the tracking module 52 of the tracking system 50 may be electrically or wirelessly coupled to and communicate with coils and / or EM sensors positioned along the catheter's length. In this configuration, the tracking module 52 may activate the transmitter pad 56 and collect data from the coils or EM sensors, which can be used to determine the 3D position of the tracking sensor, EM sensor, or coil positioned along the catheter's length. The 3D position information can then be used to determine the 3D shape of the catheter.

[0055] At box 306, the three-dimensional (3D) coordinates of the catheter are determined based on radiopaque markers tracked in the two-dimensional fluorescence fluoroscopy image. Then, at box 308, an estimation algorithm or process is used to estimate the orientation or angle of the fluorescence fluoroscopy imaging device relative to the catheter based on the 3D coordinates of the catheter. The estimation process may include solving a system of linear equations that describe the relationship between the 3D and 2D coordinates in the fluorescence fluoroscopy image.

[0056] refer to Figure 4 A flowchart of another method 400 for estimating the orientation of a fluoroscopic imaging device is provided. At box 402, a sweep is performed to capture a fluoroscopic image of the patient's body. In some aspects, the sweep includes a view of the fluoroscopic imaging device around the longitudinal axis of the patient's body at an angle greater than 30 degrees relative to the anteroposterior position.

[0057] At box 404, points or features along the length of the duct in the fluorescence fluoroscopic image are identified and tracked by applying appropriate image processing methods to the fluorescence fluoroscopic image. In some respects, points are radiopaque markers placed along the length of the duct.

[0058] At box 406, a motion-based structure-finding method is used to estimate the 3D structure of the catheter and the orientation of the fluoroscopic imaging device by motion-based imaging of the captured fluorescence fluoroscopy video images (i.e., motion of the fluoroscopic imaging device around the structure). The motion-based structure-finding method estimates the 3D structure of the catheter from a sequence of two-dimensional fluorescence fluoroscopy video images. This method may include processing multiple frames of fluorescence fluoroscopy video images captured at different angles, allowing multiple points on the catheter to be visualized in the fluorescence fluoroscopy video images. For example, the method may operate on at least four frames of the fluorescence fluoroscopy video images at a time. Furthermore, at least four points along the catheter are captured in the fluorescence fluoroscopy video images.

[0059] In all aspects, the motion-based structure-of-motion (MPS) method can be applied to any number of fluorescence fluoroscopy images and any number of points along the length of the catheter suitable for accurately estimating the 3D structure of the catheter. At least four points are sufficient for the MPS method to determine the 3D location of the points and the angle of the fluorescence fluoroscopy imaging device at each viewing point in at least four fluorescence fluoroscopy video frames. Interpolation algorithms can be applied to the 3D location of the points to obtain the 3D shape of the points, which can then be used to determine the 3D shape of the catheter including the points. In all aspects, the medical device or the catheter including these points remains stationary while the fluorescence fluoroscopy video frames are captured.

[0060] Motion-based structure acquisition methods may include detecting or identifying identical points on all fluoroscopic images and distinguishing those points from other points on all fluoroscopic images. Detecting or identifying identical points on all fluoroscopic images may include detecting or identifying a point at the catheter tip and then detecting or identifying other points by searching for points sequentially starting from the point at the catheter tip or by counting the points. Alternatively, detecting or identifying identical points on all fluoroscopic images may involve tracing the points from one fluoroscopic image to the next.

[0061] In some aspects, motion-based structure-finding methods can utilize motion signals provided by tracking sensors or coils (e.g., transmissive coils) mounted on the catheter. These methods involve finding a correspondence between fluorescence fluoroscopy images of the catheter and a 3D structural reconstruction of the catheter. To find this correspondence, features such as corner points (edges with gradients in multiple directions) can be tracked from one fluorescence fluoroscopy image to the next in a captured sequence of fluorescence fluoroscopy images.

[0062] In various aspects, motion-based structure-finding methods can use any suitable feature detector. For example, the scale-invariant feature transform (SIFT) can be used as a feature detector. SIFT uses the maximum value in the difference-of-Gaussians (DOG) pyramid as a feature. The first step of SIFT is to find the dominant gradient direction. Alternatively, the speeded-up robust feature (SURF) can be used as a feature detector. In SURF, DOG is replaced with a Hessian matrix-based blob detector. Furthermore, SURF does not evaluate the gradient histogram, but instead computes the sum of the gradient components and the sum of their absolute values.

[0063] The features detected from all fluorescence fluoroscopic images are then matched. The matching algorithm that tracks features from one fluorescence fluoroscopic image to another is the Lukas-Kanade tracker. In some respects, matched features can be filtered to remove incorrectly matched features. The random sample consensus (RANSAC) algorithm can be used to remove incorrectly matched features.

[0064] Its 3D position and the motion or orientation of the fluorescence imaging device are then reconstructed using the time-varying feature trajectory. Alternatively, a motion-based structure-finding approach can use a direct method where geometric information (e.g., the 3D structure of the catheter and the orientation of the fluorescence imaging device) is estimated directly from the fluorescence imaging image without the need for intermediate feature or corner extraction.

[0065] The motion-based structure-finding method can be performed incrementally or globally. According to the incremental structure-finding method, the attitude of the fluorescence fluoroscopy imaging device is solved, and each part is added to the set. In the global motion-based structure-finding method, the attitude of the fluorescence fluoroscopy imaging device is solved simultaneously. Alternatively, the motion-based structure-finding method can use an intermediate approach, where several partial reconstructions are calculated and integrated into the global solution.

[0066] refer to Figure 5A A flowchart is provided for another method 500a for estimating the orientation or angle of a fluoroscopic imaging device while capturing a fluoroscopic image of a patient's body. At block 501, a wide fluoroscopic scan is performed to capture a fluoroscopic image of the patient's body. At block 502, a motion-based structure-finding process is performed on the first fluoroscopic image to estimate a first 3D shape of the catheter. The motion-based structure-finding process may include alternating steps of: (1) estimating the position of the fluoroscopic imaging device that minimizes projection error given a set of corresponding 2D and 3D coordinates of the catheter, and (2) estimating (a) the 3D coordinates of the catheter and (b) the orientation of the fluoroscopic imaging device given corresponding 2D coordinates of the catheter in the captured fluoroscopic image.

[0067] At frames 503 to 505, a second 3D shape of the catheter is determined. Specifically, at frame 503, the lung airway in which the catheter is placed is identified. The lung airway in which the catheter is placed can be identified by determining the position of the catheter relative to a preoperative computed tomography (CT) image of the lungs (e.g., by using an electromagnetic positioning system and a tracking sensor or coil placed on the catheter). In other aspects, method 500a can be applied to said other body structures that are suitable for or capable of defining the shape of the catheter placed in other body structures. For example, the body structure may be a passageway within a lumen of a patient's body.

[0068] At box 504, the 3D shape of the lung airway is determined, for example, based on a preoperative CT image of the lungs. At box 505, a second 3D shape of the catheter is estimated based on the 3D shape of the body structure. At box 506, the 3D shape of the catheter is estimated based on the first and second estimated 3D shapes of the catheter. This estimation process may include correcting the first estimated 3D shape of the catheter using the second estimated 3D shape. Subsequently, at box 507, for each captured fluoroscopic image, the angle at which the estimated 3D shape of the catheter is projected onto the 2D catheter in each fluoroscopic image is determined. In various aspects, the determined angle may be used to reconstruct 3D volume data of a target region within the patient's body. Alternatively, the 3D shape of the catheter may be determined by performing a 3D shape sensing method. For example, the catheter may include at least one fiber optic sensor, such as one or more rotating multi-core optical fibers, disposed along the length of the catheter, and the 3D shape of the catheter may be determined by a 3D shape sensing method based on fiber optic sensor signals obtained from one or more fiber optic sensors.

[0069] refer to Figure 5B A flowchart 500b is provided for another method 500b for estimating the pose or angle of a fluoroscopic imaging device while capturing fluoroscopic images of a patient's body. In boxes 512-514, the three-dimensional (3D) shape of the catheter is determined. Specifically, in box 512, a wide scan of the fluoroscopic imaging device is performed to capture a first fluoroscopic image of the catheter and a marker (e.g., a sphere or other suitable marker) grid, as well as a second fluoroscopic image of the catheter. Next, in box 513, the pose of the fluoroscopic imaging device for each image of the first fluoroscopic image of the catheter is estimated based on the projection of the marker grid onto each image of the first fluoroscopic image of the catheter. Subsequently, in box 514, the 3D shape of the catheter is reconstructed based on the estimated pose of the fluoroscopic imaging device and the first fluoroscopic image. The first fluoroscopic image may correspond to the viewpoint of the fluoroscopic imaging device around the longitudinal axis of the patient, with an anteroposterior position of less than 30 degrees.

[0070] In various aspects, the 3D shape of the catheter can be determined at box 514 based on the pose or angle of the fluorescence fluoroscopic imaging device estimated according to the following method. First, a probability map is generated for each fluorescence fluoroscopic image of the first fluorescence image, the probability map indicating the probability that each pixel of each fluorescence fluoroscopic image of the first fluorescence image belongs to the projection of a structural marker or marker grid. Next, different candidates for projecting the marker structure on each fluorescence fluoroscopic image of the first fluorescence image are generated by virtually positioning the fluorescence fluoroscopic imaging device in possible different poses. Based on the image probability map, the candidate with the highest probability of becoming the projection of the marker structure on each fluorescence fluoroscopic image of the first fluorescence image is identified. Subsequently, the pose or angle of the fluorescence fluoroscopic imaging device is estimated based on the virtual pose of the fluorescence fluoroscopic imaging device used to generate the identified candidates.

[0071] At box 516, for each of the second fluorescence fluoroscopic images, the angle at which the 3D shape of the catheter is projected onto the catheter shown in the second fluorescence fluoroscopic image is determined. This angle can then be used to reconstruct 3D volumetric data of the target region including the catheter.

[0072] refer to Figure 6 The diagram illustrates a flowchart of a method 600 for constructing three-dimensional volumetric data of a target region within a patient's body based on fluorescence fluoroscopy. At block 602, a sequence of fluorescence fluoroscopy scans is performed to acquire fluorescence fluoroscopic images of the target region and radiopaque markers placed on a medical device via a fluorescence fluoroscopy imaging apparatus. The target region may include at least a portion of the lungs, and the medical device may navigate to the target region via an airway network.

[0073] At box 604, a suitable image processing algorithm is used to identify and track radiopaque markers along the length of a medical device (e.g., a catheter) in the fluoroscopic image sequence. At box 606, the three-dimensional (3D) coordinates of the medical device are determined based on the tracked radiopaque markers. At box 608, the pose or angle of the fluoroscopic imaging device is estimated based on the 3D coordinates of the medical device. At box 610, fluoroscopic-based 3D volumetric data of the target region is constructed based on the estimated pose or angle of the fluoroscopic imaging device.

[0074] In various aspects, method 600 further includes determining an offset between the medical device and the target area based on fluorescence-based three-dimensional volume data at box 612. In other aspects, method 600 includes facilitating navigation of the medical device to the target area using an electromagnetic positioning system in the display indicating the position of the medical device at box 614, and correcting the display of the position of the medical device relative to the target at box 616 based on the determined offset between the medical device and the target area.

[0075] In other aspects, the method may include displaying a 3D rendering of the target area on a display and registering the electromagnetic positioning system to the 3D rendering. Correcting the position of the medical device relative to the target may include updating the registration of the electromagnetic positioning system to the 3D rendering.

[0076] In various aspects, the method may also include generating a 3D rendering of the target region based on previously acquired CT volume data of the target region. The target region may include at least a portion of the lung, a soft tissue target region, or an ablation target.

[0077] From the foregoing and with reference to the various figures, those skilled in the art will understand that modifications can be made to this disclosure without departing from its scope. For example, although the systems and methods are described as being usable with EMN systems for navigation through luminal networks such as the lungs, the systems and methods described herein can be used with systems utilizing other navigation and treatment devices, such as percutaneous devices. Furthermore, although the aforementioned systems and methods are described for use within a patient's luminal network, it should be understood that the systems and methods can be used in other target areas such as the liver. In addition, the aforementioned systems and methods can also be used for transthoracic needle aspiration procedures.

[0078] Detailed aspects are disclosed herein. However, the disclosed aspects are merely examples of this disclosure, which may be embodied in various forms and aspects. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as the basis of the claims and as a representative basis for teaching those skilled in the art to use this disclosure in different ways with virtually any suitable detailed structure.

[0079] As will be understood, medical devices (such as biopsy tools) or energy devices (such as microwave ablation catheters) that can be located through a patient's network of luminal branches to treat tissue have proven useful in the surgical field, and this disclosure relates to systems and methods that can be used with such devices and tools. Access to the network of luminal branches can be made percutaneously or using navigation techniques through natural openings. Alternatively, image-guided systems can be used to achieve navigation through the network of luminal branches. Image-guided systems can be separate from or integrated with energy devices or separate access tools, and can include MRI, CT, fluoroscopy, ultrasound, electrical impedance tomography, optical and / or device tracking systems.

[0080] Methods for locating access tools include EM, IR, echolocation, and optics. Tracking systems can be integrated into imaging equipment, where tracking is performed in virtual space or fused with preoperative or real-time images. In some cases, treatment targets can be accessed directly from within the lumen, such as the bronchial wall for treating COPD, asthma, lung cancer, etc. In other cases, energy devices and / or other access tools may be needed to puncture the lumen and extend into other tissues to reach the target, such as for treating intramural diseases. The final location and confirmation of energy device or tool placement can be achieved using standard fluorescence fluoroscopy imaging equipment incorporating the above methods and systems, guided by imaging and / or navigation.

[0081] While several aspects of this disclosure have been shown in the accompanying drawings, it is not intended to limit the disclosure thereto, but rather to broaden its scope as is permitted in the art, and to convey the same meaning to the present specification. Therefore, the foregoing description should not be construed as restrictive, but merely as illustrative of specific aspects. Other modifications will be contemplated by those skilled in the art within the scope and spirit of the appended claims.

Claims

1. A system for estimating the attitude of a fluorescence fluoroscopic imaging device, the system comprising: One or more processors; as well as At least one memory, coupled to the one or more processors, stores instructions that, when executed by the one or more processors, cause the system to perform the following operations: Receive multiple fluorescent images of the catheter captured by scanning with the fluorescent imaging device; Identify and track multiple radiopaque markers along the length of the catheter in the multiple fluorescent transillumination images of the catheter; The 3D coordinates of the catheter are determined based on a structural method that uses 3D motion to obtain the structure of multiple tracked radiopaque markers. Determine the 3D shape of the body structure containing the catheter; as well as The orientation of the fluorescence imaging device is estimated based on the 3D coordinates of the catheter and the 3D shape of the body structure.

2. The system of claim 1, wherein the plurality of radiopaque markers comprise a plurality of corresponding tracking sensors.

3. The system according to claim 2, wherein the plurality of tracking sensors are a plurality of coils.

4. The system of claim 1, wherein the plurality of fluorescence fluoroscopic images of the catheter are captured by scanning with the fluorescence fluoroscopic imaging device, and the scanning is a wide scan that includes a viewing angle of the fluorescence fluoroscopic imaging device around the longitudinal axis relative to an anteroposterior position of greater than 30 degrees.

5. A system for estimating the attitude of a fluorescence fluoroscopic imaging device, the system comprising: One or more processors; as well as At least one memory, coupled to the one or more processors, stores instructions that, when executed by the one or more processors, cause the system to perform the following operations: Receive multiple fluorescent images of the catheter captured by scanning with the fluorescent imaging device; Identify and track multiple radiopaque markers along the length of the catheter in the multiple fluorescent fluoroscopic images of the catheter traveling through the patient's body; as well as A 3D motion-based structural method is used to estimate the first 3D shape of the catheter by performing 3D motion on multiple tracked radiopaque markers. Determine the 3D shape of the body structure containing the catheter; Estimate the second 3D shape of the catheter based on the 3D shape of the body structure in which the catheter is placed; The third 3D shape of the catheter is estimated based on the first and second 3D shapes; as well as The orientation of the fluorescence fluoroscopic imaging device is estimated based on the third 3D shape of the conduit.

6. The system of claim 5, wherein when the instructions are executed by the one or more processors, the system further causes the system to construct 3D volumetric data of the target region within the patient's body based on the estimated pose of the fluorescence fluoroscopic imaging device.

7. The system of claim 5, wherein the plurality of fluorescence fluoroscopic images of the catheter are captured by scanning with the fluorescence fluoroscopic imaging device, and the scanning is a wide scan that includes the field of view of the fluorescence fluoroscopic imaging device around the longitudinal axis of the patient's body at an anteroposterior position greater than 50 degrees.

8. The system of claim 5, wherein the 3D shape of the body structure with the catheter disposed therein is determined based on computed tomography (CT) images of the body structure with the catheter disposed therein.

9. The system of claim 5, wherein the body structure is the airway of the lungs.

10. A system for constructing fluorescence-based three-dimensional volumetric data of a target region within a patient's body, the system comprising: One or more processors; as well as At least one memory, coupled to the one or more processors, stores instructions that, when executed by the one or more processors, cause the system to perform the following operations: A sequence of fluorescent transillumination images of the target region and multiple radiopaque markers of the medical device, captured by scanning with a fluorescent transillumination imaging device; Identify and track the plurality of radiopaque markers along the length of the medical device in the sequence of fluorescent transillumination images of the medical device traveling through the patient's body; The 3D coordinates of the medical device are determined based on a 3D motion-based structural method for tracking multiple radiopaque markers. Determine the 3D shape of the body structure containing the catheter; The angle of the fluorescence imaging device is estimated based on the 3D coordinates of the medical device and the 3D shape of the body structure. as well as Based on the estimated angle of the fluorescence fluoroscopic imaging device, 3D volumetric data of the target region based on fluorescence fluoroscopic imaging is constructed.

11. The system of claim 10, wherein when the instructions are executed by the one or more processors, the system further causes the system to determine the offset between the medical device and the target region based on the fluorescence-based three-dimensional volume data.

12. The system of claim 11, wherein when the instruction is executed by the one or more processors, the system further causes... A positioning system that indicates the location of the medical device in the display is used to facilitate navigation of the medical device to the target area; and The display of the position of the medical device relative to the target area is corrected based on the determined offset between the medical device and the target area.

13. The system of claim 12, wherein the positioning system is an electromagnetic positioning system.

14. The system of claim 12, wherein when the instructions are executed by the one or more processors, the system further causes the system to: display a 3D rendering of the target region on the display; and The positioning system is registered to the 3D rendering. Correcting the position of the medical device relative to the target area includes updating the registration of the positioning system to the 3D rendering.

15. The system of claim 10, wherein when the instructions are executed by the one or more processors, the system further causes the system to generate a 3D rendering of the target region based on previously acquired CT volume data of the target region.

16. The system of claim 10, wherein the target region comprises at least a portion of the lungs, and wherein the medical device navigates to the target region via an airway lumen network.

17. The system of claim 10, wherein the target region comprises at least a portion of the lung.

18. The system of claim 10, wherein the target region includes a soft tissue target.

19. The system of claim 10, wherein the target region includes the target to be ablated.

Images