Navigating bronchial pathways

Abstract

The invention relates to navigation in bronchial pathways. To provide further improved navigation guidance, a sequence of 2D X-ray images of a region of interest of a bronchial structure with an intrathoracic device (visible in the X-ray images) inserted in the bronchial pathway is provided. The intrathoracic device is tracked in the 2D X-ray images; and directions and amplitudes of repetitive cardiovascular and respiratory induced motion are evaluated based on the tracked intrathoracic device. The evaluated motion is modeled and navigation information is generated indicating a range of the modeled motion. The navigation information is shown as a confidence reference (310) (e.g. rEBUS historical trajectory confidence reference) to a user operating the intrathoracic device. As an example, an enhanced fluoroscopic 2D image (302) of a thoracic region is registered and shown overlaid with segmentation results (304) of bronchial structures and a target lesion (312). In addition, a confidence reference (314) of a current position of the rEBUS catheter can also be shown.

Description

TECHNICAL FIELD

[0001] The present invention relates to endobronchial and transbronchial navigation of devices. The invention particularly relates to a device for facilitating navigation of an intrathoracic device inside a bronchial pathway, a system for navigation of an intrathoracic device inside a bronchial pathway, and a method for navigation of an intrathoracic device inside a bronchial pathway. BACKGROUND

[0002] Pulmonary peripheral lesions (PPLs) can be diagnosed by a non-surgical biopsy. In an example workflow, a pulmonary peripheral nodule is located and identified as requiring further diagnosis by a preoperative CT scan. An example of an endobronchial method of performing a non-surgical biopsy is minimally invasive. An example of an endobronchial method is a fluoroscopy guided intrathoracic biopsy. Conventional fluoroscopy (i.e. X-ray based) is used to help and guide transbronchial navigation and biopsy in pulmonary peripheral lesion diagnosis. The endoscopist is then guided by 2D fluoroscopy to roughly approximate where the diagnostic tool and the lesion to be diagnosed should be located and how they are positioned relative to each other. In another method, the endoscopist can be guided with an endobronchial ultrasound radial mini-probe (rEBUS). The rEBUS can be used alone or in combination with fluoroscopy. It is not necessarily a navigation method, but is able to provide valuable confirmatory information about whether the lesion has been reached. Either method helps to further guide an accurate endobronchial biopsy. Combining these methods can provide additional information, as 2D fluoroscopy is able to help guide rough navigation and positioning, while rEBUS imaging is able to provide local confirmatory imaging. WO2018215832 A2 relates to a method of 3D REM image construction and improved target localization using a radial endobronchial ultrasound probe. However, it has been shown that navigation can still become cumbersome for the operator. SUMMARY

[0003] Therefore, there is a need for further improved navigation guidance.

[0004] The objects of the present invention are solved by the subject matter of the independent claims; further embodiments are contained in the dependent claims. It should be noted that the following description aspects of the present invention apply to the device for facilitating navigation of an intrathoracic device inside a bronchial pathway, the system for navigation of an intrathoracic device inside a bronchial pathway, and the method for navigation of an intrathoracic device inside a bronchial pathway.

[0005] According to the invention, a device for facilitating navigation of an intrathoracic device inside a bronchial pathway is provided. The device comprises an image data providing section, a processor and an output data providing section. The image data providing section is configured to receive a sequence of 2D X-ray images of a region of interest of a bronchial structure with a device inserted in the bronchial pathway. The intrathoracic device is visible in the X-ray images. The processor is configured to track the intrathoracic device in at least part of the sequence of 2D X-ray images. The processor is further configured to assess directions and amplitudes of repetitive cardiovascular and respiratory induced motion based on the tracked intrathoracic device. The processor is further configured to model the assessed motion and generate navigation information indicative of a range of the modeled motion. The output data providing section is configured to provide the navigation information as a confidence reference to a user operating the intrathoracic device.

[0006] The term "bronchial pathway" relates to e.g. the lungs, but also to other parts of the pathway for breathing in and out.

[0007] The term "intrathoracic" relates to within the thorax, and relates to e.g. intrapulmonary, intrathoracic and transthoracic. Also, the term includes transbronchial. Hence, an intrathoracic device relates to an endobronchial or transbronchial device.

[0008] In an example, the processor is further configured to track a region of interest of an intrathoracic region in at least part of the sequence of 2D X-ray images.

[0009] In an example, the processor is configured to enable differentiation between repetitive motion and single event motion (e.g. single event motion caused by manipulation of the endobronchial or transbronchial device or any guiding element thereof).

[0010] In another example, the processor is further configured to enable modeling of the discovered device positioning relative to a pre-procedural imaging source.

[0011] In an example, for the motion modeling, in addition to following the device, information of the actual appearance of the bronchial tree is provided. The confidence interval positioning is related to pre-procedural knowledge of the bronchial tree positioning.

[0012] In an option, an endobronchial device is tracked, then positioned at the carina, and image data is generated. In addition, the endobronchial device is positioned under part or all of the carina of each lung segment (e.g. right upper lobe, right middle lobe, right lower lobe, left upper lobe, left lower lobe), and image data is generated. In addition, the endobronchial device is provided to register (e.g. automatically register) a pre-procedural anatomical image to a fluoroscopy image when it is known how a pre-procedural segmentation result of the lungs will fit in the fluoroscopy image.

[0013] Fluoroscopy imaging is a type of X-ray imaging, it is also known as fluoroscopy or fluorography.

[0014] In further examples, anatomical features such as the lungs are tracked based on features to improve the accuracy of the model.

[0015] This provides the user with essential information when navigating, thus meaning a mitigation and improvement of the navigation problem for bronchial and lung structures. The indication of the possible space in which the device, like the imaging probe or the extractor, is located provides guidance to the operator in an intuitive way.

[0016] In examples, coarse features in the region of interest in the thorax are considered in order to be able to give an error or adjust the confidence interval in case a dislocation of the coarser lung features is detected. This is the case for example when rotating the catheter with a predefined (passive) angle. With the rotation of the catheter, a movement of the lung segmentation segments or lesions can be seen.

[0017] In examples, the image data providing section is configured to receive pre-procedural imaging information of the patient, i.e. PET-CT, CT (or variants thereof, i.e. cone beam CT) and / or MRI.

[0018] According to examples, the confidence reference is provided as an indication of an envelope enclosing a spatial range of the modeled motion.

[0019] According to examples, the processor is configured to track the intrathoracic device in at least a portion of the sequence of 2D X-ray images; and to assess the direction and amplitude of non-repetitive lung tissue and device motion caused by intrathoracic device manipulation or its guiding element (i.e. endoscope); and to track the motion and generate navigation information adjusted for non-repetitive motion caused by intrabronchial or transbronchial device manipulation or its guiding element.

[0020] According to examples, the image data providing section is further configured to provide preoperative 3D image data of segmented bronchial structures of the subject comprised in the region of interest. The processor is configured to register the 2D X-ray images with the preoperative 3D image data. The processor is further configured to determine a spatial position of the intrathoracic device with respect to the segmented bronchial structures of the preoperative 3D image data based on 2D X-ray projections of the intrathoracic device. The output data providing section comprises a display configured to display the confidence reference in a projection of the segmented bronchial structures, both the confidence reference and the projection of the segmented bronchial structures being superimposed to a current 2D X-ray image of the sequence of 2D X-ray images.

[0021] According to an example, the image data providing section is further configured to provide 3D (reconstructed) image data, which is derived from an intraoperative X-ray fluoroscopy, e.g. cone beam computed tomography, and which further comprises a segmented bronchial structure of the object in the region of interest. The processor is configured to register the 2D X-ray image with the 3D (reconstructed) image data. The processor is further configured to determine a spatial position of the transthoracic device with respect to the segmented bronchial structure of the 3D image data based on a 2D X-ray projection of the transthoracic device. The output data providing section comprises a display configured to display the confidence reference overlaid in 3D on the display of the 3D image data.

[0022] In an example, the image data is derived from a sweep of a C-arm acquiring at least multiple projections at multiple different angles, possibly further providing a segmented bronchial structure of an object, e.g. a patient, or a 3D tool position in a region of interest.

[0023] According to an example, the image data providing section is configured to receive a live stream of 2D X-ray images as the sequence. The processor is configured to continuously track the intrathoracic device in the images. The processor is further configured to update the navigation information accordingly. The output data providing section is configured to continuously provide the navigation information as updated confidence reference to a user operating the intrathoracic device.

[0024] According to an example, the X-ray information is also updated with and / or integrated with electromagnetic tracking information.

[0025] In an example, offline operation from X-ray is provided by using electromagnetic navigation. In another example, passive impedance based localization of the intrathoracic device is used.

[0026] According to the invention, there is also provided a system for navigation of an intrathoracic device inside a bronchial pathway. The system comprises an X-ray imaging device for imaging a thoracic region of interest of a subject, e.g. the lungs of a subject, and an intrathoracic device for insertion in a bronchial pathway. The system further comprises a display and a device for facilitating navigation according to one of the preceding examples. The intrathoracic device is at least partially radiopaque. The X-ray imaging device provides a sequence of 2D X-ray images of the region of interest of the bronchial structure with the intrathoracic device inserted in the bronchial pathway. The output data providing section provides the navigation information, i.e. the visual indication, to the display. In an example, the display is a monitor or display used by an operator. The display provides at least one of: the sequence of 2D X-ray images, and the visual indication as a confidence reference overlaid to the 2D X-ray images.

[0027] According to an example, the intrathoracic device is an ultrasound imaging probe configured to provide radial endobronchial ultrasound data from within the bronchial structure, e.g. the lungs.

[0028] In an example, the intrathoracic device is an ultrasound imaging probe configured to provide radial and / or forward looking endobronchial ultrasound data from within the bronchial structure.

[0029] In an example, the intrathoracic device is an endobronchial ultrasound device, also referred to as EBUS. The device can be configured for radial and / or forward looking ultrasound imaging.

[0030] In another example, the intrathoracic device is an endobronchial ultrasound mini-probe, i.e. a radial endobronchial ultrasound mini-device or a forward looking endobronchial ultrasound mini-device. The device can be configured for radial and / or forward looking ultrasound imaging.

[0031] In the following, when referring to EBUS, also rEBUS is provided, and vice versa.

[0032] According to an example, the intrathoracic device comprises an ultrasound imaging probe and a tissue or cell extraction device. In an example, the tissue or cell extraction device is a biopsy device.

[0033] Besides navigation guidance towards a peripheral lung lesion, this also provides a further improvement of a biopsy of, e.g., a peripheral lung lesion.

[0034] According to the invention, there is also provided a method for navigation of an intrathoracic device inside a bronchial pathway. The method comprises the steps of:

[0035] receiving a sequence of live 2D X-ray images of a region of interest of a bronchial structure with an intrathoracic device inserted in a bronchial path, in which the intrathoracic device is visible in the X-ray images;

[0036] tracking the intrathoracic device in at least part of the sequence of live 2D X-ray images;

[0037] evaluating the direction and amplitude of repetitive cardiovascular and breath induced motion based on the tracked intrathoracic device;

[0038] modeling the evaluated motion and generating navigation information indicative of the modeled motion; and

[0039] providing the navigation information as a confidence reference to a user operating the intrathoracic device.

[0040] According to one aspect of the present invention, under fluoroscopy in bronchoscopy navigation, the use of the radiopaque properties of potential and / or existing devices is used for automated tracking to enhance navigation to a lesion for e.g. known catheters, devices and / or radial endobronchial ultrasound devices, providing lesion access confirmation and / or facilitating guided biopsy extraction. For example, computer vision and image processing for 2D and / or 3D tracking of catheters is deployed. In addition, it is also provided to augment the fluoroscopy images and display the results of the device tracking in 2D and / or 3D. In another example, the results of the device tracking in 2D / 3D are registered with the results of pre-procedure computed tomography (CT) planning, 2D fluoroscopy at multiple angles (continuously or at distinct angles) and / or cone beam CT scanning and / or electromagnetic body navigation bronchoscopy. In an example, a sweep with a minimum width of 30° under continuous fluoroscopy is provided to have 3D updates of lesion and device positioning. In another example, a sweep with a minimum width less than 30° is provided, for example, a sweep with a width of 20°.

[0041] According to an aspect of the present invention, the present invention relates to navigation in or through a bronchial tree. For example, the navigation relates to navigation in a bronchial path. To provide further improved navigation guidance, a sequence of 2D X-ray images of a region of interest of a bronchial structure with an (X-ray image visible) intrathoracic device inserted in the bronchial path is provided. The intrathoracic device is tracked in the 2D X-ray images; and directions and amplitudes of repetitive cardiovascular and respiratory induced motion are assessed based on the tracked intrathoracic device. The assessed motion is modeled and navigation information indicative of a range of the modeled motion is generated. The navigation information of the potential device positioning is presented to an operator, e.g. a EBUS or rEBUS historical trajectory confidence reference is presented to a user operating the intrathoracic device. As an example, an X-ray 2D image of a thoracic region is registered with a static segmentation result of a bronchial structure derived according to a use of cone beam CT (i.e. in case of X-ray imaging (with a C-arm system) usually combined with a preoperative CT) and the X-ray 2D image of the thoracic region is shown overlaid with the static segmentation result of the bronchial structure. As an example, an enhanced fluoroscopy 2D image (302) of a thoracic region is registered with a segmentation result (304) of a bronchial structure and a target lesion (312) and the enhanced fluoroscopy 2D image of the thoracic region is shown overlaid with the segmentation result of the bronchial structure and the target lesion. In addition, a confidence reference (314) of a current position of a rEBUS catheter can also be shown.

[0042] These and other aspects of the present invention will become apparent and elucidated from the embodiments described hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0043] In the following, example embodiments of the present invention will be described with reference to the accompanying drawings:

[0044] Figure 1 A device for facilitating navigation of an intrathoracic device inside a bronchial path is schematically shown.

[0045] Figure 2 An example of a system for navigation of an intrathoracic device inside a bronchial path is shown.

[0046] Figure 3 Basic steps of a method for navigation of an intrathoracic device inside a bronchial path are schematically shown.

[0047] Figure 4 An example of an enhanced live fluoroscopy 2D image of a thoracic region with a confidence reference overlaid to a 2D X-ray image around a current catheter position is schematically shown. DETAILED DESCRIPTION

[0048] Certain embodiments will now be described in greater detail with reference to the drawings. Like reference numerals used in different drawings refer to similar elements. Definitions provided in the specification are to be understood as defining, in the context of the claims, the preferred terms. Moreover, the specification provides definitions of terms that are used extensively throughout the description and claims. Such definitions are provided to facilitate understanding of the exemplary embodiments. Furthermore, the terms "comprise", "comprising", "comprises", "include", "including", "includes", "contain", "containing", "contains", "characterized by" and the like are used in the detailed description section only to describe certain embodiments. Such terms are not used, in any way, to limit the scope of the claims.

[0049] The term "subject" can also refer to an individual. The subject can also refer to a patient, although it is noted that the term does not indicate whether the subject is actually suffering from any disease or condition.

[0050] Figure 1 An apparatus 10 for facilitating navigation of an intrathoracic device inside a bronchial pathway is schematically shown. The apparatus 10 for facilitating navigation comprises an image data providing section 12, a processor 14 and an output data providing section 16. The image data providing section 12 is configured to receive a sequence of 2D X-ray images of a region of interest of a bronchial structure with an intrathoracic device inserted in the bronchial pathway. The intrathoracic device is visible in the X-ray images. The processor 14 is configured to: track the intrathoracic device in at least part of the sequence of 2D X-ray images; and estimate directions and amplitudes of repetitive cardiovascular and respiratory induced motion based on the tracked intrathoracic device; and model the estimated motion and generate navigation information indicative of a range of the modeled motion. The output data providing section 16 is configured to provide the navigation information as a confidence reference to a user operating the intrathoracic device.

[0051] The "apparatus for facilitating navigation" is also referred to as "navigation apparatus". The "output data providing section" is also referred to as "navigation facilitator". The "navigation information indicative of a range of the modeled motion" is also referred to as "visual indicator of the range of the modeled motion".

[0052] In an example, the sequence of images is a sequence of live images.

[0053] The estimated motion can also be referred to as estimated range of motion.

[0054] It is noted that repetitive tissue motion, for example induced by cardiovascular and respiratory movements, can induce tissue motion of a few centimeters maximum.

[0055] In an example, the intrathoracic device is an ultrasound imaging probe providing radial endobronchial ultrasound data from within the bronchial structure.

[0056] For tracking the intrathoracic device, three radiopaque markers are provided along the intrathoracic device in a predetermined spatial arrangement. Thus, the positioning, position and orientation can be tracked based on X-ray radiation images.

[0057] By tracking the intrathoracic device, a plane can be identified to be shown on or overlaid to an X-ray image, e.g. a fluoroscopy image.

[0058] In an example, the position of the intrathoracic device is registered with the X-ray image. In an example, the tip of the intrathoracic device and the radiopaque markers are used for the registration. Thus, the intrathoracic device is registered within the anatomy of the subject.

[0059] The confidence reference replaces a dynamic model of the anatomy. The confidence reference can also be referred to as a confidence map.

[0060] In an option, which is not further detailed, the confidence reference is provided as an indication of an envelope enclosing a spatial extent of the modeled motion.

[0061] In an example, the confidence reference is provided as an indication of an envelope enclosing a spatial extent of the modeled motion related to a target, e.g. a target lesion, and / or a current position of the catheter or a trajectory of the catheter.

[0062] As an example, the confidence reference is shown overlaid to a fluoroscopy image.

[0063] In another example, the confidence reference is provided as an indication of an envelope enclosing a spatial extent of the modeled motion related to the subject.

[0064] In a further example, the confidence reference is provided as an indication of an envelope enclosing a spatial extent of the modeled motion related to a segmented target, e.g. a lesion overlaid to a fluoroscopy image.

[0065] Thus, the extent of the motion is also displayed in relation to the X-ray imaging, e.g. the fluoroscopy image. In an example, the extent of the motion is provided as a motion envelope.

[0066] For example, as an option, as indicated by the dashed line in Figure 1 The output data providing section 16 comprises a display 18 configured to display the confidence reference overlaid to a current 2D X-ray image of a sequence of 2D X-ray images, as indicated by the dashed connecting line 22 data connecting the display 18 to the output data providing section 16.

[0067] A first arrow 20 is shown in dashed line to indicate a data input to the image data providing section 12. For example, a data connection, e.g. a wireless connection or a wired connection, to an imaging system can be provided.

[0068] As an option, the framework 24 indicates an integration of the data providing section 12, the processor 14 and the output data providing section 16. For example, they are provided in a common housing. In another example, they are combined and integrated as a separate module.

[0069] In another example, they are provided independently from each other, but are data connected to each other.

[0070] The current 2D X-ray image in the sequence of 2D X-ray images can also be referred to as a live image.

[0071] In an example, the 2D X-ray images are provided from a C-arm in a single imaging position. In another example, the 2D X-ray images are provided from a C-arm in multiple imaging positions.

[0072] In an example, the image data providing section 12 is further configured to provide preoperative 3D image data comprising segmented bronchial structures of the subject in the region of interest. The processor 14 is configured to: register the 2D X-ray images with the preoperative 3D image data; and determine a spatial position of the intrathoracic device with respect to the segmented bronchial structures of the preoperative 3D image data based on the 2D X-ray projections of the intrathoracic device. The output data providing section 16 comprises a display 18 configured to display a confidence reference in a projection of the segmented bronchial structures, both the confidence reference and the projection of the segmented bronchial structures being superimposed to the current 2D X-ray image in the sequence of 2D X-ray images. In an example, a consistent position of the 2D X-ray projections is provided, or a moving sweep for 3D reconstruction is provided.

[0073] In an example, the image data providing section 12 is further configured to provide 3D image data derived from an intraoperative X-ray fluoroscopy, for example, cone beam computed tomography, and comprising segmented bronchial structures of the subject in the region of interest. The processor 14 is configured to: register the 2D X-ray images with the 3D image data; and determine a spatial position of the intrathoracic device with respect to the segmented bronchial structures of the 3D image data based on the 2D X-ray projections of the intrathoracic device. The output data providing section 16 comprises a display 18 configured to display a confidence reference superimposed in 3D on a display content of the 3D image data.

[0074] Cone beam computed tomography is also referred to as cone beam CT, CBCT or cbCT.

[0075] In an example, the preoperative 3D CT image data enhances the fluoroscopy images.

[0076] In an example, the image data providing part 12 is configured to receive a live stream of 2D X-ray images as a sequence. The processor 14 is configured to: continuously track the intrathoracic device; and update the navigation information accordingly. The output data providing part 16 is configured to continuously provide the navigation information as an updated confidence reference to a user operating the intrathoracic device.

[0077] In an example, a single consistent position is provided for the sequence, i.e. the live stream of 2D X-ray images.

[0078] In an example, a multi-X-ray localization is provided.

[0079] The term "continuous" relates to the rate of the number of tracking steps per second such that a real-time tracking is provided in terms of user experience.

[0080] The term "operator" relates to a person operating a device for facilitating navigation. For example, the operator is a technician, a physician assistant, a staff member, a nurse, a physician, etc.

[0081] Figure 2 An example of a system 100 for navigation of an intrathoracic device inside a bronchial pathway is shown. The system 100 comprises an X-ray imaging device 102 for imaging a thoracic region of interest of a subject. The system 100 further comprises an intrathoracic device 104 for insertion in a bronchial pathway of the subject. Yet further, the system 100 comprises a device for navigation 106 according to one of the preceding examples and a display 108. The intrathoracic device 104 is at least partially radiopaque. The X-ray imaging device 102 provides a sequence of 2D X-ray images of a region of interest of a bronchial structure with the intrathoracic device 104 inserted in the bronchial pathway. The device for facilitating navigation 106 provides navigation information to the display 108. The display 108 provides at least one of: the sequence of 2D X-ray images, and the navigation information as a confidence reference overlaid to the 2D X-ray images.

[0082] The term radiopaque means at least partially opaque against X-ray radiation. In short, the term refers to being visible in (X-ray) fluoroscopic images, for example.

[0083] As an example, the X-ray imaging device 102 is provided as a C-arm structure with a C-arm 110 mounted to a movable ceiling support 112. An X-ray source 114 and an X-ray detector 116 are provided at opposite ends of the C-arm 110.

[0084] A subject support 118, e.g. a patient table, is provided. A display and interface module 120 can be arranged at the side of the patient table. Further, an adjustable lighting device 122 is provided above. A device for facilitating navigation 106 is provided as a (e.g. side) control station 124. The control station 124 comprises display and interface components (e.g. a keyboard 126, a mouse 128, a control panel 130 and a display unit 132) to allow operation and control of a plurality of instruments (e.g. in an examination or intervention or operating room).

[0085] A subject 134 can be arranged on the subject support 118. An intrathoracic device 104, e.g. an ultrasound imaging probe or an extraction device, can be inserted into a bronchial pathway (not shown in detail) of the subject 134. The X-ray imaging device 102 provides image data about conditions inside the subject, i.e. inside the body of the subject, which are not visible to the operator from the outside. The image data is provided on the display 108 as e.g. a fluoroscopy image, as shown. Figure 4

[0086] In an example, the X-ray imaging device 102 is configured to provide a sequence of 2D X-ray images of a region of interest of a bronchial structure with the intrathoracic device inserted in the bronchial pathway to the device for facilitating navigation 106. The device for facilitating navigation 106 is configured to provide navigation information to the display 108. The display 108 is configured to provide at least one of: the sequence of 2D X-ray images, and the navigation information as a confidence reference overlaid to the 2D X-ray images.

[0087] For example, distinct and spaced X-ray radiation opaque markers, which can be provided on the intrathoracic device 104, e.g. the catheter of the imaging probe, can be automatically and continuously tracked under fluoroscopy by computer vision. For example, the catheter is used for the purpose of being able to see the lung under fluoroscopy by the appearance of three markers (e.g. spaced 10 mm from each other), and a distal ultrasound transducer at a fixed distance from these markers can also be X-ray radiation opaque (i.e. radiopaque).

[0088] Then, computer vision is able to recognize the RO signature of the catheter, and automatically track the movement of the catheter in 2D, and possibly in 3D, thereby also providing spatial position and orientation.

[0089] In an option, the intrathoracic device 104 is an ultrasound imaging probe, which is configured to provide radial endobronchial ultrasound examination data from within the bronchial structure.

[0090] ​Endobronchial ultrasound is also referred to as EBUS; radial endobronchial ultrasound is referred to as rEBUS, REBUS or R-EBUS as described above. In examples, a miniature ultrasound probe (e.g. an EBUS or radial EBUS probe) is inserted through a working channel of a flexible bronchoscope or catheter, which is also referred to as a guide sheath. By imaging the surrounding tissue, a staff member (e.g. a clinician) is able to determine the location and size of a lesion.

[0091] In an option, radial endobronchial ultrasound data from within the bronchial structure is visualized and also shown on the display 108.

[0092] In examples, the intrathoracic device 104 comprises an ultrasound imaging probe and a biopsy extraction device.

[0093] Figure 3 The basic steps of a method 200 for navigation of an intrathoracic device inside a bronchial pathway are schematically illustrated. The method 200 comprises the following steps:

[0094] In a first step 202 (also referred to as step a)), a sequence of live 2D X-ray images of a region of interest of a bronchial structure with an intrathoracic device inserted in the bronchial pathway is received. The intrathoracic device is visible in the X-ray images.

[0095] In a second step 204 (also referred to as step b)), the intrathoracic device is tracked in at least part of the sequence of live 2D X-ray images.

[0096] In a third step 206 (also referred to as step c)), directions and amplitudes of repetitive cardiovascular and respiratory induced motion are evaluated based on the tracked intrathoracic device.

[0097] In a fourth step 208 (also referred to as step d)), the evaluated motion is modeled and navigation information indicative of the modeled motion is generated.

[0098] In a fifth step 210 (also referred to as step e)), the navigation information is provided as a confidence reference to a user operating the intrathoracic device.

[0099] In examples, which are not further illustrated in detail, the method further provides the following steps:

[0100] providing preoperative 3D image data of a segmented bronchial structure comprising objects in the region of interest;

[0101] registering the live 2D X-ray images with the preoperative 3D image data;

[0102] determining a spatial position of the intrathoracic device with respect to the segmented bronchial structure of the preoperative 3D image data based on 2D X-ray projections of the intrathoracic device; and

[0103] displaying the confidence reference in the projection of the segmented bronchial structure overlaid to the live 2D X-ray image.

[0104] In another example, which is also not shown in further detail, the method further provides the following steps:

[0105] providing 3D image data of a segmented bronchial structure of an object comprised in a region of interest derived from an intraoperative X-ray fluoroscopy (e.g. cone beam CT);

[0106] registering the live 2D X-ray image with the 3D image data;

[0107] determining a spatial position of the intrathoracic device with respect to the segmented bronchial structure of the 3D image data based on 2D X-ray projections of the intrathoracic device; and

[0108] displaying the confidence reference overlaid in 3D on the display content of the 3D image data.

[0109] Figure 4 The display content 300 of an enhanced live fluoroscopy 2D image 302 of a thoracic region with an overlaid segmented bronchial tree 304 and a target lesion is schematically shown. The overlay 304 comprises a bronchial structure 306. The EBUS / rEBUS path 308, which is computed after registration, is indicated in a dashed line pattern. The EBUS / rEBUS historical trajectory confidence reference is shown as an example of a confidence reference 310. In addition, also a current position confidence reference 314 is shown in relation to the current position of the EBUS / rEBUS catheter. In this example, the confidence reference is displayed as a 2D overlay, but in other examples, the confidence reference can also be projected as a 3D structure.

[0110] The enhanced X-ray image, i.e. the fluoroscopy 2D image 302, shows the anatomy with slightly visible vertebrae 316, ribs 318 and other structures.

[0111] The static overlay 304 is provided as a segmented visualization of the bronchial structure 306 and the target lesion 312. For easier distinction, a colored static overlay 304 can be provided in comparison to just more or less gray scale X-ray images. In addition, information about the current C-arm orientation 320 or object orientation indicator 322 can be shown overlaid.

[0112] Figure 4An augmented display content of live 2D fluoroscopy images with segmented bronchial paths and segmented target lesions derived from CbCT is provided in conjunction with preoperative CT and rEBUS catheters with transducers and radio-opaque markers. Confidence references support navigation of the rEBUS catheter to the target lesion, confirm its location, and then perform a biopsy sample, e.g., with a biopsy tool.

[0113] For example, based on automated tracking, rEBUS radio-opaque markers and transducers, a confidence reference in 2D and / or 3D indicated by dashed lines around the current presumed location of the ultrasound transducer can be established and further clarify to the operator from which spatial location the rEBUS imaging (not shown) can originate. Furthermore, when the rEBUS transducer is positioned close to the presumed location of the target lesion, this confidence reference can also apply to the lesion, as the presumed lesion also moves in a similar fashion.

[0114] This confidence reference can be calculated for numerous positions of the rEBUS transducer and, if these historical positions are collected, they can be combined and augmented on the fluoroscopy display to provide an overall confidence reference for the route that the rEBUS path outlines to its left and right as demonstrated in the dotted area.

[0115] Furthermore, the radio-opaque profile of the catheter is tracked in 3D and registered to the pre-planned path or the latest cone beam CT constructed position / updated path, which can provide the operator in real-time guidance on how to spatially steer the catheter in real-time and / or get a newly constructed computed path to reach the target lesion with maximum confidence and minimum cone beam CT scans.

[0116] In the interventional radiology suite, the normal clinical procedure is followed, including team introduction and reporting and timeout procedures. The patient is placed on the scan bed and sedation is initiated following the care standard. In all cases, a routine bronchoscopic evaluation of the central airways will be performed.

[0117] During the procedure for guiding navigation and other consecutive tools (e.g. brush, TBNA and / or biopsy forceps) a flexible video bronchoscope can be used in combination with a dedicated video processor. The bronchoscope can be introduced into the bronchial tree through the mouth, laryngeal mask or endotracheal tube. Navigation can be performed based on the preoperative CT scan knowledge of the pulmonologist, the video scope imaging provided by the bronchoscope and the real-time fluoroscopy imaging provided by cone beam CT. If a possible visualization by rEBUS is found and confirmed, further steps will be performed sequentially. When a sufficient tissue sample is confirmed, additional tissue samples for cytology and molecular analysis can be collected, after which the procedure can be terminated. If it is indicated that the target lesion is not reached, a switch to a variety of navigation techniques and dedicated tools will be made. The previous cbCT scan will be used to calibrate the navigation techniques to prevent radiation as much as possible.

[0118] In subjects with PPL but without bronchoscopic signs or without more than 3-4 generations, we will start navigation directly with the dedicated tools. When a peripheral lung lesion is targeted, the pulmonologist can use a steerable or non-steerable catheter or tool inserted through the working channel of the endoscope for navigation. A live update of the 3D position can be provided by intermittent 2D fluoroscopy. The fluoroscopy images can be real-time recombined with the CT scan 3D volume made earlier using image fusion and image processing, providing an update on the catheter and lesion position. After reaching the target by following the reconstructed path visualized on the screen, a probe can be inserted into the catheter (already inserted through the working channel of the endoscope) for visualization confirmation.

[0119] If a peripheral lung lesion can be seen on the image of the probe and the biopsy forceps / tool is indeed confirmed to have entered the interior of the target, a tissue sample can be obtained. For example, imaging confirmation can be based on fluoroscopy. If a peripheral lung lesion cannot be seen on the image of the probe and / or the peripheral lung lesion does not seem to be aligned with the image of the probe, a repositioning of the catheter is performed and the navigation accuracy and probe visualization can be re-evaluated.

[0120] When it seems that the target is reached, repeated tissue biopsies can be performed and the resulting outcome can be evaluated.

[0121] As an example, during navigation from central to distal, navigation at bifurcations can be cumbersome, as video scope imaging can not give positioning information, as one does not know when an angle is made and which direction is made. However, due to the confidence reference or confidence indicator provided on the display, the operator is provided with supportive information for navigation. Thus, the confidence reference compensates for 4D tissue motion caused by cardiovascular and respiratory motion.

[0122] Further, the probe (e.g. rEBUS probe) does not always provide a clear and unambiguous view of the lesion, even if due to the localization of a peripheral lung lesion in the parenchyma it should be visualized without direct contact with the bronchial tree which needs to be accessed using transbronchial and transparenchymal access under cone beam CT 3D guidance, as there is no tissue interface available for ultrasound. There can also be technical limitations in the current ultrasound design as it only images in radial direction. Often, this causes trouble when the bronchus ends in the lesion. On the other hand, there is a lack of tissue contrast for ultrasound in so-called ground glass opacity, which is an increasing part of the problem of peripheral lung lesions.

[0123] Further advantages are provided with respect to the 3D distal tip orientation. When navigating, the physician usually does not have a clear picture if the insertion of the tool causes a 1 : 1 translation in the tissue. As an example, the visualization of the movement out of plane can be poor. In addition to the orientation, it is also provided how the angle / orientation relates to the bronchial bifurcation / lesion.

[0124] Once the lesion is reached and confirmed, biopsies can be repeated. As the lesion can be as small as 5 mm and the tool is not positioned directly inside the lesion but only in the vicinity of the lesion, precise positioning is important. The confidence reference compensates for the situation of lacking knowledge of the precise positioning and angle of the lesion to the catheter, as the lesion cannot be seen in fluoroscopy and repeated cone beam CT is not feasible, e.g. in view of the increased radiation dose to the subject and the number of staff. Thus, the confidence reference can also be helpful in view of trying to avoid systematic biopsies in the region of interest cycle after cycle, if in fact no systematic biopsies are needed.

[0125] It is possible to automatically provide a tracking of the catheter. Thus, by continuously tracking the rEBUS catheter movement while the catheter is kept stationary, the tracking can assess the direction and amplitude of the motion caused by the cardiovascular and respiratory system. Preferably, the catheter should be placed in the vicinity of the lesion or the approximate path should be taken. Under fluoroscopy, one respiratory cycle (thus several heart cycles) should be monitored. It is possible to model and enhance this motion on the fluoroscopy display as a confidence reference which can be used to improve the navigation of the rEBUS catheter to the lesion and provides further information on which area around the target lesion to biopsy, as the lesion is also constantly moving.

[0126] In an example, it also facilitates rEBUS catheter navigation. As an additional option, target lesion location verification and biopsy extraction procedure are also provided, while circumventing additional X-ray exposure by augmenting spatial locations accessed by the rEBUS catheter on the fluoroscopy and registering them with pre-procedural CT planning and / or cone beam CT scan and / or electromagnetic navigation.

[0127] Furthermore, automatically tracking the profile of the radiopaque catheter enables providing more feedback to the operator. For example, by tracking multiple markers with known positions, with a priori knowledge of previous positions during the procedure, feedback can be given immediately on 3D positioning, which the operator might otherwise intuitively miss.

[0128] In an example, if the tracked catheter is used in conjunction with a biopsy extraction mechanism, real-time ultrasound visualization of the lesion and location confirmation and biopsy extraction is facilitated.

[0129] In an example, a catheter is provided that is capable of guiding a "re-entry" needle. This can be used in the same way as in EBUS transbronchial needle aspiration (TBNA), which is a staging procedure that is used to puncture a lesion and successively guide a guide wire into the lesion. Then, a second step is to direct a needle over the guide wire directly into the lesion, thereby improving local confirmation of accurate tissue biopsy. For example, almost 100% accuracy can be achieved.

[0130] According to an aspect, under fluoroscopy in bronchoscopy navigation, the radiopaque properties of potentially and / or existing equipment are used for automated tracking to enhance rEBUS catheter navigation to a lesion, lesion confirmation, and biopsy extraction. For example, computer vision and image processing for 2D and / or 3D tracking of the catheter are deployed. In addition, it is also provided to augment the fluoroscopy images and display the results of the catheter tracking in 2D and / or 3D. In another example, the results of the catheter tracking in 2D / 3D are registered with the results of pre-procedural CT planning and / or cone beam CT scan and / or electromagnetic body navigation.

[0131] Automated rEBUS tracking is provided as, for example, a bronchoscopic navigation software package. The results can be seen on a display.

[0132] For example, the confidence reference is provided to enhance existing cone beam CT system functionality and also to provide improvements in the field of lung examination and treatment, for example, improvements for endobronchial navigation. The confidence reference resulting from catheter tracking with rEBUS is provided to further enhance fluoroscopy augmented display content. As a further option, the results of rEBUS catheter tracking are registered with existing computed and displayed paths (e.g. pre-operative and / or via intra-operative cone beam CT scanning). Yet a further option is to enhance spatial / 3D navigation of rEBUS catheters through further integration with cone beam CT. Providing the confidence reference also improves endobronchial disposable (e.g. ultrasound catheter and future variants thereof) usage.

[0133] In an example, it is suggested to look at how C-arm sweeping at one or more angles can be used to localize and / or reconstruct in 3D. Further, it is suggested to look at how model information can be utilized to update intra-operative CBCT and vice versa (as well as with pre-procedural bronchial anatomy segmentation).

[0134] Additionally, in an example, the rEBUS device is integrated with electromagnetic tracking of the device.

[0135] In a further example, adjustments to 2D fluoroscopy imaging angles are provided to improve accuracy of the visualized model.

[0136] In a further example, operator induced lung / device deformations are distinguished from repetitive breathing / heart motion.

[0137] As indicated, the pre-procedural segmentation results of the bronchial tree are overlaid onto the fluoroscopy results.

[0138] In an option, once the eEBUS confirmation of lesion access is provided, its location information is provided as input to the system to track where and how the rEBUS is positioned in the model.

[0139] In an example, it is provided to extract global features such as larger blood vessels. These global features can be used to update model accuracy, for example, pre-procedural bronchial tree segmentation in combination with blood vessel segmentation and local deformations.

[0140] As an option, automated zoom and collimation functionality is provided for navigation.

[0141] In an example, a computer program is provided which enables a processor to carry out the method of one of the above examples.

[0142] In an example, a computer program or program element for controlling an apparatus according to one of the above examples is provided, which program or program element, when executed by a processing unit, is adapted for carrying out the method steps of one of the above method examples.

[0143] In an example, a computer readable medium having stored the program element of the above-described example is provided.

[0144] In another exemplary embodiment of the present application, a computer program or a computer program element is provided that is characterized by being adapted to execute the method steps of the method according to one of the preceding embodiments, on an appropriate system.

[0145] Accordingly, the computer program element might be stored on a computer unit, which might also be part of an embodiment of the present application. This computer unit can be adapted to perform or induce a performing of the steps of the method described above. Moreover, it can be adapted to operate the components of the apparatus described above. The computing unit can be adapted to operate automatically and / or it can be adapted to be controlled by a user.

[0146] Aspects of the application can be embodied in a computer program product which can be stored on a computer readable storage medium (disk, optical disk, memory, etc.) which stores the program having instructions that can be executed by a computer. The instructions of the application can be in any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs) or Java classes. The instructions can be provided as complete executable programs, partial executable programs, modifications to existing programs (e.g. updates) or extensions for existing programs (e.g. plugins). Moreover, parts of the processing of the application can be distributed over multiple computers or processors.

[0147] As discussed above, a processing unit (e.g., controller) implements the control method. The controller can be implemented in numerous ways, with software and / or hardware, to perform the various functions required. A processor is one example of a controller which employs one or more microprocessors that can be programmed using software (e.g., microcode) to perform the required functions. A controller can however be implemented with or without employing a processor, and also can be implemented as a combination of dedicated hardware to perform some functions and a processor (e.g., one or more programmed microprocessors and associated circuitry) to perform other functions.

[0148] Examples of controller components that can be employed in various embodiments of the disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0149] This exemplary embodiment of the present application covers both the computer program from the very beginning using the present application and by means of a computer program updating an existing program into a program using the present application.

[0150] Further on, the computer program element might be able to provide all the steps to complete the procedural flow of the exemplary embodiments of the method as described above.

[0151] According to a further exemplary embodiment of the present application, a computer readable medium, such as a CD-ROM, having stored thereon the computer program of this preceding paragraph is presented. A computer program can be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid state storage medium supplied together with or as a part of other hardware, but also by means of an electronic signal. The latter can be supplied on a data carrier such as an optical data carrier, a solid state storage carrier or a radio frequency carrier signal.

[0152] However, the computer program can also be presented over a network like the World Wide Web and can be downloaded into the working memory of a data processor from such a network. According to a further exemplary embodiment of the present application, a medium for making a computer program element available for downloading is provided, which computer program element is arranged to perform a method according to one of the previously described embodiments of the application.

[0153] It has to be noted that embodiments of the application are described with reference to different subjects. In particular, some embodiments are described with reference to method type claims whereas other embodiments are described by way of apparatus type claims. Unless explicitly described as methods or unless method claims are explicitly presented it is appreciated that the described features and / or steps of the described embodiments can also be implemented as apparatus features and / or as apparatus steps.

[0154] However, any combination of subject matter, presented with respect to one type of subject matter, is also contemplated as disclosed herein to the extent not inconsistent herewith. However, all features that can be implemented as a combination of subject matter with respect to one type of subject matter are also contemplated as a combination of subject matter with respect to another type of subject matter.

[0155] While the application has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description are to be considered illustrative or exemplary only. The application is not limited to the disclosed embodiments. Other variations that are within the spirit of the present application will become readily apparent to those skilled in the art from reading the foregoing description. The scope of the application is accordingly indicated by the appended claims, rather than the foregoing description, and all changes that come within the meaning and range of equivalents are intended to be embraced therein.

[0156] In the claims, the term "comprising" does not exclude other elements or steps, and the wording "a" or "an" does not exclude a plurality. A single processor or other unit can fulfil the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A device (10) for facilitating navigation of an intrathoracic device inside a bronchial pathway, the device comprising: an image data providing section (12); a processor (14); and an output data providing section (16); wherein the image data providing section is configured to receive a sequence of 2D X-ray images of a region of interest of a bronchial structure with an intrathoracic device inserted in the bronchial pathway; wherein the intrathoracic device is visible in the X-ray images; wherein the processor is configured to track the intrathoracic device in at least part of the sequence of 2D X-ray images; and to assess directions and amplitudes of repetitive cardiovascular and respiratory induced motion based on the tracked intrathoracic device; and to model the assessed motion and generate navigation information indicative of a range of the modeled motion; and wherein the output data providing section is configured to provide the navigation information as a confidence reference to a user operating the intrathoracic device.

2. The device for facilitating navigation of an intrathoracic device inside a bronchial pathway of claim 1, wherein, The confidence reference is provided as an indication of an envelope of a spatial range encompassing the modeled motion.

3. The device for facilitating navigation of an intrathoracic device inside a bronchial pathway according to claim 1 or 2, wherein, The output data providing section comprises a display (18) configured to display the confidence reference overlaid to a current 2D X-ray image of the sequence of 2D X-ray images.

4. The device for facilitating navigation of an intrathoracic device inside a bronchial pathway according to claim 1 or 2, wherein, The processor is configured to track the intrathoracic device in at least part of the sequence of 2D X-ray images; and to assess directions and amplitudes of non-repetitive lung tissue and device motion induced by intrathoracic device manipulation or its guiding elements; and to track the motion and generate navigation information adjusted for the non-repetitive motion.

5. The device for facilitating navigation of an intrathoracic device inside a bronchial pathway according to claim 1 or 2, wherein the image data providing section is further configured to provide preoperative 3D image data of a segmented bronchial structure including objects in the region of interest; wherein the processor is configured to register the 2D X-ray images with the preoperative 3D image data; and to determine a spatial position of the intrathoracic device with respect to the segmented bronchial structure of the preoperative 3D image data based on 2D X-ray projections of the intrathoracic device; and wherein the output data providing section comprises a display configured to display the confidence reference in a projection of the segmented bronchial structure, both the confidence reference and the projection of the segmented bronchial structure being overlaid to a current 2D X-ray image of the sequence of 2D X-ray images.

6. The device for facilitating navigation of an intrathoracic device inside a bronchial pathway of claim 1 or 2, wherein, the image data providing section is further configured to provide 3D image data derived from intraoperative X-ray fluoroscopy and further comprising a segmented bronchial structure of objects in the region of interest; wherein the processor is configured to register the 2D X-ray images with the 3D image data; and to determine a spatial position of the intrathoracic device with respect to the segmented bronchial structure of the 3D image data based on 2D X-ray projections of the intrathoracic device; and wherein the output data providing section comprises a display configured to display the confidence reference overlaid in 3D form on the display of the 3D image data.

7. The device for facilitating navigation of an intrathoracic device inside a bronchial pathway of claim 6, wherein, The intraoperative X-ray fluoroscopy is cone-beam computed tomography.

8. The device for facilitating navigation of an intrathoracic device inside a bronchial pathway of claim 1 or 2, wherein, The image data providing section is configured to receive a live stream of 2D X-ray images as the sequence; and wherein the processor is configured to continuously track the intrathoracic device in the image; and to update the navigation information accordingly; and wherein the output data providing section is configured to continuously provide the navigation information as updated confidence reference to a user operating the intrathoracic device.

9. The device for facilitating navigation of an intrathoracic device inside a bronchial pathway of claim 1 or 2, wherein, The X-ray information is also updated with and / or integrated with electromagnetic tracking information.

10. A system (100) for navigation of an intrathoracic device inside a bronchial pathway, the system comprising: an X-ray imaging device (102) for imaging a thoracic region of interest of a subject; an intrathoracic device (104) for being inserted in a bronchial pathway of a subject; a display (108); a device (106) for facilitating navigation according to any of the preceding claims; and wherein the intrathoracic device is at least partially radiopaque; wherein the X-ray imaging device provides a sequence of 2D X-ray images of the region of interest of the bronchial structure with the intrathoracic device inserted in the bronchial pathway; wherein the device for facilitating navigation provides the navigation information to the display; and wherein the display provides at least one of the sequence of 2D X-ray images, and the navigation information as a confidence reference overlaid to the 2D X-ray images.

11. The system according to claim 10, wherein, The intrathoracic device is an ultrasound imaging probe configured to provide radial endobronchial ultrasound data from within a bronchial structure.

12. The system of claim 11, wherein, The radial endobronchial ultrasound data from within the bronchial structure is visualized and also shown on the display.

13. The system of claim 10, 11, or 12, wherein, The intrathoracic device comprises an ultrasound imaging probe and a biopsy extraction device.

14. A method (200) for navigation of an intrathoracic device inside a bronchial pathway, the method comprising the steps of: receiving (202) a sequence of live 2D X-ray images of a region of interest of a bronchial structure with an intrathoracic device inserted in a bronchial pathway; wherein the intrathoracic device is visible in the X-ray images; tracking (204) the intrathoracic device in at least part of the sequence of live 2D X-ray images; evaluating (206) directions and amplitudes of repetitive cardiovascular and respiratory induced motion based on the tracked intrathoracic device; modeling (208) the evaluated motion and generating navigation information indicative of the modeled motion; and providing (210) the navigation information as a confidence reference to a user operating the intrathoracic device.

15. A computer program product comprising a computer program enabling a processor to perform the method for navigation of an intrathoracic device inside a bronchial pathway according to claim 14.

16. A computer readable medium having stored a computer program enabling a processor to perform the method for navigation of an intrathoracic device inside a bronchial pathway according to claim 14.

Images