Navigation in hollow anatomical structures

Abstract

The invention relates to medical navigation. In order to provide information that facilitates navigation in a hollow anatomical structure, a device (10) for navigation in a hollow anatomical structure is provided, the device comprising a data input (12), a data processor (14) and an output interface (16). The data input is configured to: provide 3D image data of a hollow structure in a region of interest of a subject, where the 3D image data includes a coordinate space; a current pose of a tool having a tool tip inserted into the hollow structure is provided. The data processor is configured to: transform an estimated current pose of the tool tip to a coordinate space of the 3D image data based on a registration of the tool tip with the coordinate space of the 3D image data; and generating a rendered image from the 3D image data, the rendered image showing a scene inside the hollow structure related to the transformed estimated current pose of the tool tip. The output interface is configured to provide the rendered image to a user.

Description

Technical Field

[0001] The present invention relates to medical navigation, and in particular to a device for navigating in a hollow anatomical structure, a system for navigating in a hollow anatomical structure, and a method for navigating in a hollow anatomical structure. Background Art

[0002] Minimally invasive surgery can be performed under the guidance of a navigation imaging device that can be inserted into the patient's body for imaging and / or navigation to an area of ​​interest. Examples include devices with RGB cameras (endoscopes, bronchoscopes, etc.) and other imaging modalities such as endobronchial ultrasound (EBUS), intravascular ultrasound (IVUS), optical coherence tomography (OCT), etc. Navigation imaging devices are sometimes used in combination with other interventional imaging modalities such as X-ray, tomosynthesis, or cone beam computed tomography (CBCT). These modalities capture a large field of view that enables the positioning of the navigation imaging device within a wider reference frame of the patient's anatomy. However, these modalities expose the patient and surgical staff to ionizing radiation, so it is critical to minimize the use of ionizing radiation. Summary of the invention

[0003] Therefore, convenient information for navigating in hollow anatomical structures may be needed.

[0004] The objects of the invention are solved by the subject-matter of the independent claims; further embodiments are incorporated in the dependent claims. It should be noted that the following described aspects of the invention also apply to an apparatus for navigating in a hollow anatomical structure, a system for navigating in a hollow anatomical structure and a method for navigating in a hollow anatomical structure.

[0005] According to the present invention, a device for navigating in a hollow anatomical structure is provided. The device includes a data input unit, a data processor and an output interface. The data input unit is configured to provide 3D image data of a hollow structure in a region of interest of a subject, wherein the 3D image data includes a coordinate space. The data input unit is also configured to provide a current posture of a tool with a tool tip inserted into the hollow structure. The data processor is configured to convert the estimated current posture of the tool tip to the coordinate space of the 3D image data based on the registration of the tool tip with the coordinate space of the 3D image data. The data processor is also configured to generate a rendered image based on the 3D image data, the rendered image showing a scene related to the converted estimated current posture of the tool tip inside the hollow structure. The output interface is configured to provide the rendered image to a user.

[0006] In a preferred embodiment of the invention, the tool does not have any imaging capabilities. A particular advantage is that while the user would not normally see what the tool would see if the tool had imaging capabilities (e.g., a camera or transducer, etc.), the user can now get a view of what the tool would see (a tool-centric view) via a rendered image generated from the 3D image data.

[0007] According to an embodiment of the present invention, the data input unit is configured to provide current image data of a region of interest acquired by an interventional imaging device arranged in a hollow structure in a current posture. The current image data includes image data related to a tool having a tool tip inserted into the hollow structure. The data processor is configured to register the interventional imaging device in the current posture with the coordinate space of the 3D image data. The data processor is also configured to estimate the current posture of the tool tip visible in the current image data. The data processor is also configured to convert the current posture of the tool tip estimated according to the current image data into the coordinate space of the 3D image data based on the registration of the interventional imaging device in the coordinate space of the 3D image data.

[0008] In practice, a detailed image of the current scene (also referred to as the current situation) is rendered within a reference frame of the 3D image data. In an example, the 3D image data is preoperative image data, i.e., pre-acquired image data. In an example, the scene inside the hollow structure associated with the converted estimated current pose of the tool tip is a rendered image representing the field of view from the converted estimated current pose of the tool tip. In an example, the rendered scene may also include an adjustable transparency of the target lesion and at least part of the structure in the rendered image.

[0009] According to an example, for registration of the interventional imaging device with the coordinate space of the 3D image data, the data processor is configured to estimate a current pose of the interventional imaging device using current image data of the region of interest acquired by the interventional imaging device.

[0010] According to an example, the 3D image data is at least one of the following group: computed tomography (CT) image data, cone beam computed tomography (CBCT) image data or magnetic resonance (MRI) image data of the subject. In an option, the current image data includes at least one of the following group: camera image data from an endoscope, colonoscope or bronchoscope, image data from an ultrasound transducer device or optical coherence tomography image data.

[0011] In an example of the apparatus, the data processor is configured to render an image representing a field of view from a tool tip. In an option, the data processor is further configured to render a target lesion in the rendered image and provide adjustable transparency for at least a portion of the structure in the rendered image.

[0012] According to an example, for registration of an interventional imaging device with a coordinate space of 3D image data, the data processor is configured to provide initial 3D image data of the interventional imaging device within a region of interest. The data processor is configured to segment the interventional imaging device in the initial 3D image data to generate an initial pose. The data processor is further configured to track the interventional imaging device. The data processor is further configured to adjust the initial pose based on the tracking.

[0013] According to an example, for registration of an interventional imaging device with a coordinate space of 3D image data, the data processor is configured to provide initial 2D image data of the interventional imaging device within a region of interest. The initial 2D image data includes at least one initial 2D image. The data processor is further configured to segment the interventional imaging device in the at least one initial 2D image. The data processor is further configured to register the at least one initial 2D image with the 3D image data. The data processor is further configured to initialize the segmented interventional imaging device within the coordinate space of the 3D image data to provide an initial pose of the interventional imaging device. The data processor is further configured to track the interventional imaging device and adjust the initial pose of the interventional imaging device based on the tracking.

[0014] According to an example, for registration of an interventional imaging device with a coordinate space of 3D image data, the data processor is configured to provide 2D image data from an interventional imaging device within a region of interest. The 2D image data includes at least two 2D images with different viewing directions. The data processor is also configured to estimate a camera pose based on the 2D image data, and triangulate a 3D structure from the at least two images based on the estimated camera pose and features visible in the at least two images. The data processor is also configured to initially segment the structure within the 3D image data. The data processor is also configured to register the triangulated 3D structure with the corresponding structure segmented in the 3D image data, and register and initialize the interventional imaging device in the coordinate space of the 3D image data. The data processor is also configured to track the interventional imaging device and adjust the initial pose of the interventional imaging device based on the tracking.

[0015] According to an example, the data processor is configured to generate a confidence estimate. The confidence estimate relates to at least one of the following group: a quality of an image used to estimate the pose, a quality of the pose estimate, and a quality of the registration. The output interface is configured to provide the confidence estimate to a user.

[0016] According to the present invention, a system for navigating in a hollow anatomical structure is provided. The system comprises: an interventional imaging device configured for insertion into a hollow structure; a tool having a tool tip configured for insertion into a hollow structure; and a device for navigating in a hollow anatomical structure according to one of the aforementioned examples. The system also comprises a display. The interventional imaging device provides current image data of a region of interest of a subject. The display shows a generated rendered image.

[0017] According to the present invention, a method for navigating in a hollow anatomical structure is also provided. The method comprises the following steps:

[0018] 3D image data of a hollow structure in a region of interest of a subject is provided. The 3D image data includes a coordinate space.

[0019] A current pose of a tool having a tool tip inserted into a hollow structure is provided.

[0020] Based on the registration of the tool tip with the coordinate space of the 3D image data, the estimated current pose of the tool tip is transformed into the coordinate space of the 3D image data.

[0021] A rendered image is generated based on the 3D image data, the rendered image showing a scene inside a hollow structure associated with the transformed estimated current pose of the tool tip.

[0022] The rendered image is provided to a user.

[0023] According to an example of this method, the following steps are provided:

[0024] The interventional imaging device is disposed in the hollow structure.

[0025] Current image data of a region of interest acquired by an interventional imaging device in a current posture is provided; wherein the current image data includes image data related to a tool having a tool tip inserted into a hollow structure.

[0026] The interventional imaging device in the current posture is registered with the coordinate space of the 3D image data.

[0027] Estimate the current pose of the tool tip.

[0028] Based on the registration of the interventional imaging device with the coordinate space of the 3D image data, the current pose of the tool tip estimated from the current image data is converted to the coordinate space of the 3D image data.

[0029] According to one aspect, the system observes tools visible in the bronchoscope field of view and outputs an estimated pose of the tool in the bronchoscope coordinate frame. Given the registration between the bronchoscope and the preoperative CT, a tool-centric view or a view representing the field of view from the tool tip can be rendered in the CT coordinate space to improve the visualization of the current state of the art. This additional visualization can enable several downstream tasks as listed below.

[0030] According to one aspect, the pose of such tools visible in the field of view of the navigation imaging device is estimated, and a view of the estimated tool tip is rendered in a coordinate frame registered to a 3D image (e.g., preoperative CT) of the navigation imaging device. These tool-centric rendered views can in turn enable other downstream tasks listed below.

[0031] According to one aspect, once registered, the navigation imaging device can be visualized in the preoperative image space via an overlay, or a rendered view in the preoperative image space can be generated from the registered imaging device pose. The rendered view allows visualization of the preoperative planned path or other targets, such as target lesions, in the preoperative image space. In an option, the pose of the tools visible in the field of view of the navigation imaging device is estimated. Although these tools do not provide any imaging feedback, since they are visible in the navigation imaging device, the pose of the tool in the coordinate space of the navigation imaging device can be estimated, and a view can be rendered from the estimated tool tip. These tool-centric rendered views can in turn enable other tasks, such as determining whether the tool can be navigated in a particular direction or whether the tool placement is optimal for subsequent tasks.

[0032] The field of use is any interventional imaging system including but not limited to bronchoscopes, endoscopes, etc. when used with pre-operative or intra-operative 3D imaging (eg, CT, CBCT, etc.).

[0033] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Exemplary embodiments of the present invention will be described below with reference to the accompanying drawings:

[0035] Figure 1 An example of a device for navigating in a hollow anatomical structure is schematically shown.

[0036] Figure 2 An example of a system for navigating in a hollow anatomical structure is shown.

[0037] Figure 3 Basic steps of an example of a method for navigating in a hollow anatomical structure are shown.

[0038] Figure 4 Shows Figure 3 Another example step of the method.

[0039] Figure 5 A fluoroscopic image with a bronchoscope and tools inserted through a working channel of the bronchoscope is shown.

[0040] Figure 6 An example of a workflow is shown.

[0041] Figure 7 Another example of a workflow is shown. DETAILED DESCRIPTION

[0042] Certain embodiments will now be described in more detail with reference to the accompanying drawings. In the following description, the same reference numerals are used for the same elements, even in different drawings. Matters defined in the specification, such as detailed construction and elements, are provided to help fully understand the exemplary embodiments. Moreover, well-known functions or constructions are not described in detail because they would obscure the embodiments with unnecessary details. In addition, when preceding a list of elements, expressions such as "at least one" modify the entire list of elements without modifying individual elements in the list.

[0043] Figure 1 An example of a device 10 for navigating in a hollow anatomical structure is schematically shown. The device 10 includes a data input 12, a data processor 14, and an output interface 16. The data input 12 is configured to provide 3D image data of a hollow structure in a region of interest of a subject. The 3D image data includes a coordinate space. The data input 12 is also configured to provide a current posture of a tool with a tool tip inserted into the hollow structure. The data processor 14 is configured to convert the estimated current posture of the tool tip to the coordinate space of the 3D image data based on the registration of the tool tip with the coordinate space of the 3D image data. The data processor 14 is also configured to generate a rendered image based on the 3D image data, which shows a scene related to the converted estimated current posture of the tool tip inside the hollow structure. The output interface 16 is configured to provide the rendered image to a user.

[0044] exist Figure 1In the example option of , the data input unit 12 is configured to provide current image data of the region of interest acquired by an interventional imaging device arranged in the hollow structure in a current posture. The current image data includes image data related to a tool with a tool tip inserted into the hollow structure. The data processor 14 is configured to align the interventional imaging device in the current posture with the coordinate space of the 3D image data. The data processor 14 is also configured to estimate the current posture of the tool tip visible in the current image data. The data processor 14 is also configured to convert the current posture of the tool tip estimated based on the current image data to the coordinate space of the 3D image data based on the alignment of the interventional imaging device with the coordinate space of the 3D image data.

[0045] The current pose of the tool having the tool tip is estimated based on current image data of the region of interest acquired by an interventional imaging device arranged in the hollow structure at the current pose.

[0046] The term "3D image data" relates to spatial data of a subject that has been acquired by a 3D medical imaging procedure such as 3D ultrasound imaging, computed tomography (CT) imaging or computed tomography angiography (CTA) imaging or cone beam computed tomography (CBCT) or 3D rotational angiography (3DRA) or X-ray tomosynthesis imaging or magnetic resonance (MRI) imaging or magnetic resonance angiography (MRA) imaging.

[0047] The term "hollow anatomical structure" refers to an anatomical structure suitable for insertion of an interventional device such as a catheter, endoscope, bronchoscope, or any intravascular or intrabronchial device. Examples of hollow structures are blood vessels, heart chambers, respiratory passages (i.e., trachea or airways), or the digestive system including the esophagus, stomach, intestines, and colon.

[0048] The term "tool" refers to an interventional device configured to be inserted into a subject's body to perform an interventional medical task. The tool may or may not have imaging capabilities, but the present invention has particular advantages for tools that do not have the ability to capture images (e.g., tools without cameras, transducers (e.g., ultrasonic transducers) or sensors suitable for obtaining images). In fact, these tools cannot provide users with images or views that the tool (if it is equipped with, for example, a camera) may see in its field of view. As a non-limiting example, such a tool may include a needle for collecting a sample by piercing the tissue to be sampled, or an aspiration needle that can aspirate a tissue sample, or a forceps that can tear a tissue sample, or a brush for scraping tissue to collect a sample. Some of these tools may be configured to be inserted into the subject's body through a working channel in an interventional imaging device, or they may be directly inserted into the subject's body. In both cases, the tool may be visible in the field of view of the interventional imaging device. The term "tool tip" refers to the farthest end of the tool.

[0049] The term "interventional imaging device" relates to a device configured for insertion into a subject's body and providing image data of the body when inserted into the body. The interventional imaging device may also be provided for other purposes, such as measurement, monitoring or treatment. In an example, the interventional imaging device is provided as an endoscope or a bronchoscope. In an example, the interventional imaging device is provided as an imaging catheter such as optical coherence tomography or intravascular ultrasound for vascular structures.

[0050] The term "current image data" relates to image data provided in a current state, for example as a real-time image during a medical procedure or intervention. In an example, the image data is provided as 2D image data in an image plane, which may be referred to as current 2D image data. In another example, the image data is provided as 3D image data, which may be referred to as current 3D image data.

[0051] The term "pose" relates to the position and orientation of, for example, a tool tip or an interventional imaging device (eg, an imaging component of an interventional imaging device).

[0052] For example, the term "estimation" relates to, for example, attempts to assess and determine the current pose.

[0053] The term "transformation" involves calculating the relationship between two reference frames (also called coordinate spaces) and determining corresponding features in one of the two reference frames, such as the current pose, in the other of the two reference frames. The term "reference frame" refers in particular to the coordinate space of the current image acquired with an interventional imaging device, or the coordinate space of the 3D image data, for example. The transformation is based on the result of the registration (also called transformation) and involves defining how the current image data needs to be transformed (i.e. changed in a broad sense) to align with the 3D data set.

[0054] The term "coordinate space" refers to the spatial grid or reference system that provides a corresponding reference frame.

[0055] The term "registration" involves calculating the spatial relationship of two different image data sets. The spatial relationship includes information about how to manipulate the corresponding other data for spatial matching. Registration includes a rigid registration part, i.e., the global registration of 2D image data in the coordinate space of 3D image data. Registration also includes a non-rigid registration part, i.e., the deformation or deformable registration process of 2D image data to 3D image data. Rigid registration involves different viewing angles and distances, such as caused by the movement of the subject support. Non-rigid registration involves deformation of the subject itself, such as caused by breathing or other activities (such as organ movement, specifically including heartbeat).

[0056] The term "rendered image" relates to an image generated by a graphics process called rendering, which basically means providing the 3D surface of an object in a 2D image by simulating the interaction of light with the object (for example, using a process like ray tracing). The rendering process can generate images with a somewhat realistic appearance by including realistic components such as light scattering and surface properties. The rendering process can also generate images showing pure wireframe models, or wireframe models with hidden lines that are not shown, by excluding computationally expensive realistic components. A rendered image may also be called a rendering.

[0057] In an example, the image data source is a data storage device storing 3D CT image data of a subject. In one option, the image data source is a CBCT system that is data connected to a device used to navigate in hollow anatomical structures during medical procedures.

[0058] The term "data input" refers to providing or supplying data for a data processing step. The data input section 12 may also be referred to as an image data input section. The data input section 12 may also be referred to as a data supply section, an image data supply section, an image input section, an input unit, or simply an input section. In an example, the image data input section 12 may be data-connected to an imaging source device. In an example, the data input section 12 may be data-connected to a data storage device that has stored image data.

[0059] The term "data processor" relates to a part of a processor or processor device that is provided to perform a computational step using the data provided by the data input. The data processor 14 may also be referred to as a data processing device, a processor unit or a processor. In the example, the data processor 14 is data connected to the data input and output interface.

[0060] The term "output interface" refers to an interface for providing processed or calculated data for further purposes. The output interface 16 may also be referred to as an output portion or an output unit. In an example, the output interface 16 may be data connected to a display device or a display apparatus. In another example, the output interface 16 is data connected to a display.

[0061] The data input section 12, the data processor 14 and the output interface 16 may be arranged in a common housing structure, as shown by the frame 18. They may also be provided in an integrated manner. However, they may also be provided as separate components.

[0062] A first arrow 20 indicates the supply of data to the data input 12. A second arrow 22 indicates the provision of data from the output interface 16.

[0063] exist Figure 1 In the example shown as an option, a display 24 is provided to display the rendered image. In the option, a current image is also shown based on the current image data.

[0064] By using 3D image data to generate rendered images, the limitation of imaging devices being able to capture only a small or partial field of view within the lumen in which they are navigated is compensated for.

[0065] In an example, the imaging device is equipped with a working channel through which other devices or tools can be inserted to treat the target.

[0066] As an example, if the navigation imaging device is large and cannot be inserted into a narrow lumen, the tools can be inserted further while being guided by the interventional imaging device. For example, a bronchoscope may be limited in how far into the airway it can navigate, such as into the central or intermediate airways, but not into the smaller peripheral airways. On the other hand, the tools and devices that can be inserted into the working channel are smaller and can be navigated further into smaller lumens. Based on the navigation interventional imaging device, these tools can be navigated towards a target area of ​​interest, or even oriented and positioned relative to a target area of ​​interest.

[0067] In this example of the apparatus, the data processor 14 is configured to estimate a current pose of the tool tip based on the current image data.In an option, the data processor 14 is configured to generate a rendered image representing the field of view from the tool tip.

[0068] In an example, a bronchoscope image processing controller is provided to estimate the current pose of a tool visible in the bronchoscope field of view. This can be done using image processing techniques (such as segmentation, optical flow, etc.) and / or adapting the latest machine / deep learning techniques to the medical domain to estimate the 3D pose of an articulated tool.

[0069] In an example, an estimate of the current pose of the tool tip is based on tracking data from the tool tip. For example, the tracking data is provided as relative tracking data between the tool tip and an interventional imaging device. Tracking data may also be provided using a tracking device (e.g., electromagnetic (EM) tracking, shape sensing, etc.).

[0070] In an example, a rendered image representing a field of view of an interventional imaging device is generated.

[0071] In the example, the current distance between the current pose of the tool tip and the determined target position is calculated. The current distance is provided to the user

[0073] In an example, if the location of the target lesion is known (e.g., by segmentation), the estimated pose of the tool can be used to calculate a distance to the lesion metric, which can be shown on the user interface. The distance can be calculated by estimating the centerline of the airway between the tool position and the lesion location and measuring the length of the centerline (e.g., in mm).

[0074] In an example, a device for navigating in a hollow anatomical structure is provided, comprising: a data input unit; a data processor; and an output interface. The data input unit is configured to: provide 3D image data of a hollow structure in a region of interest of a subject, wherein the 3D image data includes a coordinate space; provide current image data of the region of interest acquired by an interventional imaging device arranged in the hollow structure at a current posture, wherein the current image data includes image data related to a tool with a tool tip inserted into the hollow structure. The data processor is configured to: align the interventional imaging device in the current posture with the coordinate space of the 3D image data; estimate the current posture of the tool tip visible in the current image data; based on the alignment of the interventional imaging device with the coordinate space of the 3D image data, convert the current posture of the tool tip estimated according to the current image data to the coordinate space of the 3D image data; and generate a rendered image according to the 3D image data, the rendered image showing a scene inside the hollow structure related to the converted estimated current posture of the tool tip. The output interface is configured to provide the rendered image to a user.

[0075] In an example, the scene inside the hollow structure associated with the transformed estimated current pose of the tool tip is a rendered image representing a field of view from the transformed estimated current pose of the tool tip. Such a rendered image may be referred to as a tool-centric view. In an example, the rendered scene may also include an adjustable transparency of the target lesion and at least a portion of the structure in the rendered image.

[0076] In an example, for registration of a device (eg, an interventional imaging device or tool) with a coordinate space of 3D image data, the data processor 14 is configured to estimate a current pose of the device.

[0077] In an example, for registration of the interventional imaging device with the coordinate space of the 3D image data, the data processor 14 is configured to estimate a current pose of the interventional imaging device using current image data of the region of interest acquired by the interventional imaging device.

[0078] In an example, for registration of the tool with the coordinate space of the 3D image data, the data processor 14 is configured to estimate the current pose of the tool using a tracking device such as EM tracking, shape sensing or the like.

[0079] In another example, the 3D image data is CT image data of a subject.

[0080] In an option, said current image data comprises at least one of the following group: camera image data from an endoscope or bronchoscope, image data from an ultrasound transducer device or optical coherence tomography image data.

[0081] In an example, the interventional imaging device is an endoscope or bronchoscope or other intravascular or endobronchial device.

[0082] In an example, in addition to or as an alternative to the interventional imaging device with an RGB camera, other navigation imaging devices are provided, for example: ultrasound devices, such as endobronchial ultrasound (EBUS), intravascular ultrasound (IVUS), etc., or optical coherence tomography (OCT), etc. As in the above examples, these devices can also be tracked using image processing-based methods (if available) or using other tracking devices (e.g. EM tracking, shape sensing-based tracking, etc.).

[0083] In an example, for registration of the interventional imaging device with the coordinate space of the 3D image data, the data processor 14 is configured to provide initial 3D image data of the interventional imaging device within the region of interest. The data processor 14 is also configured to segment the interventional imaging device in the initial 3D image data generating an initial pose. The data processor 14 is also configured to track the interventional imaging device and adjust the initial pose based on the tracking.

[0084] In an example, for registration of an interventional imaging device with a coordinate space of 3D image data, the data processor 14 is configured to provide initial 2D image data of the interventional imaging device within a region of interest. The initial 2D image data includes at least one initial 2D image. The data processor 14 is configured to segment the interventional imaging device in the at least one initial 2D image. The data processor 14 is configured to register the at least one initial 2D image with the 3D image data. The data processor 14 is configured to initialize the segmented interventional imaging device within the coordinate space of the 3D image data to provide an initial pose of the interventional imaging device. The data processor 14 is configured to track the interventional imaging device position and adjust the initial pose of the interventional imaging device based on the tracking.

[0085] In an example, the 2D image data of the interventional device may be an X-ray image acquired during surgery. In an example, multiple 2D images from different viewing directions may be acquired. In an example, the multiple 2D images from different viewing directions may be used to perform image reconstruction to generate an intraoperative 3D tomosynthesis image or a 3D cone-beam computed tomography image prior to registration with the initial 3D image data.

[0086] In an example, for registration of an interventional imaging device with a coordinate space of 3D image data, a data processor 14 is configured to provide 2D image data from an interventional imaging device within a region of interest. The 2D image data includes at least two 2D images having different viewing directions. The data processor 14 is configured to estimate a camera pose based on the 2D image data. The data processor 14 is configured to triangulate a 3D structure from the at least two images based on the estimated camera pose and features visible in the at least two images. The data processor 14 is configured to initially segment structures within the 3D image data. The data processor 14 is configured to register the triangulated 3D structure with a corresponding structure segmented in the 3D image data. The data processor 14 is configured to register and initialize the interventional imaging device in the coordinate space of the 3D image data. The data processor 14 is configured to track the interventional imaging device. The data processor 14 is configured to adjust the initial pose of the interventional imaging device based on the tracking.

[0087] In an option, these examples also apply to tools. In other words, in the above example, the 3D image may contain a tool within a region of interest, which is segmented to provide an initial pose of the tool, which is then tracked, for example, by EM tracking, shape sensing, etc.

[0088] In an example, a device, such as an interventional imaging device and / or tool, is provided. The device may be an interventional imaging device or tool.

[0089] In an example, the data processor 14 is configured to provide said tracking by at least one of the following group: i) relative camera pose estimation based on a sequence of camera images; and ii) at least one of electromagnetic tracking, robotic manipulation data or shape sensing.

[0090] In an example, the data processor 14 is configured to provide tracking by relative pose estimation based on image sequences (eg, ultrasound images, OCT images, and other imaging devices).

[0091] Several of these navigated interventional imaging devices may also be equipped with EM trackers or shape sensing devices, e.g., Fiber Optic RealShape “FORS,” which need to be registered once, e.g., at the beginning of the procedure, and can then be tracked in real time in the preoperative image space throughout the procedure.

[0092] In an example, current fluoroscopic image data is provided and the data processor 14 is configured to provide an updated estimate of the current pose of the tool tip based on the current fluoroscopic image data.

[0093] In an example, for a situation where the tool tip is no longer visible in the bronchoscope field of view but is visible in a subsequent fluoroscopic image, the estimated tool pose from when the tool was visible in the bronchoscope is updated with a coarse pose estimate from the fluoroscopic examination. The tool pose can be optimized by minimizing the error between the fluoroscopic image and a synthetic projection image (e.g., digitally reconstructed radiographs (DRR)) generated by updating the tool pose in CT space. A tool-centric view can then be generated from the updated tool pose.

[0094] In an example, a tool inserted through a working channel of a navigation imaging device is tracked via EM tracking, shape sensing, gyroscope (orientation), etc. In this case, the bronchoscopic image processing controller of the above example is supplemented or replaced by a tracking data processing controller, where the tool pose is estimated directly from the tracking information. A tool-centric view is then rendered based on the calculated tool pose, as previously described.

[0095] In an example of the apparatus, the data processor 14 is configured to: render the target lesion in the rendered image; and provide an adjustable transparency for at least a portion of the structure in the rendered image.

[0096] In an example, a target lesion is also rendered in the bronchoscope and / or tool view as long as the lesion is within the field of view, even if the lesion is occluded by other structures. Changing the transparency in the lesion rendering can indicate the level of occlusion and / or distance from the bronchoscope / tool. That is, when the lesion is far away and occluded by many different anatomical structures, the lesion is rendered with high transparency, and when the lesion is close to the bronchoscope / tool, the lesion is rendered with low transparency.

[0097] In an example, the data processor 14 is configured to generate a confidence estimate; wherein the confidence estimate relates to at least one of the quality of the image used to estimate the pose, the quality of the pose estimate and the quality of the registration. The output interface 16 is configured to provide the confidence estimate to a user.

[0098] In the example, confidence estimation is also performed together with the pose estimation. Confidence estimation can be performed using any method available in the art, and can be based on, for example, the quality of the image used to estimate the pose (e.g., if the image contains blur, large specular reflections, etc., the confidence is low) and / or the confidence of the pose estimation method itself and / or the quality of the alignment with the coordinate space of the 3D image data. For example, in a neural network-based approach, a discard layer in the network can be used to evaluate the confidence of the pose estimate, which randomly discards the output of a specified number of nodes in the network. At inference, the pose estimate can be calculated multiple times, which will result in slightly different pose estimates (due to discarding). The variance in these estimates can be used to estimate the confidence of the network (i.e., low variance means high confidence, and high variance means low confidence). Confidence estimates can also be based on residual errors from alignment.

[0099] As an example effect, the tool can be navigated to the target area of ​​interest without fluoroscopic guidance. By rendering what can be observed from the tool tip in CT space, the reliance on fluoroscopy is minimized. The use of image-based tool tracking also eliminates the reliance on expensive EM tracking or shape sensing tools.

[0100] In one aspect, a system is provided that observes a tool in the field of view of a bronchoscope (or other navigated interventional imaging device) tracked in the coordinate space of a preoperative CT image (or other 3D coordinate space). The pose of the visible tool is estimated in the bronchoscope coordinate frame. Bronchoscope tracking and tool pose estimation are then used to render a tool-centric view in the preoperative CT coordinate space to enable downstream tasks such as determining whether the tool can be navigated in a target direction, or whether tool placement is optimal for a biopsy or other task, etc.

[0101] As an effect, the system can render the view from the perspective of a distant tool that does not have a camera.

[0102] In an example, an interventional imaging device having an RGB camera is provided that can be navigated into a patient's body to image and / or access a region of interest, such as an endoscope, bronchoscope, or the like.

[0103] In addition, preoperative or intraoperative 3D images of the patient's anatomical structure are provided, such as CT images, CBCT images, etc.

[0104] In addition, a pre-processing controller is provided that performs an alignment or registration between the interventional imaging device and the 3D image of the patient's anatomy. This may be accomplished using any of the following methods:

[0105] In an option, the bronchoscope visible in an intraoperative 3D image acquisition (e.g., tomosynthesis image, CBCT, etc.) is segmented. The segmented bronchoscope serves as the initial pose of the bronchoscope and subsequent bronchoscope tracking is performed using relative camera pose estimation from video-based techniques or tracking devices (e.g., EM, shape sensing, etc.). The relative pose estimate is continuously added to the initial bronchoscope pose to track the bronchoscope in a 3D coordinate frame.

[0106] In another option, the bronchoscope visible in the intraoperative fluoroscopic image acquisition (e.g., one X-ray image or multiple X-ray images from multiple viewpoints, tomosynthesis, CBCT, etc.) is segmented and the intraoperative image is registered with the preoperative CT image. This initializes the segmented bronchoscope in CT space, and then bronchoscope tracking can be performed as described above.

[0107] In another option, the camera pose of the interventional imaging device (e.g., bronchoscope, endoscope, etc.) is estimated, and the 3D structures visible in the interventional imaging device are triangulated from the video frames. The registration between the estimated 3D structure and the corresponding structure segmented in CT (or CBCT, etc.) is provided. Segmentation may be required only for structures that appear at the beginning of bronchoscopic tracking (e.g., the main carina in the airway) in order to align and initialize the bronchoscope in CT space. Subsequent bronchoscopic tracking can then be performed as described above. This reduces the need for complete airway segmentation in CT.

[0108] Finally, the visualization controller obtains the estimated tool pose in the bronchoscope coordinate space (from above) and the alignment between the bronchoscope and the CT coordinate space (from above) to visualize the tool in CT space. The scene from the tool's viewpoint can then be rendered and visualized (as in virtual bronchoscopy, but from the tool rather than the bronchoscope's viewpoint). Rendering can be performed by segmenting the airways and rendering the views within the segmentation using techniques such as ray tracing and calculations of ray intersections with the segmented surfaces to generate a rendered scene. Rendering can also be performed without segmentation using volume rendering techniques that rely on voxel intensity or grayscale values ​​to generate a rendered scene. The rendering can be generated using a simulated light source juxtaposed with the virtual camera (i.e., the same pose as the virtual camera). The virtual camera can be placed in the same pose as the bronchoscope to generate a virtual bronchoscope view, or the virtual camera can be placed in the same pose at the tip of the tool to generate a tool-centered view.

[0109] As an option, the post-processing controller performs downstream tasks based on the resulting tool-centric visualization. Downstream tasks may include (but are not limited to): For example, determining whether the tool is positioned in a way that it can navigate in a specific direction. For example, if the tool must be navigated into an airway branch, a viewpoint looking down at the tool may allow the physician to estimate whether pushing the tool further forward will push the tool into the airway branch.

[0110] For example, a determination of whether tool placement is optimal for a particular task such as obtaining a biopsy is provided. For example, if a segmented lung lesion is additionally rendered, visualizing the view from the tool's reference frame can allow the physician to determine whether the tool is pointing toward the lesion and whether the current tool positioning will result in the acquisition of a tissue sample from the lesion.

[0111] Figure 2 An example of a system 50 for navigating in a hollow anatomical structure is shown. The system 50 includes an interventional imaging device 52 configured for insertion into a hollow structure. The system 50 also includes a tool 54 having a tool tip 56, which is configured for insertion into the hollow structure. In addition, the system 50 includes an example of a device 10 for navigating in a hollow anatomical structure according to one of the aforementioned and following examples. In addition, a display 58 is provided. The interventional imaging device provides current image data of a region of interest of a subject. The display 58 shows the generated rendered image.

[0112] Alternatively, the interventional imaging device 52 is provided as a bronchoscope. In the example shown, the interventional imaging device 52 comprises a channel for removably receiving a tool 54.

[0113] As another option, an X-ray imaging system 60 is also provided for generating image data of 3D image data. The X-ray imaging system 60 includes an X-ray source 62 and an X-ray detector 64 mounted to the end of a movably supported C-arm 66. In the example shown, the X-ray imaging system 60 is supported by a ceiling mounted rail structure. In another example, the X-ray imaging system 60 is provided as a mobile X-ray imaging system. A subject support 68 is provided, on which a subject 70 is arranged. A control panel 72 is provided next to the subject support 68.

[0114] The device 10 for navigating in a hollow anatomical structure is data-connected to the X-ray imaging system 60 , as indicated by a first data connection line 74 .

[0115] A console 76 is provided, for example, for controlling various functions of various devices, including a display, keyboard, mouse, tablet computer and control panel. The device 10 for navigating in a hollow anatomical structure is data connected to the console 76 , as shown by a second data connection line 78 .

[0116] In an example, the interventional imaging device 52 is provided as at least one of the following group: an endoscope providing camera image data, a bronchoscope providing camera image data, a colonoscope providing camera image data, an ultrasound transducer device providing ultrasound image data, and an optical coherence tomography device providing optical coherence tomography image data.

[0117] In an option, the tool 54 is provided as at least one of the following group: a biopsy device, a guide wire, a needle, and a brush.

[0118] For example, an endoscope, bronchoscope or colonoscope includes an RGB camera operating with visible light.

[0119] For example, an ultrasound transducer generates ultrasonic waves and receives wave signals reflected by surrounding anatomical structures.

[0120] For example, optical coherence tomography uses low-coherence light waves to capture micrometer resolution images. This results in 2D and 3D images from within optically scattering media such as tissue. Optical coherence tomography is based on low-coherence interferometry. An example of an application is near-infrared light.

[0121] Figure 3 Basic steps of an example of a method 100 for navigating in a hollow anatomical structure are shown. The method 100 includes the following steps: In a first step 102, 3D image data of a hollow structure in a region of interest of a subject is provided. The 3D image data includes a coordinate space. In a second step 106, a current pose of a tool with a tool tip inserted into the hollow structure is provided. In a third step 108, based on an alignment of the tool tip with the coordinate space of the 3D image data, an estimated current pose of the tool tip is converted to the coordinate space of the 3D image data. In a fourth step 110, a rendered image is generated based on the 3D image data, the rendered image showing a scene inside the hollow structure associated with the converted estimated current pose of the tool tip. In a fifth step 112, the rendered image is provided to a user.

[0122] In an option, a further step is provided after the first step 102 and before the second step 106 , in which further step an interventional imaging device is arranged in the hollow structure.

[0123] Figure 4 Shows Figure 3 Another example of the method of providing the following steps includes the following steps:

[0124] The interventional imaging device is arranged 104' in the hollow structure. In the example, this Figure 3 An example is provided after the first step.

[0125] Current image data of the region of interest acquired by the interventional imaging device in the current posture is provided 106 ′. The current image data comprises image data related to a tool having a tool tip inserted into the hollow structure. In an example, this is provided before registration.

[0126] The interventional imaging device in the current pose is registered 114 with the coordinate space of the 3D image data.

[0127] The current pose of the tool tip is estimated 116. In an example, this is provided after the current image data is provided.

[0128] Based on the registration of the interventional imaging device with the coordinate space of the 3D image data, the current pose of the tool tip estimated from the current image data is converted 108' to the coordinate space of the 3D image data. In an example, this is provided after the estimation.

[0129] In addition, the generation Figure 3 The steps for rendering an image and providing the rendered image to Figure 3 Steps for users of the example.

[0130] In one example, a method for navigating in a hollow anatomical structure is provided, the method comprising the following steps: providing 3D image data of a hollow structure in a region of interest of a subject, wherein the 3D image data includes a coordinate space; arranging an interventional imaging device in the hollow structure; providing current image data of the region of interest acquired by the interventional imaging device in a current posture, wherein the current image data includes image data related to a tool with a tool tip inserted into the hollow structure; aligning the interventional imaging device in the current posture with the coordinate space of the 3D image data; estimating the current posture of the tool tip; based on the alignment of the interventional imaging device with the coordinate space of the 3D image data, converting the current posture of the tool tip estimated from the current image data to the coordinate space of the 3D image data; generating a rendered image from the 3D image data, the rendered image showing a scene inside the hollow structure related to the converted estimated current posture of the tool tip; and providing the rendered image to a user.

[0131] In an example of the method, an estimate of a current pose of the tool tip is based on current image data. In an option, a rendered image representing a field of view from the tool tip is generated.

[0132] In an example of the method, in order to register the interventional imaging device with the coordinate space of the 3D image data, a step of estimating a current pose of the interventional imaging device using current image data of the region of interest acquired by the interventional imaging device is provided.

[0133] In an example of the method, the 3D image data is at least one of the following group: CT image data or CBCT image data of the subject. In an option, the current image data comprises at least one of the following group: camera image data from an endoscope or a bronchoscope or a colonoscope, image data from an ultrasound transducer device, or optical coherence tomography image data.

[0134] In an example of the method, in order to align the interventional imaging device with the coordinate space of the 3D image data, the following steps are provided: providing initial 3D image data of the interventional imaging device within a region of interest; segmenting the interventional imaging device in the initial 3D image data to generate an initial pose; tracking the interventional imaging device; and adjusting the initial pose based on the tracking.

[0135] In an example of the method, in order to align the interventional imaging device with the coordinate space of 3D image data, the following steps are provided: providing initial 2D image data of the interventional imaging device within a region of interest; wherein the initial 2D image data includes at least two initial 2D images with different viewing directions; segmenting the interventional imaging device in the at least two initial 2D images; aligning the at least two initial 2D images with the 3D image data; initializing the segmented interventional imaging device within the coordinate space of the 3D image data to provide an initial posture of the interventional imaging device; tracking the interventional imaging device; and adjusting the initial posture of the interventional imaging device based on the tracking.

[0136] In an example of the method, in order to align an interventional imaging device with a coordinate space of 3D image data, the following steps are provided: providing 2D image data of an interventional imaging device within a region of interest, wherein the 2D image data includes at least two 2D images with different viewing directions; estimating a camera pose based on the 2D image data; triangulating 3D structures from the at least two images based on the estimated camera pose and the segmented structures; initially segmenting structures within the 3D image data; aligning the triangulated 3D structures with corresponding structures segmented in the 3D image data; aligning and initializing the interventional imaging device in the coordinate space of the 3D image data; tracking the interventional imaging device; and adjusting the initial pose of the interventional imaging device based on the tracking.

[0137] In an example of the method, tracking is provided by at least one of the group: i) relative camera pose estimation based on a sequence of camera images; and ii) at least one of the group of electromagnetic tracking, robotic manipulation data, and shape sensing.

[0138] In an example of the method, current fluoroscopic image data is provided. An updated estimate of a current pose of the tool tip based on the current fluoroscopic image data is provided.

[0139] In an example of the method, a target lesion is rendered in a rendered image. An adjustable transparency is provided for at least a portion of the structure in the rendered image.

[0140] In an example of the method, a confidence estimate is generated and provided to a user. The confidence estimate relates to at least one of the group of a quality of the image used to estimate the pose, a quality of the pose estimation method, and a quality of the registration.

[0141] Figure 5 A fluoroscopic image 200 is shown with a bronchoscope 202 and a tool 204 inserted through a working channel of the bronchoscope 202. A first arrow 206 indicates the distal end of the bronchoscope 202. An imaging device is mounted at the distal end. A second arrow 208 indicates the distal end of the tool 204.

[0142] Figure 6 An example of a workflow is shown. The bronchoscope provides a bronchoscopic image 210 of a hollow anatomical structure. The bronchoscopic image 210 shows a tool 212 inserted through a working channel of the bronchoscope. The tool 212 with a tool tip 214 further reaches into the hollow anatomical structure. The hollow anatomical structure is shown as having a first path 216 and a second path 218, which are branched and separated by a wall structure 220.

[0143] As a result of the above steps, a first rendered image 222 is provided, which shows the position as seen from the bronchoscope. In addition, a second rendered image 224 is provided, which shows the position as seen from the tool tip 214.

[0144] Figure 7 Another example of a workflow is shown. Figure 6 , the bronchoscope provides a bronchoscopic image 230 of a hollow anatomical structure, as shown in the upper portion of the figure. The bronchoscopic image 230 shows a tool 232 inserted through a working channel of the bronchoscope. The tool 232 with a tool tip 234 further reaches into the hollow anatomical structure. The hollow anatomical structure is shown as having a first path 236 and a second path 238, which are branched and separated by a wall structure 240.

[0145] The center portion of the figure indicates a representation of an anatomical structure 242 (i.e., a segmented airway) into which a bronchoscope 244 is partially inserted. A tool 246 having a distal end extends out of a working channel at the distal end of the bronchoscope 244. Thus, the bronchoscope 244 has a view of the tool 246 with its field of view. As described above, the bronchoscope 244 and the tool 246 are tracked in CT space. The right portion of the figure has two rendered images, the upper right rendering 248 showing a bronchoscope view and the lower right rendering 250 showing a rendering centered on the tool. The first dotted line 252 points to the location within the anatomical structure where the rendered image 248 is provided. The second dotted line 254 points to the location within the anatomical structure where the rendered image 250 is provided.

[0146] In an example, a computer program includes instructions, and when the program is executed by a computer, the computer performs the method of the above example.

[0147] In an example, a computer program or a program element for controlling an apparatus according to one of the above examples is provided, which program or program element, when executed by a processing unit, is adapted to perform the method steps of one of the above method examples.

[0148] In an example, a computer readable medium storing the computer program of the foregoing examples is provided.

[0149] The term "subject" may also be referred to as an individual. A "subject" may also be referred to as a patient, although it is noted that this term does not indicate whether the subject actually has any disease or illness.

[0150] In a further exemplary embodiment of the present invention, a computer program or a computer program element is provided, which is characterized by being adapted to execute the method steps of the method according to one of the preceding embodiments on a suitable system.

[0151] Therefore, the computer program element can be stored on a computer unit or distributed on multiple computer units, which can also be a part of an embodiment of the present invention. The computing unit can be suitable for executing or inducing the steps of the above method. In addition, it can be suitable for operating the components of the above device. The computing unit can be suitable for automatic operation and / or execution of user's commands. The computer program can be loaded into the working memory of the data processor. Therefore, the data processor can be equipped to perform the method of the present invention.

[0152] Aspects of the present invention can be implemented in a computer program product, which can be a set of computer program instructions stored on a computer-readable storage device that can be executed by a computer. Instructions of the present invention can be any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs) or Java classes. Instructions can be provided as complete executable programs, partial executable programs, modifications (e.g., updates) to existing programs or extensions (e.g., plug-ins) of existing programs. In addition, partial processing of the present invention can be distributed on multiple computers or processors.

[0153] As described above, a processing unit (e.g., a controller) implements a control method. The controller can be implemented in a variety of ways using software and / or hardware to perform the various functions required. A processor is an example of a controller that uses one or more microprocessors that can be programmed using software (e.g., microcode) to perform the required functions. However, a controller may be implemented with or without a processor, and may also be implemented as a combination of dedicated hardware for performing some functions and a processor (e.g., one or more programmed microprocessors and associated circuits) for performing other functions.

[0154] Examples of controller components that may be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0155] This exemplary embodiment of the invention comprises a computer program which uses the invention from the outset and a computer program which turns an existing program into a program which uses the invention by updating.

[0156] Furthermore, the computer program element may be able to provide all necessary steps to fulfill the procedures of an exemplary embodiment of the method described above.

[0157] According to another exemplary embodiment of the present invention, a computer readable medium, such as a CD-ROM, is provided, wherein a computer program element is stored on the computer readable medium, the computer program element being described by the previous section. The computer program can be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0158] However, the computer program may also be provided over a network such as the World Wide Web and may be downloaded from such a network into a working memory of a data processor. According to a further exemplary embodiment of the invention, a medium for making a computer program element available for downloading is provided, the computer program element being arranged to perform a method according to one of the previously described embodiments of the invention.

[0159] It must be noted that embodiments of the present invention are described with reference to different subject matters. Specifically, some embodiments are described with reference to method type claims, while other embodiments are described with reference to device type claims. However, those skilled in the art will appreciate from the above and following descriptions that, unless otherwise stated, any combination of features related to different subject matters, in addition to any combination of features belonging to one type of subject matter, is also considered to be disclosed with the present application. However, all features can be combined together to provide synergistic effects, rather than just a simple addition of features.

[0160] Although the present invention has been shown and described in detail in the drawings and the foregoing description, such illustration and description should be considered illustrative or exemplary rather than restrictive. The present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments may be understood and effected by those skilled in the art in practicing the claimed invention by studying the drawings, the disclosure, and the dependent claims.

[0161] In the claims, the word "comprising" does not exclude other elements or steps, and the attributive "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items repeatedly cited in the claims. The mere fact that certain measures are repeatedly cited in mutually different dependent claims does not mean that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A device (10) for navigating in a hollow anatomical structure, comprising: Data input unit (12); Data processor (14); as well as Output interface (16); wherein the data input section is configured to: provide 3D image data of a hollow structure in a region of interest of a subject, wherein the 3D image data includes a coordinate space; provide a current posture of a tool having a tool tip inserted into the hollow structure; wherein the tool does not have imaging capabilities; wherein the data processor is configured to: transform the estimated current posture of the tool tip into the coordinate space of the 3D image data based on the registration of the tool tip with the coordinate space of the 3D image data; and generate a rendered image based on the 3D image data, the rendered image showing a scene inside the hollow structure associated with the transformed estimated current posture of the tool tip; and Wherein, the output interface is configured to provide the rendered image to a user.

2. The device according to claim 1, in: in, The data input section is configured to: provide current image data of the region of interest acquired by an interventional imaging device arranged in the hollow structure in a current posture, wherein the current image data includes image data related to a tool having a tool tip inserted into the hollow structure; and Wherein, the data processor is configured to: align the interventional imaging device in the current posture with the coordinate space of the 3D image data; estimate the current posture of the tool tip visible in the current image data; and based on the alignment of the interventional imaging device with the coordinate space of the 3D image data, convert the current posture of the tool tip estimated based on the current image data to the coordinate space of the 3D image data.

3. The device according to claim 1 or 2, wherein: For the registration of the interventional imaging device with the coordinate space of the 3D image data, the data processor is configured to estimate a current pose of the interventional imaging device using the current image data of the region of interest acquired by the interventional imaging device.

4. The device according to any one of the preceding claims, wherein: The 3D image data is at least one of the following group: CT image data, CBCT image data, or MRI image data of the subject; and The current image data comprises at least one of the following groups: camera image data from an endoscope or a bronchoscope, image data from an ultrasound transducer device, or optical coherence tomography image data.

5. The device according to any one of the preceding claims, wherein: For the registration of the interventional imaging device with the coordinate space of the 3D image data, the data processor is configured to: provide initial 3D image data of the interventional imaging device within the region of interest; segment the interventional imaging device in the initial 3D image data to generate an initial pose; track the interventional imaging device; and adjust the initial pose based on the tracking.

6. The device according to any one of the preceding claims, wherein: For the registration of the interventional imaging device with the coordinate space of the 3D image data, the data processor is configured to: provide initial 2D image data of the interventional imaging device within the region of interest, wherein the initial 2D image data includes at least one initial 2D image; segment the interventional imaging device in the at least one initial 2D image; align the at least one initial 2D image with the 3D image data; initialize the segmented interventional imaging device within the coordinate space of the 3D image data to provide an initial pose of the interventional imaging device; track the interventional imaging device; and adjust the initial pose of the interventional imaging device based on the tracking.

7. The device according to any one of the preceding claims, wherein: For the registration of the interventional imaging device with the coordinate space of the 3D image data, the data processor is configured to: provide 2D image data from the interventional imaging device within the region of interest, wherein the 2D image data includes at least two 2D images with different viewing directions; estimate a camera pose based on the 2D image data; triangulate 3D structures from the at least two images based on the estimated camera pose and features visible in the at least two images; initially segment structures within the 3D image data; align the triangulated 3D structures with corresponding structures segmented in the 3D image data; align and initialize the interventional imaging device in the coordinate space of the 3D image data; track the interventional imaging device; and adjust the initial pose of the interventional imaging device based on the tracking.

8. The device according to any one of claims 5 to 7, wherein: The data processor is configured to provide the tracking by at least one of the following group: i) relative pose estimation based on image sequences; and ii) At least one of the group of electromagnetic tracking, robotic manipulation data, and shape sensing.

9. The device according to any of the preceding claims, wherein: Current fluoroscopic image data is provided; and Wherein the data processor is configured to provide an updated estimate of a current pose of the tool tip based on the current fluoroscopic image data.

10. The device according to any of the preceding claims, wherein: The data processor is configured to: generate a confidence estimate; wherein the confidence estimate relates to at least one of the following group: a quality of an image used to estimate the pose, a quality of the pose estimate, and a quality of the registration; and Wherein, the output interface is configured to provide the confidence estimate to the user.

11. A system (50) for navigating in a hollow anatomical structure, comprising: an interventional imaging device (52) configured for insertion into the hollow structure; a tool (54) having a tool tip (56) configured for insertion into the hollow structure; as well as Device (10) for navigating in a hollow anatomical structure according to one of the preceding claims; and Display (58); wherein the interventional imaging device provides current image data of a region of interest of a subject; and Therein, the display shows the generated rendering image.

12. A method (100) for navigating in a hollow anatomical structure, comprising the steps of: providing (102) 3D image data of a hollow structure in a region of interest of a subject, wherein the 3D image data includes a coordinate space; providing (106) a current pose of a tool having a tool tip inserted into the hollow structure, wherein the tool does not have imaging capabilities; transforming (108) the estimated current pose of the tool tip to the coordinate space of the 3D image data based on the registration of the tool tip with the coordinate space of the 3D image data; generating (110) a rendered image based on the 3D image data, the rendered image showing a scene inside the hollow structure associated with the converted estimated current pose of the tool tip; and The rendered image is provided (112) to a user.

13. The method according to claim 12, comprising the steps of: placing (104') an interventional imaging device in the hollow structure; providing (106') current image data of the region of interest acquired by the interventional imaging device in a current posture, wherein the current image data includes image data related to a tool having a tool tip inserted into the hollow structure; aligning the interventional imaging device in the current pose with the coordinate space of the 3D image data (114); estimating (116) the current pose of the tool tip; and Based on the registration of the interventional imaging device with the coordinate space of the 3D image data, a current pose of the tool tip estimated from the current image data is transformed (108') into the coordinate space of the 3D image data.

14. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method according to claim 12 or 13.

15. A computer readable medium storing a computer program according to claim 14.

Images