Guided interventional imaging device

Abstract

The invention relates to a device for guiding an interventional imaging device. In order to minimize the use of modalities such as X-ray or CT, a device (10) for guiding an interventional imaging device is provided. The device comprises a data input (12), a data processor (14) and an output interface (16). The data input is configured to provide first image data as first data from a first imaging device. The first image data includes a representation of an interventional imaging device inserted within a vascular structure of a subject. The first image data includes image data related to a first point in time. The data input is further configured to provide second data relating to movement of the interventional imaging device. The second data relates to a first point in time and at least a second point in time. The data processor is configured to estimate a pose of an interventional imaging device in the first image data. The data processor is further configured to track relative motion of the interventional imaging device based on the second data. The data processor is further configured to calculate an updated pose estimate of the interventional imaging device based on the estimated pose and the tracked relative motion. The data processor is further configured to generate an updated indicator of the interventional imaging device based on the calculated updated pose estimate. The data processor is further configured to enhance the first image data with the updated indicator. The output interface is configured to provide enhanced first image data. In an example, the first imaging device is an X-ray imaging device, and the second imaging device is a bronchoscopy imaging device.

Description

Technical Field

[0001] The present invention relates to guiding an interventional imaging device, and in particular to a device for guiding an interventional imaging device, a system for guiding an interventional imaging device, and a method for guiding an interventional imaging device. Background Art

[0002] In minimally invasive surgery, imaging devices are used that can be inserted into the subject's body to help navigate to and / or image a region of interest (ROI). Examples include devices with visible light (RGB) cameras (endoscopes, bronchoscopes, etc.) and other imaging modalities such as endobronchial ultrasound (EBUS), intravascular ultrasound (IVUS), optical coherence tomography (OCT), etc. They can be used in conjunction with other interventional imaging modalities like X-ray, tomosynthesis, cone beam computed tomography (CBCT), etc., which can provide further support in navigation. However, it has been shown that using other data sources like external X-ray imaging systems to provide information about inserted interventional devices increases exposure to ionizing radiation. Summary of the invention

[0003] Therefore, it may be desirable to minimize the use of these modalities.

[0004] The objects of the invention are solved by the subject matter of the independent claims; further embodiments are incorporated in the dependent claims. It should be noted that the following described aspects of the invention also apply to a device for guiding an interventional imaging device, a system for guiding an interventional imaging device and a method for guiding an interventional imaging device.

[0005] According to the present invention, a device for guiding an interventional imaging device is provided. The device includes a data input unit, a data processor and an output interface. The data input unit is configured to provide first image data as first data from a first imaging device. The first image data includes a representation of an interventional imaging device inserted into a vascular structure of an object, and the first image data includes image data related to a first time point. The data input unit is also configured to provide second data related to the movement of the interventional imaging device. The second data is related to the first time point and at least a second time point. The data processor is configured to estimate the posture of the interventional imaging device in the first image data. The data processor is also configured to track the relative motion of the interventional imaging device based on the second data. The data processor is also configured to calculate an updated posture estimate of the interventional imaging device based on the estimated posture and the tracked relative motion. The data processor is also configured to generate an updated indicator of the interventional imaging device based on the calculated updated posture estimate. And the data processor is configured to enhance the first image data with the updated indicator. The output interface is configured to provide enhanced first image data.

[0006] In an example use case, the first image data may be an X-ray image or a fluoroscopic sequence acquired from an X-ray imaging system, and the second image data may be a bronchoscopic image acquired from a bronchoscope as the bronchoscope is navigated within the pulmonary airway anatomy. The imaging system is used in conjunction during an endobronchial procedure in which an interventional pulmonologist navigates the bronchoscope to a lung lesion under fluoroscopic guidance for biopsy or treatment of the lesion. As an effect of the present invention, it is beneficial to the subject, the interventional pulmonologist, and other staff that the use of the bronchoscope in navigating to the lesion is maximized and the use of fluoroscopy is minimized.

[0007] As an advantage, video-based bronchoscope tracking does not require any additional hardware or workflow changes. In contrast, techniques like shape sensing and electromagnetic navigation, which can also be used to track bronchoscopes, require additional hardware, such as electromagnetic field generators, tools for electromagnetic tracking, or shape sensing catheters, etc., which can be expensive and may add additional workflow steps to incorporate into the process. As another advantage, the need for an interventional pulmonologist or other user to precisely locate their position in the patient's anatomy by viewing only the bronchoscopic image is avoided, which can be confusing to the user. These advantages also apply to other endoscopy, colonoscopy, and other video-based procedures.

[0008] According to an example, the second data is image data from a second imaging device provided by the interventional imaging device. In other words, the interventional imaging device includes the second imaging device. The image data from the second imaging device may also be referred to as second image data. The second image data includes a representation of the interior of the vascular structure. The data processor is configured to track the relative movement of the interventional imaging device within the vascular structure based on the second image data.

[0009] According to an example, in order to estimate a posture of an interventional imaging device in first image data, a data processor is configured to use an image of second image data, i.e., one or more images of multiple images of the second image data, for generating a posture estimate relating to at least a first time point for adjusting the estimated posture of the interventional imaging device in the first image data.

[0010] The second image (eg, a bronchoscopic image) is used to estimate the pose based on the image content by, for example, an image processing or image analysis procedure.

[0011] According to an example, the second image data comprises a second image stream.The data processor is configured to provide tracking of relative motion of the interventional imaging device for consecutive images of the second image stream.

[0012] As an example, during an endobronchial biopsy procedure, both bronchoscopy images and fluoroscopic images are used to navigate the physician to the lung lesion.

[0013] As an example, a machine learning system is provided that can track a camera through space with reasonable accuracy based solely on video captured by the camera (such as video captured by a bronchoscope).

[0014] According to an example, the data processor is further configured to calculate a trajectory of the interventional imaging device based on the updated pose estimate. In a first option, the data processor is configured to: i) generate a projection of the trajectory of the interventional imaging device based on the calculated trajectory of the interventional imaging device and the updated pose. The data processor is further configured to enhance the first image data based on the projection of the trajectory of the interventional imaging device to provide updated virtual first image data. In a second option, the data processor is configured to ii) project the calculated trajectory onto the first image data.

[0015] According to an example, the data processor is further configured to generate a realistic composite image depicting an updated pose of the interventional imaging device based on the first image data at the first time point and the updated pose of the interventional imaging device at the second time point using the trained generative neural network.

[0016] According to an example, the data processor is further configured to provide a confidence estimate related to the relative motion estimate from the second image data.The output interface is configured to provide a confidence indicator to a user.

[0017] According to the present invention, a system for guiding an interventional imaging device is also provided. The system comprises a first data arrangement device, the first data arrangement device comprising a first imaging device. The system further comprises a second data arrangement device and a device for guiding an interventional imaging device according to any of the foregoing examples. The first imaging device is configured to generate first image data as first data. The second data arrangement device is configured to generate second data.

[0018] According to an example, an interventional imaging device is provided. In one option, the second data arrangement apparatus is provided as a second imaging device provided by the interventional imaging device.

[0019] According to an example, the first imaging device is provided as an X-ray imaging device. The second imaging device is provided as at least one of the group consisting of: a bronchoscope, an endoscope, a colonoscope, an intravascular ultrasound, an intracardiac echocardiogram, an endobronchial ultrasound or a radial endobronchial ultrasound and an optical coherence tomography.

[0020] According to the present invention, a method for guiding an interventional imaging device is also provided. The method comprises the following steps:

[0021] First image data is provided as first data from a first imaging device. The first image data comprises a representation of an interventional imaging device inserted within a vascular structure of a subject. The first image data comprises image data relating to a first point in time.

[0022] A pose of the interventional imaging device in the first image data is estimated.

[0023] Second data related to movement of the interventional imaging device is provided. The second data is related to the first time point and at least a second time point.

[0024] Relative motion of the interventional imaging device is tracked based on the second data.

[0025] An updated pose estimate of the interventional imaging device is calculated based on the estimated pose and the tracked relative motion.

[0026] An updated indicator of the interventional imaging device is generated based on the calculated updated pose estimate.

[0027] The first image data is enhanced using the updated indicator.

[0028] Enhanced first image data is provided.

[0029] According to one aspect, camera tracking or camera pose estimation is combined with the most recent fluoroscopic images acquired during a procedure to enhance the fluoroscopic images with updated bronchoscope poses as the bronchoscope is navigated through the patient's anatomy. This enhanced view can help orient physicians and reduce their reliance on fluoroscopic images to orient themselves, thus reducing exposure to radiation. Although an overview and description related to lung lesions and bronchoscopic navigation of these lesions is provided, the solution is also applicable to other endoscopic procedures using fluoroscopy and procedures using other devices (EBUS, IVUS) in conjunction with fluoroscopy.

[0030] According to one aspect, a setup is described that provides an imager with a larger field of view (e.g., X-ray based imaging) plus an interventional device with an imager that has a small or partial field of view within the lumen while the device is being navigated. The redundancy of constantly visualizing the same anatomical structure using two imaging techniques is largely addressed by utilizing a limited field of view to track the camera on the device and thus prevent the user from becoming disoriented in complex anatomical structures.

[0031] As an effect, for example, in case of discontinuous fluoroscopic image acquisition, the pose of the bronchoscope is updated on the fluoroscopic image.

[0032] This has the effect of avoiding over-reliance on fluoroscopy and omitting the need for specialized tracked tools during interventional procedures. In addition to reducing radiation exposure, reducing reliance on fluoroscopy during interventional procedures can also achieve reduced procedure times, because, for example, in a lung lesion biopsy procedure, the user may not spend as much time repositioning the X-ray imaging system to acquire fluoroscopic images. Reducing reliance on expensive tracking equipment and tracked tools also makes the solution more accessible and, therefore, enables safer procedures to be available for more patients.

[0033] The method may be used with any interventional imaging system including but not limited to fixed and mobile X-ray imaging systems during procedures with navigated imaging devices including but not limited to bronchoscopes, endoscopes, etc.

[0034] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Exemplary embodiments of the present invention will be described below with reference to the accompanying drawings:

[0036] Figure 1 An example of a device for guiding an interventional imaging device is schematically shown.

[0037] Figure 2 An example of a system for guiding an interventional imaging device is shown.

[0038] Figure 3 Basic steps of an example of a method for guiding an interventional imaging device are shown.

[0039] Figure 4 An example of a workflow for guiding an interventional imaging device is shown.

[0040] Figure 5 An example of another workflow for guiding an interventional imaging device is shown. DETAILED DESCRIPTION

[0041] Certain embodiments will now be described in more detail with reference to the accompanying drawings. In the following description, similar reference numerals are used for similar elements even in different drawings. Matters defined in the description, such as detailed construction and elements, are provided to aid in a comprehensive understanding of the exemplary embodiments. Moreover, well-known functions or constructions are not described in detail because they would obscure the embodiments with unnecessary detail. In addition, when preceding a list of elements, expressions such as "at least one of..." modify the entire list of elements and do not modify the individual elements of the list.

[0042] Figure 1An example of a device 10 for guiding an interventional imaging device is schematically shown. The device 10 includes a data input 12, a data processor, and an output interface 16. The data input 12 is configured to provide first image data as first data from a first imaging device. The first image data includes a representation of an interventional imaging device inserted into a vascular structure of an object. In addition, the first image data includes image data related to a first time point. The data input 12 is also configured to provide second data related to movement of the interventional imaging device. The second data is related to the first time point and at least a second time point. The data processor 14 is configured to estimate the posture of the interventional imaging device in the first image data. The data processor 14 is also configured to track the relative motion of the interventional imaging device based on the second data. The data processor 14 is also configured to calculate an updated posture estimate of the interventional imaging device based on the estimated posture and the tracked relative motion. The data processor 14 is also configured to generate an updated indicator of the interventional imaging device based on the calculated updated posture estimate. The data processor 14 is also configured to enhance the first image data with the updated indicator. The output interface 16 is configured to provide the enhanced first image data.

[0043] The data input 12, the data processor 14 and the output interface 16 may be provided in a common structure, such as a common housing, as indicated by the frame 18, or even in an integrated manner. In another option (not shown), they are provided as separate components or units.

[0044] A first arrow 20 indicates a data supply, ie a supply of first image data and second data, to the data input 12. A second arrow 22 indicates a data supply, ie a supply of enhanced first image data, from the output interface 16. The data supply may be provided on a wired or wireless basis.

[0045] In the example, as an option, a display 24 is provided to present the enhanced first image. The display 24 is data connected to the output interface 16.

[0046] The term "estimate pose" relates to evaluating or determining the position and orientation of an interventional device, preferably a distal end of the interventional device, arranged within a vascular structure of a subject. As an example, the position and orientation are determined relative to a projection of the first image data.

[0047] The term "relative movement" relates to a change in the position and / or orientation of an interventional device arranged within a vascular structure relative to its position and / or orientation at a first point in time.

[0048] The term "updated pose estimate" relates to a further evaluation or determination of the pose (ie position and orientation) of the interventional device, preferably the distal end thereof.

[0049] The term "updated indicator" relates to an indicator that reflects a further evaluation or determination.The indicator may be provided as a graphical element or illustration or other visual module presented to a user.

[0050] The term "enhancement" relates to providing additional information within a corresponding image. The image content is thus enhanced. The enhanced image includes more content than the image before enhancement. Although the original image (i.e., the non-enhanced image) includes image data as seen by the corresponding imaging device, the enhanced image provides additional content presented within the original image.

[0051] The term "data input" relates to providing or supplying data for a data processing step. The data input 12 may also be referred to as an image data input. The data input 12 may also be referred to as a data supply, an image data supply, an image input, an input unit or simply an input. In an example, the image data input 12 may be data connected to an imaging source arrangement. In an example, the data input 12 may be data connected to a data storage device that has stored image data.

[0052] The term "data processor" relates to a part of a processor or processor arrangement that is provided to carry out a computational step using data supplied by a data input. The data processor 14 may also be referred to as a data processing arrangement, a processor unit or a processor. In the example, the data processor 14 is data connected to a data input and an output interface.

[0053] The term "output interface" refers to an interface for providing processed or calculated data for further purposes. The output interface 16 may also be referred to as an output portion or output unit. In an example, the output interface 16 may be data-connected to a display arrangement or display device. In another example, the output interface 16 is data-connected to a display.

[0054] In an example, for the updated indicator, a directional indicator is provided. For example, the directional indicator is provided as a shadow.

[0055] In an example, the first image data is provided as 2D image data. As an example, enhancing the first image data results in a 2D enhancement.

[0056] In an example, the second data is position and / or orientation data related to the movement of the interventional imaging device. In other words, the second data is posture data. The data processor 14 is configured to track the relative movement of the interventional imaging device within the vascular structure based on the position and / or orientation data. In an option, the position and / or orientation data is provided as tracking data, such as electromagnetic tracking. In another option, the position and / or orientation data is provided as shape sensing data of the interventional imaging device and advancement information of the interventional imaging device.

[0057] In an option, external hardware (e.g., electromagnetic "EM" tracking or shape sensing) is used to track the navigated imaging device, so that no image processing is required to estimate the pose of the device (or its distal end). In this case, the controller needs to extract the device pose from the external hardware when acquiring the fluoroscopic images in order to perform any correction of the two estimated poses. The remaining steps are performed as described above.

[0058] In another example, external tracking may be used in conjunction with image processing to estimate the pose of the interventional device.

[0059] In another example, the second data is image data from a second imaging device, i.e., second image data, which is provided by an interventional imaging device. The second image data includes a representation of the interior of the vascular structure. The data processor 14 is configured to track the relative movement of the interventional imaging device within the vascular structure based on the second image data.

[0060] In the case of imaging devices that capture RGB (red-blue-green) images, this can be achieved using any of the various methods explored in the art that estimate the (absolute or relative) pose or motion of the device based on features or changes in features observed through an image sequence. For example, in the case of bronchoscope tracking, traditional methods such as structure from motion (SfM) methods or simultaneous localization and mapping (SLAM) methods can be used as well as newer deep learning-based methods for camera tracking. Similar methods can also be used to track other navigated imaging devices. In the case of ultrasound-based imaging devices, tracking may require branch detection or other methods to roughly locate the device within the patient's anatomy.

[0061] As an example, internal or navigated interventional imaging devices may acquire images during interventional procedures such as: endoscopes, bronchoscopes, intravascular ultrasound (IVUS), intracardiac echocardiography (ICE), endobronchial ultrasound (EBUS), or radial endobronchial ultrasound (R-EBUS), etc.

[0062] As an option, a correction loop between processes is provided.

[0063] In the first example, the data processor 14 is further configured to provide computation of an updated pose estimate comprising correction of an out-of-plane pose estimate from the first image data using the pose estimate from the second image data. In the second example, additionally or alternatively, the data processor 14 is further configured to provide computation of an updated pose estimate comprising correction of an in-plane pose estimate from the second image data using the pose estimate from the first image data.

[0064] In an example, in order to estimate the posture of the interventional imaging device in the first image data, the data processor 14 is configured to use one or more of the images of the second image data provided as a plurality of images. One or more of the plurality of images are used to generate a posture estimate related to at least a first time point. Therefore, one or more of the plurality of images are used to adjust the estimated posture of the interventional imaging device in the first image data. Therefore, the second data is used to adjust or update the estimation performed using the first image. In other words, information from the second image domain is transferred to the first image domain. This information transfer compensates for the lack of corresponding information in the first image data.

[0065] In an example, the images of the second image stream show identifiable anatomical structures and a viewing direction may be estimated for the images so that the viewing direction may be transferred to the first image.

[0066] In an example, the second image data comprises a second image stream.The data processor 14 is configured to provide tracking of relative motion of the interventional imaging device for successive images of the second image stream.

[0067] In an option, a visualization controller is provided that is configured to continuously display an updated bronchoscope trajectory over the most recent fluoroscopic image.

[0068] In an example, the data processor 14 is further configured to calculate a trajectory of the interventional imaging device based on the updated pose estimate. In an option, the data processor 14 is further configured to: generate a projection of the trajectory of the interventional imaging device based on the calculated trajectory of the interventional imaging device and the updated pose; and enhance the first image data based on the projection of the trajectory of the interventional imaging device to provide updated virtual first image data. In another option, the data processor 14 is further configured to project the calculated trajectory onto the first image data (see also Figure 5 ).

[0069] In addition to or instead of calculating the updated pose estimate, providing for calculation of the trajectory. In addition to or instead of generating the updated indicator, providing for generation of a projection of the interventional imaging device. In addition to or instead of enhancing the first image data with the updated indicator, providing for superposition of the projection as an enhancement.

[0070] The updated virtual first image data may also be referred to as enhanced first image data or artificial first image data.

[0071] In an option, enhancements are provided including superimposing lines or other shapes on top of the first image, and modifying the first image using a generative neural network, such as a generative adversarial network (see below).

[0072] In an example, the data processor is configured to: generate a projection of the interventional imaging device based on the calculated trajectory and the updated pose of the interventional imaging device; and superimpose the projection of the interventional imaging device on the first image data to provide updated virtual first image data (see also Figure 5 ).

[0073] According to the example (see also Figure 5 ), the data processor 14 is further configured to use the trained generative neural network to generate a realistic composite image depicting the updated posture of the interventional imaging device based on the first image data at the first time point and the updated posture of the interventional imaging device at the second time point. The realistic composite image is then provided, for example, presented to a user.

[0074] In an option, a generative neural network (e.g., a generative adversarial network (GAN)) is used to generate fluoroscopic images with updated bronchoscope poses as the bronchoscope is navigated. This can be achieved by performing the same steps described above with the addition of a trained GAN inserted after the projection taking the superimposed augmented fluoroscopic image with the updated bronchoscope pose and producing a new fluoroscopic image with realistic bronchoscope rendering at its updated pose (e.g., Figure 5 When the next fluoroscopic image is acquired, the rendering may be updated as described above. This avoids or at least reduces the distraction that the overlay may cause to the user, i.e., avoids distracting the user from important anatomical features in the fluoroscopic image.

[0075] In an example, the first image data provides a first field of view and the second image data provides a second field of view. The first field of view is larger than the second field of view. As an example, the first field of view is at least twice as large as the second field of view.

[0076] In an example, the first image data comprises X-ray image data.The second image data comprises at least one of the group consisting of: optical camera image data, ultrasound image data, and optical coherence tomography image data.

[0077] In an example, the data processor 14 is further configured to provide a confidence estimate related to the relative motion estimate from the second image data.The output interface 16 is configured to provide a confidence indicator to a user.

[0078] In an option, a confidence estimate is provided along with the pose estimate from the bronchoscopy. The confidence estimate can be made using any method available in the art, and can be based on, for example, the quality of the image used to estimate the pose, e.g., low confidence if the image contains blur, specular reflections, etc., and / or the confidence of the pose estimation method itself. For example, in a neural network-based approach, the confidence of the pose estimate can be evaluated using a dropout layer in the network that randomly drops the output of a specified number of nodes in the network. When the relative pose estimate between a particular pair of images is then calculated multiple times, the network produces slightly different results due to the influence of slightly different nodes on the output. The mean and standard deviation of the results can be calculated. If the network is confident in its pose estimate, the standard deviation will be small, and when the network is less confident, the standard deviation will be large. If the confidence in the relative pose estimate between two frames is low, the pose estimate can be replaced by an extrapolation from the previous few poses. This low confidence or replacement of the pose estimate can be visualized with a different color or transparency, and can be an indicator to the user as to when a new fluoroscopic image may need to be acquired.

[0079] In the example, the data processor 14 is further configured to track the relative movement of the interventional imaging device within the vascular structure based on the additional tracking data. As an option, the tracking of the relative movement is made by referring to other data in addition to the imaging data plus the imaging data as the second data. As another option, the tracking of the relative movement is made by referring only to other data in addition to the imaging data (i.e., without the imaging data as the second data).

[0080] In the example, in order to display the device at the starting time point t0, an X-ray image is used. At another time point t1 or another further time point t n , providing bronchoscopy information and adding the derived positions to the X-ray images.

[0081] As an example, the fluoroscopic image processing controller acquires the most recent fluoroscopic image or sequence of images that visualizes the navigated imaging device within the patient's anatomy and estimates the pose of the tip (or distal end) of the navigated imaging device. This can be accomplished using any of the various methods explored in the art that estimate the 2D or 3D pose of the imaged device so that its projection matches that seen in the fluoroscopic image(s).

[0082] In some cases, a 3D model of the device is known and is deformed and fitted to a configuration that matches its projection in fluoroscopy. In other cases, the 3D model may not be known and the pose of the skeletonized distal end is estimated based on image features.

[0083] The image processing controller may use machine learning or deep learning techniques.Since the fluoroscopic images are 2D projection images, the estimated pose may contain inaccuracies.

[0084] As an example, the image processing controller or data processor 14 acquires a stream of frames produced by a navigated imaging device, such as a bronchoscope, as it is navigated through the patient's anatomy and estimates the relative pose or motion produced by the navigated device.

[0085] In an example, the controller is provided as data processor 14. The controller is configured to perform the following tasks (referring only to bronchoscopy as an example below):

[0086] The pose of the bronchoscope in the most recent fluoroscopic image acquired at t=t0 is estimated. As an example, a fluoroscopic image processing controller is provided as follows;

[0087] The relative motion of the bronchoscope between consecutive frames is estimated. As an example, an image processing controller is provided as follows.

[0088] The two estimates are used together to provide correction of the out-of-plane pose estimate from fluoroscopy using the pose estimated from bronchoscopy, and correction of the in-plane pose estimate from bronchoscopy using the pose estimated from fluoroscopy (using the Figure 4 (indicated by the double-headed arrow in the second panel from top to bottom).

[0089] Estimates of the relative motion of the bronchoscope between consecutive frames are made continuously as the bronchoscope moves through its position at t=t0, and these estimates are sequentially added to the pose of the device at t=t0 to compute the trajectory of the bronchoscope (in Figure 4 (shown in the third panel from top to bottom).

[0090] The calculated trajectory is projected onto the fluoroscopic image plane to augment the most recent fluoroscopic image with the updated bronchoscope pose.

[0091] If or when the next fluoroscopic image is acquired at t=t0, the process can be repeated: the estimated bronchoscope pose is combined with the fluoroscopic and bronchoscopic examinations, and the new fluoroscopic image is updated with the bronchoscope trajectory as the bronchoscope continues to move (in Figure 4 ).

[0092] Figure 2An example of a system 100 for guiding an interventional imaging device is shown. The system 100 includes: a first data arrangement device 102, which includes a first imaging device 104; and a second data arrangement device 106. In addition, the system 100 includes an example of a device 10 for guiding an interventional imaging device according to any of the foregoing and following examples. The first imaging device 104 is configured to generate first image data as first data. The second data arrangement device 106 is configured to generate second data.

[0093] In one option, the second data placement means 106 comprises a second imaging device 108. In another option (not shown), the second data placement means 106 comprises a tracking data device or a positioning data device.

[0094] In the example, an interventional imaging device 110 is provided. Alternatively, the second data arranging apparatus 106 provided as the second imaging device 108 is provided by the interventional imaging device, ie is installed on the interventional imaging device.

[0095] In the example, Figure 2 As shown, the first imaging device 104 is provided as an X-ray imaging device 111 having an X-ray source 112 and an X-ray detector 114 mounted to a movably supported C-arm. The second imaging device is provided as a bronchoscope 116. In a further example, the second imaging device is provided as at least one of the group consisting of an endoscope, a colonoscope, an intravascular ultrasound, an intracardiac echocardiogram, an endobronchial ultrasound or a radial endobronchial ultrasound, an optical coherence tomography.

[0096] An object 118, such as a patient, is arranged on an object support 120. A placement device 122 is supported by a ceiling mounted track structure 124. Furthermore, a control interface 126 is arranged beside the object support 120.

[0097] A console 128 is provided in the foreground. The console 128 includes a display, keyboard, mouse, graphic input tablet, control knobs, etc. for providing user interaction and control options. The console is provided for controlling various functions and operations of the system 100 for guiding interventional imaging equipment. The device 10 for guiding interventional imaging equipment can be arranged to be integrated in the console or as a separate device. The device 10 for guiding interventional imaging equipment is data-connected to the first data arrangement device 102, as indicated by the first data connection line 130. The device 10 for guiding interventional imaging equipment is also data-connected to the second data arrangement device 106, as indicated by the second data connection line 132. The data connection is provided based on wire ground or wirelessly.

[0098] The bronchoscope 116 is inserted into the object 118, and the X-ray imaging device 111 can provide a fluoroscopic image of the object 118 as first image data, i.e., a first image. Due to the different X-ray attenuation characteristics of the bronchoscope 116 relative to the surrounding anatomical structures, a representation of the bronchoscope 116 is visible in the X-ray image. The bronchoscope 116 provides a bronchoscopic image as second image data, i.e., as a second image.

[0099] The x-ray image is used to estimate a pose of the interventional imaging device at a first point in time. The bronchoscopic image is used to track the relative motion of the interventional imaging device. An updated pose estimate of the interventional imaging device is calculated based on the estimated pose (in the x-ray image) and the tracked relative motion (in the bronchoscopic image). An updated indicator of the interventional imaging device is generated based on the calculated updated pose estimate. The x-ray image is enhanced with the updated indicator. The enhanced x-ray image is presented to a user, for example, on a display arrangement 122.

[0100] In one option, the updated indicator comprises a graphical representation of the interventional imaging device with an indication of the camera viewing direction of the bronchoscope. The indication may be provided in the form of an overlay. This provides the advantage of an improved user understanding of the current situation. The indication of the camera viewing direction of the bronchoscope provides information about the spatial arrangement of the current situation (i.e., 3D information) in an intuitive manner. However, no current X-ray projection is required, which means a reduction / minimization of radiation dose.

[0101] In another option, the updated indicator includes a simulated X-ray projection of the interventional imaging device. In this option, a generative neural network or other technique can be used without X-ray image acquisition to generate an updated X-ray image based on an initial X-ray image with an initial pose of the interventional imaging device and based on an updated pose of the interventional imaging device. This provides an understanding of the current situation without the dose of additional or extra X-ray images. The simulated projection mimics an additional image and provides a non-distracting and intuitive source of information for the user.

[0102] In an example, a system is provided that includes an external interventional imaging system, an internal or navigated interventional imaging device, a fluoroscopic image processing controller, and an image processing controller.

[0103] In an example, a pose estimate from a navigated imaging device (e.g., a bronchoscope, endoscope, EBUS, IVUS, etc.) and a recent image from an auxiliary imaging device with a larger FOV (which can image the navigated device within the patient's anatomy, such as X-ray, CBCT, tomosynthesis, etc.) are combined to update the pose of the navigated imaging device in the recent image from the auxiliary imaging device.

[0104] Pose estimation in the context of this disclosure refers to estimating the position and viewing angle of an imaging device.

[0105] According to an aspect, it is provided that at regular intervals a first position from a first imaging system is estimated, and during the interval a second position from the interventional device is estimated and the first position is updated with the second position.

[0106] Figure 3 The basic steps of an example of a method 200 for guiding an interventional imaging device are shown. The method 200 comprises the following steps:

[0107] In a first step 202, first image data are provided as first data from a first imaging device. The first image data comprises a representation of an interventional imaging device inserted within a vascular structure of a subject. The first image data comprises image data relating to a first point in time.

[0108] In a second step 204, a pose of the interventional imaging device in the first image data is estimated.

[0109] In a third step 206, second data related to the movement of the interventional imaging device are provided. The second data are related to the first point in time and at least a second point in time.

[0110] In a fourth step 208, the relative movement of the interventional imaging device is tracked based on the second data.

[0111] In a fifth step 210 , an updated pose estimate of the interventional imaging device is calculated based on the estimated pose and the tracked relative motion.

[0112] In a sixth step 212, an updated indicator of the interventional imaging device is generated based on the calculated updated pose estimate.

[0113] In a seventh step 214, the first image data is enhanced with the updated indicator.

[0114] In an eighth step 216 enhanced first image data is provided.

[0115] In an example of the method, the first image data is provided as 2D image data.

[0116] In an example of method 200, the interventional imaging device includes a second imaging device. The second data is provided as second image data from the second imaging device. The second image data includes a representation of the interior of the vascular structure. Tracking of relative motion of the interventional imaging device within the vascular structure is based on the second image data.

[0117] In an example of method 200, to estimate a pose of an interventional imaging device in first image data, one or more images of a second image stream used to generate a pose estimate associated with at least a first time point are used to adjust the estimated pose of the interventional imaging device in the first image data.

[0118] In an example of method 200, the second image data comprises a second image stream. Tracking of relative motion of the interventional imaging device is provided for successive images of the second image stream.

[0119] In the example of method 200, the following steps are also provided: calculating a trajectory of the interventional imaging device based on the updated posture estimate; generating a projection of the trajectory of the interventional imaging device based on the calculated trajectory of the interventional imaging device and the updated posture; and enhancing the first image data based on the projection of the trajectory of the interventional imaging device onto the first image data to provide updated virtual first image data.

[0120] In another example of the method 200 , the following steps are further provided: calculating a trajectory of the interventional imaging device based on the updated pose estimate; and projecting the calculated trajectory onto the first image data.

[0121] In an example of method 200, calculation of an updated pose estimate includes at least one item from a group consisting of: correction of an out-of-plane pose estimate from the first image data using the pose estimate from the second image data; and correction of an in-plane pose estimate from the second image data using the pose estimate from the first image data.

[0122] In another example of the method 200, the first image data comprises X-ray image data. The second image data comprises at least one of the group consisting of: optical camera image data, ultrasound image data, and optical coherence tomography image data.

[0123] In an example of method 200, a confidence estimate is provided regarding the relative motion estimate from the second image data. A confidence indicator is provided to a user.

[0124] In another example of the method 200 , the tracking of the relative motion of the interventional imaging device within the vascular structure is based on additional tracking data.

[0125] Figure 4A workflow for guiding an example of an interventional imaging device is shown. In the left column 302, information from a bronchoscope is provided as an example of an interventional device for the second data placement device 106. The left column relates to second data provided by the second imaging device 108, for example in the form of a bronchoscope 116. In the right column 304, information from an X-ray imager as the first data placement device 102 is provided. The right column relates to first data provided by the first imaging device 104, for example in the form of an X-ray imaging device 111. As indicated, an estimated relative camera motion 306 at t=t0 is derived from the bronchoscope data. Furthermore, an extracted bronchoscope pose 308 is derived from the fluoroscopic image at t=t0. Next, a corrected relative camera motion 310 at t=t0 is determined. A corrected bronchoscope pose 312 at t=t0 is generated. Furthermore, from the bronchoscope, an estimated relative camera motion 306 at t=t0 is derived. n Relative camera motion 314 at t = t, n>0. An added portion 316 of the bronchoscope indicator indicates the movement of the bronchoscope. This provides an updated bronchoscope overlay 318 in the fluoroscopy, which also indicates the portion of the bronchoscope from the initial state plus an added portion 320 from the movement of the bronchoscope. Next, if at t = t n The updated fluoroscopic image is acquired at t = t n The added portion 324 of the bronchoscope indicator represents the corrected relative camera motion, while the first portion 322 indicates the bronchoscope from the initial state. Also provided is the relative camera motion from t=t n An extracted / updated bronchoscopic posture of a fluoroscopic image at , having an initial portion 326 and an added portion 328 .

[0126] The second image data from the bronchoscope 116 is provided from at least two time points: a first (eg, initial or starting) time point t=t0; and a second (eg, later) time point t=t n .

[0127] First image data from the X-ray imaging device 111 is provided from a first (eg initial or starting) point in time t=t0.

[0128] At a second (eg, later) time t=t n Enhanced first image data as shown in the third column is generated and provided, but only in fact with respect to the enhanced portion in which the indicator is superimposed. Since the enhanced first image data as shown in the third column is based on the first image data from the X-ray imaging device 116 acquired at a first (e.g., initial or starting) time point t=t0, the other portion is related to the first (e.g., initial or starting) time point t=t0.

[0129] exist Figure 4 In FIG. 1 , a single arrow between the left and right columns indicates forwarding or transfer of data and information. Within the same column, a single arrow indicates processing of data. A double arrow indicates mutual use or mutual transfer of data and information between two domains (i.e., two coordinate systems of bronchoscopic images and X-ray images).

[0130] Figure 5 Another example workflow for guiding an interventional imaging device is shown. The most recent fluoroscopic image 350 is provided in the upper region, wherein a bronchoscope 352 is shown. An updated bronchoscope posture 356 of a bronchoscope 354 is provided in the middle region. Here, the updated bronchoscope posture 536 is superimposed on the fluoroscopic image in the middle region. Thus, additional information derived from another domain (e.g., the bronchoscope domain) is added to the domain of the fluoroscopic image. The updated bronchoscope posture 356 is used to generate a simulation of the projection of the interventional imaging device in the corresponding posture. The simulation can directly generate a realistic fluoroscopic image with the updated posture of the bronchoscope, as shown in the lower region. This can be done using a generative neural network that is trained to generate a realistic rendering of the device within the anatomical structure given an updated device posture. As a result, a generated fluoroscopic image 350' with an updated bronchoscope posture 358 is generated. The generated new image should represent t=t n The image at Figure 5 This uses t = t n ' indicates that, where, ' indicates the fact that it is a simulated or synthetic image.

[0131] First image data from the X-ray imaging device 116 is provided from a first (eg, initial or starting) time point t=t0. Thus, Figure 5 The fluoroscopic image in the upper portion of is from a first (eg, initial or starting) time point t=t0.

[0132] The second image data from the bronchoscope 116 is from at least a second (eg, later) time point t=t n Provided.

[0133] exist Figure 5 The added fluoroscopic image in the middle portion of is at a second (eg, later) time point t=t n The enhanced first image data shown in the middle region is based on the first image data from the X-ray imaging device 116 acquired at a first (e.g., initial or starting) time point t=t0, so the other parts are related to the first (e.g., initial or starting) time point t=t0.

[0134] The enhanced first image data as shown in the lower region is at a second (eg, later) time point t=t n t=t0. However, since the image is based on the first image data acquired at a later time point t=t0, the other parts are related to the first (e.g., initial or starting) time point t=t0. n The bronchoscope pose at t is estimated based on the bronchoscope pose at t without X-ray image acquisition, so the image is referred to as having been acquired at time t = t n ' is generated.

[0135] exist Figure 5 In FIG. 1 , a single arrow indicates the forwarding or transfer of data and information with processing of the data.

[0136] The term "subject" may also be referred to as an individual. A "subject" may also be referred to as a patient, although it should be noted that this term does not indicate whether the subject actually has any disease or condition.

[0137] In another example, a computer program includes instructions that, when executed by a computer, cause the computer to perform the method of one of the preceding examples.

[0138] In an example, a computer program or a program element for controlling an apparatus according to one of the above examples is provided, which program or program element, when executed by a processing unit, is adapted to perform method steps of one of the above method examples.

[0139] In another example, a computer readable medium is provided, on which the computer program of the above example is stored.

[0140] In a further exemplary embodiment of the present invention, a computer program or a computer program element is provided, which is characterized by being adapted to execute the method steps of the method according to one of the preceding embodiments on a suitable system.

[0141] Therefore, the computer program element can be stored on a computer unit or distributed on more than one computer unit, which can also be part of an embodiment of the present invention. The computing unit can be suitable for executing or inducing the steps of the above method. In addition, it can be suitable for operating the components of the above device. The computing unit can be suitable for automatic operation and / or execution of user's commands. The computer program can be loaded into the working memory of the data processor. Therefore, the data processor can be equipped to perform the method of the present invention.

[0142] Aspects of the present invention can be implemented in a computer program product, which can be a collection of computer program instructions stored on a computer-readable storage device that can be executed by a computer. Instructions of the present invention can be in any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs) or Java classes. Instructions can be provided as complete executable programs, partial executable programs, modifications (e.g., updates) to existing programs, or extensions (e.g., plug-ins) to existing programs. In addition, a portion of the processing of the present invention can be distributed on multiple computers or processors.

[0143] As described above, a processing unit (e.g., a controller) implements the control method. The controller can be implemented in many ways using software and / or hardware to perform the various functions required. A processor is an example of a controller that employs one or more microprocessors that can be programmed using software (e.g., microcode) to perform the required functions. However, the controller can be implemented with or without a processor, and can also be implemented as a combination of dedicated hardware that performs some functions and a processor (e.g., one or more programmed microprocessors and associated circuits) that performs other functions.

[0144] Examples of controller components that may be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0145] This exemplary embodiment of the invention covers both a computer program that right from the beginning uses the invention and a computer program that by an update turns an existing program into a program that uses the invention.

[0146] Furthermore, the computer program element may be able to provide all necessary steps for fulfilling the procedures of an exemplary embodiment of the method described above.

[0147] According to another exemplary embodiment of the present invention, a computer readable medium, such as a CD-ROM, is provided, wherein the computer readable medium has a computer program unit stored thereon, the computer program unit being described by the aforementioned section. The computer program may be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid state medium provided together with or as part of other hardware, but the computer program may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0148] However, the computer program may also be presented over a network like the World Wide Web and may be downloaded from such a network into a working memory of a data processor. According to another exemplary embodiment of the present invention, a medium for making a computer program element available for downloading is provided, the computer program element being arranged to perform a method according to one of the aforementioned embodiments of the present invention.

[0149] It must be noted that embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method type claims, while other embodiments are described with reference to device type claims. However, a person skilled in the art will understand from the above and following descriptions that, unless otherwise notified, any combination of features relating to different subject matters, in addition to any combination of features belonging to one type of subject matter, is also considered to be disclosed by the present application. However, all features may be combined, thereby providing synergistic effects that exceed the simple sum of the features.

[0150] Although the present invention has been described and illustrated in detail in the drawings and the foregoing description, such description and illustration should be considered illustrative or exemplary rather than restrictive. The present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and implemented by those skilled in the art in practicing the claimed invention by studying the drawings, the disclosure, and the dependent claims.

[0151] In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A device (10) for guiding an interventional imaging device, comprising: Data input unit (12); Data processor (14); as well as Output interface (16); wherein the data input is configured to: provide first image data as first data from a first imaging device, wherein the first image data comprises a representation of an interventional imaging device inserted within a vascular structure of a subject, and wherein the first image data comprises image data relating to a first point in time; and provide second data relating to movement of the interventional imaging device, wherein the second data relates to the first point in time and at least a second point in time; wherein the data processor is configured to: estimate a pose of the interventional imaging device in the first image data; track relative motion of the interventional imaging device based on the second data; calculate an updated pose estimate of the interventional imaging device based on the estimated pose and the tracked relative motion; generate an updated indicator of the interventional imaging device based on the calculated updated pose estimate; and enhance the first image data using the updated indicator; and Wherein, the output interface is configured to provide enhanced first image data.

2. The device according to claim 1, wherein The second data is second image data from a second imaging device provided by the interventional imaging device; wherein the second image data comprises a representation of an interior of the vascular structure; and Wherein the data processor is configured to track relative movement of the interventional imaging device within the vascular structure based on the second image data.

3. The device according to one of the preceding claims, wherein The data processor is further configured to compute the updated pose estimate, the computing comprising at least one of the group consisting of: correction of an out-of-plane pose estimate from the first image data using the pose estimate from the second image data; as well as Correction of an in-plane pose estimate from the second image data using the pose estimate from the first image data.

4. The device according to claim 2 or 3, wherein: In order to estimate the posture of the interventional imaging device in the first image data, the data processor is configured to use one or more images of the second image data, and the one or more images are used to generate a posture estimate related to at least the first time point for adjusting the estimated posture of the interventional imaging device in the first image data.

5. The apparatus according to claim 2, 3 or 4, wherein: The second image data comprises a second image stream; and Wherein the data processor is configured to track the relative motion of the interventional imaging device with respect to consecutive images of the second image stream.

6. The device according to one of the preceding claims, wherein The data processor is further configured to: calculate a trajectory of the interventional imaging device based on the updated pose estimate; and i) generating a projection of the trajectory of the interventional imaging device based on the calculated trajectory of the interventional imaging device and the updated pose; and enhancing the first image data based on the projection of the trajectory of the interventional imaging device to provide updated virtual first image data; or ii) projecting the calculated trajectory onto the first image data.

7. The device according to one of the preceding claims, wherein The data processor is also configured to generate a realistic composite image using a trained generative neural network based on the first image data at the first time point and the updated pose of the interventional imaging device at a second time point, the realistic composite image depicting the updated pose of the interventional imaging device.

8. The device according to one of the preceding claims, wherein The first image data comprises X-ray image data; and The second image data includes at least one item from the group consisting of: optical camera image data, ultrasound image data, and optical coherence tomography image data.

9. The device according to one of the preceding claims, wherein The data processor is further configured to provide a confidence estimate related to the relative motion estimate from the second image data; and Wherein, the output interface is configured to provide a confidence indicator to a user.

10. A system (100) for guiding an interventional imaging device, the system comprising: A first data arrangement apparatus (102) comprising a first imaging device (104); Second data arrangement means (106); as well as Device (10) for guiding an interventional imaging device according to any of the preceding claims; wherein the first imaging device is configured to generate the first image data as the first data; and wherein the second data arrangement means is configured to generate the second data.

11. The system according to claim 10, wherein: providing an interventional imaging device (110); and Therein, the second data arrangement device is provided as a second imaging device provided by the interventional imaging device.

12. The system according to claim 10 or 11, wherein: The first imaging device is provided as an X-ray imaging device (111); and Wherein, the second imaging device is provided as at least one of the group consisting of: a bronchoscope (116), an endoscope, a colonoscope, intravascular ultrasound, intracardiac echocardiography, endobronchial ultrasound or radial endobronchial ultrasound, and optical coherence tomography.

13. A method (200) for guiding an interventional imaging device, comprising the steps of: providing (202) first image data as first data from a first imaging device; wherein the first image data comprises a representation of the interventional imaging device inserted within a vascular structure of a subject; wherein the first image data comprises image data relating to a first point in time; estimating (204) a pose of the interventional imaging device in the first image data; providing (206) second data related to movement of the interventional imaging device; wherein the second data is related to the first time point and at least a second time point; tracking (208) relative motion of the interventional imaging device based on the second data; calculating (210) an updated pose estimate of the interventional imaging device based on the estimated pose and the tracked relative motion; generating (212) an updated indicator of the interventional imaging device based on the calculated updated pose estimate; enhancing (214) the first image data with the updated indicator; and Enhanced first image data is provided (216).

14. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method according to claim 13.

15. A computer readable medium having stored thereon a computer program according to claim 14.

Images