Guiding during medical procedure

Abstract

The invention relates to medical imaging. In order to provide a convenient way to provide an improved image about the current situation, a device (10) for guiding during a medical procedure is provided. The device comprises the data input (12), a data processor (14) and an output interface (16). The data input is configured to provide 3D image data of a region of interest of an object and to provide current 2D image data of the region of interest. The data processor is configured to: register current 2D image data with 3D image data to determine a first transformation, identify non-linear and linear components of the determined first transformation, apply the identified linear components of the first transformation to the 3D image data, and generating a projection image from the 3D image data by applying the linear component to the 3D image data. The output interface is configured to provide the projected image as guidance during a medical procedure.

Description

Technical Field

[0001] The present invention relates to medical imaging. The present invention relates in particular to an apparatus for guidance during a medical procedure, a guidance system for use during a medical intervention and a method for guidance during a medical procedure. Background Art

[0002] Image-guided endoscopic interventions remain challenging in areas of the body where image quality and clarity are distorted by the natural motion of the patient's body. For example, bronchoscopic procedures may require a high level of skill to navigate through the airways and avoid critical structures. One of the main obstacles to further improving the results of endoscopy techniques in these areas is the image distortion caused by the natural cyclical motion of the patient's body and the motion of the table and imaging system (especially in the case of mobile fluoroscopic C-arm systems). X-ray fluoroscopy is used for intraoperative imaging guidance due to its simplicity of use, good field of view, and ability to visualize the lung airways. However, complex structures such as the lung airways are difficult to visualize and understand, especially under the influence of high-frequency cyclical motions such as respiratory and cardiac motion. During the fluoroscopic guided procedure, the motion of one part of the body relative to another becomes difficult to disentangle, and previously registered static projections may be misplaced. For example, US10682112B2 relates to the use of a 3D preoperative volume to suppress independent motion in a series of 2D X-ray fluoroscopic images. Examples include fluoroscopy-guided lung bronchoscopy, where the patient's breathing can impede clear visualization of target anatomy and surgical devices, which can reduce the diagnostic efficacy of peripheral airway biopsies, and cardiac procedures such as valve repair, where cardiac motion makes device target confirmation difficult. Summary of the invention

[0003] It may therefore be desirable to provide a convenient way to provide an improved image of the current intervention situation.

[0004] The objects of the invention are solved by the subject matter of the independent claims; further embodiments are incorporated in the dependent claims. It should be noted that the aspects of the invention described below also apply to an apparatus for guiding in a medical procedure, a system for guiding during a medical intervention and a method for guiding during a medical procedure.

[0005] According to the present invention, a device for guiding during a medical procedure is provided. The device includes a data input unit, a data processor and an output interface. The data input unit is configured to provide 3D image data of a region of interest of an object. The data input unit is also configured to provide current 2D image data of the region of interest. The data processor is configured to register the current 2D image data with the 3D image data to determine a first transformation. The data processor is also configured to identify a nonlinear component and a linear component of the determined first transformation. The data processor is also configured to apply the identified linear component of the first transformation to the 3D image data. The data processor is also configured to generate a projection image based on the 3D image data by applying the linear component to the 3D image data. The output interface is configured to provide the projection image as a guide during a medical procedure.

[0006] Thus, improved guidance is provided while avoiding an increase in radiation dose, which is implicit when, for example, image resolution and frame rate are increased. Virtual fluoroscopy incorporates patient specific information for the purpose of view stabilization. As a further effect, virtual fluoroscopy has an advantage over virtual rendering because virtual fluoroscopy is similar to live fluoroscopy images and is therefore less difficult to use. Another effect is increased confidence in the display on the user's (e.g. surgeon's) side. This also solves the problem of complex navigation in tortuous and moving vessels / airways.

[0007] According to an example, the data input is configured to provide the 3D image data as preoperative 3D image data. In one option, preoperative CT image data is provided. The data input is configured to provide the current 2D image data as 2D X-ray image data. The data processor is configured to generate a projection image whose viewing direction is aligned with the viewing direction of the 2D X-ray image data. As an option, the data processor is configured to provide the projection image as a digitally reconstructed radiographic visualization.

[0008] According to an example, the data input is configured to provide a current image, the current image comprising image data relating to an interventional device inserted into a region of interest. The data processor is configured to perform segmentation on the current 2D image data to identify a representation of the device. The data processor is further configured to apply a second transformation to the representation of the device. The data processor is further configured to combine the transformed representation of the device with the generated projection image.

[0009] According to an example, the data processor is configured to provide the second transform as an inverse of a non-linear component of the first transform.

[0010] According to an example, the data processor is configured to overlay the transformed representation of the device onto the generated projection image. In one option, the data processor is configured to provide the transformed representation to the generated projection image as an overlay similar to fluorescence imaging.

[0011] According to an example, the data processor is configured to provide a representation of the device comprising a segmented image portion of the 2D image.The data processor is further configured to apply a transformation to the segmented image portion.

[0012] According to an example, the data input is configured to provide tracking data of an external tracking device, the external tracking device tracking an interventional device inserted into the region. The data processor is configured to track the interventional device relative to the object based on the tracking data. The data processor is further configured to align a coordinate space of the tracked device with an imaging coordinate space. The data processor is further configured to apply the second transformation to a graphical representation of the device. The data processor is further configured to combine the transformed representation of the device with the generated projection image.

[0013] According to the present invention, a system for guidance during medical intervention is also provided. The system comprises an image data source, a medical imaging system, a device for guidance during medical flow according to one of the aforementioned examples, and a display device. The image data source is configured to provide 3D image data of an area in a region of interest of an object. The medical imaging system is configured to provide current 2D image data of the region of interest of the object. The device for guidance during a medical procedure is configured to provide a projection image generated based on the provided 3D image data and the provided current 2D image data. The display device is configured to present the projection image as a guide during the medical procedure.

[0014] According to an example, the medical imaging system is provided as an X-ray imaging system, which is configured to provide current 2D image data as 2D X-ray image data. In one option, the data processor is configured to generate a projection image whose viewing direction is aligned with the viewing direction of the 2D X-ray image data. In another option, the X-ray imaging system is further configured to generate 3D image data of the object.

[0015] According to an example, external tracking of the interventional device is provided, including at least one of the group of electromagnetic tracking and optical tracking. Electromagnetic tracking is applied to record and determine transformations while the object remains in place. When relative motion occurs, the current 2D image data is used for registration and determination of transformations.

[0016] According to the present invention, a method for guiding during a medical procedure is also provided. The method comprises the following steps:

[0017] Providing 3D image data of a region of interest of an object;

[0018] Providing current 2D image data of the region of interest;

[0019] registering the current 2D image data with the 3D image data to determine a first transformation;

[0020] identifying a nonlinear component and a linear component of the determined first transform;

[0021] applying the identified linear component of the first transform to the 3D image data;

[0022] generating a projected image from the 3D image data by applying the linear component to the 3D image data; and

[0023] The projection images are provided as a guide during a medical procedure.

[0024] In the example, the generated image is generally an X-ray image, i.e., in its appearance, it imitates a fluoroscopic image. The projected image can be said to simulate a live view, which leads to an increased confidence level on the part of the user (e.g., surgeon). Instead of a real X-ray image, the projected image provides an image that looks like an X-ray image. Thus, the apparatus simulates an X-ray imaging device for live X-ray imaging.

[0025] According to one aspect, a virtual stabilized view of live fluoroscopy is provided for generating from patient-specific preoperative imaging.

[0026] In an example, a software package is provided for integration into C-arm hardware. In another example, a standalone controller is provided that communicates with a C-arm system and a picture archiving and communication (PAC) system.

[0027] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Exemplary embodiments of the present invention will be described below with reference to the accompanying drawings:

[0029] Figure 1 An example of a device for guiding during a medical procedure is schematically shown.

[0030] Figure 2 An example of a system for guidance during a medical intervention is shown.

[0031] Figure 3 Basic steps of an example of a method for guiding during a medical procedure are shown.

[0032] Figure 4 An example of a view stabilization workflow for guiding an interventional imaging device is shown.

[0033] Figure 5 An example of a stabilized view is shown.

[0034] Figure 6 Another example of a view stabilization workflow for guiding an interventional imaging device is shown. DETAILED DESCRIPTION

[0035] Specific embodiments will now be described in more detail with reference to the accompanying drawings. In the following description, the same reference numerals are used for the same elements even in different drawings. Contents defined in the description, such as detailed construction and elements, are provided to assist in a comprehensive understanding of the exemplary embodiments. Furthermore, well-known functions or structures are not described in detail because they would obscure the embodiments with unnecessary detail. Furthermore, expressions such as "at least one", when preceding a list of elements, modify the entire list of elements and do not modify a single element of the list.

[0036] Figure 1 An example of a device 10 for guiding during a medical procedure is schematically shown. The device 10 includes the data input 12, a data processor 14, and an output interface 16. The data input 12 is configured to provide 3D image data of a region of interest of an object. The data input 12 is also configured to provide current 2D image data of the region of interest. The data processor 14 is configured to register the current 2D image data with the 3D image data to determine a first transformation. The data processor 14 is also configured to identify a nonlinear component and a linear component of the determined first transformation. The data processor 14 is also configured to apply the identified linear component of the first transformation to the 3D image data. The data processor 14 is also configured to generate a projection image from the 3D image data by applying the linear component to the 3D image data. The output interface 16 is configured to provide the projection image as a guide during a medical procedure.

[0037] The data input 12, data processor 14 and output interface 16 may be provided in a common structure, such as a common housing, as shown by box 18, or even in an integrated manner. In another option (not shown), they are provided as separate components or units.

[0038] The first arrow 20 indicates the data supply to the data input part 12, that is, the 3D image data is provided. The second arrow 22 indicates another data supply to the data input part 12, that is, the current 2D image data is provided. The third arrow 24 indicates the data supply from the output interface 16, that is, the projection image is provided. The data supply can be provided based on wired or wireless. In the example, as an option, a display 26 is provided to present the enhanced first image. The display 26 is connected to the output interface 16 data.

[0039] The first transform may also be referred to as a transform, an image data transform, a primary transform, a primary transform, or a situation transform.

[0040] The term "3D image data" relates to spatial data of an object acquired by a 3D medical imaging procedure, such as ultrasound imaging, X-ray imaging or MRT imaging.

[0041] The term "current 2D image data" relates to image data provided in a current state, for example a live image during a medical procedure or intervention.The image data are provided as 2D image data in an image plane.

[0042] The term "registration" involves calculating the spatial relationship of two different image data sets. The spatial relationship includes information on how to manipulate the corresponding other data for spatial matching. The registration includes a linear registration part, which is a global registration of 2D image data within the 3D image data. The registration also includes a non-linear registration part, which is a deformation registration process of 2D image data to 3D image data. Linear registration is related to different viewing angles and distances, such as those caused by the movement of the object support. Non-rigid or non-linear registration involves deformations of the object itself, such as those caused by breathing or other activities (such as organ movement, especially heartbeat).

[0043] The term "transformation" relates to defining how the two-dimensional image data needs to be transformed (ie changed in a broad sense) to align with the 3D data set.

[0044] The term "linear" refers to the linear registration part, or any subset of linear transformations, such as affine or rigid transformations.

[0045] The term "nonlinear" refers to the rest of the transformation that is not covered by the "linear" part. The nonlinear component of the transformation is related to the deformation of the tissue, thereby achieving registration.

[0046] The term "projected image" relates to an image generated by projecting an object or target onto a projection surface or projection plane. Thus, structures within the projection volume may contribute to the projected image. An example of a projected image is an X-ray radiation image.

[0047] The term "generating a projection image" relates to an artificially generated image, which for example simulates an X-ray image.

[0048] The term "data input" relates to providing or supplying data for a data processing step. The data input may also be referred to as an image data input. The data input may also be referred to as a data supply, an image data supply, an image input, an input unit or simply an input. In one example, the image data input may be data connected to an imaging source device. In one example, the data input may be data connected to a data storage device that has stored image data.

[0049] The term "data processor" relates to a processor or a part of a processor device, which is used to perform a calculation step using the data provided by the data input. A data processor may also be referred to as a data processing device, a processing unit or a processor. In an example, the data processor is data connected to the data input and the output interface.

[0050] The term "output interface" refers to an interface for providing processed or calculated data for further use. An output interface may also be referred to as an output portion or an output unit. In one example, the output interface may be connected to a display device or a display apparatus via data. In another example, the output portion is connected to a display.

[0051] In an example, a patient-specific CT-derived virtual fluoroscopy is used to provide fluoroscopic view stability.

[0052] According to one aspect, a high-resolution preoperative CT of a patient is used to generate a virtual fluoroscopic view that simulates a live fluoroscopic view whose distortion motion is stabilized. The so-called view stabilization is performed by 2D-3D fluoroscopic to CT registration, where the linear and nonlinear components are separated and the separated transformation is used to estimate the stabilized view.

[0053] Patient-specific information is used to generate a fluoroscopy-like view that is stabilized by registering the preoperative CT volume with the live fluoroscopy image. The result is a live image that resembles real anatomy fluoroscopy and surgical equipment stabilized relative to the virtual X-ray source. This view is comfortable for the clinician and solves motion issues during high-precision procedures.

[0054] Another advantage of this approach is that for procedures that require visualization of clear anatomy (e.g., lung airways or cardiac chambers), low-resolution fluoroscopic images may be sufficient for the purpose of CT-to-fluoroscopic registration. In this case, the live fluoroscopic examination serves only as a guide for registration and not for high-resolution visualization. Assuming accurate registration, the work of rendering high-quality images can be shifted to the virtual fluoroscopy or DRR generation process rather than the live images, thereby reducing intraoperative radiation usage.

[0055] According to the example, this therefore provides a virtual stabilized view of live fluoroscopy using a DRR-like visualization using the patient's own high-resolution preoperative CT scan for use in fluoroscopy-guided lung bronchoscopy applications. Rather than attempting to enhance the live fluoroscopy view after fluoroscopy is registered with a reference image, we propose generating a reconstructed view from the patient-specific CT scan at an estimated C-arm position, but with the distortion motion subtracted (equivalent to separating the linear and nonlinear components of the fluoroscopy-to-CT registration). In one option, any surgical device can be segmented live and inserted into the reconstructed virtual view, using existing methods to simulate real catheters in fluoroscopy. This produces a stabilized view in which the anatomical structures and devices do not move relative to the X-ray source, while maintaining a fluoroscopy-like view and patient-specific information that the clinician is comfortable using.

[0056] The advantage is that it provides virtual fluoroscopic visualization with a stabilized view while incorporating patient-specific detailed anatomy or images, also increasing clinician confidence. As a result, for procedures requiring high precision or complex navigation, motion compensation can be provided to stabilize the view.

[0057] In one example, the data input 12 is configured to provide the 3D image data as preoperative 3D image data. The data input 12 is configured to provide the current 2D image data as 2D X-ray image data. The data processor 14 is configured to generate a projection image whose viewing direction is aligned with the viewing direction of the 2D X-ray image data. The data processor 14 is configured to provide the projection image as a digitally reconstructed radiograph visualization.

[0058] In an example, 2D X-ray image data is acquired using an X-ray imaging system (eg, a C-arm device). The 2D X-ray image data is acquired using a relative imaging position with respect to an object. A projection image is generated using a viewing direction according to the relative imaging position.

[0059] In an example, the data input 12 is configured to provide a current image, the current image comprising image data relating to an interventional device inserted into a region of interest. The data processor 14 is configured to perform segmentation on the current 2D image data to identify a representation of the device. The data processor 14 is configured to apply a second transformation to the representation of the device. The data processor 14 is further configured to combine the transformed representation of the device with the generated projection image.

[0060] The second transform may also be referred to as a transform, a split transform, a secondary transform, a secondary transform or an auxiliary transform, or a device transform.

[0061] The interventional device may be a catheter, needle, forceps, or implant.

[0062] In an example, the current image includes image data related to a identifiable structure that is not present in the 3D image data; and the data processor is configured to: perform segmentation on the current 2D image data to determine the identifiable structure; apply a second transformation to the determined identifiable structure; and combine the transformed determined identifiable structure with the generated projection image.

[0063] In an example, the data processor 14 is configured to provide the second transform as the inverse of the non-linear component of the first transform.

[0064] In an example, the data processor 14 is configured to overlay the transformed representation of the device onto the generated projection image.The data processor 14 is configured to provide the transformed representation as a fluoroscopy-like overlay to the generated projection image.

[0065] In an example, the data processor 14 provides a representation of the device comprising a segmented image portion of the 2D image.The data processor 14 is configured to apply a transformation to the segmented image portion.

[0066] In an example, the data input 12 is configured to provide a 3D model of the device suitable for the segmented representation of the device; the data processor 14 is configured to apply the transformation to the 3D model of the device. The data processor 14 is also configured to provide a projection of the model superimposed on the generated projection image.

[0067] In another example, image data of the 3D model is added to the 3D image data before generating the projection image.

[0068] In an example, the data input 12 is configured to provide tracking data of an external tracking device, which tracks an interventional device inserted into the region. The data processor 14 is configured to track the interventional device relative to the object based on the tracking data. The data processor 14 is configured to align a coordinate space of the tracked device with an imaging coordinate space. The data processor 14 is configured to apply the second transformation to the graphical representation of the device. The data processor 14 is configured to combine the transformed representation of the device with the generated projection image.

[0069] In an example, the region of interest includes an anatomical structure including at least one of the following group: airway, lung, heart, and cardiovascular structure.

[0070] Figure 2An example of a system 100 for guiding during a medical intervention is shown. The system 100 comprises an image data source 102, a medical imaging system 104, an apparatus 10 for guiding during a medical procedure according to one of the foregoing examples, and a display device 106. The image data source 102 is configured to provide 3D image data of a region of interest of an object. The medical imaging system 104 is configured to provide current 2D image data of a region of interest of an object. The apparatus 10 is configured to provide a projection image generated based on the provided 3D image data and the provided current 2D image data. The display device 106 is configured to present the projection image as a guide during a medical procedure.

[0071] In an example, the image data source 102 is a data storage storing 3D CT image data of an object. In one option, the image data source 102 is a CT system, whose data is connected to the device for guidance during a medical procedure.

[0072] In a selection, such as Figure 2 As shown in , the medical imaging system 104 is provided as an X-ray imaging system 108, which is configured to provide the current 2D image data as 2D X-ray image data. The data processor 14 is configured to generate a projection image whose viewing direction is aligned with the viewing direction of the 2D X-ray image data. In one option, additionally or alternatively, the X-ray imaging system 108 is also configured to generate 3D image data of the object.

[0073] The X-ray imaging system 108 is provided as a C-arm imaging system, which includes an X-ray source 110 and an X-ray detector 112 mounted at opposite ends of a movably mounted C-arm 114 .

[0074] As an option, an X-ray imaging system 108 is also provided to acquire 3D image data of the object. In another example, the X-ray imaging system 108 is a mobile C-arm system.

[0075] exist Figure 2 In the embodiment of the present invention, a subject support 116 is provided. Furthermore, a control interface 118 is arranged next to the subject support 116. A subject 120 is arranged on the subject support 116. Furthermore, an interventional device 122 is partially inserted into the subject 120.

[0076] A console 124 is shown in the foreground. The console 124 is arranged to provide user interaction and control options. The console 124 includes a set of displays, a keyboard with a mouse, a graphics tablet, and control knobs, etc. The console 124 can control various functions and operations of the system 100 to guide an interventional imaging device. The device 10 for guiding an interventional imaging device can be arranged to be integrated in the console 124 or as a separate device.

[0077] The image data source 102 is data-connected to the device 10 for guiding the interventional imaging device, as indicated by the first data connection line 126. The device 10 for guiding the interventional imaging device is also data-connected to the medical imaging system 104, as indicated by the second data connection line 128. The data connection is provided on a wired or wireless basis. The device 10 for guiding the interventional imaging device is also data-connected to the console 124, as indicated by the third data connection line 130.

[0078] In an example, external tracking is provided, including at least one of the group of electromagnetic tracking and optical tracking of the interventional device. Electromagnetic tracking is applied to record and determine transformations while the object remains in place. When relative motion occurs, the current 2D image data is used for registration and determination of transformations.

[0079] When the object remains in place, no relative motion occurs. The use of electromagnetic tracking can further reduce the radiation dose. In an example, an external tracking device is provided to track an interventional device inserted into the region of interest. The data processor 14 is configured to: track the interventional device relative to the object based on data from the external tracking device; align the coordinate space of the tracked device with the imaging coordinate space; apply the second transformation to the graphical representation of the device; and combine the transformed representation of the device with the generated projection image.

[0080] According to one aspect, a stabilized view is provided by using 3D image data with improved resolution and increased detailed information and generating a projection image from the 3D data while using current (actual) image data to determine fluoroscopy. In one option, the current (actual) image data is also used to detect devices or other structures that are not present in the 3D image data and transfer them to the projection of the 3D image data. In other words, the 3D image data (e.g., preoperative or intraoperative image data) is used to provide improved guidance. The current 2D image is used to update the 3D image data to the current situation, thereby providing current guidance, i.e., navigation support.

[0081] Figure 3 The basic steps of an example of a method 200 for guiding during a medical procedure are shown. The method 200 comprises the following steps.

[0082] In a first step 202, 3D image data of a region of interest of an object is provided.

[0083] In a second step 204, current 2D image data of the region of interest is provided.

[0084] In a third step 206 , the current 2D image data is registered with the 3D image data to determine a first transformation.

[0085] In a fourth step 208, non-linear components and linear components of the determined first transformation are identified.

[0086] In a fifth step 210, the identified linear component of the first transform is applied to the 3D image data.

[0087] In a sixth step 212, a projection image is generated from the 3D image data by applying the linear component to the 3D image data.

[0088] In a seventh step 214, the projection image is provided as a guide during the medical procedure.

[0089] The first step 202 and the second step 204 may also be performed simultaneously or in reverse order.

[0090] In an example, the following steps are provided: a preoperative CT volume and a live fluoroscopic image are provided, and a CT to fluoroscopic image registration is performed. Optionally, if surgical devices are present in the image, segmentation is performed on these devices. A separation of the nonlinear component and the linear component of the transformation calculated in the registration is provided. The linear component of the transformation is applied to the preoperative CT volume, and a digitally reconstructed radiograph (DRR) is generated from this perspective. It is worth noting that moving the C-arm necessarily produces a rigid motion of the image volume, which is a linear motion. Optionally, if surgical devices are present in the image, the inverse of the nonlinear component of the transformation from the separation is applied to the segmented device, as well as any other objects segmented in the live fluoroscopic image (which do not appear in the preoperative CT), and superimposed on the generated DRR using a model or fluorescence-like overlay. Optionally, a post-processing algorithm from a C-arm image processing module can be applied.

[0091] In an example of the method, the 3D image data is pre-operative 3D image data. In one option, the pre-operative 3D image data is pre-operative CT image data. The projection image is a digitally reconstructed radiographic visualization.

[0092] In the example of method 200, the current 2D image data is provided as 2D X-ray image data. The projection image is generated with a viewing direction aligned with the viewing direction of the 2D X-ray image data.

[0093] In an example, the current image includes image data relating to identifiable structures that are not present in the 3D image data. The method 200 further includes the following steps:

[0094] performing segmentation on the current 2D image data to determine identifiable structures;

[0095] applying a second transformation to the determined identifiable structure; and

[0096] And the transformed determined identifiable structure is combined with the generated projection image.

[0097] In an example of method 200, the current image includes image data related to an interventional device inserted into the region of interest. The following steps are also provided: performing segmentation on the current 2D image data to identify a representation of the device; applying a second transformation to the representation of the device; and combining the transformed representation of the device with the generated projection image.

[0098] In one example of the method 200, the second transform is provided as the inverse of the non-linear component of the first transform.

[0099] In the example of method 200 , the transformed representation of the device is superimposed onto the generated projected image.

[0100] In an example of method 200, the representation of the device comprises a segmented image portion of the 2D image. Furthermore, the transformation is also applied to the segmented image portion. In an example of method 200, the transformed representation is provided as a fluorescence image-like overlay of the generated projection image.

[0101] In an example of method 200, a 3D model of a device is provided that fits the segmented representation of the device. The transformation is applied to the 3D model of the device. Furthermore, a projection of the model superimposed onto the generated projection image is provided.

[0102] In an example of method 200, an interventional device inserted into the region of interest is tracked by an external tracking device. The following steps are provided: tracking the interventional device relative to the object; aligning the coordinate space of the tracked device with the imaging coordinate space; applying the second transformation to the graphical representation of the device; and combining the transformed representation of the device with the generated projection image.

[0103] In an example of the method, the region of interest includes an anatomical structure including at least one of the following group: an airway, a lung, a heart, and a cardiovascular structure.

[0104] Figure 4 An example of a view stabilization workflow for guiding an interventional imaging device is shown. Preoperative CT data, i.e., a CT volume 302, and a live fluoroscopic image 304 are provided. Transformation parameters associated with CT to fluoroscopic image registration are provided. As indicated by the first arrow 306. Next, the device in the live fluoroscopy is segmented 308. In addition, the linear component R of the transformation and the nonlinear component is separated, as indicated by a first separation arrow 310 for the linearly transformed component R, and for the nonlinearly transformed component The linear component is applied to the CT volume to generate a stable view 314 in the form of a DRR. The inverse of the nonlinear component is applied to the CT volume to generate a stable view 314 in the form of a DRR. Applied to the segmented devices to superimpose them onto the stabilized DRR, also referred to as projecting the device segmentation into the stabilized view, is indicated by further arrows 316 .

[0105] From a mathematical point of view, the workflow can be described as follows, where the input is:

[0106] I0=3D model or CT image

[0107] F = Live fluoroscopic image

[0108] R = linear transformation component

[0109] φ=nonlinear transformation component

[0110] D(x) = segmented object (point set)

[0111] For a particular live fluoroscopic image F and CT volume I0, the transformation R°φ between F and I0 is calculated, as is standard for nonlinear image-to-image registration. A stable background image is generated by transforming the CT volume to R, yielding I0(R·x). A forward projection is performed through I0(R·x) to produce the DRR background.

[0112] Figure 5 An example of a stabilized view 346 is shown in the left column; and a corresponding live view 348 is shown in the right column. A first arrow 350 indicates a non-linear deformation caused by patient motion. A second arrow 352 indicates a linear deformation caused by C-arm / table motion. Anatomical structures 354, such as vascular structures, may be deformed, for example, due to motion.

[0113] Figure 5 The effect of stabilization on the background anatomy is shown. D(x) is calculated from I0 as a set of coordinate points (e.g., points along the catheter) and transformed to the DRR background space by the inverse of the nonlinear transformation components, yielding D(x)°φ -1 , which is then superimposed on the DRR background.

[0114] Figure 5The effect of stabilization on anatomical structures under fluoroscopy is shown. A sequence of four video frames from a live fluoroscopy examination from a porcine preclinical study is shown. As described above, the left column shows a stabilized virtual view and the right column shows live fluoroscopy. Between frames 1 and 2 (counted from the top), the C-arm rotates to slightly different angles. Between frames 2 and 3, the subject inhales, and the diaphragm and lungs move visibly in the live view. Between frames 3 and 4, the subject exhales, and movement is shown again. The stabilized background view generated by I0(R·x) based on the subject's preoperative CT shows the trajectory of the rigid C-arm movement, but does not move in response to the subject's anatomical movement, allowing for more stable observation of the lung airways during the surgical process.

[0115] Figure 6 Another example of a view stabilization workflow for guiding an interventional imaging device is shown. A first image 380 shows a theoretically "stabilized" live view with a device 382 superimposed. A second image 384 indicates that motion has occurred in the live view, as shown by arrow 386. In addition, a preoperative CT volume 388 is provided. A further image 390 shows a stabilized view generated by CT. Figure 6 The process of superimposing an undistorted surgical tool onto a stabilized anatomical image is shown. Figure 6 The process of superimposing a device or other object that was not present in the preoperative imaging into a stabilized view is illustrated. Starting from the top, an attempt is made to recover a theoretically stable live fluoroscopy by segmenting the image distorted by motion (second image), dewarping the device using the inverse of the nonlinear transformation components, and superimposing it on the stabilized DRR once they are in the same coordinate space (final image).

[0116] An example of an application is an interventional X-ray imaging system for endoscopic cardiac or pulmonary surgery. In addition, fluoroscopic guidance procedures that incorporate preoperative CT imaging in the workflow are suitable for generating the proposed virtual stabilized views. Examples of clinical applications also include peripheral lung nodule biopsy and heart valve repair surgery.

[0117] In the operation of the imaging system, in one example, a DRR-like rendering similar to but not identical to fluoroscopy is provided. Thus, using the above proposed techniques, it is possible to achieve stabilization of the live view under large patient motion while displaying an image similar to fluoroscopy.

[0118] In one example, a preoperative 3D model of an individual patient's lungs or anatomy to be operated on is provided, such as by a data store. The model may be in the form of a point cloud, mesh, volume image (e.g., computed tomography), or other form.

[0119] In addition, live intraoperative imaging is provided. This can be in the form of 2D projected X-rays, such as fluoroscopy from a C-arm system, or 3D imaging, such as cone beam CT or ultrasound.

[0120] In one option, when surgical instruments or other objects are known to be present in the live intraoperative image but not in the preoperative image, the live contours of these objects are detected or segmented. Objects may include catheters, needles, forceps, implants, or others. For example, detection can be calculated using active contour segmentation of surgical instruments or objects; or threshold-based segmentation of surgical instruments or objects, or neural network-based object detection (YOLO, etc.) or segmentation (U-Net, etc.).

[0121] In addition, a preoperative 3D image / model and a single intraoperative image from a live feed may also be received as input, for example, by an image registration module. As output, it produces a separable transformation between the preoperative and intraoperative images, whose components are linear and nonlinear components. In one option, this registration is repeated for each fluoroscopic image generated in the live feed. The transformation exists in the preoperative image space and has the same dimensionality (e.g., 3D if it is a CT volume). This may involve any one or more of the following groups: gradient-based preoperative to intraoperative intensity-based registration; feature- or landmark-based preoperative to intraoperative registration; neural network-based preoperative to intraoperative image registration; and, for 3D preoperative images (e.g., CT) and 2D intraoperative images (e.g., fluoroscopy), any of the above methods may be used to perform registration between CT projections and 2D fluoroscopy.

[0122] For example, a preoperative 3D image is received as input through an image transformation module and a separable transformation is provided. This applies the linear component of the transformation to the preoperative image, bringing the background anatomical structure generated from the 3D model into a "stable" coordinate space (where the anatomical structure is stable relative to the imaging source). Its output is a stable background anatomical image. In the case of intraoperative 2D projection imaging (e.g., fluoroscopy), the generated stable image will be a DRR-like rendering based on the preoperative 3D image / model.

[0123] In the case of intraoperative 3D volume imaging such as CBCT, the generated stabilized image will be a rigid transformation of the preoperative 3D image / model.

[0124] Optionally, the resulting device / object segmentation is received as input, for example by an image transformation module. The inverse of the nonlinear component of the transformation is applied to the device / object segmentation, bringing the device / object into the same coordinate space in which the anatomy and the device are stable relative to the imaging source. The output is a transformed device / object. This can take the form of a segmentation or model-based overlay in a stabilized coordinate space; or a set of markers along key points on the device / object in a stabilized coordinate space; or an image overlay of the same modality as the intraoperative imaging of the device in a stabilized coordinate space.

[0125] Additionally, a stabilized background anatomical image and a stabilized device / object rendering are received as input, such as by a visualization module, and overlaid with post-processing, and a combined image is displayed on a display monitor.

[0126] In one example, specific to intraoperative fluoroscopy, the quality of intraoperative fluoroscopy is reduced to reduce radiation dose. Intraoperative fluoroscopy is used (only) for guidance of automatic image registration, not for high-quality visualization. Since stabilized renderings generated using high-resolution preoperative CT are displayed, the resolution / energy of the fluoroscopy can be reduced as long as the registration quality remains unchanged.

[0127] In an example, external hardware (e.g., electromagnetic (EM) tracking, shape sensing, etc.) is used to track a device or tool used in a procedure. In this case, it is not necessary to track a device visible under fluoroscopy, such as by an image processing controller. Instead, a registration step is provided to align the coordinate space of the tracked device with the imaging coordinate space. For example, a registered device is received by an image transformation module, the device is transformed and brought into a stabilized coordinate space, as described above.

[0128] The term "subject" may also be referred to as an individual. A "subject" may also be referred to as a patient, but it should be noted that the term does not indicate whether the subject actually has any disease.

[0129] In an example, a computer program comprising instructions is provided, which, when executed by a computer, causes the computer to perform the method of the aforementioned example. In an example, a computer program or a program element for controlling a device according to one of the aforementioned examples is provided, which, when executed by a processing unit, is suitable for performing the method steps of one of the aforementioned method examples.

[0130] In a further exemplary embodiment of the present invention, a computer program or a computer program element is provided, characterized in that it is adapted to perform the method steps of the method according to one of the preceding embodiments on a suitable system.

[0131] Therefore, the computer program element can be stored on a computer unit or distributed on more than one computer unit, which can also be a part of an embodiment of the present invention. The computing unit can be suitable for executing the steps of the above method or causing the execution of the steps of the above method. In addition, it can be suitable for operating the parts of the above device. The computing unit can be suitable for automatically operating and / or executing the user's command. The computer program can be loaded into the working memory of a data processor. Therefore, a data processor can be equipped to implement the method of the present invention.

[0132] Aspects of the present invention may be implemented in a computer program product, which may be a collection of computer program instructions stored on a computer-readable storage device that may be executed by a computer. The instructions of the present invention may be any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs), or Java classes. The instructions may be provided as a complete executable program, a partial executable program, as a modification (e.g., an update) to an existing program, or as an extension (e.g., a plug-in) to an existing program. Furthermore, portions of the processing of the present invention may be distributed across multiple computers or processors.

[0133] As described above, a processing unit, such as a controller, implements a control method. The controller can be implemented in a variety of ways using software and / or hardware to perform the various functions required. A processor is an example of a controller that uses one or more microprocessors that can be programmed using software (e.g., microcode) to perform the required functions. However, a controller can be implemented with or without a processor, and can also be implemented as a combination of dedicated hardware for performing some functions and a processor (e.g., one or more programmed microprocessors and associated circuits) for performing other functions.

[0134] Examples of controller components that may be used in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0135] This exemplary embodiment of the invention covers both a computer program that right from the beginning uses the invention and a computer program that by means of an update turns an existing program into a program that uses the invention.

[0136] Furthermore, the computer program element may be able to provide all necessary steps to implement the procedures of an exemplary embodiment of the method as described above.

[0137] According to another exemplary embodiment of the present invention, a computer readable medium, such as a CD-ROM, is provided, wherein the computer readable medium has a computer program element stored thereon, the computer program element being described in the previous section. The computer program may be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium provided with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0138] However, the computer program may also be provided over a network like the World Wide Web and may be downloaded from such a network into a working memory of a data processor. According to a further exemplary embodiment of the invention, a medium for making a computer program element available for downloading is provided, the computer program element being arranged to perform one of the previously described embodiments of the invention.

[0139] It must be noted that embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method claims, while other embodiments are described with reference to device claims. However, it can be concluded by a person skilled in the art from the above and following descriptions that, unless otherwise indicated, any combination of features relating to different subject matters, in addition to any combination of features belonging to the same type of subject matter, is also considered to be disclosed by the present application. However, all features can be combined to provide synergistic effects that exceed the simple sum of the features.

[0140] Although the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description should be considered illustrative or exemplary rather than restrictive. The present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and implemented by those skilled in the art in practicing the claimed invention by studying the drawings, the disclosure, and the dependent claims.

[0141] In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. A single processor or other unit may perform the functions of several items recited in the claims. Although specific measures are recited in mutually different dependent claims, this does not indicate that a collection of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A device (10) for guiding during a medical procedure, comprising: Data input unit (12); Data processor (14); as well as Output interface (16); Wherein, the data input unit is configured to: provide 3D image data of a region of interest of an object; and provide current 2D image data of the region of interest; wherein the data processor is configured to: register the current 2D image data with the 3D image data to determine a first transformation; identify a nonlinear component and a linear component of the determined first transformation; apply the identified linear component of the first transformation to the 3D image data; and generate a projected image from the 3D image data by applying the linear component to the 3D image data; and Wherein, the output interface is configured to provide the projection image as a guide during a medical procedure.

2. The device according to claim 1, wherein The data input section is configured to provide the 3D image data as preoperative 3D image data; wherein the data input unit is configured to provide the current 2D image data as 2D X-ray image data; wherein the data processor is configured to generate the projection image, the viewing direction of the projection image being aligned with the viewing direction of the 2D X-ray image data; and Therein, the data processor is configured to provide the projection image as a digitally reconstructed radiographic visualization.

3. The device according to claim 1 or 2, wherein: The data input section is configured to provide the current image, the current image comprising image data related to an interventional device inserted into the region of interest; and Wherein the data processor is configured to: perform segmentation on the current 2D image data to identify a representation of the device; apply a second transformation to the representation of the device; and combine the transformed representation of the device with the generated projection image.

4. The device according to any one of the preceding claims, wherein The data processor is configured to provide the second transform as an inverse of the non-linear component of the first transform.

5. The device according to any one of the preceding claims, wherein: The data processor is configured to superimpose a transformed representation of the device onto the generated projected image; and Therein, the data processor is configured to provide the transformed representation as a fluoroscopy-like overlay to the generated projection image.

6. The device according to any one of the preceding claims, wherein The data processor is configured to: provide the representation of the device comprising a segmented image portion of the 2D image; and apply the transformation to the segmented image portion.

7. Apparatus according to any one of the preceding claims, wherein: The data input is configured to provide a 3D model of the device suitable for a segmented representation of the device; and Wherein the data processor is configured to: apply the transformation to the 3D model of the device; and provide a projection of the model as an overlay on the generated projection image.

8. Apparatus according to any one of the preceding claims, wherein The data input is configured to provide tracking data of an external tracking device that tracks an interventional device inserted into the region; and Wherein, the data processor is configured to: track the interventional device relative to the object based on the tracking data; align the coordinate space of the tracked device with the imaging coordinate space; apply the second transformation to the graphical representation of the device; and combine the transformed representation of the device with the generated projection image.

9. The device according to any one of the preceding claims, wherein: The region of interest includes an anatomical structure, the anatomical structure including at least one of the following group: airway, lung, heart, and cardiovascular structure.

10. A system (100) for guiding during a medical intervention, comprising: Image data source (102); Medical imaging systems (104); Device (10) for guiding in a medical procedure according to any of the preceding claims; as well as Display device (106); wherein the image data source is configured to provide 3D image data of a region within a region of interest of the object; wherein the medical imaging system is configured to provide current 2D image data of the region of interest of the object; wherein the device is configured to provide a projection image generated based on the provided 3D image data and the provided current 2D image data; and Wherein, the display device is configured to present the projection image as a guide during a medical procedure.

11. The system according to claim 10, wherein: The medical imaging system is provided as an X-ray imaging system (108), the X-ray imaging system being configured to provide the current 2D image data as 2D X-ray image data; wherein the data processor is configured to generate the projection image, the viewing direction of the projection image being aligned with the viewing direction of the 2D X-ray image data; and Wherein, the X-ray imaging system is further configured to generate the 3D image data of the object.

12. The system according to claim 10 or 11, wherein: providing external tracking of the interventional device, the external tracking comprising at least one of the group of electromagnetic tracking and optical tracking; wherein electromagnetic tracking is applied to record and determine the transformations while the object remains in place; and Therein, when relative motion occurs, the current 2D image data is used for registration and determination of the transformation.

13. A method (200) for guiding during a medical procedure, comprising the steps of: providing (202) 3D image data of a region of interest of an object; providing (204) current 2D image data of the region of interest; registering (206) the current 2D image data with the 3D image data to determine a first transformation; identifying (208) a nonlinear component and a linear component of the determined first transform; applying (210) the identified linear component of the first transform to the 3D image data; generating (212) a projection image from the 3D image data by applying the linear component to the 3D image data; and The projection image is provided (214) as a guide during a medical procedure.

14. A computer program comprising instructions which, when said program is run by a computer, cause said computer to perform the method according to claim 13.

15. A computer readable medium having stored thereon the computer program product according to claim 14.

Images