Guidance during medical treatment

Abstract

The present invention relates to medical guidance. An apparatus 10 for guidance during a medical procedure is provided to provide an eased method for providing an improved image regarding a current situation. The apparatus has a data input unit 12, a data processor 14, and an output interface 16. The data input unit is configured to provide 3D image data of a region of interest of a subject and current 2D image data of the region of interest. The data processor is configured to align the current 2D image data with the 3D image data to determine a first transformation, identify non-linear and linear components of the determined first transformation, apply the identified linear component of the first transformation to the 3D image data, and generate a projection image from the 3D image data together with the linear component applied to the 3D image data. The output interface is configured to provide the projection image as guidance during a medical procedure.

Description

Technical Field

[0001] The present invention relates to medical guidance. In particular, the present invention relates to an apparatus for guidance during a medical procedure, a system for guidance during a medical intervention, and a method for guidance during a medical procedure.

Background Art

[0002] Image-guided endoscopic interventions are still difficult in regions of the body where image quality and clarity are distorted by the natural movements of the patient's body. As an example, bronchoscopy procedures can require a high level of skill to navigate through the airway and avoid important structures. One of the major obstacles to further improving the outcome of endoscopic techniques in these regions is the image distortion caused by both the natural periodic movements of the patient's body and the movements of the table and imaging system (especially in the case of a mobile fluoroscopy c-arm system). Fluoroscopy is used for intraoperative imaging guidance because of its simplicity of use, favorable field of view, and ability to visualize the pulmonary airways. However, complex structures such as the pulmonary airways are difficult to visualize and understand, especially under the influence of high-frequency periodic movements such as breathing and heart movements. During fluoroscopy-guided procedures, the movement of one part of the body relative to another part becomes difficult to resolve, and previously aligned static projections can become misaligned. For example, U.S. Patent US10682112B2 relates to the suppression of independent movements in a series of two-dimensional fluoroscopic images using a 3D preoperative volume. Examples include fluoroscopy-guided bronchoscopy where the patient's breathing can reduce the diagnostic yield of biopsies in the peripheral airways, prevent clear visualization of the target anatomical structures and surgical devices, and cardiac procedures such as valve repair where heart movement makes device-target confirmation difficult.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Accordingly, there may be a need to provide an enhanced method for providing an improved image of the current intervention situation.

Means for Solving the Problem

[0004] The object of the present invention is solved by the subject matter of the independent claims, and further embodiments are incorporated in the dependent claims. It should be noted that the aspects described below of the present invention are also applicable to an apparatus for guidance during medical treatment, a system for guidance during a medical intervention, and a method for guidance during medical treatment.

[0005] According to the present invention, an apparatus for guidance during medical treatment is provided. The apparatus has a data input unit, a data processor, and an output interface. The data input unit is configured to provide 3D image data of a region of interest of a subject. The data input unit is also configured to provide current 2D image data of the region of interest. The data processor is configured to register the current 2D image data with the 3D image data in order to determine a first transformation. The data processor is also configured to identify a non-linear component and a linear component of the determined first transformation. The data processor is further configured to apply the identified linear component of the first transformation to the 3D image data. The data processor is further configured to generate a projection image from the 3D image data together with the linear component applied to the 3D image data. The output interface is configured to provide the projection image as guidance during medical treatment.

[0006] As an effect, improved guidance is provided while avoiding an implied increase in radiation dose when, for example, increasing image resolution and frame rate. Virtual fluoroscopy incorporates patient-specific information for the purpose of view stabilization. As a further effect, virtual fluoroscopy has an advantage over virtual rendering in that it is not difficult to use because of its similarity to live fluoroscopy images. Another effect is an increase in reliability on the display on the user's side, e.g., the surgeon. Also, this addresses complex navigation in tortuous moving blood vessels / airways.

[0007] According to one example, the data input unit is configured to provide 3D image data as preoperative 3D image data. Optionally, preoperative CT image data is provided. The data input unit is configured to provide current 2D image data as 2D X-ray image data. The data processor is configured to generate a projection image having an observation direction aligned with the observation direction of the 2D X-ray image data. Optionally, the data processor is configured to provide the projection image as a digitally reconstructed radiograph visualization.

[0008] According to one example, the data input unit is configured to provide a current image having image data regarding an intervention device inserted into the region of interest. The data processor is configured to perform segmentation on the current 2D image data to identify the representation of the device. The data processor is also configured to apply a second transformation to the representation of the device. The data processor is further configured to combine the transformed representation of the device with the generated projection image.

[0009] According to one example, the data processor is configured to provide the second transformation as the reciprocal of the non-linear component of the first transformation.

[0010] According to one example, the data processor is configured to overlay the transformed representation of the device onto the generated projection image. Optionally, the data processor is configured to provide the transformed representation as a fluoro-like overlay onto the generated projection image.

[0011] According to one example, the data processor is configured to provide a representation of the device having a segmented image portion of the 2D image. The data processor is also configured to apply a transformation to the segmented image portion.

[0012] According to one example, the data input is configured to provide tracking data of an external tracking device that tracks an interventional device inserted in the region. The data processor is configured to track the interventional device relative to the subject based on the tracking data. The data processor is also configured to align a coordinate space of the tracked device with the imaging coordinate space. The data processor is further configured to apply a second transformation to the graphical representation of the device. The data processor is further configured to combine the transformed representation of the device with the generated projection image.

[0013] According to the invention, a system for guidance during a medical intervention is also provided. The system comprises an image data source, a medical imaging system, an apparatus for guidance during a medical procedure according to one of the previous examples, and a display arrangement. The image data source is configured to provide 3D image data of a region of interest of the object. The medical imaging system is configured to provide current 2D image data of the region of interest of the object. The apparatus for guidance during a medical procedure is configured to provide a generated projection image based on the provided 3D image data and the provided current 2D image data. The display device is configured to present the projection image as guidance during the medical procedure.

[0014] According to one example, the medical imaging system is provided as an X-ray imaging system configured to provide current 2D image data as 2D X-ray image data. Optionally, the data processor is configured to generate a projection image having an observation direction aligned with the observation direction of the 2D X-ray image data. In a further option, the X-ray imaging system is also configured to generate 3D image data of the subject.

[0015] According to one example, external tracking of an interventional device having at least one of a group of electromagnetic tracking and optical tracking is provided. Electromagnetic tracking is applied for alignment and transformation determination when the subject remains in a predetermined position. Current 2D image data is used for alignment and transformation determination when relative movement occurs.

[0016] According to the present invention, a guidance method during a medical procedure is also provided. The method includes the following steps, namely, providing 3D image data of a region of interest of a subject; providing current 2D image data of the region of interest; aligning the current 2D image data with the 3D image data to determine a first transformation; identifying a non-linear component and a linear component of the determined first transformation; applying the identified linear component of the first transformation to the 3D image data; generating a projection image from the 3D image data together with the linear component applied to the 3D image data; providing the projection image as guidance during the medical procedure; and.

[0017] In one example, the generated image is generally an X-ray image, i.e., from its appearance, it mimics a fluoroscopic X-ray image. The projection image simulates a so-called live view, resulting in an improved level of confidence on the user's side, e.g., a surgeon. Instead of an actual X-ray image, the projection image provides an image that looks like an X-ray image. Thus, this device simulates an X-ray imaging device for live X-ray imaging.

[0018] According to one aspect, there is provided generating a virtual stabilized view of live fluoroscopy from patient-specific pre-operative imaging.

[0019] In one example, a software package is provided to be integrated with C-arm hardware. In another example, a stand-alone controller is provided that communicates with a C-arm system and an Image Archiving and Communication (PAC) system.

[0020] These and other aspects of the invention will become apparent from and be elucidated with reference to the embodiments described hereinafter.

[0021] Exemplary embodiments of the invention are described hereinafter with reference to the following drawings.

Brief Description of the Drawings

[0022]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Best Mode for Carrying Out the Invention

[0023] Here, specific embodiments will be described in detail with reference to the accompanying drawings. In the following description, like reference numerals are used for like elements in different drawings. Matters defined in this specification, such as detailed configurations and elements, are provided to assist in an overall understanding of the exemplary embodiments. Also, well-known functions or configurations are not described in detail as they would obscure the embodiments with unnecessary details. Further, expressions such as "at least one of" when preceding a list of elements modify the entire list of elements and not individual elements of the list.

[0024] FIG. 1 schematically shows an example of an apparatus 10 for guidance during a medical procedure. The apparatus 10 has a data input unit 12, a data processor 14, and an output interface 16. The data input unit 12 is configured to provide 3D image data of a region of interest of a subject. The data input unit 12 is also configured to provide current 2D image data of the region of interest. The data processor 14 is configured to register the current 2D image data with the 3D image data to determine a first transformation. The data processor 14 is also configured to identify the non-linear component and the linear component of the determined first transformation. The data processor 14 is also configured to apply the identified linear component of the first transformation to the 3D image data. The data processor 14 is further configured to generate a projection image from the 3D image data together with the linear component applied to the 3D image data. The output interface 16 is configured to provide the projection image as guidance during a medical procedure.

[0025] The data input unit 12, the data processor 14, and the output interface 16 can be provided in a common structure such as a common housing or in an integrated manner, as shown by frame 18. In a further option (not shown), they are provided as separate components or units.

[0026] The first arrow 20 indicates the supply of data to the data input unit 12, that is, the provision of 3D image data. The second arrow 22 indicates another supply of data to the data input unit 12, that is, the provision of current 2D image data. The third arrow 24 indicates the supply of data from the output interface 16, that is, the provision of a projected image. The data supply can be provided either wired or wirelessly. In one example, as an option, a display 26 is provided to present an extended first image. The display 26 is data-connected to the output interface 16.

[0027] The first transformation can also be referred to as a transformation, an image data transformation, a primary transformation, a main transformation, or a situation transformation.

[0028] The term "3D image data" relates to the spatial data of an object obtained by 3D medical imaging procedures, such as ultrasonic imaging, X-ray imaging, or MRT imaging.

[0029] The term "current 2D image data" relates to image data that is provided in the current state, for example, as a live image during a medical treatment or intervention. The image data is provided within an image plane as 2D image data.

[0030] The term "register" relates to calculating the spatial relationship between two different image data sets. The spatial relationship has information on how to manipulate each other's data for spatial matching. Registration includes a linear registration part, that is, the global registration of 2D image data within 3D image data. Registration also includes a non-linear registration part, that is, the morphing registration process of 2D image data to 3D image data. Linear registration relates to different viewing angles and distances, for example, caused by the movement of the object support. Non-rigid or non-linear registration relates to the deformation of the object itself, for example, caused by other activities such as breathing or organ movement, especially including heartbeat.

[0031] The term "transformation" relates to defining how, i.e., in what way it needs to be changed in a broad sense, 2D image data is transformed in order to be aligned with a 3D data set.

[0032] The term "linear" relates to a linear registration part, or any subset of linear transformations such as an affine transformation or a rigid transformation.

[0033] The term "non-linear" relates to the remaining part of the transformation not covered by the "linear" part. The non-linear component of the transformation relates to the morphing of tissue to achieve registration.

[0034] The term "projected image" relates to an image generated by projecting an object or objects onto a projection surface or projection plane. Thus, structures existing within the projection volume can contribute to the projected image. An example of a projected image is an X-ray emission image.

[0035] The term "generating a projected image" relates to an artificially generated image that mimics, for example, an X-ray image.

[0036] The term "data input unit" relates to providing or supplying data for data processing steps. The data input unit can also be referred to as an image data input unit. The data input unit can also be referred to as a data supply unit, an image data supply unit, an image input unit, an input unit, or simply an input. In one example, the image data input unit is data-connectable to an image source device. In one example, the data input unit is data-connectable to a data storage device storing image data.

[0037] The term "data processor" relates to a processor or a part of a processor device provided to execute computational steps using the data supplied by the data input unit. The data processor can also be referred to as a data processing device, a processor unit, or a processor. In one example, the data processor is data-connected to the data input unit and an output interface.

[0038] The term "output interface" relates to an interface for providing data that has been processed or calculated for further purposes. The output interface can also be referred to as an output part or output unit. In one example, the output interface can be data-connected to a display configuration or a display device. In another example, the output part is data-connected to a display.

[0039] In one example, stabilization of a fluoroscopic view is provided using patient-specific CT-derived virtual fluoroscopy.

[0040] According to one aspect, a patient's high-resolution preoperative CT is used to generate a virtual fluoroscopic view that mimics a live fluoroscopic view with stabilized distorted motion. So-called view stabilization is performed by aligning 2D-3D fluoroscopy to CT using separated linear and non-linear components, and the separated transformations are used to estimate the stabilized view.

[0041] The fluoroscopy-like view is generated using patient-specific information, which is stabilized by image alignment of the preoperative CT volume to live fluoroscopy. The result is a live image similar to the actual fluoroscopy of anatomical structures and surgical devices that is stable with respect to the virtual X-ray source. This view is comfortable for the clinician and solves the problem of motion during high-precision procedures.

[0042] A further advantage of this method is that for surgeries that require visualization of well-defined anatomical structures such as the airways or cardiac chambers, lower-resolution fluoroscopic images can be sufficient for the purpose of aligning CT to fluoroscopy. In this case, live fluoroscopy functions as a guide for alignment rather than for high-resolution visualization. Assuming accurate alignment, the task of rendering high-quality images can be offloaded to the virtual fluoroscopy or DRR generation process rather than the live image to enable lower intraoperative radiation usage.

[0043] According to one example, this is provided for using DRR-like visualization for the application of fluoroscopy-guided bronchoscopy of the lungs, and for using the patient's own high-resolution preoperative CT scan to generate a virtual stabilized view of live fluoroscopy, in order to enhance the live fluoroscopy view after alignment from fluoroscopy to a reference image, but rather to propose generating a view reconstructed from the patient-specific CT scan at the estimated C-arm position, where the distortion motion (which reaches the separation of the linear and non-linear components of the alignment from fluoroscopy to CT) is subtracted. Optionally, any surgical device can be inserted into the reconstructed virtual view using existing methods for segmenting live and simulating realistic catheters in fluoroscopy. This generates a stabilized view where the anatomical structures and devices are not moved relative to the X-ray source, while maintaining a fluoroscopy-like view and patient-specific information with which the clinician can work comfortably.

[0044] As an advantage, the virtual fluoroscopy visualization provides a stabilized view while incorporating detailed patient-specific anatomical structures or images, and also improves the clinician's confidence. As an effect, motion compensation for stabilizing the view for procedures requiring high-precision or complex navigation is provided.

[0045] In one example, the data input unit 12 is configured to provide 3D image data as preoperative 3D image data. The data input unit 12 is configured to provide current 2D image data as 2D X-ray image data. The data processor 14 is configured to generate a projection image in an observation direction aligned with the observation direction of the 2D X-ray image data. The data processor 14 is configured to provide the projection image as a digitally reconstructed radiography visualization.

[0046] In one example, the 2D X-ray image data is acquired using an X-ray imaging system, such as a C-arm device. The 2D X-ray image data is acquired at an imaging position relative to the subject. The projection image is generated in a line-of-sight direction corresponding to the relative imaging position.

[0047] In one example, the data input unit 12 is configured to provide a current image having image data regarding an interventional device inserted into the region of interest. The data processor 14 is configured to perform segmentation on the current 2D image data so as to identify the representation of the device. The data processor 14 is configured to apply a second transformation to the representation of the device. The data processor 14 is also configured to combine the transformed representation of the device with the generated projection image.

[0048] The second transformation can also be referred to as a transformation, a segmentation transformation, a secondary transformation, a minor transformation or an auxiliary transformation, or a device transformation.

[0049] The interventional device may be a catheter, a needle, forceps or an implant.

[0050] In one example, the current image has image data related to an identifiable structure that does not exist within the 3D image data, and the data processor performs segmentation on the current 2D image data to determine the identifiable structure, applies a second transformation to the determined identifiable structure, and combines the transformed determined identifiable structure with the generated projection image.

[0051] In one example, the data processor 14 is configured to provide the second transformation as the inverse of the non-linear component of the first transformation.

[0052] In one example, the data processor 14 is configured to overlay the transformed representation of the device on the generated projection image. The data processor 14 is configured to provide the transformed representation as a fluoroscopy-like overlay on the generated projection image.

[0053] In one example, the data processor 14 is configured to provide a representation of the device having a segmented image portion of the 2D image. The data processor 14 is configured to apply a transformation to the segmented image portion.

[0054] In one example, the data input unit 12 is configured to provide a 3D model of the device that is fitted to the segmented representation of the device. The data processor 14 is configured to apply a transformation to the 3D model of the device. The data processor 14 is also configured to provide a projection of the model that is overlaid on the generated projection image.

[0055] In another example, image data of the 3D model is added to the 3D image data prior to generating the projected image.

[0056] In one example, the data input 12 is configured to provide tracking data of an external tracking device that tracks an interventional device inserted within the region. The data processor 14 is configured to track the interventional device relative to the object based on the tracking data. The data processor 14 is configured to align a coordinate space of the tracked device to the imaging coordinate space. The data processor 14 is configured to apply a second transformation to the graphical representation of the device. The data processor 14 is configured to combine the transformed representation of the device with the generated projection image.

[0057] In one example, the region of interest comprises an anatomical structure comprising at least one of the group of airways, lungs, heart, and cardiovascular structures.

[0058] 2 shows an example of a system 100 for guidance during a medical intervention. The system 100 comprises an image data source 102, a medical imaging system 104, an apparatus 10 for guidance during a medical procedure according to one of the previous examples, and a display arrangement 106. The image data source 102 is configured to provide 3D image data of a region of interest of the object. The medical imaging system 104 is configured to provide current 2D image data of the region of interest of the object. The apparatus 10 is configured to provide a generated projection image based on the provided 3D image data and the provided current 2D image data. The display device 106 is configured to present the projection image as guidance during the medical procedure.

[0059] In one example, the image data source 102 is a data storage device storing 3D CT image data of the subject. Optionally, the image data source 102 is a CT system data-connected to a device for guidance during medical procedures.

[0060] In the option shown in FIG. 2, the medical imaging system 104 is provided as an X-ray imaging system 108 configured to provide current 2D image data as 2D X-ray image data. The data processor 14 is configured to generate a projection image in an observation direction aligned with the observation direction of the 2D X-ray image data. Optionally, additionally or alternatively, the X-ray imaging system 108 is also configured to generate 3D image data of the subject.

[0061] The X-ray imaging system 108 is provided as a C-arm imaging system having an X-ray source 110 and an X-ray detector 112 attached to both ends of a movably attached C-arm 114.

[0062] Optionally, the X-ray imaging system 108 is also provided for acquiring 3D image data of the subject. In another example, the X-ray imaging system 108 is a mobile C-arm system.

[0063] In FIG. 2, a subject support 116 is provided. Further, a control interface 118 is provided adjacent to the subject support 116. The subject 120 is placed on the subject support 116. Further, an intervention device 122 is provided partially inserted into the subject 120.

[0064] Console 124 is shown in the foreground. Console 124 is configured to provide user interaction and control options. Console 124 has a set of displays, a keyboard with a mouse, a graphics tablet, and control knobs, etc. Console 124 enables the control of various functions and operations of system 100 for guiding the interventional imaging device. The device 10 for guiding the interventional imaging device can be integrated with Console 124 or configured as a separate device.

[0065] Image data source 102 is data-connected to device 10 for guiding the interventional imaging device, as shown by the first data connection line 126. The device 10 for guiding the interventional imaging device is further data-connected to the medical imaging system 104, as shown by the second data connection line 128. The data connection is provided either wired or wirelessly. The device 10 for guiding the interventional imaging device is further data-connected to Console 124, as shown by the third data connection line 130.

[0066] In one example, external tracking having at least one of the group of electromagnetic tracking and optical tracking of the interventional device is provided. Electromagnetic tracking is applied for alignment and transformation determination when the object remains at a predetermined position. The current 2D image data is used for alignment and transformation determination when relative motion occurs.

[0067] When the object remains at a predetermined position, relative motion does not occur. Using electromagnetic tracking results in a reduced radiation dose. In one example, an external tracking device is provided for tracking the interventional device inserted into the region of interest. The data processor 14 is configured to track the interventional device with respect to the object, align the coordinate space of the tracked device with the imaging coordinate space, apply a second transformation to the graphical representation of the device, and combine the transformed representation of the device with the generated projection image, based on the data from the external tracking device.

[0068] According to one aspect, a stabilized view is provided by using 3D image data having improved resolution and increased detail information, generating a projection image from this 3D data, and using current (actual) image data to determine a perspective view. Optionally, the current (actual) image data is also used to detect devices or other structures that do not exist in the 3D image data and transfer them to the projection of the 3D image data. In other words, 3D image data, such as preoperative or intraoperative image data, is used to provide improved guidance. The current 2D image is used to update the 3D image data to the current situation and thus provide current guidance, i.e., navigation support.

[0069] Figure 3 shows the basic steps of an example of a method 200 for guidance during a medical procedure. The method 200 has the following steps. In a first step 202, 3D image data of the region of interest of the subject is provided. In a second step 204, current 2D image data of the region of interest is provided. In a third step 206, the current 2D image data is aligned with the 3D image data to determine a first transformation. In a fourth step 208, the non-linear and linear components of the determined first transformation are identified. In a fifth step 210, the identified linear component of the first transformation is applied to the 3D image data. In a sixth step 212, a projection image is generated from the 3D image data having the linear component applied to the 3D image data. In a seventh step 214, the projection image is provided as guidance during the medical procedure.

[0070] The first step 202 and the second step 204 can also be provided simultaneously or in the reverse order.

[0071] In one example, the following steps are provided. A preoperative CT volume and live fluoroscopy images are provided, and CT-fluoroscopy image registration is performed. Optionally, if surgical devices are present in the images, segmentation of those devices is performed. Separation of the non-linear and linear components of the transformation calculated in the registration is provided. The linear component of the transformation is applied to the preoperative CT volume, and a digitally reconstructed radiograph (DRR) is generated from this angle. Note that moving the C-arm necessarily generates a rigid body motion of the image volume, which is a type of linear motion. Optionally, if surgical devices are present in the images, the inverse of the non-linear component of the transformation from the separation is applied to the segmented devices, and any other object segmented in the live fluoroscopy image that would not be present in the preoperative CT is overlaid on the DRR generated using either a model or a fluoroscopy-like overlay. Optionally, a post-processing algorithm from the image processing module of the C-arm may be applied.

[0072] In one example of the method, the 3D image data is preoperative 3D image data. Optionally, the preoperative 3D image data is preoperative CT image data. The projection image is a digitally reconstructed radiograph visualization.

[0073] In one example of method 200, the current 2D image data is provided as 2D X-ray image data. The projection image is generated in an observation direction aligned with the observation direction of the 2D X-ray image data.

[0074] In one example, the current image has image data regarding a distinguishable structure that does not exist within the 3D image data. Method 200 includes performing segmentation on the current 2D image data to determine the distinguishable structure, applying a second transformation to the determined distinguishable structure, and combining the transformed determined distinguishable structure with the generated projection image.

[0075] ​ In one example of method 200, the current image has image data regarding an interventional device inserted into the region of interest. There are further provided steps of performing segmentation of the current 2D image data to identify a representation of the device, applying a second transformation to the representation of the device, and combining the transformed representation of the device with the generated projection image.

[0076] In one example of method 200, the second transformation is provided as the inverse of the non-linear component of the first transformation.

[0077] In one example of method 200, the transformed representation of the device is overlaid on the generated projection image.

[0078] In one example of method 200, the representation of the device has a segmented image portion of the 2D image. Further, the transformation is applied to the segmented image portion. In one example of method 200, the transformed representation is provided as a fluoroscopy-like overlay on the generated projection image.

[0079] In one example of method 200, a 3D model of the device adapted to the segmented representation of the device is provided. The transformation is applied to the 3D model of the device. Further, a projection of the model overlaid on the generated projection image is provided.

[0080] In one example of method 200, an interventional device inserted into the region of interest is tracked by an external tracking device. There are provided steps of tracking the interventional device with respect to the subject, aligning the coordinate space of the tracked device with the imaging coordinate space, applying a second transformation to the graphical representation of the device, and combining the transformed representation of the device with the generated projection image.

[0081] In one example of the method, the region of interest has an anatomical structure having at least one of a group of an airway, a lung, a heart, and a cardiovascular structure.

[0082] Figure 4 shows an example of a workflow for view stabilization to guide an interventional imaging device. Pre-operative CT data, i.e., CT volume 302, and live fluoroscopy image 304 are provided. Transformation parameters R·φ related to the alignment of the CT-fluoroscopy images, indicated by the first arrow 306, are provided. Next, the device in live fluoroscopy is segmented (308). Further, the linear component R of the transformation and the non-linear component φ of the transformation are separated as shown by a first separation arrow 310 for the linear transformation component R and a second separation arrow 312 for the non-linear transformation component φ. The linear component is applied to the CT volume to generate a stabilized view 314 in the form of a DRR. The inverse of the non-linear component φ -1 is applied to the segmented device to overlay the segmented device onto the stabilized DRR, which is also described as the casting device segmentation to the stabilized view, indicated by a further arrow 316.

[0083] Mathematically formulated, the workflow can be described as follows, where the inputs are as follows. I0 = D model or CT image F = live fluoroscopy image R = linear transformation component φ = non-linear transformation component D(x) = segmented object (point set)

[0084] For a particular live fluoroscopy image F and CT volume I0, the transformation R·φ between F and I0 is calculated as a reference for non-linear images for image alignment. The stabilized background image is generated by transforming the CT volume by R so as to generate I0(R·x). Forward projection is performed through I0(R·x) to generate a DRR background.

[0085] Figure 5 shows examples of the stabilized view 346 in the left column and the respective live views 348 in the right column. The first arrow 350 indicates the non-linear deformation caused by the patient's movement. The second arrow 352 indicates the linear deformation caused by the C-arm / table movement. Anatomical structures 354 such as blood vessel structures may undergo deformation, for example due to movement.

[0086] Figure 5 shows the effect of stabilization on the background anatomical structure. D(x) is calculated from I0 as a set of coordinate points (e.g., points along a catheter), transformed into the DRR background space by the inverse of the non-linear transformation component, and D(x)·φ -1 is generated and overlaid on the DRR background.

[0087] Figure 5 shows the effect of stabilization on the anatomical structure in fluoroscopy. A sequence of four video frames of live fluoroscopy is shown from a preclinical study on a pig. As described above, the left column shows the stabilized virtual view and the right column shows the live fluoroscopy. Between frame 1 and frame 2 (counting from the top), the C-arm is rotated to slightly different angles. Between frame 2 and frame 3, the subject inhales and the diaphragm and lungs move significantly in the live view. Between frame 3 and frame 4, the subject exhales and shows movement again. The stabilized background view generated from the subject's preoperative CT by I0(R·x) shows tracking of the rigid C-arm movement but does not move in response to the subject's anatomical movement, enabling a more stable view of the pulmonary airway during the procedure.

[0088] FIG. 6 shows an example of a further workflow for view stabilization to guide an interventional imaging device. The first image 380 shows a theoretical "stable" live view with the device 382 superimposed. The second image 384 shows that movement occurs in the live view as indicated by the arrow 386. Further, a preoperative CT volume 388 is provided. A further image 390 shows a stable view generated from the CT. FIG. 6 shows the process of superimposing an undistorted surgical tool onto a stabilized anatomical image. FIG. 6 shows the process of superimposing a device or other object that does not exist in the preoperative imaging onto a stabilized view. Starting from the top, an attempt is made to recover a theoretical stabilized live fluoroscopy by segmenting the image distorted by movement (the second image), correcting the distortion of the device using the inverse of the non-linear transformation component, and superimposing it onto the stable DRR when in the same coordinate space (the last image).

[0089] One example of the use is in an interventional X-ray imaging system for endoscopic heart or lung procedures. Further, a fluoroscopy-guided procedure that incorporates preoperative CT imaging into the workflow is suitable for generating the proposed virtual stabilized views. Examples of clinical uses also include peripheral lung nodule biopsy and cardiac valve repair surgery.

[0090] In the operation of the imaging system, in one example, a DRR-like rendering that is similar but not identical to fluoroscopy is provided. Thus, stabilization of the live view under large patient movement is provided using the proposed techniques above while displaying a fluoroscopy-like image.

[0091] In one example, a preoperative 3D model of the lung or anatomical structure of the individual patient to be operated on is provided, for example, by a data storage device. This model can take the form of a point cloud, a mesh, a volume image (e.g., computed tomography), or other forms.

[0092] Furthermore, intraoperative images are provided. This can take the form of 2D projection X-rays such as fluoroscopy from a C-arm system, or 3D imaging such as cone beam CT or ultrasound.

[0093] Optionally, in the presence of surgical devices or other objects that are known to be present in the intraoperative images but not in the preoperative images, the real-time contours of these objects are detected or segmented. The objects can include catheters, needles, forceps, implants, or others. Detection can be calculated, for example, using active contour segmentation of the surgical device or object, or threshold-based segmentation of the surgical device or object, or neural network-based object detection (such as YOLO) or segmentation (such as U-Net).

[0094] Furthermore, the preoperative 3D image / model and a single intraoperative image from the live feed are received as inputs, for example, by an image registration module. As an output, a separable transformation between the preoperative and intraoperative images, whose components are the linear component and the non-linear component, is generated. Optionally, this registration is repeated for each fluoroscopy image generated in the live feed. The transformation exists in the space of the preoperative image and has the same dimension (e.g., 3D in the case of a CT volume). This can include any one or more of the groups of the above methods for registration between preoperative and intraoperative images, including gradient-based intensity-based registration from preoperative to intraoperative, feature-based or landmark-based registration from preoperative to intraoperative, neural network-based image registration from preoperative to intraoperative, and in the case of 3D preoperative images such as CT and 2D intraoperative images such as fluoroscopy, registration between the CT projection and the 2D fluoroscopy.

[0095] For example, it is provided that an image conversion module receives a preoperative 3D image and a separable conversion as inputs. This applies the linear component of the conversion to the preoperative image and brings the background anatomical structure generated from the 3D model into a "stabilized" coordinate space (where the anatomical structure is stable with respect to the imaging source). The output of this is a stabilized background anatomical image.

[0096] In the case of intraoperative 2D projection imaging such as fluoroscopy, the generated stabilized image is a DRR-like rendering from the preoperative 3D image / model. In the case of intraoperative 3D volumetric imaging such as CBCT, the generated stabilized image is a rigid transformation of the preoperative 3D image / model.

[0097] Optionally, the generated device / target segmentation is received as an input, for example, by an image conversion module. The inverse of the non-linear component of the conversion is applied to the device / target segmentation, bringing the device / target into the same coordinate space where the anatomical structure and the device are stable with respect to the image source. The output of this is the transformed device / target. This can take the form of a segmentation or model-based overlay in the stabilized coordinate space, or a set of markers for key points along the device / target in the stabilized coordinate space, or an image overlay of the same modality as the intraoperative imaging of the device in the stabilized coordinate space.

[0098] Furthermore, the stabilized background anatomical image and the stabilized device / target rendering are received as inputs, for example, by a visualization module, overlaid with post-processing, and the assembled image is displayed on a display monitor.

[0099] In one example, specific to intraoperative fluoroscopy, the quality of the intraoperative fluoroscopy is reduced to reduce radiation dose. Intraoperative fluoroscopy is used (only) as a guide for automatic image registration, not for high quality visualization. Since stabilized renderings generated from high resolution preoperative CT are used for display, the fluoroscopy resolution / energy can be reduced as long as registration quality is not compromised.

[0100] In one example, a device or tool used during a procedure is tracked using external hardware, e.g., electromagnetic (EM) tracking, shape sensing, etc. In this case, it is not necessary to track the device visible under fluoroscopy, e.g., by an image processing controller. Instead, a registration step is provided to align the coordinate space of the tracked device with the imaging coordinate space. The registered device is received, e.g., by an image transformation module, which transforms and brings the device into a stable coordinate space, as described above.

[0101] The term "subject" may also be referred to as an individual. A "subject" may also be referred to as a patient, although it should be noted that this term does not indicate whether any disease or illness is actually present in the subject.

[0102] In one example, a computer program is provided having instructions which, when executed by a computer, cause the computer to perform the method of the previous example. In one example, a computer program or program element for controlling an apparatus according to one of the above examples is provided, which program or program element is configured to perform the method steps of one of the above method examples when executed by a processing unit.

[0103] In another exemplary embodiment of the invention, a computer program or a computer program element is provided, characterized in that it is arranged for, on a suitable system, to carry out the method steps of the method according to one of the previous embodiments.

[0104] Accordingly, the computer program elements may be stored in a computer unit, or may be distributed over more than one computer unit which may be part of an embodiment of the invention. This computing unit may be configured to execute or induce the execution of the steps of the method described above. Further, it may be configured to operate the components of the device described above. The computing unit can be configured to operate automatically and / or to execute a user order. The computer program may be loaded into the working memory of the data processor. Accordingly, the data processor may be configured to execute the method of the invention.

[0105] Aspects of the invention may be implemented in a computer program product which may be a set of computer program instructions stored on a computer-readable storage device executable by a computer. The instructions of the invention may be in any interpretable or executable code mechanism including, but not limited to, a script, an interpretable program, a dynamic link library (DLL) or a Java class. The instructions may be provided as a complete executable program, a partial executable program, a modification (e.g., an update) to an existing program or an extension (e.g., a plug-in) to an existing program. Further, parts of the processing of the invention may be distributed over a plurality of computers or processors.

[0106] As described above, a processing unit, such as a controller, implements a control method. This controller can be implemented in various ways using software and / or hardware to perform the various required functions. A processor is an example of a controller that uses one or more microprocessors programmed using software (e.g., microcode) to perform the required functions. However, the controller may be implemented with or without using a processor, and may be implemented as a combination of dedicated hardware for performing some functions and a processor (e.g., one or more programmed microprocessors and associated circuitry) for performing other functions.

[0107] Examples of components of a controller used in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0108] This exemplary embodiment of the present invention includes both a computer program that uses the present invention from the start and a computer program that changes an existing program to a program that uses the present invention by means of an update.

[0109] Furthermore, the computer program elements may be able to provide all the steps necessary to satisfy the procedures of the exemplary embodiments of the methods described above.

[0110] According to a further exemplary embodiment of the present invention, a computer-readable medium such as a CD-ROM is presented, the computer-readable medium having computer program elements stored thereon, the computer program elements being described by the foregoing sections. The computer program can be stored and / or distributed on a suitable medium such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless communication systems.

[0111] However, the computer program may be presented on a network such as the World Wide Web and can be downloaded from such a network into the working memory of a data processor. According to a further exemplary embodiment of the present invention, a medium for making the computer program elements downloadable is provided, the computer program elements being configured to execute a method according to one of the foregoing embodiments of the present invention.

[0112] It should be noted that the embodiments of the present invention are described with reference to different subjects. In particular, some embodiments are described with reference to method-type claims and other embodiments are described with reference to apparatus-type claims. However, those skilled in the art will understand from the above and the following description that, unless otherwise notified, any combination of features belonging to one type of subject, in addition to any combination of features relating to different subjects, is also disclosed in this application. However, all features can be combined to provide a synergistic effect higher than the simple sum of the features.

[0113] Although the present invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the dependent claims.

[0114] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. Data input section, Data processor, Output interface, In a device for guidance during medical procedures, The data input unit is configured to provide 3D image data of the target region of interest and to provide current 2D image data of the region of interest. The data processor is configured to align the current 2D image data with the 3D image data in order to determine a first transformation, to identify the nonlinear and linear components of the determined first transformation, to apply the identified linear component of the first transformation to the 3D image data, and to generate a projected image from the 3D image data together with the linear component applied to the 3D image data. The output interface is configured to provide the projected image as guidance during medical procedures. The data input unit is configured to provide the 3D image data as preoperative 3D image data. The data input unit is configured to provide the current 2D image data as 2D X-ray image data. The data processor is configured to generate a projection image having an observation direction aligned with the observation direction of the 2D X-ray image data. The data processor is configured to provide the projection image as a digitally reconstructed X-ray image visualization. The data input unit is configured to provide the current image having image data relating to the intervention device inserted into the region of interest. The data processor is configured to perform segmentation on the current 2D image data to identify the representation of the device, apply a second transformation to the representation of the device, and combine the transformed representation of the device with the generated projection image. Device.

2. The apparatus according to claim 1, wherein the data processor is configured to provide the second transformation as the inverse of the nonlinear component of the first transformation.

3. The data processor is configured to overlay the converted representation of the device onto the generated projection image. The data processor is configured to provide the converted representation as a fluorescence-like overlay on the generated projection image. The apparatus according to claim 1.

4. The apparatus according to claim 1, wherein the data processor is configured to provide a representation of the apparatus having segmented image portions of the 2D image, and to apply the transformation to the segmented image portions.

5. The data input unit is configured to provide a 3D model of the device that is adapted to the segmented representation of the device. The data processor is configured to apply the transformation to the 3D model of the device and to provide a projection of the model overlaid on the generated projection image. The apparatus according to claim 1.

6. The data input unit is configured to provide tracking data from an external tracking device that tracks the intervention device inserted into the region. The data processor is configured to track the intervention device with respect to the target based on the tracking data, align the coordinate space of the tracked device with the imaging coordinate space, apply the second transformation to the graphic representation of the device, and combine the transformed representation of the device with the generated projection image. The apparatus according to claim 1.

7. The apparatus according to claim 1, wherein the region of interest has anatomical tissue having at least one of the groups of airways, lungs, heart, and cardiovascular structures.

8. Image data source and Medical imaging system, A device for guidance during a medical procedure according to any one of claims 1 to 7, Display device and In a system for induction during medical intervention, which has the following features: The aforementioned image data source is configured to provide 3D image data of the region of interest, The medical imaging system is configured to provide current 2D image data of the region of interest of the subject, The apparatus is configured to provide the generated projection image based on the provided 3D image data and the provided current 2D image data. The display device is configured to present the projected image as guidance during medical procedures. system.

9. The medical imaging system is provided as an X-ray imaging system configured to provide the current 2D image data as 2D X-ray image data. The data processor is configured to generate the projection image having an observation direction aligned with the observation direction of the 2D X-ray image data. The X-ray imaging system is configured to also generate the 3D image data of the target. The system according to claim 8.

10. External tracking is provided, having at least one of the electromagnetic tracking and optical tracking groups of the intervention device. The electromagnetic tracking is applied for alignment and determination of the transformation when the object remains in a predetermined position. The aforementioned current 2D image data is used for alignment and determining the transformation when relative motion occurs. The system according to claim 8.

11. In methods for guidance during medical procedures, The steps include providing 3D image data of the target region of interest, The steps include providing current 2D image data of the region of interest, A step of aligning the current 2D image data with the 3D image data so as to determine a first transformation, The steps include identifying the nonlinear and linear components of the first transformation determined above, The steps include applying the identified linear component of the first transformation to the 3D image data, A step of generating a projected image from the 3D image data together with the linear component applied to the 3D image data, The steps include providing the projected image as guidance during a medical procedure, It has, The aforementioned 3D image data is provided as preoperative 3D image data. The aforementioned current 2D image data is provided as 2D X-ray image data. The projection image is generated in an observation direction aligned with the observation direction of the 2D X-ray image data. The aforementioned projection image is provided as a digitally reconstructed X-ray image visualization. The current image is provided, which has image data relating to the intervention device inserted into the region of interest. The segmentation of the current 2D image data is performed to identify the representation of the device, a second transformation is applied to the representation of the device, and the transformed representation of the device is combined with the generated projection image. method.

12. A computer program having an instruction to cause a computer to perform the method according to claim 11 when executed by a computer.

13. A computer-readable medium storing the computer program described in claim 12.

Images