Overlaying a deconstruction image with a current projection

Abstract

The present invention relates to a method for providing an overlay data set by - Providing a 3D dataset (11) depicting a vessel, - Unfolding at least a sub-area of ​​the 3D dataset (11) along a central line of the vessel into a two-dimensional unfolding image (13), - Capturing a first projection (12) of the vessel, - Register (10) the first projection (12) with the 3D data set (11) and - Providing the overlay data set comprising an overlay (14) of the two-dimensional unfolding image (13) with the first projection (12).

Description

[0001] The present invention relates to a method for providing a superimposed data set. Furthermore, the present invention relates to a navigation method, an imaging modality for vascular visualization, and a corresponding computer program.

[0002] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identities are included.

[0003] Modern imaging techniques, particularly X-ray-based imaging techniques such as fluoroscopy, are sometimes used to support interventions. In these techniques, a tool or other device inserted into the object being examined can be imaged and tracked within the object during the intervention, for example, within a vascular structure.

[0004] To enable the most accurate tracking of the device in relation to the vascular structure, and thus the most precise guidance of the device within the object, achieving the highest possible image quality is desirable. Particularly in the context of X-ray-based imaging techniques, it can sometimes be difficult to clearly identify the device and distinguish it from other components of the image, such as depictions of tissue or bone structures, or even the vascular structure itself. The same applies to the distinguishability of the vascular structure from other tissue or the like.

[0005] Navigation during procedures is often hampered by inadequate imaging, such as shortening, overlapping of different vessels, and / or suboptimal angles used to visualize vascular bends or curves. However, it is crucial to image the entire vessel to be treated in order to visualize its complete course and all its curvatures, thus enabling the selection of the appropriate instrument (material, properties, length) for the procedure. The choice of imaging settings during the procedure, however, often depends on the interventionalist's experience level. For some procedures, a preoperative 3D image dataset (e.g., CTA, MRI) is available, which can be helpful in planning the imaging angles during the procedure.Furthermore, during the procedure, usually only 2D information is available, which does not necessarily provide sufficient knowledge for precise navigation and knowledge of the total path length to the lesion to be treated, which can lead to the selection of the wrong device (stiffness, material, ...) or a device with the wrong length.

[0006] The problems encountered during surgical navigation are currently resolved through visual control and the interventionalist's experience. They often plan the imaging angles based on their own experience. Sometimes a preoperative 3D imaging dataset (e.g., CTA or MRI) is available and can be used to plan the various angles. However, a registered and superimposed 3D dataset onto the current 2D live images does not necessarily provide an overview of the vessels that is free of shortening and overlap. If, during navigation, it is determined that the device (e.g., catheter) is too short due to vessel shortening and undetected / unvisualized vessel curvature, it is usually removed and another device inserted into the vessels, incurring further costs.

[0007] From the article Rist, L., Taubmann, O., Ditt, H., Sühling, M., & Maier, A. (2023, October). Flexible Unfolding of Circular Structures for Rendering Textbook-Style Cerebrovascular Maps. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 737-746). Cham: Springer Nature Switzerland, the unfolding of vessels from a preoperative / diagnostic 3D dataset is known. Furthermore, the publication EP 3 828 836 B1 discloses a computer-implemented method for providing a two-dimensional unfolded image of at least one tubular structure. Furthermore, the publication DE 10 2004 011 154 B3 discloses a method for registering a sequence of 2D image data of a cavity organ with 3D image data of the cavity organ from 3D imaging modalities, wherein the 2D image data were acquired with an imaging endoluminal instrument at known relative displacement positions of the instrument in the cavity organ.

[0008] The object of the present invention is to improve the visualization of blood vessels, particularly for medical procedures. A corresponding method and imaging modality are proposed.

[0009] According to the invention, this problem is solved by a method for providing a superimposed data set, a method for navigation, and an imaging modality according to the independent claims. Furthermore, a corresponding computer program or computer-readable storage medium is provided.

[0010] Accordingly, a method for providing an overlay dataset is provided. The vessel in question could be, for example, a blood vessel or a duct for transporting secretions (e.g., bile ducts). It is often necessary to access these vessels using a catheter or similar instrument. Every vessel has its own predefined shape, which is why the vessel's surroundings are typically also depicted when it is visualized. Therefore, the vessel's visualization generally represents a specific target area within which the vessel is located.

[0011] First, a 3D dataset depicting a vessel is provided. This 3D dataset can be a 3D reconstruction, a 3D image, or a 3D representation, or even a 3D model of the target area containing the vessel. The 3D dataset spatially represents the target area and the vessel. This 3D dataset can be generated from preoperative data. This means that the 3D dataset will not be up-to-date during the operation and, in particular, will not depict any instruments inserted into the vessel. The 3D dataset can be provided using appropriate data storage media or via data networks.

[0012] In a further step, at least a portion of the 3D dataset is unfolded along a central line of the vessel to create a two-dimensional image. Every vessel typically has a central line, which represents a three-dimensional structure. This central line can be unfolded into a two-dimensional plane, advantageously preserving its length. If the areas surrounding the vessel's central line are also unfolded, the two-dimensional image is generated. Further information regarding this unfolding process can be found, for example, in the article by Rist et al. The two-dimensional image essentially provides a textbook-like overview map of the individual vessel's course. With its aid, interventionalists can easily orient themselves when guiding a medical instrument within the vessel.

[0013] In a further step, an initial projection of the vessel is acquired. This first projection depicts the vessel, for example, intraprocedurally, particularly while a medical device is positioned within it. The vessel is visualized, for instance, with the inserted catheter. Acquisition can involve receiving and / or recording the projection or cross-sectional image, for example, using an X-ray machine, an ultrasound device, or another imaging modality. In any case, this initial projection is independent of the dataset used for the 3D dataset. Therefore, the initial projection is more up-to-date than the 3D dataset.

[0014] The first projection is then registered with the 3D dataset. During this process, the first projection is oriented or positioned relative to the 3D dataset, or conversely, the 3D dataset can be oriented or positioned relative to the current projection image. Furthermore, the image scale is typically adjusted during registration. This registration is particularly important because the first projection and the 3D dataset are usually acquired using different modalities. Even if the first projection and the 3D dataset are acquired using X-ray technology, for example, different modalities are typically used for each image, as different acquisition sequences are required.

[0015] In a further step, the overlay dataset is provided, comprising a superimposition of the two-dimensional unfolding image with the first projection. This overlay can be achieved through at least partial superimposition. The overlay components can be at least partially or sectionally transparent. Thus, a two-dimensional projection is inserted into the two-dimensional unfolding image. The two-dimensional projection has the advantage of being more up-to-date, thereby updating the unfolding image, at least in certain areas. In particular, this allows, for example, an inserted catheter to be visualized, at least approximately, in the unfolding image during an operation. Advantageously, a two-dimensional unfolding image obtained based on preoperative data can be updated with intraoperative projection data.In particular, such an unfolded 2D view of 3D vessels can improve or facilitate endovascular (robot-assisted) navigation during minimally invasive procedures.

[0016] The main advantage is the improved overview of the current vessels and their properties, such as curvature, which are displayed without foreshortening or overlap with other vessels. With such a 2D overview, the actual length and curvature of a vessel can be visualized and used in a variety of ways for better and faster navigation. Furthermore, the combined visualization with the expanded view of a diagnostic dataset enables the comprehensive visualization of structures that might only be visible in one modality or another. This can result in, in particular, reduced cognitive load for users and easier navigation thanks to the dimensionality-reduced visualization.

[0017] According to one embodiment, the 3D dataset includes at least one additional vessel besides the main vessel, and the unfolding of the corresponding portion of the 3D dataset also occurs along a central line of this additional vessel. The unfolding thus takes place with respect to multiple vessels. This allows, for example, the two-dimensional representation of a larger, unfolded vascular tree or vascular system (e.g., the main cerebral arteries). This can significantly facilitate the surgeon's orientation.

[0018] In another embodiment, the two-dimensional unfolded image can be designed to show the vessel(s) without overlap or foreshortening. The unfolding of the 3D dataset is thus carried out in such a way that a specific vessel is displayed without overlap. This lack of overlap also improves orientation on or within the vessel.

[0019] According to another embodiment, the 3D datasets are based on a computed tomography (CT) or magnetic resonance imaging (MRI) dataset. In particular, the CT dataset can be a CT angiography dataset. Such datasets are generally well-suited for generating 3D vascular reconstructions to enable spatially accurate representation of vascular systems.

[0020] According to another embodiment, a second projection is captured and superimposed on the two-dimensional unfolded image. Thus, not only is the first projection captured, but also a second projection that differs from the first. The second projection can be captured with the same acquisition geometry as the first. In this case, the second projection would be a simple update of the first. This means that the projection is simply repeated, thereby updating it. The update can be performed repeatedly, continuously, and / or in real time. If the acquisition geometry remains the same for the different projections, the same registration can always be used. Otherwise, if the acquisition geometry changes, a new registration is generally required.Registration must always be carried out with the corresponding recording data.

[0021] According to another embodiment, a second projection is superimposed on the unfolded image in addition to the first. This means that more than one projection is visible on the unfolded image. If necessary, the interventionalist can thus superimpose several projections on the unfolded image to improve orientation. Projections may also be automatically superimposed on the unfolded image, for example, if the tip of a catheter is near a vascular bifurcation.

[0022] The first and second projections do not need to be filmed from the same projection angles. In fact, it can be helpful to control the recording angle individually for each projection.

[0023] In another embodiment, the second projection is registered separately from the 3D dataset for overlaying with the unfolded image. If a projection is simply updated, re-registration is not necessary. However, if the second projection is taken with a different acquisition geometry (different acquisition position and / or different acquisition angle), a new registration relative to the 3D dataset may be required to ensure a meaningful overlay.

[0024] According to a further embodiment, the projection, particularly the first and / or second projection, only overlays a portion of the unfolded image. This means that the projection, i.e., the projected image, is smaller than the entire unfolded image. This is particularly advantageous when significant portions of the unfolded image remain visible for general orientation, while only a smaller section of the unfolded image is overlaid by the respective projection to provide additional orientation aid in the area of ​​overlay. Thus, the projection does not overlay the entire unfolded image, but only a genuine part of it, so that its orientational function is not lost.

[0025] According to another embodiment, an artificial intelligence unit is used for unfolding, registering, and overlaying the unfolded image with the first or second projection. For example, the artificial intelligence unit can learn how a 3D dataset must be unfolded for specific interventions and / or when and where projections should be superimposed on the unfolded image. These overlay conditions can also be typical for an interventionalist, for example, and can be learned accordingly.

[0026] According to another embodiment, the first or second projection depicts a medical object. For example, the medical object could be a catheter, a wire, an implant (e.g., a stent), and the like. The first or second projection can also depict several nested medical objects, particularly medical instruments (e.g., wire inside a catheter). It is generally important to know the exact current position of these medical objects. Therefore, these medical objects, with their current position or orientation (generally: pose), can be superimposed onto the unfolded image.It may also be provided that the projections are only temporarily superimposed on the unfolding image and, for example, disappear or are deleted automatically after ten seconds, so that the original section of the unfolding image becomes visible again.

[0027] According to a further embodiment, the provision of the overlay data set includes displaying a graphical representation of the overlay data set.

[0028] Advantageously, the graphical representation of the overlay data set can be displayed using a display unit, for example a screen and / or monitor and / or smart glasses and / or a projector.

[0029] According to a further aspect of the present invention, a method for navigating a medical object (e.g., instrument, implant) is provided depending on the overlay data set supplied according to one of the methods described above. The overlay data set is translated into control information, and the object is controlled according to this information. Navigation is thus performed manually, semi-automatically, or fully automatically depending on the overlay data set or the vascular representation. For example, navigation can be performed solely based on the unfolded image, and the display of a projection can serve only for monitoring by, for example, a physician. Alternatively, information from one or more projections can also be used by image processing to support the navigation of the medical object.The superimposed unfolding image thus enables improved robot navigation or robot-assisted navigation. In particular, navigation can be supported more efficiently because the unfolding image reduces one spatial dimension.

[0030] In one embodiment of the navigation method, a maneuver for the medical object in the unfolded image is automatically determined from the overlay data set. For example, the overlay data set calculates that the medical object, particularly a catheter, must be moved a few millimeters forward and then in a specific direction. This maneuver can, for example, be automatically determined from the overlay data set.

[0031] In another embodiment of the navigation method, the maneuver can be translated into a three-dimensional movement based on a relationship between the 3D dataset and the two-dimensional unfolding image. This is achieved by translating the maneuver into three-dimensional control information, which is then used to realize the three-dimensional movement. Even if the vessel depicted in the unfolding image only moves in one plane, a movement perpendicular to this plane can be determined, since the relationship between the unfolding image and the 3D dataset is known. This allows even complex spatial maneuvers and corresponding control information to be generated or supported based on the two-dimensional unfolding image.

[0032] The above-mentioned problem is also solved according to the invention by an imaging modality for providing a superimposed data set, comprising - a storage device for providing a 3D dataset depicting a vessel, - a computing device for unfolding at least a sub-area of ​​the 3D dataset along a central line of the vessel into a two-dimensional unfolded image and - a detection device for capturing a first projection of the vessel, wherein - the computer system is trained for: ◯ Register the first projection with the 3D dataset and ◯ Providing the overlay dataset comprising a superimposition of the two-dimensional unfolding image with the first projection.

[0033] The storage device can, for example, comprise one or more memory modules and, optionally, its own processor. Furthermore, the storage device can be located locally within the housing of the imaging modality or externally, for example, in a data network.

[0034] The computing unit may itself contain one or more processors. Suitable image processing algorithms may also be assigned to the computing unit.

[0035] The scanning device can be based on different technologies to obtain a specific projection of the vessel. For example, the scanning device can be based on X-ray technology or ultrasound technology.

[0036] The advantages and variations described above in connection with the method according to the invention also apply analogously to the imaging modality according to the invention. The described method features can accordingly be interpreted as functional features of the imaging modality.

[0037] According to the present invention, a computer program or a computer-readable medium can also be provided which includes commands which, when executed by the imaging modality described above, cause it to execute the method also described above.

[0038] The present invention will now be explained in more detail with reference to the accompanying drawings, which show: Fig. 1 a schematic representation of an exemplary embodiment of an imaging device; and Fig. 2 a flowchart of an embodiment of a presentation method for a vessel according to the present invention.

[0039] The exemplary embodiments described in more detail below represent preferred embodiments of the present invention.

[0040] In Fig. Figure 1 schematically depicts an exemplary embodiment of an imaging device 1 (i.e., imaging modality), which is configured, for example, as an X-ray imaging device. In the example of the Fig. Figure 1 shows a construction of the X-ray imaging device based on the principle of a C-arm device with a rotatable and movable C-arm 6, which can be rotated and moved accordingly to image an object 4 from different directions, i.e., with different angles of view. However, an imaging device 1 according to the invention can also be constructed according to other designs. In particular, the concept according to the invention is not fundamentally limited to X-ray-based imaging methods.

[0041] Imaging device 1 of the Fig. The imaging device 1 includes, for example, an X-ray source 2, which is configured to generate X-rays and emit them towards the object 4. On the opposite side of the object 4 from the X-ray source 2, a sensor 3 of the imaging device 1 is arranged, which, for example, contains a detector array of photodiodes to detect X-ray quanta penetrating the object 4. The sensor 3 can then transmit the corresponding detector signals, for example, to a processing unit 5 of the imaging device 1 for further processing.

[0042] The imaging device 1 can be configured, in particular, to perform a rotational angiography procedure, for example, based on the principle of subtraction angiography. In this case, the processing unit 5 can, for example, generate a large number of two-dimensional projections taken from different angles, and the processing unit 5 can calculate a three-dimensional reconstruction from these and make it available to a display unit 9 (e.g., a screen).

[0043] The following section explains in more detail the functioning of the imaging device 1 with reference to various embodiments of a method for providing a superimposed data set and a method for navigating a medical object according to the concept of the invention, in particular with reference to Fig. 2.

[0044] For interventional navigation, a preoperative 3D dataset can be combined with an unfolding technique (described in Rist et al.) according to one embodiment, to enable better and more precise navigation while simultaneously displaying vessels free of overlap and shortening. The proposed workflow includes, for example, the following steps (see Fig. 2) which are described in detail below: A) Registration 10 of preoperative 3D data 11 (e.g. CTA or MRI) with current 2D intervention images 12; B) Development of 13 of the registered data for the “textbook” visualization of the vessels; C) Visualization 14 of overlap- and shortening-free vessels and their use for precise navigation.

[0045] A prerequisite for an exemplary workflow, as described in step A), is robust (multimodal) registration 10 between preoperative 3D data 11 and the interventional 2D images 12, which, for example, show an instrument in its current position within a vessel. Various approaches to solving this problem are described in the literature, e.g., the articles by Park et al., Gouveia et al., and Zhu et al. mentioned at the beginning.

[0046] During the unfolding process according to step B), the 3D dataset 11 is first unfolded, for example, using the method described in Rist et al. Registration 10 establishes the relationship between the diagnostic 3D data 11 and the currently acquired interventional image 12. An AI can be trained to use all this information (unfolding parameters of the diagnostic data, registration transformation, interventional data) to output an unfolded view of the currently visualized vessels, which can be (partially) overlaid on the current unfolded view of the diagnostic data. As new images 15 are acquired, the unfolded view 13 can be gradually completed (see update 16 in [reference]). Fig. 2) are displayed, with more and more vessels from the intervention data 12, 15 and / or updated vessel segments being shown.

[0047] The result of step C) is a superimposed version of the unfolded vessels, providing a textbook-like 2D overview of the vessels, derived from both a 3D dataset (especially diagnostic and / or preoperative data)11 and current interventional data12, 15, even using currently deployed devices. The vessels in this overview are shown without foreshortening or overlap. This overview can be used in various ways to support and enable more precise navigation, including: a. Calculation of path lengths for navigation, which can be used directly for predicting device lengths and sizes (diameters). b. Calculation of “simple” path maneuvers of the device, e.g., 3 mm forward, then turn 30° to the left. These can be used for better visualization to guide the interventionalist (e.g., a color map indicating how much further the device needs to be moved forward), or can be directly translated into precise 3D maneuvers in a robot-assisted procedure for automatic navigation. c. Providing a comprehensive 2D overview 13 together with diagnostic and interventional image data 12, 15 can precisely identify stent landing zones, e.g., making calcifications on the diagnostic image more visible, and the overlay of the interventional image 12, 15 simultaneously shows, for example, the specific stent position. If the stent has already been deployed, it can be seen whether it has advanced sufficiently or not. d. By updating the registration 10 and unfolding 13 with current interventional images 12, 15, the device movements can also be displayed live on the comprehensive 2D overview 14, 16, leading to a better understanding of the device behavior and a faster reaction in difficult navigation situations.

[0048] According to an advantageous embodiment, the control of an endovascular robot can be simplified. By displaying the interventional data 12 (2D) and preoperative data 11 (3D) in an unfolded or unfolded, i.e., 2D, view, the dimensions of the movement space are implicitly reduced. In particular, there are only two dimensions. Therefore, the robot can be controlled solely by considering these two dimensions (up / down, left / right). Since the relationship between the unfolded 2D view 13 and the actual 3D view 11 is known, movements in the unfolded 2D space can be automatically converted into actual 3D movements by the robot / software.

[0049] Thus, (robot-assisted) navigation can be advantageously improved using the novel method for unfolding vessels from a 3D dataset, particularly preoperative and / or diagnostic dataset (see Rist et al.). This method is capable of visualizing the vessels of interest, along with the surrounding parenchyma, in a comprehensive, coherent 2D overview and displaying vessel length and curvature for a rapid assessment of vascular topology. Combining this approach with live interventional images during the treatment of, for example, stroke patients, can provide a better overview of the current vessels and their properties, such as curvature, leading to improved overall visibility, more precise navigation, and better selection of treatment devices.

[0050] The expanded display of the vessels thus provides a better overview of the current vessels and their properties, such as curvature, which are shown without foreshortening or overlapping with other vessels. Furthermore, as mentioned above, this 2D overview allows the actual length and curvature of a vessel to be visualized and used in various ways for better and faster navigation.

[0051] Furthermore, the combined visualization with the unfolded view of a diagnostic dataset enables the comprehensive visualization of structures that may only be visible on one modality or the other.

[0052] Furthermore, a lower cognitive load for users and easier navigation can be expected due to the reduced-dimensionality visualization.

Claims

[1] Method for providing an overlay data set by - Providing a 3D dataset (11) depicting a vessel, - Unfolding at least a sub-area of ​​the 3D dataset (11) along a central line of the vessel into a two-dimensional unfolding image (13), - Capturing a first projection (12) of the vessel, - Register (10) the first projection (12) with the 3D data set (11) and - Providing the overlay data set comprising an overlay (14) of the two-dimensional unfolding image (13) with the first projection (12). [2] Method according to claim 1, wherein the 3D data set (11) depicts at least one further vessel in addition to the vessel, and the unfolding of the partial area of ​​the 3D data set (11) also takes place along a further central line of the further vessel. [3] Method according to claim 1 or 2, wherein the two-dimensional unfolding image (13) shows the vessel or vessels without overlap and without shortening. [4] Method according to any of the preceding claims, wherein the 3D data set (11) is based on a computed tomography data set or a magnetic resonance tomography data set. [5] Method according to one of the preceding claims, wherein a second projection (15) is captured and superimposed on the two-dimensional unfolding image (13). [6] Method according to claim 5, wherein in addition to the first projection (12) the second projection (15) is superimposed on the unfolding image. [7] Method according to claim 5 or 6, wherein for superimposing with the unfolding image (13) the second projection (15) is registered separately with the 3D data set (11). [8] Method according to one of the preceding claims, wherein the projection only superimposes a part of the unfolding image (13) during superimposition. [9] Method according to any of the preceding claims, wherein an artificial intelligence unit is used for unfolding, registering and superimposing the unfolded image (13) with the first projection (12). [10] Method according to any of the preceding claims, wherein the first projection (12) represents a medical object. [11] Method according to any of the preceding claims, wherein providing the overlay data set includes displaying a graphical representation of the overlay data set. [12] Imaging modality for providing an overlay dataset, comprising - a storage device for providing a 3D data set (11) depicting a vessel, - a computing device for unfolding at least a sub-area of ​​the 3D data set (11) along a central line of the vessel into a two-dimensional unfolded image (13) and - a detection device for capturing a first projection (12) of the vessel, wherein - the computer system is trained for: • Register the first projection (12) with the 3D data set (11) and • Providing the overlay data set comprising an overlay of the two-dimensional unfolding image (13) with the first projection (12). [13] Imaging modality according to claim 12, which is configured to navigate a medical object depending on the provided overlay data set, wherein the imaging modality is designed to translate the overlay data set into control information and to control the medical object according to the control information. [14] Imaging modality according to claim 13, which is configured to automatically determine a maneuver for the medical object in the unfolding image (13) from the overlay data set. [15] Imaging modality according to claim 14, which is configured to translate the maneuver into a three-dimensional movement based on a relationship between the 3D data set (11) and the two-dimensional unfolding image (13) by translating the maneuver into three-dimensional control information according to the relationship, with which the three-dimensional movement is realized. [16] Computer program comprising instructions which, when the program is executed by an imaging modality according to any one of claims 12 to 15, cause the imaging modality to execute the method according to any one of claims 1 to 11.

Images