Target visualization

Abstract

The invention relates to object visualization. To provide a convenient visualization of the internal structure of an object, an object visualization apparatus (10) is provided, which comprises an image supply (12), a data processor (14) and an output (16). The image supply provides a 2D X-ray image of the object and 3D image data of the object. The data processor registers the 3D image data to the 2D X-ray image and detects a viewing direction used for generating the 2D X-ray image. The data processor further determines a viewing direction used for projecting the 3D image data to generate a matching view of the 3D image data based on the viewing direction used for generating the 2D X-ray image. The data processor generates a stereoscopic image pair (24) of the 3D image data for the determined viewing direction and combines the 2D X-ray image with one of the images of the stereoscopic image pair to result in an enhanced first stereoscopic image (26), while the other image of the stereoscopic image pair remains unmodified as a generated second stereoscopic image (28). The output provides the enhanced first stereoscopic image and the generated second stereoscopic image as an enhanced stereoscopic image pair (30) to provide an enhanced 3D view of the object. Although only one stereoscopic image is provided with additional X-ray image content information, the X-ray information is also transferred to the other unmodified image in the binocular combined view. Thus, the X-ray information is spread into the spatial content. One advantage is that only one X-ray image is needed, which means that the object is exposed to less X-rays.

Description

Technical Field

[0001] This invention relates to a target visualization device, a system for target visualization, and a method for visualizing a target of interest, as well as computer programs and computer-readable media. Background Technology

[0002] For medical procedures such as interventional procedures, in addition to user observation, various imaging techniques are used to provide users with information related to the target. For example, X-ray imaging is used to visualize regions of interest inside an object, such as vascular structures. It is also possible to use image data acquired based on X-rays to generate 3D models for further visualization (e.g., for spatial representation). For example, US 2018 / 0310907 A1 relates to providing visualization of a target. According to its abstract, a computer system for providing simulated 2D fluorescence fluoroscopy images is described. During operation, the computer system can generate simulated 2D fluorescence fluoroscopy images based on data in a predetermined 3D image associated with an individual's body. For example, generating simulated 2D fluorescence fluoroscopy images can involve orthographic projection. Furthermore, orthographic projection can involve calculating the cumulative absorption corresponding to the density of X-rays passing through pixels in the predetermined 3D image. The computer system can then provide the simulated 2D fluorescence fluoroscopy images with a 3D background associated with predefined sections in the individual's body. However, it has been shown that the setup is quite complex. Summary of the Invention

[0003] Therefore, it may be necessary to make the internal structure of the object easy to visualize.

[0004] The object of the invention is achieved through the subject matter of the independent claims; further embodiments are included in the dependent claims. It should be noted that the aspects described below also apply to target visualization devices, systems for target visualization, and methods for visualizing targets of interest.

[0005] According to the present invention, a target visualization device is provided, comprising an image supply unit, a data processor, and an output unit. The image supply unit is configured to provide a 2D X-ray image of the target. The image supply unit is also configured to provide 3D image data of the target. The data processor is configured to register the 3D image data to the 2D X-ray image. The data processor is further configured to detect an observation direction for generating the 2D X-ray image. The data processor is also configured to determine an observation direction for projecting the 3D image data to generate a matching view of the 3D image data based on the observation direction for generating the 2D X-ray image. The data processor is further configured to generate a stereo image pair of the 3D image data for the determined observation direction and combine the 2D X-ray image with one image from the stereo image pair to obtain an enhanced first stereo image, while the other image in the stereo image pair remains unchanged as a generated second stereo image. The output unit is configured to provide the enhanced first stereo image and the generated second stereo image as an enhanced stereo image pair, thereby providing an enhanced 3D view of the target.

[0006] One advantage is that it provides users with spatial (i.e., three-dimensional) information as well as the content of the X-ray image, while only modifying one image in the stereoscopic image.

[0007] The term "registration" refers to linking 3D image data of a target to a 2D X-ray image, aligning the 2D and 3D image data of the target with each other in terms of viewing angle. For example, the 2D X-ray image is taken with a specific imaging angle (i.e., a specific image acquisition geometry, resulting in a specific 2D X-ray image with a certain image plane (i.e., the plane of the detector that collects the projected and thus attenuated X-ray radiation beam)) and the X-ray radiation direction of the generated X-ray beam. As an example, the 3D image data is arranged relative to the 2D X-ray image such that image generation operations using the 3D image data when acquiring X-ray images with the same geometry will produce similar X-ray images.

[0008] For example, 3D data of a target refers to 3D image data of the target. The term "registration" can therefore refer to registering the 3D image data of the target to a 2D X-ray image, such that the 2D X-ray image and 3D image data of the target are aligned within the same spatial reference, that is, they are registered to the same coordinate system.

[0009] The term "observation direction for detecting 2D X-ray images" refers to the imaging direction of an X-ray imaging system that has an X-ray source that generates X-ray radiation and an X-ray detector that detects the generated X-ray radiation, which is attenuated by the target as it passes through it.

[0010] For example, detecting the observation direction of an X-ray imaging system used to generate 2D X-ray images.

[0011] The term “determining the viewing direction for a matching view of 3D data based on the viewing direction of a 2D X-ray image” refers to determining the projection direction to be applied to 3D data (e.g., 3D image data of a target) to generate a projection in the image plane that will show an image result similar to the 2D X-ray image.

[0012] For example, the following observation direction is determined for a simulated projection onto an X-ray imaging system, which will yield an image that matches a 2D X-ray image. For example, a "matching view" refers to a "matching image" or an equivalent image result.

[0013] As an example, a target visualization device is provided, comprising an image supply unit, a data processor, and an output unit. The image supply unit is configured to provide a 2D X-ray image of the target and to provide 3D image data of the target. The data processor is configured to: register the 3D data to the 2D X-ray image; detect the viewing direction of the 2D X-ray image; determine the viewing direction of a matching view for the 3D data based on the viewing direction of the 2D X-ray image; generate a stereo image pair of the 3D data for the determined viewing direction; and combine the 2D X-ray image with one image from the stereo image pair to obtain an enhanced first stereo image, while the other image in the stereo image pair remains unchanged as a generated second stereo image. The output unit is configured to provide the enhanced first stereo image and the generated second stereo image as an enhanced stereo image pair, thereby providing an enhanced 3D view of the target.

[0014] In the example, augmented reality (AR) or mixed reality (MR) technologies are provided to display the combined image set.

[0015] This provides the effect that information is presented to the user's field of view from any working position. By providing a stereoscopic 3D dataset, stereoscopic information can be added to the user's field of view from any working position. Head-mounted displays (HMDs) such as Google Glass or Microsoft HoloLens are provided as examples.

[0016] For interventional applications, this technology enables applications to map 3D anatomical models and (live) X-ray streams in space. Such capabilities facilitate a better understanding of clinical anatomy structures navigating, for example, in vascular cases. This supports situations requiring surgeons to visualize or track devices advancing through blood vessels.

[0017] In addition to generating 3D image pairs, providing regular X-ray images also enables visualization of the device in two dimensions, and also gives a sense of how the device, vascular system, organs and other tissues are related to each other in 3D.

[0018] One advantage is that even if the X-rays displayed as superimposed on a stereoscopic image only provide us with a 2D image from a single viewpoint, it is possible to provide stereoscopic rendering of scenes that require two viewpoints (one for each of the observer's eyes).

[0019] The example provides a visualization engine for 3D rendering of the scene.

[0020] In the example, two offset views of the 3D volume are created based on the line of sight from each corresponding eye to the volume, which will be displayed to the observer's left and right eyes, respectively. The X-ray image is overlaid on only one of these views.

[0021] As a consequence, because it is a single 2D X-ray image, depth cannot be implied through binocular parallax. As another consequence, because it is blended with a 3D model (which is rendered for two perspective views and has correct binocular parallax), the user combines the two offset views (i.e., one with a 2D X-ray overlay and the other without) without visual incongruity. In the example, a small single-view target is provided for easier user perception.

[0022] Note that the X-ray image is used to determine the viewing direction for the projection of the stereoscopic image. The X-ray image is the starting point for the computational space setup and therefore determines the subsequent image generation steps.

[0023] According to the example, the enhanced first stereo image is a dual-source image, and the generated second stereo image is a single-source image.

[0024] The term "dual image source" refers to two distinct sources providing two different types of images, which are then combined to form an enhanced first stereoscopic image. The term "single image source" refers to a single source providing a single type of image, which is then used to create a second stereoscopic image.

[0025] In the example, the enhanced first stereo image is an image that includes image data from two different image sources, and the generated second stereo image includes image data from a single image source.

[0026] In the example, the enhanced first stereo image is an image that includes the following: an image from a first image source and an image from a second image source different from the first image source. The generated second stereo image includes only the image from the first image source.

[0027] In the example, the enhanced first stereoscopic image is constructed by combining an image from a first image source and an image from a second image source different from the first image source. This combination is provided as a superposition of two different images. The resulting second stereoscopic image includes only the image from the first image source.

[0028] According to an example, the data processor is configured to combine the 2D X-ray image with one of the images in the stereo image pair, including superimposing the 2D X-ray image onto the one of the images in the stereo image pair.

[0029] In another example, the data processor is configured such that combining a 2D X-ray image with one image from a stereo image pair includes overlaying one image from the stereo image pair onto the 2D X-ray image.

[0030] According to the example, the data processor is configured to provide the combination such that in the enhanced first stereoscopic image, i) the image portion covered by the 2D X-ray image is smaller than the portion covered by the generated stereoscopic image. As an additional or alternative option, the data processor is configured to provide the combination such that in the enhanced first stereoscopic image, ii) the 2D X-ray image is provided with greater transparency than the generated stereoscopic image.

[0031] According to the present invention, a system for target visualization is also provided. The system includes a first image display, a second image display, and a target visualization device according to any one of the preceding claims. One of the first image display and the second image display is configured to present the enhanced first stereoscopic image to one of the user's two eyes. The other of the first image display and the second image display is configured to present the generated second stereoscopic image to the other of the user's two eyes.

[0032] In the example, the system includes a first image display, a second image display, and a target visualization device according to any one of the preceding claims. One of the first and second image displays is configured to present an enhanced first stereoscopic image to a user. The other of the first and second image displays is configured to present a generated second stereoscopic image to the user.

[0033] According to the example, a display device is provided, the display device including a first image display and a second image display. Alternatively, the display device is a head-mounted display providing presentation for each eye of the user; and providing the enhanced first stereoscopic image to one eye of the user, and providing the generated second stereoscopic image to the other eye of the user.

[0034] According to the example, the display device is an augmented reality display that provides the enhanced stereoscopic image pair as an enhanced 3D view of the target superimposed on a view of the current live scene.

[0035] According to the present invention, a method for visualizing a target of interest is also provided. The method includes the following steps:

[0036] Provide a 2D X-ray image of the target;

[0037] Provide 3D data of the target, such as 3D image data;

[0038] Register the 3D image data to the 2D X-ray image;

[0039] Detect the viewing direction used to generate the 2D X-ray image;

[0040] The observation direction for projecting the 3D image data to generate a matching view of the 3D image data is determined based on the observation direction used to generate the 2D X-ray image;

[0041] Generate stereo image pairs of the 3D image data for the determined viewing direction;

[0042] The 2D X-ray image is combined with one image from the stereo image pair to obtain an enhanced first stereo image, while the other image from the stereo image pair remains unchanged as the generated second stereo image; and

[0043] The enhanced first stereoscopic image and the generated second stereoscopic image are provided as an enhanced 3D view of the target.

[0044] According to the example, the registration of the 3D image data to the 2D X-ray image takes into account the reverse perspective of the 2D X-ray image, which includes a focal point and a converging beam with respect to the observation direction of the target and the X-ray radiation.

[0045] The term "reverse X-ray imaging" refers to the fact that when a 2D X-ray image is generated from an X-ray beam, the 2D X-ray image is based on a fan-shaped or cone-shaped beam that has a focal point at its origin and diverges towards the detector. Because the radiation direction is from the source to the detector (i.e., from the focal point to the wider beam), the "observation" direction of such X-rays is also divergent. From a visual perspective, this provides the opposite situation in a certain way: the observation direction is towards the focal point, that is, it converges in the observation direction.

[0046] In the example, the registration of 3D data to a 2D X-ray image takes into account the inverse perspective of the 2D X-ray image. According to one aspect, a single 2D X-ray image is superimposed on only one offset view in an offset view used to create a stereoscopic view of the 3D volume. The offset view provides visual depth information (also known as spatial information) of the target, and the single 2D X-ray provides additional information about the target's region of interest. By providing this 2D X-ray image against the background of a stereoscopic view within the stereoscopic view, the information from the 2D X-ray image is transferred from the two-dimensional world of the X-ray image to the three-dimensional world of the stereoscopic view.

[0047] According to one aspect, a first display image (for one eye) is generated that blends a 3D model with a 2D (live) X-ray image. Additionally, a second display image (for the other eye) is generated that only shows the 3D model but is presented from a different perspective. Furthermore, the orientation of the other eye is matched.

[0048] According to one aspect, a target visualization device is provided, comprising an image supply unit, a data processor, and an output unit. The image supply unit provides a 2D X-ray image of the target and also provides 3D data of the target. The data processor registers the 3D data to the 2D X-ray image and detects the viewing direction of the 2D X-ray image. The data processor also determines the viewing direction of a matching view for the 3D data based on the viewing direction of the 2D X-ray image. The data processor generates a stereo image pair of the 3D data for the determined viewing direction and combines the 2D X-ray image with one image from the stereo image pair to obtain an enhanced first stereo image, while the other image in the stereo image pair remains unchanged as a generated second stereo image. The output unit provides the enhanced first stereo image and the generated second stereo image as an enhanced stereo image pair, thereby providing an enhanced 3D view of the target. Although only one stereo image is provided with additional X-ray image content information, in the binocular combined view, X-ray information is also transferred to the other unmodified image. Therefore, X-ray information is extended into the spatial content. One advantage is that only one X-ray image is needed, which means that the object is exposed to less X-rays.

[0049] These and other aspects of the invention will become apparent and elucidated with reference to the embodiments described below. Attached Figure Description

[0050] Exemplary embodiments of the present invention will now be described with reference to the following figures:

[0051] Figure 1 An example of a target visualization device is shown schematically.

[0052] Figure 2 An example of a system for target visualization is illustrated schematically.

[0053] Figure 3 An example of a stereoscopic image pair presented to the user is shown.

[0054] Figure 4 An example of a system for target visualization is shown in an interventional room environment equipped with an X-ray imaging system.

[0055] Figure 5 An example of a head-mounted device is shown for displaying an enhanced 3D view of an enhanced first stereoscopic image and a generated second stereoscopic image as a target.

[0056] Figure 6 The basic steps of an example method for visualizing a target of interest are shown. Detailed Implementation

[0057] Certain embodiments will now be described in more detail with reference to the accompanying drawings. In the following description, similar reference numerals are used for similar elements even in different drawings. What is defined in the description (e.g., detailed constructions and elements) is provided to aid in a comprehensive understanding of the exemplary embodiments. Furthermore, well-known functions or structures are not described in detail, as this would obscure the embodiments with unnecessary detail. Additionally, expressions such as “at least one of” modify the entire list of elements when preceding a list of elements, rather than modifying individual elements in that list.

[0058] Figure 1 An example of a target visualization device 10 is shown schematically. The target visualization device 10 includes an image supply unit 12, a data processor 14, and an output unit 16.

[0059] The image supply unit 12, data processor 14, and output unit 16 can be provided in an integrated manner within a common structure such as a housing, as indicated by frame 18. Alternatively, the image supply unit 12, data processor 14, and output unit 16 can also be provided separately with data connections to each other.

[0060] Image supply unit 12 is configured to provide a 2D X-ray image of the target, indicated by a first dashed line 20. Image supply unit 12 is also configured to provide 3D data of the target, indicated by a second dashed line 22. In this example, the 3D data of the target is provided as 3D image data of the target.

[0061] Data processor 14 is configured to register 3D data (e.g., 3D image data) to a 2D X-ray image. Data processor 14 is also configured to: detect an observation direction for generating the 2D X-ray image, and determine an observation direction for projecting the 3D image data to generate a matching view of the 3D image data based on the observation direction for generating the 2D X-ray image. Data processor 14 is further configured to: generate a stereo image pair 24 of the 3D image data for the determined observation direction, and combine the 2D X-ray image with one image from the stereo image pair to obtain an enhanced first stereo image 26, while the other image from the stereo image pair remains unchanged as the generated second stereo image 28.

[0062] The output unit 16 is configured to provide an enhanced first stereoscopic image 26 and a generated second stereoscopic image 28 as an enhanced stereoscopic image pair 30, thereby providing an enhanced 3D view of the target.

[0063] The term "target visualization" refers to providing a presentation (or image) of a target. Visualization of the target is achieved through this presentation, which is provided to the user in a visual manner. An enhanced 3D view of the target can also be referred to as target visualization. The image supply unit 12 can also be referred to as an image data supply unit, an image input unit, an input unit, or an input unit.

[0064] In this example, the image supply unit 12 can be connected to an imaging source arrangement (such as an X-ray imaging device that provides a 2D X-ray image of a target). In this example, the image supply unit can be connected to a data storage device that has already stored a 2D X-ray image of the target.

[0065] In this example, the image supply unit can connect to an imaging source arrangement (such as a CT X-ray imaging device that provides 3D data of the target). In this example, the image supply unit can also connect to a data storage device that already stores the 3D data of the target.

[0066] The data processor 14 can also be referred to as a data processing arrangement, processor unit, or processor. In this example, the data processor may be connected to an image supply unit and an output unit. The data processor 14 is provided as a visualization engine that renders two views of a 3D model from two stereoscopic views, based on a perspective that matches the orientation of the acquired X-rays.

[0067] Output unit 16 can also be referred to as an output unit. In one example, output unit 16 can be connected to a display arrangement. In another example, output unit 16 can be connected to a display arrangement. For example, output unit 16 is configured to provide an enhanced stereoscopic image pair to a clinician, thereby providing an enhanced 3D view of the target.

[0068] The data processor 14 is configured to combine the 2D X-ray image with only one image from the stereo image pair.

[0069] The term "stereo image pair" refers to two separate images depicting the same object: a left-eye view and a right-eye view. Although presented as images on a 2D surface, the difference between the resulting views due to parallax provides the user with depth information, i.e., spatial information. An enhanced stereo image pair comprises a first image of a first generated view with 3D data and a second image of a second generated view with 3D data.

[0070] In the example, the data processor 14 is configured to: acquire the generated stereo image pair, combine one image from the image pair with a 2DX-ray image, and maintain the other image from the image pair, wherein the image pair is thus provided to include an X-ray-enhanced generated image data and a purely generated stereo image.

[0071] In the example, the data processor 14 is configured to: acquire a pair of generated stereoscopic images depicting the same target, including a left-eye view and a right-eye view, and combine one of the left-eye view and the right-eye view with a 2D X-ray image, while maintaining the other of the left-eye view and the right-eye view, thus providing a pair of eye views that includes an X-ray information-enhanced eye view and a purely 3D data-generated eye view.

[0072] By providing spatial information in the form of stereoscopic views of image pairs and by providing additional X-ray information in only one of the images, enhanced stereoscopic image pairs provide a partially enhanced 3D view of the target.

[0073] In the example, the data processor 14 is configured to: generate a stereo image pair of a first stereo image and a second stereo image with 3D data for a determined viewing direction; combine a 2D X-ray image with one of the first stereo image and the second stereo image to obtain an enhanced first stereo image; and maintain the other of the first stereo image and the second stereo image as the generated second stereo image.

[0074] In the example, the data processor 14 is configured to generate a stereo image pair consisting of a first stereo image and a second stereo image with 3D data for a determined viewing direction, and to combine a 2D X-ray image with one of the first and second stereo images. The resulting stereo image pair includes a first image and a second image, wherein the first image and the second image include different types of image content. One of the two images includes additional X-ray image content. For example, one of the two images includes additional X-ray fluorescence fluoroscopic image content combined with the generated 3D data view. Both images show the same options, only with a slight shift in the observation point to provide a stereo view for the user. Therefore, the second item of the first and second stereo images is provided as a purely generated second stereo image.

[0075] In the example, the second item in the first stereoscopic image and the second stereoscopic image is provided as an uncombined second image, while the first item in the first stereoscopic image and the second stereoscopic image is provided as a combined first image (also referred to as a combined first image). The uncombined second image can also be referred to as a single-source image, while the combined first image can also be referred to as a dual-source image or a multi-source image.

[0076] In the example, the second item in the first stereo image and the second stereo image is provided as the second image without superposition, while the first item in the first stereo image and the second stereo image is provided as the superimposed first image (also referred to as the superimposed first image).

[0077] In the example, the first item of the first stereoscopic image and the second stereoscopic image is provided as a dual-source image, wherein the dual data sources include data from a 2D X-ray image of the target and 3D data of the target. The second item of the first stereoscopic image and the second stereoscopic image is provided as a single-source image, wherein the single data source is the 3D data of the target.

[0078] Note that the terms "first" and "second" are provided to distinguish two images in a corresponding pair of images provided for a stereoscopic view. These terms do not imply any chronological order or other priority.

[0079] It should also be noted that the terms "right" and "left" are provided to define two images for the user's two shifted eyes, thus providing the basis for stereoscopic (i.e., spatial observation). These terms do not provide for or limit the user's actual left or right view. "First and second" or "one and the other" can also be used instead of "left and right".

[0080] Optionally, 3D data can be provided as preoperatively generated image data for the target procedure. Alternatively, 2D X-ray images can be provided as intraoperative images, current images, or live images, such as images taken during examinations, interventions, or other medical procedures.

[0081] For example, 3D data is based on CT X-ray images. Alternatively, or in addition to X-ray imaging, other imaging sources (e.g., ultrasound imaging or MRI) can be used to generate 3D data.

[0082] In the example not shown in further detail, the enhanced first stereo image is a dual-source image, while the generated second stereo image is a single-source image.

[0083] In an example not shown in further detail, the data processor 14 is configured such that combining a 2D X-ray image with one of the images in a stereo image pair includes overlaying the 2D X-ray image onto one of the images in the stereo image pair.

[0084] In an example not shown in further detail, the data processor 14 is configured to provide the combination such that in the enhanced first stereoscopic image, the portion of the image covered by the 2D X-ray image is smaller than the portion covered by the generated stereoscopic image.

[0085] Additionally or alternatively, the 2D X-ray image is provided with greater transparency than the generated stereoscopic image.

[0086] The X-ray image content is presented in a reduced manner (e.g., by smaller portions or increased transparency) so that the stereoscopic effect is largely unaffected by the combination of X-ray information with only one view in the view.

[0087] In an example not shown in further detail, the data processor 14 is configured to adjust the portion of the generated stereoscopic image covered by the 2D X-ray image to a maximum of 75%.

[0088] Therefore, the portion of the generated stereoscopic image covered by the 2D X-ray image is at most 75% of the surface of the first stereoscopic image actually provided to the user. At least 25% of the surface of the first stereoscopic image actually provided to the user remains uncovered by the 2D X-ray image.

[0089] Therefore, the enhanced first stereoscopic image includes an image region or image surface that shows at least a portion of the first stereoscopic image that is not superimposed with the X-ray image, but which is arguably unaffected by the combination of the two images. Thus, the enhanced first stereoscopic image includes: a main portion having a superposition of 2D X-ray images on the first stereoscopic image, and a secondary portion of the first stereoscopic image that is purely a superposition of 2D X-ray images without any superposition of 2D X-ray images.

[0090] In the example, the portion of the image covered by the 2D X-ray image is at least half the portion covered by the generated stereoscopic image.

[0091] In the example, the data processor is configured to adjust the transparency of the 2D X-ray image to be greater than the transparency of the generated stereo image (e.g., at least 25% greater) when the 2D X-ray image is projected onto one of the images in a stereo image pair.

[0092] In an alternative example, the data processor 14 is therefore configured to adjust the transparency of the generated stereo image to be greater than that of the 2D X-ray image (e.g., at least 25% or one-third greater) when one of the images in the stereo image pair is projected onto the 2D X-ray image.

[0093] In examples not shown in further detail, registration involves the spatial registration of a 2D X-ray image with respect to 3D image data. The term "spatial" registration refers to linking a 2D X-ray image to a reference frame or spatial coordinate system of the 3D image data.

[0094] Figure 2An example of a system 50 for target visualization is schematically illustrated. System 50 includes a first image display 52, a second image display 54, and an example of a target visualization device 10 from the foregoing examples. One of the first image display 52 and the second image display 54 is configured to present an enhanced first stereoscopic image to one of the user's two eyes. The other of the first image display 52 and the second image display 54 is configured to present a generated second stereoscopic image to the other of the user's two eyes.

[0095] exist Figure 2 In the example indicated as an option, a display device 56 (indicated by a frame) is provided, which includes a first image display 52 and a second image display 54.

[0096] In one option, display device 56 is a head-mounted display that provides presentation for each of the user's eyes (see below). An enhanced first stereoscopic image is provided to one of the user's eyes, and a generated second stereoscopic image is provided to the other of the user's eyes.

[0097] Figure 3 An example of a pair of stereoscopic images presented to the user is shown. Figure 3 The diagram shows a generated image pair 80, which includes, for example, a first image 82 or a portion of a first image and a second image 84 or a portion of a second image (i.e., an offset view for creating a stereoscopic view of a 3D volume). The first image 82 and the second image 84 each show a projection of a 3D model of the generated vascular system structure. This projection has slightly different viewing orientations to achieve a stereoscopic effect for the user. A single 2D X-ray image 86 is superimposed on only one of the offset views. The 2D X-ray image 86 shows the vascular structures (e.g., visible due to angiography). The first image 82 can be provided as a left-eye image, while the second image 84 can be provided as a right-eye image.

[0098] Figure 4An example of a system 50 for target visualization in an interventional room 100 environment is shown. The X-ray imaging system 102 includes a C-arm 104 with an X-ray source 106 and an X-ray detector 108. A target of interest (e.g., object 110) is arranged on an object support 112. A display and illumination arrangement 114 is also indicated. A console 116 is shown in the foreground, comprising a target visualization device 10 and forming the system 50 for target visualization together with a display device 56 having a first image display 52 and a second image display 54. The console can also be configured to control the operation of various instruments in the interventional room 100. For example, the X-ray imaging system 102 provides the console 116 with a current (i.e., live) image, as indicated by a first data connection line 118. A second data connection line 120 indicates the data supply from the visualization device 10 to the display device 56. For example, the display device 56 is provided as shown... Figure 5 The head-mounted device shown.

[0099] Figure 5 An example of a display device 56 as a head-mounted display 58 is schematically shown, which provides a presentation for each of the user's eyes. The head-mounted display 58 is configured to display an enhanced first stereoscopic image 26 and a generated second stereoscopic image 28 as an enhanced 3D view of the target. The enhanced first stereoscopic image 26 is provided to one of the user's eyes, and the generated second stereoscopic image 28 is provided to the other of the user's eyes. Such virtual images presented to the user are indicated by a projection 59, which provides spatial information to the user in the form of a pair of two stereoscopic images (one for each eye). The projection 59 also provides the user with additional information in the form of a superimposed X-ray image visible to the user's right or left eye.

[0100] For example, in addition to the object 62 on the object support, a user 60, such as a surgeon, is shown. Other staff members are also indicated. A monitor 64 is shown in the background, and a C-arm X-ray imaging system 66 is indicated in the right-hand portion of the illustration.

[0101] For example, a head-mounted display includes two separate displays.

[0102] Examples of head-mounted displays include Microsoft's HoloLens or Magic Leap glasses.

[0103] Alternatively, the display device includes a single monitor, but projects completely different images to each of the user's two eyes. For example, filters or optical elements can be provided to the user to allow specific image content to be delivered to only each eye.

[0104] In one option, the display device is an augmented reality display that provides an enhanced stereoscopic image pair as an enhanced 3D view of the target superimposed on a view of the current live scene.

[0105] In an alternative option, the display device is a virtual reality display that provides an enhanced stereoscopic image of an enhanced 3D view of the target.

[0106] Figure 6 The basic steps of an example method 200 for visualizing a target of interest are shown. Method 200 includes the following steps: In a first step 202, a 2D X-ray image of the target is provided. In a second step 204, 3D data of the target is provided. The first step 202 and the second step 204 can be performed in this order, in reverse, or simultaneously. In a third step 206, the 3D data is registered to the 2D X-ray image. In a fourth step 208, the viewing direction of the 2D X-ray image is detected. In a fifth step 210, the viewing direction of the matching view for the 3D data is determined based on the viewing direction of the 2D X-ray image. In a sixth step 212, a stereo image pair of the 3D data is generated for the determined viewing direction. In a seventh step 214, the 2D X-ray image is combined with one image from the stereo image pair to obtain an enhanced first stereo image, while the other image in the stereo image pair remains unchanged as the generated second stereo image. In an eighth step 216, the enhanced first stereo image and the generated second stereo image are provided as an enhanced 3D view of the target.

[0107] One image in a stereo image pair is also called the first stereo image, while the other image in the pair is called the second stereo image. The first stereo image and the second stereo image are also referred to as the left stereo image and the right stereo image (or the right stereo image and the left stereo image).

[0108] The stereo image pair includes a left-eye view image and a right-eye view image.

[0109] In the example, combining a 2D X-ray image with one image from a stereo image pair involves projecting one image from the stereo image pair onto the 2D X-ray image.

[0110] In the example, one image in the stereo image pair includes active image pixels on a clear / transparent background. For example, active image pixels include black or colored pixels.

[0111] In the example, registration includes spatial registration of 2D X-ray images with respect to 3D data.

[0112] In the example, the registration of 3D data to 2D X-ray images takes into account the inverse perspective of the 2D X-ray images.

[0113] In one option, reverse perspective includes a focal point and a converging beam about the direction of observation of the target and the X-ray radiation.

[0114] In the example, in the enhanced first stereoscopic image, the portion of the image covered by the 2D X-ray image is smaller than the portion covered by the generated stereoscopic image. Alternatively or additionally provided examples, in the enhanced first stereoscopic image, the 2D X-ray image is provided with greater transparency than the generated stereoscopic image.

[0115] In the example, the portion of the image covered by the 2D X-ray image is at least approximately 30% smaller than the portion covered by the generated stereoscopic image. For example, the portion of the image covered by the 2D X-ray image is approximately 50% smaller. The term "approximately" means a deviation of up to + / - 10% from the corresponding percentage value.

[0116] In the example, the transparency of the 2D X-ray image is at least approximately 30% higher than that of the generated stereoscopic image. For example, the transparency of the 2D X-ray image is approximately 50% higher. The term "approximately" means a deviation of up to + / - 10% from the corresponding transparency percentage value.

[0117] In an example of this method, an enhanced first stereoscopic image is presented to the user's first eye, and a generated second stereoscopic image is presented to the user's second eye.

[0118] In an example of this method, an enhanced first stereoscopic image and a generated second stereoscopic image are provided as a virtual screen projected onto a head-mounted viewing device.

[0119] In the example, the visualization engine is used to render the 3D model from a perspective that matches the orientation of the X-rays. For one of the two eyes, a scene is created that blends the 3D model with a 2D (live) X-ray image. For the other eye, only the 3D model is used to create a scene, but this scene is drawn from a different perspective that matches the orientation of this eye.

[0120] In the example, a way of superimposing X-rays is provided without implying depth: there is no depth of the plane in which the image is placed, nor is there depth within the image (e.g., blood vessels extending at different depths).

[0121] This applies to augmented reality and virtual reality concepts, as well as other variations that utilize stereoscopic observation. The idea that a target can be presented to a single eye, provided there is sufficient stereoscopic correction data, can be used for other medical and non-medical purposes. For example, it could be possible to project a viewpoint onto a 3D scene using a single eye.

[0122] In another exemplary embodiment of the present invention, a computer program or computer program unit is provided, characterized in that it is adapted to run method steps of a method according to an embodiment of the foregoing embodiments on a suitable system.

[0123] In the example, a computer program or program unit is provided for controlling an apparatus according to one of the above examples, the program or program unit being adapted, when run by a processing unit, to perform the method steps of one of the above method examples.

[0124] Therefore, the computer program unit may be stored in the computer unit or distributed across more than one computer unit, and this computer program unit may also be part of an embodiment of the present invention. The computing unit may be adapted to perform or cause the execution of steps of the above-described method. Furthermore, the computing unit may be adapted to operate components of the above-described apparatus. The computing unit is capable of automatically operating and / or executing user commands. The computer program may be loaded into the working memory of the data processor. Therefore, the data processor can be equipped to execute the method of the present invention.

[0125] Various aspects of the present invention can be implemented in a computer program product, which may be a collection of computer program instructions stored on a computer-readable storage device and executable by a computer. The instructions of the present invention can be any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs), or Java classes. The instructions can be provided as a complete executable program, a partial executable program, a modification (e.g., an update) of an existing program, or an extension (e.g., a plugin) of an existing program. Furthermore, some processing of the present invention can be distributed across multiple computers or processors.

[0126] As discussed above, the processing unit (e.g., a controller) implements the control method. A controller can be implemented in various ways, using software and / or hardware, to perform a variety of required functions. A processor is an example of a controller employing one or more microprocessors, which can be programmed using software (e.g., microcode) to perform the required functions. However, a controller can be implemented with or without a processor, and can also be implemented as a combination of dedicated hardware for performing some functions and a processor (e.g., one or more programmed microprocessors and associated circuitry) for performing other functions.

[0127] Examples of controller components that may be used in various embodiments of this disclosure include, but are not limited to, conventional microprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0128] This exemplary embodiment of the invention covers both computer programs that use the invention from the outset and computer programs that use the invention by means of updating existing programs.

[0129] Additionally, the computer program unit may be able to provide all the necessary steps to complete the process of an exemplary embodiment of the method as described above.

[0130] According to another exemplary embodiment of the invention, a computer-readable medium, such as a CD-ROM, is provided, wherein the computer-readable medium has computer program units stored on the computer-readable medium, the computer program units being described in the preceding sections. The computer program may be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0131] However, computer programs can also exist on networks (such as the World Wide Web) and can be downloaded from such networks to the working memory of a data processor. According to another exemplary embodiment of the invention, a medium is provided for making computer program units available for download, said computer program units being arranged to perform a method according to one of the foregoing embodiments of the invention.

[0132] It should be noted that embodiments of the present invention are described with reference to different subjects. In particular, some embodiments are described with reference to method claims, while others are described with reference to apparatus claims. However, unless otherwise stated, those skilled in the art will infer from the above and below that any combination of features relating to different subjects, in addition to any combination of features belonging to one type of subject matter, is also considered to be disclosed in this application. However, all features can be combined to provide synergistic effects beyond the simple addition of features.

[0133] Although the invention has been illustrated and described in detail in the accompanying drawings and the foregoing description, such illustrations and descriptions should be considered illustrative or exemplary, and not restrictive. The invention is not limited to the disclosed embodiments. Those skilled in the art, through studying the drawings, the disclosure, and the claims, will understand and implement other variations of the disclosed embodiments in practicing the claimed invention.

[0134] In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality. A single processor or other unit can perform the functions of several items recited in the claims. Although certain measures are recited in different dependent claims, this does not indicate that combinations of these measures cannot be advantageously used. No reference numerals in the claims should be construed as limiting the scope.

Claims

1. A target visualization medical device (10), comprising: Image supply department (12); Data processor (14); as well as Output section (16); The image supply unit is configured to: provide a 2D live X-ray image of the target; and provide 3D image data of the target as preoperatively generated image data; The data processor is configured to: register 3D preoperative image data to the 2D live X-ray image; detect the observation direction used to generate the 2D live X-ray image; determine, based on the observation direction used to generate the 2D live X-ray image, the observation direction for projecting the 3D preoperative image data to generate a matching view of the 3D preoperative image data; generate a stereoscopic image pair (24) of the 3D preoperative image data for the determined observation direction; and combine the 2D live X-ray image with one image from the stereoscopic image pair to obtain an enhanced first stereoscopic image (26), while the other image from the stereoscopic image pair remains unchanged as the generated second stereoscopic image (28); The output unit is configured to provide the enhanced first stereoscopic image and the generated second stereoscopic image as an enhanced stereoscopic image pair, thereby providing an enhanced 3D view of the target.

2. The device according to claim 1, wherein, The data processor is configured to combine the 2D live X-ray image with one of the images in the stereo image pair, including superimposing the 2D live X-ray image onto the image in the stereo image pair.

3. The device according to claim 1 or 2, wherein, The data processor is configured to provide the combination such that in the enhanced first stereoscopic image i) The portion of the image covered by the 2D live X-ray image is smaller than the portion covered by the generated stereoscopic image; and / or ii) The 2D live X-ray image is provided with greater transparency than the generated stereoscopic image.

4. The device according to claim 3, wherein, The data processor is configured to adjust the portion of the generated stereoscopic image covered by the 2D live X-ray image to a maximum of 75%.

5. The device according to any one of claims 1-4, wherein, The registration includes the spatial registration of the 2D live X-ray image relative to the 3D image data.

6. A system (50) for target visualization, the system comprising: First image display (52); Second image display (54); as well as The target visualization medical device (10) according to any one of claims 1-5; Wherein, one of the first image display and the second image display is configured to present the enhanced first stereoscopic image to one of the user's two eyes; and the other of the first image display and the second image display is configured to present the generated second stereoscopic image to the other of the user's two eyes.

7. The system according to claim 6, wherein, A display device (56) is provided, the display device comprising the first image display and the second image display; The display device is a head-mounted display that provides presentation for each eye of the user; and wherein the enhanced first stereoscopic image is provided to one eye of the user, and the generated second stereoscopic image is provided to the other eye of the user.

8. The system according to claim 7, wherein, The display device is an augmented reality display that provides the enhanced stereoscopic image pair as an enhanced 3D view of the target superimposed on a view of the current live scene.

9. A method (200) for visualizing a medically relevant target, the method comprising the following steps: Provide a 2D live X-ray image of the target (202); Provide 3D image data of the target (204) as preoperatively generated image data; Register the 3D preoperative image data (206) to the 2D live X-ray image; The observation direction (208) is detected to generate the 2D live X-ray image; The observation direction for projecting the 3D preoperative image data to generate a matching view of the 3D preoperative image data is determined (210) based on the observation direction used to generate the 2D live X-ray image. Generate stereoscopic image pairs of the 3D preoperative image data (212) for the determined observation direction; The 2D live X-ray image is combined with one image from the stereo image pair (214) to obtain an enhanced first stereo image, while the other image from the stereo image pair remains unchanged as the generated second stereo image; and Provide (216) the enhanced first stereoscopic image and the generated second stereoscopic image as an enhanced 3D view of the target.

10. The method according to claim 9, wherein, The registration of the 3D image data to the 2D live X-ray image takes into account the reverse perspective of the 2D live X-ray image, which includes a focal point and a converging beam about the observation direction of the target and the X-ray radiation.

11. A computer program product comprising a computer program, said computer program causing the processor to perform the method according to claim 9 or 10 when executed by a processor.

12. A computer-readable medium having a computer program stored thereon, the computer program causing the processor to perform the method according to claim 9 or 10 when executed by a processor.

Images