Visualization of objects

JP2023552781A5Active Publication Date: 2024-11-29KONINKLIJKE PHILIPS NV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023534170
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-12-08
Filing Date
2021-11-24
Publication Date
2024-11-29
Estimated Expiration
2041-11-24

AI Technical Summary

Technical Problem

Existing medical imaging technologies for interventional treatments are complex and do not effectively allow for easy visualization of the internal structure of a subject, particularly in providing both spatial information and X-ray image content.

Method used

An object visualization device that aligns 3D image data with a 2D X-ray image, detects the viewing direction, and generates a pair of stereoscopic images, combining one with the 2D X-ray image to provide an enhanced 3D view, using augmented or mixed reality technology for display.

Benefits of technology

Enables the user to view internal structures in 3D with overlaid X-ray information, reducing X-ray exposure and enhancing understanding of clinical anatomy during procedures like vascular navigation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

The present invention relates to the visualization of objects. To facilitate visualization of a subject's internal structure, an object visualization device 10 is provided, comprising an image provider 12, a data processor 14, and an output device 16. The image provider provides 2D X-ray images of the object and also 3D image data of the object. The data processor registers the 3D image data with the 2D X-ray images and detects the viewing direction used to generate the 2D X-ray images. Based on the viewing direction used to generate the 2D X-ray images, the data processor also determines a viewing direction for projecting the 3D image data to generate a matching view of the 3D image data. The data processor generates a pair of stereoscopic images 24 of the 3D image data for the determined viewing direction, and combines the 2D X-ray image with one of the pair of stereoscopic images to produce an enhanced first stereoscopic image 26, while the other of the pair remains as a generated second stereoscopic image 28. The output device provides the enhanced first stereoscopic image and the generated second stereoscopic image as an enhanced pair of stereoscopic images 30, providing an augmented 3D view of the object. While only one of the stereoscopic images contains the additional X-ray image content information, in a combined binocular view the X-ray information is also transferred to the other unmodified image. In this way, the X-ray information is extended to spatial content. The advantage is that only one X-ray image is required, reducing the subject's X-ray exposure.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an object visualization device, a system for object visualization, a method for visualizing an object of interest, and a computer program and a computer-readable medium.

Background Art

[0002] In medical procedures such as interventional therapy, various imaging techniques are used to provide a user with information related to an object in addition to the user's own field of vision. For example, X-ray imaging is used for visualizing an area of interest within a subject, such as visualizing a blood vessel structure. The image data acquired based on X-rays can also be used to generate a 3D model for further visualization, such as for spatial presentation. As an example, US Patent Application Publication No. 2018 / 0310907A1 relates to the provision of object visualization. According to its abstract, a computer system that provides a simulated 2D fluoroscopic image is described. During operation, the computer system can generate a simulated 2D fluoroscopic image based on data within a predetermined 3D image related to an individual's body. For example, the generation of the simulated 2D fluoroscopic image can include forward projection. Further, the forward projection can include calculating a cumulative absorption corresponding to the density along an X-ray line passing through pixels within the predetermined 3D image. Then, the computer system can assign a 3D context associated with a predefined cross-section within an individual's body to the simulated 2D fluoroscopic image. However, this configuration has been found to be very complex.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Thus, it would be necessary to easily visualize the internal structure of a subject.

Means for Solving the Problems

[0004] The object of the present invention is resolved by the subject matter of the independent claims, and further embodiments are incorporated into the dependent claims. It should be noted that the embodiments described below of the present invention also apply to object visualization devices, systems for object visualization, and methods for visualizing objects of interest.

[0005] The present invention provides an object visualization apparatus having an image supply unit, a data processor, and an output unit. The image supply unit is configured to supply a 2DX line image of an object. The image supply unit is also configured to supply 3D image data of an object. The data processor is configured to align the 3D image data with the 2DX line image. The data processor is also configured to detect the viewing direction (observation direction) used to generate the 2DX line image. The data processor is further configured to determine a viewing direction for projecting the 3D image data to generate a matching view of the 3D image data, based on the viewing direction used to generate the 2DX line image. The data processor is further configured to generate a pair of stereoscopic images of the 3D image data for the determined viewing direction, and to combine the 2DX line image with one of the pair of stereoscopic images to produce an enhanced first stereoscopic image, while leaving the other of the pair of stereoscopic images as a generated second stereoscopic image. The output unit is configured to supply the enhanced first stereoscopic image and the generated second stereoscopic image as an enhanced pair of stereoscopic images providing an extended 3D view of the object.

[0006] The advantage is that users are provided with both spatial information (i.e., 3D information) and the content of the X-ray image by simply changing one of the three-dimensional images.

[0007] The term "align" refers to linking 3D image data of an object to a 2D X-ray image of the object so that the 2D X-ray image and the 3D image data are aligned with respect to the perspective observed. For example, a 2D X-ray image is taken at a specific imaging perspective (viewpoint), i.e., a specific image acquisition geometry, resulting in a specific 2D X-ray image with respect to a certain image plane, i.e., the surface of the detector that collects the projected and therefore attenuated X-ray beam, and the X-ray emission direction of the generated X-ray beam. As an example, 3D image data is positioned relative to a 2D X-ray image such that image generation using the 3D image data yields a similar X-ray image if acquired at the same geometric structure.

[0008] For example, 3D data of an object refers to 3D image data of that object. Therefore, the term "align" may refer to aligning the 3D image data of an object with a 2D X-ray image, so that the 2D X-ray image and the 3D image data of the object are aligned within the same spatial reference, that is, they are aligned within the same coordinate system.

[0009] The term "detecting the viewing direction of a 2D X-ray image" refers to detecting the imaging direction of an X-ray imaging system that includes an X-ray source that emits X-rays and an X-ray detector that detects the emitted X-rays that are attenuated by an object as they pass through that object.

[0010] For example, the viewing direction of the X-ray imaging system used to generate the 2D X-ray image is detected.

[0011] The phrase "determining the viewing direction for matching 3D data based on the viewing direction of a 2DX line image" refers to determining the projection direction to be applied to 3D data, such as 3D image data of an object, in order to generate a projection in the image plane that will produce similar image results to the 2DX line image.

[0012] For example, the viewing direction for a simulated projection of an X-ray imaging system that will produce an image matching the 2D X-ray image is determined. For example, "matching view" refers to the "matching image" or an equivalent image result.

[0013] As an example, an object visualization device is provided having an image supply unit, a data processor, and an output unit. The image supply unit is configured to supply a 2DX line image of an object and 3D data of the object. The data processor is configured to: align the 3D data with the 2DX line image; detect the viewing direction of the 2DX line image; determine the viewing direction of the matching view of the 3D image data based on the viewing direction of the 2DX line image; generate a pair of stereoscopic images of the 3D data for the determined viewing direction; and combine the 2DX line image with one of the pair of stereoscopic images to produce an enhanced first stereoscopic image, while leaving the other of the pair of stereoscopic images as a generated second stereoscopic image. The output unit is configured to supply the enhanced first stereoscopic image and the generated second stereoscopic image as an enhanced pair of stereoscopic images that provide an extended 3D view of the object.

[0014] In one example, augmented reality (AR) or mixed reality (MR) technology is provided to display the combined image set.

[0015] This results in the effect of providing information to the user's field of view regardless of their working position. By providing a 3D dataset for stereoscopic viewing, stereoscopic information can be added to the user's field of view regardless of their working position. As an example, a head-mounted display (HMD) such as Google Glass or Microsoft HoloLens is provided.

[0016] For interventional therapeutic applications, this technology enables applications that render 3D anatomical models and (live) X-ray streams in space. Such capabilities help to better understand clinical anatomical structures for navigation, for example, in the case of blood vessels. This technology assists in cases where, for example, a surgeon needs to be able to visualize or track the progression of an instrument through a blood vessel.

[0017] By providing conventional X-ray images in addition to the generated 3D image pairs, the device is not only visualized in two dimensions, but also provides a sense of how the device, vascular system, organs, and other tissues relate to each other in three dimensions.

[0018] One advantage is that while the X-ray image overlaid on the stereoscopic image only provides a 2D image from a single viewpoint, it offers stereoscopic rendering of scenes that require two perspectives (one for each observer's eye).

[0019] In one example, a visualization engine is provided for rendering a scene in 3D.

[0020] In one example, two offset views of the 3D volume are created based on the line of sight from each eye to the volume, which should be displayed to the observer's left and right eyes, respectively. The X-ray image is overlaid on only one of these views.

[0021] As an effect, because it is a single 2DX line image, depth due to binocular parallax is not included. Furthermore, as an effect, by mixing it with a 3D model (presented for two perspective views and having correct binocular parallax), the user can visually combine two offset views—one with a 2DX line overlay and the other without the overlay—without any visual distortion. In one example, a small-sized single-view object is provided to facilitate user perception.

[0022] The X-ray image is used to determine the viewing direction for the projection of the stereoscopic image. The X-ray image is a starting point for a computational spatial setup and thus determines subsequent image generation steps.

[0023] According to one example, the enhanced first stereoscopic image is a dual-image source image, and the generated second stereoscopic image is a single-image source image.

[0024] The term dual-image source refers to two different image sources that supply two different types of images that are then combined as the enhanced first stereoscopic image. The term single-image source refers to a single image source that supplies a single type of image that is supplied as the second stereoscopic image.

[0025] In one example, the enhanced first stereoscopic image is an image having image data from two different image sources, and the generated second stereoscopic image has image data from a single image source.

[0026] In one example, the enhanced first stereoscopic image is an image having an image from a first image source and an image from a second image source different from the first image source. The generated second stereoscopic image has only an image from the first image source.

[0027] In one example, the enhanced first stereoscopic image is an image constructed by combining an image from a first image source and an image from a second image source different from the first image source. The combination is made as an overlay of two different images. The generated second stereoscopic image has only an image from the first image source.

[0028] According to one example, the data processor is configured such that a combination with one of a pair of stereoscopic images of a 2D X-ray image includes overlaying the 2D X-ray image on one of the pair of stereoscopic images.

[0029] In another example, the data processor is configured such that the combination with one of the pair of stereoscopic images of the 2D X-ray image includes overlaying one of the pair of stereoscopic images on the 2D X-ray image.

[0030] According to an example, in the enhanced first stereoscopic image, the data processor is configured to perform the combination such that: i) an image portion covered by the 2D X-ray image is smaller than a portion covered by the generated stereoscopic image. As an additional or alternative option, in the enhanced first stereoscopic image, the data processor is configured to perform the combination such that: ii) a transparency greater than the transparency of the generated stereoscopic image is imparted to the 2D X-ray image.

[0031] According to the present invention, a system for object visualization is also provided. The system has a first image display, a second image display, and one object visualization device described above. One of the first image display and the second image display is configured to present the enhanced first stereoscopic image to one of the user's two eyes. The other of the first image display and the second image display is configured to present the generated second stereoscopic image to the other of the user's two eyes.

[0032] In one example, the system includes a first image display, a second image display, and one object visualization device described above. One of the first image display and the second image display is configured to present the enhanced first stereoscopic image to the user. The other of the first image display and the second image display is configured to present the generated second stereoscopic image to the user.

[0033] In one example, a display device is provided that includes a first image display and a second image display. Optionally, the display device is a head-mounted display that presents to each of the user's eyes, wherein the enhanced first stereoscopic image is supplied to one of the user's eyes, and the generated second stereoscopic image is supplied to the user's other eye.

[0034] For example, the display device is an augmented reality display that provides the enhanced pair of stereoscopic images as an augmented 3D view of an object overlaid on a view of the current live situation.

[0035] The present invention also provides a method for visualizing an object of interest. The method is: - A step of supplying a 2DX line image of the object; - A step of supplying 3D data of an object, such as 3D image data; - A step to align 3D image data with a 2D line image; - A step to detect the viewing direction used to generate the 2DX line image; - A step of determining the viewing direction for projecting the 3D image data in order to generate a matching view of the 3D image data, based on the viewing direction used to generate the 2D line image; - A step of generating a pair of stereoscopic images of 3D image data for the determined viewing direction; - A step of combining a 2D X-ray image with one of a pair of stereoscopic images to produce an enhanced first stereoscopic image, while leaving the other of the pair of stereoscopic images as the generated second stereoscopic image; and - A step of supplying the enhanced first stereoscopic image and the generated second stereoscopic image as an enhanced 3D view of the object; It has.

[0036] For example, the step of aligning the 3D image data to a 2D X-ray image takes into account inverse projection, including the focal point and focused beam relative to the object, as well as the viewing direction of the X-ray emission.

[0037] The term "reverse fluoroscopy" relates to the fact that 2DX-ray images, when generated by an X-ray beam, are based on a fan-shaped or conical beam that has a focal point at the source and diverges toward the detector. Since the radiation direction is from the source to the detector, i.e., from the focal point toward the broader beam, the "viewing" direction of such X-rays also diverges. In visual fluoroscopy, the reverse is done in a specific way; that is, the viewing direction is toward the focal point (i.e., converges toward the viewing direction).

[0038] In one example, the alignment of 3D data to a 2DX X-ray image considers inverse perspective of the 2DX X-ray image. According to one embodiment, a single 2DX X-ray image is overlaid onto only one of the offset views used to generate a stereoscopic view of the 3D volume. These offset views provide visual depth information (also called spatial information) of the object, and the single 2DX X-ray image provides additional information about the object's region of interest. By providing this 2DX X-ray image in one of the stereoscopic views' front-to-back contexts, the information from the 2DX X-ray image is transferred from the two-dimensional world of the X-ray image to the three-dimensional world of the stereoscopic view.

[0039] In one embodiment, a first display image (for one eye) is generated by mixing a 3D model with a 2D (live) X-ray image. Furthermore, a second display image (for the other eye) is generated that displays only the 3D model but is rendered from a different viewpoint. Furthermore, the orientation of the other eye is aligned.

[0040] According to one embodiment, an object visualization device is provided comprising an image supply unit, a data processor, and an output unit. The image supply unit also supplies a 2D X-ray image of the object and 3D data of the object. The data processor aligns the 3D data with the 2D X-ray image and detects the viewing direction of the 2D X-ray image. The data processor also determines the viewing direction of the matching view of the 3D data based on the viewing direction of the 2D X-ray image. The data processor generates a pair of stereoscopic images of the 3D data for the determined viewing direction and combines the 2D X-ray image with one of the stereoscopic images to produce an enhanced first stereoscopic image, while the other stereoscopic image remains as the generated second stereoscopic image. The output unit supplies the enhanced first stereoscopic image and the generated second stereoscopic image as a pair of enhanced stereoscopic images that provide an extended 3D view of the object. Although only one of the stereoscopic images contains additional X-ray image content information, in a combined binocular view, the X-ray information is also transferred to the other unmodified image. In this way, the X-ray information is extended to spatial content. One advantage is that only one X-ray image is required, which means that the subject's X-ray exposure is reduced.

[0041] These and other aspects of the present invention will become apparent from the embodiments described below and will be explained by reference to these embodiments.

[0042] Exemplary embodiments of the present invention will be described with reference to the following accompanying drawings. [Brief explanation of the drawing]

[0043] [Figure 1] Figure 1 schematically shows an example of an object visualization device. [Figure 2] Figure 2 schematically shows an example of a system for object visualization. [Figure 3] Figure 3 shows an example of a pair of three-dimensional images presented to the user. [Figure 4] Figure 4 shows an example of a system for object visualization in a contextual setting within an intervention treatment room equipped with an X-ray imaging system. [Figure 5] Figure 5 shows an example of a head-mounted device for displaying an enhanced first stereoscopic image and a generated second stereoscopic image as an augmented 3D view of an object. [Figure 6] Figure 6 shows the basic steps of one example of a method for visualizing an object of interest. [Modes for carrying out the invention]

[0044] Specific embodiments are described below in detail with reference to the accompanying drawings. In the following description, similar elements are given the same reference numerals in different drawings. Details of configurations and elements, etc., defined in this description are provided to aid in a comprehensive understanding of the exemplary embodiments. Furthermore, well-known functions or configurations are not described in detail, as this could unnecessarily obscure the embodiments. In addition, expressions such as "at least one" preceding a list of elements modify the entire list of elements, not the individual elements in the list.

[0045] Figure 1 schematically shows an example of an object visualization device 10, which comprises an image supply unit 12, a data processor 14, and an output unit 16.

[0046] The image supply unit 12, data processor 14, and output unit 16 can be integrally provided within a common structure such as a housing, as shown in frame 18. Alternatively, they can be provided separately, with data connected to each other.

[0047] The image supply unit 12 is configured to supply a 2D line image of an object, as shown by the first dashed line 20. The image supply unit 12 is also configured to supply 3D data of the object, as shown by the second dashed line 22. In one example, the 3D data of the object is supplied as 3D image data of the object.

[0048] The data processor 14 is configured to align 3D data, such as 3D image data, with a 2DX line image. The data processor 14 is also configured to detect the viewing direction used to generate the 2DX line image and to determine the viewing direction for projecting the 3D image data to generate a matching view of the 3D image data, based on the viewing direction used to generate the 2DX line image. The data processor 14 further generates a pair of stereoscopic images 24 of the 3D image data for the determined viewing direction and combines the 2DX line image with one of the stereoscopic images to produce an enhanced first stereoscopic image 26, while leaving the other of the stereoscopic images as the generated second stereoscopic image 28.

[0049] The output unit 16 is configured to supply the enhanced first stereoscopic image 26 and the generated second stereoscopic image 28 as enhanced paired stereoscopic images 30 that provide an augmented 3D view of the object.

[0050] The term "object visualization" refers to providing a presentation (or image) of an object. The object is visualized by the presentation, which is provided to the user in a visible manner. An extended 3D view of an object can also be referred to as object visualization. The image supply unit 12 may also be called an image data supply unit, image input unit, input unit, or input unit.

[0051] In one example, the image supply unit 12 can be data-connected to an image source device, such as an X-ray imaging device, which supplies 2DX-ray images of an object. In another example, the image supply unit can be data-connected to a data storage unit that stores 2DX-ray images of an object.

[0052] In one example, the image supply unit can be connected to an image source device, such as a CTX-ray imaging device, which provides 3D data of an object. In another example, the image supply unit can be connected to a data storage unit that stores 3D data of an object.

[0053] The data processor 14 may also be called a data processing device, processor unit, or processor. In one example, the data processor is data-connected to the image supply unit and output unit. The data processor 14 is provided as a visualization engine that renders two views of a 3D model relating to two stereoscopic views from viewpoints matching the acquired X-ray direction.

[0054] The output unit 16 can also be called an output unit. In one example, the output unit 16 is data-connected to a display device. In another example, the output unit 16 can be data-connected to a display device. For example, the output unit 16 is configured to provide a pair of enhanced stereoscopic images to a clinician, thereby providing an augmented 3D view of an object.

[0055] The data processor 14 is configured to combine the 2DX line image with only one of the pair of stereoscopic images.

[0056] The term “pair of stereoscopic images” refers to two separate images representing the left-eye view and the right-eye view of the same object. Even when presented as images on a 2D surface, the resulting parallax between views provides the user with depth information, i.e., spatial information. Such an enhanced pair of stereoscopic images includes a first image having a first generated view of the 3D data and a second image having a second generated view of the 3D data.

[0057] In one example, the data processor 14 is configured to take in a pair of generated stereoscopic images, combine one image of the pair with a 2D X-ray image, while retaining the other image of the pair, thereby yielding a pair of images containing one X-ray information-enhanced generated image data and one purely generated stereoscopic image.

[0058] In one example, the data processor 14 is configured to take in the pair of generated stereoscopic images representing the left-eye and right-eye views of the same object, and to combine one of the left-eye and right-eye views with a 2D X-ray image while maintaining the other left-eye and right-eye view, thereby providing a pair of visual views including one X-ray information-enhanced visual view and one visual view generated based purely on 3D data.

[0059] The enhanced pair of stereoscopic images described above provide a partially augmented 3D view of an object by providing spatial information in the form of a stereoscopic view of the pair of images, and by providing additional X-ray information in only one of these images.

[0060] In one example, the data processor 14 is configured to generate a pair of stereoscopic images having first and second stereoscopic images of 3D data for a determined viewing direction; to combine a 2DX line image with one of the first and second stereoscopic images to generate an enhanced first stereoscopic image; and to maintain the other of the first and second stereoscopic images as the generated second stereoscopic image.

[0061] In one example, the data processor 14 is configured to generate a pair of stereoscopic images comprising first and second stereoscopic images of 3D data for a determined viewing direction, and to combine a 2D X-ray image with one of the first and second stereoscopic images. The resulting pair of stereoscopic images includes the first and second images, which contain different types of image content. One of these two images has additional X-ray image content. As an example, one of these two images includes additional X-ray fluorescence fluoroscopy image content combined with the view of the generated 3D data. Both of these two images show the same option with a slightly displaced viewpoint, providing the user with a stereoscopic view. Thus, the second image of the first and second stereoscopic images is provided purely as a generated second stereoscopic image.

[0062] In one example, the second image of the first and second stereoscopic images is provided as the uncombined second image, while the first image of the first and second stereoscopic images is provided as the combined first image (also called the combined first image). The uncombined second image can also be called a single-source image, while the combined first image can also be called a dual-source image or a multi-source image.

[0063] In one example, the second of the first and second stereoscopic images is provided as the second image without overlay, while the first of the first and second stereoscopic images is provided as the first image with overlay (also called the first image with overlay).

[0064] In one example, the first of the first and second stereoscopic images is provided as a dual data source image, the dual data source including data from a 2D X-ray image of the object and 3D data of the object. The second of the first and second stereoscopic images is provided as a single data source image, the single data source being 3D data of the object.

[0065] Please note that the terms "First" and "Second" are provided to distinguish the two images in each pair of images provided for stereoscopic viewing. These terms do not imply any temporal order or other priority.

[0066] Furthermore, it should be noted that the terms “right” and “left” are provided to define that the two images are supplied to the user’s two misaligned eyes, and that this misalignment provides the basis for three-dimensional, i.e., spatial observation. These terms do not impose any restriction on the user’s actual left or right view. Instead of “left and right,” you could also use “first and second” or “one and the other.”

[0067] Optionally, the 3D data is supplied as image data generated preoperatively of the object. Further optional, the 2D X-ray images are supplied as intraoperative, current, or live images, for example, taken during an examination, interventional treatment, or other medical procedure.

[0068] For example, the 3D data is based on CT X-ray images. Alternatively, or in addition to X-ray imaging, other imaging sources such as ultrasound imaging or MRI may be used to generate the 3D data.

[0069] In one example, although not shown in further detail, the enhanced first stereoscopic image is a dual-source image, and the generated second stereoscopic image is a single-source image.

[0070] In one example, although not shown in further detail, the data processor 14 is configured such that the combination of a 2DX line image and one of a pair of stereoscopic images includes overlaying the 2DX line image onto one of the stereoscopic images.

[0071] In one example, although this is not shown in detail, the data processor 14 is configured to perform the above combination such that, in the enhanced first stereoscopic image, the portion of the image covered by the 2DX-ray image is smaller than the portion covered by the generated X-ray image.

[0072] In addition, or instead, the 2DX line image is given greater transparency than the transparency of the resulting stereoscopic image.

[0073] As a result of providing X-ray image content in a reduced manner, for example, by a smaller allocation or increased transparency, the three-dimensional effect is not substantially affected by the combination with X-ray information in only one of the views.

[0074] In one example, although not shown in further detail, the data processor 14 is configured to adjust the image portion of the generated stereoscopic image so that it is covered by a maximum of 75% of the 2DX line image.

[0075] The portion of the generated stereoscopic image covered by the 2DX line image thus represents a maximum of 75% of the surface of the first stereoscopic image actually provided to the user. The remaining minimum of 25% of the surface of the first stereoscopic image actually provided to the user remains uncovered by the 2DX line image.

[0076] Thus, the enhanced first stereoscopic image has an image region or image surface that represents at least a portion of the first stereoscopic image that is not overlaid by the X-ray image, and this portion is, so to speak, unaffected by the combination of the two images. Therefore, the enhanced first stereoscopic image has a primary portion that has a 2D X-ray image overlay on the first stereoscopic image, and a secondary portion that does not have a 2D X-ray image overlay on the first stereoscopic image and is purely the first stereoscopic image.

[0077] In one example, the portion of the image covered by the 2DX image is at least half the portion covered by the generated stereoscopic image.

[0078] In one example, the data processor is configured to adjust the transparency of the 2DX line image to be greater (for example, at least 25% greater) than the transparency of the generated stereoscopic image when the 2DX line image is projected onto one of a pair of stereoscopic images.

[0079] In the alternative example, the data processor 14 is configured to adjust the transparency of the generated stereoscopic image to be greater than (for example, at least 25% or one-third greater than) the transparency of the 2DX line image when one of the pair of stereoscopic images is projected onto the 2DX line image.

[0080] In one example, though not shown in further detail, the alignment includes the spatial alignment of a 2DX line image to a 3D image. The term “spatial” alignment relates to linking the 2DX line image to a reference frame or spatial coordinate system of the 3D image.

[0081] Figure 2 schematically shows an example of a system 50 for object visualization. The system 50 comprises a first image display 52, a second image display 54, and an example of one of the object visualization devices 10 described above. One of the first image display 52 and the second image display 54 is configured to present the enhanced first stereoscopic image to one of the user's two eyes. The other of the first image display 52 and the second image display 54 is configured to present the generated second stereoscopic image to the other of the user's two eyes.

[0082] In the example shown in Figure 2, an optional display device 56 (indicated by the frame) is provided, which includes a first image display 52 and a second image display 54.

[0083] In an example of the options, the display device 56 is a head-mounted display (see below) that presents to each of the user's eyes. The enhanced first stereoscopic image is supplied to one of the user's eyes, and the generated second stereoscopic image is supplied to the user's other eye.

[0084] Figure 3 shows an example of a pair of stereoscopic images presented to the user. Figure 3 shows, as an example, a pair of generated images 80 having a first image 82 or first image portion and a second image 84 or second image portion (i.e., an offset view used to generate a stereoscopic view of a 3D volume). The first image 82 and the second image 84 each show projections of a generated 3D model of a vascular structure. These projections have slightly different viewing directions to achieve a stereoscopic effect for the user. A single 2D X-ray image 86 is overlaid on only one of these offset views. The 2D X-ray image 86 shows, for example, a vascular structure visible after contrast injection. The first image 82 may be provided as the left-eye image, and the second image 84 as the right-eye image.

[0085] Figure 4 shows an example of a system 50 for object visualization in context with an intervention room 100 equipped with an X-ray imaging system 102 having a C-arm 104 with an X-ray source 106 and an X-ray detector 108. The object of interest, for example, a patient 110, is placed on a patient support 112. A display and illumination device 114 is also shown. A console 116 is shown in the foreground with the object visualization device 10, forming the system 50 for object visualization together with a display device 56 having a first image display 52 and a second image display 54. The console may also be configured to control the operation of various devices within the intervention room 100. As an example, the X-ray imaging system 102 provides the console 116 with a current image, i.e., a live image, as indicated by a first data connection line 118. A second data connection line 120 indicates the supply of data from the visualization device 10 to the display device 56. For example, the display device 56 may be provided as a head-mounted device as shown in Figure 5.

[0086] Figure 5 schematically shows an example of a display device 56 as a head-mounted display 58 that presents to each of the user's eyes. The head-mounted display 58 is configured to display an enhanced first stereoscopic image 26 and a generated second stereoscopic image 28 as an augmented 3D view of an object. The enhanced first stereoscopic image 26 is supplied to one of the user's eyes, and the generated second stereoscopic image 28 is supplied to the user's other eye. Such virtual images provided to the user are shown by a projection 59 that provides the user with spatial information in the form of a pair of stereoscopic images, one for each eye. The projection 59 also provides the user with additional information in the form of an overlaid X-ray image visible to the user's right or left eye.

[0087] As an example, a user 60, such as a surgeon, is shown next to a subject 62 on a subject support. Other staff members are also shown. A display 64 is shown in the background, and a C-arm X-ray imaging system 66 is shown on the right side of the figure.

[0088] For example, the head-mounted display in question has two separate displays.

[0089] One example of such a head-mounted display is the glasses known as Microsoft's HoloLens or Magic Leap.

[0090] As another example, the display device may have a single display that projects different images to each of the user's eyes. For example, filters or optical elements may be provided to enable the user to receive only the specified image content for each eye.

[0091] In an example of an option, the display device is an augmented reality display that provides an enhanced pair of stereoscopic images as an augmented 3D view of an object overlaid on a view of the current live situation.

[0092] In an alternative option, the display device is a virtual reality display that provides an enhanced pair of stereoscopic images as an augmented 3D view of an object.

[0093] Figure 6 shows the basic steps of an example of method 200 for visualizing an object of interest. Method 200 has the following steps: In the first step 202, a 2DX line image of the object is supplied. In the second step 204, 3D data of the object is supplied. The first step 202 and the second step 204 can be performed in this order, in reverse, or simultaneously. In the third step 206, the 3D data is aligned with the 2DX line image. In the fourth step 208, the viewing direction of the 2DX line image is detected. In the fifth step 210, the viewing direction for a matching view of the 3D data is determined based on the viewing direction of the 2DX line image. In the sixth step 212, a pair of stereoscopic images of the 3D data for the determined viewing direction is generated. In the seventh step 214, the 2DX line image is combined with one of the pair of stereoscopic images to produce an enhanced first stereoscopic image, while the other of the pair of stereoscopic images remains as the generated second stereoscopic image. In step 216 of the eighth step, the enhanced first stereoscopic image and the generated second stereoscopic image are supplied as an augmented 3D view of the object.

[0094] One of the pair of stereoscopic images is also referred to as the first stereoscopic image, and the other of the pair of stereoscopic images is also referred to as the second stereoscopic image. The first and second stereoscopic images are also referred to as the left and right stereoscopic images (or the right and left stereoscopic images).

[0095] The pair of stereoscopic images includes a left-eye view image and a right-eye view image.

[0096] In one example, the operation of combining a 2DX line image with one of a pair of stereoscopic images includes the operation of projecting one of the stereoscopic images onto the 2DX line image.

[0097] In one example, one of the pair of stereoscopic images includes active image pixels on a clear / transparent background. For example, these active image pixels may be black or colored.

[0098] In one example, the alignment operation includes spatial alignment of a 2D line image with respect to 3D data.

[0099] In one example, the operation of aligning 3D data to a 2DX line image takes into account the inverse perspective of the 2DX line image.

[0100] In the example options, inverse projection includes a beam that is focused and converged with respect to the viewing direction of the object and the X-ray emission.

[0101] In one example, in the enhanced first stereoscopic image, the portion of the image covered by the 2DX line image is smaller than the portion covered by the generated stereoscopic image. In an alternative or additional example provided, in the enhanced first stereoscopic image, the 2DX line image has greater transparency than the generated stereoscopic image.

[0102] In one example, the portion of the image covered by the 2DX image is at least approximately 30% smaller than the portion covered by the generated stereoscopic image. For example, the portion of the image covered by the 2DX image is approximately 50% smaller. The term "approximately" refers to a deviation of up to + / - 10% from the percentage value of each portion.

[0103] In one example, the transparency of the 2DX image is at least approximately 30% higher than the transparency of the resulting stereoscopic image. For example, the transparency of the 2DX image is approximately 50% higher. The term "approximately" refers to a deviation of up to + / - 10% from each transparency percentage value.

[0104] In one example of the method, the enhanced first stereoscopic image is supplied to the user's first eye, while the generated second stereoscopic image is supplied to the user's second eye.

[0105] In one example of the method, the enhanced first stereoscopic image and the generated second stereoscopic image are provided as a virtual screen projected onto a head-mounted observation device.

[0106] In one example, a visualization engine is used to render a 3D model from a viewpoint that matches the direction of the X-rays. For one eye, a scene is generated that mixes this 3D model with a 2D (live) X-ray image. For the other eye, the scene is generated using only the 3D model, but it is rendered from another viewpoint that matches the direction of that eye.

[0107] In one example, the method of overlaying X-rays is provided without depth, i.e., without depth on the plane in which the image is placed, and without depth within the image (for example, blood vessels extending at different depths).

[0108] This is well-suited to the concepts of augmented reality, virtual reality, and other variations that utilize stereoscopic vision. The idea that one object can be rendered for a single eye, as long as there is sufficient stereoscopically correct data, can also be used for other medical and non-medical purposes. For example, a viewpoint could be projected onto a 3D scene using one eye.

[0109] In another exemplary embodiment of the present invention, a computer program or computer program element is provided, characterized in that it is adapted to perform the steps of the method according to one of the embodiments described above on a suitable system.

[0110] In one example, a computer program or program element for controlling a device according to one of the above examples is provided, and the program or program element is adapted to perform the method steps of the example of the method described above when executed by a processing unit.

[0111] The above-described computer program elements can therefore be stored on a computer unit or distributed across multiple computer units, which can also be part of embodiments of the present invention. This computer unit can be adapted to perform or induce the execution of the steps of the method described above. Furthermore, the computer unit can be adapted to operate the components of the apparatus described above. The computer unit can be adapted to operate automatically and / or to execute user instructions. The computer program can be loaded into the working memory of a data processor. The data processor can thus be equipped to perform the method of the present invention.

[0112] Aspects of the present invention can be implemented in a computer program product, which is a collection of computer program instructions stored on a computer-readable memory device that can be executed by a computer. Such instructions of the present invention may be any interpretable or executable code mechanism, including, but are not limited to, scripts, interpretable programs, dynamic link libraries (DLLs), or Java classes. These instructions may be provided as a complete executable program, a partial executable program, a modification to an existing program (e.g., an update), or an extension to an existing program (e.g., a plug-in). Furthermore, some of the processing of the present invention may be distributed across multiple computers or processors.

[0113] As described above, a processing unit (e.g., a controller) implements the control method. The controller can be implemented in various ways using software and / or hardware to perform various required functions. A processor is an example of a controller using one or more microprocessors that can be programmed using software (e.g., microcode) to perform the required functions. However, a controller can be implemented with or without a processor, and can also be implemented as a combination of dedicated hardware for performing certain functions and processors (e.g., one or more programmed microprocessors and associated circuits) for performing other functions.

[0114] Examples of controller components that may be used in various embodiments of this disclosure include, but are not limited to, common microprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0115] This exemplary embodiment of the present invention covers both computer programs that use the present invention from the outset and computer programs that, through updates, transform an existing program into one that uses the present invention.

[0116] Furthermore, the computer program element may be capable of providing all the necessary steps to satisfy the procedure of the exemplary embodiment of the method described above.

[0117] According to a further exemplary embodiment of the present invention, a computer-readable medium such as a CD-ROM is provided, which stores the computer program elements described in the previous paragraph. The computer program can be stored and / or distributed not only on a suitable medium such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but also in other forms, such as via the Internet or other wired or wireless communication systems.

[0118] However, computer programs may be provided via a network such as the World Wide Web, and can also be downloaded from such a network into the working memory of a data processor. According to a further exemplary embodiment of the present invention, a medium is also provided that makes a computer program element downloadable, and the computer program element is configured to perform a method according to one of the above-described embodiments of the present invention.

[0119] It should be noted that embodiments of the present invention are described in relation to different subject matter. In particular, some embodiments are described in relation to claims of a type of method, while others are described in relation to claims of a type of apparatus. However, those skilled in the art will see from the above and below descriptions that, unless otherwise stated, any combination of features belonging to a certain type of subject matter, as well as any combination of features relating to different subject matter, are disclosed in this application. However, all features can be combined in such a way that they provide a synergistic effect beyond a simple collection of these features.

[0120] Although the present invention has been illustrated and described in detail in the drawings and above, such illustrations and descriptions should be considered explanatory or illustrative and not limiting. That is, the present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and realized by examining the drawings, disclosures and claims when carrying out the invention described in the claims.

[0121] In the claims, the phrase “having (including)” does not exclude other elements or steps, and the singular form does not exclude the plural. A single processor or other unit may perform the functions of several items described in the claims. The mere fact that certain means are described in different dependent claims does not imply that combinations of these means cannot be used advantageously. No reference numeral in the claims should be construed as limiting the scope.

Claims

1. 1. A medical object visualization device having an image providing unit, a data processor and an output unit, the image provider provides 2D live x-ray images of an object and provides 3D image data of the object as preoperatively generated image data; the data processor registers 3D pre-operative image data to the 2D live X-ray image, detects a viewing direction used to generate the 2D live X-ray image, determines a viewing direction for projecting the 3D pre-operative image data to generate a matching view of the 3D pre-operative image data based on the viewing direction used to generate the 2D live X-ray image, generates a pair of stereoscopic images of the 3D pre-operative image data for the determined viewing direction, and combines the 2D live X-ray image with one image of the pair of stereoscopic images to produce an enhanced first stereoscopic image, while leaving the other image of the pair of stereoscopic images as a generated second stereoscopic image; the output unit providing the enhanced first stereo image and the generated second stereo image as an enhanced pair of stereo images providing an augmented 3D view of the object. Object visualization medical device.

2. The object visualization medical device of claim 1 , wherein the data processor's operation of combining the 2D live X-ray image with one of the pair of stereoscopic images includes an operation of overlaying the 2D live X-ray image on one of the pair of stereoscopic images.

3. The data processor processes the enhanced first stereoscopic image. i) the image portion covered by the 2D live X-ray image is smaller than the portion covered by the generated volumetric image; and / or ii) the 2D live x-ray image is given a transparency that is greater than the transparency of the generated volumetric image; The medical object visualization apparatus according to claim 1 or 2, wherein the combined operations are performed as follows.

4. The object visualization medical apparatus of claim 3 , wherein the data processor adjusts the image portion of the generated volumetric image covered by the 2D live X-ray image to be at most 75%.

5. The object visualization medical device of claim 1 , wherein the registration comprises spatial registration of the 2D live X-ray image to the 3D pre-operative image data.

6. A system for object visualization comprising a first image display, a second image display and a medical object visualization device according to any one of claims 1 to 5, one of the first image display and the second image display presents the enhanced first stereoscopic image to one of the user's two eyes, and the other of the first image display and the second image display presents the generated second stereoscopic image to the other of the user's two eyes. system.

7. the system comprises a display device comprising the first image display and the second image display; the display device being a head mounted display providing a presentation to each eye of a user, the enhanced first stereoscopic image being provided to one eye of the user and the generated second stereoscopic image being provided to the other eye of the user; The system of claim 6.

8. The system of claim 7 , wherein the display device is an augmented reality display that provides the enhanced pair of stereoscopic images as an augmented 3D view of the object overlaid on a view of a current live situation.

9. A method for visualizing an object of medical interest, comprising: providing a 2D live X-ray image of the object; providing 3D image data of the object as pre-operatively generated image data; registering 3D pre-operative image data to the 2D live x-ray image; detecting a viewing direction used to generate the 2D live X-ray image; determining a viewing direction for projecting the 3D pre-operative image data to generate a matching view of the 3D pre-operative image data based on the viewing direction used to generate the 2D live X-ray image; generating a pair of stereoscopic images of the 3D pre-operative image data relative to the determined viewing direction; combining the 2D live x-ray image with one image of the pair of stereoscopic images to produce an enhanced first stereoscopic image, while the other image of the pair of stereoscopic images remains as a generated second stereoscopic image; providing the enhanced first stereo image and the generated second stereo image as an enhanced 3D view of the object; The method comprising:

10. 10. The method of claim 9, wherein the step of registering the 3D pre-operative image data to the 2D live X-ray image takes into account an inverse fluoroscopy of the 2D live X-ray image, the inverse fluoroscopy including focal and converging beams and viewing directions of X-ray radiation relative to the object.

11. A computer program enabling a processor to carry out the method according to claim 9 or 10.

12. A computer readable medium having stored thereon a computer program according to claim 11.