Apparatus for assisting user in image-based planning and performing surgical procedure

Abstract

An object of the present invention is to provide a device (110) that allows improved assistance in image-based planning and performing of a surgical procedure by a user. An image providing unit (111) provides an anatomical volume image of an anatomy of a patient. A contour enhancement unit (113) enhances an anatomical contour in the defined volume of interest based on the anatomical volume image, thereby producing an enhanced volume image. A projection image generation unit (114) generates an enhanced projection image of the anatomical structure. A composite image generation unit (115) generates a composite image based on the enhanced projection image of the anatomical structure and an original projection image acquired by projection through the anatomical structure along a respective projection ray determined by the projection position, angle, and direction. An interface unit (116) presents the composite image to a user.

Description

Technical Field

[0001] The present invention relates to a device, a method, a computer program product, and a system including such a device for assisting a user in planning and performing a surgical operation based on images. Background Art

[0002] In many surgical operations, providing an accurate image of the anatomical structure to be treated is important not only for planning the surgical operation but also for performing the surgical operation. Providing such an accurate image allows, for example, planning and scrutinizing the path followed by a catheter or the position for placing an instrument. In this context, different kinds of medical images are typically used to plan a surgical operation and to plan itself during the surgical operation. For example, to plan a surgical operation, three-dimensional volume images are typically used, which allow an accurate (especially 3D) assessment of the anatomical structure, while during the operation, projection images are utilized. Since the anatomical structures overlap in the projection images, it is often more difficult for a surgeon to accurately find and follow the anatomical structure of interest in the projection images compared to the situation in the 3D planning images. Therefore, it would be useful to assist the surgeon by providing projection images that accurately show the anatomical structure of interest both during the planning and during the performance of the surgical operation. In addition, it would be useful to give the surgeon a high degree of flexibility in the presentation of accurate projection images (which is only possible if the generation of such accurate projection images is computationally inexpensive). Summary of the Invention

[0003] It is an object of the present invention to provide a device, a method, a computer program product, and a system including such a device for assisting a user in planning and performing a surgical operation based on images. In particular, it is an object of the present invention to provide and improve anatomical projection images generated with low computational resources to the user.

[0004] In a first aspect of the present invention, there is provided a device for assisting a user in planning and performing a surgical operation based on an image, wherein the device comprises a) an image providing unit for providing an anatomical volume image of the anatomical structure of a patient to undergo the surgical operation, b) a volume of interest defining unit for defining a volume of interest in the anatomical volume image, the volume of interest comprising at least a part of the anatomical structure shown in the anatomical volume image, c) a contour enhancement unit for enhancing an anatomical contour in the defined volume of interest based on the anatomical volume image, thereby generating an enhanced volume image of the volume of interest, d) a projection image generating unit for generating an enhanced projection image of the anatomical structure for a predetermined projection position, angle and direction, wherein the enhanced projection image is generated by determining a projection through the enhanced volume image along a respective projection ray determined by the projection position, angle and direction, e) a composite image generating unit for generating a composite image based on the enhanced projection image and an original projection image of the anatomical structure, wherein the original projection image is acquired by projecting through the anatomical structure along a respective projection ray determined by the projection position, angle and direction, f) an interface unit for presenting the composite image to the user.

[0005] Since the anatomical contour is enhanced in the volume image (i.e., 3D image), an enhancement algorithm with low computational cost can be utilized and the anatomical structure can be accurately enhanced. Further, since the projection image is generated based on the enhanced volume image and since the composite image is generated based on the enhanced projection image and the original projection image (e.g., a projection image generated based on a non-enhanced volume image, or a projection image directly acquired from a projection image acquisition unit), a composite image can be provided such that the composite image not only accurately shows the enhanced anatomical contour but also shows other anatomical contours, wherein the presentation of the enhanced anatomical contour relative to other structures can be very flexibly adjusted within the composite image to allow a user (e.g., a surgeon) to find an optimal view for planning or scrutinizing the surgical operation. Thus, the device allows for improved assistance to the user in planning and performing a surgical operation based on an image.

[0006] Typically, the device is configured to assist a user in image-based planning and execution of a surgical procedure (in particular by presenting an improved composite image to the user). The device can implement any form of any hardware and / or software provided by a general or special-purpose computer system. In particular, the device can also be implemented in distributed computing (e.g., can be implemented as part of a computer network), where the functions of the device are provided by different processors, servers, or computer systems. The surgical procedure can be any surgical procedure planned and / or executed using medical projection images. Preferably, the surgical procedure refers to a transcatheter aortic valve implantation or replacement procedure (in particular a minimally invasive procedure), where a new heart valve is inserted, e.g., without removing the old damaged valve.

[0007] The image providing unit is configured to provide an anatomical volume image of the anatomy of a patient to undergo a surgical procedure. Typically, the image providing unit can refer to a storage unit or can be communicatively coupled to a storage unit, where the anatomical volume image has been stored on the storage unit, and the image providing unit is configured to provide the anatomical volume image stored on the storage unit. Additionally, the image providing unit can also receive the anatomical volume image, for example, from an input unit or directly from a corresponding imaging device via an interface, and the image providing unit is then configured to provide the received anatomical volume image. Furthermore, the image providing unit can also refer to a corresponding image acquisition device, e.g., a CT device, an MRI device, etc. The anatomical volume image can be any image indicating the anatomy of the patient that is of interest for the surgical procedure. Preferably, the anatomical volume image is a CT image or an MRI image. The anatomy can be any anatomy that is of interest for the surgical procedure, preferably, the anatomy refers to at least a part of the patient's heart (e.g., the valve region of the patient's heart). In a preferred embodiment, the image is a contrast-enhanced image of the patient's heart. Preferably, the anatomical volume image is a preoperative image acquired for planning the surgical procedure.

[0008] The region of interest definition unit is configured to define a region of interest in an anatomical volume image. Generally, the region of interest includes at least a part of the anatomical structure shown in the anatomical volume image. However, the region of interest can also refer to the entire anatomical image, in which case, the region of interest definition unit simply defines the entire image as the region of interest. The definition of the region of interest by the region of interest definition unit can be automatically performed via user input or a machine-guided user interaction process. For example, in an automated process, the region of interest definition unit can be configured to: define a region of interest based on a predetermined anatomical structure of interest, using the segmentation of the anatomical structure or one or more known image characteristics of the predefined anatomical structure of interest. In addition, other techniques (such as machine learning techniques for defining the region of interest) can be used. In this context, it should be noted that the definition of the region of interest does not have to be very accurate. For example, a rough contour of the region of interest is suitable for the following processes. Therefore, the automated methods and algorithms for defining the region of interest do not have to be very complex. Additionally, the region of interest definition unit can define the region of interest, for example, by presenting the anatomical volume image to the user, where the user can then use the corresponding input device to input the corresponding contour of the region of interest into the volume image. In addition, an interaction process between the user and the region of interest definition unit can be utilized. For example, the region of interest definition unit can automatically determine the region of interest in the anatomical volume image and present the automatically determined result to the user, where the user can then adjust the region of interest in the anatomical volume image.

[0009] The contour enhancement unit is configured to enhance an anatomical contour in a defined volume of interest based on an anatomical volume image. Generally, all known contour enhancement algorithms can be used to enhance the anatomical contour in the defined volume of interest. Preferably, the contour enhancement unit is configured to: enhance the anatomical contour in the volume of interest based on the anatomical volume image by applying a noise-suppressing contour filter to the anatomical volume image in the volume of interest. The use of a noise-suppressing contour filter has the advantage that the enhancement can be performed at low computational cost (i.e., very quickly). Generally, any known noise-suppressing contour filter can be used and applied to the anatomical volume image to enhance the corresponding anatomical contour. In particular, a bilateral edge detection algorithm can be used, for example, as described in the articles “A 3D Image Filter for Parameter-Free Segmentation of Macromolecular Structures from Electron Tomograms” (A. RA et al., PLoS ONE 7(3): e33697, 2012) and “Bilateral edge detectors” (Jose et al., International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE, pp. 1449-1453, 2013). However, the inventors have found that it can be particularly useful if applying the noise-suppressing contour filter includes the following: using a local Hessian matrix calculated for the voxels of the volume of interest; and determining an enhancement value for a corresponding voxel based on the calculated Hessian matrix of the voxels of the volume of interest to enhance the anatomical contour in the volume of interest. Generally, the Hessian matrix is a sparse matrix of second-order partial derivatives of a scalar field. Thus, for an anatomical volume image, known numerical methods for calculating the corresponding derivatives can be used to calculate the Hessian matrix of a voxel based on the value of the voxel of the volume image and also based on the values of the voxels adjacent to that voxel. Accordingly, the Hessian matrix of the voxel takes into account not only the value of the voxel but also the values of the adjacent voxels, which allows for enhanced noise suppression. Preferably, calculating the enhancement value for a voxel includes: using the highest positive eigenvalue of the Hessian matrix of the voxel that is zero-clamped. In particular, in this preferred embodiment, the contour enhancement unit is configured to calculate the eigenvalues of the Hessian matrix of a voxel. Generally, the inventors have found that the three eigenvalues of the Hessian matrix include information about the local gray value curvature around the corresponding voxel (for which the Hessian matrix has been calculated). In particular, to identify cardiac structures such as cardiac valve leaflets (represented as dark contours (especially dark planes) in a contrast-enhanced image), it has been found that these contours can be identified by using the positive eigenvalues in the corresponding Hessian matrices of the voxels belonging to the cardiac valve.Thus, preferably, in the enhanced volume image, the value of a voxel is set to the highest positive eigenvalue of the Hessian matrix for that voxel, clamped by zero. Accordingly, if none of the eigenvalues for a voxel are positive, the voxel will be set to zero in the enhanced volume image, while voxels including one or more positive eigenvalues are set to the highest positive eigenvalue. In particular, this allows for a fast and easy calculation of the enhanced contours of structures, which can include darker gray values in the corresponding anatomical volume image (such as a heart valve). However, if the surgical procedure is interested in other anatomical structures (e.g., structures that appear bright in the anatomical volume image), other rules for setting the values of the voxels in the enhanced volume image based on the Hessian matrix or specifically on the eigenvalues of the Hessian matrix can also be applied. As an alternative to using a contour filter (especially the contour filter described above), the contour enhancement unit can be configured to: enhance the anatomical contours in the volume of interest based on the anatomical volume image by using a trained machine learning-based model, the trained machine learning-based model being configured to: enhance the anatomical contours in the image based on the provided volume image. Corresponding trained machine learning-based models configured to enhance the anatomical contours in the image based on the provided volume image are known. For example, a trained machine learning algorithm can be parameterized using multiple labeled volume images (e.g., volume images that have been provided with enhancement results), where a corresponding machine learning model can then be trained based on these multiple labeled volume images. Examples of such machine learning algorithms that can be used here can be found in the article "Hough-CNN: Deep Learning for Segmentation of Deep Brain Regions in MRI and Ultrasound" (F. Milletari et al., arXiv:1601.07014, 2016). Generally, the enhancement of an anatomical volume image results in an enhanced volume image.

[0010] The projection image generation unit is configured to generate an enhanced projection image of the anatomical structure for a predetermined projection position, angle, and orientation. The enhanced projection image is generated by determining the projection through the enhanced volume image along the respective projection rays determined by the predetermined projection position, angle, and orientation. Generally, methods for generating a projection image from a volume image along the respective predetermined projection position, angle, and orientation are known and can be utilized by the projection image generation unit. For example, a method described in the article “A cost effective and high fidelity fluoroscopy simulator using the Image - Guided Surgery Toolkit (IGSTK)” (R. Gong et al., 《Progress in Biomedical Optics and Imaging》, SPIE Proceedings, 9036.903618, 2014) can be utilized. In particular, since the enhanced projection image is generated based on the enhanced volume image, the enhanced anatomical contours will also be enhanced in the enhanced projection image. However, the enhanced anatomical structure will not be seen very well in the enhanced projection image.

[0011] The composite image generation unit is configured to generate a composite image based on the enhanced projection image and the original projection image of the anatomical structure. The original projection image is acquired by projecting through the anatomical structure along the respective projection rays determined by the predetermined projection position, angle, and orientation that are also used to generate the enhanced projection image. The original projection image can refer to a projection image acquired by a projection image acquisition unit before or during a surgical procedure. In this case, a projection image that accurately shows the anatomical structure can be utilized. For example, in this case, a fluoroscope or an X - ray system or any other type of projection acquisition unit can be utilized to acquire the projection image. However, in a preferred embodiment, the projection image generation unit is further configured to generate the original projection image of the anatomical structure for the predetermined projection position, angle, and orientation, wherein the original projection image is generated by determining the projection through the anatomical volume image along the respective projection rays determined by the projection position, angle, and orientation. In this case, the same or any other known algorithm for determining a projection image from a volume image can be used to generate the original projection image based on the provided anatomical volume image. This allows providing a projection image similar to (e.g., during a surgical procedure) the projection image that has already been used in the planning phase of the surgical procedure without exposing the patient to further imaging procedures, especially projection imaging procedures using ionizing radiation. Thus, it is possible to support and assist the surgeon in planning the surgery while increasing patient safety and comfort.

[0012] Then, the original projection image and the enhanced projection image are utilized to generate a composite image that includes aspects of both the projection image and the enhanced projection image. To generate the composite image, any known algorithm that allows for the fusion of two images (particularly two medical images) can be utilized. Typically, a user can input corresponding preferences for the composite image. For example, the enhanced projection image and the original projection image can be utilized in different ways during the fusion. Additionally, different colors can be used for the images, or the corresponding projection images can be fused with different grayscales. For example, voxels mainly defined by the enhanced projection image can be provided with a different color than voxels mainly defined by the original projection image. This allows for the customization of the composite image such that an image that best suits the user's intent can be provided. This becomes possible because the computational cost of fusing two 2D images is typically very low, and the more computationally intensive task of enhancing the contours has already been performed based on the volume image. Thus, if the user wants to change the composite image (e.g., the appearance of the composite image or the projection direction of the composite image), the task does not need to be repeated. Generally, the composite image allows for the important anatomical structures to be shown with enhanced contrast in the projection image, but also shows other anatomical structures that were not enhanced during the enhancement process. Therefore, the composite image allows for an easy orientation in the anatomical structure for the user while making it easier to find the relevant anatomical structures (i.e., the subsequently enhanced anatomical contours). Accordingly, when the interface unit configured to present the composite image to the user presents the composite image, it aids the user in planning or performing a surgical procedure.

[0013] In one embodiment, the contour enhancement unit is configured to: further apply Gaussian smoothing before enhancing the anatomical contours in the volume of interest. Generally, Gaussian smoothing allows for a more accurate determination of the anatomical contours by removing artifacts that may be caused by a noisy image.

[0014] In one embodiment, the apparatus further includes a registration unit configured to register the composite projection image with the anatomical volume image such that the position of an interest point in the anatomical volume image is associated with the position of the interest point in the combined projection image, and such that the interest point in the composite projection image is associated with a projection ray including the interest point in the anatomical volume image. Since the composite image is generated based on the enhanced projection image, which itself is generated by projecting through an enhanced image volume based on the anatomical volume image, the relationship between the composite image and the anatomical volume image is known. Thus, registration can be easily performed by tracking the position of voxels in the anatomical volume image during generation of the enhanced volume image and projecting through the enhanced image to reach the position in the composite projection image. Additionally, if the original projection image is also generated by projecting through the anatomical volume image, the relationship between these two images can also be used to register the composite projection image with the anatomical volume image. Further, since at least some information about the position of the interest point is lost during projection from the anatomical volume image to the composite projection image, registration is provided such that the interest point in the composite projection image is associated with a projection ray including the interest point in the anatomical volume image.

[0015] Furthermore, preferably, the apparatus further includes an interactive navigation unit configured to provide interactive navigation of the anatomical image volume to a user based on the composite projection image and the registration, wherein the interactive navigation includes: determining the position of the interest point in the composite projection image in the anatomical volume image by determining the voxel that contributes the most along the projection ray associated with the interest point in the composite projection image. Since the interest point indicated by the user in the composite projection image is more likely to refer to a structure presented at an interest point with good visibility (e.g., enhanced) in the composite projection image rather than any anatomical structure surrounding the structure with the best visibility, providing the voxel that contributes the most along the projection ray as the interest point in the anatomical volume image (which is the indicated interest point in the composite projection image) is the most useful and easiest way to associate the interest point in the composite projection image with the interest point in the anatomical volume image. Preferably, in order to determine the voxel that contributes the most along the projection ray, voxels belonging to the enhanced contour are given a higher weight than other voxels along the projection ray. Since the user is most likely interested in the enhanced contour, this allows the interest point to be easily found in the anatomical volume image if the enhanced contour exists at the interest point. This allows for intuitive and easy navigation of the anatomical volume image based on the composite projection image.

[0016] In one embodiment, the device further includes a simulated volume image generation unit, which is configured to generate a simulated volume image based on the acquired projection images and based on the registration between the anatomical volume image and the composite image. Since the registration between the anatomical volume image and the composite image is known (in particular since the projection rays associated with the points of interest in the composite image are known), the acquired projection images (e.g., acquired during a surgical procedure) can be used to generate the simulated volume image. In particular, based on the registration, the anatomical volume image can be modified to fit the acquired projection images in order to generate the simulated volume image. For example, an iterative process can be utilized, in which, through the registration, it is possible to easily iterate back and forth from the modified anatomical volume image to the projection images until the projection images are equivalent to the acquired projection images and the modified anatomical volume image can thus be regarded as the simulated volume image. In particular, this allows for the rapid acquisition of up-to-date volume images during complex surgical procedures (e.g., when 3D replanning is required) without having to interrupt the surgical procedure for acquiring the true 3D volume image of the anatomical structure.

[0017] In a further aspect of the present invention, there is provided a method for assisting a user in image-based planning and execution of a surgical procedure, wherein the method comprises: a) providing an anatomical volume image of the anatomical structure of a patient to undergo the surgical procedure, b) defining a volume of interest in the anatomical volume image, the volume of interest including at least a part of the anatomical structure shown in the anatomical volume image, c) enhancing the anatomical contours in the defined volume of interest based on the anatomical volume image, thereby generating an enhanced volume image of the volume of interest, d) generating an enhanced projection image of the anatomical structure for a predetermined projection position, angle, and orientation, wherein the enhanced projection image is generated by determining the projection through the enhanced volume image along the respective projection rays determined by the projection position, angle, and orientation, e) generating a composite image based on the enhanced projection image and the original projection image of the anatomical structure, wherein the original projection image is acquired by projecting through the anatomical structure along the respective projection rays determined by the projection position, angle, and orientation, f) presenting the composite image to the user. Generally, the method refers to a computer-implemented method.

[0018] In a further aspect of the present invention, there is provided a system for assisting a user in image-based planning and execution of a surgical procedure, wherein the system comprises: a) an image acquisition unit configured to acquire an anatomical volume image, and b) a device as described above.

[0019] In a further aspect of the present invention, there is provided a computer program product for assisting a user in image-based planning and execution of a surgical operation, wherein the computer program product causes the device as described above to perform the method as described above.

[0020] It should be understood that the above-mentioned device, the above-mentioned system, the above-mentioned method, and the above-mentioned computer program product have similar and / or identical preferred embodiments, in particular the embodiments defined in the dependent claims.

[0021] It should be understood that the preferred embodiments of the present invention can also be any combination of the dependent claims or the above-mentioned embodiments and the corresponding independent claims.

[0022] These and other aspects of the present invention will be apparent with reference to the embodiments described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In the following drawings:

[0024] Figure 1 There is schematically and exemplarily shown a system for assisting a user in image-based planning and execution of a surgical operation.

[0025] Figure 2 There is schematically and exemplarily shown a flowchart of a method for assisting a user in image-based planning and execution of a surgical operation, and

[0026] Figure 3 There is schematically and exemplarily shown a CT image generated by using the above-mentioned device and / or method. DETAILED DESCRIPTION

[0027] Figure 1 There is schematically and exemplarily shown a system for assisting a user in image-based planning and execution of a surgical operation. In particular, system 100 includes a volumetric image acquisition unit 120 and a device 110 for assisting a user in image-based planning and / or execution of a surgical operation. The volumetric image acquisition unit 120 (e.g., a CT imaging unit) acquires an anatomical volumetric image of a patient 121 lying on a patient table 122. Then the acquired anatomical volumetric image is provided to the device 110. The device 110 includes an image providing unit 111, a volume of interest definition unit 112, a contour enhancement unit 113, a projection image generation unit 114, a composite image generation unit 115, and an interface unit 116. Additionally, the device can include an input unit 117 and an output unit 118. Generally, the device 110 can be implemented in any form of software / hardware combination of a computing device, in particular, the device 110 can be implemented in the form of distributed computing, where multiple processors at different locations execute various functions of the device 110 described below.

[0028] The image providing unit 111 is configured to provide, for example, an anatomical volume image acquired from the volume image acquisition unit 120. For example, the volume image acquisition unit 120 can provide the acquired anatomical volume image to the storage unit, and the image providing unit 111 can be configured to access the storage unit to provide, for example, the anatomical volume image to the region of interest definition unit 112. However, the image providing unit 111 can also directly receive the anatomical volume image from the anatomical volume image acquisition unit 120, for example, via a wireless communication interface.

[0029] [[ID=z3]]The region of interest definition unit 112 is then configured to define a region of interest in the anatomical volume image, the region of interest including at least a part of the anatomical structure shown in the anatomical volume image. For example, the region of interest definition unit 112 can use an output unit (e.g., the display 117) to present the region of interest to the user. Then, the user can define the corresponding region of interest in the anatomical volume image. However, the region of interest definition unit 112 can also automatically define the region of interest (e.g., using predefined rules or algorithms, especially machine learning algorithms, segmentation algorithms, or corresponding filtering algorithms).

[0030] The contour enhancement unit 113 is then configured to enhance the anatomical contours in the defined region of interest based on the anatomical volume image, thereby generating an enhanced volume image of the region of interest. In particular, the contour enhancement unit 113 can use any known contour enhancement algorithm (e.g., a contour enhancement filter or a trained machine learning algorithm). Details of the preferred examples will be further provided below. The projection image generation unit 114 is then configured to generate an enhanced projection image of the anatomical structure based on the enhanced volume image. In particular, the projection position, angle, and direction are predetermined (e.g., using an input unit 118 implemented in the form of a keyboard, mouse, or any other user interface) to determine the projection through the enhanced volume image along the corresponding projection rays determined by the projection position, angle, and direction. Generally, the methods and algorithms for determining a projection image from a volume image are known and can be used by the projection image generation unit 114.

[0031] Then, the composite image generation unit 115 is capable of generating a composite image based on the enhanced projection image and based on the original projection image. The original projection image of the anatomical structure can be a projection image that has been acquired by a projection image acquisition unit (e.g., an X-ray imaging unit or a fluoroscopic imaging unit), or can be a projection image that has been generated before enhancement (e.g., generated based on an anatomical volume image). Then, the composite image refers to the fusion of the enhanced projection image and the original projection image such that, in addition to the enhanced contours, other anatomical structures are also visible in the composite image. For example, the composite image can be generated as a superposition of the corresponding enhanced projection image and the original projection image using a predetermined color, weight, and / or gray value scale. Moreover, other fusion algorithms for determining the value of a voxel based on two projection images can be used. For example, the value can be determined by adding or weighted adding the corresponding image values, by multiplying the corresponding image values, by selecting the maximum / minimum image value, etc. In particular, regarding the projection images, the user can use the input unit 118 and the output unit 117 to determine different types of composites, such as determining different colors, schemes, different gray levels, etc. of the corresponding projection images for fusion. Then, the output unit 117 can present the composite image generated in this way to the user via the interface unit 116 (implemented in the form of a display, in particular).

[0032] Figure 2 A method 200 for assisting a user in image-based planning and performing a surgical operation is schematically and exemplarily shown. Generally, the computer-implemented method 200 can be performed using the device 110 as Figure 1 described. The method 200 includes a step 110 of providing an anatomical volume image of the anatomical structure of a patient on whom a surgical operation should be performed. Additionally, in step 120, a volume of interest in the anatomical volume image is defined, the volume of interest including at least a part of the anatomical structure shown in the anatomical volume image. In step 230, the anatomical contours in the defined volume of interest are enhanced based on the anatomical volume image, thereby generating an enhanced volume image of the volume of interest. In a further step 240, an enhanced projection image of the anatomical structure is generated for a predetermined projection position, angle, and direction. Generally, the enhanced projection image is generated by determining the projection of the enhanced volume image along the corresponding projection rays determined by the projection position, angle, and direction. Additionally, the method includes a step 250 of generating a composite image based on the enhanced projection image and the original projection image of the anatomical structure. The original projection image is a projection acquired by passing through the anatomical structure along the corresponding projection rays determined by the projection position, angle, and direction. In a final step 260, the composite image is then presented to the user, thereby allowing the user to be assisted in planning / performing the corresponding surgical operation.

[0033] In the following, some preferred embodiments are described in more detail. Preferred applications of the above-described devices and methods relate to transcatheter aortic valve surgery. Transcatheter aortic valve implantation or replacement (TAVI or TAVR) is a minimally invasive procedure in which a new heart valve is inserted without removing the old damaged valve. TAVI procedures are typically performed under intraoperative image guidance based on fluoroscopy (preferably, dynamic X-ray projection images), but preoperative planning is typically performed in preoperative CT image volumes. Although there are techniques aimed at rendering interventional imaging to approximate 3D CT imaging, this information is often not available prior to the intervention. Additionally, with the development of specific transcatheter techniques for treating the mitral and tricuspid / mitral valves, precise anatomical planning for percutaneous valve interventions has become crucial. In order to anticipate the expected fluoroscopy projection images, it is possible to calculate simulations of these images from preoperative CT image volumes for pre-viewing. However, optimal visual assessment / evaluation is necessary for preoperative planning. Although it is possible to simulate the expected fluoroscopy projection images from preoperative CT images, it is often difficult to discern detailed information about fine structures of interest (e.g., valve and cusp / cusplet contours) due to the superposition of multiple anatomical structures and strongly opaque contrast agents. Additionally, automatic segmentation (e.g., adapted geometric models) may be available, but may have delineation / localization defects or unknown accuracy / certainty, such that the possible graphical overlays from the segmentation may make medical users skeptical about how reliable these overlays are. Additionally, machine learning (ML) techniques and artificial intelligence (AI) techniques can also be used to extract structures of interest, but they require a sufficient amount of highly accurate annotations, i.e., training data collected and curated under a careful sampling strategy (preferably from independent experts). All of these problems can be overcome by using, for example, the invention described with respect to Figure 1 and Figure 2 which is used to provide a familiar overall visual representation (i.e., a composite image) of the region of interest to a medical user, the composite image being enriched to a subtle and optionally adjustable degree by additionally emphasizing anatomical contours, without cluttering or overburdening the standard familiar representation. To this end, as described above, automatic generation of the region of interest can be utilized, which does not need to be an exact, flawless delineation of the structure of interest (in this example, the valve, aorta, and outflow tract). Instead, an approximate envelope is sufficient. Then, the enhancement in the composite image allows for accelerated and intuitive navigation to the points of interest to be visually apprehended and measured. Since analytical algorithms (such as contour filters) are preferably utilized for enhancement, there is no need for training data for machine learning (which is accompanied by annotations, sampling, imaging protocol coverage, and regulatory efforts, respectively).

[0034] Thus, the object of this application of the invention (e.g., as already in Figure 1 and Figure 2The one described in (is to provide an intuitive graphical representation to assist in visually comprehending and navigating pre-operative measurements and the points of interest for planning. Examples of preferred embodiments of the method of the present invention include the following computer-implemented steps that can be performed by, for example, the device discussed in Figure 1 . In one step, using X-ray projection forward simulation, a fluoroscopy-like projection image can be calculated based on the CT image volume at a corresponding predetermined projection angle. Additionally, a corresponding algorithm can be used to define an approximate volume of interest (such as around the aortic valve) in the CT volume image. For example, model-based segmentation (MBS) or other machine learning-based semantic segmentation (such as deep CNN) can be used for region-of-interest definition. Additionally, a noise-suppressing contour filter is applied to the CT image volume to collect filter responses along the anatomical contours within the region of interest (preferably having an inherent noise-suppressing property and invariance to contrast agent concentration). For example, contour filtering can include determining the eigenvalues of the Hessian matrix of the second derivative. Then, a composite volume rendering can be used to generate a composite image for unobtrusively embedding the contour filter responses into the simulated projection image. For example, soft alpha blending that gradually decreases at the region-of-interest margin can be used. Then, optionally, the generated projection image can be spatially co-registered to relate the projection image to the 3D CT image volume (especially considering enhanced contours) to enable interactive navigation to the structures at the points of interest in the 3D CT volume image, such as by mouse clicks. Figure 3 illustrates the application of the above method to cardiac CT images. The top row shows image slices re-formatted obliquely from the original scanner grid to rotate around the axis passing through the aortic heart valve. In the left column, before enhancement, the generated fluoroscopy projection image does not visibly show the valve structures (such as the mitral and tricuspid leaflets) due to the superposition of other adjacent anatomical structures. The same simulated rotational projection image is shown in the right column, this time with contour enhancement embedded within the region of interest around the valve.

[0035] In the following, some additional preferred embodiments are described. In one embodiment, a filter response for enhancing a contour in a volume image is calculated based on differential geometric properties. Preferably, after applying an optional initial Gaussian smoothing, a local Hessian matrix of spatial partial derivatives is calculated for each voxel in the region of interest. The derivatives of the Hessian matrix are invariant with respect to the absolute intensity level (e.g., from the varying contrast agent concentration in the ascending aorta and left ventricle). In order to select certain local anatomical structures, combinations or conditions such as planar, tubular or spherical structures of the three real eigenvalues of the symmetric Hessian matrix can be utilized. However, the inventors have found that it is particularly advantageous to sort the eigenvalues of the Hessian matrix not by their absolute magnitude but by their signed magnitude, and to use the value of the largest positive eigenvalue clamped by zero. This eigenvalue designates a locally fairly smooth planar structure, e.g., a low-radiopacity patch typical for a cardiac valve leaflet embedded in contrast-enhanced blood, which is at the same time hardly affected by the stray local noise captured in the smaller positive eigenvalues. In an alternative preferred embodiment, instead of an analytical contour filter from differential geometry, a trained machine learning contour enhancement can also be employed.

[0036] In one embodiment, preferably, during a navigation interaction, if the user clicks on a salient point in the enhanced projection image, the corresponding 3D position (e.g., indicated by a crosshair) and a switch to the position of the slice containing the volume image are shown in the standard slice viewport of the volume image. Generally, the correspondence between the projection rays involving pixels in the enhanced projection image and the 3D positions of the voxels in the CT image volume is not bijective. In order to provide the corresponding voxels, preferably, the most contributing point (e.g., voxel) along the projection ray can be taken as the point of interest. A means for identifying the maximum contribution point can be, for example, the point along the ray having the highest radiopacity among all points belonging to the ray. Preferably, a means for providing more weight to the enhanced contour points along the projection ray is used, such that the user's selection of the enhanced projection image is most likely to result in an enhanced contour point in the three-dimensional CT volume.

[0037] In one embodiment, the enhanced contour is also integrated into the intraoperative image. For example, instead of the generated projection image, intraoperative projection images can also be used to generate a composite image. For example, the forward coordinate correspondence obtained from the image registration between the preoperative CT image and the interventional X-ray image can be used to combine the enhanced projection image with the intraoperative projection image, such that the CT-based contour can be added as an overlay to the interventional X-ray image to enhance these images.

[0038] In one embodiment, the 2D interventional X-ray images can be rendered into 2.5D or 3D images using the forward coordinate correspondence obtained from the image registration between the preoperative CT volume images and the interventional X-ray images. Additionally, the 2D+t interventional X-ray images / sequences can be enhanced into 2.5D+t or 3D+t interventional X-ray sequences by using the available geometric background from the preoperative 3D CT volume images. This allows virtual change of the viewing angle of the 2D+t interventional X-ray acquisition during the intervention.

[0039] In one embodiment, a switch between automatic segmentation and edge-enhanced visualization is preferably provided. For example, as described above, the user can switch between the enhanced analysis contours within the region of interest and the overlays of the automatic segmentation or planning results in order to visually examine the consistency / accuracy / certainty.

[0040] In particular, the above invention can be integrated into imaging workstations and PACS viewers dedicated to the diagnosis and / or preoperative surgical planning for (e.g., structural heart disease (SHD) or valvular heart disease (VHD), transcatheter aortic valve implantation or replacement (TAVI or TAVR)).

[0041] Although the above embodiments mainly relate to CT volume images, other imaging modalities can also be utilized. In particular, the imaging modalities are related to cardiac 3D imaging (e.g., MRI, US, SPECT, PET).

[0042] Although the aortic valve is referred to as the structure of interest in the above examples, the present invention is also applicable in a similar manner to any other organ or anatomical structure.

[0043] By studying the drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement other variations of the disclosed embodiments when practicing the claimed invention.

[0044] In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality.

[0045] A single unit or device can perform the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used advantageously.

[0046] The processes performed by one or several units or devices (e.g., providing anatomical volume images, defining volumes of interest, enhancing anatomical contours, generating enhanced projection images, generating composite images, presenting composite images, etc.) can be performed by any other number of units or devices. These processes can be implemented as program code units of a computer program and / or dedicated hardware.

[0047] A computer program product can be stored / distributed on a suitable medium (e.g., an optical storage medium or a solid-state medium) provided together with or as part of other hardware, but can also be distributed in other forms (e.g., via the Internet or other wired or wireless telecommunication systems).

[0048] Any reference signs in the claims shall not be construed as limiting the scope.

[0049] The object of the present invention is to provide a device that allows for improving the assistance to the user in planning and performing a surgical operation based on an image. An image providing unit provides an anatomical volume image of the anatomical structure of a patient. A contour enhancing unit enhances anatomical contours in a defined volume of interest based on the anatomical volume image, thereby generating an enhanced volume image. A projected image generating unit generates an enhanced projected image of the anatomical structure. A composite image generating unit generates a composite image based on the enhanced projected image and an original projected image of the anatomical structure, wherein the original projected image is acquired by projecting through the anatomical structure along corresponding projection rays determined by a projection position, an angle, and a direction. An interface unit presents the composite image to the user.

Claims

1. A device for assisting a user in image-based planning and performing a surgical operation, wherein, The device (110) includes: - An image providing unit (111) configured to provide an anatomical volume image of the anatomical structure of a patient (121) to undergo the surgical procedure, - A volume of interest defining unit (112) configured to define a volume of interest in the anatomical volume image, the volume of interest including at least a part of the anatomical structure shown in the anatomical volume image, - A contour enhancement unit (113) configured to enhance anatomical contours in the defined volume of interest based on the anatomical volume image, thereby generating an enhanced volume image of the volume of interest, - A projection image generating unit (114) configured to generate an enhanced projection image of the anatomical structure for a predetermined projection position, angle, and direction, wherein the enhanced projection image is generated by determining a projection through the enhanced volume image along a respective projection ray determined by the projection position, angle, and direction, - A composite image generating unit (115) configured to generate a composite image based on the enhanced projection image and an original projection image of the anatomical structure, wherein the original projection image is acquired by projecting through the anatomical structure along a respective projection ray determined by the projection position, angle, and direction, - An interface unit (116) configured to present the composite image to a user.

2. The device according to claim 1, wherein The contour enhancement unit (113) is configured to: enhance the anatomical contours in the volume of interest based on the anatomical volume image by applying a noise-suppressing contour filter to the anatomical volume image in the volume of interest.

3. The device according to claim 2, wherein Applying the noise-suppressing contour filter includes: utilizing a local Hessian matrix calculated for a voxel of the volume of interest; and determining an enhancement value for the respective voxel based on the calculated Hessian matrix of the voxel of the volume of interest for enhancing the anatomical contours in the volume of interest.

4. The device according to claim 3, wherein Calculating the enhancement value for a voxel includes: utilizing the highest positive eigenvalue of the Hessian matrix for the voxel clamped by zero.

5. The device according to claim 1, wherein The contour enhancement unit (113) is configured to: enhance the anatomical contours in the volume of interest based on the anatomical volume image by utilizing a trained machine learning-based model, the trained machine learning-based model being configured to: enhance anatomical contours in an image based on a provided volume image.

6. The device according to any one of the preceding claims, wherein, The contour enhancement unit (113) is configured to: further apply Gaussian smoothing before enhancing the anatomical contours in the volume of interest.

7. The device according to any one of the preceding claims, wherein, The device (110) further includes a registration unit configured to register the composite projection image with the anatomical volume image such that a position of an interest point in the anatomical volume image is associated with a position of the interest point in the composite projection image and such that the interest point in the composite projection image is associated with a projection ray including the interest point in the anatomical volume image.

8. The apparatus according to claim 7, wherein, The device (110) further includes an interactive navigation unit configured to provide interactive navigation of the anatomical image volume to a user based on the composite projection image and the registration, wherein the interactive navigation includes: determining a position of an interest point in the composite projection image within the anatomical volume image by determining a voxel that contributes the most along a projection ray associated with the interest point in the composite projection image.

9. The device according to claim 8, wherein, To determine the voxel that contributes the most along the projection ray, voxels belonging to an enhanced contour are provided with a higher weight than other voxels along the projection ray.

10. The device according to any one of claims 7 to 9, wherein, The device (110) further includes a simulated volume image generation unit (114) configured to generate a simulated volume image based on the acquired projection images and based on the registration between the anatomical volume image and the composite image.

11. The device according to any one of the preceding claims, wherein, The projection image generation unit (114) is further configured to generate the original projection image of the anatomical structure for a predetermined projection position, angle, and orientation, wherein the original projection image is generated by determining a projection through the anatomical volume image along a respective projection ray determined by the projection position, angle, and orientation.

12. The apparatus according to any one of the preceding claims, wherein, The original projection image is acquired by a projection image acquisition unit before or during the surgery.

13. The apparatus according to any one of the preceding claims, wherein, The anatomical volume image is a pre-operative image acquired for planning the surgery.

14. A method for assisting a user to plan and perform a surgical operation based on an image, wherein, The method (200) includes: - providing (210) an anatomical volume image of an anatomical structure of a patient (121) to be subjected to the surgery, - defining (220) an interest volume in the anatomical volume image, the interest volume including at least a portion of the anatomical structure shown in the anatomical volume image, - enhancing (230) an anatomical contour in the defined interest volume based on the anatomical volume image, thereby generating an enhanced volume image of the interest volume, - generating (240) an enhanced projection image of the anatomical structure for a predetermined projection position, angle, and orientation, wherein the enhanced projection image is generated by determining a projection through the enhanced volume image along a respective projection ray determined by the projection position, angle, and orientation, - generating (250) a composite image based on the enhanced projection image and the original projection image of the anatomical structure, wherein the original projection image is acquired by projecting through the anatomical structure along a respective projection ray determined by the projection position, angle, and orientation, - presenting (260) the composite image to the user.

15. A computer program product for assisting a user in image-based planning and performing a surgical operation, wherein, The computer program product causes the device (110) according to any one of claims 1 to 13 to perform the method (200) according to claim 14.

Images