Interactive image editing

Abstract

The invention relates to a renderer (100) and an interactive method for image editing of medical 3D anatomical data. The method comprises: receiving (S1) a dataset having volumetric image data, the volumetric image data having been acquired from an image acquisition modality (M), the received volumetric image data; and providing (S2) a signed distance field data structure of the received dataset. Further, performing (S3) an editing operation on a user interface (140) for editing at least a portion of the provided signed distance field data structure, and calculating (S5) a visualization of the editing operation, and displaying (S6) the calculated visualization on a display (130).

Description

Technical Field

[0001] The present invention relates to medical volume editing, in particular to interactive editing of anatomical data, and rendering techniques for editing and visualization of editing operations. Background Art

[0002] Interactive volume rendering, as a subfield of visualization, has matured over the past decade. Several approaches exist that allow domain experts to visualize and explore volumetric datasets at interactive frame rates on standard graphics hardware.

[0003] In particular, in the field of medicine, quality is crucial, since diagnostic and / or therapeutic measures may be based on image data.

[0004] For example, vascular diseases are one of the leading causes of death worldwide. This has driven the development of a wide range of medical imaging techniques dedicated to the visualization of vascular structures and the blood flowing inside, such as magnetic resonance imaging (MRI) and X-ray computed tomography (CT). Moreover, for example, induced angiographic images provide valuable aiding information for clinical applications such as diagnosis and surgical planning. However, these images are complex to analyze due to the sparsity of blood vessels, their complex topological and geometrical properties, low signal-to-noise ratio, and the presence of noise and artifacts inherent to the flowing blood signal. In this challenging context, computer-aided tools dedicated to image interaction (making annotations and labeling in the volume) as well as processing and analysis are indeed mandatory to aid clinicians in their daily practice.

[0005] Typically, many existing volume editing or segmentation systems directly modify the binary voxel map or label map, resulting in interpolation artifacts at the segmentation boundaries. In many cases, the drawn region is only used as a seed for the automatic segmentation algorithm, and the actual drawing and its rendering quality are not a primary concern. Some prior art systems may have limitations that the quality of the rendered image volume needs to be improved.

[0006] Existing systems can derive smoothed masks or anti-aliased signed distance fields (SDFs) from binary segmentation, which can reduce rendering artifacts. Typically, such algorithms are not suitable for use during interactive editing or segmentation. Summary of the invention

[0007] It is therefore an object of the present invention to improve prior art systems, in particular to improve the quality of interactive editing and visualization.

[0008] The object is accordingly solved by a method, a renderer or a rendering system, a computer program product and a computer readable medium for interactive editing according to a first aspect of the invention. Advantageous aspects, features and embodiments are described in other aspects and in the following description together with advantages.

[0009] In the following, the solution according to the invention is described with respect to the claimed method. Features, advantages or alternative embodiments of this document may also be assigned to other claimed subjects (e.g., to a renderer or a computer program product), and vice versa. In other words, the aspects claimed for the renderer may be improved using the features described or claimed in the context of the method. In this case, the functional features of the method are correspondingly embodied by the structural elements of the system, and vice versa. In general, in computer science, software implementations and corresponding hardware implementations are equivalent, at least from the perspective of computability. Thus, for example, the method steps for "displaying" data may be performed using a display unit (monitor, screen) and corresponding instructions. To avoid redundancy, although the renderer may also be used in the alternative embodiments described with reference to the method, these embodiments are no longer explicitly described for the device.

[0010] In general, the features and embodiments mentioned or claimed herein may be selectively combined.

[0011] In one aspect, the invention relates to a method for interactive editing of volumetric image data, comprising the following method steps:

[0012] - receiving a dataset having volumetric image data acquired from an image acquisition modality;

[0013] - displaying the received volumetric image data;

[0014] - providing a signed distance field data structure of the received data set, wherein in particular the signed distance field is an aid to the volume image data (or anatomical volume data); it is a representation of the distance to the segmented objects, or a specific structure of the anatomical data;

[0015] - performing an editing operation on a user interface for editing at least a portion of the provided signed distance field data structure;

[0016] - Visualization of computational editing operations.

[0017] Preferably, the method can be further performed by:

[0018] - Displaying or rendering the computed visualization on a display.

[0019] Compared with existing technologies, the differentiation aspects involve:

[0020] - Apply ray casting algorithms for preview and continuous / layer depth picking; and / or apply path tracing algorithms for photorealistic final rendering;

[0021] - Build (one or more) SDFs on the fly, combining segmentation algorithms and SDF / geometry / data-assisted brushes.

[0022] The technique described in this article concerns high-quality rendering with SDF, but in the context of interactive volume editing. Following are the different parts of the algorithm:

[0023] Interactive Editing:

[0024] -Build SDF in real time, combining data from pre-segmentation algorithms and SDF / geometry / data-assisted user-controlled brushes;

[0025] - Fusion of registered SDFs from different sources, e.g. high-resolution pre-segmentation, low-resolution and / or high-resolution from interactive editing, layering, adaptive resolution, etc.

[0026] Rendering:

[0027] - Direct and / or indirect SDF-based visualization techniques that support interactive editing / annotation workflows, such as using rendering to define and pre-visualize regions marked for masking.

[0028] - Support for interactive / lightweight volume editing workflows with different rendering algorithms relative to the final render (e.g. with or without a depth image locked from an interactive raycaster during editing used to drive brush strokes).

[0029] Compared to several existing systems, the entire segmentation pipeline, and in particular the method or system described herein, uses signed distance fields (SDFs), which allow for subvoxel accuracy, significantly improved rendering quality, and many special effects that are difficult to achieve with traditional voxel masks. The main focus is on interactive editing of medical data and mask creation. Rendering of the resulting data, i.e., computational visualization of the editing operations, is optional.

[0030] The SDF encodes the distance to a specific object contained in the received dataset, for example, the SDF is a distance transform for organ segmentation. In general, one or more SDFs may be used.

[0031] The system can also start with an empty SDF, which is then fully created through editing operations.

[0032] As opposed to direct binary voxel rendering of segmented objects, signed distance fields as a local data structure enable reconstruction of smooth boundaries even for low-resolution data and enable real-time editing and segmentation workflows with high quality.

[0033] Preferably, the method and system supports real-time volume rendering, sculpting and erasing as part of both pre-processing and real-time annotation workflows.

[0034] In preferred embodiments, the method and system can be used to support educational and clinical workflows.

[0035] A major advantage of the solution described herein is the use of SDF for interactive editing operations, which enables, for example, annotation, advanced cropping and volume editing at subvoxel resolution independent of the imaging data. Furthermore, it allows combining segmentation data with SDF operations. Furthermore, the method does not require pre-processing of the volumetric image data, for example by providing segmented image data.

[0036] The volumetric image data may be medical image data acquired from a medical image modality, such as, for example, MRI, CT, PET, SPET, ultrasound, and others. The volumetric image data may represent anatomical structures of a human or animal. Other embodiments relate to non-anatomical objects (implants and / or artificial structures) in a medical or anatomical volume, or to non-medical uses for industrial applications, such as, for example, fault analysis in chip technology.

[0037] The calculation of the visualization is performed by executing at least one rendering algorithm. Typically, there is a first visualization of the received volumetric image data, which is the basis for the editing operation. After or during the execution of the editing operation, a second visualization is calculated and displayed, encoding the editing operation with the respective functions (marking, erasing, etc.).

[0038] Interactive rendering can use any volume rendering technique, but high-performance volume rendering techniques may be more suitable for maintaining interactivity, such as interactive volume ray casting. For more details on interactive volume ray casting, see J., WESTERMANN R., "Acceleration techniques for GPU-based volume rendering" (in Proc. of IEEE Visualization 2003 (2003), pp. 287-292). The final rendering can use the same or a different volume rendering algorithm that provides more realistic rendering, such as Monte Carlo volume path tracing, see, for example, Kroes T, Post FH, Botha CP (2012) "Exposure Render: An Interactive Photo-Realistic Volume Rendering Framework" (PLoS ONE 7 (7): e38586).

[0039] In other embodiments, volumetric ray casting with ray traced shadows, depth shadow maps, ambient occlusion, and / or other global illumination effects such as single or multiple scattering approximations, reflections, color bleeding is used.

[0040] Providing a signed distance field data structure means generating a second data structure based on the volumetric image data, which second data structure is subjected to editing and rendering. The term "second" is to be interpreted as "secondary to the anatomical image (received volumetric image data) and based on the anatomical image". In other words, in addition to the main data structure having or representing the volumetric data, an additional data structure is generated.

[0041] The editing of volumetric image data and the visualization of the edited volumetric image data calculated therefrom can be done in a completely unrestricted manner. The user can freely determine the part of the volume to be edited and / or freely determine the type of editing. The editing can be marking, shading, texturing, coloring and / or other types of editing of a partial volume or the entire volume. The editing can also be highlighting or other types of interaction with the volume (e.g., temporarily highlighting without saving). In addition, the function set can be used, for example, by activating a keyboard shortcut combination before or during the execution of the editing operation and / or by the user's mouse input. Therefore, a specific function for processing the marked structure in the volumetric image data is determined. The processing can include erasing the edited substructure, changing and / or annotating the substructure. The editing is performed at runtime. The user interacts directly with the SDF data structure. In the case of performing multiple editing operations, multiple SDFs are generated and stored. The SDF is updated upon user interaction.

[0042] In addition, multiple operations can also be performed on the same SDF. For example, multiple drawing operations (as edit operations) can be performed on the SDF, and multiple drawing operations can be performed to draw ribs in one SDF and the skull in another SDF. This provides more flexibility. Different edit results can be stored separately. Therefore, the basic anatomical structure on which the edit operation (edit) has been performed is redundantly stored.

[0043] The SDF is a separate 3D image that is registered to the received volumetric image data, which is used as the primary anatomical image (or primary volume). All operations are performed only on the SDF, but during rendering of the primary anatomical image, the SDF is sampled to achieve various effects, such as changing the optical opacity of the anatomical structure based on the distance encoded in the SDF.

[0044] The visualization of the editing operation (i.e., rendering) may be configured. For example, the edits may be visualized as a 3D texture in the edited volume and / or colored in a specific pre-configured color. Different representations of the final effect of the editing operation may be provided (e.g., semi-transparency during erasure versus full transparency for the final rendering).

[0045] In some embodiments, the visualization of the calculations may be direct and / or indirect or a combination thereof. This may be set up depending on the specific workflow.

[0046] Direct visualization of the SDF shows the distance values ​​encoded in the SDF, e.g. as colors during rendering, or as text labels under the mouse cursor, e.g.. For example, displaying the distance values ​​themselves as text is a direct visualization. Ray casting of the SDF volume as an isosurface is another example of a direct visualization. Isosurfaces or any level set rendering technique can also be used. Visualizing the SDF using colored glyphs where the color encodes the distances is also an example of a direct visualization.

[0047] Using distance values ​​to color the anatomical structure would be an indirect rendering. Indirect visualization modifies the primary volume rendering based on the distance information, for example, by changing the optical color and / or opacity of the anatomical data during volume rendering to preview an erase operation.

[0048] The difference is that no other parts of the rendering process are modified and no color is applied to the primary anatomical data (volume image data).

[0049] In a preferred embodiment, the editing operation is performed by masking the volume image data based on the values ​​of the SDF during the editing operation, wherein the values ​​of the SDF may in particular be sampled using some reconstruction filter. Thus, masking is an example of an operation that may be performed using distance data from the SDF.

[0050] Masking or mask-based algorithms use masks to edit specific parts of the volume. The mask can be reconstructed using linear or nonlinear filters to achieve predefined smoothing characteristics. In a preferred embodiment, B-spline filtering can be used. For more details, refer to "Fast third-ordertexture filtering" by Sigg, Christian, Hadwiger, Markus (GPU Gems 2, pp. 313-329, 2005). Preferably, filtering is performed when the SDF is sampled during a rendering operation. Examples of nonlinear filters may include, for example, anisotropic, viewpoint-dependent, morphological and / or random filters. Alternatively, or cumulatively, a mask can be generated by using a segmentation algorithm.

[0051] In a preferred embodiment, the SDF is initialized from, or combined into, an existing segmentation (e.g., a skeleton or organ mask). In this way, the segmentation mask can be refined using interactive editing tools.

[0052] In another preferred embodiment, the editing operation is performed by using a brush-based algorithm. In other embodiments, other algorithms can be used to implement user-defined editing operations, i.e., the editing operation is based on user interaction. Alternatively, the editing operation can be automatic. The editing operation is then implemented by algorithm instructions and is not based on user interaction.

[0053] According to a preferred embodiment, the rendering system or method establishes a mapping between the rendered image (computed visualization) and the 3D patient space (e.g., through the viewing and projection matrices of a perspective camera). Thus, for example, by using the depth data generated during volume editing, brush strokes on the screen are then mapped to brush strokes in the 3D patient space.

[0054] In a preferred embodiment, the anatomical structure and the SDF are registered to each other in patient space, and this registration is used when applying editing operations on the SDF. This registration allows, among other things, the anatomical structure and the SDF to have different resolutions. For example, a high-resolution SDF from an offline segmentation algorithm is combined with a lower-resolution SDF for faster interactive painting / editing; in addition, anisotropic data from a scanner (e.g., thick slices) is combined with an isotropic SDF for more consistent handling of brush operations.

[0055] Brush-based algorithms can use brush strokes in the volume by using an interaction tool (such as, for example, a mouse) and modifier keys, where the modifier keys have certain functions, such as for marking volume structures and / or erasing volume structures. A brush stroke is a user interaction signal that can be considered as a drawing brush in the volume.

[0056] The brush-based algorithm uses brush strokes in the volume as input and performs calculations in response to the voxel data, which corresponds to locations in the SDF data structure, identifying the locations in the SDF.

[0057] When brush input is received, the algorithm updates all voxels near the brush position with the distance from the brush position. This information is later used during rendering to determine the edges of the brush stroke.

[0058] When rendering a volume (with or without an SDF), a depth image is also generated, which can optionally be used to perform drawing or editing operations on the "visible surface" of the volume dataset when used with, for example, a mouse and a 2D screen. Although strictly speaking this is not necessary. Other embodiments involve editing or drawing using a virtual reality (VR) headset and a 3D controller, and this information can be used to create 3D strokes. In this case, the depth of the volume data is irrelevant because the brush input has its own depth information.

[0059] Therefore, when a brush stroke is received within a specific portion in the volume, it can be automatically determined to which anatomical part the brush stroke is associated or at which depth the brush stroke is performed so that the editing signal in the SDF structure can be converted into corresponding volumetric image data (positional correlation between the displayed volumetric image data and the SDF data structure).

[0060] In the preferred embodiment, two types of editing operations are performable:

[0061] 1. First, organ-specific editing operations can be performed. Thus, it can be automatically determined which organ, e.g. a rib or a bone, or an organ (heart, liver, etc.) an editing operation (e.g. performed by a brush stroke) involves. However, this requires pre-processing of the data, in particular by means of performing whole or part body segmentation algorithms.

[0062] 2. Second, the solution proposed in this paper also covers the case where voxel classification modifies the SDF editing operation (e.g., only erase voxels within a given Hounsfield unit (HU) range) during volume rendering (computation of visualization). This can be done with a separate classification function without the need for pre-segmentation.

[0063] According to another preferred embodiment, the method can be used recursively.Recursive editing requires making the SDF into a main three-dimensional volume using some direct visualization method, and then using editing operations directly on it.

[0064] An example of fully recursive editing is using SDF tools on anatomical data to create a first SDF, then switching to direct rendering of the SDF (e.g., using isosurfaces or other level set methods), defining a secondary SDF, and using SDF tools to modify the first SDF itself.

[0065] However, in a preferred embodiment of the present invention, use of the secondary rendering operation, for example to undo the first rendering operation, is implemented as another logical operation between two independent rendering operations on the primary anatomical volume before submission to the SDF (and therefore considered to be applied iteratively).

[0066] According to another preferred embodiment, the method can be used iteratively by using the calculated visualization, encoding the performed editing operations as volume image data for performing further editing operations in the calculated visualization, the further editing operations being iteratively displayed as further calculated visualizations. This has the technical effect that corrective measures can be applied to the calculated visualization by performing further editing operations for, for example, erasing volume parts or marking volume parts.

[0067] An example of an iterative editing operation may be given by the following sequence: taking an image of the received volumetric image data, applying the edits / drawings / annotations etc., re-rendering the image, and then using the new image to define further editing operations. A specific medical example is first performing skull stripping, and then in an iterative step, and based on the rendering with the skull removed, drawing voxels of blood vessels in the now exposed brain or head.

[0068] A separate or second iterative editing operation will "bake" the results of the first editing step into the volume data. The edits are thus embedded in the volume data, and in particular are unavoidably embedded therein. Thus, preferably, the editing operation has modified the original anatomical voxels. The editing will then continue based on the rendering of the modified data. This accumulates the edits into the anatomical volume data, whereas the previous example accumulated the edits into the SDF data.

[0069] According to another preferred embodiment, a plurality of corresponding or different editing operations are performed on different parts of a volume to be jointly represented in the calculated visualization.

[0070] For example, an editing operation may refer to highlighting a particular first anatomical structure in a volume using a first visualization, and highlighting a second structure in the same volume using a second visualization. Thus, for example, a broken rib may be colored red, while an intact rib may be colored green.

[0071] According to another preferred embodiment, the visualization of the computationally edited volumetric image data uses a segmentation algorithm or an SDF-based brush or manually indicated editing.

[0072] The segmentation algorithm can be a fully automatic segmentation algorithm, in particular, for example, based on deep learning or model-based automatic organ contouring. Alternatively, the segmentation algorithm can be performed at least in part by user interaction, in particular, for example, by applying region growing segmentation. Alternatively, a SDF-based brush can be applied. The brush can be a static shape (e.g., a sphere, a cylinder, etc.) that is voxelized during an SDF editing operation, for example, updating the distance values ​​of the SDF voxels covered by the sphere. Alternatively, the brush itself can be defined as a static or dynamic SDF, for example, the brush SDF is adapted based on some anatomical image metrics (such as gradient magnitude). Applying the brush to the SDF then involves SDF logical operations.

[0073] According to another preferred embodiment, the visualization of the computational editing operations and thereby the computational edited volumetric image data is performed by directly transforming and embedding the user-defined editing operations in the three-dimensional space in the signed distance field data structure of the volumetric image data. The term "directly embedded" relates to the fact that rendering the computational visualization is performed in parallel or together with the editing. There is no first rendering of the volumetric image data and a subsequent second separate rendering of the editing operation(s), instead these two steps are performed jointly.

[0074] In another preferred embodiment, the visualization of the computationally edited volumetric image data is based on a rendering algorithm, in particular a ray casting algorithm. For details of the ray casting algorithm, please refer to J., WESTERMANN R., "Acceleration techniques for GPU-based volume rendering" (in Proc. of IEEE Visualization 2003 (2003), pp. 287–292). Other possibilities include volumetric ray casting with ray-traced shadows, deep shadow maps, and / or ambient occlusion.

[0075] Preferably, a subset of volume rendering lighting techniques is selected for application and use during interactive rendering, i.e. techniques with lower rendering and refresh times.

[0076] Alternatively or cumulatively, other rendering algorithms may be applied to apply the Monte Carlo path tracing algorithm, see, for example, Daniel "A Survey of Volumetric IlluminationTechniques for Interactive Volume Rendering" by et al. (COMPUTER GRAPHICS forum, Volume 0 (1981), Number0, pp. 1–25).

[0077] In a further preferred embodiment, for displaying the received volumetric image data, a first rendering algorithm is used, and for computing a visualization of the edited volumetric image data with the performed editing operations, a second rendering algorithm is used, in particular a level set algorithm for computing a smooth segmentation boundary representation from the SDF volume.

[0078] The first rendering algorithm and the second rendering algorithm can be the same. The second rendering algorithm can be a level set algorithm that discretizes the entire volume and defines a function phi instead of an evolution curve. Typically, the level set method in rendering involves, for example, thin isosurfaces and thick isosurfaces for visualizing scalar fields. More details of the level set algorithm can be found in the following literature: "Interactive deformation and visualization of level set surfaces using graphics hardware" by AE Lefohn, JMKniss, CD Hansen and RT Whitaker (IEEE Visualization, 2003.VIS 2003, Seattle, WA, USA, 2003, pp. 75-82, doi: 10.1109 / VISUAL.2003.1250357).

[0079] The SDF is an implicit representation of a segmented object, where the zero level set is the surface of the segmented object, and for each 3D position, the SDF can be sampled to compute the distance to the segmented object. Level sets are used directly during rendering.

[0080] Sampling the SDF using a reconstruction filter can recover the smooth distance to the boundary for any localization in patient space. According to embodiments of the solution described herein, this is used together with sampling and classification of anatomical data to enable volume editing, cropping and / or annotation.

[0081] In yet another preferred embodiment, the visualization of the computationally edited volumetric image data uses a set of voxel classifiers to classify different volumetric characteristics, including different coloring values, different opacity values, and / or different shading techniques.

[0082] Classification (or, respectively, classifiers) in volume rendering maps scalar values ​​in the imaging data (e.g., in HU for CT data) to optical properties (e.g., color, opacity, absorption / extinction coefficient for 1D transfer functions) that are integrated during rendering. A transfer function can map data gradients or local shape descriptors to optical properties.

[0083] In a further preferred embodiment, the editing operations are encoded and visualized as a drawing mask, in particular a semi-transparent drawing layer, in the displayed visualization of the calculation, wherein the drawing mask can be individually configured.

[0084] In yet another preferred embodiment, an undo / redo function is provided for editing operations. The technical advantageous effect of this feature is that computing power can be saved and efficiency can be improved. This is because, as part of the undo / redo, the SDF can be rebuilt from the stored brush strokes and SDF operations. Furthermore, the method can preferably be optimized by dynamically computing a set of checkpoints so that only a subset of the operations need to be reapplied.

[0085] In general, compared to traditional voxel-based volume editing, the above solution using SDF allows for editing operations performed with sub-voxel accuracy that are also fully decoupled from the volume data. This approach is more efficient than purely procedural volume editing and scales to more operations.

[0086] External segmentation, static / geometric brushes, and static / adaptive SDF-based brushes are combined in a unified SDF-based volume editing system.

[0087] In another aspect, the solution presented herein relates to a renderer for interactive editing of volumetric image data, which is configured to perform the method for interactive editing as described above, comprising:

[0088] - an image interface configured to receive a data set having volumetric image data acquired from an image acquisition modality;

[0089] - a memory interface to a memory storing a signed distance field data structure of the received data set;

[0090] - a display interface to a display for displaying the received volumetric image data;

[0091] - a user interface configured for performing an editing operation for editing at least a portion of the displayed volumetric image data;

[0092] - A processing unit for computing visualizations of the editing operations performed.

[0093] The processing unit may be further configured to instruct the second display to display the visualization of the calculation.The display and the second display may be identical.

[0094] In another preferred embodiment, the processing unit may be a graphics processing unit or a central processing unit. The processing unit is configured to execute at least one algorithm, and is particularly configured to execute the method for interactive editing as described above. The processing unit provides a control signal for instructing a display device to show a visualization of the calculation.

[0095] In another preferred embodiment, the display and the user interface can be implemented in the same module or computer.For example, the display can be provided as a graphical user interface and therefore also serves as a user interface.

[0096] According to preferred embodiments and uses of the present invention, the method may be used in educational and / or clinical workflows.

[0097] In particular, in a distributed scenario where multiple users work at different computing locations (e.g., if a teacher is located at a first location and students are located at a different second location, which may be the case in a home teaching scenario or virtual application), this may result in a very advantageous effect, that is, a user interface configured for performing editing operations on a signed distance field is implemented on a first computing entity, while a second display for displaying the editing operations is implemented on a second computing entity. With this deployment, editing operations can be performed on the first computing entity, and the results of the editing operations are displayed on the second computing entity, which the teacher can use to evaluate and track the students' editing operations. With this setup, the teacher can instruct the students to perform specific markings on the volume, and the students are then required to apply the markings. The results will be displayed on the display of the teacher at a distance.

[0098] The proposed approach can be applied to collaborative annotation in both didactic and clinical scenarios as well as other scenarios of distributed editing / rendering. For example, an instructor with a tablet running a preview rendering can make real-time annotations for high-quality display during a lecture.

[0099] Distributed annotation, for example as used in education, is an important use case, and the approach presented in this paper has specific technical advantages:

[0100] (1) Unified processing of pre-segmented data and real-time annotations (e.g., during lectures).

[0101] (2) Fusion and unified handling of SDFs from different sources, e.g., low-resolution SDFs edited during live lectures from a low-power tablet device combined with collaborative editing / annotation from exports on a workstation (enabled by supporting multiple registered SDFs).

[0102] (3) Fusion of multiple rendering algorithms for different display devices during editing, e.g., ray casting on a tablet, ray casting combined with global illumination on a workstation, and path tracing on a lecture hall display.

[0103] According to another aspect, the solution described herein relates to a computer program product comprising a program element which, when loaded into a memory of a computer, causes the computer to perform the steps of the method for interactive editing of volumetric image data as described above.

[0104] According to a further aspect, the solution described herein relates to a computer-readable medium having a program element stored thereon, which can be read and / or loaded and executed by a computer so as to perform the steps of the method for interactive editing of volumetric image data as described above when the program element is executed by the computer.

[0105] An advantage of implementing the present invention by means of a computer program product and / or a computer readable medium is that already existing workstations (eg post-processing servers) or clients can be easily adapted by a software update to work as proposed by the present invention.

[0106] The computer used to perform the interactive editing method as described above, in particular to perform the above-mentioned related algorithms, can be a personal computer or a workstation in a computer network, and can include a processing unit, a system memory, and a system bus that couples various system components including the system memory to the processing unit. The computer can be implemented in a data processing system or as a data processing system. The data processing system can therefore include a computing device or a cluster thereof. The data processing system includes a processing unit and a storage component that communicate data with each other. The processing unit can be composed of or include a CPU and / or a GPU, and includes several modules configured to perform the steps of the method of the present invention.

[0107] The computer program may be stored on a suitable medium (such as an optical storage medium or a solid-state medium) provided with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. The computer program may be or may include an executable program for execution on a processor or processing unit. The computer program may be provided in a distributed manner and may therefore be distributed between several computing entities such as processors. For example, the computer program may be distributed (stored) on different computing entities, such as a client set (working in a distributed medical system with multiple medical treatment devices and servers), so that a part of the computer program (e.g., the steps of receiving and displaying the received volume image data) resides on a first computing entity, and another part (e.g., the rest of the method steps, especially providing, executing, calculating and displaying the visualization of the calculation) resides on a second entity. The computer program may include program modules - program modules may include routines, programs, objects, components, data structures, etc. - to instruct the computer to perform (one or more) tasks through data processing, especially data processing of abstract data types.

[0108] The display may be provided as a monitor. The display may be connected to the renderer by wire or wirelessly with the aid of a display interface. It may be a touch display. The display may be directly connected to the processor (e.g., medical scanners, VR / AR headsets).

[0109] The present invention makes use of a set of algorithms. An algorithm is in particular a set, e.g. a sequence, of instructions for processing input information to obtain output information. The instructions of the algorithm can be implemented in a computer and executed by, e.g., a processor of a data processing system according to the present invention. In particular, the input information of the algorithm is encoded in input data (here: a data set with volumetric image data), which can be accessed by a processor executing the algorithm. In particular, the processor processes the input data according to the instructions of the algorithm to generate output information, which is typically encoded in output data (here: a visualization of the calculation to be displayed on a display). In particular, according to the present invention, the algorithm processes data when it comprises instructions, which, when executed by a processor, cause the processor to process these data.

[0110] The order according to which the steps of the method of the present invention are described in this specification does not necessarily reflect the temporal order according to which the steps are performed. In particular, the steps of displaying the received volumetric image data and calculating the signed distance field data structure may be performed in parallel or in another order. Furthermore, the steps of displaying the received volumetric image data and displaying the editing operation may preferably be performed online and in parallel, which allows highly interactive editing.

[0111] It should be understood that the preferred embodiments of the present invention may also be any combination of other aspects of the present invention or the above embodiments with the first aspect of the present invention.

[0112] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] Figure 1 is a schematic overview diagram of a renderer for interactive editing according to a preferred embodiment of the present invention;

[0114] Figure 2 is another schematic overview diagram of a renderer according to another preferred embodiment;

[0115] Figure 3 is yet another schematic overview diagram of a renderer according to another preferred embodiment;

[0116] Figure 4 is a flow chart of a method for interactive editing showing different embodiments of the present invention;

[0117] Figure 5 A preferred embodiment of representing computational visualization;

[0118] Figure 6 Represents a preferred embodiment for performing an editing operation;

[0119] Figure 7 Another preferred embodiment of representing computational visualization;

[0120] Figure 8 presents a preferred embodiment for displaying received volumetric image data;

[0121] Fig. 9

[0046] Yet another preferred embodiment for computational visualization is shown.

[0122] Fig.10 is an example image of the 3D image dataset, and

[0123] Fig.11 Is the first edit operation with visualization Fig.10 The example shown in , and

[0124] Fig.12 is an erase operation with the effect of a first editing operation, in particular a visualization of the calculation by displaying Fig.10 and Fig.11 The example shown in , and

[0125] Fig.13 A second editing operation with visualization, in particular two different labeling operations for two different types of structures in volume image data Figures 10 to 12 The example shown in . DETAILED DESCRIPTION

[0126] The present invention relates to interactive editing, and in particular to interactive volume editing and rendering. In contrast to several existing systems, the entire segmentation pipeline uses signed distance fields (hereinafter also abbreviated as SDF), which allows for sub-voxel accuracy. This significantly improves the display / rendering quality of the performed editing operations.

[0127] Firstly, annotation in volumetric image data is possible. In particular, the system provides interactive marking of findings, highlighting of structures, switching of structures, smooth boundary treatment at segmented surfaces, and the option of combining user-defined and algorithmic (ie automatic) annotations.

[0128] Second, advanced clipping is possible. In particular, the system provides for the use of complex analytical shapes with sharp clipping surfaces without artifacts (e.g., stair-step artifacts).

[0129] Third, volume editing is possible. In particular, the system provides removal of artifacts or substructures in the volume by performing editing operations. In addition, structural enhancement, isolation and color isolation of substructures are possible. Furthermore, SDFs generated from user interaction and algorithms can be combined.

[0130] exist Figures 10 to 13 In the example shown, the approach is demonstrated by using the proposed SDF-based volume editing system together with a movie renderer.

[0131] The examples explained below show a multi-SDF workflow, including, for example, undoing drawing or editing operations in some SDFs.

[0132] Typically, marking may include coloring. However, coloring is not permitted in the patent literature, so hatching is used. The color "red" may be represented in the figure by a left diagonal line or hatching (starting from the lower left to the upper right), and the color "green" may be represented by a right diagonal hatching.

[0133] Fig.10 Volumetric image data is shown, showing a scapula and a catheter by way of example. In the volumetric image data, a user can perform editing operations by applying different labels to different structures in the displayed image. The operations can generally be directed to or applied to anatomical structures (e.g., organs, bones, etc.) shown in the image or to artificial structures (e.g., catheters, implants, etc.) also shown in the image.

[0134] like Fig.11As shown, a first editing operation is performed, which is here a preliminary marking for performing a subsequent second editing operation, and the second editing operation is here an erasing operation for subsequently erasing or removing the shoulder blade and the catheter. The result of the first editing operation (marking) is shown in Fig.11 Shown in a cross pattern.

[0135] The result of the second editing operation (erasing) is represented in the calculated visualization. Fig.12 Here, it can be seen that the scapula and catheter are no longer present in the image and have been removed according to the user defined editing operation instructions.

[0136] like Fig.13 As shown, the second type of editing operation, namely marking or highlighting, is applied to visually distinguish between two different structures in the image, in particular to visually distinguish between an intact and a broken rib. Thus, a first marking operation (here: colored in green, which is indicated in the figure by hatching to the upper right) as an editing operation is applied to two intact ribs, shown in the displayed volumetric image data, located to the left of the rib structure, in Fig.13 In addition, as the second marking operation of the editing operation (here: colored in red, in Fig.13 ) is applied to the remaining fractured rib, as shown in the displayed volumetric image data, to the right of the rib structure. Fig.13 The visualization results of the calculations of these two different markings or highlights are shown. As can be seen, two ribs are intact. These two ribs are shown with a first marking (e.g., green); the remaining eight ribs are shown with a second marking (e.g., red), both of which are shown in a common single representation.

[0137] The system is used for interactive editing, which means that after receiving the editing operations, the calculated visualization of these editing operations is calculated and rendered on the fly. Therefore, the system is an online system with high performance during interactive editing. In particular, different volumetric editing operations, such as drawing, erasing, carving, complex clipping, are visualized with high quality. In general, high image quality is becoming more and more important throughout the industry, especially with the adoption of high-end display devices such as OLED panels in the medical field, and by providing photo-realistic editing techniques such as volumetric path tracing of medical data - where low-quality segmentation renderings may not meet the growing user expectations.

[0138] In particular, each voxel of the SDF encodes the signed distance to the nearest point of the segmentation surface, in contrast to a binary mask which only describes the inclusion of the voxel center in the segmented object. Especially during rendering, using an SDF allows for smooth surfaces to be reconstructed using fewer voxels than would be the case with a binary representation. The distance transform of the binary mask can be pre-computed, or some other smoothing algorithm applied in preprocessing and / or interactively, but the additional processing degrades performance during editing (in general). The reconstruction filter can then restore a smooth segmentation surface with significantly reduced voxel resolution, while the arithmetic operations allow for a wide range of editing possibilities.

[0139] The rendering or method for interactive editing as suggested herein targets both physically based volume rendering (e.g., a volume path tracing renderer) and interactive ray casting. Furthermore, although the segmentation workflow described is primarily for anatomical education with real-time volume rendering, portions of the method and / or system generalize well to surface-based models, fused volume + surface models, non-medical volumetric data (e.g., industrial CT scans for failure analysis in chip technology), and / or non-real-time systems, e.g., for producing pre-rendered videos.

[0140] In a preferred embodiment, the method and / or system is GPU-based with a graphics-based implementation of the proposed technology. Other embodiments may employ a GPU (graphics processing unit) computing pipeline (e.g., CUDA, OpenCL) or a non-GPU-based rendering pipeline (e.g., Intel OSPRay).

[0141] According to a preferred embodiment, the system and / or renderer comprises the following components:

[0142] -3D volume rendering engine that supports auxiliary 3D textures in addition to anatomical 3D textures;

[0143] - Real-time generation, management and update of multiple SDFs encoded in 3D textures;

[0144] - Runtime interaction with SDF, including user-based editing operations of SDF (e.g., by applying SDF brush strokes) and algorithmic editing operations of SDF (e.g., by means of segmentation algorithms);

[0145] - Direct and indirect visualization of SDF during volume rendering.

[0146] The basic volume editing primitive in the methods, systems, or renderers described herein is an SDF that defines both a volume editing mask and an editing brush. A brush can be a pure mathematical operation on a mask SDF, or in more complex cases, it can be to rasterize a shape into a separate SDF. Some examples include:

[0147] Various 3D or extruded 2D geometric shapes (e.g., sphere, cube, cylinder)

[0148] 3D spline extrusions (e.g. a circular extrusion with a 3D centerline with a spherical cap)

[0149] Complex shapes described by curves and dynamic SDFs adapt to the volume data being edited. For example, when editing a portion of an anatomical volume containing blood vessels, a vessel detection algorithm can be used to automatically select, size, and orient a cylindrical brush.

[0150] In particular, vessel detection and / or vessel segmentation, including accurate extraction of the entire vessel volume or the vessel median axis, is a critical task that introduces complex image processing problems and obstacles to the basic data processing. In this context, non-exhaustive, model-based, region-growing level set and machine learning methods can be cited. For vessel filtering, some methods rely on contrast and geometric properties of the vessels. Thus, for example, it can usually be assumed that observation of local curvature via Hessian matrix analysis can allow for the determination of the location and orientation of putative vascular structures.

[0151] Thus, in a preferred embodiment, centerline tracking and vessel diameter estimation are performed from the vessel lumen mask, and this information can then be used to define the brush size, e.g., calculating the distance from the edit position to the vessel centerline, and using the vessel diameter at that position to drive the heuristics for the brush size. Alternatively, the SDF of the vessel tree can be calculated to speed up the query. Typically, in another preferred embodiment, the pre-segmented structure is converted into an SDF, and this processing step and / or information is used to adapt the brush.

[0152] In some embodiments, the volume of effect of a given brush can be constrained to a particular iso-value of the brush SDF; alternatively, the distance to the zero level set can modify the brush's characteristics (e.g., a smooth falloff of brush hardness), or a global masking operation can be defined (e.g., a mask punch effect across the entire volume).

[0153] The basic editing workflow involves creating a paint mask SDF consisting of one or more brush strokes, each of which is defined by one or more brush inputs. Once a brush stroke has been applied to the paint mask, it can be committed to a permanent mask, or it can be discarded. It is also possible to perform the reverse operation on the paint mask, causing the brush stroke to remove a 3D shape from the paint SDF. This is accomplished by using Boolean operations (Jones, Baerentzen, & Sramek, 2006) between existing values ​​in the paint mask and the brush inputs. For example, given SDF values ​​d1 and d2, the resulting value of an SDF subtraction is defined as max(-d1, d2). An already committed SDF mask can also be edited in a similar manner, by copying the SDF values ​​from the mask into the paint SDF, performing the edit operation, and committing the result to the permanent SDF mask.

[0154] The following paragraphs describe an example implementation of a (draw) edit operation:

[0155] The inputs are two textures - the submitted SDF and the active draw SDF;

[0156] Initially, the submitted SDF is copied to the active drawing SDF;

[0157] During editing:

[0158] Updates the active mask based on the drawing mode and stroke mode;

[0159] By comparing the active texture with the submitted texture, the unsubmitted drawing area is rendered differently (e.g., in a different color);

[0160] When editing is complete, the active drawing SDF is copied directly into the submitted SDF;

[0161] Alternatively, the active drawing SDF is initially empty, the brush is applied, and a final step submits the new drawing using the SDF operations.

[0162] Depending on the system architecture, the draw edit operations may be implemented as OpenGL compute shaders, CUDA kernels, or various other methods (e.g., running purely on the CPU) that are performed on the portion of the voxel data covered by the brush stroke. Various optimization methods may be implemented, including hierarchical data structures for SDFs, adaptive resolution of active draw SDFs and / or submitted SDFs, and progressive updating of SDFs.

[0163] In a preferred embodiment, various compression schemes may be used, such as range data quantization or wavelet compression.

[0164] In an alternative preferred implementation, a fully automatic segmentation algorithm (e.g., organ segmentation) or a partially automatic algorithm (e.g., flood fill of regions or isovalue contours grown from user-placed seed points) can be used to create the mask SDF. A combination of SDF from an automatic algorithm and user-directed drawing or editing operations can also be used.

[0165] As a variation on the basic workflow, volume data classification can be used to restrict the paint area during a paint or edit operation to achieve selective paint effects. This can be used to accurately paint or remove certain types of structures while avoiding other nearby structures (e.g., painting individual ribs).

[0166] One embodiment of the system uses volumetric hit testing to define strokes in 2D anatomical data. The system generates one or more depth images during the rendering process (see patent application number US20180227568A1) and performs hit testing by querying the depth value at a specific pixel location.

[0167] In some scenarios, this problem can be solved by creating a copy of the depth texture each time it is updated during rendering, and using the copied texture for fetching. This allows efficient fetching operations to be performed without blocking the renderer running on a separate thread.

[0168] Alternative embodiments may generate one or more isosurfaces from the volume data and select 3D brush stroke positioning based on mesh picking.

[0169] Because the drawing mask can be configured for different visualization effects, such as color drawing and erasing, the depth buffer is continuously updated when visualizing with the eraser. This allows continuous erasing as the upper layers of the volume are erased. In an embodiment, once the stroke begins, the depth buffer copy is not updated, which allows the user to pick and erase only the part of the volume that was visible before the stroke began and protect the remaining volume data from unintentional editing. This is useful when erasing structures that are very close to other structures and overlapping structures.

[0170] Hierarchical operations using a draw mask allow a set of operations to be discarded and before and after versions of the mask to be compared before committing to a permanent mask. By using the active draw SDF as a draw clear operation using clear mode, operations can be undone after committing to a mask, which is a manual operation and may not produce the exact same mask as before the operation.

[0171] The system implements undo operations by tracking brush strokes and regenerating SDF masks from a subset of strokes. In some embodiments, the cost of regenerating the full SDF each time may be prohibitively high, and the process is accelerated by caching the SDF after a certain number of strokes. When performing an undo operation, the SDF mask checkpoint closest to the new stroke list (after removing some strokes) is loaded, and only the missing strokes are reapplied using SDF operations.

[0172] As a variation of the basic workflow, multiple individual masks representing various segmentation structures of a volumetric dataset can be combined into a single mask using Boolean operations for specific use cases, allowing for reduced memory footprint and performance impact of operating with multiple masks during runtime while maintaining the flexibility and simplicity of individual masks.

[0173] SDF-based rendering

[0174] SDF allows for a number of visual effects to be implemented as part of the volume rendering pipeline, which can be implemented as single selected options, or all at once, and their use affects various components of the main volume rendering algorithm:

[0175] 1. Ray generation - for example, different or adaptive sampling rates for painted textures or painted regions for anatomy; adaptive sampling rates during interactions; SDF-based erase operations may affect empty space skipping.

[0176] 2. Reconstruction Filters - Various types of linear or nonlinear interpolation filters can be used to achieve various smoothing properties for SDF reconstruction. For example, high-order B-spline interpolation for distance reconstruction can be applied during rendering.

[0177] 3. Volume Data Classification - Queries to the SDF are used to modify the optical properties derived from the anatomical voxel data, such as specifying different shading (painting), different opacity (erasing and carving), or other color pattern kernels can be applied to highlights. Alternatively, masked areas can be rendered using a different classification than that used to render the volume.

[0178] 4. Volumetric Data Shading - Modify different material and lighting properties based on SDF, including material glossiness, refraction, and locally adapted light direction.

[0179] Because the masks in the methods and systems according to the solution presented in this application are encoded as SDFs, when using high-order reconstruction filters (e.g., GPU-accelerated B-spline interpolation), accurate segmentation boundaries can be recovered with sub-voxel accuracy even for volumes with low voxel resolution and anisotropic voxel dimensions. In the case of high-resolution anatomical data, this feature can be exploited to achieve visually similar results when using lower resolution SDFs.

[0180] In an embodiment, the difference between multiple persistent masks is visualized at runtime using Boolean operations as a guide for user-created paint brush strokes. For example, an SDF is generated that contains the subtraction between the automatic body segmentation SDF and the current mask. This SDF is visualized as a semi-transparent paint layer whose color is different from the current paint color.

[0181] In an embodiment, certain SDF effects, such as erasing, may be applied to the volumetric image data as pre-processing or pre-application effects, and the resulting volume is used for direct volume rendering. Masking and / or other editing operations are baked into the volumetric data. In particular, drawing / shading and / or masking may be baked into the volumetric data, but with a significant impact on volume rendering quality and efficiency (rendering may be faster). Thus, rendering performance is improved (no more sampling of auxiliary volumes is required) at the expense of reduced visual fidelity (signed distance information is not available during rendering, e.g., boundary smoothing cannot be used). In some applications, the above effects may be applied based on a separate SDF during rendering, even without caching pre-processed volume editing (referring to the stage within the volume ray casting loop, and the data is not pre-sorted before being passed to the rendering algorithm).

[0182] The system supports visualizations defined by multiple submitted and temporary SDFs, where the effects of each SDF can be switched individually and all effects are composited at runtime. For example, weighted alpha blending can be used to combine the color and opacity of multiple overlapping drawing layers.

[0183] Figure 1 A renderer 100 for interactive editing of three-dimensional image data is shown. The image data may be acquired by a medical modality, such as a computed tomography device, a magnetic resonance imaging device, an ultrasound device, and / or other devices. A data set having volumetric image data is received via an image interface 110, which is configured to receive the volumetric image data. The renderer 100 further uses a memory 120 for storing a signed distance field data structure that has been created for the received data set. The memory 120 may be directly embedded in the renderer 100 (e.g., Figure 1The renderer 100 may be provided as a separate unit and connected via an interface. The renderer 100 is further connected to a display 130 for displaying the received volume image data. The user interface 140 is configured to perform an editing operation for editing at least a portion of the displayed volume image data. Fig.10 and 11 As described in the example of , the editing operation may involve highlighting or marking certain substructures of the volumetric image data, such as ribs. In particular, a first function may be applied to perform a first editing operation (green marking) on ​​a first portion of the volume (intact ribs), and another second function may be applied to perform a second editing operation (red marking) on ​​a second portion of the volume (broken ribs). In general, different functions f for performing editing operations may be provided. For example, a function may be provided for annotating data, another function may be applied to marking data with a first marking, another function may be applied to marking data with a second marking, etc., and further, a function may be provided for cropping data, etc.

[0184] The renderer 100 further comprises a processing unit 150 configured to execute an algorithm for computing a visualization of the editing operation. The algorithm may further instruct the second display to display the computed visualization, i.e., the editing operation embedded with the volumetric image data (e.g., Fig.11 , with hatching and dashed markings). In this example, the display 130 and the user interface 140 are distinct entities, which brings the advantage of flexibility.

[0185] like Figure 2 As shown, display 130 may be transferred to another computing entity and thus may be externalized, while user interface 140 remains within the computing entity, which also houses processing unit 150 and memory 120 for storing signed distance field data structures. Functionality regarding application editing operations may be provided on user interface 140.

[0186] This setup can be beneficial for educational systems where the teacher is located at a different computing entity than the students. For example, the teacher can use a computing system where the display 130 resides, and the students can use a computing system where the user interface 140 can reside. Another use case involves a lecture hall setup with a high quality renderer (used as an image generator) and a large display coupled to a tablet running a real-time drawing / annotation user interface and preview quality rendering.

[0187] There are also distributed annotation scenarios where multiple devices run annotation user interfaces against a shared renderer. The computed visualization is the output of the interactive editing and / or rendering algorithm. The output may be provided on the display 130 or the user interface 140. In another embodiment, the output may be provided on both the display 130 and the user interface 140. To implement an educational system, an additional communication channel may be provided between the teacher and the students for transmitting instructions for editing the three-dimensional volume. The system may also be deployed as a client-server system.

[0188] Reference Figure 3 Another arrangement is described. In this case, the display 130 and the user interface 140 are arranged in the same unit. Thus, the display 130 is configured as a graphical user interface for inputting and outputting data.

[0189] Figure 4 is a flow chart of an interactive editing method. After starting the method, in step S1, a data set with volume image data is received. In step S2, the received volume image data is displayed, in particular on the display 130. In step S3, based on the received data set, a signed distance field data structure is calculated and provided. In step S4, editing operations can be performed on the user interface 140 for editing at least a part of the volume image data. These editing operations are then converted into operations in the provided signed distance field data structure. In step S5, a visualization of the editing operations that have been performed is calculated. Figure 4 As indicated by dashed lines in FIG. 1 , steps S4 and S5 may be iterated repeatedly. In step S6 , the visualization of the calculation is displayed on the display 130 as a result.

[0190] The steps of displaying S2 and providing S3 can be changed in order, such as Figure 4 Thus, first a signed distance field data structure may be provided / calculated and then the received volume image data may be displayed in step S2. In a preferred embodiment, the steps of performing the editing operation in step S4, calculating S5 and displaying the result in step S6 are performed in parallel to provide a real-time interactive editing system.

[0191] like Figure 4 As depicted, the method can be used iteratively by using the output result with the computed visualization, encoding the performed editing operation again as input for performing a further or second editing operation in the computed visualization. Thereafter, the second result so generated can be displayed as a further computed visualization. This feature has the technical effect that modifications and corrections can be applied to the computed visualization of the editing operation. This helps to improve the quality of the displayed image data.

[0192] After step S6, the method may end.

[0193] Figure 5 An embodiment of the step of calculating a visualization of the editing operation in step S5 is shown. Preferably, a segmentation algorithm S51 can be used for this.

[0194] Figure 6 An embodiment related to how to perform the editing operation in step S4 is shown. Preferably, a brush-based algorithm S41 can be used. Corresponding to the other functions f implementing the editing operation, different functions can be configured, such as annotation, clipping, marking, as already mentioned above. For example, thin or thick brushes and / or brushes of different colors can be used.

[0195] Figure 7 A preferred embodiment of the step of calculating a visualization of the editing operation in step S5 is shown. Preferably, a rendering algorithm may be applied, in particular a ray casting algorithm S52.

[0196] Figure 8 A preferred embodiment of the step of displaying the received volumetric image data in step S2 is shown.Preferably, a first rendering algorithm S21 may be applied and used for displaying the received volumetric image data.

[0197] A second rendering algorithm may be used in step S53 to visualize the editing operation, such as Fig. 9 As depicted in .

[0198] Any reference signs in the claims should not be construed as limiting the scope.

[0199] Where not explicitly described, the individual embodiments described in relation to the figures or their various aspects and features may be combined or exchanged with one another without limiting or expanding the scope of the described invention, as long as such combination or exchange makes sense and is in the sense of the invention. The advantages described with respect to a specific embodiment of the invention or with respect to a specific figure are also, where applicable, advantages of the other embodiments of the invention.

Claims

1. A computer-implemented method for interactive editing of volumetric image data, comprising the following method steps: - receiving a data set having volumetric image data, the volumetric image data having been acquired from an image acquisition modality; - displaying the received volumetric image data; - providing a signed distance field data structure of the received data set by generating a second data structure based on the received volumetric image data, wherein the signed distance field is an auxiliary to the volumetric image data; wherein the signed distance field is a representation of distances to structures of segmented objects or anatomical data; - performing an editing operation on a user interface for editing at least a portion of the provided signed distance field data structure; - rendering a visualization of said editing operation; wherein a plurality of corresponding or different editing operations are performed on different portions of a volume to be collectively represented in a rendered visualization, wherein organ-specific editing operations are performed which require pre-processing of said data by means of a whole or partial body segmentation algorithm; - Displaying the rendered visualization on a display.

2. The method of claim 1, wherein the editing operation is performed by masking the volumetric image data based on a value of a signed distance field data structure during the editing operation, the signed distance field data structure being sampled using a reconstruction filter.

3. The method according to claim 1 or 2, wherein the editing operation is performed by using a brush-based algorithm.

4. The method of claim 1 or 2, wherein the method is used iteratively by using the rendered visualization, encoding the performed editing operations as volumetric image data for performing further editing operations in the rendered visualization, the further editing operations being iteratively displayed as further rendered visualizations.

5. The method of claim 1 or 2, wherein rendering the visualization of the edited volumetric image data uses a segmentation algorithm or an SDF-based brush or manually indicated edits.

6. The method of claim 1 or 2, wherein rendering the visualization of the edited volumetric image data is performed by directly transforming and embedding user-defined editing operations in three-dimensional space in a signed distance field data structure of the volumetric image data.

7. The method according to claim 1 or 2, wherein rendering the visualization of the edited volumetric image data is based on a ray casting rendering algorithm.

8. The method according to claim 1 or 2, wherein: For displaying the received volumetric image data, a first rendering algorithm is used and / or for rendering a visualization of the editing operation, a second rendering algorithm is used.

9. The method of claim 1 or 2, wherein rendering the visualization of the editing operation uses a set of voxel classifiers to classify different volume characteristics, the volume characteristics comprising different shading values, different opacity values ​​and / or different shading techniques.

10. The method of claim 1 or 2, wherein the editing operations are encoded and visualized as drawing masks in the displayed rendered visualization, wherein the drawing masks are individually configured.

11. The method according to claim 1 or 2, wherein an undo / redo function is provided for the editing operation.

12. The method of claim 8, wherein the second rendering algorithm is a level set algorithm for rendering a smooth segmentation boundary representation from a signed distance field volume. The method according to claim 10 , wherein the drawing mask is a semi-transparent drawing layer.

14. Renderer for interactive editing of volumetric image data, including: - an image interface configured for receiving a data set having volumetric image data, the volumetric image data having been acquired from an image acquisition modality; - a memory interface to a memory configured for storing a signed distance field data structure of the received data set, wherein the signed distance field data structure is provided by generating a second data structure based on the received volumetric image data, wherein the signed distance field is an auxiliary to the volumetric image data; wherein the signed distance field is a representation of distances to structures of segmented objects or anatomical data; - a user interface configured for performing editing operations for editing at least a portion of the stored signed distance field data structure, configured for rendering a visualization of said editing operations, and configured for performing a plurality of corresponding or different editing operations on different portions of a volume to be jointly represented in the rendered visualization, and configured for performing organ-specific editing operations, which require pre-processing of said data by means of a whole or part body segmentation algorithm; - a display interface to a display configured to display the editing operations performed; - a processing unit configured for rendering a visualization of the editing operation for instructing the display to show the rendered visualization.

15. A computer program product comprising a program element which, when loaded into a memory of a computer, causes the computer to perform the steps of the method for interactive editing of volumetric image data according to any of claims 1-13.

16. A computer-readable medium having stored thereon a program element, the program element being read and executed by a computer, so as to perform the steps of the method for interactive editing of volumetric image data according to any of claims 1 to 13 when the program element is executed by the computer.

Images