Stereoscopic rendering of medical image sets

Abstract

The present disclosure particularly relates to a computer-implemented method for stereoscopic rendering of a medical image set of a patient. The method includes S10 acquiring a set of medical images defining a voxel grid including voxels. Each voxel is associated with a respective value (hereinafter referred to as an intensity value). The method includes S20 segmenting a voxel grid defined by a set of medical images by associating a respective tag in a set of tags with each voxel, where each tag corresponds to a respective region of interest of a patient. The method comprises S30 modifying the value of the voxel grid by adding an offset value to each value to be modified, where the offset value depends on the respective tag associated with the voxel. This method provides an improved solution for stereoscopic rendering of a medical image set of a patient.

Description

Technical Field

[0001] The present disclosure relates to the field of computer programs and systems, and more particularly, to a method, system, and program for performing stereoscopic rendering of a set of medical images of a patient. Background Art

[0002] Several widely used medical imaging techniques, such as computed tomography (CT scan) or MRI (magnetic resonance imaging), can provide 3D acquisitions of anatomical structures. The output of these medical imaging examinations is typically a vertically stacked set of two-dimensional image slices, which can be interpreted as a volumetric representation of the patient's anatomical structure. Generating 3D visual renderings of these volumes has become crucial for an increasing number of applications, such as medical research, surgical planning, surgical guidance, or patient communication.

[0003] Generally speaking, there are two main ways to represent 3D object visualization: volumetric representation and surface representation.

[0004] The most common one is surface representation, which represents an object by a surface approximated by a mesh of geometric primitives (such as triangles or quadrilaterals). Since it is hollow, this representation is very sparse and only retains the boundaries of the object of interest. Rasterization and ray tracing algorithms are able to generate visualizations of these surfaces. For medical applications, in order to define the boundaries of the object, the object needs to be segmented and masked, and then unique colors or realistic textures are applied to the surfaces of each individual mesh to generate the final rendering. Alternatively, the surface can be defined as an isosurface of a user-defined intensity level. Surface representation is not suitable for representing anatomical structures without clear boundaries, nor for representing intensity variations within the volume of the object. For example, fine details such as thin blood vessels appear blurred in CT scans and cannot be accurately rendered using surface representation.

[0005] The second one is volumetric representation, which is particularly suitable for 3D medical images. It involves representing a 3D object by a voxel grid filled with density and color values. Converting the 3D medical image volume into a volumetric representation is performed by mapping the obtained intensity values to color values and opacity values according to a so-called transfer function. Then, a volume ray casting algorithm is used to generate the visualization. This visualization is called direct stereoscopic rendering or simplified stereoscopic rendering. In contrast to surface representation, stereoscopic rendering is able to display smooth intensity variations in body tissues.

[0006] One of the main limitations of known stereoscopic rendering solutions is that they do not allow representing anatomical structures sharing overlapping intensity value ranges with a unique appearance. In fact, the transfer function depends only on intensity values, so they are spatially blind: different anatomical structures with similar intensity values located at different positions will have the same appearance (color and opacity). In addition, due to occlusion problems, they also do not allow visualizing anatomical structures when located behind tissues with overlapping intensity values.

[0007] One solution is to combine stereoscopic rendering with a surface representation of selected objects of interest. In fact, the surface representation is spatially aware, so each object surface has its own texture or color. However, as mentioned above, the surface representation is not suitable for every anatomical structure. In addition, compared with the volume representation, it results in a loss of granularity.

[0008] In this context, there is still a need for an improved solution for stereoscopic rendering of a patient's medical image set. Summary of the Invention

[0009] Accordingly, there is provided a computer-implemented method for stereoscopic rendering of a patient's medical image set. The method includes obtaining the medical image set defining a voxel grid including voxels, each voxel being associated with a value respectively. The method includes segmenting the voxel grid defined by the medical image set by associating a corresponding label from a label set with each voxel, wherein each label corresponds to a respective region of interest of the patient. The method includes modifying the values of the voxel grid by adding an offset value to each value to be modified, wherein the offset value depends on the corresponding label associated with the voxel.

[0010] The method may include one or more of the following:

[0011] - The offset value added to each value is equal to the result of multiplying the corresponding label associated with the voxel by a predetermined factor;

[0012] - The modification includes calculating an offset voxel grid V based on the following formula m :

[0013] V m = V + tS,

[0014] wherein V is the voxel grid defined by the medical image set, t is the predetermined factor, and S is the segmented voxel grid;

[0015] - The method further includes transmitting the calculated offset voxel grid to a viewer;

[0016] - Each segment is associated with a corresponding transfer function, and for each voxel of the segment associated with the function, the corresponding transfer function outputs corresponding values of at least two sets of appearance parameters including color parameters and opacity parameters, and the at least two sets of appearance parameters optionally include at least one additional appearance parameter, such as a metallicity parameter and / or a roughness parameter;

[0017] - One or more transfer functions are predetermined;

[0018] - One or more transfer functions are user-defined;

[0019] - The method further includes:

[0020] o Combining corresponding transfer functions in a single piecewise transfer function;

[0021] o Transmitting the single piecewise transfer function to the viewer;

[0022] - The single piecewise transfer function has the form:

[0023]

[0024] where f m is the single piecewise transfer function, f i is the corresponding transfer function associated with segment i, v min and v max are the minimum and maximum values of the voxel grid, respectively, and t is a predetermined factor;

[0025] - The predetermined factor is greater than the magnitude of the values of the voxel grid;

[0026] - Selecting the region of interest of the patient from:

[0027] o One or more organs, such as the brain, heart, lungs, liver, pancreas, kidneys, skin, and / or intestines;

[0028] o One or more tissues, such as epithelial tissue, connective tissue (such as adipose tissue), muscle tissue, and / or nerve tissue;

[0029] o One or more tubes, such as blood vessels; and / or

[0030] o One or more pathological forms, such as tumors; and / or

[0031] - The medical image set is generated by a CT scanner or an MRI scanner.

[0032] There is also provided a computer program product including instructions for performing the method.

[0033] There is also provided a computer-readable storage medium having recorded thereon the computer program product.

[0034] There is also provided a system including a processor coupled to a memory, on which the computer program product is recorded.

[0035] The system may further include a viewer. The viewer may include a graphical user interface for displaying a stereoscopic rendering of the medical image set. The viewer may be a medical image viewer capable of performing direct stereoscopic rendering.

[0036] There is also provided an apparatus that includes a data storage medium on which a computer program product is recorded. The apparatus may form or serve as a non-transitory computer-readable medium, such as on SaaS (Software as a Service) or other servers, cloud-based platforms, etc. The apparatus may optionally include a processor coupled to the data storage medium. Thus, the apparatus may form a computer system, in whole or in part (e.g., the apparatus is a subsystem of the entire system). The system may also include a viewer. The viewer may include a graphical user interface for a single image set. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Now, non-limiting examples will be described with reference to the drawings, wherein:

[0038] - Figure 1 A flowchart illustrating an example of the method is shown.

[0039] - Figure 2 An example of an occlusion problem in stereoscopic rendering using a known solution is shown.

[0040] - Figure 3 An example of calculating an offset voxel grid is shown.

[0041] - Figure 4 Another flowchart illustrating an example of the method is shown.

[0042] - Figure 5 and Figure 6 An example of a single piecewise transfer function is shown.

[0043] - Figure 7 and Figure 8 An example of stereoscopic rendering using the method is shown.

[0044] - Figure 9 An example of the system is shown. DETAILED DESCRIPTION

[0045] Referring to Figure 1 the flowchart, a computer-implemented method for stereoscopic rendering of a medical image set of a patient is proposed. The method includes, in S10, obtaining a medical image set defining a voxel grid including voxels. Each voxel is respectively associated with a value (hereinafter also referred to as "intensity value"). The method includes, in S20, segmenting the voxel grid defined by the medical image set by associating a corresponding label in a label set with each voxel, wherein each label corresponds to a corresponding region of interest of the patient. The method includes, in S30, modifying the values of the voxel grid by adding an offset value to each value to be modified, wherein the offset value depends on the corresponding label associated with the voxel.

[0046] This method provides an improved solution for stereoscopic rendering of a patient's medical image set.

[0047] Notably, the method allows for stereoscopic rendering of a patient's medical image set for different regions of interest, giving it an arbitrary unique appearance. This volumetric representation exhibits smooth intensity variations between body tissues, which is an improvement over surface representation solutions. In particular, the stereoscopic rendering of the medical image set allows for the display of different tissues and anatomical structures.

[0048] Furthermore, the method allows for the representation of different anatomical structures represented in the medical image set with a unique appearance. In fact, the increased offset values highlight the different regions of interest resulting from the S20 segmentation, thus highlighting the different anatomical structures represented by these segments. In particular, the method allows for the representation of anatomical structures with a unique appearance even if these anatomical structures share overlapping intensity value ranges. The S20 segmentation allows for the differentiation of different anatomical structures represented in the medical image set by associating labels with voxels. Based on these assigned labels, the method can then S30 differentiate the different anatomical structures in the voxel values by increasing the offset values, thus allowing for the representation of these different anatomical structures represented in the medical image set with a unique appearance.

[0049] The method is for stereoscopic rendering of a patient's medical image set. This means that the method can generate a new voxel grid (i.e., the offset voxel grid) whose values are modified relative to the initial voxel grid defined by the acquired medical image set. In particular, these values are modified to highlight the different anatomical structures defined by the provided segmentation mask. It is capable of visualizing all regions of interest shown in multiple medical image sets as a single group. The method can then include displaying this generated new voxel grid (e.g., using a 3D viewer, as described below). The display can be used by medical practitioners (such as doctors or nurses) to study the patient regions represented in the medical image set or to show it to the patient.

[0050] The method is implemented by a computer. This means that the steps (or substantially all steps) of the method are performed by at least one computer or any similar system. Thus, the steps of the method are performed by a computer, which may be fully automated or semi-automated. In an example, the triggering of at least some steps of the method can be performed through user-computer interaction. The required level of user-computer interaction may depend on the desired level of automation and be balanced with the needs of the user. In an example, this level can be user-defined and / or pre-defined.

[0051] A typical example of the computer implementation of a method is to execute the method using a system suitable for this purpose. The system may include a processor coupled to a memory and a Graphical User Interface (GUI), and a computer program product including instructions for executing the method is recorded on the memory. The memory may also store a database. The memory is any hardware suitable for such storage and may include several physically distinct parts (e.g., one for the program and possibly one for the database).

[0052] S10 Obtain a medical image set, which may include obtaining the medical image set. For example, the medical image set may be obtained by CT-scanning or any other imaging medical device (such as MRI scanning). In this case, S10 obtaining may include using CT-scanning or other imaging medical devices to obtain the medical image set. Alternatively, when the method is executed, the medical image set may already have been obtained. In this case, S10 obtaining may include retrieving the already obtained medical image set. For example, the medical image set may have been recorded on the memory, and S10 obtaining may include retrieving the obtained medical image set from the memory.

[0053] The medical image set may cover (i.e., image) an area of the patient's body. Imaging an area of the patient's body using such a medical image set is well known. For example, it is well known that an examination of the scanner type may involve regularly taking medical images along the patient's body (e.g., a part), and each image represents a slice of the patient's body. These medical images are then combined and can be used to reconstruct layer by layer a 3D volume representing the patient's body (e.g., part). Thus, the medical image set may together represent a 3D volume including the patient's area. Each medical image may be a 2D image and may represent a corresponding slice along the 3D volume represented by the set. Thus, a 3D volume can be formed by assembling consecutive slices represented by the different medical image sets.

[0054] The medical image set covers an area of the patient. In other words, the area of the patient is represented in at least a part (e.g., all) of each medical image set. The area covered by the medical image set may be any area of the patient's body. For example, the area of the body may include part or all of the head, neck, trunk (chest, abdomen, and pelvis), one or both upper limbs, and / or one or both lower limbs.

[0055] The medical image set can define a voxel grid. The voxel grid can be longitudinally aligned with the patient body region covered by the medical image set. The voxel grid can include voxel layers stacked on top of each other along the longitudinal direction. Each voxel layer can include voxels of the grid located at the same position along the longitudinal direction. Each voxel layer can represent a corresponding slice of the 3D volume represented by the set. Each medical image set can be perpendicular to the longitudinal direction. Each voxel can have a value defined by the medical image set. In particular, each medical image can define the value of the voxels belonging to the voxel layer located at the same position along the longitudinal direction as the medical image. When the voxel layer and the medical image are superimposed, each voxel can have a value corresponding to the value in the part of the medical image included in the voxel.

[0056] The values within the voxel grid can be decimal numbers. The values within the voxel grid can be intensity values acquired during the acquisition of the medical image set. When acquired by CT scan, the medical image set can be generated by passing X-rays through the human body. The generated signals can be read and analyzed to reconstruct the dense volume of the body. In this case, the voxel values defined by the medical image set can be Hounsfield Unit (HU) values. HU is a relative quantitative measure of radiodensity used by radiologists when interpreting Computed Tomography (CT) images. The values within the voxel grid can include at least two different values. For voxels belonging to the background of the medical image set (i.e., not included in the patient region), these values may be lower, close to a predetermined value (e.g., - 1000, representing air).

[0057] The medical image set can cover regions of interest. The regions of interest can be included in the region of the patient body covered by the medical image set. Each region of interest can be a corresponding part of the region of the patient body. Each region of interest can be any type of region of the human body that can be the target of a class of medical images (i.e., whose appearance, content, and / or shape can be revealed by such medical images). For example, the regions of interest can include one or more (e.g., a part of) organs such as the brain, heart, lungs, liver, pancreas, kidneys, skin, and / or intestines. Alternatively or additionally, the regions of interest can include one or more tissues such as epithelial tissue, connective tissue (such as adipose tissue), muscle tissue, and / or nerve tissue. Alternatively or additionally, the regions of interest can include one or more ducts such as blood vessels. Alternatively or additionally, the regions of interest can include one or more pathological forms such as tumors.

[0058] The medical image set can cover at least one region of interest among all the possible regions of interest listed above. This means that the intensity values within the voxel grid defined by the medical image set can capture the content and / or shape of at least a portion of the at least one region of interest. For example, voxels belonging to a region of interest may have values that vary within a specific range (e.g., depending on the density at the voxel position within the region of interest). Voxels that do not belong to a region of interest can have a predetermined value, which is the value of the image background (e.g., corresponding to air).

[0059] The S20 segmentation of the medical image set can include computing a segmentation mask for the voxel grid defined by the medical image set. The segmentation mask is referred to as the "segmented voxel grid". The segmented voxel grid can have the same size as the voxel grid defined by the medical image set, but for each voxel, it can include a label indicating the region of interest to which the voxel belongs. The label is one of a set of labels, with each label corresponding to a respective region of interest of the patient. For example, the regions of interest can include n 对象 regions of interest (n 对象 being an integer), and the set of labels can include {1,..., n 对象}(one corresponding to each region of interest). Additionally, the set of labels can include another label corresponding to the background (e.g., the label "0"). Thus, each voxel (z, y, x) of the segmented voxel grid can be filled with an integer value S(z, y, x) from 0 to n objects, which can be the number of segmented regions of interest of any of the types of anatomical structures listed above (e.g., tumor, fat, or organ). S(z, y, x) = i ∈ {1,..., n 对象} can indicate that the voxel (z, y, x) belongs to the region of interest labeled i. S(z, y, x) = 0 can indicate that the voxel (z, y, x) belongs to the background.

[0060] The computation of the segmentation mask can be performed in any way. For example, the computation of the segmentation mask can include creating an empty voxel grid having the same dimensions as the voxel grid defined by the medical image set and filling each voxel of the empty voxel grid with a label according to the region of interest to which the voxel belongs. For each region of interest covered in the medical image set, filling the voxels with labels can include detecting the voxels that belong to the region of interest and filling the corresponding voxels (i.e., located at the same position) in the empty voxel grid with the label associated with the region of interest to which the respective voxel belongs.

[0061] Voxel detection belonging to the region of interest can be performed based on segmentation techniques. For example, voxels belonging to the region of interest can be detected according to threshold segmentation based on intensity values in the voxels. Alternatively, voxels belonging to the region of interest can be detected based on the spatial variation of intensity values in the voxels. A significant change (e.g., higher than a given threshold between two voxels) can mean a change in the region of interest. Then, filling can include filling the detected voxels belonging to the same region of interest with a label associated with the region of interest to which the voxels belong. For the remaining voxels (e.g., having values below the threshold), filling can include filling the voxels of the empty voxel grid corresponding to these remaining voxels with a label associated with the background (e.g., the label "0").

[0062] The detection of labeled voxels can be automatically performed using known algorithms, such as neural networks, e.g., the U-Net described in the paper "U-Net: Convolutional Networks for Biomedical Image Segmentation" published in "Medical Image Computing and Computer-Assisted Intervention" (MICCAI 2015, edited by N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi) in "Lecture Notes in Computer Science" (Cham: Springer International Publishing, 2015, pp. 234–241. doi:10.1007 / 978-3-319-24574-4_28, which is incorporated herein by reference) by O. Ronneberger, P. Fischer, and T. Brox. The U-Net has a classical automatic segmentation architecture. Alternatively, the detection of labeled voxels can be performed semi-automatically. For example, filling the voxels with labels can involve the user performing certain actions, such as determining the contour of the region of interest to be considered or validating the region of interest automatically detected by a known algorithm. For example, the determination of the contour of the region of interest can be performed by the user. The determination can include displaying the medical image set (e.g., displaying each medical image in sequence), and through user interaction, selecting the voxels belonging to different regions of interest visible on the display (e.g., by defining the contour of each region of interest on each medical image).

[0063] The S30 modification can include, for each voxel to be modified, adding an offset value depending on the corresponding label associated with the voxel to the value of the voxel. Only a part of the voxels can be modified. For example, the method can include modifying the values of the voxels belonging to the region of interest and can include not modifying the values of the voxels belonging to the background.

[0064] The offset value can depend on the region of interest in which the voxel is located. In the generated voxel grid, each region of interest can include only voxels having values generated within their respective (i.e., non-overlapping) ranges. For each voxel to be modified, the increased offset value is equal to the result of multiplying the corresponding label associated with the voxel by a predetermined factor. For all voxels, the predetermined factor can be the same. The predetermined factor can depend on the magnitude of the intensity values within the voxel grid defined by the medical image set. For example, the predetermined factor can be strictly higher than the magnitude of the values within the voxel grid defined by the medical image set. The predetermined factor can be defined manually. Alternatively, the factor can be calculated automatically, for example, as the difference between the maximum and minimum intensities in the voxel grid (e.g., before the modification step S30) plus any positive value (e.g., 1).

[0065] The S30 modification can be performed by sequentially considering all voxels one by one. For example, for each voxel, the S30 modification can include identifying the label associated with the voxel and determining the offset value to be added based on the identified label (e.g., by multiplying the identified label by the previous predetermined factor). Then, the S30 modification can include, for each voxel, assigning a new value to the voxel by adding the offset value determined for the voxel to the value of the voxel. When the voxel does not belong to the region of interest, the new value may be the same as the value the voxel already had. For example, for voxels belonging to the background, the label may be zero, which means that the result of multiplying the identified label by the predetermined factor is also zero, and the value of each of these voxels remains unchanged.

[0066] Alternatively, the S30 modification can be performed by considering all voxels in the same mathematical operation. In this case, the S30 modification can include calculating the offset voxel grid V m :

[0067] V m = V + tS,

[0068] where V is the voxel grid defined by the medical image set, t is the predetermined factor, and S is the segmented voxel grid. The offset voxel grid can be the voxel grid with modified values.

[0069] In an example, the method may include, after the modification at S30, rendering a voxel grid with the modified values (i.e., the offset voxel grid). Rendering the offset voxel grid may include, by means of a 3D viewer, e.g., a known 3D viewer, preparing data for 3D visualization of the offset voxel grid. Known 3D viewers such as 3D Slicer (described in the paper by A. Fedorov et al., "3DSlicer as an Image Computing Platform for the Quantitative Imaging Network", Magn Reson Imaging, Vol. 30, No. 9, pp. 1323–1341, Nov. 2012, doi:10.1016 / j.mri.2012.05.001, which is incorporated herein by reference); ITK Snap (described in the paper by Paul A. Yushkevich, Joseph Piven, Heather Cody Hazlett, Rachel Gimpel Smith, Sean Ho, James C. Gee and Guido Gerig, "User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability", Neuroimage, 2006.07.01, 31(3):1116-28, which is incorporated herein by reference); or OsiriX (described on the website homepage "https: / / www.osirix-viewer.com / osirix / osirix-md / ", which is incorporated herein by reference). The viewer may be a medical image viewer capable of performing direct volume rendering. The prepared data may include the computed offset voxel grid. After rendering, the method may include sending (or transmitting) the prepared data to the 3D viewer (i.e., including the computed offset voxel grid). The 3D viewer may be used to display the offset voxel grid based on the sent data. In an example, the method may stop after sending the data to the 3D viewer. In other examples, the method may further include, after sending the data to the 3D viewer, displaying, by the 3D viewer, the resulting voxel grid based on the sent data. For example, the 3D viewer may include a graphical user interface, and the method may include displaying the resulting voxel grid on the graphical user interface.

[0070] The 3D viewer displays the offset voxel grid and can be performed by applying a stereoscopic rendering algorithm (e.g., a ray casting algorithm). During the execution of the stereoscopic rendering algorithm, the intensity values of the offset voxel grid can be converted into at least two appearance values according to a transfer function (i.e., by applying the transfer function to the values of the offset voxel grid), including color (e.g., RGB values) and opacity values (e.g., a proportion representing the voxel opacity in the final rendering). These appearance values can also include additional values, such as metallicity values and / or roughness values. The transfer function can be defined as a concatenation of multiple transfer functions that form a single piecewise transfer function. Each of these multiple transfer functions can be defined over a range of intensity values in the offset voxel grid associated with the region of interest or background (i.e., the region outside any region of interest) resulting from the S20 segmentation. Converting the intensity values of the offset grid according to the single piecewise transfer function enables the different regions of interest that may strictly contain anatomical structures to be clearly highlighted when displaying the voxel grid.

[0071] In an example, the application of the stereoscopic rendering algorithm can be performed by the 3D viewer. In this case, the data sent to the 3D viewer can include one or more transfer functions to be applied. Applying one or more transfer functions to the offset voxel grid can be performed by the 3D viewer. Each transfer function can take the intensity value of a given voxel as input and can output the corresponding value of each appearance parameter for that given set of voxels. Each transfer function can be applied to a corresponding set of voxels. For example, each segment (or region of interest) can be associated with a corresponding transfer function, and the corresponding transfer function can be applied to the voxels of that segment. These segments are the segments (i.e., regions of interest) obtained as a result of the S20 segmentation. Assigning values can include, for each segment, applying the corresponding transfer function associated with that segment to the voxels belonging to that segment. It can represent the different anatomical structures represented in the medical image set with a unique appearance. In fact, each segment includes voxels representing the same anatomical structure, and the method can apply the corresponding function to each segment of the voxels representing the same anatomical structure, thereby rendering each anatomical structure with a unique appearance.

[0072] In an example, one or more (e.g., all) of the multiple transfer functions can be user-defined. In this case, the method can include determining one or more user-defined transfer functions by a user (e.g., a medical practitioner or an operator). Determining each user-defined transfer function can include determining the graph of the transfer function. For example, the transfer function can be defined as a line graph, in which case determining the graph of the function can include adding, deleting, and / or moving the breakpoints of the line graph through a digital user interface.

[0073] In an example, one or more (e.g., all) of the plurality of transfer functions may be predetermined. In this case, one or more predetermined transfer functions may have been determined before the method is executed. For example, the shape of the function and / or its characteristic points may have been recorded in a memory, and the method may include retrieving the function shape and / or its characteristic points of each predetermined transfer function from the memory.

[0074] In an example, before sending data, the method may combine the respective transfer functions into a single piecewise transfer function. Each part of the single transfer function may correspond to a respective region of interest and may have its own color and intensity scale. For example, the form of the single piecewise transfer function may be:

[0075]

[0076] where, f m is the single piecewise transfer function, f i is the corresponding transfer function associated with segment i, v min and v max are the minimum and maximum values of the voxel grid respectively, and t is a predetermined factor. The single transfer function makes the method compatible with known general direct volume rendering tools.

[0077] Each corresponding transfer function f i may take an intensity value as input and may output a corresponding value of each appearance parameter of the set based on the input intensity value. For example, the form of each corresponding transfer function f i may be:

[0078]

[0079] where, c is the RGB color, σ is the opacity value, and p includes one or more potential additional appearance parameter values (e.g., including a metallicity parameter, a specular parameter, and / or a roughness parameter, e.g., as at https: / / en.wikipedia.org / wiki / Specular_highlight). v max and v min may be the maximum and minimum intensity values within the voxel grid respectively.

[0080] Then, the method may include transmitting a single-piece transfer function to a viewer. For example, the single-piece transfer function may be included in the data sent to a 3D viewer. The data can be sent to the 3D viewer in any way. For example, the method may be executed on a computer on which the 3D viewer is installed. This connection can be used to send the data. Then, the viewer can be used to display the calculated offset voxel grid based on the transmitted single-piece transfer function (e.g., using a known volume ray casting algorithm).

[0081] Volume rendering refers to the process of generating a 2D image corresponding to a 3D scene view when the underlying geometry of the scene is densely represented by a 3D matrix (i.e., a voxel grid), and the values of the 3D matrix describe the local appearance properties of the volume. Common volume rendering algorithms include ray casting or maximum intensity projection algorithms. Volume rendering is different from surface rendering, which refers to the process of generating a 2D image corresponding to a 3D scene view when the underlying geometry of the scene is sparsely represented by the surfaces of the objects it contains. These surfaces are usually decomposed into tilable primitives (triangles, quadrilaterals) to form a so-called mesh. Common surface rendering algorithms include rasterization or ray tracing algorithms. In contrast to surface rendering, volume rendering can take into account the opacity changes of scene elements in the depth direction. By performing volume rendering, the method thus improves the rendering of the patient area.

[0082] In an example, the method may include preparing for surgery on a patient using a voxel grid with modified values (i.e., an offset voxel grid). For example, the method may include, during the surgical preparation process, which may include after performing the method, preparing for surgery on the patient based on the offset voxel grid. Preparing for surgery may include sending the offset voxel grid to a 3D viewer to display the offset voxel grid (e.g., as described above), and determining the surgical operations to be performed based on the displayed offset voxel grid (e.g., identifying the areas to be treated, determining the operations to be performed on these areas and / or determining the tool paths to be used to perform these operations). Alternatively or additionally, the method may include communicating with the patient using the offset voxel grid. For example, the method may include sending the offset voxel grid to a 3D viewer to display the offset voxel grid to the patient (e.g., as described above), so that medical practitioners (e.g., doctors or nurses) can explain to the patient what is visible in the displayed offset voxel grid. By being able to display the offset voxel grid to the patient, the method also improves surgical preparation and / or patient communication. In fact, the method allows different anatomical structures represented in medical images to be represented in a unique appearance, which is particularly beneficial for surgical preparation and / or communication with the patient.

[0083] Referring to Figures 2 to 9 , implementation examples of the method will now be described.

[0084] In known volume rendering solutions, the intensity values of medical images (e.g., Hounsfield units of a CT scan) are converted into color and opacity values according to a transfer function (also called a color map or lookup table). Depending on the selected transfer function, direct volume rendering can display different tissues and anatomical structures. However, only a single transfer function can be used for the entire volume at a time. In fact, transfer functions only depend on intensity values, so they are spatially blind. In other words, transfer functions do not depend on the position of the voxels. Different anatomical structures located at different positions and having similar intensities will have the same appearance (color and opacity).

[0085] Therefore, due to occlusion problems, different anatomical structures cannot be visualized in volume rendering (e.g., in the volume rendering of a CT scan, visceral fat is usually hidden by the skin). More generally, it is not possible to render anatomical structures such as abdominal organs or different muscles with specific unique colors because they have overlapping intensity ranges. For example, in Figure 2 it is difficult to distinguish the liver 101 and the kidney 103 because they have similar intensity values.

[0086] This method solves these occlusion problems by making the volume rendering spatially aware. In particular, this method requires little coding and can be easily embedded into existing modern volume rendering R & D and production systems.

[0087] Now an explanation of the terms used is provided.

[0088] A CT scan (Computed Tomography) is a specific type of medical image generated by passing X-rays through the human body. The signals are read and analyzed to reconstruct a dense volume of the body.

[0089] The Hounsfield unit (HU) is a relative quantitative measure of radiodensity used by radiologists when interpreting Computed Tomography (CT) images (see, for example, https: / / www.ncbi.nlm.nih.gov / books / NBK547721 / ). Figure 3 An example of the Hounsfield scale for a CT scan is shown. It gives an example of the range corresponding to each tissue (bone: 400 - 1000 HU, soft tissue: 40 - 80 HU, …).

[0090] A medical image set can be a set of CT scan images acquired simultaneously in a single acquisition. They can define a voxel grid representing the HU values of the patient's anatomical structure.

[0091] The Region of Interest (ROI) is the anatomical structure that the method aims to highlight. For example, the region of interest can be an anatomical structure that includes a contrast agent product.

[0092] A transfer function (also known as a Look-Up Table (LUT) or color map) is a function used to map intensity values of a medical image volume to color and opacity values for volume rendering.

[0093] Direct volume rendering (or volume rendering) refers to the process of generating a 2D image from 3D volume data. The direct volume rendering method is a method that requires converting the source volume into a 3D mesh of color and opacity values based on a transfer function.

[0094] Ray casting is the baseline direct volume rendering algorithm. It involves casting viewing rays from an observation position into the volume. Color and opacity values are sampled along these rays. The color of each pixel in the resulting 2D image is a blend of the colors of the sampled points on its associated viewing ray weighted by the opacity values.

[0095] A spatially blind function is a function that does not depend on spatial information such as position.

[0096] A spatially aware function is a function that depends on spatial information, contrary to a spatially blind function.

[0097] Figure 3 An example of calculating an offset voxel grid is shown.

[0098] In the figure, a medical image set 200 is input for the method (step S10). The medical image set defines a volume grid V of size (D, H, W), which stores the intensity data of 3D medical image acquisitions. D is the number of acquired slices (i.e., the number of medical images in the set), and H and W are the height and width of each slice, respectively. The values within V (referred to as V values) are between v min and v max . The magnitude of the V value is represented as v amp , where, v amp = v max - v min .

[0099] The figure shows a segmented voxel grid 210, which is a volume grid S of size (D, H, W) representing a manually or automatically created segmentation mask V. S is filled with integer values from 0 to n 对象 , i.e., the number of segmented objects of interest that can be any type of anatomical structure (such as tumors, fat, organs, etc.). S(z, y, x) = i ∈ {1,..., n 对象} indicates that the voxel V(z, y, x) belongs to the object labeled i. S(z, y, x) = 0 indicates that the voxel V(z, y, x) belongs to the background.

[0100] In this example, the medical image set includes two regions of interest. In the segmented voxel grid 210, the voxels 211 belonging to the first region of interest are associated with the label "1" (the kidney in this example). The voxels 212 belonging to the second region of interest are associated with the label "2" (the liver in this example). The voxels 210 belonging to the background are associated with the label "0".

[0101] The figure also shows the resulting offset voxel grid 220. The method calculates the offset voxel grid 220 by adding the result of multiplying the segmented voxel grid 210 by a predetermined factor t to the initial voxel grid 200. As shown, the figure shows that the resulting offset voxel grid 220 has intensity values in a non-overlapping range in the first and second regions of interest (221 and 222). It can highlight these two different regions of interest in the final rendering.

[0102] Figure 4 Another flowchart showing an example of the method is presented. In this example, the method includes the following steps.

[0103] In the first step, the method includes determining an offset t > v amp (e.g., for a standard CT scan t = 5000).

[0104] In the second step (at runtime), the method includes calculating an offset volume V m = V + tS (as shown in the reference Figure 3 ). The calculation of the offset volume includes S10 obtaining the medical image set, S20 segmenting the voxel grid, and S30 modifying the voxel grid values to obtain the offset volume V m .

[0105] In the third step, the method includes designing or having the user select a preset transfer function f specific to each object i (including the background) i , e.g.:

[0106]

[0107] where c is the RGB color, σ is the opacity value, and p are potential other appearance attribute parameters (e.g., specularity or roughness). This step can be performed interactively, e.g., using a medical image viewer (such as Slicer described at https: / / www.slicer.org).

[0108] In the fourth step, the method includes combining the segmented object transfer functions into a single piecewise transfer function f m that is partially defined and adapted to the modified V m , e.g.:

[0109]

[0110] Thus, if the voxel (z, y, x) belongs to object i, then f m (V m (z, y, x)) = f i (V(z, y, x)). Figure 5 And Figure 6 shows an example of a piecewise transfer function.

[0111] In the fifth step, the method includes transmitting the modified volume V m and the transfer function f m to a general-purpose volume rendering tool (such as the Open3D library described at http: / / www.open3d.org / docs / release / #, or the VTK library described at https: / / vtk.org / ) to perform a volume ray casting algorithm to obtain the desired result visualization (see Figure 7 and Figure 8 ).

[0112] Figure 5 And Figure 6 shows an example of a single piecewise transfer function f m . Figure 5 Shows in graphical form the RGB values output by the single piecewise transfer function f m as a function of the modified intensity value. In this example, the method calculates a single piecewise transfer function f m that outputs three RGB color channels: red, green, and blue. The first peak 301 associated with the color "red" in the figure corresponds to the region of interest "kidney". The second peak 302 associated with the color "green" in the figure corresponds to the region of interest "liver". The third peak 303 associated with the color "blue" in the figure corresponds to the region of interest "lung". Figure 6 Shows in graphical form the opacity values output by the single piecewise transfer function f m as a function of the modified intensity value. The first peak corresponds to the non-segmented part of the scan, and each subsequent peak corresponds to a segmented region (such as Figure 5 the kidney, liver, and lung shown). The single piecewise transfer function f m produced can be decomposed into several (e.g., four in this example) transfer functions, each within its own intensity value range (e.g., separated by an offset of 5000 HU in this example) and corresponding to the body regions divided by the segmentation mask (in this example, the non-segmented region, kidney, liver, and lung), which can highlight different regions of interest in the final rendering.

[0113] Figure 7 AndFigure 8 An example of volume rendering using this method is shown. In particular, Figure 7 an example of a CT scan obtained by this method and using Figure 5 and Figure 6 the transfer function shown is the final volume rendering. This method can assign different colors to the liver 402, kidneys 401, and lungs 403 according to the segmentation masks (green, red, blue). In particular, this method can clearly render the lungs 403 because their density values are close to air, which is particularly difficult to render in known volume rendering solutions. Figure 8 The final volume rendering of an example CT scan of a segmentation mask with visceral fat 404 obtained by this method is shown.

[0114] Figure 9 An example of the system is shown, where the system is a client computer system, such as the user's workstation.

[0115] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, and a random-access memory (RAM) 1070 also connected to the bus. The client computer also provides a graphical processing unit (GPU) 1110, which is associated with a video random-access memory 1100 connected to the bus. The video RAM 1100 is also known as a frame buffer in the art. A mass storage device controller 1020 manages access to a mass storage device such as a hard disk drive 1030. Mass storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, by way of example, including semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks. Any of the above functions can be supplemented or integrated by a specially designed ASIC (application-specific integrated circuit). A network adapter 1050 manages access to a network 1060. The client computer may also include a haptic device 1090, such as a cursor control device, a keyboard, etc. A cursor control device is used in the client computer to allow the user to selectively position the cursor at any desired location on a display 1080. In addition, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes a plurality of signal generating devices for inputting control signals to the system. Generally, the cursor control device can be a mouse, and the buttons of the mouse are used to generate signals. Alternatively or additionally, the client computer system may include a touch pad and / or a touch screen.

[0116] A computer program may include instructions executable by a computer, which instructions include means for causing the above system to perform the method. The program may be recorded on any data storage medium, including the memory of the system. For example, the program may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations thereof. The program may be implemented as an apparatus, such as a product tangibly embodied in a machine-readable storage device for execution by a programmable processor. Method steps may be performed by a programmable processor that performs an instruction program to perform the functions of the method by operating on input data and generating output. Thus, the processor may be programmable and coupled to receive data and instructions from, and to send data and instructions to, a data storage system, at least one input device, and at least one output device. An application program may be implemented in a high-level procedural or object-oriented programming language, or, if desired, in assembly or machine language. In any case, the language may be a compiled or interpreted language. The program may be a complete installation program or an update program. The application of the program on the system in any case results in the execution of the instructions of the method. Alternatively, the computer program may be stored and executed on a server in a cloud computing environment that communicates with one or more clients via a network. In such a case, the processing unit executes the instructions contained in the program, thereby causing the method to be performed on the cloud computing environment.

Claims

1. A computer-implemented method for stereoscopic rendering of a set of medical images of a patient, characterized in that, The method includes: - S10 obtaining the medical image set defining a voxel grid including voxels, each voxel being associated with a value respectively; - S20 segmenting the voxel grid defined by the medical image set by associating corresponding labels in a label set with each voxel, wherein each label corresponds to a respective region of interest of the patient; - S30 modifying the values of the voxel grid by adding an offset value to each value to be modified, wherein the offset value depends on the corresponding label associated with the voxel.

2. The method according to claim 1, wherein The offset value added to each value is equal to the result of multiplying the corresponding label associated with the voxel by a predetermined factor.

3. The method according to claim 1 or 2, characterized in that, The modification includes calculating an offset voxel grid V based on the following formula m :[[]]END]] V m = V + tS, Wherein, V is the voxel grid defined by the medical image set, t is the predetermined factor, and S is the segmented voxel grid; And wherein the method further includes transmitting the calculated offset voxel grid to a viewer.

4. The method according to any one of claims 1-3, characterized in that Each segment is associated with a corresponding transfer function, and for each voxel of the segment associated with the function, the corresponding transfer function outputs corresponding values of at least two sets of appearance parameters including color parameters and opacity parameters, and the at least two sets of appearance parameters optionally include at least one additional appearance parameter, such as a metallicity parameter and / or a roughness parameter.

5. The method according to claim 4, wherein One or more transfer functions are predetermined.

6. The method according to claim 4 or 5, characterized in that, One or more transfer functions are user-defined.

7. The method according to any one of claims 4-6, characterized in that The method further includes: - Combining the corresponding transfer functions in a single piecewise transfer function; - Transmitting the single piecewise transfer function to the viewer.

8. The method according to claim 7, wherein The form of the single piecewise transfer function is: where f m is the single-piecewise transfer function, f i is the corresponding transfer function associated with segment i, v min and v max are the minimum and maximum values of the voxel grid, respectively, and t is a predetermined factor.

9. The method according to any one of claims 2-8, characterized in that, The predetermined factor is greater than the magnitude of the values of the voxel grid.

10. The method according to any one of claims 1-9, characterized in that, Select the region of interest of the patient from the following: One or more organs, such as the brain, heart, lungs, liver, pancreas, kidneys, skin, and / or intestines; One or more tissues, such as epithelial tissue, connective tissue (such as adipose tissue), muscle tissue, and / or nerve tissue; One or more tubes, such as blood vessels; and / or One or more pathological forms, such as tumors.

11. The method according to any one of claims 1-10, characterized in that, The medical image set is generated by a CT scanner or an MRI scanner.

12. A computer program product, characterized in that, Includes instructions for performing the method according to any one of claims 1-11.

13. A computer-readable storage medium, characterized in that, A computer program product according to claim 12 is recorded thereon.

14. A system, characterized in that, Includes a processor coupled to a memory, and a computer program product according to claim 12 is recorded on the memory.

15. The system according to claim 14, wherein, Further includes: A viewer, the viewer includes a graphical user interface for displaying a stereoscopic rendering of the medical image set.

Images