Volume rendering of set of medical images

Abstract

SOLUTION: The present disclosure relates particularly to a computer-implemented method for volume-rendering a set of medical images of a patient. The method comprises obtaining (S10) the set of medical images defining a voxel grid comprising voxels. The voxels are each associated with a value (hereinafter also referred to as "intensity value"). The method comprises segmenting (S20) the voxel grid defined by the set of medical images by associating with each voxel a respective label among a set of labels each corresponding to a respective region of interest of the patient. The method comprises modifying (S30) values of the voxel grid by adding to each value to be modified an offset value which depends on the respective label associated with the voxel. Such a method provides an improved solution for volume-rendering a set of medical images of a patient.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present disclosure relates to the field of computer programs and systems, and more particularly to methods, systems, and programs for volume rendering of a medical image set of a patient. [Background technology]

[0002] Several widely used medical imaging techniques, such as computed tomography scans (CT scans) or magnetic resonance imaging (MRI), provide 3D acquisition of anatomical structures. The output of these medical imaging tests is typically a vertical stack of 2D image slices that can be interpreted as a volumetric representation of the patient's anatomy. Generating 3D visual renderings of these volumes is becoming important for an increasing number of applications, such as medical research, surgical planning, surgical guides, or patient communication.

[0003] In general, there are two main ways to represent 3D objects for visualization: volumetric and surface representations.

[0004] The most common is surface representation, which consists of representing an object with its surface approximated by a mesh of geometric primitives such as triangles or quadrilaterals. This representation is hollow and therefore very sparse, preserving only the boundaries of the object of interest. Rasterization and ray tracing algorithms can be used to generate visualizations of these surfaces. For medical applications, a segmentation mask of the object to be represented is required to define its boundaries, and then a distinctive color or realistic texture is applied to the surface of each individual mesh to generate the final rendering. Alternatively, surfaces can be defined as isosurfaces with user-defined intensity levels. Surface representations are not suitable for representing anatomical structures with poorly defined boundaries or for representing intensity variations within an object's volume. For example, small details such as thin blood vessels appear blurred in CT scans, but they cannot be accurately rendered using surface representations.

[0005] Another method, volumetric representation, is particularly suited to 3D medical images. It involves representing 3D objects using a voxel grid filled with density and color values. The conversion of a 3D medical image volume into a volumetric representation is performed by mapping the intensity values of the capture to color and opacity values according to a so-called transfer function. A volumetric ray-casting algorithm is then used to generate the visualization. This type of visualization is called direct volume rendering, or simply volume rendering. Unlike surface representations, volumetric rendering can reveal smooth intensity variations throughout the body's tissues.

[0006] One of the main limitations of known solutions for volume rendering is their inability to represent anatomical structures that share overlapping ranges of intensity values with a distinctive appearance. Indeed, because transfer functions depend only on intensity values, they are spatially blind: different anatomical structures at different locations with similar intensity values will have the same appearance (color and opacity). Furthermore, they cannot visualize anatomical structures when they are surrounded by tissues with overlapping intensity values due to occlusion issues.

[0007] One solution is to combine volume rendering with surface representation for selected objects of interest. Indeed, surface representation is spatially aware, so each object surface can have its own texture or color. However, as mentioned above, surface representation is not suitable for all anatomical structures. Furthermore, it causes a loss of granularity compared to volume representation. Summary of the Invention [Problem to be solved by the invention]

[0008] In this context, there remains a need for improved solutions for volume rendering of a patient's medical image set. [Means for solving the problem]

[0009] Accordingly, there is provided a computer-implemented method for volume rendering a set of medical images of a patient. The method comprises acquiring a set of medical images defining a voxel grid comprising voxels, each voxel associated with a value. The method comprises segmenting the voxel grid defined by the set of medical images by associating with each voxel a respective label from a set of labels, each corresponding to a respective region of interest of the patient. The method comprises modifying the values of the voxel grid by adding to each value to be modified an offset value that is dependent on the respective label associated with the voxel.

[0010] The method may comprise one or more of the following: The offset value added to each value is equal to the result of multiplying the respective label associated with the voxel by a given factor. - Correct the voxel grid offset based on the following formula: m Calculating V m =V+tS where V is the voxel grid defined by the medical image set, t is a predetermined coefficient, and S is the segmentation voxel grid. The method further comprises transmitting the calculated offset voxel grid to a viewer. - each segment is associated with a respective transfer function, each transfer function outputting, for each voxel of the segment with which the function is associated, a respective value for at least two sets of appearance parameters comprising a color parameter and an opacity parameter, the at least two sets of appearance parameters optionally comprising at least one additional appearance parameter such as a metallicity parameter and / or a surface roughness parameter. The one or more transfer functions are predetermined. One or more transfer functions are user-defined. The method further comprises: Combining each transfer function into a single piecewise transfer function, and Transmitting a single piecewise transfer function to the viewer Equipped with. - A single piecewise transfer function is of the form ∀i∈{0,…,n objects},f m (v)=f i (v-ti),v∈[v min +ti,v max +ti] where f m is a single piecewise transfer function, and f iis the respective transfer function associated with segment i, and v min and v max are the minimum and maximum values of the voxel grid, respectively, and t is a predetermined coefficient. The predetermined coefficient is greater than the range of variation of the values in the voxel grid. The patient's area of interest is selected from the following: one or more organs, such as the brain, heart, lungs, liver, pancreas, kidneys, skin, and / or intestines; one or more tissues, such as epithelial tissue, connective tissue (e.g., adipose tissue), muscle tissue, and / or nervous tissue; one or more vessels, such as blood vessels, and / or one or more pathological forms, such as tumors; and / or The medical image set was generated by a CT or MRI scanner.

[0011] Further provided is a computer program comprising instructions for carrying out the method.

[0012] Additionally, a computer readable storage medium having a computer program recorded thereon is provided.

[0013] Additionally provided is a system comprising a processor coupled to a memory, the memory having a computer program recorded thereon.

[0014] The system may further comprise a viewer. The viewer may comprise a graphical user interface configured to display a volume rendering of the medical image set. The viewer may be a medical image viewer capable of performing direct volume rendering.

[0015] Further provided is a device comprising a data storage medium having a computer program recorded thereon. The device may form or function as a non-transitory computer-readable medium, such as in a Software as a Service (SaaS) or other server or cloud-based platform. The device may alternatively comprise a processor coupled to the data storage medium. Thus, the device may form, in whole or in part, a computer system (e.g., the device is a subsystem of the overall system). The system may further comprise a viewer. The viewer may comprise a graphical user interface configured to display a single image set. [Brief explanation of the drawings]

[0016] Non-limiting examples will now be described with reference to the accompanying drawings.

[0017] [Figure 1] 1 shows a flowchart of an example method. [Figure 2] 1 shows an example of an occlusion problem in volume rendering using known solutions. [Figure 3] 10 shows an example of a calculated offset voxel grid. [Figure 4] 10 shows another flowchart of an example method. [Figure 5] An example of a single piecewise transfer function is shown below. [Figure 6] An example of a single piecewise transfer function is shown below. [Figure 7] An example of volume rendering using the method is shown. [Figure 8] An example of volume rendering using the method is shown. [Figure 9] An example of a system is shown. DETAILED DESCRIPTION OF THE INVENTION

[0018] With reference to the flowchart of Figure 1, a computer-implemented method for volume rendering a medical image set of a patient is proposed. The method comprises acquiring S10 a medical image set defining a voxel grid comprising voxels, each of which is associated with a value (hereinafter also referred to as an "intensity value"). The method comprises segmenting S20 the voxel grid defined by the medical image set by associating with each voxel a respective label from a set of labels, each corresponding to a respective region of interest of the patient. The method comprises modifying S30 the values of the voxel grid by adding to each value to be modified an offset value that depends on the respective label associated with the voxel.

[0019] Such a method provides an improved solution for volume rendering a medical image set of a patient.

[0020] In particular, the method allows for volume rendering of a set of medical images of a patient with arbitrary characteristic appearances for various regions of interest. Such volume representations reveal smooth intensity variations across body tissues, which is an improvement over surface representation solutions. In particular, volume rendering of a set of medical images can reveal various tissues and anatomical structures.

[0021] Furthermore, the method allows various anatomical structures represented in the medical image set to be represented with a distinctive appearance. Indeed, the added offset values highlight various regions of interest resulting from the segmentation S20, and therefore the various anatomical structures represented by these segments. In particular, the method allows anatomical structures to be represented with a distinctive appearance even when these anatomical structures share overlapping ranges of intensity values. The segmentation S20 allows different anatomical structures represented in the medical image set to be distinguished by associating labels with voxels. Based on these assigned labels, the method allows different anatomical structures to be distinguished in the voxel values by adding offset values S30, thereby allowing these various anatomical structures represented in the medical image set to be represented with a distinctive appearance.

[0022] The method is for volume rendering a set of medical images of a patient. This means that the method may generate a new voxel grid (i.e., the offset voxel grid described above) whose values are modified relative to the initial voxel grid defined by the acquired medical image set. In particular, the values are modified to highlight various anatomical structures defined by a provided segmentation mask. This allows all regions of interest revealed by multiple medical image sets to be visualized in a single set. The method may then comprise displaying this generated new voxel grid (e.g., using a 3D viewer described below). This display may be used by a medical professional (e.g., a doctor or nurse) to examine the patient's area represented in the medical image set or to show it to the patient.

[0023] The method is computer-implemented. This means that the steps (or substantially all steps) of the method are performed by at least one computer, or any similar system. Thus, the method steps are performed by a computer, possibly fully automatically or semi-automatically. In an example, triggering of at least some of the method steps may be performed by user / computer interaction. The level of user / computer interaction required may depend on the level of automation that is anticipated and commensurate with the need to implement the user's wishes. In an example, this level may be user-defined and / or pre-defined.

[0024] A typical example of a computer implementation of the method is executing the method on a system adapted for this purpose. The system may include a processor and a graphical user interface (GUI) coupled to a memory having recorded thereon a computer program comprising instructions for carrying out the method. The memory may also store a database. The memory is any hardware adapted for such storage, possibly comprising multiple physically distinct parts (e.g., one for the program, perhaps another for the database).

[0025] Acquiring S10 the medical image set may comprise acquiring the medical image set. For example, the medical image set may be acquired by a CT scan or any other imaging medical device (e.g., an MRI scan). In that case, acquiring S10 may comprise acquiring the medical image set using the CT scan or other imaging medical device. Alternatively, the medical image set may already be acquired at the time the method is performed. In that case, acquiring S10 may comprise retrieving the already acquired medical image set. For example, the medical image set may be stored in a memory, and acquiring S10 may comprise retrieving the acquired medical image set from the memory.

[0026] A set of medical images may cover (i.e., image) a region of a patient's body. The use of such a set of medical images to image a region of a patient's body is well known. For example, it is known that a scanner-type examination may include taking medical images periodically along (e.g., a portion of) a patient's body, with each medical image representing a slice of the patient's body. When assembled together, these medical images may then be used to reconstruct a 3D volume that represents (e.g., a portion of) the patient's body slice by slice. Thus, the medical images of the set may together represent a 3D volume that includes a region of the patient. Each medical image may be a 2D image and may represent a respective slice along this 3D volume represented by the set. Thus, a 3D volume may be formed by assembling consecutive slices represented by different medical images of the set.

[0027] The medical image sets cover a region of the patient. In other words, the patient's region is represented by at least some (e.g., all) of the medical images in each set. The region covered by the medical image sets can be any region of the patient's body. For example, the body region can include, partially or entirely, the head, neck, trunk (chest, abdomen, pelvis), one or both upper limbs, and / or one or both lower limbs.

[0028] The set of medical images may define a voxel grid. This voxel grid may be aligned with the longitudinal direction of the region of the patient's body covered by the set of medical images. The voxel grid may comprise layers of voxels overlaid on one another along this longitudinal direction. Each voxel layer may comprise voxels of the grid that are co-located along the longitudinal direction. Each voxel layer may represent a respective slice of the 3D volume represented by the set. Each medical image of the set may be perpendicular to this longitudinal direction. Each voxel may have a value defined by the set of medical images. In particular, each medical image may define values for voxels belonging to a voxel layer that is co-located with the medical image along the longitudinal direction. When the voxel layer and the medical images are overlaid, each voxel may have a value that corresponds to the value of the portion of the medical image that the voxel contains.

[0029] The values in the voxel grid may be decimal numbers. The values in the voxel grid may be intensity values acquired during the acquisition of a medical image set. When acquired by a CT scan, the medical images of the set may be generated by irradiating a human body with X-rays. The resulting signals may be read and analyzed to reconstruct a dense volume of the body. In that case, the values of voxels defined by the medical image set may be Hounsfield Unit (HU) values. HU is a relative quantitative measurement of radiation density used by radiologists to interpret computed tomography (CT) images. The values in the voxel grid may comprise at least two different values. For voxels that belong to the background of the medical image set (i.e., not included in the patient's region), the value may be a low value close to a predetermined value (e.g., −1000, which represents air).

[0030] The set of medical images may cover regions of interest. The regions of interest may be included in the region of the patient's body covered by the set of medical images. Each region of interest may be a respective portion of this region of the patient's body. Each region of interest may be any type of region of the human body that can be targeted by some type of medical imaging (i.e., whose appearance, content, and / or shape can be revealed by this medical imaging). For example, the regions of interest may include one or more organs (e.g., portions thereof), such as the brain, heart, lungs, liver, pancreas, kidneys, skin, and / or intestines. Alternatively, or in addition, the regions of interest may include one or more tissues, such as epithelial tissue, connective tissue (e.g., adipose tissue), muscle tissue, and / or nerve tissue. Alternatively, or in addition, the regions of interest may include one or more vessels, such as blood vessels. Alternatively, or in addition, the regions of interest may include one or more pathological forms, such as tumors.

[0031] The medical image set may cover at least one region of interest among all possible regions of interest listed above. This means that the intensity values within the voxel grid defined by the medical image set may capture the content and / or shape of at least a portion of this at least one region of interest. For example, voxels belonging to the region of interest may have values that vary within a certain range (e.g., depending on the density of voxel positions within the region of interest). Voxels not belonging to the region of interest may have a predetermined value that is the value of the background of the image (e.g., a value corresponding to air).

[0032] Segmenting the medical image set S20 may comprise calculating a segmentation mask of the voxel grid defined by the medical image set. The segmentation mask is referred to as a "segmented voxel grid." The segmented voxel grid may have the same dimensions as the voxel grid defined by the medical image set, but may comprise, for each voxel, a label indicating the region of interest to which the voxel belongs. The labels are from a set of labels each corresponding to a respective region of interest of the patient. For example, the region of interest may be n objectsRegions of interest (n objects may comprise a label set of {1,...,n objects} (one for each region of interest). Additionally, the label set may include another label corresponding to the background (e.g., label "0"). Thus, each voxel (z, y, x) in the segmentation voxel grid may be assigned a value between 0 and n objects , n, which is the number of segmentation regions of interest, which can be any type of anatomical structure (e.g., tumor, fat, or organ) mentioned above. S(z,y,x)=i∈{1,...,n objects} may indicate that voxel (z,y,x) belongs to the region of interest labeled i. S(z,y,x)=0 may indicate that voxel (z,y,x) belongs to the background.

[0033] The computation of the segmentation mask may be performed in any manner. For example, the computation of the segmentation mask may comprise creating an empty voxel grid having the same dimensions as the voxel grid defined by the medical image set, and filling each voxel of this empty voxel grid with a label according to which region of interest the voxel belongs to. Filling the voxels with labels may comprise, for each region of interest covered in the medical image set, detecting voxels that belong to the region of interest, and filling corresponding voxels (i.e., at the same location) in the empty voxel grid with labels associated with the region of interest to which they belong.

[0034] The detection of voxels belonging to the region of interest may be performed based on a segmentation technique. For example, the detection of voxels belonging to the region of interest may be based on threshold-based segmentation based on intensity values within the voxels. Alternatively, the detection of voxels belonging to the region of interest may be based on spatial variations in intensity values within the voxels. A large variation (e.g., exceeding a given threshold between two voxels) may indicate a change in the region of interest. Then, filling may comprise filling voxels detected as belonging to the same region of interest with a label associated with the region of interest to which they belong. For remaining voxels (e.g., having values lower than the threshold), filling may comprise filling voxels of the empty voxel grid corresponding to these remaining voxels with a label associated with the background (e.g., label "0").

[0035] The detection of labeled voxels can be performed automatically using known algorithms, such as neural networks, such as the U-Net described in the paper "U-Net: Convolutional Networks for Biomedical Image Segmentation" by O. Ronneberger, P. Fischer, and T. Brox (Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015, edited by N. Navab, J. Hornegger, W. M. Wells, and A. F. Rangi, Lecture Notes in Computer Science. Cham: Springer International Publishing, 2015, pp. 234-241. doi:10.1007 / 978-3-319-24574-4_28), which is incorporated herein by reference. U-Net has a classic automatic segmentation architecture. Alternatively, the detection of labeled voxels can be performed semi-automatically. For example, filling voxels with labels may involve a user performing a specific action, such as determining the outline of a region of interest to be considered or verifying a region of interest automatically detected by a known algorithm. For example, determining the outline of the region of interest may be performed by a user. The determination may comprise displaying a set of medical images (e.g., each medical image in turn) and selecting, through user interaction, voxels belonging to various regions of interest that can be displayed on the display (e.g., by defining the outline of each region of interest on each medical image).

[0036] The modifying S30 may comprise, for each voxel to be modified, adding to the value of the voxel an offset value that depends on the respective label associated with the voxel. Only a portion of the voxels may be modified. For example, the method may comprise modifying values of voxels that belong to the region of interest and leaving values of voxels that belong to the background unmodified.

[0037] The offset value may depend on the region of interest in which the voxel is located. In the resulting voxel grid, each region of interest may exclusively include voxels with values generated in their respective (i.e., non-overlapping) ranges. For each voxel to be modified, the added offset value may be equal to the result of multiplying the respective label associated with the voxel by a predetermined coefficient. The predetermined coefficient may be the same for all voxels. The predetermined coefficient may depend on the range of intensity values within the voxel grid defined by the medical image set. For example, the predetermined coefficient may be a value that is strictly greater than the range of values within the voxel grid defined by the medical image set. This predetermined coefficient may be manually defined. Alternatively, the coefficient may be automatically calculated, for example, as the difference between the maximum and minimum intensity within the voxel grid (e.g., the previous modification step S30) plus an arbitrary positive value (e.g., 1).

[0038] The correction S30 may be performed by sequentially considering all voxels one by one. For example, for each voxel, the correction S30 may include identifying a label associated with the voxel and determining an offset value to be added based on the identified label (e.g., by multiplying the identified label by a previous predetermined coefficient). Then, for each voxel, the correction S30 may include attributing a new value to the voxel by adding the offset value determined for the voxel to the voxel's value. If the voxel does not belong to the region of interest, the new value may be the same as the value the voxel already has. For example, for voxels belonging to the background, the label may be zero, which means that the result of multiplying the identified label by the predetermined coefficient is also zero, and the value for each of these voxels remains the same.

[0039] Alternatively, the correction S30 can be performed by considering all voxels in the same mathematical operation. In that case, the correction S30 calculates the offset voxel grid V according to the following formula: m The method may comprise calculating:

[0040] Vm =V+tS

[0041] where V is a voxel grid defined by the medical image set, t is a predetermined coefficient, and S is a segmentation voxel grid. The offset voxel grid can be a voxel grid with modified values.

[0042] In an example, the method may comprise, after the modification S30, rendering a voxel grid with the modified values (ie, an offset voxel grid). Rendering the offset voxel grid may comprise preparing data for 3D visualization of the offset voxel grid by a 3D viewer, for example, known 3D viewers such as 3D Slicer (described in the article "3D Slicer as an Image Computing Platform for the Quantitative Imaging Network" by A. Fedorov et al., Magn Reson Imaging, vol. 30, no. 9, pp. 1323-1341, Nov. 2012, doi:10.1016 / j.mri.2012.05.001, which is incorporated herein by reference), ITK Snap (Paul A. Yushkevich, Joseph Piven, Heather Cody Hazlett, Rachel Gimpel Smith, Sean Ho, James C. Gee, Guido Gerig. User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability. Neuroimage 2006 Jul. 2006), and others. 1;31(3):1116-28, incorporated herein by reference) or OsiriX (described on the website page https: / / www.osirix-viewer.com / osirix / osirix-md / , incorporated herein by reference). The viewer may be a medical image viewer capable of performing direct volume rendering. The prepared data may comprise a calculated offset voxel grid. After rendering, the method may comprise sending (or transmitting) the prepared data (i.e., including the calculated offset voxel grid) to a 3D viewer. The 3D viewer may be configured to display the offset voxel grid based on the transmitted data. In an example, the method may stop after sending the data to the 3D viewer.In another example, the method may include, after transmitting the data to the 3D viewer, displaying the resulting voxel grid by the 3D viewer based on the transmitted data. For example, the 3D viewer may include a graphical user interface, and the method may include displaying the resulting voxel grid on the graphical user interface.

[0043] Displaying the offset voxel grid by the 3D viewer may be performed by applying a volume rendering algorithm (e.g., a ray casting algorithm). During the volume rendering algorithm, the intensity values of the offset voxel grid may be converted into at least two appearance values, including an RGB value and an opacity value (e.g., a ratio representing the opacity of a voxel in the resulting rendering), according to a transfer function (i.e., by applying a transfer function to the values of the offset voxel grid). These appearance values may further include additional values, such as a metallicity value and / or a surface roughness value. The transfer function may be defined as a combination of multiple transfer functions forming a single piecewise transfer function. Each of these multiple transfer functions may be defined over a range of intensity values in the offset voxel grid associated with a region of interest or background (i.e., a region other than any region of interest) resulting from the segmentation S20. By converting the intensity values of the offset grid according to a single piecewise transfer function, various regions of interest, which may strictly include anatomical structures, may be clearly highlighted in the display of the voxel grid.

[0044] In an example, the application of the volume rendering algorithm may be performed by a 3D viewer. In that case, the data transmitted to the 3D viewer may include one or more transfer functions to be applied. The application of the one or more transfer functions to the offset voxel grid may be performed by the 3D viewer. Each transfer function may receive as input the intensity value of a given voxel and output a respective value of each appearance parameter for the given set of voxels. Each transfer function may be applied to a respective set of voxels. For example, each segment (or region of interest) may be associated with a respective transfer function, which may be applied to the voxels of the segment. A segment is a segment (i.e., a region of interest) obtained as a result of segmentation S20. The value assignment may comprise, for each segment, applying a respective transfer function associated with the segment to the voxels belonging to the segment. This allows various anatomical structures represented in the medical image set to be represented with characteristic appearances. Indeed, since each segment comprises voxels representing the same anatomical structure, the method makes it possible to apply a respective function to each segment of voxels representing the same anatomical structure, thus rendering each anatomical structure with a characteristic appearance.

[0045] In an example, one or more (e.g., all) of the plurality of transfer functions may be user-defined. In that case, the method may comprise a user (e.g., a medical professional or operator) determining one or more user-defined transfer functions. Determining each user-defined transfer function may comprise determining a graph of the transfer function. For example, the transfer functions may be defined as dashed-line graphs, in which case determining the graph of the function may comprise adding, deleting, and / or moving breakpoints of the dashed-line graph via a digital user interface.

[0046] In examples, one or more (e.g., all) of the plurality of transfer functions may be predetermined. In that case, the one or more predetermined transfer functions may be determined prior to performing the method. For example, the shape of the function and / or its characteristic points may be recorded in memory, and the method may comprise retrieving the shape of the function and / or its characteristic points from memory for each of the predetermined transfer functions.

[0047] In an example, before transmitting the data, the method may combine the respective transfer functions into a single piecewise transfer function, where each portion of the single transfer function may correspond to a respective region of interest and may have its own color and intensity scale. For example, the single piecewise transfer function may be of the form:

[0048] ∀i∈{0,…,n objects},f m (v)=f i (v-ti),v∈[v min +ti,v max +ti]

[0049] where f m is a single piecewise transfer function, f i is the respective transfer function associated with segment i, v min and v max are the minimum and maximum values of the voxel grid, respectively, and t is a predetermined coefficient. The single transfer function makes our method compatible with known general-purpose direct volume rendering tools.

[0050] Each transfer function f i may take intensity values as input and output a respective value for each appearance parameter of the set based on the input intensity values. For example, each transfer function f i can be of the form:

[0051] ∀v∈[v min ,v max ],f i (v)=(c,σ,p)

[0052] where c is an RGB color, σ is an opacity value, and p includes one or more values of possibly one or more additional appearance parameters (e.g., metallicity parameters, specularity parameters, and / or roughness parameters, including those described, for example, at https: / / en.wikipedia.org / wiki / Specular_highlight). max and v min may be the maximum and minimum intensity values in the voxel grid, respectively.

[0053] The method may then comprise transmitting the single piecewise transfer function to a viewer. For example, the single piecewise transfer function may be included in data transmitted to the 3D viewer. The transmission of the data to the 3D viewer may be performed in any manner. For example, the method may be performed on a computer on which the 3D viewer may be installed. The transmission of the data may be performed using this connection. The viewer may then be configured to display an offset voxel grid calculated based on the transmitted single piecewise transfer function (e.g., using a known volume ray casting algorithm).

[0054] Volume rendering refers to the process of generating 2D images corresponding to views of a 3D scene when the underlying geometry of the scene is densely represented by a 3D matrix (i.e., a voxel grid) whose values describe the local appearance properties of the volume. Common volume rendering algorithms include ray-casting or maximum intensity projection algorithms. Volume rendering differs from surface rendering, which refers to the process of generating 2D images corresponding to views of a 3D scene when the underlying geometry of the scene is sparsely represented by the surfaces of the objects contained within it. These surfaces are typically decomposed into tileable primitives (triangles, quadrilaterals), forming so-called meshes. Common surface rendering algorithms include rasterization or ray-tracing algorithms. Unlike surface rendering, volume rendering can account for variations in opacity of scene elements with depth. By enabling volume rendering, the method improves the rendering of regions of a patient.

[0055] In an example, the method may comprise using a voxel grid having modified values (i.e., an offset voxel grid) to prepare a patient for surgery. For example, the method may be included in a surgery preparation process, which may comprise, after performing the method, preparing a patient for surgery based on the offset voxel grid. Preparing for surgery may comprise sending the offset voxel grid to a 3D viewer to display the offset voxel grid (e.g., as described above) and determining surgical procedures to be performed based on the displayed offset voxel grid (e.g., identifying areas to be treated, determining procedures to be performed on these areas, and / or determining tool paths to use to perform these procedures). Alternatively, or in addition, the method may comprise using the offset voxel grid for communication with a patient. For example, the method may comprise sending the offset voxel grid to a 3D viewer to display the offset voxel grid to a patient (e.g., as described above), so that a medical professional (e.g., a doctor or nurse) can explain to the patient the conditions seen in the displayed offset voxel grid. By allowing the offset voxel grid to be displayed to the patient, the method also improves surgical preparation and / or patient communication. Indeed, the method allows various anatomical structures represented in a medical image set to be represented with distinctive appearances, which is particularly useful in preparing for a surgical procedure and / or communicating with the patient.

[0056] An example implementation of this method will be described with reference to FIGS.

[0057] In known solutions for volume rendering, intensity values in a medical image (e.g., Hounsfield units for CT scans) are converted to color and opacity values according to a transfer function (also called a color map or lookup table). Depending on the selected transfer function, direct volume rendering reveals different tissues and anatomical structures. However, only one transfer function can be used for the entire volume at a time. In fact, because transfer functions depend only on intensity values, they are spatially blind. In other words, transfer functions do not depend on the voxel's location. Different anatomical structures at different locations with similar intensities will have the same appearance (color and opacity).

[0058] As a result, different anatomical structures cannot be visualized in volume renderings due to occlusion issues (e.g., in volume renderings of CT scans, visceral fat is usually hidden by the skin). More generally, anatomical structures such as abdominal organs or various muscles cannot be rendered with specific characteristic colors because they share overlapping intensity ranges. For example, in Figure 2, the liver 101 and kidneys 103 are difficult to distinguish because their intensity values are similar.

[0059] Our method solves these occlusion problems by making volume rendering spatially aware. Notably, our method requires little coding and can be easily embedded into existing modern volume rendering pipelines.

[0060] An explanation of the terms used is provided here.

[0061] A CT scan (computed tomography scan) is a specific type of medical image produced by shining X-rays at the body. The signals are read and analyzed to reconstruct a dense volume of the body.

[0062] The Hounsfield unit (HU) is a relative quantitative measurement of radiological density used by radiologists in interpreting computed tomography (CT) images (see, for example, https: / / www.ncbi.nlm.nih.gov / books / NBK547721 / ). Figure 3 shows an example of the Hounsfield scale for a CT scan. It shows an example of the range that corresponds to each tissue (bone: 400-1000 HU, soft tissue: 40-80 HU, …).

[0063] A medical image set may be a set of CT scan images acquired simultaneously in a single acquisition, which may define a voxel grid of HU values representing the patient's anatomy.

[0064] A region of interest (ROI) is an anatomical structure that is intended to be highlighted by the method. For example, the region of interest may be an anatomical structure that includes a contrast agent.

[0065] A transfer function (also called a look-up table (LUT) or color map) is a function configured to map intensity values of a medical image volume to color and opacity values to enable volume rendering.

[0066] Direct volume rendering (or volume rendering) refers to the process of generating a 2D image from 3D volume data. Direct volume rendering methods involve converting the source volume into a 3D grid with color and opacity values via a transfer function.

[0067] Ray casting is the baseline direct volume rendering algorithm. It consists of casting lines of sight from an observation position towards the volume. Color and opacity values are sampled along these lines of sight. The color of each pixel in the resulting 2D image is obtained as a blend of the colors of the sampled points along its associated line of sight, weighted by the opacity value.

[0068] A spatially blind function is a function that does not depend on spatial information such as location.

[0069] A spatially aware function, as opposed to a spatially blind function, is a function that depends on spatial information.

[0070] Figure 3 shows an example of a calculated offset voxel grid.

[0071] The diagram shows a medical image set 200 received as input by the method (acquired in step S10). The medical image set defines a volume grid V of dimensions (D, H, W) that holds the intensity data of a 3D medical image acquisition, where D is the number of acquired slices (i.e., the number of medical images in the set), and H and W are the height and width of each slice, respectively. The values in V (called V values) are expressed as v min From v max The range of the V value is v amp and v amp =v max -v min becomes equal to

[0072] The figure shows a segmentation voxel grid 210, which is a volume grid S of dimensions (D, H, W) that represents a manually or automatically created segmentation mask of V. S contains a number of voxels ranging from 0 to n objects Filled with integer values up to n objects is the number of segmented objects of interest, which can be any type of anatomical structure (e.g., tumor, fat, organ, etc.). S(z,y,x)=i∈{1,...,n objects} indicates that voxel (z,y,x) of V belongs to the object labeled i. S(z,y,x)=0 indicates that voxel (z,y,x) of V belongs to the background.

[0073] In this example, the medical image set comprises two regions of interest. In the segmentation voxel grid 210, voxels 211 belonging to the first region of interest are associated with the label "1" (in this example, the kidneys). Voxels 212 belonging to the second region of interest are associated with the label "2". Voxels 210 belonging to the background are associated with the label "0" (in this example, the liver).

[0074] The figure also shows the resulting offset voxel grid 220. The method calculates the offset voxel grid 220 by multiplying the segmented voxel grid 210 by a predetermined factor t and adding the result to the initial voxel grid 200. The figure shows that the intensity values of the resulting offset voxel grid 220 are in the non-overlapping ranges of the first and second regions of interest (221 and 222), allowing these two distinct regions of interest to be highlighted in the final rendering.

[0075] 4 shows another flowchart of an example method. In this example, the method comprises the following steps:

[0076] In the first step, the method finds offset t>v amp (e.g., t=5000 for a standard CT scan).

[0077] In the second step (runtime), the method calculates the offset volume V m Calculating the offset volume comprises acquiring a medical image set S10, segmenting a voxel grid S20, and modifying values of the voxel grid S30, thereby obtaining the offset volume V m is obtained.

[0078] In the third step, the method calculates a preset transfer function f specific to each object i (including the background) such that iThe user can design or select the

[0079] ∀v∈[v min ,v max ],f i (v)=(c,σ,p)

[0080] where c is the RGB color, σ is the opacity value, and p is a parameter for possible other appearance properties (e.g., specular reflectance or roughness). This step can be performed interactively, for example, using a medical image viewer (such as the slicer described at https: / / www.slicer.org).

[0081] In a fourth step, the method modifies the segmented object transfer function V by a modified V defined by the part: m A single piecewise transfer function f fitted to m It is possible to combine it with.

[0082] ∀i∈{0,…,n objects},f m (v)=f i (v-ti),v∈[v min +ti,v max +ti]

[0083] So if voxel (z,y,x) belongs to object i, then f m (V m (z,y,x))=f i (V(z,y,x)). Figures 5 and 6 show examples of piecewise transfer functions.

[0084] In a fifth step, the method calculates the corrected volume V m and the transfer function f mto a generic volume rendering tool (such as the Open3D library described at http: / / www.open3d.org / docs / release / # or the VTK library described at https: / / vtk.org / ) to execute a volume ray casting algorithm and obtain a visualization of the desired results (see Figures 7 and 8).

[0085] Figures 5 and 6 show the relationship between a single piecewise transfer function f m Figure 5 shows an example of the resulting single piecewise transfer function f m , as a function of the corrected intensity value. In this example, the method generates a single piecewise transfer function f that outputs the three RGB color channels: red, green, and blue. m The first peak 301, associated with the color "red" on the graph, corresponds to the region of interest "kidney." The second peak 302, associated with the color "green" on the graph, corresponds to the region of interest "liver." The third peak 303, associated with the color "blue" on the graph, corresponds to the region of interest "lung." Figure 6 shows the resulting single piecewise transfer function f m , as a function of the corrected intensity values. The first peak corresponds to the unsegmented portion of the scan, and each of the following peaks corresponds to a segmented region (kidney, liver, lung as described with reference to FIG. 5). The resulting single piecewise transfer function f m can be decomposed into multiple (e.g., four in the example) transfer functions, each contributing within a respective range of intensity values (e.g., separated by an offset of 5000 HU in the example), corresponding to regions of the body separated by segmentation masks (non-segmented regions, kidneys, liver, and lungs in the example), allowing different regions of interest to be clearly highlighted in the final rendering.

[0086] 7 and 8 show examples of volume rendering using the present method. In particular, FIG. 7 shows the resulting volume rendering obtained by the present method for an example CT scan using the transfer functions shown in FIGS. 5 and 6. The present method allows for the assignment of characteristic colors to the liver 402, kidneys 401, and lungs 403 according to the segmentation masks (green, red, and blue). In particular, the present method allows for a sharp rendering of the lungs 403, which are particularly difficult to render with known volume rendering solutions due to their density values being close to air. FIG. 8 shows the volume rendering results obtained by the present method for an example CT scan enhanced with a segmentation mask of visceral fat 404.

[0087] FIG. 9 shows an example of a system, where the system is a client computer system, for example a user's workstation.

[0088] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000 and a random access memory (RAM) 1070 connected to the bus. The client computer further includes a graphical processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the bus. The video RAM 1100 is also known in the art as a frame buffer. A mass storage controller 1020 manages access to mass storage devices such as a hard drive 1030. Mass storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, such as semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, and magneto-optical disks. Any of the foregoing may be supplemented by, or incorporated in, specially designed application-specific integrated circuits (ASICs). A network adapter 1050 manages access to a network 1060. The client computer may also include a haptic device 1090, such as a cursor control device, keyboard, or the like. A cursor control device is used in the client computer to allow a user to selectively position a cursor at any desired location on the display 1080. Furthermore, the cursor control device allows a user to select various commands and input control signals. The cursor control device includes several signal generating devices for inputting control signals to the system. Typically, the cursor control device may be a mouse, with the mouse buttons used to generate the signals. Alternatively, or in addition, the client computer system may include a sensitive pad and / or a sensitive screen.

[0089] A computer program may comprise computer-executable instructions, which comprise means for causing the system to perform the method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuitry, or computer hardware, firmware, software, or a combination thereof. The program may be implemented as an apparatus, such as an article of manufacture tangibly embodied in a machine-readable storage device, executed by a programmable processor. The method steps may be performed by a programmable processor executing a program of instructions that performs the functions of the method by operating on input data and generating output. The processor is thus programmable and may be coupled to receive data and instructions from, and transmit data and instructions to, a data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language, as appropriate. In either case, the language may be a compiled or interpreted language. The program may be a full installation program or an update program. In either case, applying the program to a system generates instructions for performing the method. The computer program may alternatively be stored and executed on a server in a cloud computing environment, the server communicating with one or more clients via a network, in which case the processing unit executes the instructions contained in the program, thereby performing the method in the cloud computing environment.

Claims

1. 1. A computer-implemented method for volume rendering a medical image set of a patient, comprising: - acquiring (S210) said set of medical images defining a voxel grid comprising voxels, each voxel being associated with a value; - segmenting (S20) the voxel grid defined by the set of medical images by associating with each voxel a respective label from a set of labels, each corresponding to a respective region of interest of the patient; - modifying the values of the voxel grid by adding to each value to be modified an offset value that depends on the respective label associated with the voxel (S30); A method comprising:

2. The method of claim 1 , wherein the offset value added to each value is equal to the respective label associated with the voxel multiplied by a predetermined coefficient.

3. The correction is performed by offsetting the voxel grid V according to the following formula: ｍ Calculating V ｍ =V+tS where V is the voxel grid defined by said medical image set, t is said predetermined coefficient, and S is the segmentation voxel grid; The method of claim 1 or 2, wherein the method further comprises transmitting the calculated offset voxel grid to a viewer.

4. 4. The method of claim 1, wherein each segment is associated with a respective transfer function, said respective transfer function outputting, for each voxel of the segment with which it is associated, a respective value for at least two sets of appearance parameters comprising a color parameter and an opacity parameter, said at least two sets of appearance parameters optionally comprising at least one additional appearance parameter such as a metallicity parameter and / or a surface roughness parameter.

5. The method of claim 4 , wherein one or more of the transfer functions are predetermined.

6. The method of claim 4 or 5, wherein one or more of the transfer functions are user-defined.

7. - combining said respective transfer functions into a single piecewise transfer function; and - transmitting said single piecewise transfer function to said viewer; The method of any one of claims 4 to 6, further comprising:

8. The single piecewise transfer function is of the form: For all \(i\in\{0,\ldots,n\}\), ｏｂｊｅｃｔｓ \(f\) ｍ \((v)=f\) ｉ \((v - t_i)\) for \(v\in[v\) ｍｉｎ \(+t_i,v\) ｍａｘ \(+t_i]\) where f ｍ is the single piecewise transfer function, and f ｉ is the respective transfer function associated with the segment i, and v ｍｉｎ and v ｍａｘ 8. The method of claim 7, wherein t is the predetermined coefficient and t is the minimum and maximum values of the voxel grid, respectively.

9. The method according to any one of claims 2 to 8, wherein the predetermined factor is greater than the variation range of the values in the voxel grid.

10. The region of interest of the patient is one or more organs, such as the brain, heart, lungs, liver, pancreas, kidneys, skin and / or intestines, one or more tissues, such as epithelial tissue, connective tissue (e.g. adipose tissue), muscle tissue and / or nervous tissue, one or more vessels, such as blood vessels, and / or - one or more pathological forms, such as tumors The method according to any one of claims 1 to 9, wherein the compound is selected from the group consisting of:

11. The method according to any one of claims 1 to 10, wherein the set of medical images is produced by a CT scanner or an MRI scanner.

12. A computer program comprising instructions for carrying out the method according to any one of claims 1 to 11.

13. A computer-readable storage medium having the computer program according to claim 12 recorded thereon.

14. 13. A system comprising a processor coupled to a memory having the computer program of claim 12 recorded thereon.

15. The system of claim 14 , further comprising a viewer, the viewer comprising a graphical user interface configured to display the volume rendering of the set of medical images.

Images