Single rendering of sets of medical images

Abstract

SOLUTION: The present disclosure relates particularly to a computer-implemented method for single-rendering at least two sets of medical images of a patient. The at least two sets of medical images cover an area of the patient. The method comprises obtaining (S10) the at least two sets of medical images. Each set of medical images covers one or more respective regions of interest. The method comprises extracting (S20) the one or more respective regions of interest of each of the obtained at least two sets of medical images. The method comprises assembling (S30) the extracted respective regions of interest into a single set of images. Such a method forms an improved solution for single-rendering sets of medical images.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present disclosure relates to the field of computer programs and systems, and more particularly to methods, systems, and programs for single rendering of at least two medical image sets of a patient. [Background technology]

[0002] The use of multiple sets of medical images to cover the same area of a patient is currently very common in medical applications. For example, in the case of computed tomography scans (CT scans), radiologists can observe a particular type of structure in the CT scan before and after the injection of a contrast agent into the body. In that case, a medical imaging examination may involve acquiring a first set of medical images before the injection of the contrast agent and a second set of medical images after the injection of the contrast agent, thereby generating two sets of medical images covering the same area of the patient. These differ significantly in the method of injecting the radiological contrast agent, such as intravenously, orally, or rectally. For example, in oncology, the injection of the contrast agent is primarily performed into blood vessels.

[0003] Individually, each acquired medical image set can be visualized in 3D using standard volume rendering algorithms. For example, a standard volume rendering algorithm called "ray casting" can be used to render CT scans. In this algorithm, density values from the CT scan are mapped to color and opacity values using lookup tables. These lookup tables depend on the tissues intended to be displayed.

[0004] The main problem is that different sets of medical images covering the same region (e.g., acquired during different phases of a CT scan) may reveal different anatomical structures in the volume rendering. Therefore, for some applications, such as surgical planning or patient communication, it is useful to see all these anatomical structures in a single view. However, known standard volume rendering algorithms are unable to compute a rendering that can be used for different sets of medical images in a single view. Summary of the Invention [Problem to be solved by the invention]

[0005] In this context, there remains a need for improved solutions for single rendering sets of medical images. [Means for solving the problem]

[0006] Accordingly, there is provided a computer-implemented method for single-rendering at least two medical image sets of a patient, the at least two medical image sets covering a region of the patient. The method comprises acquiring the at least two medical image sets, each medical image set covering one or more respective regions of interest. The method comprises extracting one or more respective regions of interest from each of the at least two acquired medical image sets. The method comprises assembling the extracted respective regions of interest into a single image set.

[0007] The method may comprise one or more of the following: Extracting (S20) includes: Segmenting (S21) the at least two acquired medical image sets, thereby generating a respective preliminary binary segmentation mask for each medical image set; Calculating (S22) the intersection between all the generated preliminary binary segmentation masks, thereby obtaining a common mask of one or more common regions of the at least two acquired medical image sets; and For each medical image set, subtracting (S23) the obtained common mask from a respective preliminary binary segmentation mask generated for the medical image set, thereby obtaining a respective final binary segmentation mask for one or more respective regions of interest of the medical image set. The calculation of the crossover (S22) is based on the following formula:

number

number

number

[0008] Further provided is a computer program comprising instructions for carrying out the method.

[0009] Additionally, a computer readable storage medium having a computer program recorded thereon is provided.

[0010] Additionally provided is a system comprising a processor coupled to a memory, the memory having a computer program recorded thereon.

[0011] The system may further comprise a viewer, which may comprise a graphical user interface configured to display the single image set.

[0012] Further provided is a device comprising a data storage medium having a computer program recorded thereon. The device may form or function as a non-transitory computer-readable medium, such as in a Software as a Service (SaaS) or other server or cloud-based platform. The device may alternatively comprise a processor coupled to the data storage medium. Thus, the device may form, in whole or in part, a computer system (e.g., the device is a subsystem of the overall system). The system may further comprise a viewer. The viewer may comprise a graphical user interface configured to display a single image set. [Brief explanation of the drawings]

[0013] Non-limiting examples will now be described with reference to the accompanying drawings.

[0014] [Figure 1] 1 shows a flowchart of an example method. [Figure 2] An example of a single image set is shown. [Figure 3] An example of the Hounsfield scale for a CT scan is shown. [Figure 4] An example of assembling each extracted region of interest into a single image set is shown. [Figure 5] 10 shows another flowchart of an example method. [Figure 6] 1 shows an example flowchart of extracting regions of interest and assembling the extracted regions of interest into a single medical image set. [Figure 7] 1 shows an example of a single rendering produced by the method. [Figure 8] 1 shows an example of a single rendering produced by the method. [Figure 9] An example of a transfer function is shown below. [Figure 10] An example of a transfer function is shown below. [Figure 11]An example of a transfer function is shown below. [Figure 12] An example of a system is shown. DETAILED DESCRIPTION OF THE INVENTION

[0015] Referring to the flowchart of Figure 1, a computer-implemented method for single rendering of at least two medical image sets of a patient is proposed. The at least two medical image sets cover a region of the patient. The method comprises acquiring S10 the at least two medical image sets, each covering one or more respective regions of interest. The method comprises extracting S20 one or more respective regions of interest from each of the at least two acquired medical image sets. The method comprises assembling S30 the extracted respective regions of interest into a single image set.

[0016] Such a method provides an improved solution for a single rendering set of medical images.

[0017] In particular, the method allows rendering at least two sets of medical images of a patient in a single view. In particular, the single rendering calculated by the method is particularly useful. In fact, the method allows rendering different regions of interest of each set of medical images in a single rendering. This single rendering is therefore particularly useful for applications such as surgical planning or patient communication. In fact, this allows all anatomical structures located in / constituting the region of interest specific to each set of medical images to be seen in a single view.

[0018] Furthermore, the method is particularly efficient in terms of computation, and the unique rendering is particularly comprehensive. Indeed, first, the respective regions of interest of each medical image set are extracted, and then these extracted regions of interest are assembled to form a single resulting image set. Thus, all regions of interest covered in the different medical image sets acquired as input can be efficiently and accurately incorporated. In particular, the method allows this single rendering to be realized in a computer-automated manner, whatever the medical image set considered as input.

[0019] The method is for single rendering of at least two medical image sets of a patient, and may comprise a volume compositing step consisting of creating a single medical image set (i.e., only one set) from the multiple medical image sets, thereby allowing all regions of interest revealed by the multiple medical image sets to be visualized in a single visualization result.

[0020] The method may then comprise rendering and displaying (e.g., using a 3D viewer described below) this created single medical image set so that a medical professional (e.g., a doctor or nurse) can examine the area of the patient or show it to the patient.

[0021] The method is computer-implemented. This means that the steps (or substantially all steps) of the method are performed by at least one computer, or any similar system. Thus, the method steps are performed by a computer, possibly fully automatically or semi-automatically. In an example, triggering of at least some of the method steps may be performed by user / computer interaction. The level of user / computer interaction required may depend on the level of automation that is anticipated and commensurate with the need to implement the user's wishes. In an example, this level may be user-defined and / or pre-defined.

[0022] A typical example of a computer implementation of the method is executing the method on a system adapted for this purpose. The system may include a processor and a graphical user interface (GUI) coupled to a memory having recorded thereon a computer program comprising instructions for carrying out the method. The memory may also store a database. The memory is any hardware adapted for such storage, possibly comprising multiple physically distinct parts (e.g., one for the program, perhaps another for the database).

[0023] In an example, the method may comprise, after assembling S30, rendering the single image set. For example, rendering the single image set may comprise applying an algorithm for volume rendering (hereinafter referred to as a "volume rendering algorithm") to the single image set to obtain data that can be visualized in 2D for display on a screen (e.g., using a standard viewer). The volume rendering algorithm may be a known standard volume rendering algorithm (e.g., an algorithm implementing ray casting or maximum intensity projection). In fact, the data structure of the single image set may be the same as the data structure of each medical image set obtained as input by the method. For example, the single image set may define a voxel grid comprising voxels, each voxel having an associated value.

[0024] Prior to applying the volume rendering algorithm, the method may comprise assigning a respective color and opacity value to each voxel of the single image set according to values associated with the voxels in the single image set. The assignment of color and opacity values may be performed in any manner, for example, using transfer function(s) or lookup table(s). The transfer function(s) or lookup table(s) may be used to map intensity values from the acquisition (e.g., HU for a CT scan) to graphic / visual properties (color vector and opacity as a minimum) to enable the desired graphic rendering. The functions may be manually defined, for example, so that desired tissues are visible and distinguishable when they correspond to intensity ranges.

[0025] A volume rendering algorithm may be applied to the resulting voxel grid (i.e., including the respective color and opacity values assigned to each voxel) and configured to project these 3D data onto a 2D representation. The result may then be visualized using a 3D viewer. For example, the method may further comprise displaying the obtained 2D representation on a screen.

[0026] Volume rendering refers to the process of generating a 2D image corresponding to a view of a 3D scene when the underlying geometry of the scene is densely represented by a 3D matrix (i.e., a voxel grid) whose values represent the local appearance properties of the volume. Common volume rendering algorithms include ray-casting or maximum intensity projection algorithms. Volume rendering differs from surface rendering, which refers to the process of generating a 2D image corresponding to a view of a 3D scene when the underlying geometry of the scene is sparsely represented by the surfaces of the objects contained within it. These surfaces are typically decomposed into tileable primitives (triangles, quadrilaterals), forming a so-called mesh. Common surface rendering algorithms include rasterization or ray-tracing algorithms. Unlike surface rendering, volume rendering can take into account the depth-dependent opacity variations of scene elements. By enabling volume rendering of distinctive regions, the method therefore improves visualization of the patient's region.

[0027] In an example, the method may comprise using a single image set to prepare a patient for surgery. For example, the method may be included in a surgery preparation process, which may comprise, after performing the method, preparing a patient for surgery based on the single image set. Preparing for surgery may comprise displaying the single image set and determining surgical procedures to be performed based on the displayed single image set (e.g., identifying areas to be treated, determining procedures to be performed on those areas, and / or determining tool paths to use to perform those procedures). Alternatively, or in addition, the method may comprise using the single image set for patient communication. For example, the method may comprise displaying the single image set to the patient (e.g., using a 3D viewer as described above) so that a medical professional (e.g., a doctor or nurse) can explain to the patient what is seen in the displayed single image set. By enabling grouped rendering of different medical image sets, the method also improves surgery preparation and / or patient communication. Indeed, this is particularly useful for surgical preparation and / or patient communication, as all anatomical structures revealed in the different medical image sets can be viewed in a single view.

[0028] Acquiring S10 the at least two medical image sets may comprise obtaining the at least two medical image sets. For example, each medical image set may be acquired by any other imaging medical device, such as a CT scanner or a magnetic resonance imaging (MRI) scanner. In that case, acquiring S10 may comprise acquiring the at least two medical image sets using the CT scanner or other imaging medical device. Alternatively, the at least two medical image sets may already be acquired at the time of performing the method (e.g., using a CT scanner). In that case, acquiring S10 may comprise retrieving the at least two already acquired medical image sets. For example, the at least two medical image sets may be stored in memory, and acquiring S10 may comprise retrieving the acquired at least two medical image sets from memory.

[0029] The at least two medical image sets provided in the input (S10) may have been acquired consecutively. For example, the at least two medical image sets may have been acquired at different phases during a medical imaging examination, as is known. For example, the medical imaging examination may include the injection of a contrast agent (e.g., as described in https: / / radiologyassistant.nl / more / ct-protocols / ct-contrast-injection-and-protocols and / or https: / / radiopaedia.org / articles / contrast-phases). In that case, the at least two medical image sets may have been acquired at different phases distributed before and / or after the injection of the contrast agent (i.e., by a CT scanner or other imaging medical device). For example, the at least two medical image sets may include one or more sets of medical images acquired before the injection of the contrast agent and one or more sets of medical images acquired after the injection of the contrast agent. Before or after injection means that each set may have been acquired with or without the presence of a contrast agent in the patient's body. The contrast agent may be absent from the patient's body prior to injection and may be present after injection (eg, varying in abundance as a function of time after the start of injection).

[0030] In an example, the at least two medical image sets may include at least one medical image set acquired in a non-enhanced phase. Contrast may not have been injected into the patient during this phase. For example, the medical imaging examination may not include an injection of contrast, or an injection of contrast may not have been performed yet. Alternatively, or in addition, the at least two medical image sets may include at least one medical image set acquired in an arterial phase. The arterial phase may be after the injection of contrast. The arterial phase may be 25-50 seconds after the injection of contrast (i.e., after the start of the injection). For example, the arterial phase may be 35-40 seconds after the injection of contrast. Alternatively, or in addition, the at least two medical image sets may include at least one medical image set acquired in a portal vein phase. The portal vein phase may be 60-90 seconds after the injection of contrast. For example, the portal vein phase may be 70-80 seconds after the injection of contrast.

[0031] We now consider an arbitrary set of medical images. The use of such medical image sets is well known. For example, it is known that a scanner-type examination may involve taking medical images periodically along a patient's body (e.g., a portion thereof), with each image representing a slice of the patient's body. When assembled together, these medical images may then be used to reconstruct a 3D volume that represents the patient's body (e.g., a portion thereof) slice by slice. Thus, the set of medical images may together represent a 3D volume that includes a region of the patient. Each medical image may be a 2D image and may represent a respective slice along this 3D volume represented by the set. Thus, a 3D volume may be formed by assembling consecutive slices represented by different medical images of the set.

[0032] The set of medical images may define a voxel grid. The voxel grid may be aligned with a vertical direction of the patient's region. The voxel grid may comprise layers of voxels overlaid on one another along the vertical direction. Each voxel layer may comprise voxels of the grid that are co-located along the vertical direction. Each voxel layer may represent a respective slice of the 3D volume represented by the set. Each medical image of the set may be perpendicular to the vertical direction. Each voxel may have a value defined by the set of medical images. In particular, each medical image may define values for voxels belonging to a voxel layer that is co-located with the medical image along the vertical direction. When the voxel layer and the medical image are overlaid, each voxel may have a value that corresponds to the value of the portion of the medical image that the voxel contains.

[0033] When acquired by a CT scanner, a medical image set may be generated by irradiating the human body with X-rays. The resulting signals may be read and analyzed to reconstruct a dense volume of the body. In this case, the values of the voxels defined by the medical image set may be Hounsfield Units (HU) values. HU may be a relative quantitative measure of radiation density used by radiologists in interpreting computed tomography (CT) images.

[0034] Each medical image set covers a region (i.e., a section or part) of the patient. This means that each set images said region of the patient. In other words, the patient region is represented in at least some (e.g., all) of the medical images in each set. The region covered by the at least two medical image sets can be any region of the patient's body. For example, the body region may include, partially or entirely, the head, neck, trunk (chest, abdomen, and pelvis), one or both upper limbs, and / or one or both lower limbs. The region is a common region covered by each of the at least two medical image sets. This means that each medical image set may also cover another region of the patient's body, but each of the at least two medical image sets covers at least this common region. Here, the term region is synonymous with the term part, for example, each medical image set covers a part of the patient.

[0035] Each medical image set covers one or more respective regions of interest. Each of the one or more regions of interest is included in the region of the patient's body covered by the respective medical image set. Each region of interest may be a respective portion of this region of the patient's body. Each region of interest may be any type of region of the human body that can be targeted by some type of medical imaging (i.e., whose appearance, content, and / or shape can be revealed by this medical imaging). The regions of interest may include any one or any combination of the regions of the human body listed on the website page https: / / www.radiologyinfo.org / en / info / safety-contrast. For example, the regions of interest may include one or more internal organs (e.g., portions thereof), such as the brain, breast, heart, lungs, liver, adrenal glands, kidneys, pancreas, gallbladder, spleen, uterus, and / or bladder. Alternatively, or in addition, the regions of interest may include the gastrointestinal tract, including, for example, the stomach, small intestine, and / or large intestine. Alternatively, or in addition, the region of interest may include one or more arteries and veins of the body, such as blood vessels in the brain, neck, chest, abdomen, pelvis, and / or legs. Alternatively, or in addition, the region of interest may include one or more other parts of the body, including muscle and / or bone.

[0036] Preferably, the region of interest may be a region of the body that can be enhanced by a contrast agent, for example, the region of interest may include an artery, a vein, an intestine, a stomach, a urinary tract, a tumor(s), and / or a bladder.

[0037] Each medical image set may cover one or more respective regions of interest from all possible regions of interest listed above. These one or more respective regions of interest may be covered by only one of the medical image sets. The one or more respective regions covered by each set may depend on the phase in which the set is acquired. For example, in a medical image set acquired in a non-enhanced phase, the one or more respective regions may be regions that are not revealed by the presence of contrast agent. Conversely, in a medical image set acquired in an arterial or venous phase, the one or more respective regions may be regions that are revealed by the injection of contrast agent (which becomes apparent later in the venous phase). Each medical image set may also cover one or more common regions (i.e., covered in at least two medical image sets). In other words, the one or more common regions are portions or portions that are not specifically revealed in a single medical image set (i.e., everything that is not included in each region of interest). The one or more common regions may be regions that are revealed regardless of whether contrast agent is present in the body. For example, the one or more common regions may include all bones present in regions of the patient's body covered by at least two medical image sets.

[0038] For each of the at least two sets of medical images, the extraction S20 of one or more respective regions of interest may be performed in any manner. The extraction S20 may be performed based on any segmentation of the set of medical images. For example, the extraction S20 may be performed manually by a medical professional (e.g., a doctor or nurse). For each set of medical images, the extraction S20 may comprise displaying the set of medical images (e.g., each medical image in turn) and selecting, by user interaction (e.g., by defining the outline of a respective region of interest on each medical image), one or more respective regions of interest covered by the set of medical images. The result of the segmentation is a respective final binary segmentation mask for each set of medical images.

[0039] Alternatively, extraction S20 may be performed automatically or semi-automatically. For example, extraction S20 may include segmenting S21 the acquired at least two medical image sets, thereby generating a respective preliminary binary segmentation mask for each medical image set. Each generated preliminary binary segmentation mask may also define a voxel grid. The voxel grid defined by each generated preliminary binary segmentation mask may have the same spatial layout as that defined by the medical image set, except that the values in the voxel grid may be of binary type (e.g., "0" or "1"). The binary value may define whether a corresponding voxel in the grid defined by the medical image set (i.e., located in the same location in the patient's body region) belongs to the region of interest. For example, the binary value may define whether the value of this corresponding voxel is above or below a predetermined threshold (e.g., 300 HU for a CT scan) indicating that the voxel belongs to the region of interest. Segmentation S21 of each medical image set may be performed in any manner. Segmentation S21 may comprise assigning binary values to a voxel grid according to the values of the medical image set (the voxel grid having the same dimensions as those defined by the medical image set). For example, segmentation S21 may comprise determining an empty voxel grid having the same dimensions as those defined by the medical image set but with no values, and determining for each voxel in this empty voxel grid whether the value of the corresponding voxel in the grid defined by the medical image set is below or above a predetermined threshold, and assigning a binary value to the voxel accordingly (e.g., a value of "0" if the value is above the predetermined threshold, and a value of "1" otherwise). The resulting voxel grid may be a respective preliminary binary segmentation mask calculated for the medical image set.

[0040] After segmentation S21, extraction S20 may comprise calculating S22 an intersection between all generated preliminary binary segmentation masks, thereby obtaining a common mask of one or more common regions of at least two acquired medical image sets. The common mask may be calculated when the preliminary binary segmentation masks of the previous step contain both a region of interest that is unique to at least two sets and a region of interest (e.g., bone) that is common to all sets. This may occur, for example, when performing threshold segmentation.

[0041] In an example, the intersection calculation S22 may be performed by calculating the intersection of all the generated respective preliminary binary segmentation masks. For example, the intersection calculation S22 may be based on the following formula:

[0042]

number

[0043] where M Common is the common mask, and for each medical image set labeled k in the label set {0,…,#series-1}, M k ROI+Common is each generated preliminary binary segmentation mask. The multiplication Π may be an element-wise multiplication. This formula may be applied during calculation S22 and may include, for each voxel location, multiplying the values of all respective binary segmentation masks at this voxel location and assigning the result of this multiplication to the same voxel location in the common mask. For each voxel location where all of the respective preliminary binary segmentation masks have a value of "1", this results in a value of "1" at the same voxel location in the common mask, and a value of "0" otherwise.

[0044] After the calculation S22, the extraction S20 may comprise, for each medical image set, subtracting S23 the common mask of the obtained common regions from the respective preliminary binary segmentation mask generated for the medical image set, thereby obtaining a respective final binary segmentation mask of one or more respective regions of interest for the medical image set. Each final binary segmentation mask is the result of the difference S23 (in the sense of set theory) between the common mask and the respective preliminary binary segmentation mask. In other words, each final binary segmentation mask may define a voxel grid whose value is "1" for voxels that hold a value "1" in the respective binary segmentation mask and a value "0" in the common mask, and whose value is "0" for other voxels. The values of each mask may represent only one or more regions of interest of the medical image set (i.e., not one or more common regions).

[0045] In an example, the subtraction S23 may be based on the following formula:

[0046]

number

[0047] where M Common is a common mask, and for each medical image set labeled k in the label set {0,…,#series}, M k ROI+Common is each generated preliminary binary segmentation mask, and M k ROI is the respective final binary segmentation mask obtained. In this formula, 1 may be a voxel grid with the same dimensions but containing only positive binary values (i.e., "1"), and the operator × may indicate element-wise multiplication.

[0048] Assembling S30 may comprise determining a single image set. Determining the single image set may comprise determining a voxel grid (i.e., a grid defined by the single image set) and assigning values to this voxel grid according to each extracted region of interest. The voxel grid defined by the single image set may have the same dimensions as that defined by the at least two medical image sets. Assembling S30 may comprise assigning values to this voxel grid to identify each region of interest in the at least two medical image sets. For example, the method may comprise modifying values from each extracted region of interest so that they fall within different intensity ranges, where each intensity range is associated with a unique region of interest.

[0049] In an example, the assignment of the respective value may be performed for each voxel. Assigning the respective value to each voxel of the voxel grid may include determining for each voxel whether the voxel belongs to one of the respective regions of interest of one of the at least two medical image sets, and assigning the respective value to the voxel accordingly. For example, if the voxel belongs to one of the regions of interest, the method may include assigning to the voxel the value of a voxel at a corresponding position in the voxel grid defined by one of the medical image sets, which medical image set is associated with the region of interest in which the voxel is located. Otherwise, if the voxel does not belong to one of the respective regions of interest, the method may include assigning to the voxel the value of a voxel at a corresponding position in a medical image set selected as a reference (the method may include selecting a medical image set from the at least two medical image sets to be considered as a reference).

[0050] We now describe different methods for assigning values to voxels belonging to a region of interest (hereinafter referred to as "assignment methods"), any of which may be used by the present method.

[0051] The first assignment method may include assigning the same label to voxels belonging to the same medical image set. In this case, the first assignment method may consider a set of labels corresponding to each medical image set. Each label of the set may be associated with each of at least two medical image sets. In this first assignment method, the value assigned to each voxel may be the respective label associated with the medical image set having the respective region of interest to which the voxel belongs. For example, each medical image set may be associated with a respective number (e.g., a positive integer n), and the assigned value may be equal to the respective number associated with the medical image set having the respective region of interest to which the voxel belongs. Next, the method may include displaying the resulting single image set by applying the volume rendering algorithm detailed above (e.g., using a specific transfer function). Thus, this first assignment method may enable regions of interest in different medical image sets to be highlighted. For example, different organs may be highlighted when they are revealed in different medical image sets.

[0052] The second assignment method may comprise assigning to each voxel the same value as a corresponding position in the respective region of interest to which the voxel belongs (i.e., for each voxel, in the medical image set including the respective region of interest to which the voxel belongs). In this case, the method may comprise assigning to each voxel, according to the respective mask, the value of a voxel at a corresponding position in a voxel grid defined by the medical image set associated with the region of interest to which the voxel belongs. For example, after determining for each voxel whether the voxel belongs to one of the respective regions of interest of one of the at least two medical image sets, the method may comprise assigning to the voxel a respective value equal to the value of the corresponding position in the medical image set having the respective region of interest to which the voxel belongs. The corresponding position may be the same voxel position in the voxel grid defined by the medical image set and the single image set. Therefore, this second assignment method allows accurate reproduction of nuances of values obtained in different medical image sets. In particular, because only the values of the highlighted region(s) of interest in each set are kept, the values of the different sets can be cleanly assembled, thus avoiding unnecessary and inaccurate summing of values in these regions (as would be the case if the values of all sets were simply summed everywhere).

[0053] According to the third assignment method, each value assigned to a voxel may be equal to the sum of the value of the corresponding location and an offset. The sum may be the value of the medical image set to which the voxel belongs, with the respective region of interest (as in the second method). The offset may depend on the medical image set to which the voxel belongs, with the respective region of interest. This third assignment method combines the advantages of both the first and second assignment methods, i.e., it can highlight regions of interest in different medical image sets and also reproduce value nuances. More generally, this allows volume rendering to be performed using transfer functions specific to each image set.

[0054] In an example, for the second and third allocation methods, the assembly may be based on the following formula:

[0055]

number

[0056] where V fusion is a single image set, V k is a medical image set labeled with k from the label set {0,…,#series-1}, V0 is a reference set of medical images with regions of interest from other medical image sets added, M k ROI is the final binary segmentation mask for each of the k labeled regions of interest in the medical image set in the label set {0,…,#series-1}, δ distinctive is a parameter, and kt is an offset obtained by multiplying the value of label k by parameter t. In the second assignment method, parameter δ distinctive can be equal to 0, thereby staying within the range of the original intensity values. In the third assignment method, the parameter δ distinctive may be equal to 1, so that each region of interest is associated with a new, distinct range of intensity values.

[0057] In an example, the method may further comprise registering the at least two medical image sets prior to the extraction. In that case, the extraction may be performed on the registered at least two medical image sets. The registration of the at least two medical image sets may be performed in any manner. For example, the registration may be performed by applying a linear transformation (translation, rotation) or a non-linear transformation to register the at least two medical image sets together. The objective of the registration step may be to find a transformation that minimizes the gap between the at least two medical image sets. Because the medical image sets may be acquired in different phases and the patient may (at least unconsciously) move between these different phases, the registration is involved in providing an accurate single image set.

[0058] In an example, the method may be performed dynamically. For example, medical image sets may be continuously acquired, and each time a new medical image set is acquired, the method may comprise updating the single image set to incorporate the newly acquired medical image set. The updating may comprise extracting one or more respective regions of interest from the newly acquired medical image set and assembling these one or more respective regions of interest with previously assembled regions of interest. This allows for live analysis of the generated images.

[0059] An implementation example of this method will be described with reference to FIGS.

[0060] Different tissues will appear the same in a volume rendering if their density values are the same. It is difficult to highlight specific organs or tissues differently. For example, areas highlighted by a contrast agent and bones have the same density value. Multiphase CT scans output multiple image series, each containing different meaningful visual information. Known methods make it impossible to display all of this information in a single visualization. This method allows for the creation of a fusion of multiple CT scans into a single 3D rendering, avoiding the need to visualize multiple 3D renderings (one for each phase).

[0061] The present method aims to perform CT scan fusion to visualize different tissues from multi-phase CT scans in a single view. The present method aims to highlight these tissues while not highlighting tissues with similar density values as the region of interest, such as bone. This allows the relative position of each structure to be visualized. Figure 2 shows an example of a single image set generated by the present method. The figure shows a first set of medical images 101 and a second set of medical images 102 considered in this example. In this example, the first set of medical images 101 and the second set of medical images 102 are acquired by a CT scanner. In particular, the first set of medical images 101 is acquired during the arterial phase, so arteries are visible in the first set of medical images 101. The second set of medical images 102 is acquired during a later phase, so the urinary tract is visible. The figure shows a single image set 103 generated by the present method and obtained by fusing the first set of medical images 101 and the second set of medical images 102. The figure shows that the assembly performed by the method is particularly accurate, since both the arteries and the urinary tract are visible. In this example, the method uses the second assignment method (i.e., including only value nuances).

[0062] Different regions of interest are highlighted in different ways, which is useful for preparing for surgery and communicating with the patient. In particular, the method overcomes the limitations of known solutions. In particular, the method allows all details of a CT scan to be displayed. For example, the method allows observing small blood vessels or detecting small changes in blood vessels (e.g., thrombosis or blood clots). The method also allows for volume rendering, which allows for visualization of details of the CT scan and information from different phases.

[0063] An explanation of the terms used is provided here.

[0064] A CT scan (computed tomography scan) is a specific type of medical image produced by shining X-rays at the body. The signals are read and analyzed to reconstruct a dense volume of the body.

[0065] The Hounsfield unit (HU) is a relative quantitative measurement of radiological density used by radiologists in interpreting computed tomography (CT) images (see, for example, https: / / www.ncbi.nlm.nih.gov / books / NBK547721 / ). Figure 3 shows an example of the Hounsfield scale for a CT scan. It shows an example of the range that corresponds to each tissue (bone: 400-1000 HU, soft tissue: 40-80 HU, …).

[0066] A medical image set may be a set of CT scan images acquired simultaneously in a single acquisition, which may define a voxel grid of HU values representing the patient's anatomy.

[0067] Multiphase CT scans are CT scans acquired at different times during the same medical exam. The two acquisitions may be separated by a few seconds. Each acquisition enhances one or more respective regions of interest (also called "ROIs") depending on the localization of contrast agent within the body at the time of acquisition.

[0068] A contrast agent (also called a contrast material or contrast medium) is a substance used in medical imaging to enhance the contrast of structures or fluids within the body. Examples of contrast agents are barium-, gadolinium-, or iodine-based. The HU value corresponding to an area of contrast agent can have different values depending on the concentration of the agent. Further details can be found, for example, at https: / / en.wikipedia.org / wiki / Contrast_agent or https: / / www.radiologyinfo.org / en / info / safety-contrast.

[0069] Registration is the transformation of different images of a scene into the same coordinate system. The transformation can be rigid, affine, homographic, or complex large deformation, and gradient descent is used to find the optimal transformation.

[0070] A region of interest (ROI) is an anatomical structure that is intended to be highlighted by the method. For example, the region of interest may be an anatomical structure that includes a contrast agent.

[0071] A transfer function (also called a look-up table (LUT) or color map) is a function configured to map intensity values of a medical image volume to color and opacity values to enable volume rendering.

[0072] A region that cannot be distinguished from the ROI (called an indistinguishable region) is a region whose set of intensity values overlaps with the intensity values of the ROI.

[0073] Direct volume rendering or volume rendering refers to the process of generating a 2D image from 3D volume data. Direct volume rendering methods involve using a transfer function to convert the source volume into a 3D grid of color and opacity values.

[0074] Ray casting is the baseline direct volume rendering algorithm. It consists of casting a line of sight from an observation position towards the volume. Color and opacity values are sampled along these rays. The color of each pixel in the resulting 2D image is obtained as a blend of the colors of the sampled points in its associated line of sight, weighted by the opacity value.

[0075] Binary morphological opening is a mathematical operation used in computer vision and image processing. Opening removes small objects from the foreground of an image (see, for example, details provided at https: / / en.wikipedia.org / wiki / Opening_(morphology)).

[0076] Binary morphological closing is a mathematical operation used in computer vision and image processing. Closing removes small holes (see, for example, details provided at https: / / en.wikipedia.org / wiki / Closing_(morphology)).

[0077] FIG. 4 illustrates an example of assembling each extracted region of interest into a single image set. In this example, the method comprises acquiring S10 two sets of medical images acquired from CT scans (two CT scan series). The first set is acquired in an arterial phase V0, and the second set is acquired in a later phase V1. For each set, the diagram illustrates a respective preliminary binary segmentation mask resulting from segmenting S21 the two sets. In particular, the diagram illustrates, for each set, the binary values of the layers of the respective preliminary binary segmentation mask corresponding to one of the medical images of the set. Layer 110 corresponds to the first set, and layer 120 corresponds to the second set.

[0078] The figure shows the first set of regions of interest M0 extracted by the method. ROI 114 and the second set of respective regions of interest M1 ROI124 are also shown. In particular, extraction S20 comprises segmenting S21 each set to generate a respective preliminary binary segmentation mask 110 and 120, and calculating S22 the intersection 132 between each of the two preliminary binary segmentation masks 110 and 120. Next, extraction S20 comprises subtracting S23, for the first set, the obtained common mask 132 from the respective preliminary binary segmentation mask 110, thereby generating a respective final binary segmentation mask M0 for each region of interest of the first set. ROI Similarly, the extraction S20 comprises, for the second set, subtracting S23 the obtained common mask 132 from the respective preliminary binary segmentation mask 120, thereby obtaining respective final binary segmentation masks M1 for each region of interest of the second set. ROI 124 is obtained.

[0079] The figure also shows the extracted regions of interest M0 ROI 114 and M1 ROI The figure also shows assembling S30 layers 110 and 124 into a single image set. In particular, the figure shows the result of assembling layers 110 and 120. The result of the assembly is image 130. In this example, the method applies the obtained common mask 132 to each extracted region of interest M0 ROI 114 and M1 ROI 124. The extracted regions of interest M0 obtained for the first set are also assembled. ROI 114 includes the arteries, and each extracted region of interest M1 ROI124 contains the urinary tract, while common mask 132 contains bones. Thus, the method allows for a single rendering of all these body parts. In this example, the method uses the first assignment method. For each voxel in single set 130, the value assigned to the voxel is the respective label associated with the medical image set from which the voxel belongs (e.g., "0" or "1" in this example). The method also comprises assigning a different label to all voxels that belong to common mask 132. The method can then render these values by applying a volume rendering algorithm for highlighting the regions of interest of the different medical image sets.

[0080] Figure 5 shows an example of a flow chart of the method, which, by following this flow chart, allows a single visualization of CT scans in different phases.

[0081] The method computes a medical image V k In this example, the method considers V0 as a reference series, and assembling S30 comprises adding ROIs (i.e., respective regions of interest) from the other series to this reference series V0.

[0082] The method comprises registering (i.e., aligning) all these series so that they can be compared. After registration, the method comprises calculating a merged series that is used to generate a volume rendering and visualized. Calculating the merged series comprises extracting S20 a respective region of interest from each series and assembling S30 the extracted respective regions of interest into a single medical image set.

[0083] FIG. 6 shows an example flow chart for extracting each region of interest for each series S20 and assembling each extracted region of interest into a single medical image set S30.

[0084] The extraction S20 comprises a set of segmentations S21, which results in a preliminary binary segmentation mask M containing the ROIs for each series and the indistinguishable (i.e., common) regions. k ROI+Common To do so, the method comprises defining a range that includes the HU values of the ROI (e.g., HU values greater than 300 HU). The defined value may depend on the concentration and type of contrast agent. Within this range of values, the method comprises capturing the contrast-enhanced region (ROI) and bone (indistinguishable region). To obtain a smoother result, the method may comprise applying a normalization filter, such as binary morphological opening or closing, to the result of segmentation S21.

[0085] The method then comprises calculating S22 the intersection between all previously calculated masks in order to obtain a mask of the common region (which is an indistinguishable region). The calculation of the intersection S22 is carried out according to the following formula:

[0086]

number

[0087] where M Common is the common mask, and for each medical image set labeled with k in the label set {0,…,#series-1}, M k ROI+Common is each generated preliminary binary segmentation mask.

[0088] These are common regions (e.g., bones) that have HU values within the range defined above across the input CT scan series. As a result, these common regions do not contain anatomical structures enhanced by contrast agents, because the enhanced anatomical structures are not the same across the entire series of multiphase CT scan examinations (as shown in the example in Figure 4).

[0089] Therefore, the method calculates a final binary segmentation mask M for each series, which contains only the ROI region, as the difference S23 between the common mask and the (ROI+indistinguishable region) preliminary binary segmentation mask. k ROI The difference calculation S23 is based on the following formula:

[0090]

number

[0091] where M Common is a common mask, and for each medical image set labeled k in the label set {0,…,#series}, M k ROI+Common is each generated preliminary binary segmentation mask, and M k ROI is the respective final binary segmentation mask obtained.

[0092] The method comprises defining an offset t (for example t=5000 HU) beyond the amplitude of the HU values of all series of the multi-phase examination.

[0093] Finally, the method calculates a fusion series V based on the reference series. fusion The creation of the fusion series may be based on the following algorithm: If voxel (x,y,z) belongs to no ROI other than one of the reference series, i.e., ∀k∈{1,…,#series},Mk ROI If (x,y,z)=0, receive the HU value of that corresponding voxel in the reference series. V fusion (x,y,z)=V0(x,y,z). Otherwise, if voxel (x,y,z) belongs to an ROI of series k>0, then either 〇V fusion (x,y,z)=V k (x,y,z). In that case, the method uses the normal transfer function of the CT scan to find V fusion This second method of assignment is called a blend mode (see FIG. 7). ○ or V fusion (x,y,z)=V k (x,y,z)+kt. This allows the adapted piecewise transfer function to generate a volume rendering in which different series of ROIs are displayed in distinctive colors (see Figure 8). This third assignment method is called the distinctive mode.

[0094] In summary, for the characteristic mode, δ distinctive = 1 for blend mode, and 0 for blend mode, the assemble S30 may be based on the following formula:

[0095]

number

[0096] where V fusion is a single image set, V k is a medical image set labeled with k from the label set {0,…,#series-1}, V0 is a reference set of medical images with regions of interest from other medical image sets added, M k ROI is the final binary segmentation mask for each of the regions of interest in the medical image set labeled k in the label set {0,…,#series-1}.

[0097] FIG. 7 shows a first example of a single rendering. In this example, the method generates a blended fused volume rendering of the arterial phase and the later phase. The method preserves the ROIs from both phases. The method allows the urinary tract from the later phase and the arterial tract from the arterial phase to be rendered in a single visualization.

[0098] FIG. 8 shows a second example of a single rendering. In this example, the method generates a distinctive fused volume rendering of the arterial phase and the later phase. The method can highlight the later phase ROI in a different color. In this example, the urinary tract 310 is displayed in a first color, while the rest of the volume is displayed in a different color. The figure shows that the method can preserve the shading of the urinary tract 310.

[0099] After computing the fusion series, the method may comprise defining an appropriate transfer function to emphasize each ROI extracted from each series (see examples in Figures 9, 10 and 11). Figure 9 shows how the method can be applied to the second assignment method (i.e., δ distinctive 10 and 11 show examples of transfer functions that the method may use for rendering a single image set when using the third allocation method (i.e., δ = 0 blend mode). distinctive 1 shows an example of a transfer function that the method can use for rendering a single image set when using a characteristic mode (i.e., .times. ...

[0100] The method may then comprise visualizing the CT scan in 3D using a standard volume rendering algorithm with an appropriate transfer function depending on whether the user selected a blend mode or a characteristic mode.

[0101] FIG. 12 shows an example of a system, which may be a client computer system, such as a user's workstation.

[0102] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000 and a random access memory (RAM) 1070 connected to the bus. The client computer further includes a graphical processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the bus. The video RAM 1100 is also known in the art as a frame buffer. A mass storage controller 1020 manages access to mass storage devices such as a hard drive 1030. Mass storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, such as semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, and magneto-optical disks. Any of the foregoing may be supplemented by, or incorporated in, specially designed application-specific integrated circuits (ASICs). A network adapter 1050 manages access to a network 1060. The client computer may also include a haptic device 1090, such as a cursor control device, keyboard, or the like. A cursor control device is used in the client computer to allow a user to selectively position a cursor at any desired location on the display 1080. Furthermore, the cursor control device allows a user to select various commands and input control signals. The cursor control device includes several signal generating devices for inputting control signals to the system. Typically, the cursor control device may be a mouse, with the mouse buttons used to generate the signals. Alternatively, or in addition, the client computer system may include a sensitive pad and / or a sensitive screen.

[0103] A computer program may comprise computer-executable instructions, which comprise means for causing the system to perform the method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuitry, or computer hardware, firmware, software, or a combination thereof. The program may be implemented as an apparatus, such as an article of manufacture tangibly embodied in a machine-readable storage device, executed by a programmable processor. The method steps may be performed by a programmable processor executing a program of instructions that performs the functions of the method by operating on input data and generating output. The processor is thus programmable and may be coupled to receive data and instructions from, and transmit data and instructions to, a data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language, as appropriate. In either case, the language may be a compiled or interpreted language. The program may be a full installation program or an update program. In either case, applying the program to a system generates instructions for performing the method. The computer program may alternatively be stored and executed on a server in a cloud computing environment, the server communicating with one or more clients via a network, in which case the processing unit executes the instructions contained in the program, thereby performing the method in the cloud computing environment.

Claims

1. 1. A computer-implemented method for single rendering at least two sets of medical images of a patient, the at least two sets of medical images covering a region of the patient; acquiring (S10) said at least two sets of medical images, each set of medical images covering one or more respective regions of interest; Extracting (S20) the one or more respective regions of interest in each of the acquired at least two medical image sets; and Assembling each of the extracted regions of interest into a single image set (S30). A method comprising:

2. The extraction (S20) Segmenting (S21) the acquired at least two sets of medical images, thereby generating a respective preliminary binary segmentation mask for each set of medical images; calculating (S22) an intersection between all the generated preliminary binary segmentation masks, thereby obtaining a common mask of one or more common regions of the at least two acquired medical image sets; and and subtracting (S23) for each medical image set the obtained common mask from the respective preliminary binary segmentation mask generated for the medical image set, thereby obtaining a respective final binary segmentation mask for the one or more respective regions of interest of the medical image set. The method of claim 1 , comprising:

3. Calculating the intersection (S22) is based on the following formula: [Equation 1] Here, M Ｃｏｍｍｏｎ is the common mask, and for each medical image set labeled k in the label set {0, . . . , #series-1}, M ｋ ＲＯＩ＋Ｃｏｍｍｏｎ The method of claim 2 , wherein each of the generated preliminary binary segmentation masks is a binary segmentation mask.

4. The subtraction (S23) is based on the following formula: [Equation 2] Here, M Ｃｏｍｍｏｎ is the common mask, and for each medical image set labeled k in the label set {0,...,#series}, M ｋ ＲＯＩ＋Ｃｏｍｍｏｎ is each of the generated preliminary binary segmentation masks, and M ｋ ＲＯＩ 4. The method of claim 2, wherein each of the final binary segmentation masks obtained is a final binary segmentation mask.

5. The single image set defines a voxel grid, and the assembling (S30) involves, for each voxel: determining whether the voxel belongs to one of the respective regions of interest of one of the at least two sets of medical images; and assigning a respective value to said voxel if said voxel belongs to one of said respective regions of interest according to said medical image set from which said respective mask was acquired, said respective region of interest to which said voxel belongs; A method according to any one of claims 1 to 4, comprising assigning a respective value to each voxel of the voxel grid by performing:

6. acquiring the at least two sets of medical images further comprises acquiring a set of labels corresponding to each of the sets of medical images; The method of claim 5 , wherein assigning the respective value comprises assigning the respective label associated with the medical image set having the respective region of interest to which the voxel belongs.

7. The method of claim 5 , wherein assigning the respective values comprises assigning the respective values equal to the values of corresponding locations in the set of medical images having the respective region of interest to which the voxel belongs.

8. The respective values assigned to the voxels are: a value of a corresponding position in the set of medical images having the respective region of interest to which the voxel belongs; and an offset dependent on the medical image set having the respective region of interest to which the voxel belongs; 6. The method of claim 5, wherein the sum is equal to the sum of

9. The assembly is based on the following formula: [Equation 3] Here, V ｆｕｓｉｏｎ is the single image set, and V ｋ is the medical image set labeled with k in the label set {0, ..., #series-1}, and V ０ is a reference set of medical images to which the regions of interest from other sets of medical images have been added, 1 is an indicator function, and M ｋ ＲＯＩ is the final binary segmentation mask of each of the regions of interest in the medical image set labeled k in the label set {0, ..., #series-1}, and δ ｄｉｓｔｉｎｃｔｉｖｅ 9. The method according to claim 7, wherein kt is a parameter equal to 1 for a characteristic mode and 0 for a blend mode, and kt is an offset obtained by multiplying the value of the label k by a parameter t.

10. 10. The method of claim 1, further comprising registering the at least two sets of medical images prior to said extraction, wherein said extraction is performed on the registered at least two sets of medical images.

11. The at least two sets of medical images include: at least one set of medical images acquired by a CT scanner, preferably in a non-enhanced phase; at least one set of medical images acquired by a CT scanner, preferably in an arterial phase, said arterial phase preferably being 25-50 seconds after injection of a contrast agent, preferably 35-40 seconds after injection of a contrast agent; and / or At least one medical image set acquired by a CT scanner, preferably in a portal vein phase, said portal vein phase being preferably 60-90 seconds after injection of a contrast agent, preferably 70-80 seconds after injection of a contrast agent. The method according to any one of claims 1 to 10, comprising:

12. A computer program comprising instructions for carrying out the method according to any one of claims 1 to 11.

13. A computer-readable storage medium having the computer program of claim 12 recorded thereon.

14. 13. A system comprising a processor coupled to a memory, the memory having the computer program of claim 12 recorded thereon.

15. The system of claim 14 , further comprising a viewer, the viewer comprising a graphical user interface configured to display a single image set.

Images