Single rendering of medical image sets

Abstract

The present disclosure particularly relates to a computer-implemented method for single rendering on at least two medical image sets of a patient. The at least two medical image sets cover an area of the patient. The method comprises the following steps: S10, obtaining at least two medical image sets; each medical image set covers one or more corresponding regions of interest. The method comprises the following steps: S20, extracting one or more corresponding regions of interest of each medical image set in the at least two acquired medical image sets; the method includes S30 combining the extracted respective regions of interest into a single image set. This method provides an improved solution for single rendering of a set of medical images.

Description

Technical Field

[0001] The present invention relates to the field of computer programs and systems, and more particularly, to a method, system, and program for performing a single rendering on at least two medical image sets of a patient. Background Art

[0002] Nowadays, it is very common to use multiple medical image sets in medical applications to cover the same area of a patient. For example, in the case of computed tomography (CT-scan), a radiologist can observe certain types of structures on a CT-scan before and after injecting a contrast agent into the body. In this case, a medical imaging examination may include acquiring a first medical image set before injecting the contrast agent and a second medical image set after injecting the contrast agent, thus generating two medical image sets covering the same area of the patient. They are significantly different in the way of injecting the contrast agent, such as intravenous injection, oral administration, or rectal injection. For example, in oncology, the injection of the contrast agent is mainly carried out intravenously.

[0003] Individually, each acquired medical image set can be visualized in 3D using a standard stereoscopic rendering algorithm. For example, a standard stereoscopic rendering algorithm called "ray casting" can be used to render a CT-scan. In this algorithm, a look-up table is used to map the density values of the CT-scan to color and opacity values. These look-up tables depend on the tissue to be visualized.

[0004] The main problem is that different medical image sets covering the same area (for example, acquired at different phases of a CT-scan) may display different anatomical structures in stereoscopic rendering. Therefore, for some applications, such as surgical planning or communicating with the patient, it is useful to view all these anatomical structures in a single view. However, the known standard stereoscopic rendering algorithms cannot calculate the available renderings of different medical image sets in a single view.

[0005] In this context, there is still a need for an improved solution for performing a single rendering on medical image sets. Summary of the Invention

[0006] Accordingly, there is provided a computer-implemented method for performing a single rendering on at least two medical image sets of a patient. The at least two medical image sets cover an area of the patient. The method includes acquiring the at least two medical image sets. Each medical image set covers one or more corresponding regions of interest. The method includes extracting one or more corresponding regions of interest from each of the acquired at least two medical image sets. The method includes combining the extracted corresponding regions of interest into a single image set.

[0007] The method may include one or more of the following:

[0008] -The extraction described in S20 includes:

[0009] o S21: Segmenting the at least two obtained medical image sets to generate corresponding initial binary segmentation masks for each medical image set;

[0010] o S22: Calculating the intersection between all the generated initial binary segmentation masks to obtain a common mask for one or more common regions of the at least two obtained medical image sets;

[0011] o S23: For each medical image set, subtracting the obtained common mask from the corresponding initial binary segmentation mask generated for the medical image set to obtain a corresponding final binary segmentation mask for one or more corresponding regions of interest of the medical image set;

[0012] -S22 calculates the intersection based on the following formula:

[0013]

[0014] where M Common is the common mask, and for each medical image set labeled k in the label set {0,..., #series - 1}, is the corresponding initial binary segmentation mask generated;

[0015] -S23 performs the subtraction based on the following formula:

[0016]

[0017] where M Common is the common mask, and for each medical image set labeled k in the label set {0,..., #series - 1}, is the corresponding initial binary segmentation mask generated, is the corresponding final binary segmentation mask obtained;

[0018] -The single image set defines a voxel grid. The combination described in S30 includes, for each voxel of the voxel grid, assigning a corresponding value to the voxel by performing the following operations on each voxel:

[0019] o Determining whether the voxel belongs to one of the corresponding regions of interest of one of the at least two medical image sets; and

[0020] o If the voxel belongs to one of the corresponding regions of interest, then assigning a corresponding value to the voxel according to the medical image set, where a corresponding mask having the corresponding region of interest to which the voxel belongs is obtained from the medical image set;

[0021] - The obtaining of at least two medical image sets further includes obtaining a tag set, where each tag corresponds to a corresponding medical image set;

[0022] - The assigning of the corresponding value includes assigning a corresponding tag associated with the medical image set having the corresponding region of interest to which the voxel belongs;

[0023] - The assigning of the corresponding value includes assigning a corresponding value equal to the value at the corresponding position of the medical image set having the corresponding region of interest to which the voxel belongs;

[0024] - The corresponding value assigned to the voxel is equal to the result of the following summation:

[0025] o The value at the corresponding position of the medical image set having the corresponding region of interest to which the voxel belongs;

[0026] o Depending on the offset of the medical image set having the corresponding region of interest to which the voxel belongs;

[0027] - The combination is based on the following formula:

[0028]

[0029] where V fusion is a single image set, V k is the medical image set labeled k in the tag set {0, …, #series - 1}, V0 is the reference medical image set with regions of interest added from other medical image sets, 1 is the indicator function, is the corresponding final binary segmentation mask of the region of interest of the medical image set labeled k in the tag set

[0030] {0, …, #series - 1}, δ distinctive is a parameter when the discrimination mode is equal to 1 and in the mixing mode

[0031] is equal to 0, and kt is the offset obtained by multiplying the value of tag k by parameter t; - The method further includes, before the extraction:

[0032] o Aligning the at least two medical image sets, and the extraction is performed in the aligned at least two medical image sets; and / or

[0033] - The at least two medical image sets include:

[0034] o Preferably, at least one medical image set is acquired by a CT scanner during the non - enhanced phase;

[0035] o Preferably, at least one set of medical images is acquired by the CT scanner during the arterial phase, which is preferably 25 to 50 seconds, preferably 35 to 40 seconds after the injection of the contrast agent; and / or

[0036] o Preferably, at least one set of medical images is acquired by the CT scanner during the portal venous phase, which is preferably 60 to 90 seconds, preferably 70 to 80 seconds after the injection of the contrast agent.

[0037] There is also provided a computer program product comprising instructions for performing the method.

[0038] There is also provided a computer-readable storage medium having recorded thereon the computer program product.

[0039] There is also provided a system comprising a processor coupled to a memory having recorded thereon the computer program product.

[0040] The system may further include a viewer. The viewer may include a graphical user interface for displaying a single set of images.

[0041] There is also provided a device comprising a data storage medium having recorded thereon a computer program. The device may form or serve as a non-transitory computer-readable medium, such as on SaaS (Software as a Service) or other servers, cloud-based platforms, etc. The device may optionally include a processor coupled to the data storage medium. Thus, the device may form a computer system, in whole or in part (e.g., the device is a subsystem of the entire system). The system may further include a viewer. The viewer may include a graphical user interface for a single set of images. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Now, non-limiting examples will be described with reference to the accompanying drawings, wherein:

[0043] - Figure 1 A flowchart showing an example of the method is shown.

[0044] - Figure 2 An example of a single set of images is shown.

[0045] - Figure 3 An example of the Hounsfield scale of a CT scan is shown.

[0046] - Figure 4 An example of combining the extracted regions of interest into a single set of images is shown.

[0047] - Figure 5 Another flowchart showing an example of the method is shown.

[0048] -Figure 6 A flowchart showing an example of extracting regions of interest and combining the extracted regions of interest into a single medical image set.

[0049] - Figure 7 and Figure 8 Showing an example of a single rendering generated by the method.

[0050] - Figure 9 、 Figure 10 and Figure 11 Showing an example of a transfer function.

[0051] - Figure 12 Showing an example of the system. Detailed Description of the Invention

[0052] Referring to Figure 1 the flowchart, a computer-implemented method for performing a single rendering on at least two medical image sets of a patient is proposed. The at least two medical image sets cover regions of the patient. The method includes S10 obtaining at least two medical image sets. Each medical image set covers one or more corresponding regions of interest. The method includes S20 extracting one or more corresponding regions of interest from each of the at least two obtained medical image sets. The method includes S30 combining the extracted corresponding regions of interest into a single image set.

[0053] This method provides an improved solution for performing a single rendering on medical image sets.

[0054] It is worth noting that the method allows for the rendering of at least two medical image sets of a patient in a single view. In particular, the single rendering calculated by the method is particularly useful. In fact, the method allows for the rendering of different regions of interest of each medical image set in a single rendering. Therefore, such a single rendering is particularly useful for applications such as surgical planning or communicating with the patient. In fact, it enables the viewing of all anatomical structures located within / constituting the specific regions of interest of each medical image set in one view.

[0055] Furthermore, the method is particularly computationally efficient and the unique rendering is particularly detailed. In fact, first, the corresponding regions of interest of each medical image set are extracted, and then, these extracted regions of interest are combined to form the final single image set. Therefore, it can effectively and accurately integrate all the regions of interest covered by different medical image sets used as input. In particular, regardless of which medical image sets are regarded as input, the method allows for the implementation of such a single rendering in a computer-automated manner.

[0056] The method is for performing a single rendering on at least two medical image sets of a patient. It may include a volume synthesis step that includes generating a single medical image set (i.e., only one set) from a number of medical image sets. It can visualize all regions of interest shown in the number of medical image sets as a single visualization.

[0057] Then, the method may include rendering and displaying the generated single medical image set (e.g., using a 3D viewer explained below) so that a medical practitioner (e.g., a doctor or a nurse) can study the region of the patient, or they can show it to the patient.

[0058] The method is implemented by a computer. This means that the steps (or substantially all steps) of the method are performed by at least one computer or any similar system. Thus, the steps of the method are performed by a computer, which may be fully automatic or semi-automatic. In an example, the triggering of at least some steps of the method can be performed through user-computer interaction. The required level of user-computer interaction may depend on the expected level of automation and be balanced with the needs of the user's wishes. In an example, this level can be user-defined and / or pre-defined.

[0059] A typical example of a computer implementation of a method is to execute the method using a system suitable for this purpose. The system may include a processor coupled to a memory and a Graphical User Interface (GUI), and a computer program including instructions for executing the method is recorded on the memory. The memory may also store a database. The memory is any hardware suitable for such storage and may include several physically different parts (e.g., one for the program and possibly one for the database).

[0060] In an example, the method may include rendering a single image set after assembly in S30. For example, the rendering of the single image set may include applying an algorithm for stereoscopic rendering (hereinafter referred to as "stereoscopic rendering algorithm") to the single image set to obtain data that can be visualized in 2D for display on a screen (e.g., using a standard viewer). The stereoscopic rendering algorithm may be a known standard stereoscopic rendering algorithm (e.g., an algorithm implementing ray casting or maximum intensity projection). In fact, the data structure of the single image set may be the same as the data structure of each medical image set that is input to the method. For example, the single image set may define a voxel grid including voxels respectively associated with values.

[0061] Before applying the volume rendering algorithm, the method may include assigning corresponding color and opacity values to each voxel in a single image set according to the values associated with the voxels in the single image set. The assignment of the color and opacity values may be performed in any way, such as using a transfer function or a look-up table. The transfer function or the look-up table can be used to map the acquired intensity values (e.g., HU of a CT scan) to the graphical / visual attributes (minimum being the color vector and opacity) to achieve the desired graphical rendering. For example, these functions can be defined manually so that the desired tissues corresponding to the intensity ranges can be viewed and distinguished.

[0062] The volume rendering algorithm can be applied to the resulting voxel grid (i.e., including the corresponding color and opacity values assigned to each voxel), and can be used to project the 3D data into a 2D representation. Then, a 3D viewer can be used to visualize the result. For example, the method may further include displaying the obtained 2D representation on a screen.

[0063] Volume rendering refers to the process of generating a 2D image corresponding to a 3D scene view when the underlying geometry of the scene is densely represented by a 3D matrix (i.e., a voxel grid), and the values of the 3D matrix describe the local appearance attributes of the volume. Common volume rendering algorithms include ray casting or maximum intensity projection algorithms. Volume rendering is different from surface rendering, which refers to the process of generating a 2D image corresponding to a 3D scene view when the underlying geometry of the scene is sparsely represented by the surfaces of the objects it contains. These surfaces are usually decomposed into tilable primitives (triangles, quadrilaterals) to form a so-called mesh. Common surface rendering algorithms include rasterization or ray tracing algorithms. In contrast to surface rendering, volume rendering is able to take into account the opacity changes of scene elements in the depth direction. By performing volume rendering on different regions, the method thus improves the visualization of the patient region.

[0064] In an example, the method may include preparing a surgery for a patient using a single image set. For example, the method may include, during the surgery preparation and after performing the method, the process may include preparing a surgery for the patient based on the single image set. Preparing the surgery may include displaying the single image set and determining the surgical operations to be performed based on the displayed single image set (e.g., identifying the areas to be treated, determining the operations to be performed on these areas and / or determining the tool paths for performing these operations). Alternatively or additionally, the method may include communicating with the patient using the single image set. For example, the method may include displaying the single image set to the patient (e.g., using the 3D viewer as described above) so that a healthcare practitioner (e.g., a doctor or a nurse) can explain to the patient what is visible in the displayed single image set. By implementing grouped rendering of different medical image sets, the method also improves surgery preparation and / or patient communication. In fact, all the anatomical structures shown in different medical image sets can be viewed in a single view, which is particularly useful for preparing the surgery and / or communicating with the patient.

[0065] S10 obtaining at least two medical image sets may include acquiring at least two medical image sets. For example, each medical image set may be acquired by a CT scanner or any other imaging medical device (such as a Magnetic Resonance Imaging (MRI) scanner). In this case, S10 obtaining may include acquiring at least two medical image sets using a CT scanner or other imaging medical device. Alternatively, at the time of performing the method, at least two medical image sets may have been acquired (e.g., using a CT scanner). In this case, S10 obtaining may include retrieving the at least two medical image sets that have been acquired. For example, the at least two medical image sets may have been recorded in a memory, and S10 obtaining may include retrieving the at least two acquired medical image sets from the memory.

[0066] At least two medical image sets may have been successively acquired from the input provided by S10. For example, it is known that during a medical imaging examination, at least two medical image sets may have been acquired at different phases. For example, a medical imaging examination may include an injection of a contrast agent (e.g., as described in https: / / radiologyassistant.nl / more / ct-protocols / ct-contrast-injection-and-protocol s and / or https: / / radiopaedia.org / articles / contrast-phases). In such a case, at least two medical image sets may have been acquired at different phases before and / or after the injection of the contrast agent (i.e., by a CT scanner or other imaging medical device). For example, at least two medical image sets may include one or more medical image sets acquired before the injection of the contrast agent and one or more medical image sets acquired after the injection of the contrast material. Before or after injection means that each set may have been acquired with or without the presence of the contrast agent in the patient's body. There may be no contrast agent in the patient's body before injection, and there may be contrast agent after injection (e.g., after the start of injection, its presence rate changes over time).

[0067] In an example, at least two medical image sets may include at least one medical image set acquired during a non-enhanced phase. During this phase, the contrast agent may not have been injected into the patient's body. For example, a medical imaging examination may not include an injection of a contrast agent, or the injection of the contrast agent may not have been performed yet. Alternatively or additionally, at least two medical image sets may include at least one medical image set acquired during an arterial phase. The arterial phase may be after the injection of the contrast agent. The arterial phase may be 25 to 50 seconds after the injection of the contrast agent (i.e., after the start of injection). For example, the arterial phase may be 35 to 40 seconds after the injection of the contrast agent. Alternatively or additionally, at least two medical image sets may include at least one medical image set acquired during a portal venous phase. The portal venous phase may be 60 to 90 seconds after the injection of the contrast agent. For example, the portal venous phase may be 70 to 80 seconds after the injection of the contrast agent.

[0068] Now discuss any medical image set. The use of such a medical image set is well known. For example, it is known that an examination of a scanner type may involve regularly taking medical images along a patient's body (e.g., a part), and each image represents a slice of the patient's body. These medical images are combined together and can then be used to reconstruct a 3D volume representing the patient's body (e.g., a part) slice by slice. Therefore, the medical images in the set can together represent a 3D volume including the patient's region. Each medical image may be a 2D image and may represent a corresponding slice along the 3D volume represented by the set. Therefore, a 3D volume can be formed by assembling consecutive slices represented by different medical images in the set.

[0069] The medical image set can define a voxel grid. The voxel grid can be longitudinally aligned with a region of a patient. The voxel grid can include voxel layers stacked on top of each other along the longitudinal direction. Each voxel layer can include voxels of a grid located at the same position along the longitudinal direction. Each voxel layer can represent a corresponding slice of the 3D volume represented by the set. Each medical image of the set can be perpendicular to the longitudinal direction. Each voxel can have a value defined by the medical image set. In particular, each medical image can define the value of a voxel belonging to a voxel layer that is located at the same position along the longitudinal direction as the medical image. When the voxel layer and the medical image are superimposed, each voxel can have a value corresponding to the value in the portion of the medical image included in the voxel.

[0070] When acquired by CT-scanning, the medical images of the set can be generated by passing X-rays through the human body. The generated signals can be read and analyzed to reconstruct a dense volume of the body. In this case, the voxel values defined by the medical image set can be Hounsfield Unit (HU) values. HU can be a relative quantitative measure of radio density used by radiologists when interpreting Computed Tomography (CT) images.

[0071] Each medical image set covers a region (i.e., a slice or a portion) of a patient. This means that each set images a region of the patient. In other words, the region of the patient is represented in at least a part (e.g., all) of each medical image set. The region covered by at least two medical image sets can be any region of the patient's body. For example, the region of the body can include part or all of the head, neck, trunk (chest, abdomen, and pelvis), one or both upper limbs, and / or one or both lower limbs. The region is a common region covered by each of the at least two medical image sets. This means that each medical image set can also cover another region of the patient's body, but each of the at least two medical image sets covers at least this common region. Here, the term "region" is a synonym for the term "portion", e.g., each medical image set covers a portion of the patient.

[0072] Each medical image set covers one or more corresponding regions of interest. The one or more corresponding regions of interest are included in the region of the patient's body covered by each medical image set. Each region of interest may be a corresponding part of that region of the patient's body. Each region of interest may be any type of region of the human body that may be the target of a medical imaging (i.e., whose appearance, content, and / or shape can be revealed by such medical imaging). The regions of interest may include any one or any combination of the regions of the human body indicated in the website page https: / / www.radiologyinfo.org / en / info / safety-contrast. For example, the regions of interest may include one or more (e.g., a part of) visceral organs such as the brain, breast, heart, lungs, liver, adrenal glands, kidneys, pancreas, gallbladder, spleen, uterus, and / or bladder. Alternatively or additionally, the regions of interest may include the digestive tract, such as including the stomach, small intestine, and / or large intestine. Alternatively or additionally, the regions of interest may include one or more arteries and veins of the body, including blood vessels in the brain, neck, chest, abdomen, pelvis, and / or legs. Alternatively or additionally, the regions of interest may include one or more other parts of the body, including muscles and / or bones.

[0073] Preferably, the region of interest may be a region of the body that can be enhanced by a contrast agent. For example, the regions of interest may include arteries, veins, intestines, stomachs, urinary tracts, tumors, and / or bladders.

[0074] Each medical image set may cover one or more corresponding regions of interest among all the possible regions of interest listed above. These one or more corresponding regions of interest may be covered by only one of the medical image sets. The one or more corresponding regions covered by each set may depend on the phase during which the set is acquired. For example, for a medical image set acquired during the non-enhanced phase, the one or more corresponding regions may be the regions shown in the absence of a contrast agent. In contrast, for a medical image set acquired during the arterial or venous phase, the one or more corresponding regions may be the regions shown due to the injection of a contrast agent (especially in the later stage of the venous phase). Each medical image set may also cover one or more common regions (i.e., regions covered by at least two medical image sets). In other words, the one or more common regions refer to the parts not specifically shown by a single medical image set (i.e., all the content not in the corresponding regions of interest). The one or more common regions may be those regions that are shown regardless of the presence or absence of a contrast agent in the body. For example, the one or more common regions may include each bone present in the region of the patient's body covered by at least two medical image sets.

[0075] For each of at least two medical image sets, the extraction of one or more corresponding regions of interest S20 can be performed in any manner. The S20 extraction can be performed based on any segmentation of the medical image set. For example, the S20 extraction can be performed manually by a medical practitioner (e.g., a doctor or a nurse). For each medical image set, the S20 extraction can include displaying the medical image set (e.g., displaying each medical image in sequence), and selecting one or more corresponding regions of interest covered by the medical image set through user interaction (e.g., by defining the contour of each region of interest on each medical image). For each medical image set, the result of the segmentation is a corresponding final binary segmentation mask.

[0076] Alternatively, the S20 extraction can be performed automatically or semi-automatically. For example, the S20 extraction can include segmenting S21 the at least two medical image sets obtained to generate a corresponding initial binary segmentation mask for each medical image set. The corresponding initial binary segmentation masks generated can also define a voxel grid. The voxel grid defined by the generated corresponding initial binary segmentation mask can have the same spatial layout as the spatial layout defined by the medical image set, except that the values within the voxel grid can be of binary type (e.g., "0" or "1"). The binary values can define whether the corresponding voxels (i.e., at the same position in the patient's body region) of the grid defined by the medical image set belong to the region of interest. For example, the binary values can define whether the value in the corresponding voxel is above or below a predetermined threshold, and the predetermined threshold can mark the voxel as belonging to the region of interest (e.g., 300 HU for a CT-scan). The S21 segmentation of each medical image set can be performed in any manner. The S21 segmentation can include assigning binary values to the voxel grid according to the values of the medical image set (the size of the voxel grid is the same as the size defined by the medical image set). For example, the S21 segmentation can include determining an empty voxel grid with the same size as the voxel grid defined by the medical image set and without values; and for each voxel in the empty voxel grid, determining whether the value of the corresponding voxel in the grid defined by the medical image set is below or above a predetermined threshold; and accordingly assigning a binary value to the voxel (e.g., "0" when the value is above the predetermined threshold, otherwise "1"). The resulting voxel grid can be the corresponding initial binary segmentation mask calculated for the medical image set.

[0077] After the S21 segmentation, the S20 extraction can include calculating S22 the intersection between all the generated initial binary segmentation masks to obtain a common mask for one or more common regions of the at least two medical image sets obtained. The common mask can be calculated when the initial binary segmentation masks in the previous step contain two regions of interest specific to at least two sets and regions of interest common to all two sets (e.g., bones). For example, this can occur when performing threshold segmentation.

[0078] In the example, the S22 calculating the intersection can be performed by calculating the intersection of all the generated corresponding initial binary segmentation masks. For example, the S22 calculating the intersection can be based on the following formula:

[0079]

[0080] where M Common is the common mask, and for each medical image set labeled with k in the label set {0, …, #series - 1}, is the corresponding generated initial binary segmentation mask. The multiplication Π can be an element-wise multiplication. This formula can be applied during the S22 calculation, and for each voxel position, it can involve multiplying the values of all the corresponding binary segmentation masks at that voxel position and assigning the result of the multiplication to the same voxel position in the common mask. For each voxel position where the values of all the corresponding initial binary segmentation masks are "1", the value of the same voxel position in the common mask is "1", otherwise it is "0".

[0081] After the S22 calculation, the S20 extraction can include, for each medical image set, subtracting the common mask of the obtained common region from the corresponding initial binary segmentation mask generated for the medical image set to obtain the corresponding final binary segmentation mask for one or more corresponding regions of interest for the medical image set. The corresponding final binary segmentation mask is the result of the difference (in the sense of set theory) between the S23 common mask and the corresponding initial binary segmentation mask. In other words, the corresponding final binary segmentation mask can define a voxel grid, where for voxels that remain with the value "1" in the corresponding binary segmentation mask and are "0" in the common mask, the value is "1", otherwise it is "0". The values of the respective masks can represent only one or more regions of interest of the medical image set (i.e., not one or more common regions).

[0082] In the example, the S23 subtraction can be based on the following formula:

[0083]

[0084] where M Common is the common mask, and for each medical image set labeled with k in the label set {0, …, #series - 1}, is the corresponding generated initial binary segmentation mask, is the obtained corresponding final binary segmentation mask. In this formula, 1 can be a voxel grid with the same dimensions but only containing positive binary values (i.e., "1"), and the operator × can represent element-wise multiplication.

[0085] The S30 combination may include determining a single image set. Determining the single image set may include determining a voxel grid (i.e., the voxel grid defined by the single image set) and assigning values to the voxel grid according to the corresponding regions of interest extracted. The voxel grid defined by the single image set may have the same dimensions as the voxel grid defined by at least two medical image sets. The S30 combination may include assigning values to the voxel grid in order to identify the corresponding regions of interest of at least two medical image sets. For example, the method may include modifying the values of the corresponding regions of interest extracted so that they are included in different intensity ranges. In this case, each intensity range is associated with a unique region of interest.

[0086] In an example, the assignment of the corresponding values may be performed voxel by voxel. Assigning the corresponding values to each voxel of the voxel grid may include, for each voxel, performing a determination of whether the voxel belongs to one of the corresponding regions of interest of at least two medical image sets and accordingly assigning the corresponding value to the voxel. For example, if the voxel belongs to one of the regions of interest, the method may include assigning the value of the voxel at the corresponding position in the voxel grid defined by one of the medical image sets to the voxel. The medical image set is one of the medical image sets associated with the region of interest where the voxel is located. Otherwise, if the voxel does not belong to one of the corresponding regions of interest, the method may include assigning the value of the voxel at the corresponding position in the medical image set selected as the reference to the voxel (the method may include a step of selecting the medical image set regarded as the reference from at least two medical image sets).

[0087] Now discuss different methods (hereinafter referred to as "assignment methods") for assigning values to the voxels belonging to the regions of interest. The method may use any one of these assignment methods.

[0088] The first assignment method may include assigning the same label to voxels belonging to the same medical image set. In this case, the first assignment method may consider a set of labels, where each label corresponds to a respective medical image set. Each label of the set may be associated with a respective one of at least two medical image sets. In this first assignment method, the value assigned to each voxel may be the respective label associated with the medical image set having the respective region of interest to which the voxel belongs. For example, each medical image set may be associated with a respective number (e.g., a positive integer n), and the assigned value may be equal to the respective number associated with the medical image set having the respective region of interest to which the voxel belongs. Then, the method may include displaying the resulting single image set by applying the volume rendering algorithm as described above (e.g., using a specific transfer function). Thus, the first assignment method allows highlighting the regions of interest of different medical image sets. For example, when different organs are displayed in different medical image sets, it may highlight these different organs.

[0089] The second assignment method may include assigning the same value at the respective positions of the respective regions of interest to which the voxels belong (i.e., for each voxel, in the medical image set including the respective region of interest to which the voxel belongs). In this case, the method may include, according to a respective mask, assigning to each voxel the value of the voxel at the corresponding position in the voxel grid defined by the medical image set associated with the region of interest to which the voxel belongs. For example, for each voxel, after determining whether the voxel belongs to one of the respective regions of interest of at least two medical image sets, the method may include assigning to the voxel a respective value equal to one of the values at the corresponding positions of the medical image set having the respective region of interest to which the voxel belongs. The corresponding position may be the same voxel position in the voxel grid defined by the medical image set and the single image set. Thus, the second assignment method allows precisely reproducing the subtle differences in the values obtained in different medical image sets. In particular, it allows clearly combining the values of different sets, since only the values of the regions of interest highlighted by each set are retained, thus avoiding unnecessary and inaccurate addition of the values of these regions (which would occur if the values of all sets were simply added together anywhere).

[0090] According to the third assignment method, the corresponding value assigned to a voxel can be equal to the sum of the value at the corresponding position and an offset. The values being added can be the values of the medical image set of the corresponding region of interest to which the voxel belongs (as in the second method). The offset can depend on the medical image set of the corresponding region of interest to which the voxel belongs. This third assignment method can combine the advantages of the first and second assignment methods, namely, allowing the highlighting of regions of interest in different medical image sets and reproducing the subtle differences in values. More generally, this can be performed for volume rendering using a transfer function specific to each image set.

[0091] In an example, for the second and third assignment methods, the combination can be based on the following formula:

[0092]

[0093] where V fusion is a single image set, V k is the medical image set labeled k in the label set {0, …, #series - 1}, V0 is the reference medical image set with regions of interest from other medical image sets added, is the corresponding final binary segmentation mask of the region of interest of the medical image set labeled k in the label set {0, …, #series - 1}, δ distinctive is a parameter, and kt is the offset obtained by multiplying the value of label k by the parameter t. For the second assignment method, the parameter δ distinctive can be equal to 0 to stay within the original intensity value range. For the third assignment method, the parameter δ distinctive can be equal to 1 such that each region of interest is associated with a new and different intensity value range.

[0094] In an example, the method can further include, before extraction, aligning at least two medical image sets. In this case, the extraction can be performed on the at least two aligned medical image sets. The at least two medical image sets can be aligned in any way. For example, the at least two medical image sets can be registered together by applying a linear transformation (translation, rotation) or a non - linear transformation. The purpose of the alignment step can be to find a transformation that minimizes the gap between the at least two medical image sets. The alignment contributes to providing an accurate single image set because these medical image sets may be taken at different time points and the patient may have moved (at least involuntarily) between these different time points.

[0095] In an example, the method can be performed dynamically. For example, a set of medical images can be acquired continuously, and each time a new set of medical images is acquired, the method can include updating a single image set to incorporate the newly acquired set of medical images. The update can include extracting one or more corresponding regions of interest from the newly acquired set of medical images and combining these one or more corresponding regions of interest with the regions of interest that have already been combined. This allows for real-time analysis of the resulting images.

[0096] Reference Figures 2 to 12 , an example implementation of the method will now be described.

[0097] If the density values of different tissues are the same, the appearance of the different tissues in a stereoscopic rendering will also be the same. This makes it difficult to highlight a particular organ or tissue in different ways. For example, the highlighted regions of a contrast agent and bone have the same density value. A multi-phase CT-scan examination outputs a number of image series, each of which contains different meaningful visual information. Using existing methods, it is not possible to view all of this information in a single visualization. The method allows for the fusion of a number of CT-scans into a single 3D rendering to avoid the visualization of multiple 3D renderings (one for each phase).

[0098] The method is aimed at performing CT-scan fusion so as to be able to visualize different tissues from multi-phase CT scans in a single view. The method is aimed at highlighting these tissues without highlighting tissues having density values similar to the regions of interest, such as bone. It is able to visualize the relative positions of each structure. Figure 2 An example of a single image set generated by the method is shown. The figure shows a first set of medical images 101 and a second set of medical images 102 considered in this example. In this example, the first set of medical images 101 and the second set of medical images 102 are acquired by a CT-scanner. In particular, the first set of medical images 101 is acquired in the arterial phase, so the arteries are visible in the first set of medical images 101. The second set of medical images 102 is acquired in the late phase, so the urinary tract is visible. The figure shows a single image set 103 generated by the method and formed by the fusion of the first set of medical images 101 and the second set of medical images 102. The figure shows that the combination performed by the method is particularly accurate because both the arteries and the urinary tract are visible. In this example, the method uses a second assignment method (i.e., only including minor differences in values).

[0099] Different regions of interest are highlighted in different ways. This helps with surgical preparation and communication with the patient. It is worth noting that this method overcomes the limitations of known solutions. In particular, this method allows for viewing of all details of a CT-scan. For example, this method can see small blood vessels or detect small changes in blood vessels (such as thrombosis or blood clots). This method also allows for performing stereoscopic rendering, enabling visualization of details of the CT-scan and information from different phases.

[0100] An explanation of the terms used is now provided.

[0101] A CT scan (Computed Tomography) is a specific type of medical image produced by passing X-rays through the human body. The signals are read and analyzed to reconstruct a dense volume of the body.

[0102] The Hounsfield unit (HU) is a relative quantitative measure of radiodensity used by radiologists when interpreting Computed Tomography (CT) images (see, for example, https: / / www.ncbi.nlm.nih.gov / books / NBK547721 / ). Figure 3 An example of the Hounsfield scale for a CT scan is shown. It gives an example of the range corresponding to each tissue (bone: 400 - 1000 HU, soft tissue: 40 - 80 HU, …).

[0103] A medical image set can be a group of CT scan images acquired simultaneously in a single acquisition. They can define a voxel grid of HU values representing the patient's anatomy.

[0104] A multi-phase CT scan is a CT-scan acquired at different times during the same medical examination. The two acquisitions can be separated by a few seconds. Each acquisition enhances one or more corresponding regions of interest (also called “ROI”), depending on the localization of the contrast agent in the body at the time of acquisition.

[0105] A contrast agent (also called a contrast reagent or contrast medium) is a substance used to increase the contrast of structures or fluids in the body in medical imaging. Examples of contrast agents include barium, gadolinium, iodine-based. The HU values corresponding to regions with a contrast reagent can take different values depending on the concentration of the contrast reagent. For example, more details are also given in https: / / en.wikipedia.org / wiki / Contrast_agent or https: / / www.radiologyinfo.org / en / info / safety-contrast.

[0106] Registration is the transformation of different images of a scene into the same coordinate system. The transformation can be rigid, affine, homomorphic, or complex large deformation, which uses gradient descent to find the best transformation.

[0107] A Region of Interest (ROI) is an anatomical structure that the method aims to highlight. For example, the region of interest can be an anatomical structure that includes a contrast agent product.

[0108] A transfer function (also known as a Look-Up Table (LUT) or color map) is a function used to map the intensity values of a medical image volume to color and opacity values for volume rendering.

[0109] A region indistinguishable from the ROI (referred to as an indistinguishable region) is a region whose set of intensity values overlaps with the set of intensity values of the ROI.

[0110] Direct volume rendering or volume rendering refers to the process of generating a 2D image from 3D volume data. The direct volume rendering method requires a method for converting the source volume into a 3D mesh of color and opacity values based on a transfer function.

[0111] Ray casting is a baseline direct volume rendering algorithm. It involves casting lines of sight from an observation position into the volume. Color and opacity values are sampled along these rays. The color of each pixel of the resulting 2D image is a blend of the colors of the sampled points on its associated line of sight weighted by the opacity values.

[0112] Binary morphological opening is a mathematical operation used in computer vision and image processing. The opening operation removes small objects from the foreground of an image (see, for example, the details provided at https: / / en.wikipedia.org / wiki / Opening_(morphology)). Binary morphological closing is a mathematical operation used in computer vision and image processing. The closing operation removes small holes (see, for example, the details provided at https: / / en.wikipedia.org / wiki / Closing_(morphology)).

[0113] Figure 4 An example of combining the extracted individual regions of interest into a single image set is shown. In this example, the method includes, S10 obtaining two medical image sets acquired from CT-scans (two CT-scan series): a first set acquired at the arterial phase V0 and a second set acquired at the late phase V1. The figure shows, for each set, the corresponding initial binary segmentation masks generated by the segmentation of the two sets at S21. Specifically, the figure shows that, for each set, the binary of the corresponding initial binary segmentation mask layer corresponds to one of the medical images of the set. Layer 110 corresponds to the first set and layer 120 corresponds to the second set.

[0114] The figure also shows the corresponding regions of interest of the first set extracted by this method 114 and the corresponding regions of interest of the second set 124. Specifically, S20 extraction includes S21 segmenting each set to generate corresponding initial binary segmentation masks 110 and 120, and S22 calculating the intersection 132 between the two corresponding initial binary segmentation masks 100 and 120. Then, for the first set, S20 extraction includes S23 subtracting the obtained common mask 132 from the corresponding initial binary segmentation mask 110 to obtain the corresponding final binary segmentation mask for the corresponding region of interest of the first set 114. Similarly, for the second set. S20 extraction includes subtracting the obtained common mask 132 from the corresponding initial binary segmentation mask 120 to obtain the corresponding final binary segmentation mask for the corresponding region of interest of the second set 124.

[0115] The figure also shows that S30 combines the corresponding regions of interest extracted 114 and 124 into a single image set. Specifically, the figure shows the combined result of layers 110 and 120. The combined result is image 130. In this example, the method further includes combining the obtained common mask 132 with the corresponding regions of interest extracted 114 and 124 together. The corresponding region of interest extracted for the first set 114 contains arteries, the corresponding region of interest extracted for the second set 124 contains the urinary tract, and the common mask 132 contains bones. Therefore, the method allows for a single rendering of all these body parts. In this example, the method uses the first assignment method. For each voxel of the single set 130, the value assigned to the voxel is the corresponding label associated with the medical image set to which the voxel belongs to the corresponding region of interest (e.g., "0" or "1" in this example). The method also includes assigning another label to all voxels belonging to the common mask 132. Then, the method allows for rendering these values by applying a stereoscopic rendering algorithm to highlight the regions of interest of different medical image sets

[0116] Figure 5 Shows an example of the flowchart of the method. By following this flowchart, the method can achieve a single visualization of different-phase CT scans

[0117] The method includes S10 obtaining a number of CT scan medical image sets V with k ∈ {0, …, #series} from different phases k(hereinafter also referred to as the series). In this example, the method takes V0 as the reference series, and the S30 combination includes adding the ROIs (i.e., the corresponding regions of interest) from other series to the reference series V0.

[0118] The method includes registering (i.e., aligning) all these series for comparison. After registration, the method includes calculating a combined series, which is used to generate a volume rendering and will be visualized. Calculating the combined series includes S20 extracting the corresponding regions of interest of each series, and S30 combining the extracted corresponding regions of interest into a single set of medical images.

[0119] Figure 6 An example of a flowchart showing that S20 extracts the corresponding regions of interest of each series, and S30 combines the extracted corresponding regions of interest into a single set of medical images is shown.

[0120] S20 extraction includes S21 segmenting the set to generate an initial binary segmentation mask 210, which contains the ROIs and indistinguishable (i.e., common) regions of each series. To this end, the method includes defining a range of HU values (e.g., HU values above 300 HU) that contain the ROIs. The defined values may depend on the concentration and type of the contrast agent. Within this value range, the method includes capturing the contrast regions (ROIs) and the bones (indistinguishable regions). To obtain smoother results, the method may include applying a regularization filter, such as a binary morphological opening or closing operation, to the result of the S21 segmentation.

[0121] Then, the method includes S22 calculating the intersection between all the previously calculated masks to obtain a mask of the common region (i.e., the indistinguishable region). S22 calculates this intersection based on the following formula:

[0122]

[0123] where M Common is the common mask, and for each set of medical images labeled k in the label set {0,…,#series - 1}, is the corresponding initial binary segmentation mask generated.

[0124] These are the common regions (e.g., bones) within the above - mentioned range of HU values in all the input CT - scan series. Therefore, these common regions do not include the anatomical structures enhanced by the contrast agent, because the enhanced anatomical structures are not the same in a series of multi - phase CT - scan examinations (as shown in the example of Figure 4 ).

[0125] Therefore, the method includes calculating the corresponding final binary segmentation mask The mask contains only the ROI region of each series as the difference between the common mask and the initial binary segmentation mask (ROI + indistinguishable area). The S23 difference is calculated based on the following formula:

[0126]

[0127] Among them, M Common is a common mask, for each medical image set labeled k in the label set {0,…,#series}, is the corresponding initial binary segmentation mask generated, is the corresponding final binary segmentation mask obtained.

[0128] The method comprises defining an offset t, for example t=5000 HU, which is superior to the amplitude of the HU values of all series of the multi-phase examination.

[0129] Finally, the method comprises, S30, creating a fused series V based on the reference series fusion The creation of a fusion series can be based on the following algorithms:

[0130] If the voxel (x,y,z) does not belong to any ROI outside the reference series, that is Then receive the HU value of the corresponding voxel in the reference series: V fusion (x,y,z)=V0(x,y,z).

[0131] Otherwise, if voxel (x,y,z) belongs to a ROI with series k>0, then:

[0132] οV fusion (x,y,z)=V k (x,y,z). In this case, the method involves rendering V using a conventional transfer function fusion This second allocation method is called hybrid mode (see Figure 7 );

[0133] Or, V fusion (x,y,z)=V k (x,y,z)+kt. This enables the generation of stereo renderings where different series of ROIs appear in unique colors due to the adaptive piecewise transfer function (see Figure 8 ). This third allocation method is called the differentiation mode.

[0134] In summary, for discriminative patterns use δ distinctive =1, use delta for blending mode distinctive =0, the S30 combination can be based on the following formula:

[0135]

[0136] wherein, V fusion is a single image set, V k is a medical image set labeled as k in the label set {0, …, #series - 1}, and V0 is a reference medical image set with regions of interest added from other medical image sets. is the corresponding final binary segmentation mask of the regions of interest of the medical image set labeled as k in the label set {0, …, #series - 1}.

[0137] Figure 7 Shows a first example of a single rendering. In this example, the method generates a hybrid fusion stereo rendering of the arterial phase and the late phase. The method ensures the ROIs from both phases. The method allows the rendering of the urinary tract in the late phase and the arterial path in the arterial phase in one visualization.

[0138] Figure 8 Shows a second example of a single rendering. In this example, the method generates a differential fusion stereo rendering of the arterial phase and the late phase. The method allows the ROIs in the late phase to be highlighted in different colors. In this example, the urinary tract 310 is shown in a first color while the rest of the volume is shown in another color. The figure shows that the method can maintain shadows in the urinary tract 310.

[0139] After calculating the fusion series, the method may include defining an appropriate transfer function (see Figure 9 , Figure 10 and Figure 11 for examples), which enhances each ROI extracted from each series. Figure 9 Shows an example of the transfer function that the method can use to render a single image set when the method uses a second assignment method (i.e., the hybrid mode where δ distinctive = 0). Figure 10 and Figure 11 Show examples of the transfer function that the method can use to render a single image set when the method uses a third assignment method (i.e., the differential mode where δ distinctive = 1).

[0140] Then, the method may include visualizing the CT - scan in 3D by using a standard stereo rendering algorithm with an appropriate transfer function, where the transfer function depends on whether the user has selected the hybrid mode or the differential mode.

[0141] Figure 12 Shows an example of the system, where the system is a client computer system, such as the user's workstation.

[0142] The client computer of this example includes a Central Processing Unit (CPU) 1010 connected to an internal communication bus 1000, and a Random-Access Memory (RAM) 1070 also connected to the bus. The client computer also provides a Graphical Processing Unit (GPU) 1110, which is associated with a video random access memory 1100 connected to the bus. The video RAM 1100 is also known as a frame buffer in the art. A mass storage device controller 1020 manages access to a mass storage device such as a hard disk drive 1030. Mass storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, by way of example, including semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks. Any of the above functions can be supplemented or integrated by a specially designed ASIC (Application Specific Integrated Circuit). A network adapter 1050 manages access to a network 1060. The client computer may also include a haptic device 1090, such as a cursor control device, a keyboard, etc. A cursor control device is used in the client computer to allow a user to selectively position a cursor at any desired location on a display 1080. Additionally, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes a plurality of signal generation devices for inputting control signals to the system. Typically, the cursor control device can be a mouse, and the buttons of the mouse are used to generate signals. Alternatively or additionally, the client computer system may include a touch pad and / or a touch screen.

[0143] A computer program may include instructions executable by a computer, which instructions include means for causing the above system to perform the method. The program may be recorded on any data storage medium, including the memory of the system. For example, the program may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations thereof. The program may be implemented as an apparatus, for example a product tangibly embodied in a machine-readable storage device for execution by a programmable processor. Method steps may be performed by a programmable processor that performs an instruction program to perform the functions of the method by operating on input data and generating output. Accordingly, the processor may be programmable and coupled to receive data and instructions from, and to send data and instructions to, a data storage system, at least one input device, and at least one output device. An application program may be implemented in a high-level procedural or object-oriented programming language, or, if desired, in assembly or machine language. In any case, the language may be a compiled or interpreted language. The program may be a complete installation program or an update program. The application of the program on the system will in any case result in the execution of the instructions of the method. Alternatively, the computer program may be stored and executed on a server in a cloud computing environment that communicates with one or more clients via a network. In such a case, the processing unit executes the instructions contained in the program, thereby causing the method to be performed on the cloud computing environment.

Claims

1. A computer-implemented method for performing a single rendering on at least two medical image sets of a patient, the at least two medical image sets covering regions of the patient, the method comprising: - S10 Obtaining the at least two medical image sets, each medical image set covering one or more corresponding regions of interest; - S20 Extracting the one or more corresponding regions of interest from each of the at least two medical image sets obtained; and - S30 Combining the extracted corresponding regions of interest into a single image set.

2. The method according to claim 1, wherein The extraction in S20 includes: - S21 Segmenting the at least two medical image sets obtained to generate corresponding initial binary segmentation masks for each medical image set; - S22 Calculating the intersection between all the generated initial binary segmentation masks to obtain a common mask for one or more common regions of the at least two medical image sets obtained; - S23 For each medical image set, subtracting the obtained common mask from the corresponding initial binary segmentation mask generated for the medical image set to obtain a corresponding final binary segmentation mask for the one or more corresponding regions of interest of the medical image set.

3. The method according to claim 2, characterized in that, The calculation of the intersection in S22 is based on the following formula: where M Common is the common mask, and for each medical image set labeled k in the label set {0, …, #series-1}, is the corresponding generated initial binary segmentation mask.

4. The method according to claim 2 or 3, characterized in that, The subtraction in S23 is based on the following formula: where M Common is the common mask, and for each medical image set labeled k in the label set {0, …, #series-1}, is the corresponding generated initial binary segmentation mask, is the corresponding obtained final binary segmentation mask.

5. The method according to any one of claims 1-4, characterized in that, The single image set defines a voxel grid, and the combination in S30 includes, for each voxel of the voxel grid, assigning a corresponding value by performing the following operations on each voxel: - Determining whether the voxel belongs to one of the corresponding regions of interest of one of the at least two medical image sets; and - If the voxel belongs to one of the corresponding regions of interest, assigning a corresponding value to the voxel according to the medical image set, wherein a corresponding mask having the corresponding region of interest to which the voxel belongs is obtained from the medical image set.

6. The method according to claim 5, wherein The obtaining of the at least two medical image sets further includes obtaining a tag set, each tag corresponding to a corresponding medical image set respectively; wherein, the assigning of the corresponding value includes assigning a corresponding tag associated with the medical image set having the corresponding region of interest to which the voxel belongs.

7. The method according to claim 5, wherein The assigning of the corresponding value includes assigning a corresponding value equal to the value at the corresponding position of the medical image set having the corresponding region of interest to which the voxel belongs.

8. The method according to claim 5, wherein The corresponding value assigned to the voxel is equal to the result of the following addition: - The value at the corresponding position of the medical image set having the corresponding region of interest to which the voxel belongs; - An offset depending on the medical image set having the corresponding region of interest to which the voxel belongs.

9. The method according to claim 7 or 8, characterized in that The combination is based on the following formula: where, V fusion is a single image set, V k is the medical image set labeled as k in the label set {0, …, #series-1}, V0 is the reference medical image set with regions of interest added from other medical image sets, 1 is the indicator function, is the corresponding final binary segmentation mask of the region of interest of the medical image set labeled as k in the label set {0, …, #series-1}, δ distinctive is the parameter when the discrimination mode is equal to 1 and the mixing mode is equal to 0, and kt is the offset obtained by multiplying the value of label k by parameter t.

10. The method according to any one of claims 1-9, characterized in that, Before the extraction: - Aligning the at least two medical image sets, and the extraction is performed on the aligned at least two medical image sets.

11. The method according to any one of claims 1-10, characterized in that, The at least two medical image sets include: - Preferably, at least one medical image set is acquired by a CT scanner during the non-enhanced phase; - Preferably, at least one medical image set is acquired by the CT scanner during the arterial phase, the arterial phase being preferably 25 to 50 seconds after injection of a contrast agent, preferably 35 to 40 seconds after injection of the contrast agent; and / or -Preferably, at least one set of medical images is acquired by the CT scanner during the portal venous phase, which is preferably 60 to 90 seconds, preferably 70 to 80 seconds after injection of the contrast agent.

12. A computer program product comprising instructions for performing the method according to any one of claims 1-11.

13. A computer-readable storage medium, characterized in that, A computer program product according to claim 12 is recorded thereon.

14. A system comprising a processor coupled to a memory, on which is recorded a computer program product according to claim 12.

15. The system according to claim 14, wherein Further comprising: A viewer, the viewer comprising a graphical user interface for displaying the single set of images.

Images