Method for visualizing at least a zone of an object in at least one interface - Patents.com

Abstract

The invention relates to a method implemented by computer means for visualizing at least a zone of an object in at least one interface, the method comprising the following steps: obtaining at least one image of the zone, the image comprising at least one channel, the image being a two-dimensional or three-dimensional image comprising pixels or voxels, a value being associated with each channel of each pixel or voxel of the image, a representation of the image being displayed in the interface, obtaining at least one annotation from a user, the annotation defining a group of selected pixels or voxels of the image, calculating a transfer function based on the selected pixels or voxels and applying the transfer function to the values ​​of each channel of the image, and updating a representation of the image in the interface, the color and transparency of the pixels or voxels of the representation depending on the transfer function.

Description

[Technical field]

[0001] The present invention relates to a method for visualizing at least a zone of an object, more particularly for visualizing at least a zone of a patient, in at least one interface. [Background technology]

[0002] Medicine makes use of medical imaging techniques to generate pictures of the inside of a patient's body, as well as visual representations of some organs or tissues, for clinical analysis and intervention. Medical imaging aims to reveal internal structures of the body that are hidden, for example, by skin or bone, in order to diagnose and treat disease or to prepare for some surgery.

[0003] For example, medical images produced by computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET) or ultrasound can be two-dimensional (2D) or three-dimensional (3D) images. 3D images can be calculated from a stack of 2D images or can be obtained directly.

[0004] Each image includes pixels (for 2D images) or voxels (for 3D images), and a value is associated with each channel of each pixel or voxel of the image. For example, in the case of a monochromatic image, the image may include only one channel. In contrast, in the case of a color image, the image may include multiple channels, for example, one for each primary color (red, green, blue). The terms voxel and pixel may be used for either 2D or 3D images, as an extension or analogy.

[0005] 2D medical images are often slice-based representations that enable radiologists to perform diagnoses, measure structures of interest, and evaluate treatment strategies. However, surgeons benefit from detailed, life-size representations of the patient as an "avatar" or "digital twin" in natural 3D viewing situations, such as those realized by virtual and augmented reality immersive visualization techniques.

[0006] Volume rendering of 3D medical images is often used in several healthcare contexts, but the quality of the rendered image as well as the contrast between the anatomical structures of the subject strongly depends on the type of transfer function applied to the image.

[0007] A transfer function applied to an image is intended to modify the representation or rendering of that image by altering the color (i.e., emission properties) and transparency (also called opacity, i.e., absorption properties) of the pixels or voxels of that representation. Summary of the Invention [Problem to be solved by the invention]

[0008] Conventional software allows a user to apply predefined transfer functions that impart optical properties to pixels or voxels, but such predefined transfer functions are not always applied to the corresponding analysis.

[0009] Manual tuning of some hyperparameters of the transfer function is sometimes allowed by the software, but the manual design of the transfer function can be a tedious and time-consuming task and does not always result in the best optimization of the transfer function for the case of interest.

[0010] There is also a need to facilitate communication between medical professionals who use different interfaces: in particular, radiologists are trained to use a 2D viewing interface, while surgeons use a 3D interface that accurately resembles the patient in the operating room. Such interaction can benefit from the expertise of both to facilitate the diagnosis of a disease or the preparation of a surgery. [Means for solving the problem]

[0011] For these purposes, the invention proposes a method implemented by computer means for visualizing at least a zone of an object in at least one interface, said method comprising the following steps: - acquiring at least one image of said zone, said image comprising at least one channel, said image being a two-dimensional or three-dimensional image comprising pixels or voxels, a value being associated with each channel of each pixel or voxel of said image, and a representation of said image being displayed on an interface; - obtaining at least one annotation from a user, the annotation defining a selected group of pixels or voxels of the image; - calculating a transfer function based on the selected pixels or voxels and applying said transfer function to the values ​​of each channel of the image; - updating said representation of the image in an interface, the color and transparency of the pixels or voxels of said representation depending on the transfer function; Includes.

[0012] The invention is applicable to medical imaging as well as other types of imaging.The invention is not limited to visualizing at least a zone of a patient's body.

[0013] Contrary to the prior art, in the present invention the transfer function is not predefined or modified only by manually fine-tuning its hyper-parameters. In fact, in the present invention, annotations of the image are used to apply the transfer function and thus obtain the resulting representation of the image. Such a process is called semi-automatic definition or parameterization of the transfer function.

[0014] Such ergonomic and dynamic modification of the transfer function is particularly effective for quickly and optimally modifying the transfer function based on image content.

[0015] The interface may be a graphical user interface (GUI) and a user may interact with the interface through at least one controller, such as, for example, a mouse, a keyboard, and / or a virtual reality controller.

[0016] The method comprises the following steps: - acquiring at least one two-dimensional or 2D image of the zone, the 2D image comprising pixels and at least one channel, a value being associated with each channel of each pixel of the 2D image, and a representation of the 2D image being displayed on a first interface; - acquiring at least one three-dimensional or 3D image of the zone, the 3D image comprising voxels and at least one channel, a value being associated with each channel of each voxel of the 3D image, at least some of the voxels of the 3D image corresponding to some pixels of the 2D image, and a representation of the 3D image being displayed on a second interface; - obtaining at least one annotation from a user, the annotation defining a selected group of pixels of the 2D image or a selected group of voxels of the 3D image; - propagating a selection of said group of pixels or voxels selected in the 2D or 3D image to a 3D or 2D image, respectively, by selecting voxels or pixels of said 3D or 2D image that correspond to the selected pixels or voxels of said 2D or 3D image, respectively; - calculating a first transfer function based on the selected pixels of the 2D image and applying the first transfer function to values ​​of each channel of the 2D image; - updating a representation of the 2D image in a first interface, the color and transparency of the pixels of said representation being dependent on the first transfer function; - calculating a second transfer function based on the selected voxels of the 3D image and applying the second transfer function to values ​​of each channel of the 3D image; - updating a representation of the 3D image in a second interface, the color and transparency of the voxels of said representation being dependent on the second transfer function; may include.

[0017] According to these features, the first interface and the representation of the 2D image (or the 2D representation) can be adapted to the needs of a first user, e.g. a radiologist, and the second interface and the representation of the 3D image (or the 3D representation) can be adapted to the needs of a second user, e.g. a surgeon.

[0018] In this way, an element to be detected, which may possibly not be seen in one of the 2D or 3D representations, will have a better chance of being detected on the other representation. This double verification using both the 2D and 3D interfaces thus improves detection. Of course, both the first and second interfaces can be used by the same user to visualize and interact with both the 2D and 3D representations.

[0019] The propagation of selected groups of pixels or voxels from one image to the other allows for the dynamic and simultaneous application of the first and second transfer functions. In other words, annotations on one interface have an impact on the representation of images on both interfaces.

[0020] Furthermore, according to the above mentioned features, annotations (e.g. regions of interest) are transferred from one interface to the other. However, the transfer function is not transferred, which is recalculated for each interface with each annotation of the user. This makes it possible to obtain the best transfer function for each of the 2D and 3D interfaces in an interactive and dynamic way.

[0021] Such behavior facilitates interaction between interfaces, for example between two users.

[0022] A selected group of pixels or voxels of the corresponding image may be updated by obtaining at least one additional annotation from a user through at least one interface.

[0023] In other words, updating the representation of the image on both interfaces can be an iterative process where each successive annotation has an impact on the representation.

[0024] A group of pixels may be selected in the 2D interface or the first interface, and a group of voxels may be selected in the 3D interface or the second interface, and a union of corresponding pixels and voxels selected in both interfaces may be used to calculate corresponding first and second transfer functions.

[0025] In this situation, if the initial transfer function parameterization is not optimal, the annotation position can be readjusted in either the first or second interface, and the transfer function reparameterization allows convergence towards an optimal image representation.

[0026] The annotation may involve adding or removing pixels or voxels to or from a selected group of pixels or voxels.

[0027] More specifically, some pixels or voxels may be added through annotation in one interface, while some previously added pixels or voxels may be removed through annotation in the other interface.

[0028] At least one of the transfer functions may be calculated according to the following steps. - selecting a first region A of the object and a second region B of the object, each region comprising a group of pixels or voxels based on the selected regions; - creating a first feature tensor and a second feature tensor based on pixels or voxels of a first region A of the object and a second region B of the object, respectively; - defining a statistical test that will distinguish the first region A from the second region B through an optimal maximum mean discrepancy (MMD) of statistics of features associated with the first region A and the second region B; For each pixel or voxel of the image, the color C of said pixel or voxel is calculated according to the following formula:

[0029]

number

[0030] The step defined in where f * (v) is the witness function of the value of pixel or voxel v defined by:

[0031]

number

[0032] where k is the pixel or voxel x that belongs to region A. i and y j is a kernel that defines a value that represents the distance between the feature associated with A and the feature associated with pixel or voxel v, where m is the number of pixels or voxels in A and n is the number of pixels or voxels in region B. - for each pixel or voxel of the image, defining the transparency T of said pixel or voxel by the following formula:

[0033]

number

[0034] Where: h A(v)=(k*ρ)(v) is the smoothed density of a feature associated with voxel v of the first region A with kernel k, the convolution of the kernel with the density of the feature; h B (v)=(k*ρ)(v) is the smoothed density of a feature associated with voxel v of the first region A with kernel k, the convolution of the kernel with the density of the feature; Z A,B is c≦1 and max(Z A,B is a normalization constant that ensures that σ=c, a predetermined constant that defines the maximum transparency factor.

[0035] The method for calculating the transfer function requires little computational effort and is suitable for many applications.

[0036] At least one of the transfer functions may be calculated according to the following steps. - selecting a first region A of the object and a second region B of the object, each region comprising a group of pixels or voxels based on the selected regions; - creating a first feature tensor and a second feature tensor based on a first region A of the object and a second region B of the object, respectively; -

[0037]

number

[0038] and 2n A ≦n max Sampling pixels or voxels in a first region A of the object and a second region of the object such that n A is the number of sample pixels or voxels in the first region A, and n B is the number of sample pixels or voxels in the second region B, and n max is a predetermined value. For each pixel or voxel of the image, the normalized product of the shifted probabilities for the pixel or voxel v is that the pixel or voxel belongs to region A due to the value of one feature, g(v), of the pixel or voxel v, f * (v)=(βp A (v)+1)g(v) is used to calculate the color C of the pixel or voxel,

[0039]

number

[0040] where β is a predetermined constant; For each pixel or voxel of the image, the transparency T of said pixel or voxel is calculated according to the following formula:

[0041]

number

[0042] Step defined in Where: h A (v)=(k*ρ)(v) is the smoothed density of a feature associated with voxel v of the first region A with kernel k, the convolution of the kernel with the density of the feature; h B (v)=(k*ρ)(v) is the smoothed density of a feature associated with voxel v of the first region A with kernel k, the convolution of the kernel with the density of the feature; Z A,B is c≦1 and max(Z A,B is a normalization constant that ensures that σ=c, a predetermined constant that defines the maximum transparency factor.

[0043] The method for calculating the transfer function provides capabilities that enable a user to better identify parts of an object in a representation, even when such identification for a 2D or 3D image appears particularly difficult or ambiguous.

[0044] A first region of interest may be defined as a group of pixels or voxels selected based on the annotation.

[0045] A second region of interest may be automatically defined as another group of pixels or voxels that does not include pixels or voxels from the group of pixels or voxels selected based on the annotation.

[0046] Alternatively, the second region of interest may be defined as another group of pixels or voxels selected by the user based on complementary annotations.

[0047] Each feature tensor may define, for each pixel of the corresponding region of interest, at least one feature value selected from the list of features below: The value of the pixel or voxel v ∇ l v is the normalized gradient (over scale l) of the pixel or voxel value, and the normalization is the Gaussian convolution ∇ l v = ∇(N l *I), where N is the standard deviation of the Gaussian and l of the null mean, and I is the image stack. ·S l (v) is the entropy of a patch of size l around pixel or voxel v ·

[0048]

number

[0049] is the difference of the convolved image at pixel or voxel v where (l1,l2) are the two scales associated with the Gaussian. I is the image stack σ l (v) is the standard deviation of a patch of size l centered at pixel or voxel v ·KLl,m (v) is the Kullback-Leibler distance between a patch of size l centered at pixel or voxel v and surrounding patches of size l+m ·

[0050]

number

[0051] , the median value of voxels in a patch of size l centered at pixel or voxel v ∇log(N l *I) is the logarithmic derivative of the convolved image at pixel or voxel v d p-UMAP,l,m (v) is the low-dimensional Euclidean distance in the latent space generated by parametric UMAP for a patch of size l centered at pixel or voxel v and a patch around size l+m. This feature is used in professional solutions where a database of medical images is collected. Parametric UMAP is trained on a predefined set of patch sizes on 2D image slices in the three principal axes (sagittal, coronal, and axial). Training is done to reduce the dimensions of the patches to the 2D latent space. At evaluation time, the distances l and l+m are approximated as the closest values ​​used in training. (r,θ) p-UMAP (v) is the polar coordinate of pixel or voxel v in the latent space generated by the parametric UMAP centered on pixel or voxel v ·S p-UMAP,l (v) is the convex hull of a region of size l around pixel v

[0052] The kernel k may be selected from the following list of kernels:

[0053]

number

[0054] Where: - x and x' are the features of the corresponding pixel or voxel, vectors of size up to 8, - {σ,l,p,σ b ,σ v ,α,γ} are the hyperparameters of the kernel, which may be predefined or set automatically or by the user.

[0055] The above-mentioned hyperparameters may be modified or set by at least one user through at least one interface.

[0056] According to the present invention, the same medical image or medical image series may be used to acquire slice-based 2D and 3D images.

[0057] The medical image series can be a Digital Imaging and Communications in Medicine (DICOM) image series.

[0058] A representation of the 3D image can be obtained through volume ray casting techniques.

[0059] The first interface may be displayed on a computer screen.

[0060] The second interface may be displayed on a computer screen and / or on a display of a virtual reality (VR) device, which may include a stereoscopic motion-tracked head-mounted display and a VR controller. Such a VR device allows for natural 3D visualization and interaction with the representation of 3D images.

[0061] The semi-automatic transfer function definition or parameterization process described above is compatible with further manual options. The user may still interact with the transfer function and / or add or remove parts of the transfer function. The user may change the shape of the transfer function in the feature space of the transfer function and may substitute the function optimized by a custom function. All parameters associated with the optimized transfer function may be accessible.

[0062] In this manual situation, the user may manually determine the transparency and color properties of the transfer function, for example, in a dedicated interface or in a dedicated area of ​​the first or second interface, which interface or area consists of an interactive absorption curve and a corresponding color mapping.

[0063] The simplest form of transfer function is a one-dimensional version that relates opacity and color based on pixel intensity value. What is important about this transfer function is the smoothness of the opacity and color changes between pixel intensity values. As a result, one-dimensional splines or Gaussian profiles can be used to transition between the colors and various functions to define the opacity.

[0064] A 1D interface or area (where D refers to the dimension of the transfer function) may allow the user to construct transfer functions using either a controlled spline or a set of pre-defined template functions including Gaussian, asymmetric Gaussian, log-normal, Gaussian blend, etc. Transparency and color may be rendered independently.

[0065] While adequate in many cases, such a simple one-dimensional description of the transfer function can lead to problems, especially in highly heterogeneous medical data: in effect, a one-dimensional transfer function such as that described above cannot distinguish between different portions of the volumetric data if they happen to have the same raw pixel or voxel intensity values.

[0066] For this reason, two-dimensional transfer functions can be used, which take into account both the data values ​​(pixel or voxel values) and the gradient magnitude. This approach makes it possible to see interfaces between parts of a volume (e.g., between bone and tissue) as well as to better distinguish homogeneous regions.

[0067] The most common forms of 2D transfer functions are those that relate pixel or voxel intensity to the magnitude of the gradient (i.e., the first derivative of the pixel or voxel intensity) or to the Laplacian (i.e., the second derivative of the pixel or voxel intensity). For example, in the case of CT images, these types of transfer functions are particularly useful for separating tissues, such as skin and bone. The interfaces between homogeneous regions in this type of transfer function appear as circular arcs in the corresponding 2D histogram. This allows for a more complete separation (but not segmentation) of structures in the image, since the gradient or Laplacian are additional parameters to the pixel or voxel intensity to optimize the representation.

[0068] 2D interfaces or regions may allow for the construction of transfer functions in a similar manner to 1D interfaces or regions, but with more complex dictionaries. Interfaces or regions may allow the combined use of shapes, functions, and geometric limits to design the transfer function.

[0069] According to an embodiment of the present invention, synchronization between a first interface and a second interface may be obtained through at least one manager including a set of procedures, classes, and messages that generate a program that interfaces with and synchronizes with the interfaces and stores variables that control visualization-related information (e.g., annotations, transfer functions, image transformations) between the user interfaces.

[0070] The manager may include a data store that holds information in a universal format that is readable by all user interfaces.

[0071] There may be a separate manager for each type of visualization-related information, with each manager implementing at least one or each of the following general functions: - export data from the manager data store to a format readable outside the application (e.g. CSV text, PNG images, STL meshes); - Import data (e.g. CSV text, PNG images, STL meshes) from outside the application into the manager data store, - receiving data from at least one interface, storing said data in a manager data store and updating all other interfaces with a visual readout containing this data; - removing the data from at least one interface, removing the data from the manager data store and updating all other interfaces with a visual readout excluding this data; - Updating the visual readout of a given interface adding new visual elements (e.g. landmarks, ruler measurements, volume measurements), - Removing visual readouts of a given interface that have removed existing visual elements (e.g. landmarks, ruler measurements, volume measurements).

[0072] Each data element that is stored (eg, landmarks, ruler measurements, volume measurements) may have a unique identifier associated with it that is used to ensure synchronization between each interface.

[0073] Data can be input by the user using a mouse or a VR controller in a corresponding 2D or 3D interface. Position information as well as button presses on either the mouse or the VR controller can be received by an input handler that translates them into actions. For example, a user click on a particular area in a 2D image slice can result in an action defined by the creation of a landmark. A user annotating with the mouse can result in the creation of both an annotation in the 2D representation and, through propagation, a corresponding annotation in the 3D representation for annotation creation on the VR controller, and vice versa.

[0074] All actions in one interface may have their corresponding actions in the other. Actions may include image transformations such as region cropping, region removal, orientation adjustment, and scaling or zooming. Other types of actions may be the addition of annotations to an image such as landmarks, ruler measurements, and pixel-by-pixel tagging. Similarly, the removal of annotations may constitute an action. Actions may correspond to adjustments to transfer functions that control the visual properties of a 2D or 3D representation, that is, adjusting the brightness, color, and transparency of an image as a function of features associated with a pixel or voxel.

[0075] After its creation, a visual readout of the action may be reflected in each interface in updating the interface. For example, an action in a first interface that creates a landmark may render a graphical representation of the landmark in the first interface.

[0076] The interface updates can be followed by the occurrence of events via event triggers that notify the manager of user actions.

[0077] The interfaces can be fully synchronized to ensure data consistency and streamlined communication between both interfaces and their respective users. The means of doing so is through the use of a manager, which can have two main functions: The manager can connect both interfaces via event firing to ensure that actions in one interface are visible in the other. Creation of an action can result in an event in each interface that notifies the manager and then processes it. The manager can then receive event arguments from each interface and perform two operations in turn. The manager can first save the information stored in the arguments in a centralized internal data store in a format readable by both interface types. For example, this data can be the position of an annotation or an image transformation. As a second step, the manager can raise an event in the other interface and send the associated arguments. These arguments can be received by the interface update unit and directly translated into the appropriate graphical elements.

[0078] As mentioned above, a manager may be associated with each type of annotation: for example, a manager may handle adding and removing landmarks, and a separate manager may handle changing transfer functions.

[0079] Such a manager is the best vehicle for data consistency, integrity, synchronization, and security across multiple user interfaces.

[0080] The fact of having all the relevant visual information from different user interfaces stored in a centralized location and in a universal format eliminates conflicts between the data types present in each interface.

[0081] The effect of an action in one user interface (adding, deleting, transforming data) is immediately reflected in every other interface.

[0082] Data export is always consistent because annotations from all user interfaces are stored in one centralized location.

[0083] Functions that require access to multiple data types (e.g., transfer functions that require access to tagged pixel or voxel information) can do so directly by accessing the associated manager type.

[0084] The radiologist can make annotations in the first interface (the user interface the radiologist is familiar with) and the exact same annotations will appear in the second interface, represented in three dimensions, as appreciated by the surgeon, so that there is no ambiguity in the visualized annotations. In addition, the surgeon can make annotations in the second interface and the same annotations will appear in the first interface. User navigation of the data in the first interface can be visualized in the second interface in the form of three moving 2D planes. The position of the mouse in the first interface can be shown in the second interface as a highlighted sphere.

[0085] User navigation of the data in the 3D interface can be visualized in the 2D interface in the form of autonomous movements in the sagittal, axial, and coronal axial planes. The position of the VR controller can be indicated in the form of a moving colored circle.

[0086] More generally, the position of the mouse or the position of the VR controller in one interface can be quickly seen in the other interface.

[0087] The centralized data feature ensures that the data visualized in different user interfaces is the correct data. The manager can be tailored to ensure modular implementation. The manager model can ensure that each visualization related information is accessed in the same interface. The incorporation of new features may not require changes to the manager. This abstraction lends itself to easy implementation of new types of data types that should be synchronized across user interfaces.

[0088] Managers can use event triggers that update all interfaces upon addition or removal of visualization-related data types in real time, ensuring instantaneous visibility of data types across all interfaces.

[0089] Annotations can be information that is overlaid on the visual representation. That is, the same annotations can be represented in both the first and second user interfaces. While the precision of position and orientation is the same, their appearance can be adjusted for each interface. Below are listed some of the annotations that can be added or removed by the user in the corresponding interface.

[0090] A landmark is an annotation placed by a user at a particular pixel or voxel location in an image or representation, which may be represented by a sphere, a flag, or an arrow.

[0091] A ruler annotation is a 2D or 3D distance measurement that is superimposed on an image or representation. It is typically represented by a straight line between two ends that define the measurement.

[0092] A volumetric annotation is a volume measurement within a 3D image or representation. It may be represented by a sphere, ellipse, or other 3D geometric shape. Volumetric annotations may also serve as a means to measure the surface area of ​​a region.

[0093] Tags are annotations that represent individual pixels or voxels in an image or representation, which may be indicated by a semi-transparent color that overlays the original pixel or voxel in the image.

[0094] A bounding box is an annotation used to identify a region or object in an image or representation. It can be indicated by a box-shaped outline in 2D images or representations, and by a 3D box in 3D images or representations.

[0095] The path is a spline interpolated curve defined in either the first or second interface along which automatic display procedures can be created. The path may appear as a colored line on all interfaces.

[0096] Annotations can be used to add positional information inside the volumetric image stack to ensure its proper visualization in 2D and 3D images or representations. Such annotations can include information regarding the number of tumors (e.g., with landmarks), tumor size (e.g., with ruler measurements), procedure volume (e.g., via volume annotations), landmark locations, automatic or semi-automatic segmentation of structures of interest, and segmented objects (surgical instruments, foreign bodies).

[0097] The use of annotations such as landmarks and ruler measurements is a means for radiologists to communicate clinical information contained in medical images to surgeons. Such information can include tumor size, volume, and number, as well as resection margins. Landmarks can also indicate areas of interest, such as candidate blood vessels for oncoplastic surgery, the presence of congenital heart defects, and the exact location of bone fragments.

[0098] Surgeons can annotate medical images to plan surgical maneuvers. For example, the location of incision planes can be determined using rulers and tagging tools. In laparoscopic procedures, path annotation allows intuitive visualization of the surgical path.

[0099] Tagging and bounding box annotations in the 2D representation or image play a pivotal role in defining optimal transfer functions for other interfaces. In effect, as described above, the local information contained in the tagged or bounded regions indicates regions of interest (groups of pixels or voxels) that can be easily incorporated into the transfer function parameterization. Image transformations, as described above, are geometric edits performed on an image.

[0100] Image transformation may include image cropping or deletion.

[0101] Image cropping involves removing the outside of a region in an image. The region to be cropped is typically selected with a common geometric shape, such as a square, cube, or sphere.

[0102] The cropping region may appear in both the first and second interfaces and may be modified from both interfaces. Synchronization of the interfaces allows the cropping region to move simultaneously on all interfaces.

[0103] Image deletion involves removing the interiors of regions in an image. The regions to be deleted may typically be selected by a common geometric shape, such as a square, cube, or sphere.

[0104] Image transformation prepares medical images for unobstructed visualization by an expert. For example, cropping routines can isolate specific regions or organs of interest. Removal routines can enable removal of regions that are not relevant to the patient's condition or pathology.

[0105] Medical images, such as MRI and CT scans, typically capture significant image information that is not related to the patient's condition or pathology. For example, a CT scan of the liver captures signals about the heart, intestines, lungs, and rib cage that may otherwise obscure the desired visualization of the liver. Upon cropping or removing irrelevant portions of the image, unobscured medical visualization may be possible in both the first and second interfaces.

[0106] Furthermore, in many situations, images (more specifically, medical images) are multi-channel. For example, the same zone of the same patient may be imaged multiple times with different acquisition parameters or imaging modalities. Each image of the zone may then constitute a channel of the image. This may be the case of multiple MRI weightings (e.g., T1, T1-injected, T2, etc.) imaged in the same session. It may also correspond to the case of the same patient imaged with MRI and CT scan.

[0107] Multimodal image fusion is the act of correlating the information from these multiple channels.

[0108] Image fusion can be performed using channels, where each channel can be associated with a dedicated transfer function.

[0109] The channel and multiple transfer functions can be handled by a manager.

[0110] All image channels can be visualized independently by activating the individual channels.

[0111] Subsets of the image channels can be visualized together by activating multiple channels.

[0112] The associated transfer function can be applied to the 3D image in a second interface to obtain a 3D representation.

[0113] In the first interface, multiple image channels may be blended, but the present implementation is to display only one channel in the first interface.

[0114] Annotations made in one image channel can be propagated to all channels of the image simultaneously. Annotations in one channel can be used to generate transfer functions in all channels. Transfer functions defined in one channel can be applied to other channels.

[0115] When alignment of the various imaging is not ensured by the recording technique or when the patient's morphology changes between different imaging, the procedure can be updated using landmarks that can be located in the 2D or 3D representation or images in multiple channels, and the channels are realigned from one to the other using such annotations.

[0116] Fusion of multiple images is useful for aggregating together information contained in different image sequences or modalities, where each sequence or modality defines a channel. Matching these types of information together provides an extended picture of the patient's condition.

[0117] An example of multi-sequence would include, in the context of prostate imaging, relating complementary information contained in T1 weighted MRI to that in T2 weighted MRI. In the T2 weighted image, the prostate is typically well defined. In the T1 weighted image, visualization of treatment effects, particularly related to blood vessels, is depicted. Combining these two information types into a single visualization by using the same annotations can provide expanded understanding on the part of the expert when evaluating treatment options.

[0118] An example of a multi-sequence can also include related MRI images of the same patient taken at different times during treatment. For example, a chest MRI before neoadjuvant treatment can be correlated with a post-chemotherapy one to assess tumor response. Correlating these two types of information in the same visualization assists the thoracic cancer surgeon in defining appropriate tumor resection margins.

[0119] An example of multi-acquisition would include the fusion of MRI and CT scan sequences. An application in this context would be to evaluate tumor infiltration in bone tissue. The MRI with tissue contrast enhancement allows visualization of the tumor, while the CT scan allows clear visualization of the bone tissue.

[0120] The invention also relates to a computer program comprising instructions for carrying out at least part of the method described above, when the software is executed by a processor.

[0121] The present invention relates to - input means for receiving at least one image of a zone of the object; - a memory for storing at least instructions of said computer program; - a processor accessing the memory to read said instructions and then executing said method; - interface means for displaying a representation of an image obtained by carrying out said method; The present invention also relates to a computing device comprising:

[0122] The invention also relates to a computer readable non-transitory storage medium in which computer software is registered so as to carry out the above-mentioned method when the computer software is executed by a processor.

[0123] The invention also relates to a method for generating a 3D model of the anatomy of a medical patient, the method comprising: - implementing by computer means a method for visualizing at least a zone of an object as disclosed herein, the object being a patient's anatomical structure represented on a medical 3D image and comprising a zone of medical interest, the medical 3D image being a Magnetic Resonance Imaging (MRI) image, a Computed Tomography (CT) scan image, a Positron Emission Tomography (PET) scan image or a numerically processed ultrasound recording image; - displaying a 3D model of the patient's anatomy, including the zones of medical interest, for example using virtual reality means, preferably a virtual reality headset; Includes.

[0124] In some embodiments of the method for generating a 3D model, a user provides at least one annotation to the medical 3D image, where the at least one annotation selects pixels or voxels in a zone of medical interest to improve visualization of the zone or / and visualization of the boundary of the zone of interest, and / or the at least one annotation selects pixels or voxels outside the zone of medical interest to enable image transformation by cropping or removal of interfering structures such as surrounding tissue, bone, muscle, or blood vessels in the zone of interest.

[0125] In some embodiments the zone of interest is or comprises a medical image of a lesion or group of lesions, a tumor, particularly a cancer, or an anatomical abnormality.

[0126] A user may be a person, particularly a medical professional, trained to visualize, annotate, and / or interact with data gleaned from medical images.

[0127] In some embodiments of the method the 3D model is generated using raw 3D image capture data, or alternatively using segmented 3D image data.

[0128] The invention therefore also relates to a method for analyzing a 3D model obtained from medical 3D images previously acquired of a patient for the need to determine his / her condition, illness or health status, said method comprising: - implementing by computer means a method for visualizing at least a zone of an object as disclosed herein, the object being a patient's anatomical structure represented on a medical 3D image and comprising a zone of medical interest, the medical 3D image being a Magnetic Resonance Imaging (MRI) image, a Computed Tomography (CT) scan image, a Positron Emission Tomography (PET) scan image or a numerically processed ultrasound recording image; - displaying a 3D model of the patient's anatomy, including zones of medical interest; - analyzing the displayed 3D model, in particular visualization, tagging, manipulation and / or measurement metrics based on the imaging data of the 3D model, thereby characterizing the patient's anatomical structures in the zone of medical interest; Includes.

[0129] The present invention also relates to a method of diagnosing or monitoring a condition, disease or health status of a patient, the method comprising: - carrying out a method as disclosed above, - collecting data on measured metrics based on the imaging data, the measurements allowing mapping of morphological, geometrical or positional features of the medical object to a patient's condition, disease or health status; Includes.

[0130] In some embodiments, the collected data is determinative for drawing a conclusion about a patient's condition, disease or health status.

[0131] In some embodiments, the geometrical characteristics include the ratio of the volume of the zone of interest to the volume of the anatomical structure, in particular the lesion or tumor or anatomical abnormality. In some embodiments, the location characteristics of the zone of interest include the relative location of the zone to the contour of the anatomical structure or element that constitutes it.

[0132] In some embodiments, morphological, geometric or location features allow for detection or visualization of multicentric or multifocal zones of interest, particularly multicentric or multifocal lesions.

[0133] In some embodiments, the morphological, geometric, or positional features are evaluated in the natural context of the zone of interest displayed on the 3D model, such as in the presence of represented vessels, muscles, or other tissues, i.e., without preprocessing of the displayed data. In other embodiments, the morphological, geometric, or positional features are evaluated after an image transformation performed prior to displaying the 3D model, such transformation including or consisting of cropping or removal of interfering structures, such as surrounding tissues, bones, muscles, or vessels, in the zone of interest.

[0134] In some embodiments, the method of the invention is for the detection or monitoring of a zone of interest that is a tumor (including a tumor containing multiple lesions) in a patient, the visualized or measured metric being selected in the group of: spatial positioning of the tumor in a specific organ or in a specific part thereof, determination of the number of lesions, location of the tumor relative to the affected anatomical structure or body part, in particular the contour of the affected organ, in particular determination of the tumor borders (margins), determination of the tumor volume or the respective volume ratio of the tumor and the resident volume of the organ that contains it. In a particular embodiment, the tumor is a breast cancer.

[0135] The method of the invention can be used to assist in pre-operative planning of medical care, including to assist in preparation for surgery, to assist in medical decisions related to the patient's state of health, disease. In particular, the method can be used for applications in radiology, surgery-specific topics, medical diagnosis or monitoring of a patient's state or disease. Applications in the field of surgery have been found to improve the accuracy of medical image visualization and analysis, and as a result provide useful data and information for designing or performing a strategy for surgery. Thus, the invention can be applied in the field of cancer treatment, in particular in the diagnosis of cancer or in the planning of oncoplastic surgery, in particular in the field of breast cancer treatment by surgery, but also more broadly in the field of trauma treatment, in particular in the planning of trauma surgery, in particular in the field of treatment of anatomical diseases such as cardiac diseases that affect the anatomy of the heart, in the field of liver diseases. The invention can also be applied in the field of pediatric surgery or orthopedic surgery.

[0136] Other features, details and advantages will be set forth in the following detailed description and figures. [Brief description of the drawings]

[0137] [Figure 1] FIG. 2 illustrates a first example interface showing a 2D representation of a 2D image prior to pixel or voxel selection through annotation. [Diagram 2] FIG. 13 illustrates a second example interface showing a 3D representation of a 3D image prior to pixel or voxel selection through annotation. [Diagram 3] FIG. 2 corresponds to FIG. 1 after updating the 2D representation of the 2D image based on a semi-automatic definition or parameterization of a transfer function by selecting pixels or voxels through annotations. [Figure 4] 3 corresponds to FIG. 2 after updating the 3D representation of the 3D image based on a semi-automatic definition or parameterization of the corresponding transfer function. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0138] 1 and 2 respectively represent a first and a second interface showing a first or 2D representation of a 2D image of a particular zone of a patient and a second or 3D representation of a 3D image of the particular zone. The zone of interest in these figures is a breast cancer tumor located in the right breast of the patient. The tumor is clearly visible in the 2D interface representation, but to visualize the tumor in 3D, an accurate parameterization of the transfer function is required. FIG. 3 and FIG. 4 illustrate the same zone of the patient after updating the transfer function in the interface. A first region A and a second region B of the object are represented in FIG. 3. Letters and lines (hatched and solid) representing the regions A and B of the object are added to FIG. 3.

[0139] As mentioned above, the 2D and 3D images may be acquired through the same volumetric image data, which may be multi-channel. The 2D images may be slice-based images of the volumetric image.

[0140] In the first and / or second interfaces, a first user (e.g., a radiologist) and a second user (e.g., a surgeon) can select groups of pixels or voxels on the 2D and 3D representations through annotations. Such selections can be made and modified by both users, since annotations can be updated, visualized, and modified in both interfaces and representations.

[0141] The transfer function can be calculated for each interface or for just one interface, based on the selection.

[0142] A transfer function is defined to optimize an objective function L. The transfer function is defined as a mapping Φ:V→G, where V is the space of pixel or voxel features, G is the set of function spaces in which the two functions are defined (this naturally depends on the application), T(v) is the transparency function, C(v) is the color mapping function, and v is the feature associated with the pixel or voxel.

[0143] In the embodiments or applications described below, an objective function will define L and a procedure to optimize to obtain (T,C).

[0144] In a first embodiment or application, a statistical test is used to define the transfer function. More specifically, the test relies on the Maximum Mean Discrepancy (MMD).

[0145] The following list of kernels is defined:

[0146]

number

[0147] Where: - x and x' are the features of the corresponding pixels or voxels, - {σ,l,p,σ b ,σ v ,α,γ} are the hyperparameters of the kernel, which may be predefined or set automatically or by the user.

[0148] The above-mentioned hyperparameters may be modified or set by a user through at least one interface.

[0149] Maximum average discrepancy relies on the concept of embedding probability measures in a reproducing kernel Hilbert space (RKHS), which is a generalization of kernel methods that operate on embedding points of the input space as elements in a RKHS.

[0150] Given a probability P defined on a separable topological space ξ and a continuous positive definite real kernel k (H should be the corresponding RKHS), P is P = ∫k(.,x)dP(x) and it is called the kernel mean.

[0151] Based on the embedding on P, we define a distance, the maximum mean discrepancy (MMD), on the space of probability measures as the distance between corresponding mean elements, i.e.

[0152]

number

[0153] Kernel K(x,x')=k G Let (x-x') be h(v), and define the normalized density, the convolution of the histogram of the features.

[0154] While the features can be adapted for a particular application, common features associated with voxels for stack l are: The value of the pixel or voxel v ∇ l v, the normalized gradient (over scale l) of pixel or voxel values. The normalization is a Gaussian with N, the null mean and standard deviation of l, convolved with the Gaussian ∇ l v = ∇(N l *I), where I is the image stack. ·S l (v), the entropy of a patch of size l around pixel or voxel v ·

[0155]

number

[0156] , the difference of the convolved image at pixel or voxel v where (l1,l2) are the two scales associated with the Gaussian. I is the image stack σ l (v), the standard deviation of a patch of size l centered at pixel or voxel v ·KL l,m(v), the Kullback-Leibler distance between a patch of size l centered at pixel or voxel v and surrounding patches of size l+m ·

[0157]

number

[0158] , the median value of voxels in a patch of size l centered at pixel or voxel v ∇log(N l *I) The logarithmic derivative of the convolved image at pixel or voxel v d p-UMAP,l,m (v), the low-dimensional Euclidean distance in the latent space generated by parametric UMAP for a patch of size l centered at pixel or voxel v and a patch around size l+m. This feature is used in professional solutions where a database of medical images is collected. Parametric UMAP is trained on a predefined set of patch sizes on 2D image slices in the three principal axes (sagittal, coronal, and axial). Training is done to reduce the dimensions of the patches to the 2D latent space. At evaluation time, the distances l and l+m are approximated as the closest values ​​used in training. (r,θ) p-UMAP (v), the polar coordinates of pixel or voxel v in the latent space generated by the parametric UMAP centered on pixel or voxel v ·S p-UMAP,l (v), the convex hull of a region of size l around pixel v

[0159] Depending on the expected run time, features can be evaluated on 2D images or slices of interest (those slices where the radiologist decides to annotate the data) or on the full volumetric 3D image. It does not change the following algorithm.

[0160] At least one of the transfer functions is calculated according to the following steps. - selecting a first region A of the object and a second region B of the object, each region including a group of pixels or voxels based on the selected pixels. a first feature tensor v based on pixels or voxels of a first region A of the object and a second region B of the object, respectively; A and the second feature tensor v B Step 2: Create a feature tensor v A or v B defines at least one feature value selected from the list of features mentioned above for each pixel of the corresponding region of interest A or B. If a 16-bit graphics card is used, both feature tensors v A or v B The total number of features in can be up to 4 features. If a 32-bit graphics card is used, both feature tensors v A or v B The total number of features can be up to eight features. v A or v B is (n A ,m A ) and (n B ,m B ), where: n A is the number of pixels or voxels in region A, m A is the number of features for each pixel or voxel of region A, n B is the number of pixels or voxels in region B, m B is the number of features in each pixel or voxel of region B, n A is B may be different from m A is m B may differ from. - the maximum number n of pixels or voxels (A and B) to be considered, depending on machine limitations max Steps to define - selecting one kernel from the list of kernels predefined above and calculating the optimal MMD test to distinguish A from B using the following definition and formula: For P, let f be a witness function. * (v) ∝ μ P -μ Q and empirical feature average

[0161]

number

[0162] has. φ(x)=[…φ i (x)…] is a feature map, φ∈F, k(x,x')=<φ(x),φ(x')> F , and F is a class of functions. - for each pixel or voxel of the image, defining a color C of said pixel or voxel according to the formula: where f * (v) is the witness function of the value of pixel or voxel v, defined by:

[0163]

number

[0164] where k is the pixel or voxel x that belongs to region A. i and y j is a kernel that defines a value that represents the distance between the feature associated with A and the feature associated with pixel or voxel v, where m is the number of pixels or voxels in A and n is the number of pixels or voxels in region B. - calculating the colors C and T defined by the following formulas:

[0165]

number

[0166]

number

[0167] Where: h A (v)=(k*ρ)(v) is the smoothed density of a feature associated with voxel v of the first region A with kernel k, the convolution of the kernel with the density of the feature; h B (v)=(k*ρ)(v) is the smoothed density of a feature associated with voxel v of the first region B with kernel k, the convolution of the kernel with the density of the feature; Z A,B is c≦1 and max(Z A,B is a normalization constant that ensures that σ=c, a predetermined constant that defines the maximum transparency factor.

[0168] In a second embodiment or application, a probability distribution over the features is used to define the transfer function.

[0169] The algorithm defines the color of a pixel or voxel from the product of the feature of interest and the shifted probability of the voxel belonging to one of the structures (A, B), performed by one-time learning.

[0170] Here, the learning is performed only on the data that the radiologists and surgeons are working with, not from a dataset: some features used for one-off learning can be derived from inference trained on a specific dataset.

[0171] The following list of kernels is defined:

[0172]

number

[0173] Define a binary classifier using Gaussian processes. Associate region A with the "label" y=+1, and region B with the "label" y=-1. Define a latent variable f(v) and a logistic function π(v)≡p(y=+1|v)=S(f(v)). S is the logistic function. Define a set V of tagged pixels or voxels (corresponding to region (A,B) in the following).

[0174] pixel or voxel v * Regarding this, the following is true.

[0175]

number

[0176] Where:

[0177]

number

[0178] Where:

[0179] At least one of the transfer functions is calculated according to the following steps. - selecting a first region A of the object and a second region B of the object, each region including a group of pixels or voxels based on the selected pixels. - the maximum number of pixels or voxels n that can be used in a procedure on a computing platform max Steps to define - List of target features {...g i (v) Defining the {...} a first feature tensor v based on pixels or voxels of a first region A of the object and a second region B of the object, respectively; A and the second feature tensor v B Step 2: Create a feature tensor v A or v Bdefines, for each pixel of the corresponding region A or B of interest, at least one feature value selected from the list of features mentioned above for the first embodiment or application. v A or v B is (n A ,m A ) and (n B ,m B ), where: n A is the number of pixels or voxels in region A, m A is the number of features for each pixel or voxel of region A, n B is the number of pixels or voxels in region B, m B is the number of features in each pixel or voxel of region B, n A is B may be different from m A is m B may differ from. - Sample voxels or pixels for (A,B)

[0180]

number

[0181] and

[0182]

number

[0183] Step of making it so that Where:

[0184]

number

[0185] is the number of pixels or voxels sampled in region A,

[0186]

number

[0187] is the number of pixels or voxels sampled in region B - Define the classifier as a binary Gaussian process classifier (P,π). - Step to calculate the Laplace approximation - calculating the colors C and T defined by the following formulas:

[0188]

number

[0189] f * (v)=(βp A When the normalized product of shifted probabilities for a pixel or voxel v to belong to region A due to the value of one feature, g(v), of the pixel or voxel v, we use f*(v)=(βpA(v)+1)g(v), where β is a predetermined constant.

[0190]

number

[0191] Where: h A (v)=(k*ρ)(v) is the smoothed density of a feature associated with voxel v of the first region A with kernel k, the convolution of the kernel with the density of the feature; h B (v)=(k*ρ)(v) is the smoothed density of a feature associated with voxel v of the first region B with kernel k, the convolution of the kernel with the density of the feature; Z A,Bis c≦1 and max(Z A,B is a normalization constant that ensures that σ=c, a predetermined constant that defines the maximum transparency factor.

[0192] In a software implementation, the above procedure is performed for the set of kernels defined above. In at least one interface, the kernel can be changed by the corresponding user. By default, the kernel k G may be displayed.

[0193] Such a Bayesian approach allows for better generalization of results when there is a very small number of tagged pixels or voxels. When there are a large number of tagged pixels or voxels, the MMD (first embodiment or application) will allow for faster computation and greater efficiency than the Bayesian approach (second embodiment or application).

Claims

1. A method implemented by computer means for visualizing at least a zone of an object in at least one interface, comprising the following steps: - acquiring at least one image of said zone, said image comprising at least one channel, said image being a two-dimensional or three-dimensional image comprising pixels or voxels, a value being associated with each channel of each pixel or voxel of said image, and a representation of said image being displayed on said interface; - obtaining at least one annotation from a user, said annotation defining a selected group of pixels or voxels of said image; - calculating a transfer function based on the selected pixels or voxels and applying said transfer function to the values ​​of each channel of the image; - updating the representation of the image in the interface, the color and transparency of the pixels or voxels of the representation being dependent on the transfer function; The method comprises the following steps: - acquiring at least one two-dimensional or 2D image of the zone, the 2D image comprising pixels and at least one channel, a value being associated with each channel of each pixel of the 2D image, and a representation of the 2D image being displayed on a first interface; - acquiring at least one three-dimensional or 3D image of said zone, said 3D image comprising voxels and at least one channel, a value being associated with each channel of each voxel of said 3D image, at least some of said voxels of said 3D image corresponding to some pixels of said 2D image, and a representation of said 3D image being displayed on a second interface; - obtaining at least one annotation from a user, said annotation defining a selected group of pixels of said 2D image or a selected group of voxels of said 3D image; - propagating the selection of the group of pixels or voxels selected in the 2D or 3D image to the 3D or 2D image, respectively, by selecting the voxels or pixels of the 3D or 2D image that correspond to the selected pixels or voxels of the 2D or 3D image, respectively; - calculating a first transfer function based on the selected pixels of the 2D image and applying the first transfer function to the values ​​of each channel of the 2D image; - updating the representation of the 2D image in the first interface, the color and the transparency of the pixels of the representation depending on the first transfer function; - calculating a second transfer function based on the selected voxels of the 3D image and applying the second transfer function to the values ​​of each channel of the 3D image; - updating the representation of the 3D image in the second interface, the color and the transparency of the voxels of the representation being dependent on the second transfer function; A method comprising:

2. The method of claim 1 , wherein the group of selected pixels or voxels of the corresponding image is updated by obtaining at least one additional annotation from a user through the at least one interface.

3. At least one of the transfer functions is determined by the following steps: - selecting a first region A of the object and a second region B of the object, each region comprising a group of pixels or voxels based on the selected pixels; - creating a first feature tensor and a second feature tensor based on pixels or voxels of said first region A of the object and said second region B of the object, respectively; - defining a statistical test that will distinguish the first region A from the second region B through an optimal maximum mean discrepancy (MMD) of statistics of features associated with the first region A and the second region B; for each pixel or voxel of said image, the color C of said pixel or voxel is calculated according to the following formula: [0010] The step of: f * (v) is the witness function of the value of said pixel or voxel v defined by: [0025] k is the pixel or voxel x that belongs to region A i and y j and the feature associated with the pixel or voxel v, m is the number of pixels or voxels in A, and n is the number of pixels or voxels in region B; k is the number of pixels or voxels x i and y j a kernel defining a value representing a distance between the feature associated with v and the feature associated with the pixel or voxel v; for each pixel or voxel of said image, the transparency T of said pixel or voxel is calculated using the following formula: [0030] The step of: h A (v) is the smoothed density of the feature associated with voxel v in the first region A using the kernel k; h B (v) is the smoothed density of the feature associated with voxel v in the second region B using the kernel k; Z A,B is c≦1 and max(Z A,B a normalization constant that ensures that θ = θ, a predetermined constant that defines the maximum transparency factor, and The method according to claim 1 or 2, wherein the calculated value is:

4. At least one of the transfer functions is determined by the following steps: - selecting a first region A of the object and a second region B of the object, each region comprising a group of pixels or voxels based on the selected pixels; - creating a first feature tensor and a second feature tensor based on a first region A of the object and a second region B of the object, respectively; [0045] and 2n A ≦n max Sample pixels or voxels in a first region A of the object and a second region B of the object such that: A is the number of sample pixels or voxels in the first region A, and n B is the number of sample pixels or voxels in the second region B, and n max is a predetermined value; and for each pixel or voxel of the image, the normalized product of the shifted probabilities for the pixel or voxel v is a function of the value of one feature, g(v), of the pixel or voxel v belonging to region A, * (v)=(βp A (v)+1)g(v) is used to find the color C of the pixel or voxel, [0050] Defining for each pixel or voxel of said image, the transparency T of said pixel or voxel is calculated according to the following formula: [006] The step of: -h A (v)=(k * ρ)(v) is the smoothed density of the feature associated with voxel v of the first region A with the kernel k, the convolution of the kernel with the density of the feature; -h B (v)=(k * ρ)(v) is the smoothed density of the feature associated with voxel v of the second region B with the kernel k, the convolution of the kernel with the density of the feature; -Z A,B is c≦1 and max(Z A,B a normalization constant that ensures that θ = θ, a predetermined constant that defines the maximum transparency factor, and The method according to any one of claims 1 to 3, wherein the calculation is performed according to:

5. Each feature tensor contains, for each pixel in the corresponding region of interest, the following list of features: the value of said pixel or voxel v, ・∇ l v is the normalized gradient (over scale l) of the pixel or voxel value, where the normalization is the Gaussian convolution ∇ l v = ∇(N l * I), where N is the standard deviation of the Gaussian and l of the null mean. ・S l (v) is the entropy of a patch of size l around pixel or voxel v, ・ [0070] is the convolved image difference at pixel or voxel v, Here, (l 1 ,l 2 ) are the two scales associated with the Gaussian, I is the image stack, σ l (v) is the standard deviation of the patch of size l centered at pixel or voxel v, KL l,m (v) is the Kullback-Leibler distance between the patch of size l and a surrounding patch of size l+m centered at pixel or voxel v; ・ [0080] is the median value of the voxels in the patch of size l centered at pixel or voxel v, ・∇log(N l * I) is the logarithmic derivative of the convolved image at pixel or voxel v; ・d p-UMAP,l,m (v) is the low-dimensional Euclidean distance in the latent space generated by parametric UMAP for a patch of size l centered at pixel or voxel v and surrounding patches of size l+m. ・(r,θ) p-UMAP (v) is the polar coordinate of the pixel or voxel v in the latent space generated by a parametric UMAP centered on the pixel or voxel v; ・S p-UMAP,l (v) is the convex hull of a region of size l around pixel v The method according to claim 3 or 4, further comprising defining at least one characteristic value selected from:

6. The kernel k may be chosen from the following list of kernels: [0090] is selected from x and x' are features of the corresponding pixels or voxels, {σ,l,p,σ b ,σ v 6. The method according to claim 3, wherein {α,γ} are hyperparameters of the kernel, these parameters being predefined or set automatically or by a user.

7. 7. The method of claim 1, wherein an action generated on one of the first and second devices is transmitted to the other device through at least one manager that stores data in a memory, the data including at least one parameter representing the action, and the first representation and / or the second representation is updated based on the stored data and a set of original images.

8. Each manager has the following generic functions: - export data from the manager data store to a format readable outside the application; - importing data into the manager data store from outside the application; - receiving data from at least one interface, storing said data in said manager data store and updating all other interfaces with a visual readout containing this data; - removing the data from at least one interface, removing said data from said manager data store and updating all other interfaces with a visual readout excluding this data; - updating said visual readout of a given interface adding new visual elements; - Remove the visual readout of a given interface by removing existing visual elements The method of claim 7 , implementing at least one or each of:

9. 9. The method of claim 8, wherein each stored data element has a unique identifier associated with it that is used to ensure synchronization between each interface.

10. The method according to claim 1 , wherein the representation of the 3D image is obtained through a volume ray casting method.

11. 11. The method of claim 1, wherein the first interface is displayed on a computer screen and the second interface is displayed on a computer screen and / or on a display of a virtual reality device.

12. A computer program comprising instructions for carrying out at least part of the method according to any one of claims 1 to 11, when the software is executed by a processor.

13. - input means for receiving at least one image of a zone of the object; - a memory for storing at least instructions of a computer program according to claim 12; a processor for accessing said memory to read said instructions and then executing the method according to any one of claims 1 to 11; - interface means for displaying said representation of said image obtained by performing said method; A computing device comprising:

14. A computer readable non-transitory storage medium, wherein said computer software is registered to perform the method of any one of claims 1 to 11 when said computer software is executed by a processor.

15. 1. A method for generating a 3D model of a patient's anatomy, comprising: - implementing, by computer means, the method according to any one of claims 1 to 11 on a medical 3D image of an object, said object being a patient's anatomy with a zone of medical interest, said medical 3D image being a Magnetic Resonance Imaging (MRI) image, a Computed Tomography (CT) scan image, a Positron Emission Tomography (PET) scan image or a numerically processed ultrasound recording image; - displaying a 3D model of the patient's anatomy including the zone of medical interest; The method includes:

16. 16. The method of claim 15, wherein a user provides at least one annotation to the medical 3D image, the at least one annotation selecting pixels or voxels in the zone of the medical object to improve visualization of the zone or / and visualization of the boundary of the zone of the object, and / or the at least one annotation selecting pixels or voxels outside the zone of the medical object to enable image transformation by cropping or removal of interfering structures such as surrounding tissue, bone, muscle, or blood vessels in the zone of the object.

17. 17. The method of claim 15 or 16, wherein the method is performed on raw 3D image capture data or on segmented 3D image data.

18. 1. A method for analyzing a 3D model obtained from a medical 3D image previously acquired of a patient for the need to determine his / her condition, disease or health status, comprising: - implementing, by computer means, the method according to any one of claims 1 to 11 on a medical 3D image of an object, said object being a patient's anatomy including a zone of medical interest, said medical 3D image being a Magnetic Resonance Imaging (MRI) image, a Computed Tomography (CT) scan image, a Positron Emission Tomography (PET) scan image or a numerically processed ultrasound recording image; - displaying a 3D model of the patient's anatomy including the zone of medical interest; - analyzing the displayed 3D model, in particular visualization, tagging, manipulation and / or measurement metrics based on the imaging data of the 3D model, thereby characterizing the patient's anatomy in the zone of medical interest. The method includes:

19. 1. A method for diagnosing or monitoring a condition, disease or health status in a patient, comprising: - implementing the method according to claim 18; - collecting said data on said metrics measured based on imaging data, said metrics allowing mapping of morphological, geometric or positional features of a medical object to a condition, disease or health status of said patient; The method includes:

20. 20. The method of claim 19, wherein for detection or monitoring of a tumor in a patient, the visualized or measured metric is selected in the group of: localization of the tumor in a particular organ or part thereof, determination of the number of lesions, location of the tumor relative to the affected anatomical structure or body part, in particular the outline of the affected organ, determination of the tumor volume or the respective volume ratio of the tumor and the normal volume of the affected organ.

Images