Deformation of medical images

Abstract

This invention provides a computer implementation method for generating template images of organs. [Solution] The method includes obtaining a dataset containing a set of medical images and an autoencoder pre-trained on the set of medical images. The method includes determining a template image in the latent space by minimizing a loss that is a function of a latent vector, parameters of a differential homeomorphism applied to a template image to obtain a medical image for each medical image in the dataset. The loss includes a first term that penalizes the discrepancy between the posterior probability of the differential homeomorphism parameters and the Gaussian prior probability of the differential homeomorphism parameters, given a medical image. The loss includes a second term that penalizes the negative log-likelihood of the set of medical images in the dataset, given a latent vector and parameters of the differential homeomorphism.

Description

Technical Field

[0001] The present disclosure relates to the field of computer programs and systems, and more specifically to a method, a data structure, a computer-readable storage medium and a system relating to generating an organ template image.

Background Art

[0002] Several widely used medical imaging and scanning techniques, such as computed tomography scans (CT scans), magnetic resonance imaging (MRI), and positron emission tomography (PET), acquire medical images representing an anatomical target (e.g., an organ) of a patient.

[0003] The use of medical images to visualize a patient's anatomical target is common in today's clinical practice, and is useful, for example, for the diagnosis and treatment of diseases. The use of medical images makes it possible to observe specific types of structures in a patient's anatomical target and / or acquire various types of medical information relevant to clinical practice.

[0004] However, in the field of medical image analysis, medical images of an anatomical target can vary greatly between patients. Such variation results from inter-patient differences including anatomical structure, age, sex, and medical condition.

[0005] In this context, there remains a need for improved solutions for processing medical images representing a patient's organ.

Summary of Invention

[0006] Therefore, we provide a computer-implemented method for generating template images of organs, hereafter referred to as the “generation method”. The generation method includes acquiring a dataset containing sets of medical images representing organs of different patients, and acquiring an autoencoder trained on sets of medical images representing organs of different patients, the autoencoder comprising an encoder and a decoder. The generation method includes determining a template image as a result of applying the decoder to a latent vector, the latent vector being defined in latent space by minimizing a loss that is a function of the latent vector and, for each medical image in the dataset, the parameters of a differential homeomorphism applied to the template image to acquire the medical image. The latent vector and the parameters of the differential homeomorphism are variable during minimization. Minimization is performed in latent space. The loss includes a first term that penalizes the discrepancy between the posterior probability of the parameters of the differential homeomorphism applied to the template image to acquire each medical image in the dataset and the Gaussian prior probability of the parameters of the differential homeomorphism applied to the template image to acquire each medical image in the dataset, given each medical image. The loss includes a second term that penalizes the set of medical images in the dataset with the negative log-likelihood, given the parameters of the differential homeomorphism applied to the template image to acquire each medical image in the dataset and the latent vector.

[0007] The generation method may include one or more of the following features: - Determining the template image involves predicting the parameters of the differential homeomorphic deformation applied to the template image and training a neural network to acquire each medical image in the dataset. - The neural network takes the latent vector and the result of applying the encoder to the medical image as input for each medical image. - The first term is the Kullback-Leibler divergence between the posterior probability and the Gaussian prior probability. - Differential in-phase deformation is the result of a power of a smooth static velocity field, which is obtained by computing a smoothing kernel at control points of the medical image, with each control point being associated with the respective parameter of the differential in-phase deformation applied to the medical image, and the power of the smooth static velocity field involves scaling and squarering methods. - The smoothing kernel is a Gaussian kernel. - The Gaussian prior probability is centralized. - Acquiring an autoencoder involves learning an autoencoder, and / or, The medical imaging set includes automated refraction testing, automated refraction testing, angiography, bone densitometry (US), biomagnetic imaging, bone densitometry (X-ray), color flow Doppler, cine fluorescence fluoroscopy, vaginoscopy, computed radiography, cystoscopy, computed tomography, duplex Doppler, digital fluoroscopy, fluoroscopy, digital microscopy, digital subtraction angiography, digital radiography, echocardiography, electrocardiogram, cardiac electrophysiology, endoscopy, fluorescein angiography, fiducial imaging, fundus examination, general microscopy, hardcopy, hemodynamic waveforms, intraoral radiography, intraocular lens data, intravascular optical coherence tomography, intravascular ultrasound, corneal measurement, lentometry, laparoscopy, laser surface scan, magnetic resonance angiography, mammography, magnetic resonance imaging, MR T1-weighted images, and MR T2-weighted imaging, MR proton density-weighted imaging, MR steady-state free precession, MR effective T2, MR susceptibility-weighted imaging, MR short tau inversion recovery imaging, MR fluid-attenuated inversion recovery imaging, MR double inversion recovery imaging, MR conventional diffusion-weighted imaging, MR apparent diffusion coefficient imaging, MR diffusion tensor imaging, MR dynamic susceptibility contrast imaging, MR arterial spin contrast imaging, MR dynamic contrast-weighted imaging, MR blood oxygen-dependent imaging, MR time-of-flight imaging, MR phase contrast imaging, magnetic resonance spectroscopy, nuclear medicine, axial length measurement, optical coherence tomography (non-ophthalmology), ophthalmic photography, ophthalmic mapping, ophthalmic refraction testing, ophthalmic tomography, eye This includes images from any of the following medical imaging diagnostic methods: field of view, optical surface scan, other, positron emission tomography (PET), panoramic X-ray, respiratory waveform, fluoroscopy, radiographic images (conventional film / screen), radiation therapy dose, radiation therapy images, radiation therapy plans, radiation therapy records, radiation therapy structural sets, segmentation, slide microscopy, stereometric measurements, single-photon emission computed tomography (SPECT), automated slide staining devices, thermography, ultrasound, A-mode US, B-mode US, M-mode US, visual acuity, videofluoroscopy, X-ray angiography, and external camera imaging.

[0008] Further, we provide a method for using template images of organs generated according to a generation method implemented on a computer. The method includes obtaining input medical images representing the patient's organs. The method also includes determining the parameters of a differential homeomorphism that, when applied to the template image, becomes the input medical image.

[0009] The method of use may include one or more of the following features: - The diffeomorphism is of the same class as the diffeomorphism deformation of the generation method, and optionally, the diffeomorphism includes parameters predicted by the neural network described above, and / or - The template image includes data representing the medical segmentation of the template image and / or one or more medical annotations on the template image, and the method further includes projecting the data onto an input medical image based on determined parameters of a diffeomorphism.

[0010] The present invention further provides a data structure that includes a computer program which includes (i) instructions that cause the computer to perform a generation method when the program is executed by the computer, and / or (ii) instructions that cause the computer to perform a method for using a template image of an organ when the program is executed by the computer, and / or (iii) a template image generated according to the generation method.

[0011] Further, we provide a computer-readable storage medium on which data structures are recorded.

[0012] The present invention provides a system including a processor coupled to memory, wherein the memory stores data structures. [Brief explanation of the drawing]

[0013] Here, a non-limiting example will be explained with reference to the attached diagram. [Figure 1]FIG. 1 is a flowchart of an example of a method for generating an organ template image. [Figure 2] FIG. 2 is a flowchart of an example of a method of using an organ template image generated according to the method for generating an organ template image. [Figure 3] FIG. 3 is a diagram illustrating an example of a system. [Figure 4] FIG. 4 is a diagram illustrating the method. [Figure 5] FIG. 5 is a diagram illustrating the method. [Figure 6] FIG. 6 is a diagram illustrating the method. [Figure 7] FIG. 7 is a diagram illustrating the method. [Figure 8] FIG. 8 is a diagram illustrating the method. [Figure 9] FIG. 9 is a diagram illustrating the method. [Figure 10] FIG. 10 is a diagram illustrating the method. DETAILED DESCRIPTION OF EMBODIMENTS

[0014] Referring to the flowchart of FIG. 1, a computer-implemented method for generating a template image of an organ (e.g., a human organ such as the heart, liver, brain, lung, etc.) is proposed, which is hereinafter referred to as the "generation method".

[0015] The generation method includes acquiring a dataset D (S100). The dataset includes a set of medical images representing (the same) organ from different patients (e.g., actual medical images obtained from the same medical image diagnostic modality such as CT scan, MRI or PET).

[0016] The generation method includes obtaining an autoencoder A (e.g., a variational autoencoder) that has been trained on a set of actual medical images obtained from the same medical image diagnostic modality (e.g., the same medical image diagnostic modality as the set of medical images in the dataset acquired in S100), said medical images being (for example, CT scans, MRI or PET, etc.) representing an organ (the same organ) of different patients (S100). The autoencoder may optionally be trained on the (same) dataset acquired in S100. The autoencoder comprises an encoder and a decoder. The generation method may optionally include training the autoencoder. Alternatively, the autoencoder has been trained separately, and can be predetermined when the generation method is executed.

[0017] The generation method includes determining a template image as a result of applying a decoder to a latent vector by minimizing a loss (S120). The latent vector is defined in a latent space (i.e., the domain of definition thereof). The loss is a function of (i) the latent vector and, for each medical image in the dataset, (ii) parameters of a diffeomorphic deformation applied to the template image to obtain the medical image. The latent vector and the parameters of the diffeomorphic deformation are variable (i.e., can and actually do change), meaning that their values can (and actually do) change as the minimization is performed. Furthermore, the latent vector is a variable that is directly searched during minimization. In other words, the minimization is performed in the latent space (i.e., the latent space is directly searched or spanned during loss minimization to obtain an appropriate latent vector).

[0018] The loss includes a first term and a second term. Each term corresponds to a respective penalty. The loss may combine the two penalty terms in any manner, for example, by summing the first term and the second term.

[0019] The first term imposes a penalty (e.g., minimizes) for the discrepancy between the posterior probability and the Gaussian prior probability. The posterior probability represents the probability of the parameters of the differential homeomorphism applied to the template image to obtain each medical image in the dataset, given each medical image (i.e., conditioned by the medical images). The Gaussian prior probability represents the probability of the parameters of the differential homeomorphism applied to the template image to obtain each medical image in the dataset.

[0020] The second term imposes a penalty (e.g., minimizes) the negative log-likelihood of the set of medical images in the dataset, given the parameters of the differential homeomorphism applied to the template image to obtain each medical image in the dataset and the latent vector.

[0021] Such generation methods form an improved solution for processing medical images representing a patient's organs.

[0022] In particular, such a generation method determines a template image for an organ when given a set of medical images representing organs of different patients. The template image determined in S120 is, in particular, a representative or reference image of the set of medical images in the dataset. Each medical image in dataset D can, in fact, be obtained by a differential homeomorphic deformation applied to the template image. The distribution of differential homeomorphic deformations around the template image is called an "atlas".

[0023] "Differential homeomorphism" refers to a smooth, reversible transformation that can be applied to a template image to transform it into another medical image, such as any medical image in a dataset or an input medical image representing an organ of a new patient. The smoothness of differential homeomorphism ensures that the transformation does not introduce discontinuities and artifacts (e.g., pseudo-defects) into the transformed medical image. Furthermore, the reversibility of differential homeomorphism ensures that the transformation cannot be undone.

[0024] For each medical image in the dataset, the differential homeomorphism applied to the template image to acquire the medical image can be defined by the parameters of that differential homeomorphism. Therefore, these parameters may follow a distribution.

[0025] For each medical image, the differential homeomorphism applied to the template image to obtain the medical image is defined (belongs to) a class of differential homeomorphisms. The class of differential homeomorphisms explored during loss minimization can be any class of all differential homeomorphisms. Computationally, a class of differential homeomorphisms can be represented by any parameterization and / or approximation (i.e., representative constraint) of any class of all differential homeomorphisms. An approximate class is large enough that the template image can be effectively transformed into each medical image in the dataset with sufficient accuracy (it has at least one differential homeomorphism of the approximate class). "With sufficient accuracy" means that the mismatch between the transformed medical image and the medical image in the dataset, calculated by a particular similarity metric, falls below a given (and sufficiently low) threshold. Approximate classes / constraints may include non-rigid differential homeomorphisms (including, but not limited to, differential rotation, local scaling, local translation, etc.). An approximation class / constraint may include a set of parameters that define a constrained (e.g., approximate or allowed) class of differential homeomorphic deformations. For example, an implementation of the constraint may include differential homeomorphic deformations arising from powers of a smooth stationary velocity field.

[0026] Therefore, the generation method utilizes differential homeomorphism applied to template images to obtain medical images from the dataset, thereby ensuring spatial regularity and consistency of the atlas. In other words, differential homeomorphism provides medically relevant correspondences between template images and other medical images, that is, it correlates anatomical sites of two patients with the same biological function and / or anatomical location. As a result, the template images, along with the atlas, form a reference frame (corresponding to the template patient or reference patient, i.e., the patient with the template image of the organ under consideration) onto which medical images of organs from different actual patients can be projected, and any relevant medical analysis or processing can be performed on the reference patient. By applying the results to actual patients, appropriate medical treatment can be provided to those actual patients.

[0027] A dataset may contain 100, 200, 500, or more than 1000 medical images, each potentially representing the same organ from a different patient. The dataset may include 2D or 3D medical images. The organs of interest may be coronary arteries, heart, brain, liver, lungs, or other anatomical targets. The set of medical images may include images from a single (e.g., the same) medical imaging modality, such as CT scans, MRI, or PET. The medical images in the dataset may be alignable; that is, the generation method may include aligning the medical images in the dataset, meaning they may be pre-processed to have the same dimensions and / or orientation and / or size. This alignment ensures consistency within the dataset, guaranteeing that the medical images in the dataset represent the same anatomical target within the patient's body. Additionally or alternatively, the dataset may include annotations and / or segmentation for each medical image. Additionally or alternatively, the dataset may include patient demographic information (e.g., age, sex, health status, disease status, etc., where applicable) and image parameters for each medical image. Thus, by using a richer dataset that includes ground truth annotation and / or segmentation, the generation method can better learn (recognize) organ regions of interest (e.g., heart valves).

[0028] The generation method includes obtaining an autoencoder A (e.g., a variational autoencoder) that is trained on a set of medical images, each representing an organ of a different patient (S100). In this way, the generation method utilizes the latent representation of the autoencoder.

[0029] As is known in the art, an autoencoder compresses input data (e.g., representing a medical image) into a lower-dimensional latent space using each encoder. The input data is then represented in the latent space by latent vectors (also called latent codes) that capture the key characteristics of the input data in its compressed form. The autoencoder then reconstructs output data (e.g., representing another medical image) from these latent vectors using each decoder.

[0030] An autoencoder (e.g., a variational autoencoder) may include a deep convolutional network in both the encoder and decoder, for example, with 5 to 20 convolutional layers and 0 to 5 interleaved attention layers.

[0031] The presence of an autoencoder allows the generation method to leverage a more compact representation of the latent vector in the latent space and the key properties extracted by the autoencoder (while simultaneously avoiding interference from other properties, i.e., other properties not captured by the latent vector that only form noise during decision-making (S120) and degrade the accuracy of the result). In fact, the autoencoder (e.g., a variational autoencoder) is trained (i.e., aligned) on a set of medical images and learns their latent representations. Furthermore, the generation method benefits from decoding the properties of the autoencoder (e.g., a variational autoencoder) to reconstruct template images that resemble the actual medical images in the dataset. Thus, the generated template images correspond to realistic medical images representing organs of different patients.

[0032] The generation method further leverages the latent space of an autoencoder (e.g., a variational autoencoder) to minimize a loss, which is a function of the latent vector and, for each medical image in the dataset, the parameters of the differential homeomorphic deformation applied to the template image to acquire the medical image. In other words, loss minimization is performed in the latent space of the autoencoder, and therefore the low dimensionality of the latent space can be used to improve performance and reduce computational cost. Loss minimization can be performed by any gradient descent method (e.g., stochastic gradient descent). Since the loss is a function of the latent vector and the parameters of the differential homeomorphic deformation, the minimization space (i.e., the variable domain explored or spanned during minimization) is the product of a set of domains including the latent space. Thus, the values ​​of the variables change in their respective domains during the minimization of the function. Therefore, minimization is achieved by directly changing the latent vector in the latent space and by changing the parameters in their respective domains (i.e., exploring classes of differential homeomorphic deformations).

[0033] Alternatively, or at will, to maximize the use of the latent space during minimization, the parameters of the differential homeomorphisms may be the result of a neural network that takes a latent vector and a representation of the medical images of the encoded dataset as inputs. In this way, the parameters may be a function of the neural network parameters defined in the latent space of the autoencoder. Minimization is therefore achieved by directly changing the latent vector in the latent space and by changing the neural network parameters in the latent space (i.e., by indirectly changing the parameters of the differential homeomorphisms, i.e., by exploring classes of differential homeomorphisms). The generation method employs a loss with first and second terms in determining the template image (S120). The first term penalizes the discrepancy between the posterior probability of the parameters of the differential homeomorphisms applied to the template image to obtain each medical image of the dataset, given each medical image, and the Gaussian prior probability of the parameters of the differential homeomorphisms applied to the template image to obtain each medical image of the dataset (e.g., minimizing the difference). In other words, the first term of the loss function guarantees that the posterior probability follows a Gaussian distribution in the latent space.

[0034] The second term imposes a penalty (e.g., minimizes) the negative log-likelihood of the set of medical images in the dataset, given the parameters of the differential homeomorphic deformation applied to the template image to acquire each medical image in the dataset from the latent vector. In other words, the second term of the loss function measures how well the decoder can reconstruct the input data (e.g., representing medical images) from the parameters of the differential homeomorphic deformation applied to the latent vector and the template image.

[0035] The generation method therefore corresponds to a combination of a probabilistic approach and a deformation (e.g., geometric) approach. Such a combination is implemented in the latent space of a pre-trained autoencoder, discarding unnecessary details from the image in order to generate a template image.

[0036] The optional characteristics are explained here.

[0037] In determining the template image (S120), the generation method may include training a neural network (e.g., a registration network). In other words, the loss minimized by determining (S120) is expressed as a function of the neural network, and determining (S120) performs loss minimization while (simultaneously) changing the parameters of the neural network (thus becoming a "trained" version of the neural network). The neural network may take as input, for each medical image, the result of applying a latent vector and an encoder to the medical image (i.e., another latent vector representing the medical image in latent space). The neural network may predict the parameters of the diffeomorphic deformation applied to the template image to obtain each medical image in the dataset. The neural network may have an architecture like a U-net and may have 5 to 20 convolutional layers and 0 to 5 attention layers in between.

[0038] Because neural networks learn with latent data (i.e., data with information encoded in the latent space) representing latent vectors associated with template images and other latent vectors associated with input medical images, the neural network thus leverages the latent representation of the autoencoder. In other words, the generation method learns differential homeomorphisms from the latent space of the autoencoder (e.g., variational autoencoder), thereby learning accurate deformations of medical images in the dataset by leveraging more compact representations and key elements extracted by the autoencoder.

[0039] Furthermore, the parameters of a differential homeomorphism can be approximated by applying a neural network to the loss function for each medical image in the dataset and a given template image. The neural network has its own set of parameters (e.g., weights and biases), which can be the variables to be minimized. In other words, during loss minimization, the neural network parameters change in each region, and loss minimization can be achieved. The parameters of a differential homeomorphism can be a function of the neural network parameters, and searching for (spanning) these neural network parameters is equivalent to searching for (spanning) a class of differential homeomorphisms.

[0040] Determining the template image by minimizing the loss function (S120) means that the first term of the loss function, i.e., the term that penalizes the discrepancy between the posterior probability and the Gaussian prior probability, can be the Kullback-Leibler divergence or other discrepancy between the posterior probability and the Gaussian prior probability.

[0041] Gaussian prior probabilities can be centered, meaning the mean value of the Gaussian prior probabilities can be set to zero.

[0042] As is known in the art, divergence, particularly the Kullback-Leibler divergence, is a quantitative indicator of how a given probability distribution differs from another expected (e.g., approximated) probability distribution.

[0043] The differential homeomorphic deformations of a generation method may be the result of a power of a smooth resting velocity field. A power of a smooth resting velocity field guarantees the smoothness (i.e., regularity) and reversibility of the differential homeomorphic deformation. In other words, a class of differential homeomorphic deformations may be defined by a power of a smooth resting velocity field. In this way, a class of differential homeomorphic deformations may be defined by the properties of the smooth resting velocity field (e.g., stationarity, smoothing kernel, and other parameters of the smooth resting velocity field). A “velocity field” refers to a set of vectors, each assigned to a point in a medical image (e.g., a pixel in the case of a 2D medical image, a voxel in the case of a 3D medical image), indicating how a point in one medical image moves to align with a point in another medical image. A “resting velocity field” is a velocity field that does not change over time at each point in a medical image. A “smooth resting velocity field” is a continuously differentiable (i.e., regular, without jumps, bumps, or defects) resting velocity field. The generation method, therefore, differs significantly from approaches that modify pixels or voxels to perform deformations. This allows for template images that are realistic medical images that are consistent with the dataset (i.e., look like the medical images in the dataset) and accurately reproduce the anatomical details of organs.

[0044] A smooth static velocity field can be obtained by computing a smoothing kernel at the control points of a medical image.

[0045] A smoothing kernel can be a mathematical function defined on the input data (e.g., points in a medical image). Smoothing kernels help reduce noise in the input data, thereby improving the signal-to-noise ratio. Furthermore, smoothing kernels ensure the smoothness of the stationary velocity field before exponentiation. In other words, smoothing kernels prevent unrealistic or abrupt deformations.

[0046] The smoothing kernel can be a Gaussian kernel.

[0047] Medical images in this specification may be discrete. That is, the generation method samples the medical image at intervals (e.g., regular intervals) to create a grid of points. Each point in the grid may correspond to a pixel or voxel in the medical image. Each point in the grid may be associated with a value representing the intensity of a pixel in the case of a 2D medical image, or a voxel in the case of a 3D medical image (e.g., a discrete value from 0 to 255 for an 8-bit medical image).

[0048] Control points in a medical image can be points on a grid of medical images that control (e.g., parameterize) the differential homeomorphic deformation applied to the medical image. In other words, control points can be reference points (e.g., landmarks) on a grid of points in a medical image.

[0049] The generation method may involve sampling control points on a grid of points and assigning each control point to the respective parameters of the differential in-phase deformation applied to the medical image. In this way, i.e., by parameterizing the differential in-phase deformation at each control point, the generation method reduces the degrees of freedom (i.e., the number of independent parameters that parameterize the deformation). The degrees of freedom may include translation, rotation, scaling, and other complex deformations. In other words, computing a smoothing kernel at control points, each associated with each parameter of the differential in-phase deformation, is equivalent to reducing the computational load on the processor (i.e., CPU) of the underlying system (e.g., a computer system including a processor). Furthermore, in this way the generation method adopts a target smoothing kernel, thereby improving the accuracy and realism of the deformation and reducing bias (e.g., caused by pixels or voxels with high intensity values).

[0050] A set of medical images representing organs of different patients may include images from any of the following medical imaging modalities (e.g., actual medical images are obtained): automated refraction, angiography, bone densitometry (US), biomagnetic imaging, bone densitometry (X-ray), color flow Doppler, cine fluoroscopy, vaginoscopy, computed radiography, cystoscopy, computed tomography, duplex Doppler, digital fluoroscopy, fluoroscopy, digital microscopy, digital subtraction angiography, digital radiography, echocardiography, electrocardiogram, cardiac electrophysiology, endoscopy, fluorescein angiography, fiducial, fundus examination, general microscopy, hardcopy, hemodynamic waveforms, intraoral radiography, intraocular lens data, intravascular optical coherence tomography, intravascular ultrasound, corneal measurement, lentometry, laparoscopy, laser surface scan, magnetic resonance angiography, mammography, magnetic resonance, MR T1-weighted images, MR T2-weighted imaging, MR proton density-weighted imaging, MR steady-state free precession, MR effective T2, MR susceptibility-weighted imaging, MR short tau inversion recovery imaging, MR fluid-damped inversion recovery imaging, MR double inversion recovery imaging, MR conventional diffusion-weighted imaging, MR apparent diffusion coefficient imaging, MR diffusion tensor imaging, MR dynamic susceptibility contrast imaging, MR arterial spin contrast imaging, MR dynamic contrast-weighted imaging, MR blood oxygen-dependent imaging, MR time-of-flight imaging, MR phase contrast imaging, magnetic resonance spectroscopy, nuclear medicine, axial length measurement, optical coherence tomography (non-ophthalmology), ophthalmic photography, ophthalmic mapping, ophthalmic refraction testing, ophthalmology Medical imaging diagnostic methods including tomography, ophthalmic field of view, optical surface scanning, positron emission tomography (PET), panoramic X-ray, respiratory waveform, radiofluoroscopy, radiographic imaging (conventional film / screen), radiation therapy dose, radiation therapy images, radiation therapy planning, radiation therapy records, radiation therapy structural sets, segmentation, slide microscopy, stereometric measurements, single-photon emission computed tomography (SPECT), automated slide staining devices, thermography, ultrasound, A-mode US, B-mode US, M-mode US, visual acuity, videofluoroscopy, X-ray angiography, and external camera imaging.

[0051] "Medical imaging modality" refers to a type of imaging diagnostic technique that utilizes specific physical methods to detect signals within a patient's body and observe anatomical structures or physiological phenomena. Therefore, images produced by a specific medical imaging modality reflect the biological, structural, and physiological characteristics of the patient's tissues in an intensity space that reflects the desired characteristics (generally...).

[0052]

number

[0053] This is the result obtained by the transfer function to ). Medical imaging modalities may differ depending on the physical mechanism used, the physical sensors used for image acquisition, the sensor parameters at the time of acquisition, the use of contrast agents, the delay between contrast agent injection and acquisition, or post-acquisition signal processing.

[0054] The generation method may include acquiring an autoencoder (e.g., a variational autoencoder) (S100) and training the autoencoder. The autoencoder (e.g., a variational autoencoder) can be trained on the dataset acquired in S100, i.e., the same dataset containing a set of medical images.

[0055] Referring to the flowchart in Figure 2, a method for using template images of organs generated according to a generation method implemented by a computer is shown, and this will hereafter be referred to as the "method of use". The method of use includes obtaining input medical images representing the patient's organs (S200) and determining the parameters of the diffeomorphism that becomes the input medical image when applied to the template image (S210).

[0056] Such usage forms an improved solution for using organ template images.

[0057] In particular, such a method, given an input medical image (e.g., one obtained by a specific medical imaging technique) and a template image (e.g., one generated from a dataset containing medical images obtained from the same medical imaging technique), determines the parameters of a differential homeomorphism that transforms the template image into the input medical image. In other words, the method predicts the parameters of the differential homeomorphism by instantiating (i.e., transforming) the template image into the input medical image.

[0058] The usage method involves performing a patient-specific task: using template images that can serve as reference images representing organs, and determining the parameters of a differential homeomorphism that transforms these template images into input medical images representing the organs of a specific patient.

[0059] Furthermore, by applying a differential homeomorphism to a template image, the method provides accurate and reliable instantiation of the template image that is aligned to the input medical image representing the patient's organs. In fact, the method provides the most likely parameters of the differential homeomorphism applied to the template image, given the input medical image. The most likely parameters of the differential homeomorphism are those that maximize the posterior probability.

[0060] Further details about its usage are explained here.

[0061] The diffeomorphism in determining the parameters (S210) may be from the same class (e.g., one determined from among) as the diffeomorphisms explored during the determination (S120), or it may be from a different class. Optionally, the parameters determined in S210 may be predicted by a neural network learned during the execution of the determination (S120). In other words, the usage may involve a neural network learned during the determination (S120) that receives an input template latent vector (i.e., a latent vector corresponding to a template image) and another latent vector resulting from the application of an encoder to an input medical image. The parameters are therefore the result (i.e., output) of the neural network, and given an input medical image and a template image, the relevant diffeomorphism is the most likely deformation.

[0062] The method of use further includes aligning the input medical image and the template image. The alignment of the input medical image occurs before the decision (S210). In one example, the alignment of the input medical image may occur before acquiring the input medical image (S200) or after the decision (S210). Alignment improves the accuracy of the decision (S210).

[0063] Depending on the method of use, the template images of organs used may include data representing the medical segmentation of the template image and / or one or more medical annotations to the template image.

[0064] Medical image annotation refers to any type of spatial indication added manually or automatically to describe, supplement, or analyze spatial information. Examples of annotation include any type of indication (boxes, arrows, etc.) that shows regions of interest with measurements of image elements and associated labels. Medical image annotation helps identify and classify different parts of an anatomical target; for example, it can mark a region as a tumor, or label regions of interest as indicating a specific condition. For example, annotation can mark the location and size of a tumor in a medical image obtained from an MRI scan, or label a fracture in a medical image obtained from an X-ray scan.

[0065] Medical image segmentation refers to the division of a region of interest within a medical image. Segmentation can utilize annotations previously applied to the region of interest within the medical image.

[0066] The usage may further include projecting data onto an input medical image (e.g., medical annotation and / or medical segmentation) based on the parameters of the determined diffeomorphism.

[0067] The method of use may further include displaying a transformed (also called instantiated) representation of a medical image on the screen of a computer system. Displaying may include showing the input medical image, the template image, and the instantiated medical image (i.e., the medical image resulting from the application of a diffeomorphism to the template image) on the same screen (e.g., on different parts of the screen). Displaying allows a healthcare professional viewing the screen to compare the input medical image, the template image, and the instantiated medical image, or parts thereof. Displaying may trigger medical actions such as diagnosis, additional medical examinations based on the comparison results, and / or medical treatment (e.g., a new treatment or indication of treatment) (e.g., by a healthcare professional viewing the screen).

[0068] Additionally or alternatively, the usage may further include (i) analyzing instantiated medical images (e.g., segmentation), (ii) extracting medically relevant features from the analyzed instantiated medical images, (iii) classifying the extracted medically relevant features (benign or malignant), and (iv) outputting diagnostic information and / or treatment.

[0069] The generation and usage methods are implemented on a computer. This means that the steps (or substantially all steps) of the generation and usage methods are performed by at least one computer, or any system, etc. Therefore, the steps of the generation and usage methods are performed by a computer, and may be fully automated or semi-automatic. For example, at least some of the steps of the method are triggered by user-computer interaction. The required level of user-computer interaction varies depending on the expected level of automation and is balanced with the need to fulfill the user's requirements. For example, this level can be defined by the user or predefined.

[0070] A typical example of implementing a method on a computer is to execute the method using a system suited to that purpose. This system includes a processor linked to memory and a graphical user interface (GUI), where the computer program containing the instructions for executing the method is stored. Memory can also be used for database storage. Memory can be any hardware suitable for such storage and may consist of multiple physically distinct parts (e.g., one for the program and possibly one for the database).

[0071] Figure 3 shows an example of a system, where the system is a client computer system, such as a user's workstation.

[0072] The computer in question includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000, and random access memory (RAM) 1070 also connected to the bus. The computer is further provided with a graphical processing unit (GPU) 1110 associated with video random access memory 1100 connected to the bus. The video RAM 1100 is also known in the art as a frame buffer. A mass storage device controller 1020 manages access to mass storage devices, such as a hard drive 1030. Mass storage devices suitable for tangibly executing computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices, magnetic disks such as internal hard disks and removable disks, and magneto-optical disks. Any of the above may be complemented by or incorporated into a specially designed ASIC (Application-Specific Integrated Circuit). A network adapter 1050 manages access to the network 1060. The computer may also include tactile devices 1090, such as a cursor control device and a keyboard. A cursor control device is used in a client computer to allow the user to selectively position the cursor at any desired location on the display 1080. Furthermore, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes several signal generating devices for inputting control signals to the system. Typically, the cursor control device may be a mouse, with the mouse buttons used to generate signals. Alternatively, or additionally, the computer system may include a sensitive pad and / or a sensitive screen.

[0073] A computer program may include instructions that can be executed by a computer, and the instructions include means for causing the system described above to execute a generation method and / or usage method. A program may be recordable on any data storage medium, including the system's memory. A program may be implemented, for example, in a digital electronic circuit, or in computer hardware, firmware, software, or a combination thereof. A program may be implemented as a device, for example, as a product tangibly implemented in a machine-readable storage device for execution by a programmable processor. The steps of a method may be executed by a programmable processor that executes a program of instructions that perform the functions of the method by manipulating input data and producing outputs. The processor may therefore be programmable and coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to transmit data and instructions to them. An application program may be implemented in a high-level procedural programming language or an object-oriented programming language, or, if preferred, in assembly language or machine code. In any case, the language may be a compiled language or an interpreted language. A program may be a complete installation program or an update program. The application of a program on a system in any case results in instructions for executing a method. Alternatively, computer programs can be stored and executed on servers in a cloud computing environment, where the servers communicate with one or more clients over a network. In such cases, processing units execute instructions composed of the program, thereby executing the method on the cloud computing environment.

[0074] For the sake of explanation, an example of an implementation of the generation method is summarized below.

[0075] The generation method may depend on an unsupervised deep learning model (e.g., a deep convolutional network) that registers medical images and training deformation templates. "Registering medical images" means the process of aligning a first image with a second image, that is, transforming the first image into the second image.

[0076] For this purpose, the generation method includes obtaining a dataset containing a set of medical images representing organs of different patients (S100).

[0077] The generation method includes acquiring an autoencoder (S100). The autoencoder may be a variational autoencoder. The autoencoder (e.g., a variational autoencoder) includes an encoder and a decoder. The autoencoder (e.g., a variational autoencoder) learns the latent representation of the medical images (e.g., the shape and contour of the organ represented in the medical image) by training on a set of medical images, each representing an organ of a different patient. The autoencoder (e.g., a variational autoencoder) may be trained on a dataset acquired in S100, which includes a set of medical images.

[0078] The latent representation of an autoencoder is used around (i.e., coupled with) a registration model (e.g., a registration neural network), taking input latent codes (e.g., latent vectors) as input to an input medical image and outputting parameters for a deformation map (e.g., a transformation). More specifically, the resulting deformation is computed using a diffeomorphic deformation model and applied to the decoded medical image. "Decoded medical image" means the result of applying a decoder to the latent codes. Used in this way, the registration model can learn deformations to register images using only their latent representations.

[0079] The generation method employs a probabilistic model, which will be described in detail later. In this way, the generation method ensures that the generated template image becomes a medical image from the dataset, while preventing it from becoming excessively noisy. The probabilistic model further learns deformations that transform the template image into a medical image from the dataset, and as a result obtains an atlas of deformations around the template image that can be sampled to generate new medical images.

[0080] An example of implementation is described in detail here.

[0081] The generation method includes obtaining a dataset containing sets of medical images representing organs of different patients (S100). The set of medical images has a data distribution,

[0082]

number

[0083] followed by,

[0084] Independent and identically distributed data samples,

[0085]

number

[0086] This may include, where N represents the total number of samples, for example, 100, 200, 500, or 1000.

[0087]

number

[0088] teeth,

[0089]

number

[0090] or

[0091]

number

[0092] (D, H, and W represent depth, height, and width (i.e., the spatial dimensions of the medical image), and d represents the number of channels in the medical image.)

[0093] Each element is data representing a medical 2D or 3D image. The medical image is used for geometric purposes, specifically as a grid region of the medical image.

[0094]

number

[0095] from

[0096]

number

[0097] This can be viewed as a discretization of a function to . Such discretizations are useful for the deformation models described later.

[0098] The generation method includes obtaining an autoencoder (S100). The autoencoder may be a variational autoencoder.

[0099] Figure 4 shows a schematic diagram of a variational autoencoder.

[0100] The variational autoencoder includes an encoder 400 and a decoder 410. The autoencoder (e.g., autoencoder) may include a deep convolutional network using, for example, 5 to 20 convolutional layers and 0 to 5 interleaved attention layers for both the encoder network and the decoder network.

[0101] Variational autoencoders have two models.

[0102] Observed values

[0103]

number

[0104] A latent code given (i.e., a conditional) input data representing a medical image.

[0105]

number

[0106] The posterior probability of a probability encoder

[0107]

number

[0108] And, latent code

[0109]

number

[0110] The given data

[0111]

number

[0112] The probability decoder is the likelihood of

[0113]

number

[0114] This could be a latent probability model that includes ,

[0115] posterior distribution

[0116]

number

[0117] Calculating or estimating the maximum likelihood can often be very difficult. One approach to overcome this problem is to use variational inference: posterior probability

[0118]

number

[0119] This is another distribution selected from the parametric distribution family (e.g., Gaussian distribution).

[0120]

number

[0121] It is approximated by this.

[0122] Posterior probability

[0123]

number

[0124] to a different distribution

[0125]

number

[0126] Approximating with this is equivalent to maximizing the lower bound of evidence, assuming a correct probability (for example,

[0127]

number

[0128] teeth,

[0129]

number

[0130] The average is given by,

[0131]

number

[0132] It can be a Gaussian distribution whose standard deviation is given by,

[0133]

number

[0134] The prior probability can be a centralized and scaled Gaussian distribution, belonging to the family of parametric distributions.

[0135]

number

[0136] is the average

[0137]

number

[0138] and,

[0139] dispersion

[0140]

number

[0141] Under a family of Gaussian distributions parameterized by , this corresponds to minimizing the following loss function of the variational autoencoder:

[0142]

number

[0143] Here,

[0144]

number

[0145] This is a function parameterized by a neural network that gives the mean of the decoder's probability distribution,

[0146]

number

[0147] And,

[0148]

number

[0149] This is the approximated posterior distribution

[0150]

number

[0151] A function parameterized by another neural network that gives the mean and variance, respectively.

[0152]

number

[0153] This is obtained by sampling this posterior distribution,

[0154]

number

[0155] is the average

[0156]

number

[0157] and distribution

[0158]

number

[0159] Gaussian distribution

[0160]

number

[0161] This is a standard Gaussian distribution with a mean of 0 and a variance of 1.

[0162]

number

[0163] Represents the Kullback-Leibler divergence between [parameter] and [parameter].

[0164]

number

[0165] is the variance of the probabilistic decoder in an autoencoder network. A larger value indicates greater noise around the decoded value, and therefore the decoder prioritizes regularization over reconstruction of the decoded image.

[0166]

number

[0167] Typical values ​​range from 1 to 10, depending on the desired degree of regularization. -6 It could fall within that range.

[0168] The generation method can therefore use this variational ode encoder to learn the latent space of the regularized input data.

[0169] The generation method may include the differential homeomorphism model described here.

[0170] One example of a differential homeomorphism model may include a smooth vector field (e.g., a stationary velocity field).

[0171]

number

[0172] This is in the field of medical imaging.

[0173]

number

[0174] Therefore, the body is in contact with the space

[0175]

number

[0176] It is a mapping to . In other words, a smooth velocity field is one in which each point in a medical image is associated with a smooth velocity vector.

[0177] A smooth velocity field is an element of tangent space, which is a vector space containing vectors tangent to points in a medical image. A smooth velocity field can also be a stationary velocity field.

[0178] Differential homeomorphisms associated with a smooth static velocity field are used in the medical imaging domain.

[0179]

number

[0180] A mapping from itself

[0181]

number

[0182] The differential homeomorphism is defined by the following system of equations.

[0183]

number

[0184] Here, the first equation corresponds to the evolution equation of the differential homeomorphism, and the second equation is the initial condition (i.e., the differential homeomorphism applied to the medical image at the initial time t=0 is the medical image region).

[0185]

number

[0186] (This is the identity function above). This provides a geometric method for parameterizing spatial deformations via a velocity field (e.g., a smooth static velocity field). Furthermore, having a static velocity field makes it possible to accelerate (i.e., speed up) the calculation of diffeomorphic deformations using the scaling-and-squaring method. The scaling-and-squaring method depends on the algebraic properties of the exponential function, i.e.,

[0187]

number

[0188] and

[0189]

number

[0190] If they are commutative,

[0191]

number

[0192] and small speed field

[0193]

number

[0194] It depends on the approximation of the exponential function for . Using this property and approximation, the following equation is associated with velocity.

[0195]

number

[0196] So, differential homeomorphism

[0197]

number

[0198] It can be calculated.

[0199] Here,

[0200]

number

[0201] This is the scaling and squaring factor. The scaling and squaring factor can be set to a predetermined number (for example, an integer between 5 and 10, or between 7 and 12).

[0202] In order to improve the performance of diffeomorphic transformation and ensure a smooth stationary velocity field, the smooth stationary vector field

[0203] [Math.]]

[0204] for control points of a medical image

[0205] [Math.]]

[0206] wherein (for example, the control points are points on a grid or a sub-grid of a medical image), a smoothing kernel

[0207] [Math.]]

[0208] can be obtained by calculating (for example, evaluating) . Each control point

[0209] [Math.]]

[0210] can be

[0211] [Math.]]

[0212] associated with each parameter of a diffeomorphic transformation applied to a medical image, represented by . The smooth stationary velocity field is

[0213] [Math.]]

[0214] can be parameterized as. The smoothing kernel is a scale that can be set to one tenth of the image size

[0215] <Math.>

[0216] having, for example,

[0217] <Math.>

[0218] , which may be a Gaussian kernel. The scale

[0219] <Math.>

[0220] may be an adjustable hyperparameter that controls regularization of the diffeomorphic transformation. When the image size is normalized, for example considered to be 1, a typical value is

[0221] <Math.>

[0222] in the range of

[0223] A diffeomorphic transformation (e.g., a spatial transformation) resulting from a diffeomorphic transformation model is a medical image region

[0224] <Math.>

[0225] (also called ambient space) by deforming the medical image region

[0226] <Math.>

[0227] It acts on, and as a result,

[0228]

number

[0229] Obtained, here,

[0230]

number

[0231] These are medical images,

[0232]

number

[0233] This represents a differential homeomorphism, where the points on the left represent the deformation action on the medical image (i.e., the action of the deformation group on the medical image), and the circle on the right represents the composition of functions. For simplicity, the generation method is the inverse deformation.

[0234]

number

[0235] Calculate the differential homeomorphism associated with it and remove unnecessary differential homeomorphisms.

[0236]

number

[0237] This could prevent a reversal.

[0238] We will explain the probability model here.

[0239] The purpose of the generation method is to generate template images of organs.

[0240]

number

[0241] This will be used as the template image. Each medical image in the dataset

[0242]

number

[0243] Regarding the generation method, the medical images of the dataset

[0244]

number

[0245] Template image to be as close as possible

[0246]

number

[0247] Differential homeomorphism that deforms

[0248]

number

[0249] It is possible to calculate each differential homeomorphism.

[0250]

number

[0251] According to the differential homeomorphism model explained earlier, the parameters

[0252]

number

[0253] It can be parameterized by [the specified method].

[0254] Figure 5 shows a graphical representation of the probability model, where the dashed line represents the deterministic parameterization of the latent variable (which can be considered a Dirac distribution in a probabilistic context), and the solid line represents the conditional probability of the probability model.

[0255] Probability model

[0256]

number

[0257] This can predict both the template image and the velocity field (e.g., a smooth static velocity field) (e.g., simultaneously). In other words, the probabilistic model can predict both the template image and the parameters of the diffeomorphic deformation applied to the template image to obtain the medical image (e.g., simultaneously).

[0258]

number

[0259] Let the elements (i.e., latent codes or latent vectors) be the elements of the latent space of a pre-trained variational autoencoder, and the template image

[0260]

number

[0261] This latent code (i.e.,

[0262]

number

[0263] ) Likelihood distribution associated with

[0264]

number

[0265] This is the average. In this way, the template image

[0266]

number

[0267] This is because the variational autoencoder is properly trained, and the prior distribution is imposed on the latent space of the variational autoencoder.

[0268]

number

[0269] This is the standard Gaussian distribution, that is,

[0270]

number

[0271] If so, it can represent (i.e., appear to represent) medical images in the dataset.

[0272] The probabilistic model can also define prior probability distributions for the parameters of the differential homeomorphic deformation. Such prior distributions may be Gaussian prior probabilities for the parameters of the differential homeomorphic deformation applied to a template image to acquire each medical image in the dataset. The Gaussian prior probabilities of the parameters facilitate smooth differential homeomorphic deformation through the use of a smoothing kernel introduced into the parameterization of the stationary velocity field. The Gaussian prior probabilities can be centered. Collectively,

[0273]

number

[0274] The Gaussian prior probability of the parameters of a differential homeomorphism, known as,

[0275] Template variables

[0276]

number

[0277] It can become independent from,

[0278]

number

[0279] It can be expressed as, and here,

[0280]

number

[0281] This is a smoothing kernel matrix containing the smoothing kernel values ​​for each pair of points on the grid (e.g., control points). In fact, this is,

[0282]

number

[0283] This can be rewritten as follows: This prior distribution is the smoothed kernel norm

[0284]

number

[0285] The first parameter is regularized by the average, which promotes the smoothness of the velocity field and, consequently, the associated flow.

[0286]

number

[0287] It promotes smoothness.

[0288] Therefore, template variables

[0289]

number

[0290] and parameters of differential homeomorphism

[0291]

number

[0292] Since they can be independent, the prior probability of the parameters of the differential homeomorphism applied to the template image to acquire each medical image in the dataset (for example,

[0293]

number

[0294] ) is (i) the prior distribution imposed on the latent space of the variational autoencoder

[0295]

number

[0296] (ii) Prior probability of the parameters of the differential homeomorphism

[0297]

number

[0298] It can be a product of these two things.

[0299] Given the parameters of the differential homeomorphism applied to the template image to obtain each medical image in the dataset and the latent vector, the data likelihood of the probabilistic model, i.e., the likelihood of the set of medical images in the dataset, is:

[0300]

number

[0301] It is expressed as follows, and here,

[0302]

number

[0303] This represents a template image,

[0304]

number

[0305] is a smooth stationary velocity field

[0306]

number

[0307] Differential homeomorphisms arising from known parameters

[0308]

number

[0309] This is a template image transformed by the parameters.

[0310]

number

[0311] This represents the variance of the probabilistic decoder in the registration network. A larger value indicates greater noise around the decoded value, and therefore the decoder prioritizes regularization over reconstruction of the decoded image.

[0312]

number

[0313] Typical values ​​range from 1 to 10, depending on the desired degree of regularization. -6 It could fall within that range.

[0314] latent vector

[0315]

number

[0316] and parameters of differential homeomorphism

[0317]

number

[0318] For the purpose of predicting (for example, simultaneously),

[0319]

number

[0320] Given each medical image represented as follows, the posterior probability of the parameters of the differential homeomorphism applied to the template image to obtain each medical image in the dataset can be calculated using variational inference:

[0321]

number

[0322] It can be approximated by another expected distribution expressed as . Approximated posterior distribution

[0323]

number

[0324] These could be Gaussian distributions, each with its own mean and variance.

[0325] The probabilistic model actually involves two neural networks (for example, two separate neural networks), and each neural network is independent and distributed

[0326]

number

[0327] and

[0328]

number

[0329] These correspond to the first neural network, which is the template image probability distribution.

[0330]

number

[0331] This parameterizes the approximate posterior distribution.

[0332]

number

[0333] The average of each

[0334]

number

[0335] and their respective variances

[0336]

number

[0337] It has a second neural network (e.g., a registration model or registration network)

[0338]

number

[0339] This predicts the parameters of the differential homeomorphic deformation applied to the template image to acquire each medical image in the dataset. Both neural networks can therefore predict the mean and variance of their respective posterior distributions. The registration network has an architecture similar to a U-net, and can have 5 to 20 convolutional layers and 0 to 5 attention layers in between. In this way, the predicted parameters can have the same spatial resolution as the latent image (i.e., the latent vector representing the encoded medical image), and furthermore, this provides a sufficient number of degrees of freedom for the probabilistic model.

[0340] The generation method is latent vector

[0341]

number

[0342] And the dataset

[0343]

number

[0344] Each medical image

[0345]

number

[0346] Regarding the parameters of the differential homeomorphism applied to the template image to acquire medical images,

[0347]

number

[0348] By minimizing the loss, which is a function of the latent vector, the decoder can be transformed into a latent vector.

[0349]

number

[0350] The template image is determined as a result of applying it (S120).

[0351] The loss is an approximate posterior probability.

[0352]

number

[0353] and the true (e.g., actual or exact) posterior probability

[0354]

number

[0355] The difference (e.g., discrepancy) between the two can be minimized. The loss is the Kullback-Leibler divergence between the approximate posterior probability and the true posterior probability, e.g.,

[0356]

number

[0357] It is possible.

[0358] Minimization is the latent vector of a neural network that predicts the parameters of the differential homeomorphic deformation applied to the template image to acquire each medical image in the dataset.

[0359]

number

[0360] and parameters

[0361]

number

[0362] It can be performed against. The loss is therefore,

[0363]

number

[0364] It can be expressed as follows.

[0365] The loss (in the second row of the above equation) includes the first and second terms.

[0366] The first term of the loss (the second line of the equation above) is the posterior probability of the parameters of the differential homeomorphism applied to the template image to obtain each medical image in the dataset, given each medical image (for example, the true posterior probability is another expected posterior probability).

[0367]

number

[0368] (approximated by)

[0369] Gaussian prior probabilities of the parameters of the differential homeomorphism applied to the template image to acquire each medical image in the dataset (e.g.,

[0370]

number

[0371] This could be the Kullback-Leibler information between ) and ).

[0372] In the implementation example, since both distributions are Gaussian, the first term can be computed in a closed form (i.e., analytically exact).

[0373] The second term (the second line of the above equation) is the negative log-likelihood of the set of medical images in the dataset (for example, given the parameters of the differential homeomorphism applied to the template image to obtain each medical image in the dataset, and the latent vector.

[0374]

number

[0375] This is imposed as a penalty. In this implementation example, the second term reduces to a simple mean squared error data attachment term, but by choosing different assumptions about the data likelihood, it may also reduce to other types of loss functions.

[0376] Figure 6 shows a schematic representation of the probabilistic model. The gray boxes represent neural networks, and the white boxes represent non-learning layers that make up the differential homeomorphic deformation model. "ST" represents spatial transformation.

[0377] When the loss function is minimized, that is, the optimal parameters of the training dataset are found.

[0378]

number

[0379] and latent vector

[0380]

number

[0381] Once that is decided,

[0382] latent vector

[0383]

number

[0384] The result of applying the decoder to, that is,

[0385]

number

[0386] The template image is determined by (S120).

[0387]

number

[0388] Therefore, it provides representative medical images of organs, which can be considered reference medical images for the dataset.

[0389] Figure 7 shows an example of medical images obtained from a dataset acquired in S100. The dataset may contain 1000 (e.g., between 800 and 1200, e.g., 1000) medical images representing the hearts of different patients. The dataset includes a set of medical images obtained from CT scans. In the example, the medical images in the dataset may be divided into a training volume (e.g., 800 training volumes), a validation volume (e.g., 16 validation volumes), and a test volume (e.g., 184 test volumes). The generation method corresponds to calculating template medical images and their deformations and geometrically matching them to the medical images in the dataset (e.g., by means of differential homeomorphism).

[0390] Figure 8 shows a comparison of template images of the heart. Template image 800 was calculated using an existing method. Template image 810 was calculated using the generation method of the present disclosure. The comparison between template image 800 and template image 810 shows that template image 810 better represents the heart, i.e., template image 810 is more similar to the actual medical images in the dataset. In other words, template image 810 corresponds to the actual medical images, which are consistent with the dataset. Furthermore, template image 810 is more accurate (e.g., detailed) in representing the anatomical details of the organ (e.g., the heart). The higher level of accuracy and realism of template image 810 calculated by the generation method of the present disclosure is achieved by parameterizing the differential homeomorphism. For example, template image 810 yields a medical image with reduced bias (e.g., bias caused by pixels or voxels with high intensity values ​​is reduced and averaged, resulting in a blurry, unrealistic image like template image 800).

[0391] An example of how to implement the use of organ template images is described here.

[0392] Instructions for use: Input medical images representing the patient's organs.

[0393]

number

[0394] This includes obtaining (S200).

[0395] Instructions for use are provided in the template image.

[0396]

number

[0397] When applied, input medical images

[0398]

number

[0399] The parameters of the differential homeomorphism are such that (for example,

[0400]

number

[0401] ) is determined (S210).

[0402] The differential homeomorphism of the method of use may be a differential homeomorphism of the generation method described above in this disclosure. Optionally, the parameters determined in S210 of the method of use may be the neural network described above.

[0403]

number

[0404] These may be parameters predicted by the input medical image. In this case, the template image is used as the input medical image.

[0405]

number

[0406] To instantiate it, the usage is that when applied to the template image, it becomes the input medical image, and the parameters of the differential homeomorphism are...

[0407]

number

[0408] The decision is made (S210).

[0409] parameters

[0410]

number

[0411] The input medical image

[0412]

number

[0413] and latent vector

[0414]

number

[0415] Given the posterior probability

[0416]

number

[0417] This maximizes the posterior probability of the parameters of the differential homeomorphism.

[0418]

number

[0419] A neural network trained to approximate the true posterior probability by predicting the mean and variance.

[0420]

number

[0421] This can be achieved through a registration model / module (also known as a registration network).

[0422] The usage method includes acquiring input medical images (S200). Acquisition (S200) is the input medical image

[0423]

number

[0424] as a latent vector (for example,

[0425]

number

[0426] This may include encoding (by calculating) (for example, by the encoder of a variational autoencoder). Acquiring (S200) the input medical image

[0427]

number

[0428] latent vectors and template latent vectors

[0429]

number

[0430] registration module

[0431]

number

[0432] This may include supplying to.

[0433] The usage is to determine the parameters (e.g., prediction) of the diffeomorphism (S210). The parameters of the diffeomorphism (e.g., parameters of the diffeomorphism deformation) are the output of the registration module, i.e.,

[0434]

number

[0435] It is possible.

[0436] The parameters of a diffeomorphism (e.g., the parameters of a diffeomorphism deformation) can be the mean (i.e., with respect to the maximum value) of an approximate posterior probability distribution. Related diffeomorphism deformations

[0437]

number

[0438] Therefore, this represents the most likely deformation given the input medical image and prior distribution.

[0439] Figure 9 shows an example of instantiation (e.g., deformation) of a template image performed according to the usage method. Medical image 900 represents the template image calculated according to the generation method. Medical image 910 represents the input medical image (also called the target medical image). Medical image 920 represents the instantiated medical image, i.e., the medical image resulting from the application of a differential homeomorphism (e.g., differential homeomorphism deformation) to the template image. The instantiated medical image 920 captures the characteristics (e.g., contour, structure, intensity) of the input medical image 910. The deformation grid 930 shows a three-dimensional grid illustrating the transformation from template image 900 to instantiated medical image 920.

[0440] For example, before using the method, the template image may contain data representing the medical segmentation of the template image and / or one or more medical annotations on the template image. The method may further include projecting the data onto the input medical image based on the parameters of the determined differential homeomorphism. In other words, the method may determine the parameters of a differential homeomorphism that, when applied to a template image which may contain data representing the medical segmentation of the template image and / or one or more medical annotations on the template image, results in an input medical image containing the data (e.g., transformed data).

[0441] Figure 10 shows how the usage method transforms data representing the segmentation and / or annotations of a template image into an instantiated medical image. Image 940 represents the template image computed by the generation method. Image 950 represents the template image 940 having data representing the medical segmentation and / or one or more medical annotations for the template image 940. Image 950 includes the medical segmentation and / or annotations, i.e., the top of image 950a), the right side of image 950b, the left side of image 950c, and the bottom of image 950d. Image 960 represents the input medical image. Image 970 is the instantiated medical image, i.e., the medical image resulting from the application of a differential homeomorphism (e.g., a differential homeomorphism) to the template image 950 having the medical segmentation and / or annotations. Image 970 includes data (e.g., transformed data, converted data) representing the medical segmentation and / or annotation of the template image 950, namely, transformed segmentation 970a of segmentation 950a, transformed 970b of segmentation 950b, transformed segmentation 970c of segmentation 950c, and transformed segmentation 970d of segmentation 950d. Image 980 represents the input medical image having ground truth segmentation and / or annotations 980a, 980b, 980c, and 980d. A comparison of the instantiated medical image 970 with the input medical image 980 demonstrates an accurate and valid instantiation (e.g., transformation) of the template image, including data representing the segmentation and / or one or more medical annotations for the template image.

Claims

1. A computer-implemented method for generating template images of organs, wherein the method is A dataset (D) containing a set of medical images representing organs of different patients, An autoencoder that is trained on a set of medical images representing organs of different patients, wherein the autoencoder includes an encoder and a decoder, an autoencoder (A), To obtain (S100), (S120) Determining the template image as a result of applying the decoder to the latent vector, wherein the latent vector is defined in latent space by minimizing a loss which is a function of the latent vector and the parameters of the differential homeomorphic deformation applied to the template image to acquire the medical image for each medical image in the dataset, wherein the latent vector and the parameters of the differential homeomorphic deformation are variable during the minimization, the minimization is performed in latent space, and the loss is Given each of the aforementioned medical images, the posterior probability of the parameters of the differential homeomorphism applied to the template image to acquire each medical image in the dataset, The Gaussian prior probability of the parameters of the differential homeomorphism applied to the template image in order to acquire each medical image of the dataset, The first clause imposes a penalty for any discrepancy between the two, Given the parameters of the differential homeomorphism applied to the template image to acquire the latent vector and each medical image of the dataset, a second term imposes a penalty of the negative log-likelihood of the set of medical images in the dataset, (S120) includes, Methods that include...

2. The method according to claim 1, wherein determining the template image includes predicting the parameters of a differential homeomorphic deformation applied to the template image and training a neural network to acquire each medical image in the dataset.

3. The neural network takes the latent vector and the result of applying the encoder to the medical image as input for each medical image. The method according to claim 2.

4. The method according to any one of claims 1 to 3, wherein the first term is the Kullback-Leibler divergence between the posterior probability and the Gaussian prior probability.

5. The method according to any one of claims 1 to 4, wherein the differential in-phase deformation is the result of a power of a smooth static velocity field, the smooth static velocity field is obtained by computing a smoothing kernel at control points of a medical image, each control point being associated with the respective parameter of the differential in-phase deformation applied to the medical image, and the power of the smooth static velocity field comprises a scaling and squarering method.

6. The method according to claim 5, wherein the smoothing kernel is a Gaussian kernel.

7. The method according to any one of claims 1 to 6, wherein the Gaussian prior probability is centered.

8. The method according to any one of claims 1 to 7, wherein acquiring the autoencoder includes learning the autoencoder.

9. The aforementioned set of medical images includes automated refraction testing, angiography, bone densitometry (US), biomagnetic imaging, bone densitometry (X-ray), color flow Doppler, cine fluorescence fluoroscopy, vaginoscopy, computed radiography, cystoscopy, computed tomography, duplex Doppler, digital fluoroscopy, fluoroscopy, digital microscopy, digital subtraction angiography, digital radiography, echocardiography, electrocardiogram, cardiac electrophysiology, endoscopy, fluorescein angiography, fiducial, fundus examination, general microscopy, hardcopy, hemodynamic waveforms, intraoral radiography, intraocular lens data, intravascular optical coherence tomography, intravascular ultrasound, corneal measurement, lentometry, laparoscopy, laser surface scan, magnetic resonance angiography, mammography, magnetic resonance, MR T1-weighted images, and MR T2-weighted images, MR proton density-weighted images, MR steady-state free precession, MR effective T2, MR susceptibility-weighted images, MR short tau inversion recovery images, MR fluid-attenuated inversion recovery images, MR double inversion recovery images, MR conventional diffusion-weighted images, MR apparent diffusion coefficient images, MR diffusion tensor images, MR dynamic susceptibility contrast images, MR arterial spin contrast images, MR dynamic contrast-weighted images, MR blood oxygen-dependent images, MR time-of-flight images, MR phase contrast images, magnetic resonance spectroscopy, nuclear medicine, axial length measurement, optical coherence tomography (non-ophthalmology), ophthalmic photography, ophthalmic mapping, ophthalmic refraction testing, ophthalmic tomography, eye This includes images from any of the following medical imaging diagnostic methods: field of view, optical surface scan, other, positron emission tomography (PET), panoramic X-ray, respiratory waveform, radiofluoroscopy, radiographic imaging (conventional film / screen), radiation therapy dose, radiation therapy images, radiation therapy planning, radiation therapy records, radiation therapy structural sets, segmentation, slide microscopy, stereometric measurements, single-photon emission computed tomography (SPECT), automated slide staining devices, thermography, ultrasound, A-mode US, B-mode US, M-mode US, visual acuity, videofluoroscopy, X-ray angiography, and external camera imaging. The method according to any one of claims 1 to 8.

10. A method for using an organ template image generated according to any one of claims 1 to 9, which is implemented on a computer, wherein the method of use is To acquire an input medical image representing the aforementioned organs of the patient (S200), Determining the parameters of the differential homeomorphism that becomes the input medical image when applied to the template image (S210), Instructions for use, including those for which instructions are provided.

11. The method according to claim 10, wherein the differential homeomorphism is a differential homeomorphism according to any one of claims 1 to 9, and optionally, the differential homeomorphism is of the same class as the class of differential homeomorphisms that include the parameters predicted by the neural network according to claim 2 or 3.

12. The method according to claim 10 or 11, wherein the template image includes data representing a medical segmentation of the template image and / or data representing one or more medical annotations on the template image, and the method further includes projecting the data onto the input medical image based on determined parameters of the differential homeomorphism.

13. A computer program that includes instructions causing the computer to perform the method according to any one of claims 1 to 9, and / or the usage method according to any one of claims 10 or 12, when the program is executed by the computer, and / or A template image generated according to the method described in any one of claims 1 to 9, A data structure that includes this.

14. A computer-readable storage medium recording the data structure described in claim 13.

15. A system comprising a processor connected to memory, wherein the memory records the data structure described in claim 13.

Images