Method and device for generating a three-dimensional synthetic image from a three-dimensional input image

Abstract

The present invention relates to a method for generating at least one three-dimensional synthetic image from a three-dimensional input image, said three-dimensional input image representing a part of a body of a subject, said part comprising a bone portion. Figure 11a

Description

Title of the invention: Method and device for generating a three-dimensional synthetic image from a three-dimensional input image DOMAIN

[0001] The present invention relates to the field of medical image generation.

[0002] Advantageously, the invention relates to the generation of images of a part of the body of a subject, and more particularly of a bony portion of the body, in particular of a bony portion having complex anatomical specificities, such as the pelvis.

[0003] More specifically, the invention relates to a method and a device for generating a three-dimensional synthetic image from a three-dimensional input image. It also relates to a computer program and a computer-readable medium storing such a computer program. CONTEXT

[0004] Methods involving deep neural networks are implemented for image analysis. In particular, deep neural networks are increasingly used in the medical field, for example to assist in diagnosis.

[0005] In order to make a neural network efficient, a training process must be conducted before implementing the corresponding neural network. Such a training process requires substantial and qualitative data sets in order to ensure good performance of the corresponding neural network.

[0006] However, in the medical field, accessing a large number of images can be difficult. This is because data collection is costly and labor-intensive. In addition, medical data privacy concerns impose restrictions on data sharing. These drawbacks therefore limit access to publicly available medical data sets, which hinders the rapid implementation of cutting-edge methods in the fields of medicine and diagnosis. Summary

[0007] The present invention thus aims to increase the medical data sets that can be used to train neural networks for medical image analysis. The present invention is particularly advantageous for medical images that include a bony portion of a patient's body.

[0008] One aspect of the invention relates to a method for generating at least one three-dimensional synthetic image from a three-dimensional input image, the method comprising:

[0009] - receiving a three-dimensional input image representing a part of a body of a subject, said part comprising a bony portion,

[0010] - converting the three-dimensional input image into a first vector which has dimensions smaller than the dimensions of the three-dimensional input image,

[0011] - determination, using a diffusion module, of a vector modified by ap applying a diffusion processing to the first vector, the modified vector corresponding to a noisy representation of the first vector, said diffusion module being configured to receive, as input, the first vector and to provide, as output, the modified vector,

[0012] - determination, using an inverse diffusion module, of a second vector by applying an inverse diffusion processing to the modified vector, the second vector corresponding to a denoised representation of the modified vector, said inverse diffusion module being configured to receive, as input, the modified vector and to provide, as output, the second vector, and

[0013] - conversion of the second vector into a three-dimensional synthetic image which has the same dimensions as the three-dimensional input image, the three-dimensional synthetic image representing said part of the subject's body, said part comprising the bony portion.

[0014] By "three-dimensional input image" is meant a real image acquired during the examination of a patient. The real image can be acquired, for example, by computed tomography (or CT-scan for "Computed Tomography scan" according to the commonly used acronym of Anglo-Saxon origin).

[0015] By "three-dimensional synthetic image" is meant an image generated by the method of the invention. In other words, a synthetic image is a simulated image, by the method of the invention, which is not directly obtained by examining a patient.

[0016] By “vector” we mean a three-dimensional matrix representation (and not a column vector).

[0017] As detailed below, the inventors have developed a method for generating synthetic images from real input images representing a bone portion of the subject's body. This generation method makes it possible to preserve the anatomical details of the bone portion. Consequently, starting from a real image comprising the bone portion, the method of the invention makes it possible to generate a corresponding synthetic image which preserves the main features of the real image. This makes it possible to preserve the spatial structure of medical images. This point is crucial in medical imaging, because the spatial arrangement of bone structures is essential for performing accurate diagnosis and analysis.

[0018] Until now, to the inventors' knowledge, no one has thought of generating such synthetic images comprising a bony portion and in particular representing the pelvis of a subject. According to the inventors' knowledge, the method of the invention is the first to be developed to efficiently generate synthetic images of the pelvis.

[0019] Furthermore, due to diffusion processing and inverse diffusion processing (and the successive addition and elimination of noise components), a single input image can be used multiple times to generate a plurality of different synthetic images that will retain the main features of the input image.

[0020] The method according to the invention is particularly advantageous for medical images which are CT scans. Indeed, CT scans do not usually have a resolution as good as that of MRI images. It is therefore difficult to generate high-quality synthetic images from real CT scans. As presented below, the generation method according to the invention makes it possible to generate synthetic images from CT scans which preserve all the anatomical details of the bone portion concerned (here the pelvis). In other words, the method of the invention makes it possible to generate synthetic images which faithfully reproduce the anatomical details of the bone portion concerned, even if the resolution of the input images is not very high.According to the inventors' knowledge, the method of the invention is the first to be developed for generating three-dimensional synthetic images from three-dimensional input images having average quality.

[0021] Therefore, the generated synthetic images can be used to enrich the training datasets of neural networks in order to effectively train the corresponding neural networks which are used in the medical field.

[0022] Other non-limiting and advantageous characteristics of the invention, taken individually or in all technically possible combinations, are the following:

[0023] - the bony portion is the pelvis;

[0024] - the conversion of the three-dimensional input image into the first vector is implemented work using an encoding portion of an artificial neural network, the encoding portion being configured to receive, as input, the three-dimensional input image and to provide, as output, the first vector;

[0025] - the conversion of the second vector into a three-dimensional synthetic image is implemented implemented using a decoding portion of the artificial neural network, the decoding portion being configured to receive, as input, the second vector and to provide, as output, the three-dimensional synthetic image;

[0026] - the artificial neural network is an autoencoder;

[0027] - the diffusion processing comprises a composition of diffusion functions ap replicated to the first vector, each diffusion function corresponding to the addition of a noise component;

[0028] - the inverse diffusion processing comprises a composition of functions of inverse diffusion applied to the modified vector, each diffusion function corresponding to the elimination of the noise component;

[0029] - the noise component is a component with a Gaussian distribution;

[0030] - each diffusion function corresponds to a Markov chain;

[0031] - the diffusion processing comprises the application of a plurality of chains of Successive Markov chains, two successive Markov chains of the plurality of successive Markov chains being separated by a time step, the diffusion processing being associated with a number T of time steps;

[0032] - the inverse diffusion module comprises a convolutional neural network, the convolutional neural network being configured to implement a corresponding inverse diffusion function;

[0033] - the number T of time steps is greater than 100;

[0034] - the number T of time steps is greater than 500;

[0035] - the number T of time steps is greater than 1000;

[0036] - the method comprises, before the conversion of the three-dimensional input image, a preprocessing the three-dimensional input image, in order to obtain a preprocessed three-dimensional image, the first vector being obtained by converting said preprocessed three-dimensional image;

[0037] - the preprocessing of the three-dimensional input image comprises an extraction of the part comprising the bone portion, such that the preprocessed three-dimensional image is focused on said part comprising the bone portion;

[0038] - the extraction of the part comprising the bone portion comprises:

[0039] a) segmentation of the three-dimensional input image in order to identify said part comprising the bone portion,

[0040] b) determining a mask corresponding to said part comprising the bone portion on the basis of the segmented three-dimensional input image, and

[0041] c) generating an intermediate pre-processed three-dimensional image by applying said mask to the three-dimensional input image;

[0042] - preprocessing of the three-dimensional input image includes cropping and resizing the intermediate preprocessed three-dimensional image; and

[0043] - the preprocessing of the three-dimensional input image comprises an execution of an adjustment of a spatial orientation of the intermediate preprocessed three-dimensional image.

[0044] Another aspect of the invention relates to a method of generating at least one three-dimensional synthetic image from a three-dimensional input image using an image processing device which includes one or more processors, one or more memories including machine-executable instructions stored in the one or more memories for implementing a first neural network, a diffusion model and an inverse diffusion neural network, the method comprising:

[0045] - a reception, by the image processing device, of a three-dimensional input image mental representing a part of a subject's body, said part comprising a bony portion,

[0046] - a conversion, by the first neural network, of the three-dimensional input image sional into a first vector which has dimensions smaller than the dimensions of the three-dimensional input image,

[0047] - a determination, using the diffusion model, of a vector modified by ap applying a diffusion processing to the first vector, the modified vector corresponding to a noisy representation of the first vector, said diffusion model being configured to receive, as input, the first vector and to provide, as output, the modified vector,

[0048] - a determination, using the inverse diffusion neural network, of a second vector by applying an inverse diffusion processing to the modified vector, the second vector corresponding to a denoised representation of the modified vector, said inverse diffusion neural network being configured to receive, as input, the modified vector and to provide, as output, the second vector, and

[0049] - a conversion, with the first neural network, of the second vector into a three-dimensional synthetic image having the same dimensions as the three-dimensional input image, the three-dimensional synthetic image representing said part of the subject's body, said part comprising the bony portion.

[0050] In one embodiment, the conversion of the three-dimensional input image into the first vector is performed by the encoding portion of the first neural network.

[0051] In one embodiment, the conversion of the second vector into a three-dimensional synthetic image is performed by the decoding portion of the first neural network.

[0052] In one embodiment, the diffusion model is based on a diffusion function corresponding to one or more Markov chains.

[0053] Another aspect of the invention relates to a method for generating three-dimensional images of a bone portion for training a neural network of a medical image analysis system, the method comprising:

[0054] - the generation of at least one three-dimensional synthetic image from a three-dimensional input image using an image processing device that includes one or more processors, one or more memories including machine-executable instructions stored in the one or more memories for implementing a first neural network, a diffusion model and an inverse diffusion neural network, the generation comprising:

[0055] - receiving, by the image processing device, a three-dimensional input image dimensional representing a part of a subject's body, said part comprising the bony portion,

[0056] - the conversion, by the first neural network, of the three-dimensional input image sional into a first vector which has dimensions smaller than the dimensions of the three-dimensional input image,

[0057] - the determination, using the diffusion model, of a vector modified by ap applying a diffusion processing to the first vector, the modified vector corresponding to a noisy representation of the first vector, said diffusion model being configured to receive, as input, the first vector and to provide, as output, the modified vector,

[0058] - the determination, using the inverse diffusion neural network, of a second vector by applying an inverse diffusion processing to the modified vector, the second vector corresponding to a denoised representation of the modified vector, said inverse diffusion neural network being configured to receive, as input, the modified vector and to provide, as output, the second vector, and

[0059] - the conversion, with the first neural network, of the second vector into an image three-dimensional synthetic image which has the same dimensions as the three-dimensional input image, the three-dimensional synthetic image representing said part of the subject's body, said part comprising the bony portion, and

[0060] - training the neural network of the medical image analysis system with said three-dimensional synthetic image.

[0061] In one embodiment, the conversion of the three-dimensional input image into the first vector is performed by the encoding portion of the first neural network.

[0062] In one embodiment, the conversion of the second vector into the three-dimensional synthetic image is performed by the decoding portion of the first neural network.

[0063] In one embodiment, the diffusion model is based on a diffusion function corresponding to one or more Markov chains.

[0064] Another aspect of the invention relates to a device for generating at least one three-dimensional synthetic image from a three-dimensional input image, the device comprising a control unit configured to:

[0065] - receiving a three-dimensional input image representing a part of a body of a subject, said part comprising a bony portion,

[0066] - converting the three-dimensional input image into a first vector which has dimensions smaller than the dimensions of the three-dimensional input image,

[0067] - determine, using a diffusion module, a vector modified by application from a diffusion processing to the first vector, the modified vector corresponding to a noisy representation of the first vector, said diffusion module being configured to receive, as input, the first vector and to provide, as output, the modified vector,

[0068] - determine, using an inverse diffusion module, a second vector by ap applying an inverse diffusion processing to the modified vector, the second vector corresponding to a denoised representation of the modified vector, said inverse diffusion module being configured to receive, as input, the modified vector and to provide, as output, the second vector, and

[0069] - convert the second vector into a three-dimensional synthetic image which presents the same dimensions as the three-dimensional input image, the three-dimensional synthetic image representing said part of the subject's body, said part comprising the bony portion.

[0070] Another aspect of the invention relates to an image processing device for generating at least one three-dimensional synthetic image from a three-dimensional input image, the image processing device comprising a control unit which includes one or more processors, one or more memories including machine-executable instructions stored in the one or more memories for implementing a first neural network, a diffusion model and an inverse diffusion neural network and for performing the method comprising:

[0071] - receiving, by the image processing device, a three-dimensional input image sional representing a part of a body of a subject, said part comprising a bony portion,

[0072] - convert, by the first neural network, the three-dimensional input image into a first vector which has dimensions smaller than the dimensions of the three-dimensional input image,

[0073] - determine, using the diffusion model, a vector modified by applying a diffusion processing to the first vector, the modified vector corresponding to a noisy representation of the first vector, said diffusion model being configured to receive, as input, the first vector and to provide, as output, the modified vector,

[0074] - determine, using an inverse diffusion neural network, a second vector by applying an inverse diffusion processing to the modified vector, the second vector corresponding to a denoised representation of the modified vector, said inverse diffusion neural network being configured to receive, as input, the modified vector and to provide, as output, the second vector, and

[0075] - convert, by the first neural network, the second vector into a syn image three-dimensional image having the same dimensions as the three-dimensional input image, the three-dimensional synthetic image representing said part of the subject's body, said part comprising the bony portion.

[0076] In one embodiment, the conversion of the three-dimensional input image into the first vector is performed by the encoding portion of the first neural network.

[0077] In one embodiment, the conversion of the second vector into a three-dimensional synthetic image is performed by the decoding portion of the first neural network.

[0078] In one embodiment, the diffusion model is based on a diffusion function corresponding to one or more Markov chains.

[0079] The invention also provides a computer program comprising instructions executable by a processor and configured so that the processor carries out a method as presented above when these instructions are executed by the processor.

[0080] Finally, the invention provides a computer-readable (non-transitory) medium storing such a computer program.

[0081] Other features and benefits of the method and apparatus disclosed herein will become apparent from the following description of non-limiting embodiments, with reference to the accompanying drawings. Brief description of the drawings

[0082] The present invention is illustrated by way of example, and not limitation, in the figures of the accompanying drawings, in which like reference numerals refer to like elements and in which:

[0083] - [Fig.l] [Fig.l] represents, in a functional form, a device for ge generation of at least one three-dimensional synthetic image from a three-dimensional input image configured to implement a method for generating at least one three-dimensional synthetic image from a three-dimensional input image according to the invention;

[0084] - [Fig.2] [Fig.2] represents an example of the architecture of a neural network ar global materials used in the present invention;

[0085] - [Fig.3] [Fig.3] represents an example of a flowchart of the generation process of at least one three-dimensional synthetic image from a three-dimensional input image according to the invention;

[0086] - [Fig.4a], [Fig.4b], [Fig.4c] Figures 4a, 4b, 4c represent three examples of input images used in the generation method of the invention;

[0087] - [Fig.5] [Fig.5] represents an example of a flowchart of a pre-process treatment according to the invention;

[0088] - [Fig.6] [Fig.6] shows an example of a binary mask obtained when setting implementation of the pre-treatment process of [Fig.5];

[0089] - [Fig.7] [Fig.7] shows an example of a first pre-processed intermediate image obtained after an extraction step of the pre-processing process of [Fig.5];

[0090] - [Fig.8] [Fig.8] shows an example of a second intermediate image preprocessed obtained after the cropping and resizing steps of the preprocessing process in [Fig.5];

[0091] - [Fig.9] [Fig.9] shows an example of a third intermediate image preprocessed obtained after a step of adjusting the spatial orientation of the preprocessing process of [Fig.5];

[0092] - [Fig. 10] [Fig. 10] shows an example of a preprocessed image obtained by the setting implementing the pre-processing processes of [Fig.5]; and

[0093] - [Fig.11a], [Fig.11b], [Fig.11c] Figures 11a to 11c represent examples of synthetic images generated by the generation process of [Fig.3]. DETAILED DESCRIPTION

[0094] The present invention aims to increase the medical data sets that can be used to train neural networks for medical image analysis in order to improve the reliability and efficiency of such neural networks in this medical context.

[0095] In particular, the present invention provides a method for generating three-dimensional synthetic images from three-dimensional input images.

[0096] In order to simplify the description, the “three-dimensional synthetic image” is also referred to as the “synthetic image” in the following. The “three-dimensional input image” is also referred to as the “input image”.

[0097] In the present description, an input image corresponds to a real image acquired during the examination of a patient. The input image is thus an original image obtained from machines intended to carry out medical examinations. For example, the input images are CT scans acquired from computed tomography (CT) sources.

[0098] On the contrary, a synthetic image corresponds to an image generated by the method of the invention. In other words, a synthetic image corresponds to an image obtained by the method of the invention. The synthetic image comprises the characteristics of an input image without being directly obtained by examining a patient. In other words, a synthetic image is an image simulated, by the method of the invention, on the basis of the input image.

[0099] In the following, each image (input image or synthetic image) is represented by a plurality of voxels. A three-dimensional image here means that the image is defined as a matrix. For example, an average size of each image is at least 512 x 512 x 500 voxels. Preferably, the size of each image considered is at least 128 x 128 x 128 voxels.

[0100] In the present description, a "voxel" corresponds to an elementary volume defined in the corresponding three-dimensional image. A voxel is the three-dimensional equivalent in a three-dimensional image of a two-dimensional pixel in a two-dimensional image.

[0101] Here, the input image is for example a CT scan.

[0102] The present invention is particularly advantageous for medical images that include a bony portion of a patient's body. More particularly, it is applicable for complex bony portions, i.e. for bony portions with a plurality of anatomical details. Preferably here, the bony portion is the patient's pelvis (as seen in Figures 4a to 4c). In the following, the invention is illustrated using pelvic images, however the present invention can be applied to each bony portion of a patient's body.

[0103] [Fig.l] represents, in a functional form, an example of a device 1 for generating at least one synthetic image Ims from an input image Imr configured to implement the present invention.

[0104] This device 1 comprises a control unit 2 with a processor 5 and a memory 7. The device 1 may be an image processing device and comprises electronic circuitry for executing its various modules.

[0105] The device 1 is configured to execute a set of functional modules. It comprises for example a training module 10, a pre-processing module 12 and a generation module 14. As visible in [Fig.l], the generation module 14 comprises a diffusion module 15 and a reverse diffusion module 16.

[0106] Each of these modules is for example executed using a processor executing software instructions corresponding to the module concerned and stored in the memory 7 associated with the processor 5 of the control unit 2. Several modules can in this context be executed by the same processor, which can execute different software portions corresponding respectively to the different modules. Alternatively, there could be more than one memory for storing the processor executing software instructions associated with the different modules and more than one processor for executing the processor executing software instructions of the different modules of the device 1. The one or more memories can also be used to store the three-dimensional input images received by the device 1.

[0107] For example, the processor comprises a central processing unit (CPU) and / or a graphics processing unit (GPU). Preferably, the device 1 here comprises a CPU and a GPU.

[0108] According to a possible variant, some of the modules can each be executed by means of a dedicated electronic circuit, such as an application-specific integrated circuit.

[0109] The processor 5 is also configured to implement a global artificial neural network NN (also referred to as global neural network NN in the following), involved in the method of generating the synthetic image Ims from the input image Imr as described in the following.

[0110] [Fig.2] represents an example of the structure of the global neural network NN implemented in the present invention.

[0111] This global neural network NN comprises a first artificial neural network AutoNN, a diffusion model Diff and an inverse diffusion network RDiff. Here, the global neural network NN is defined as a generative model.

[0112] The first artificial neural network AutoNN is configured to provide, as output, a representation (here an Ims image) which corresponds as closely as possible to the representation provided as input (here also an Imr image). In practice, this correspondence is evaluated by minimizing a cost function based on the output representation and on the input representation.

[0113] As visible in [Fig.2], the first artificial neural network AutoNN comprises an encoding part Enc and a decoding part Dec.

[0114] The encoding part Enc is a neural network configured to compress the data it receives as input (here the image Imr) while maintaining reliable characteristics of the data received as input. In other words, the encoding part Enc is configured to transform the input data (here the image Imr) into a representation that is defined in a specific space of lower dimensions (this specific space is usually called "latent space").

[0115] Considering that the encoding part Enc receives, here, an image Imr as input, the encoding part Enc is configured to provide, as output, a vector z of dimensions smaller than those of the image Imr received as input. This vector z (also noted as “latent vector z” in the following) corresponds to another representation of the image Imr received as input. This other representation preserves the most relevant characteristics of the image Imr received as input, that is to say the characteristics which best characterize this image, and with dimensions smaller than those of this image Imr.

[0116] According to an embodiment here, the encoding part Enc receives the image Imr as input. The dimensions of this image Imr are for example at least 128 x 128 x 128 voxels. In practice, the encoding part Enc progressively reduces the dimensions of the image Imr through a series of layers. The encoding part Enc thus produces the latent vector z, which is a projection of the image Imr into space latent. The z vector is thus a latent representation of the Imr image. Here, the dimensions of the latent vector z are, for example, at least 30 x 24 x 30 voxels.

[0117] The operation of the encoding part Enc can be written as follows: Z = cr(WX + b), X representing the image Imr (X is a matrix representation of the image Imr), W a weight matrix associated with the encoding part Enc, b a bias associated with the encoding part Enc and o an activation function. The activation function o is for example a sigmoid function or a Rectified Linear Unit function (commonly noted ReLU).

[0118] The decoding part Dec works in reverse to the encoding part Enc. The decoding part Dec is a neural network configured to construct output data from a vector (denoted £ in [Fig.2]) of lower dimensions. In practice, the decoding part Dec is configured to generate an output that is as close as possible to the input provided to the encoding part Enc. The similarities are evaluated by considering the optimization of a cost function taking into account the output of the decoding part Dec and the input of the encoding part Enc.

[0119] In practice, the decoding part Dec is a neural network which is the transposition of the encoding part Enc.

[0120] As an example here, the first artificial neural network AutoNN is an autoencoder. In other words, the first artificial neural network AutoNN is trained to provide, as output, an image Ims that is as close as possible to the input image Imr. Further details on this type of neural networks can be found in the article “Autoencoders” written by Bank D., Koenigstein N. and Giryes R., 2020.

[0121] As shown in [Fig.2], the global neural network NN also comprises the diffusion model Diff which is configured to implement diffusion processing. The diffusion model Diff is here implemented by the processor 5 of the control unit 2.

[0122] More particularly, this Diff diffusion model is configured to determine a modified vector zT of the latent vector z. In other words, the Diff diffusion model is configured to introduce a noise component eT into the latent vector z. In other words, the Diff diffusion model is configured to perturb the latent vector z with the noise component eT in order to obtain a modified vector zT which represents a noisy representation of the input image Imr.

[0123] Here, the noise component eT is for example a component with a Gaussian distribution.

[0124] The diffusion model Diff is based on a diffusion function which corresponds to a Markov chain applied to the latent vector z.

[0125] Here, the diffusion model Diff is configured to perturb the latent vector z on the based on a composition of diffusion functions applied to the latent vector z. Each diffusion function corresponds to the addition of a noise component eT. Therefore, the diffusion model Diff is configured to perturb the latent vector z by introducing a plurality of noise components eT. In other words, the diffusion model Diff is configured to introduce a plurality of successive noise components eT into the latent vector z.

[0126] The operation implemented (i.e., the composition of diffusion functions) in the Diff diffusion model can be written as follows: p(^)=p^z)... avec des vecteurs intermediate latents zt into which successive noise components eT are introduced, and a transition distribution associated with the introduction of the noise component between the intermediate latent vector zt.i and the following obtained latent vector zt.

[0127] In practice, each introduction of the noise component is implemented here by means of a Markov chain (i.e. each diffusion function corresponds to a Markov chain).

[0128] More particularly, in order to successively introduce a plurality of noise components eT, the diffusion model Diff comprises the application of successive Markov chains. Two successive Markov chains (of the plurality of Markov chains) are separated by a time step. In practice, a Markov chain is applied to the intermediate latent vector zt.i and provides, as output, the other intermediate latent vector zt corresponding to the intermediate latent vector zt.i in which the noise component eT is introduced.

[0129] Here, the diffusion model Diff comprises a number T of time steps. In other words, a Markov chain is applied T times in order to perform the addition of the noise component eT T times. This number T is greater than 100. Preferably, this number T is greater than 500. Even more preferably, the number T is greater than 1000. The use of a plurality of Markov chain applications is very advantageous because it ensures that the latent vector z is “completely destroyed” and replaced by a corresponding noisy representation.

[0130] The diffusion model Diff is coupled to the inverse diffusion network RDiff. The inverse diffusion network RDiff is configured to recover a latent vector associated with the input image Imr from the modified vector zT (associated with the noisy representation). In other words, the inverse diffusion network RDiff is configured to eliminate the noise component eT that was introduced by the diffusion model Diff. In other words, the inverse diffusion network RDiff is configured to denoise the modified vector zT in order to restore the original characteristics of the latent vector z (and the input image Imr).

[0131] In practice, the inverse diffusion network RDiff is configured to apply the inverse transformation (to the modified vector zT) of that which was applied to the latent vector z by the diffusion model Diff, in order to obtain a restored vector 2-

[0132] The inverse diffusion network RDiff comprises a multi-step chain applied to the modified vector zT.

[0133] Here, the inverse diffusion network RDiff is configured to restore the original characteristics of the latent vector z, generating the restored vector z, by recursively eliminating the noise components eT that were introduced by the diffusion model Diff. In practice, the inverse diffusion network RDiff is configured to apply a composition of inverse diffusion functions to the modified vector zT. Each inverse diffusion function corresponds to the elimination of the noise component eT. In other words, the inverse diffusion network RDiff is configured to iteratively eliminate the noise component eT from the modified vector zT. In other words, the modified vector zT is denoised by a plurality of successive denoising operations. The restored vector z is therefore a denoised version of the modified vector zT.

[0134] Each denoising operation aims to eliminate the noise component eT to obtain an intermediate modified vector zTi.

[0135] In practice, each elimination of the noise component is here implemented by means of a convolutional neural network. In other words, in order to successively eliminate a plurality of noise components eT, the inverse diffusion network RDiff comprises a plurality of convolutional neural networks which are executed successively. Each convolutional neural network receives, as input, the intermediate modified vector zTi.i and provides, as output, another intermediate modified vector zTi corresponding to the intermediate modified vector zTi.i in which the noise component eT is eliminated.

[0136] Preferably, the convolutional neural network here is a U-Net network.

[0137] In order to eliminate all the noise components that were introduced by the diffusion model Diff and to recover the original characteristics of the latent vector z (i.e. to compensate for the T applications of the Markov chain(s) that previously introduced the plurality of noise components eT), the inverse diffusion network RDiff comprises the same number T of convolutional neural networks. As previously indicated, this number T is greater than 100. Preferably, this number is greater than 500. Even more desirably, the number T is greater than 1000.

[0138] Finally, the association of the diffusion model Diff and the inverse diffusion network RDiff makes it possible to continuously perturb the latent vector z (representing the main characteristics of the input image Imr) by adding a noise component (here with a Gaussian distribution for example) in order to obtain a modified vector zT of the latent vector z corresponding to a noisy representation of the latent vector z and then to reverse this process to generate the restored vector z which presents the original characteristics of the latent vector z.

[0139] The device 1 and the global neural network NN previously presented are used to implement the method of generating at least one synthetic image Ims from the input image Imr. [Fig.3] is a diagram showing steps of an example method of generating at least one synthetic image Ims from the input image Imr according to the present invention.

[0140] Before implementing the generation method according to the invention, a preliminary method is implemented in order to train the global neural network NN.

[0141] More particularly, this preliminary method is implemented in order to train the different parts of the global neural network NN, i.e. the first artificial neural network AutoNN and the inverse diffusion network RDiff.

[0142] In practice here, the first artificial neural network AutoNN is trained separately from the inverse diffusion network RDiff.

[0143] The first artificial neural network AutoNN is trained using reference images that comprise the relevant bony portion of the subject's body. For example, such reference images comprise the pelvis. These reference images are, for example, images obtained during the examination of patients.

[0144] The preliminary method therefore consists of adjusting the weights of the nodes of the first AutoNN artificial neural network in order to reduce the difference between the images obtained at the output of the first AutoNN artificial neural network (when the reference images are provided as input) and the reference images.

[0145] In practice here, the training of the first artificial neural network AutoNN consists of optimizing loss functions such as the Ll loss function, the perceptual loss function and the KL (Kullback-Leibler) regularization function. The optimization of these loss functions is implemented using an Adam optimizer with, for example, a learning rate equal to 1.10 4.

[0146] Training the first AutoNN neural network involves repeating the weight adjustment a large number of times, while also successively applying a plurality of distinct reference images. A plurality of training epochs are therefore performed to train the first AutoNN neural network. The number of training epochs is, for example, greater than 500, preferably greater than 1000. Performing at least 2000 training epochs makes it possible to determine optimal hyperparameters (such as node weights, batch size, and learning rate) of the first AutoNN neural network.

[0147] The preliminary method is also implemented to train the reverse diffusion network RDiff.

[0148] To this end, the diffusion model Diff and the inverse diffusion network RDiff are executed using reference latent vectors which are associated with reference images which comprise the relevant bone portion of the subject's body. Here, the reference latent vectors are for example associated with the reference images used to train the first artificial neural network AutoNN (as described previously). Advantageously according to the present invention, the reference latent vectors are associated with the pelvis. The global neural network NN is therefore optimally trained to manage images representing the subject's pelvis. Alternatively, if another bone portion is considered, the reference images are adapted to this other bone portion considered.

[0149] The preliminary method therefore consists of adjusting the weights of the nodes of the inverse diffusion network RDiff in order to reduce the distance between the restored vectors obtained at the output of the inverse diffusion network RDiff (when the reference latent vectors are provided as input to the Diff diffusion model) and the corresponding reference latent vectors. In other words, during the preliminary method, the weights of the nodes of the inverse diffusion network RDiff are adjusted so that the restored vectors (obtained at the output of the inverse diffusion network RDiff) converge towards the reference latent vectors (provided as input to the Diff diffusion model).

[0150] Training the inverse diffusion network RDiff involves repeating the weight adjustment many times, while also successively applying a plurality of distinct reference modified vectors as input to the inverse diffusion model Diff (the reference modified vectors are for example associated with corresponding reference latent vectors). A plurality of training epochs are thus performed to train the diffusion model Diff and the inverse diffusion network RDiff. The number of training epochs is greater than 500, preferably greater than 1000. Performing at least 2000 training epochs makes it possible to determine optimal hyperparameters (such as node weights) of the inverse diffusion network RDiff.

[0151] The method for generating a synthetic image Ims from an input image Imr (also referred to as “generation method” in the following) is thus implemented following the preliminary method presented.

[0152] The generation method is for example implemented by the processor 5 of the control unit 2. Generally, the generation method is implemented by computer.

[0153] As shown in [Fig.3], the generation method begins with a Rec step of receiving at least one input image Imr. This input image Imr re shows a part of a subject's body that includes a bony portion 50 of the body. Here, the bony portion 50 is the subject's pelvis 50. Figures 4a to 4c show three examples of input images representing the subject's pelvis 50.

[0154] In practice, a plurality of input images Imr are received by the processor 5 of the control unit 2. This plurality of input images Imr is for example stored in the memory 7 of the control unit 2. For the sake of clarity, the generation method is described by considering a single input image Imr but this generation method applies in the same way to all the input images Imr of the plurality of input images Imr.

[0155] The generation method then comprises a pre-processing process Pre-Proc of the input image Imr. The pre-processing process Pre-Proc aims to process the input image Imr in order to make the following steps of the generation method more efficient. The pre-processing process Pre-Proc is here configured to obtain a pre-processed version of the input image Imr (also called pre-processed image Imr' in the following). [Fig.5] is a diagram showing the successive steps of an example of the pre-processing process Pre-Proc according to the present invention.

[0156] As visible in [Fig.5], the pre-processing process Pre-Proc firstly comprises a step Ext of extracting the part comprising the bony portion 50 (here the pelvis) so that the pre-processed image Imr' is focused on this part of the body comprising the bony portion 50.

[0157] In practice, this extraction step Ext comprises a step Seg of segmenting the input image Imr in order to identify the part comprising the bone portion 50. In practice, this segmentation step Seg is implemented using a segmentation convolutional neural network. This segmentation convolutional neural network is configured to identify different anatomical regions such as soft tissues and bone portions and highlight the bone portions.

[0158] The segmentation convolutional neural network is here configured to receive the input image Imr and to provide, as output, a plurality of multi-class masks. The formulation “multi-class” here means that each class is associated with a corresponding anatomical region. The segmentation convolutional neural network is thus configured to predict a multi-class mask corresponding to bone portions (for example here in the pelvis).

[0159] In a practical implementation, the segmentation convolutional neural network here is a U-Net network.

[0160] It is noted here that the segmentation convolutional neural network may also be trained before implementing the generation method. For example, the segmentation convolutional neural network is trained during the preliminary method described above.

[0161] Advantageously according to the present invention, the segmentation convolutional neural network is trained using the reference images introduced previously. Such reference images advantageously include the pelvis. These reference images are for example images obtained during the examination of patients.

[0162] As visible in [Fig.5], the pre-processing process Pre-Proc then comprises a Mask step for determining a binary mask corresponding to the part comprising the bone portion. This binary mask is obtained on the basis of the multi-class mask predicted (for the bone portions) by the segmentation convolutional neural network during the segmentation step Seg. In other words, here, the processor 5 of the control unit 2 converts the multi-class mask predicted for the bone portions into a binary mask. This step makes it possible to determine a mask focusing only on the bone portion (here the pelvis). [Fig.6] shows an example of such a binary mask.

[0163] Finally, the extraction step Ext comprises a step Gen of processing an intermediate pre-processed image Iml on the basis of this determined binary mask. More particularly, the processor 5 of the control unit 2 generates the intermediate pre-processed image Iml by applying the determined binary mask to the input image Imr. More particularly, the intermediate pre-processed image Iml is obtained by multiplying the determined binary mask and the input image Imr. This step Gen makes it possible to eliminate the other anatomical regions which are not of interest here (since the generation method focuses on the bone portions). In other words, the bone portion is extracted thanks to the multiplication of the determined binary mask and the input image Imr.

[0164] By means of this extraction step Ext, the input image Imr is modified in order to be focused on this part of the body comprising the bone portion 50 (as indicated, the other anatomical regions are eliminated from the preprocessed image Imr'). [Fig.7] shows an example of an intermediate preprocessed image Iml obtained after the extraction step Ext.

[0165] As shown in [Fig.5], the pre-processing process Pre-Proc then comprises a Crop step of cropping the intermediate pre-processed image Iml obtained after the extraction step Ext. During this step, the processor 5 of the control unit 2 performs a central cropping of the intermediate pre-processed image Iml obtained after the extraction step Ext. In other words, the intermediate pre-processed image Iml is cut in order to preserve the representation of the bone portion and of tens of voxels around the bone portion 50 (in order to obtain the desired dimensions for the pre-processed image Imr', for example the dimensions of 128 x 128 x 128 voxels). This Crop cropping step aims to eliminate the irrelevant parts of the intermediate pre-processed image Iml obtained after the extraction step Ext and then allows you to reduce the size of the cropped Imc image.

[0166] The pre-processing process Pre-Proc then comprises a Size step of resizing the cropped image Imc in order to standardize the size of the images that are involved in the generation process. As an example, the processor 5 of the control unit 2 resamples the cropped image Imc to a uniform size of 128 x 128 x 128 voxels (using a standard voxel spacing of 1.5 x 1.5 x 1.5). [Fig.8] shows an example of another intermediate pre-processed image Im2 obtained after the cropping and resizing steps. The frame shown in Figures 7 and 8 illustrates the reduction of the image after the cropping and resizing steps.

[0167] As visible in [Fig.5], the pre-processing process Pre-Proc also includes an Orient step for adjusting the spatial orientation of the part comprising the bone portion 50. More particularly, the processor 5 of the control unit 2 adjusts the spatial orientation of the part comprising the bone portion 50 in order to obtain a predetermined orientation of this part. This spatial orientation is adjusted in the standard RAS (right anteroposterior) orientation system. More explicitly, this RAS orientation system means that the first dimension points to the right side of the bone portion (here the pelvis), the second dimension points to the anterior face of the bone portion, and the third dimension points to the top of the bone portion. The directions are considered to be from the subject's perspective.Figures 4a to 4c show examples of input images (the bony portion being the pelvis) in three different anatomical orientations.

[0168] This Orient adjustment step aims to adjust the orientation of the images in order to ensure the consistency of the spatial alignment when implementing the generation method, thus improving the efficiency of this generation method. [Fig.9] shows an example of an intermediate preprocessed image Im3 obtained after the Orient step of adjusting the spatial orientation.

[0169] Finally, the pre-processing process Pre-Proc includes a Norm step for normalizing the intensity of the intermediate pre-processed image Im3. This Norm normalization step aims to scale the intensity associated with the voxels of the intermediate pre-processed image Im3 between 0 and 1. In practice, very high intensities are filtered using a known filter or filters in order to eliminate potential anomalies and artifacts.

[0170] The preprocessed image Imr' is obtained after this normalization step Norm. [Fig. 10] shows an example of the preprocessed image Imr'.

[0171] The pre-processing process is particularly advantageous because it allows the standardization of input images acquired by different examination systems, which do not have the same format or the same acquisition characteristics. This standardization of the input images helps improve the efficiency of the generation process implementation.

[0172] Furthermore, the pre-processing process makes it possible to reduce the size of the images which are stored in the memory 7 of the control unit 2 and which are processed in the generation method, which makes it possible to preserve memory and computing resources.

[0173] It should be noted that the preprocessing process is optional. If it is implemented, the following steps of the generation method are implemented on the preprocessed image Imr'. If it is not implemented, the following steps of the generation method are directly implemented on the input image Imr.

[0174] As shown in [Fig.3], the generation method then comprises a step S4 of converting the input image Imr (or the preprocessed image Imr') into a latent vector z. This conversion step S4 is implemented using the encoding part Enc of the first artificial neural network AutoNN described previously. The latent vector z thus has dimensions smaller than the dimensions of the input image Imr (or the preprocessed image Imr'). Here, the dimensions of the latent vector z are for example 30 x 24 x 30 voxels (while the dimensions of the input image Imr are 512 x 512 x 500 voxels and the dimensions of the preprocessed image Imr' are 128 x 128 x 128 voxels).

[0175] The generation method then continues with a step S6 of determining a modified vector zT by applying a diffusion processing to the latent vector z. The diffusion processing here uses the diffusion model Diff (which is implemented by the processor 5 of the control unit 2). As described previously, the diffusion processing aims to add a noise component eT to the latent vector z in order to obtain a noise representation of this latent vector z.

[0176] More particularly here, as described previously, the diffusion process comprises the composition of diffusion functions which is applied to the latent vector z . Each diffusion function corresponds to the addition of the noise component eT. Each diffusion function is here implemented by means of a Markov chain (as described previously). As indicated previously, the noise component eT is for example here a component with a Gaussian distribution.

[0177] Therefore, after this determination step S6, the output of the diffusion module is a noisy representation of the latent vector z (this noisy representation is represented by the modified vector zT).

[0178] The generation method then comprises step S8 of determining the restored vector z by applying an inverse diffusion processing to the modified vector zT. The inverse diffusion processing is here implemented by the inverse diffusion network RDiff. As described previously, the inverse diffusion processing aims to eliminate recursively the noise components eT that were introduced by the diffusion model Diff in order to restore the original characteristics of the latent vector z (thus generating the restored vector 2)- During this inverse diffusion processing, the modified vector zT is denoised by a plurality of successive denoising operations. The restored vector z is thus a denoised representation of the modified vector zT.

[0179] More particularly here, as previously described, the inverse diffusion process comprises the composition of inverse diffusion functions which is applied to the modified vector zT. Each inverse diffusion function corresponds to the elimination of the noise component eT. Each inverse diffusion function is here implemented by a corresponding convolutional neural network (previously described).

[0180] Therefore, after this determination step S8, the output of the inverse diffusion module is the restored vector z which includes the original characteristics of the latent vector z. The restored vector z has the same dimensions as the latent vector z.

[0181] As shown in Figure 3, the generation method finally comprises a step S10 of converting the restored vector z into a synthetic image Ims. This conversion step S10 is implemented using the decoding part Dec of the first artificial neural network AutoNN described previously. The synthetic image Ims has the same dimensions as the input image Imr (or the preprocessed image Imr'). Here, the dimensions of the synthetic image Ims are for example 512 x 512 x 500 voxels (or 128 x 128 x 128 voxels).

[0182] Advantageously according to the present invention, the synthetic image Ims represents the part of the subject's body which comprises the bony portion 50. Here, the synthetic image Ims represents the pelvis 50. Figures 1 la to 1 le represent examples of synthetic images generated respectively from the input images represented in Figures 4a to 4c.

[0183] By comparing the input images of Figures 4a to 4c with the respectively generated synthetic images shown in Figures 11a to 11e, it can be seen that the method of the invention makes it possible to preserve the anatomical details of the pelvis. Therefore, starting from a real image comprising the bone portion 50, the invention makes it possible to generate a corresponding synthetic image which preserves the main characteristics of the real image. It makes it possible to preserve the spatial structure of medical images. This point is crucial in medical imaging because the spatial arrangement of bone structures is essential for performing accurate diagnosis and analysis. According to the inventors' knowledge, the method of the invention is the first to be developed to efficiently generate synthetic images of the pelvis.

[0184] The method according to the invention is particularly advantageous for medical images which are CT scans. Indeed, CT scans are three-dimensional images which generally do not have a resolution as good as that of MRI images for example. Consequently, it is difficult to generate high-quality synthetic images from real CT scans (high-quality images here correspond to images in which all the anatomical details are preserved).

[0185] According to the invention, the generation method makes it possible to generate synthetic images from CT scans which preserve all the anatomical details of the bone portion concerned (here the pelvis). In other words, the method of the invention makes it possible to generate three-dimensional synthetic images which faithfully reproduce the anatomical details of the bone portion concerned, even if the resolution of the input images is not very high.

[0186] Advantageously, the method according to the invention makes it possible to preserve all the main characteristics of the input images while reducing the size (related to the memory) of the generated synthetic images by at least one hundred times. For example, the input images have a size of approximately 110 MB while the size of the corresponding synthetic images is approximately 820 kB.

[0187] Furthermore, in practice, due to diffusion processing and inverse diffusion processing (and the successive addition and removal of noise components), a single input image may be used multiple times to generate a plurality of different synthetic images that will preserve the main characteristics of the input image.

[0188] The similarities between the input images and the corresponding generated synthetic images can be quantitatively assessed by statistical analysis. For example, the Kullbach-Leibler (KL) divergence values ​​and the Euclidean distance values ​​are evaluated for several pairs of input images and corresponding generated synthetic images. It should be noted here that the generation method was implemented using more than 430 input images (here CT scans) from medical examinations performed on patients.

[0189] For example, for five pairs, the KL divergence values ​​are between 0.00511 and 0.01071. The KL divergence values ​​are thus close to zero, suggesting that the voxel distributions between the input (i.e., original) image and the synthetic image are slightly different from a statistical point of view. This indicates that the distribution of voxel values ​​in the generated synthetic image is very close to that of the real image.

[0190] If we consider the Euclidean distance for the same five couples, the values ​​of this Euclidean distance are between 0.12656 and 0.18573. These values ​​of the Euclidean distance illustrates relatively high proximities between voxels of generated images and those of real images.

[0191] Such statistical analyses demonstrate the high efficiency of the method of the invention for generating synthetic images very similar to the real images which are used for their generation. The generated synthetic images have the advantage of preserving the anatomical details and the main characteristics of the original images.

[0192] Advantageously, the generated synthetic images can thus be used to enrich the training data sets in order to effectively train the neural networks which are used in the medical field.

Claims

1.

2.

3. Claims Method for generating at least one three-dimensional synthetic image (Ims) from a three-dimensional input image (Imr), the method comprising: - receiving (S2) a three-dimensional input image (Imr) representing a part of a body of a subject, said part comprising a bone portion (50), - conversion (S4) of the three-dimensional input image (Imr) into a first vector (z) which has dimensions smaller than the dimensions of the three-dimensional input image (Imr), - determination (S6), using a diffusion module, of a modified vector (zT) by applying a diffusion processing to the first vector (z), the modified vector (zT) corresponding to a noisy representation of the first vector (z), said diffusion module being configured to receive, as input, the first vector (z) and to provide, as output, the modified vector (zT), - determination (S8), using an inverse diffusion module, of a second vector (z) by applying an inverse diffusion processing to the modified vector (zT), the second vector corresponding to a denoised representation of the modified vector (zT), said inverse diffusion module being configured to receive, as input, the modified vector (zT) and to provide, as output, the second vector (2), and - converting the second vector (2) into a three-dimensional synthetic image (Ims) which has the same dimensions as the three-dimensional input image (Imr), the three-dimensional synthetic image (Ims) representing said part of the subject's body, said part comprising the bone portion (50). The method of claim 1, wherein the bony portion (50) is the pelvis. Method according to claim 1 or 2, wherein the conversion of the three-dimensional input image (Imr) into the first vector (z) is implemented using an encoding part (Enc) of an artificial neural network (AutoNN), the encoding part (Enc) being configured to receive, as input, the three-dimensional input image (Imr) and to provide, as output, the first vector (z), and wherein the conversion of the second vector (z) into a three-dimensional synthetic image (Ims) is implemented using a part of decoding (Dec) of the artificial neural network (AutoNN), the decoding part (Dec) being configured to receive, as input, the second vector (2) and to provide, as output, the three-dimensional synthetic image (Ims).

4. The method of claim 3, wherein the artificial neural network (AutoNN) is an autoencoder.

5. A method according to any one of claims 1 to 4, wherein the diffusion processing comprises a composition of diffusion functions applied to the first vector (z), each diffusion function corresponding to the addition of a noise component, and wherein the inverse diffusion processing comprises a composition of inverse diffusion functions applied to the modified vector (zT), each diffusion function corresponding to the elimination of the noise component.

6. The method of claim 5, wherein the noise component is a component with a Gaussian distribution.

7. The method of claim 5 or 6, wherein each diffusion function corresponds to a Markov chain, the diffusion processing comprising applying a plurality of successive Markov chains, two successive Markov chains of the plurality of successive Markov chains being separated by a time step, the diffusion processing being associated with a number T of time steps, and wherein the inverse diffusion module comprises a convolutional neural network configured to implement a corresponding inverse diffusion function.

8. Method according to claim 7, in which the number T of time steps is greater than 100, preferably greater than 500, and even more preferably greater than 1000.

9. Method according to any one of claims 1 to 8, further comprising, before the conversion (S4) of the three-dimensional input image, a pre-processing (Pre-Proc) of the three-dimensional input image (Imr), in order to obtain a pre-processed three-dimensional image (Imr'), the first vector (z) being obtained by conversion of said pre-processed three-dimensional image (Imr').

10. Method according to claim 9, wherein the pre-processing (Pre-Proc) of the three-dimensional input image (Imr) comprises an extraction (Ext) of the part comprising the bone portion (50) so that the pre-processed three-dimensional image (Imr') is focused on said part comprising the bone portion (50).

11. Method according to claim 10, wherein the extraction (Ext) of the part comprising the bone portion (50) comprises: - segmentation (Seg) of the three-dimensional input image (Imr) in order to identify said part comprising the bone portion (50), - determination (Mask) of a mask corresponding to said part comprising the bone portion (50) on the basis of the segmented three-dimensional input image, and - generation (Gen) of an intermediate pre-processed three-dimensional image by applying said mask to the three-dimensional input image (Imr).

12. The method of claim 11, wherein the pre-processing (Pre-Proc) of the three-dimensional input image (Imr) comprises cropping (Crop) and resizing (Size) of the intermediate pre-processed three-dimensional image.

13. A method according to any one of claims 10 to 12, wherein the pre-processing (Pre-Proc) of the three-dimensional input image (Imr) comprises performing (Orient) an adjustment of a spatial orientation of the intermediate pre-processed three-dimensional image.

14. Device (1) for generating at least one three-dimensional synthetic image (Ims) from a three-dimensional input image (Imr), the device (1) comprising a control unit (2) configured to: - receive a three-dimensional input image (Imr) representing a part of a body of a subject, said part comprising a bone portion (50), - convert the three-dimensional input image (Imr) into a first vector (z) which has dimensions smaller than the dimensions of the three-dimensional input image (Imr), - determine, using a diffusion module, a modified vector (zT) by applying a diffusion processing to the first vector (z), the modified vector (zT) corresponding to a noisy representation of the first vector (z), said diffusion module being configured to receive, as input, the first vector (z) and to provide, as output, the modified vector (zT), - determine, using an inverse diffusion module,a second vector (2) by applying an inverse diffusion processing to the modified vector (zT), the second vector (2) corresponding to a denoised representation of the modified vector (zT), said diffusion module, inverse being configured to receive, as input, the modified vector (zT) and to provide, as output, the second vector (2), and - convert the second vector (z) into a three-dimensional synthetic image (Ims) which has the same dimensions as the three-dimensional input image (Imr), the three-dimensional synthetic image (Ims) representing said part of the subject's body, said part comprising the bone portion (50).

15. A computer program comprising instructions executable by a processor (5) and configured such that the processor (5) performs a method according to any one of claims 1 to 13 when these instructions are executed by the processor (5).

16. A computer-readable medium storing a computer program according to claim 15.

Images