Method for training a model for semantic segmentation of medical images

Abstract

The invention relates to a method for training a segmentation neural network for segmenting anatomical structures in a medical image. The segmentation neural network (30) is trained in a supervised manner (107) using supervised training data (25) that comprises segmented real medical images (11, 12) and their associated segmentation masks (21, 22), as well as synthetic images (13) generated from artificial segmentation masks (23). In parallel, a teacher network (30'), initially corresponding to a clone of the segmentation neural network, is updated using an exponential moving average. The teacher network is used to generate (110) segmentation pseudo-masks (24) from semi-supervised training data (26) comprising unsegmented real medical images (14) that are each associated with a weak annotation. The unsegmented real medical images and their associated segmentation pseudo-masks are added (111) to the supervised training data to continue training the segmentation neural network.

Description

[0001] Training method for a semantic segmentation model of medical images

[0002] Field of invention

[0003] The present application belongs to the field of semantic segmentation of medical images. In particular, a method for training a segmentation neural network is proposed, as well as a computer-readable storage medium comprising a segmentation neural network trained by such a method.

[0004] State of the art

[0005] Machine learning algorithms for semantic segmentation of medical images are generally implemented by artificial neural networks. In particular, it is known that training a convolutional neural network based on a U-net type model allows semantic image segmentation to be performed. Semantic segmentation involves labeling each pixel of an image with a class corresponding to what is represented in the image. In the field of medical images, a class can represent a particular anatomical structure, such as an anatomy of interest (this can be an organ such as the liver, a lung or a kidney, or another anatomical structure such as a bone or a blood vessel), an anomaly within this anatomy of interest (for example a cyst, a tumor or an ablation zone), or a surgical artifact (for example a part of a medical instrument).

[0006] Training a segmentation neural network requires a large amount of training data to achieve good prediction quality. Also, underrepresentation of a class in the training data can produce a significant prediction bias against that class.

[0007] This problem is particularly important in the medical field. Indeed, to train the neural network in a supervised manner, the training data must contain a large number of segmented medical images. Each medical image must then be segmented manually, or possibly semi-automatically, by a medical expert. This segmentation of medical images is particularly costly in terms of time and expertise.

[0008] It is even more difficult to collect a large number of medical images showing rare anatomical abnormalities (due to the rarity of diseases, patient confidentiality, the effort and expense of conducting medical imaging operations, etc.).

[0009] The paper “Deep semi-supervised segmentation with weight-averaged consistency targets,” Christian S. PERONE et al., describes a method for semi-supervised training of a segmentation neural network using the “Mean Teacher” approach.

[0010] Several prior art solutions are also based on a relatively similar approach in which the segmentation neural network that one seeks to train is reproduced to form a teaching network. In particular, patent applications WO 2021 / 140426 A1, US 2022 / 0292689 A1, US2021 / 0407656 A1 and US2022 / 0358658 A1 describe different methods for training a medical image segmentation neural network.

[0011] However, these solutions have several drawbacks, including overfitting of the segmentation algorithm on supervised training data, sometimes insufficient segmentation performance, learning biases against the segmentation of rare classes, and / or an insufficient level of involvement of medical expertise in the training of the segmentation neural network.

[0012] Patent application WO 2022 / 238640 A1 describes a method for generating synthetic images showing rare anatomical anomalies using generative adversarial networks.

[0013] Statement of the invention

[0014] The present application aims to propose a solution to all or part of the drawbacks of the prior art, in particular those set out above.

[0015] To this end, and according to a first aspect, a method is proposed for training a segmentation neural network to segment anatomical structures on a medical image representing an anatomy of interest of a patient. The method comprises the following steps:

[0016] - obtaining segmented real medical images, each associated with a segmentation mask, including: o majority real medical images associated with majority segmentation masks and representing the anatomy of interest of a patient without anomaly, o minority real medical images associated with minority segmentation masks and representing the anatomy of interest of a patient with an anomaly,

[0017] - training a generator neural network to generate a synthetic image from a segmentation mask,

[0018] - obtaining artificial segmentation masks, each artificial segmentation mask being obtained by combining a majority segmentation mask with a segmentation of an anomaly of a minority segmentation mask,

[0019] - obtaining synthetic images, from artificial segmentation masks, using the trained generator neural network,

[0020] - obtaining real, non-segmented medical images, each representing the anatomy of interest of a patient and each associated with a weak annotation,

[0021] - an identical reproduction of the segmentation neural network to form a teaching network.

[0022] The method further involves several iterations of the following steps:

[0023] - supervised training of the segmentation neural network from supervised training data comprising real segmented medical images and their associated segmentation masks, as well as synthetic images and their associated artificial segmentation masks,

[0024] - an update of the weights of the gradient descent segmentation neural network,

[0025] - an update of the w weights T of the teacher network by an exponential moving average calculated based on the previous weights w Tprec of the teaching network and weights updated segmentation neural network,

[0026] - a generation of pseudo-segmentation masks corresponding to predictions of the teaching network obtained from semi-supervised training data comprising real non-segmented medical images and their associated weak annotations,

[0027] - an addition to the supervised training data of the generated pseudo-segmentation masks and the real unsegmented medical images which allowed their generation.

[0028] The construction of artificial segmentation masks and the generation of associated synthetic images makes it possible to limit overfitting of the segmentation neural network. Indeed, it is possible to construct an infinite number of different artificial segmentation masks by a random combination of majority segmentation masks with minority segmentation masks, and possibly by a transformation of the segmentation of an anomaly of a minority segmentation mask before its combination with a majority segmentation mask. Each synthetic image thus generated can then be used only once in the training of the segmentation algorithm.

[0029] Real medical images can be segmented by medical experts. Weak annotations can also be provided by medical experts. This ensures a sufficient level of involvement of medical expertise in training the segmentation neural network.

[0030] A weak annotation is significantly less expensive to produce (in time and expertise) than a segmentation mask. A weak annotation corresponds to the marking of one or more pixels of the image. It can correspond to an approximate segmentation (of low precision compared to the segmentations of the majority or minority segmentation masks) or to a simple localization of a structure to be segmented. The use of weak annotations makes it possible to refine the predictions of the pseudo-segmentation masks made by the teaching network.

[0031] Updating the weights of the teacher network using an exponential moving average (EMA) allows the accumulation of different versions of the weights of the segmentation neural network (which acts as a student network) over time. The "conservative" predictions of the teacher network thus remain slightly different, which has a regularizing effect on the training of the student network (the neural network that we are trying to train) and limits overfitting (in other words, we are trying to avoid the predictions of the student network being too close to certain segmentation masks that are too specific to the supervised training data).

[0032] The combination in the supervised training data of real segmented medical images, synthetic images with rare anomalies, and real unsegmented medical images for which pseudosegmentation has been generated, makes it possible to obtain very good segmentation performances at the end of training.

[0033] In particular embodiments, the training method may further comprise one or more of the following characteristics, taken individually or in all technically possible combinations.

[0034] In particular modes of implementation, the method comprises, at each iteration:

[0035] - a calculation of a supervised cost function C srepresentative of a dissimilarity between a prediction of the segmentation neural network, made from an image belonging to the supervised training data, and the segmentation mask associated with this image,

[0036] - a calculation of a semi-supervised cost function C ss representative of a dissimilarity between a prediction of the teaching network and a prediction of the segmentation neural network made for an image belonging to the semi-supervised training data,

[0037] - a calculation of a global cost function C as a function of the supervised cost function C s and the semi-supervised cost function C ss ,

[0038] - weight update of the gradient descent segmentation neural network being performed based on the global cost function.

[0039] The combination of the supervised cost function with the semi-supervised cost function contributes to the good segmentation performance obtained after training.

[0040] In particular implementation modes, a prediction of the teaching network carried out on an image belonging to the semi-supervised training data is enriched with the weak annotation associated with said image. The enriched prediction, noted y T ' , can be written in the form y T ' = (1 - ^)y T + py F , expression in which y T is a raw prediction from the teacher network, y F corresponds to the weak annotation, and p is a contribution ratio to the weak annotation.

[0041] In particular implementations, the contribution ratio p of the weak annotation is between 0.45 and 0.55.

[0042] This combination of weak annotations with the predictions of the teaching network makes it possible to refine the generation of pseudo-segmentation masks.

[0043] In particular embodiments, each weak annotation comprises an approximate location of an anatomical structure to be segmented. This approximate location may be represented by one or more points of the anatomical structure to be segmented, a geometric shape covering a region within the anatomical structure to be segmented, or a geometric shape surrounding or covering the anatomical structure to be segmented.

[0044] In particular implementations, the global cost function C is calculated as a weighted sum of the supervised cost function C s and the semi-supervised cost function C ss . In particular implementations, the overall cost function is written in the form C = (1 - y)Cs + yC ss , with y between 0.05 and 0.15.

[0045] In particular modes of implementation, the supervised cost function C s is calculated as a linear combination of a real cost function C Re calculated from real segmented medical images and their associated segmentation masks and a synthetic cost function C Syn calculated from synthetic images and their associated artificial segmentation masks.

[0046] In particular implementations, the overall cost function C is written in the form C = (1 - + apC Syn + yC ss , expression in which y is a contribution ratio of the semi-supervised cost function C ss , a is a contribution ratio of the synthetic cost function C Syn, and p is a realism score of synthetic images calculated using a discriminator neural network forming with the generator neural network a pair of generative adversarial networks.

[0047] In particular modes of implementation, the contribution ratio a is between 0.35 and 0.45 and the contribution ratio y is between 0.05 and 0.15.

[0048] In particular modes of implementation, obtaining artificial segmentation masks involves a transformation of the segmentation of the anomaly of the minority segmentation mask.

[0049] In particular embodiments, the transformation of the anomaly segmentation corresponds to a rotation, an enlargement, a reduction, a deformation and / or a displacement of the anomaly segmentation.

[0050] According to a second aspect, a method for automatically segmenting a medical image is provided. This automatic segmentation method comprises a step of training a segmentation neural network according to any of the preceding implementation modes, then using the trained segmentation neural network to segment the medical image.

[0051] According to a third aspect, there is provided a computer-readable storage medium comprising a segmentation neural network trained according to any of the preceding embodiments.

[0052] Presentation of figures

[0053] The invention will be better understood by reading the following description, given by way of non-limiting example, and made with reference to figures 1 to 12 which represent:

[0054] [Fig. 1] a schematic representation of the main steps of an implementation mode of a method for training a segmentation neural network according to the invention,

[0055] [Fig. 2] A schematic representation of the training of a generative neural network to generate synthetic images from a segmentation mask,

[0056] [Fig. 3] a schematic representation of the method for training a segmentation neural network according to the invention,

[0057] [Fig. 4] a schematic representation of the calculation of a supervised cost function representative of a dissimilarity between a prediction of the segmentation neural network and the segmentation mask associated with the image from which the prediction is made,

[0058] [Fig. 5] a schematic representation of the calculation of a semi-supervised cost function representative of a dissimilarity between a prediction of the teaching network and a prediction of the segmentation neural network,

[0059] [Fig. 6] a schematic representation of updating the segmentation neural network weights and the teaching network weights from a global cost function calculated based on the supervised cost function and the semi-supervised cost function,

[0060] [Fig. 7] a schematic representation of a step of obtaining real medical images and their associated segmentation masks,

[0061] [Fig. 8] a schematic representation of a step of obtaining artificial segmentation masks from majority segmentation masks and minority segmentation masks,

[0062] [Fig. 9] a schematic representation of a step of obtaining artificial segmentation masks with transformation of the segmentation of the anomaly,

[0063] [Fig. 10] an illustration of obtaining an artificial segmentation mask from a majority segmentation mask and a minority segmentation mask, [Fig. 1 1] an illustration of a real medical image, its associated segmentation mask, and a synthetic medical image generated by the generator neural network from the segmentation mask, for a majority case (at the top of the figure) and for a minority case (at the bottom of the figure),

[0064] [Fig. 12] An illustration, for three different majority real medical images, of the majority real medical image, its associated majority segmentation mask, an artificial segmentation mask generated from the majority segmentation mask, and a synthetic medical image generated by the generator neural network from the artificial segmentation mask. In these figures, identical references from one figure to another designate identical or similar elements. For reasons of clarity, the elements represented are not necessarily to the same scale, unless otherwise indicated.

[0065] Detailed description of the invention

[0066] Figures 1 and 3 schematically represent the main steps of an implementation mode of a method 100 for training a segmentation neural network 30 to segment anatomical structures on a medical image representing an anatomy of interest of a patient.

[0067] In the present application, a “neural network” corresponds to a model, implemented by a computer, whose operation is inspired by that of neurons in the human brain. It is a variety of deep learning technology, which itself is part of machine learning algorithms. Machine learning algorithms form a category in the field of artificial intelligence.

[0068] The anatomy of interest may correspond to an organ (e.g., liver, pancreas, gallbladder, lung, or kidney) or another anatomical structure (e.g., bone or blood vessel). An anomaly within the anatomy of interest generally corresponds to a lesion, such as a tumor, cyst, ablation zone, aneurysm, etc. An ablation zone corresponds to a lesion that has undergone ablation treatment using a known method (microwave, laser, radiofrequency, etc.); it is a necrotic region. The anomaly may also correspond to a surgical artifact, such as part of a medical instrument.

[0069] The segmentation neural network 30 is trained to take as input a real medical image and to provide as output a segmentation mask associated with this image. A segmentation mask is an image in which each voxel provides particular information about an element represented at the position of the voxel on the real medical image. A voxel can thus take a particular numerical value associated with the element represented at the position of the voxel on the real medical image. Different specific numerical values ​​are for example defined respectively for a healthy part of the anatomy of interest, for an anomaly within the anatomy of interest, for other anatomical structures (for example bones or blood vessels), for the background of the image, etc. Different numerical values ​​thus represent different classes of elements visible on the real medical image.It should be noted that the term "voxel" is used generically to define a particular area of ​​an image (a voxel identifies a position of said area on the image and takes a value representative of what is represented in said area on the image). It can be a two-dimensional or three-dimensional image. If it is a two-dimensional image, the term "voxel" then takes on the same meaning as the term "pixel".

[0070] A "real medical image" means a medical image of a patient acquired using a medical imaging device, for example, computed tomography (CT), positron emission tomography (PET), magnetic resonance imaging (MRI), ultrasound, or X-ray.

[0071] The segmentation neural network 30 takes the form, for example, of a U-Net convolutional neural network. The segmentation neural network 30 may also be based on a model derived from the U-Net architecture, such as the RITnet model, which combines the U-Net and DenseNet models (see “RITnet: Real-time Semantic Segmentation of the Eye for Gaze Tracking,” A. K. Chaudhary et al.). However, the segmentation neural network 30 could also be based on other models. For example, it could be a transformer neural network such as the “Swin Transformer” network (see, for example, “Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images,” A. Hatamizadeh et al.).

[0072] As illustrated in Figures 1 and 3, the method 100 comprises a step 101 of obtaining segmented real medical images 11, 12 each associated with a segmentation mask 21, 22. These real medical images comprise: o majority real medical images 11 associated with majority segmentation masks 21 and representing the anatomy of interest of a patient without anomaly, o minority real medical images 12 associated with minority segmentation masks 22 and representing the anatomy of interest 41 of a patient with an anomaly.

[0073] The terms "majority" and "minority" are used because there are generally a significantly greater number of medical images depicting anatomy of interest without abnormality than medical images depicting anatomy of interest with abnormality.

[0074] Figure 7 schematically represents the step of obtaining 101 real medical images 11, 12 and their associated segmentation masks 21, 22. On the left of Figure 7 is illustrated the generation of majority segmentation masks 21 from majority real medical images 11. The anatomy of interest 41 is visible on a majority real medical image 11. On a so-called “majority” real medical image, there is no anomaly within the anatomy of interest. A majority segmentation mask 21 comprises a segmentation of the anatomy of interest.

[0075] On the left of Figure 7 is illustrated the generation of minority segmentation masks 22 from minority real medical images 12. The anatomy of interest 41 and an anomaly 42 present within the anatomy of interest 41 are visible on a minority real medical image 12. A minority segmentation mask 22 comprises a segmentation of the anatomy of interest and a segmentation 44 of the anomaly.

[0076] Segmentation can be implemented manually by a medical expert. In this case, the medical expert defines the contours of the different anatomical regions on the medical image using a graphical interface (mouse, stylus, touch screen, etc.) of an electronic device (computer, tablet, etc.) on which the image is displayed. Segmentation can also be implemented semi-automatically, using a conventional image segmentation algorithm. For example, bones visible on medical images can be segmented by the voxel intensity thresholding method with an intensity window of width 1800 HU and center 400 HU (HU is the acronym for "Hounsfield Unit", it is a quantitative scale describing radio-density, i.e. a unit of measurement representative of the opacity of a material to X-rays).

[0077] As illustrated in Figures 1 and 3, Method 100 comprises:

[0078] - a training step 102 of a generator neural network 31 to generate a synthetic image 13 from a segmentation mask,

[0079] - a step of obtaining 103 artificial segmentation masks 23,

[0080] - a step of obtaining 104 synthetic images 13, from the artificial segmentation masks 23, using the trained generator neural network 31.

[0081] Figure 2 schematically represents an example of implementation of the training step 102 of the generator neural network 31. The generator neural network 31 is trained to generate a synthetic image from a segmentation mask associated with a real medical image. The majority 11 and minority 12 real medical images and their associated segmentation masks 21, 22 can be used to train the generator neural network 31. It should however be noted that nothing would prevent the generator neural network 31 from being trained with segmentation masks other than the majority 21 and minority 22 segmentation masks.

[0082] Different types of neural networks can be considered to generate a synthetic image from a segmentation mask. For example, it is possible to use a neural network of the autoencoder type ("Variational Autoencoder" or VAE in the English literature). However, it is preferable, as illustrated in Figure 2, to use a generator of a pair of generative adversarial neural networks (GAN). These types of neural networks allow the generation of images with a high degree of realism. A GAN is a generative model where two neural networks are placed in competition in a zero-sum game scenario. The first network, the generator 31 , generates an image, and its adversary, the discriminator 32, tries to detect whether the generated image is real or whether it is a synthetic image generated by the generator 31 .

[0083] In a first step, the generator 31 is trained to generate synthetic images from the segmentation masks. In a second step, the synthetic images generated by the generator are analyzed by the discriminator 32 which has been previously trained to recognize, by taking as input an image and its associated segmentation mask, whether the pair formed by the image and the segmentation mask is real. There is therefore at the output of the discriminator 32 a “True” or “False” decision depending on whether the pair formed by the image and the segmentation mask is considered real or not. By a backpropagation loop, depending on the veracity of the decision taken by the discriminator 32, the parameters of the generator 31 are modified until the generated synthetic images are considered real by the discriminator 32.

[0084] Generator 31 is a neural network for translating one image into another image. For example, it is a convolutional neural network of the “pix2pix” type as described in the document “image-to-image translation with conditional adversial networks” by Isola, P et al. In the example considered, the neural network comprises a first part of the encoder type composed of “Batch Normalization Leaky ReLU” convolution layers with 4x4 convolution filters of sizes 64, 128, 256, 512, 512, 512, 512 and a second part of the decoder type composed of “Batch Normalization Dropout ReLU” convolution layers with 4x4 convolution filters of sizes 512, 512, 512 then of “Batch Normalization ReLU” convolution layers with 4x4 convolution filters of sizes 256, 128, 64.The image size reduction in the encoder part is produced by strides of two pixels and the image size increase in the decoder part is produced by a 2D upsampling layer (Nearest Neighbors method) of size 2x2. The output is produced by a hyperbolic tangent (Tanh) activation layer.

[0085] The discriminator 32 is a neural network. For example, it is a convolutional neural network of the “PatchGAN” type as described in the paper “Image-to-image translation with conditional adversial networks” by Isola, P et al, modified to accept two input images which are concatenated into a single image. The rest of the neural network consists of a “Leaky ReLU” convolution layer with 4x4 convolution filters of size 64 and then four “Batch Normalization Leaky ReLU” convolution layers with 4x4 convolution filters of sizes 128, 256, 512, 512. The output is produced using a sigmoid activation layer.

[0086] The generator neural network 31 for translating an image into another image (“pix2pix”) is combined with the PatchGAN type discriminator 32 in such a way that the output predictions of the generator 31 (the synthetic medical images) constitute the second input of the discriminator 32. The first input of the discriminator corresponds to the medical image associated with the segmentation mask which is provided as input to the generator 31. The output is a 70x70 probability matrix. The weights of the neurons of the discriminator 32 are not modifiable during the training of the generator 31. The weights of the generator 31 can be updated during training. The cost calculation function is composed of the cross entropy and the 1 norm in a ratio of 1 to 100.

[0087] The discriminator 32 and the generator 31 are trained alternately, in turn, on a training set. The output of the discriminator 32 is optimized using the Adam-type stochastic gradient algorithm (Adaptive Moment Estimation, beta_1: 0.9, beta_02: 0.999, epsilon: 1 e-08) against a 70x70 matrix of values ​​1 when its input is a "true" pair (i.e., a pair comprising a segmentation mask and its associated real medical image), and of value 0 when its input is a "false" pair (i.e., a pair comprising a segmentation mask and a synthetic medical image produced by the generator 31 from the segmentation mask).The output of generator 31 is optimized using the Adam-type stochastic gradient algorithm against a 70x70 matrix of 1 values ​​so that the weights of the neurons in generator 31 are updated, but not those of discriminator 32, when discriminator 32 detects that the input pair is not close enough to the "true" pairs already encountered. Updating the weights of discriminator 32 alternately allows it to be ahead of generator 31 and forces it to update.

[0088] The method 100 also comprises a step 103 of obtaining artificial segmentation masks 23 from the majority segmentation masks 21 and the minority segmentation masks 22. As illustrated in FIG. 8, this step 103 may comprise:

[0089] - a selection of a majority segmentation mask 21,

[0090] - a selection of a minority segmentation mask 22,

[0091] - an identification, on the selected minority segmentation mask 22, of a set of voxels whose numerical value encodes the anomaly,

[0092] - a replacement, on the selected majority segmentation mask 21, of the numerical value of the identified voxels by the numerical value encoding the anomaly.

[0093] The majority segmentation mask 21 and the minority segmentation mask 22 can be randomly selected. The number of combinations of artificial segmentation masks can thus be the product of the number of majority segmentation masks 21 by the number of minority segmentation masks 22.

[0094] Several minority segmentation masks 21 with different semantic values ​​(i.e. with different types of anomaly: tumor, cyst, ablation region, artifact, ...) can be combined with the majority segmentation masks 22 and thus increase the number of potential combinations. Manipulating the proportions of the different minority masks 21 that will be used in the combinations of artificial segmentation masks 23 makes it possible to control the characteristics of the synthetic images produced. Similarly, several majority masks 22 with different semantic values ​​(liver, lung, pancreas, gallbladder, ...) can be combined with the minority segmentation masks 21.These different majority masks 22 make it possible to control in which organ or structure the minority segmentation masks 21 can appear using Boolean logic rules (for example rules of the “AND”, “OR”, “NOT” type) between the different majority masks 22 and minority masks 21 for each pixel of the artificial segmentation masks 23. This characteristic makes it possible to control, for example, in which organ the random distribution of a minority segmentation mask 21 of a tumor can appear (“AND”) and in which structure or organ this minority segmentation mask 21 cannot appear (“NOT”).

[0095] The combinations of majority segmentation mask 21 and minority segmentation mask 22 of greatest interest can also be selected according to an estimated distance between a segmentation of the anomaly 25 on the minority segmentation mask 22 and a segmentation of interest of the majority segmentation mask 21, such as for example the segmentation of the gallbladder, the vessels, the hilum or even the liver capsule.

[0096] As illustrated in Figure 9, the step 103 of obtaining artificial segmentation masks 23 may also comprise a transformation of the segmentation 44 of the anomaly of the selected minority segmentation mask 22. Such arrangements make it possible to generate a greater number of different artificial segmentation masks.The transformation of the segmentation 44 of the anomaly corresponds for example to a rotation (as illustrated for the artificial segmentation mask 23-1), a displacement (as illustrated for the artificial segmentation mask 23-2), an enlargement (as illustrated for the artificial segmentation mask 23-3), a reduction (as illustrated for the artificial segmentation mask 23-4), a deformation (as illustrated for the artificial segmentation mask 23-5) or a combination of these different possible transformations (as illustrated for the artificial segmentation mask 23-6 for which the segmentation 44 of the anomaly has been simultaneously displaced, reduced, and rotated).

[0097] Figure 10 illustrates the generation 103 of an artificial segmentation mask 23 from a majority segmentation mask 21 and a minority segmentation mask 22. On the majority segmentation mask 21, the segmentation 43 of the anatomy of interest can be seen. On the minority segmentation mask 22, both the segmentation 43 of the anatomy of interest and the segmentation 44 of the anomaly can be seen. On the generated artificial segmentation mask 23, the segmentation 43 of the anatomy of interest of the majority segmentation mask 21 can be seen combined with the segmentation 44 of the anomaly of the minority segmentation mask 22 (in the example considered, the segmentation 44 of the anomaly has also been moved).

[0098] This step 103 of generating the artificial segmentation masks 23 is implemented by a computer. The computer comprises a memory with a set of program code instructions which, when the program is executed, configure one or more processors of the computer to generate, as described above with reference to FIGS. 8 to 10, artificial segmentation masks 23 from the majority segmentation masks 21 and the minority segmentation masks 22.

[0099] As illustrated in Figures 1 and 3, the method 100 comprises a step 104 of obtaining synthetic images 13, from the artificial segmentation masks 23, using the trained generator neural network 31.

[0100] Figure 11 is an illustration of a real medical image 11 (respectively 12), its associated segmentation mask 21 (respectively 22), and a synthetic medical image 13 generated by the trained neural network 31, from the segmentation mask 21 (respectively 22), for a majority case without anomaly (respectively for a minority case with anomaly).

[0101] Figure 12 illustrates by way of example, for three different majority real medical images 11: the majority real medical image 11, its associated majority segmentation mask 21, an artificial segmentation mask 23 generated from the majority segmentation mask 21, and a synthetic medical image 13 generated from the artificial segmentation mask 23 by the trained generator neural network 31.

[0102] It is thus possible to generate a very large diversity of artificial segmentation masks 23. This then makes it possible to generate a very large diversity of synthetic medical images 13 presenting anatomical anomalies.

[0103] It is possible to train the segmentation neural network 30 from the supervised training data 25 comprising on the one hand the real segmented medical images 11, 12 and their associated segmentation masks 21, 22, and on the other hand the synthetic images 13 and their associated artificial segmentation masks 23. The supervised training of the segmentation neural network 30 corresponds to step 107 in FIGS. 1 and 3.

[0104] The solution proposed in this application goes further and proposes to further enrich the supervised training data 25.

[0105] For this, the method 100 comprises a step 105 of obtaining real non-segmented medical images 14 each representing the anatomy of interest 41 of a patient and each being associated with a weak annotation. The non-segmented medical images 14 may comprise images with anomaly and / or images without anomaly.

[0106] A weak annotation may correspond to an approximate segmentation (of low precision compared to the aforementioned majority or minority segmentations) or to a simple localization of a structure to be segmented. This approximate localization may be materialized by one or more points of the anatomical structure to be segmented, by a geometric shape covering a region inside the anatomical structure to be segmented, or by a geometric shape surrounding or covering the anatomical structure to be segmented (bounding box). The case where a weak annotation is materialized by a geometric shape covering a region inside the anatomical structure to be segmented is particularly advantageous. Weak annotations can advantageously be generated or at least validated by a medical expert.

[0107] Real unsegmented medical images 14 and their associated weak annotations form semi-supervised training data 26.

[0108] As illustrated in Figures 1 and 3, the method 100 comprises an identical reproduction 106 (a cloning) of the segmentation neural network 30 to form a teaching network 30'. Thus, the teaching network 30' is initialized with weights equal to those of the segmentation neural network 30 at the time of the reproduction 106.

[0109] As we will see later, thanks to the use of weak annotations, it is not necessary to train the segmentation neural network 30 before cloning it to train the teaching network 30'. Weak annotations make it possible to initiate segmentation even from a naive teaching network (little or not trained).

[0110] The weights of the segmentation neural network 30 (which acts as a student network) and the weights of the teacher network 30' are updated iteratively. Each iteration corresponds to a training phase based on the use of a batch of images from the training data.

[0111] At each iteration, the weights of the segmentation neural network 30 are updated by gradient descent. The weights of the teacher network 30' are updated by moving exponential average based on the weights of the teacher network 30' calculated at the previous iteration and the weights of the segmentation neural network 30 calculated at the current iteration.

[0112] The teaching network 30' can then be used to generate segmentation pseudomasks 24 which correspond to predictions of the teaching network 30' obtained from semi-supervised training data 26. The segmentation pseudomasks 24 thus generated can then be used as supervised training data to continue the training 107 of the segmentation neural network during a following iteration.

[0113] Each iteration therefore comprises: - a supervised training step 107 of the segmentation neural network 30 from supervised training data 25,

[0114] - a step of updating 108 the weights of the segmentation neural network 30 by gradient descent,

[0115] - a step of updating 109 of the weights of the teacher network 30' by moving exponential average,

[0116] - a generation step 1 10 of pseudo-segmentation masks 24 by the teaching network 30',

[0117] - a step of adding 1 11 to the supervised training data 25 the generated segmentation pseudomasks 24 and the real non-segmented medical images 14 which allowed their generation 110.

[0118] It should be noted that the above steps could be performed in a different order (i.e., it is not essential to perform the steps listed above in the proposed order). Nothing would prevent, for example, for a current iteration, updating the weights of the teacher network 30' by moving exponential average before updating the weights of the segmentation neural network 30 by gradient descent (in this case, the update of the weights of the teacher network 30' by moving exponential average is done according to the weights of the segmentation neural network 30 updated in the previous iteration).

[0119] As illustrated in Figure 4, the supervised training 107 comprises a calculation of a supervised cost function C s representative of a dissimilarity between a prediction ÿ s of the segmentation neural network 30, carried out from an image belonging to the supervised training data 25, and the segmentation mask y s associated with this image.

[0120] The supervised cost function C s is for example calculated according to the Sorensen-Dice index:

[0121] In parallel, and as illustrated in Figure 5, the method 100 may also include a calculation of a semi-supervised cost function C ss representative of a dissimilarity between a prediction ÿ T from the teacher network 30' and a prediction ÿ s of the segmentation neural network 30 carried out for an image belonging to the semi-supervised training data 26.

[0122] The semi-supervised cost function C ss is for example calculated by binary cross entropy:

[0123] C ss = E[(1 - y T log( - ÿs) - y T log ys \ The predictions of the teacher network 30' are a source of truth for the calculation of the semi-supervised cost function (as are the segmentation masks 21, 22 for the calculation of the supervised cost function).

[0124] As illustrated in Figure 6, a global cost function C can then be calculated based on the supervised cost function C s and the semi-supervised cost function C ss .

[0125] The global cost function C can notably be calculated in the form of a weighted sum of the supervised cost function C s and the semi-supervised cost function C ss . For example, the overall cost function can be written as:

[0126] The combination of the supervised cost function C s with the semi-supervised cost function C ss contributes to the good segmentation performance obtained after training

[0127] It has been empirically observed that an optimal value for the coefficient y is between 0.05 and 0.15.

[0128] Weights Update 108 of the segmentation neural network 30 by gradient descent can then be performed based on a result of the global cost function C.

[0129] The weights w T of the 30' teacher network are updated in the form of an exponential moving average calculated based on the previous weights w Tp rec of the teacher network 30' and weights updated segmentation neural network 30.

[0130] As illustrated in Figure 6, the weights w Tof the 30' teacher network are for example calculated in the form:

[0131] In this expression, the term S is a parameter whose value is predetermined. It has been observed empirically that an optimal value for the parameter S is between 0.85 and 0.95.

[0132] Update 109 of the w weights T of the teacher network 30' using an exponential moving average allows the accumulation of different versions of the weights of the student network over time. The conservative predictions of the teacher network thus remain slightly different; this has a regularization effect for the training of the student network (i.e. for the segmentation neural network 30 that we are trying to train), and this helps limit overfitting.

[0133] It is also important to note that, again with the aim of limiting overfitting, the supervised training data 25 are not consumed by the teaching network 30'. Thus, the weights of the teaching network 30' are updated from a training dataset different from the data that the teaching network 30' consumes to generate the pseudo-annotations 24. This limits overfitting.

[0134] Advantageously, and as illustrated in FIG. 5, a prediction of the teaching network 30' carried out on an image belonging to the semi-supervised training data 26 is enriched with the weak annotation associated with said image (the prediction obtained is merged with the weak annotation).

[0135] The enriched prediction, noted y T ' , can be written in the form: y T ' = (.1 - P)ÿ T + Py F

[0136] In this expression, ÿ Tis a raw prediction of the teacher network 30', y F corresponds to the weak annotation, and p is a contribution ratio of the weak annotation. It has been observed that an optimal value for the contribution ratio p of the weak annotation is between 0.45 and 0.55. The enriched prediction y T ' is used instead of the prediction ÿ T in the calculation of the semi-supervised cost function C ss .

[0137] As previously indicated, weak annotations correspond to location information of the structures to be segmented, this information being produced by medical experts. Combining the predictions of the 30' teaching network with weak annotations thus makes it possible to guide the training of the segmentation on valid structures. This makes it possible to refine the generation of pseudo-segmentation masks by the 30' teaching network.

[0138] In the example considered, the probability maps formed by the combination ÿ F of the prediction T of the 30' teacher network and weak annotation y F are binarized at a threshold value of 0.5.

[0139] The supervised training 107 of the student network 30 then makes it possible to update the teacher network 30' which in turn can produce more precise pseudo-segmentation masks 24 thanks to the weak annotations.

[0140] The supervised cost function C s can be calculated as a linear combination of the following:

[0141] - a real cost function C Re calculated from the real segmented medical images 11, 12 and their associated segmentation masks 21, 22,

[0142] - a synthetic cost function C Syn calculated from the synthetic images 13 and their associated artificial segmentation masks 23.

[0143] The overall cost function C can then be written in the form:

[0144] In this expression, / is a contribution ratio of the semi-supervised cost function C ss , a is a contribution ratio of the synthetic cost function C Syn , and p is a realism score of the synthetic images 13 calculated using the discriminator neural network 32 forming with the generator neural network 31 a pair of generative adversarial networks. The more confident the discriminator 32 is that the synthetic images produced by the generator 31 are real, the more these synthetic images contribute to the total cost function. It has been observed that an optimal value for the contribution ratio a is between 0.35 and 0.45. It has also been observed that an optimal value for the contribution ratio y is between 0.05 and 0.15.

[0145] Training of the segmentation neural network 30 can then continue until the model converges by cross-validation. The predictions of the segmentation neural network 30 are binarized using a threshold to produce the segmentation masks of the images to be segmented.

[0146] Each of the different artificial intelligence algorithms (the segmentation neural network 30, the teaching network 30', the generator neural network 31, and the discriminator neural network 32) is implemented by a computer. It is conceivable to use a single computer to implement all of these algorithms. Alternatively, some of these algorithms can be implemented on separate computers.

[0147] Once the segmentation neural network 30 is sufficiently trained, it can be used in a method for automatically segmenting a medical image. The trained segmentation neural network 30 can be stored on a computer-readable storage medium. One or more processors of the computer can then be configured to take as input a medical image to be segmented, and to produce as output a segmentation mask associated with the medical image using the trained segmentation neural network 30.

[0148] The above description clearly illustrates that, through its various characteristics and their advantages, the present invention achieves the set objectives.

[0149] In particular, the construction of the artificial segmentation masks 23 and the generation of the associated synthetic images 13 makes it possible to limit the overtraining of the segmentation neural network 30. Indeed, it is possible to construct an infinity of different artificial segmentation masks 23 by a random combination of the majority segmentation masks 21 with the minority segmentation masks 22, and by a transformation of the segmentation 44 of an anomaly of a minority segmentation mask 22 before its combination with a majority segmentation mask 21. Each synthetic image 13 thus generated can then be used only once in the training of the segmentation algorithm.

[0150] The majority 21 and minority 22 segmentation masks, as well as the weak annotations, are preferably obtained by medical experts. This ensures a sufficient level of involvement of medical expertise in the training of the segmentation neural network.

[0151] Weak annotations, while being significantly less expensive to produce than a segmentation mask, allow the predictions of pseudo-segmentation masks by the teaching network 30' to be refined.

[0152] The pseudo-segmentation masks generated by the teaching network 30' make it possible to enrich the supervised training data set 25. The supervised training data 25 can thus contain a very large number of different images.

[0153] The fact that the 30' teaching network does not consume the supervised training data 25 also helps to limit overfitting.

[0154] The combination in the supervised training data of real segmented medical images 11, 12, synthetic images 13 comprising rare anomalies, and real unsegmented medical images 14 for which a pseudo-segmentation has been generated, makes it possible to obtain very good segmentation performances at the end of the training.

Claims

Claims 1. Method (100) for training a segmentation neural network (30) to segment anatomical structures on a medical image representing an anatomy of interest (41) of a patient, the method (100) comprising the following steps: - obtaining (101) segmented real medical images (11, 12), each associated with a segmentation mask (21, 22), among which: o majority real medical images (11) associated with majority segmentation masks (21) and representing the anatomy of interest (41) of a patient without anomaly, o minority real medical images (12) associated with minority segmentation masks (22) and representing the anatomy of interest (41) of a patient with an anomaly (42), - training (102) a generator neural network (31) to generate a synthetic image from a segmentation mask, - obtaining (103) artificial segmentation masks (23), each artificial segmentation mask (23) being obtained by combining a majority segmentation mask (11) with a segmentation (44) of an anomaly of a minority segmentation mask (12), - obtaining (104) synthetic images (13), from the artificial segmentation masks (23), using the trained generator neural network (31), - obtaining (105) real non-segmented medical images (14) each representing the anatomy of interest (41) of a patient and each being associated with a weak annotation, - an identical reproduction (106) of the segmentation neural network (30) to form a teaching network (30'), the method further comprising several iterations of the following steps: - supervised training (107) of the segmentation neural network (30) from supervised training data (25) comprising the segmented real medical images (11, 12) and their associated segmentation masks (21, 22), as well as the synthetic images (13) and their associated artificial segmentation masks (23), - an update (108) of the weights of the segmentation neural network (30) by gradient descent, - an update (109) of the weights w T of the teacher network (30') by an exponential moving average calculated according to the previous weights w Tprec of the teaching network (30') and weights updated segmentation neural network (30), - a generation (1 10) of pseudo-segmentation masks (24) corresponding to predictions of the teaching network (30') obtained from semi-supervised training data (26) comprising the real non-segmented medical images (14) and their associated weak annotations, - an addition (1 11 ) to the supervised training data (25) of the pseudo-segmentation masks (24) generated and of the real non-segmented medical images (14) which allowed their generation (110).

2. Method (100) according to claim 1, comprising, at each iteration: - a calculation of a supervised cost function C s representative of a dissimilarity between a prediction of the segmentation neural network (30), made from an image belonging to the supervised training data (25), and the segmentation mask associated with this image, - a calculation of a semi-supervised cost function C ssrepresentative of a dissimilarity between a prediction of the teaching network (30') and a prediction of the segmentation neural network (30) made for an image belonging to the semi-supervised training data (26), - a calculation of a global cost function C as a function of the supervised cost function C s and the semi-supervised cost function C ss , - the update (108) of the weights of the segmentation neural network (30) by gradient descent being carried out according to the global cost function.

3. Method (100) according to claim 2 in which a prediction of the teaching network (30') carried out on an image belonging to the semi-supervised training data (26) is enriched with the weak annotation associated with said image, the enriched prediction, noted y T ' , which can be written in the form y T ' = (1 - ?)ÿ T + py F, expression in which y T is a raw prediction of the teacher network (30'), y F corresponds to the weak annotation, and p is a contribution ratio to the weak annotation.

4. Method (100) according to claim 3 wherein the contribution ratio p of the weak annotation is between 0.45 and 0.

55.

5. Method (100) according to any one of claims 2 to 4 wherein each weak annotation comprises an approximate location of an anatomical structure to be segmented, said approximate location being materialized by one or more points of the anatomical structure to be segmented, a geometric shape covering a region inside the anatomical structure to be segmented, or a geometric shape surrounding or covering the anatomical structure to be segmented.

6. Method (100) according to any one of claims 2 to 5 in which the global cost function C is calculated in the form of a weighted sum of the supervised cost function C s and the semi-supervised cost function C ss .

7. Method (100) according to claim 6 in which the overall cost function is written in the form C = (1 - y)C s + yC ss , with y between 0.05 and 0.

15.

8. Method (100) according to any one of claims 2 to 7 in which the supervised cost function C s is calculated as a linear combination of a real cost function C Re calculated from the segmented real medical images (11, 12) and their associated segmentation masks (21, 22) and a synthetic cost function C Syn calculated from synthetic images (13) and their associated artificial segmentation masks (23).

9. Method (100) according to claim 8 in which the global cost function C is written in the form C = (1 - + apC Syn + yC ss , expression in which y is a contribution ratio of the semi-supervised cost function C ss , a is a contribution ratio of the synthetic cost function C Syn , and p is a realism score of the synthetic images (13) calculated using a discriminator neural network (32) forming with the generator neural network (31) a pair of generative antagonistic networks.

10. Method (100) according to claim 9 wherein the contribution ratio a is between 0.35 and 0.45 and the contribution ratio y is between 0.05 and 0.

15.

11. Method (100) according to any one of claims 1 to 10 in which obtaining (103) artificial segmentation masks (23) comprises a transformation of the segmentation (44) of the anomaly of the minority segmentation mask.

12. Method (100) according to claim 11 wherein the transformation of the segmentation (44) of the anomaly corresponds to a rotation, an enlargement, a reduction, a deformation and / or a displacement of the segmentation (44) of the anomaly.

13. Method for automatic segmentation of a medical image comprising: - training a segmentation neural network (30) according to any one of claims 1 to 12, - a use of the trained segmentation neural network (30) for segmenting the medical image.

14. Computer-readable storage medium comprising a trained segmentation neural network (30) according to any one of claims 1 to 12.

Images