How to train a model for semantic segmentation of medical images

Abstract

The present invention relates to a method for training a segmentation neural network for segmenting anatomical structures in medical images. The segmentation neural network (30) is trained using a supervised method (107) with supervised training data (25) which includes a composite image (13) generated from segmented real medical images (11, 12) and associated segmentation masks (21, 22), as well as an artificial segmentation mask (23). In parallel, a teacher network (30') initially corresponds to a clone of the segmentation neural network and is updated using an exponential moving average. The teacher network is used to generate (110) segmentation pseudomasks (24) from semi-supervised training data (26) which includes unsegmented real medical images (14), each associated with a weak annotation. The unsegmented real medical images and associated segmentation pseudomasks are added (111) to the supervised training data to continue training the segmentation neural network.

Description

[Technical Field]

[0001] Field of Invention This application relates to the field of semantic segmentation of medical images. In particular, a method for training a segmentation neural network and a computer-readable storage medium containing a segmentation neural network trained by such a method are proposed. [Background technology]

[0002] prior art Machine learning algorithms for semantic segmentation of medical images are generally implemented using artificial neural networks. In particular, it is known that training convolutional neural networks based on U-net type models enables the performance of semantic image segmentation. Semantic segmentation involves labeling each pixel of an image with a class corresponding to what is shown on the image. In the field of medical imaging, a class may represent a specific anatomical structure, such as a target anatomical site (which may be a liver, lung, or kidney, or even another anatomical structure such as bone or blood vessels), an abnormality within this target anatomical site (e.g., a cyst, tumor, or ablation zone), or a surgical artifact (e.g., part of a medical device).

[0003] Training segmentation neural networks requires a large amount of training data to achieve good prediction quality. Furthermore, low representativeness of a class in the training data can lead to significant prediction bias for that class.

[0004] This issue is particularly important in the medical field. In fact, training a neural network using supervised methods requires training data containing numerous segmented medical images. Each medical image must then be segmented manually, or optionally semi-automatically, by a medical professional. This medical image segmentation is costly, especially in terms of time and expertise.

[0005] Collecting a large number of medical images showing rare anatomical abnormalities is all the more difficult (due to the rarity of the disease, patient confidentiality, and the effort and cost required to perform medical imaging operations).

[0006] A paper by Christian S. Perone et al. titled "Deep semi-supervised segmentation with weight-averaged consistency targets" describes a method for semi-supervised training of segmentation neural networks using the "mean teacher" approach.

[0007] Several prior art solutions also rely on relatively similar methods in which the segmentation neural network being trained is replicated to form a teacher network. In particular, patent applications, International Publication No. 2021 / 140426A1, U.S. Patent Application No. 2022 / 0292689A1, U.S. Patent Application No. 2021 / 0407656A1, and U.S. Patent Application No. 2022 / 0358658A1 describe various methods for training neural networks for segmenting medical images.

[0008] However, these solutions have several drawbacks, including overfitting of segmentation algorithms to supervised training data, sometimes insufficient segmentation performance, learning bias for segmenting rare classes, and / or insufficient involvement of medical expertise in training segmentation neural networks.

[0009] On the other hand, International Patent Application Publication No. 2022 / 238640A1 describes a method of generating synthetic images showing rare anatomical abnormalities using a adversarial generation network. SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0010] DISCLOSURE OF THE INVENTION The object of the present application is to propose solutions to all or some of the drawbacks of the prior art, particularly those disclosed above. MEANS FOR SOLVING THE PROBLEM

[0011] For this purpose, according to a first aspect, a method of training a segmentation neural network to segment an anatomical structure on a medical image representing a target anatomical site of a patient is proposed. The method is - obtaining segmented actual medical images, each associated with a segmentation mask, the actual medical images being 〇 a majority actual medical image associated with a majority segmentation mask and representing a target anatomical site of a patient without abnormalities, and 〇 obtaining, including a minority actual medical image associated with a minority segmentation mask and representing a target anatomical site of a patient with abnormalities, - training a generation neural network to generate a synthetic image based on the segmentation mask; - obtaining an artificial segmentation mask, each artificial segmentation mask being obtained by combining a majority segmentation mask with the abnormal segmentation of a minority segmentation mask, - obtaining a synthetic image based on the artificial segmentation mask using the trained generation neural network; - A step of obtaining unsegmented medical images in which each represents a target anatomical site of the patient and each is associated with weak annotation, - The step of identically duplicating a segmentation neural network to form a training network.

[0012] The method is, - A supervised training step of a segmentation neural network based on supervised training data including segmented real medical images and associated segmentation masks, and synthesized images and associated artificial segmentation masks, - Using gradient descent to calculate the weights of the segmentation neural network w S Steps to update, - Pre-weights of the teacher network

number

[0013] The construction of artificial segmentation masks and the generation of associated synthetic images limit overfitting of segmentation neural networks. In fact, by randomly combining majority and minority segmentation masks, and optionally by transforming the segmentation of anomalous minority segmentation masks before combining them with the majority mask, it is possible to construct an infinite number of different artificial segmentation masks. Therefore, each synthetic image is generated to be used only once when training the segmentation algorithm.

[0014] Real-world medical images can be segmented by medical professionals. Weak annotations can also be provided by medical professionals. This ensures a sufficient level of involvement of medical expertise in the training of the segmentation neural network.

[0015] Weak annotations are significantly less expensive to produce than segmentation masks (in terms of time and expertise). Weak annotations correspond to labeling one or more pixels in an image. This can correspond to approximate segmentation (which has lower accuracy compared to segmentation using majority or minority segmentation masks) or to the simple location of the structure being segmented. The use of weak annotations makes it possible to refine the predictions of pseudo-segmentation masks produced by training networks.

[0016] By updating the weights of the teacher network using an exponential moving average (EMA), it becomes possible to accumulate the weights of various versions of the segmentation neural network (which behaves like the student network) over time. Thus, the "conservative" predictions of the teacher network remain slightly different, which has a regularizing effect for training the student network (the neural network being trained) and limits overfitting (in other words, the intention is to prevent the student network's predictions from becoming too close to a particular segmentation mask that is too specialized for the supervised training data).

[0017] By combining segmented real medical images, composite images containing rare abnormalities, and unsegmented real medical images from which pseudo-segmentation has been generated in supervised training data, it becomes possible to obtain extremely high segmentation performance capabilities upon completion of training.

[0018] In certain embodiments, the training method may further include one or more of the following features, which are considered individually or in all technically possible combinations:

[0019] In a particular embodiment, the method, in each iteration, - A supervised cost function C that represents the difference between the predictions of a segmentation neural network performed on images belonging to supervised training data and the segmentation mask associated with the images. S Calculating and - A semi-supervised cost function C that represents the difference between the predictions of the supervising network and the predictions of the segmentation neural network performed on images belonging to the semi-supervised training data. SS Calculating and - Supervised cost function C S and semi-supervised cost function C SS This includes calculating the overall cost function C as a function of, - The weights w of the segmentation neural network using gradient descent method S are updated as a function of the overall cost function.

[0020] Combining the supervised cost function with the semi-supervised cost function contributes to the excellent segmentation performance ability obtained at the end of training.

[0021] In certain embodiments, the prediction of the teacher network performed on an image belonging to the semi-supervised training data is enhanced with the weak annotation associated with the said image. The enhanced prediction

Number

Number

Number

[0022] In certain embodiments, the contribution rate β of the weak annotation is in the range of 0.45 to 0.55.

[0023] By combining this weak annotation with the prediction of the teacher network, it becomes possible to refine the generation of the pseudo-segmentation mask.

[0024] In certain embodiments, each weak annotation includes the approximate position of the anatomical structure to be segmented. This approximate position can be embodied by one or more points of the anatomical structure to be segmented, a geometric shape encompassing an area inside the anatomical structure to be segmented, or a geometric shape surrounding or encompassing the anatomical structure to be segmented.

[0025] In a particular embodiment, the overall cost function C is the supervised cost function C S and semi-supervised cost function C SS It is calculated in the form of a weighted sum.

[0026] In a particular embodiment, the overall cost function is C = (1-γ)C S +γC SS It is written that γ is in the range of 0.05 to 0.15.

[0027] In a particular embodiment, the supervised cost function C S This is the real cost function C calculated based on segmented medical images and associated segmentation masks. Re , as well as the composite cost function C calculated based on the composite image and associated artificial segmentation mask. Syn It is calculated in the form of a linear combination of .

[0028] In a particular embodiment, the overall cost function C is C = (1 - γ)(C Re +αρC Syn )+γC ss It is written that equation γ is the semi-supervised cost function C SS This is the contribution rate, where α is the composite cost function C Syn The contribution rate is ρ, where ρ is the reality score of the synthetic image calculated using a discriminator neural network that forms a pair of generative and adversarial neural networks together with the generative neural network.

[0029] In certain embodiments, the contribution rate α is in the range of 0.35 to 0.45, and the contribution rate γ is in the range of 0.05 to 0.15.

[0030] In certain embodiments, obtaining an artificial segmentation mask involves transforming the segmentation of anomalies in a minority segmentation mask.

[0031] In certain embodiments, transforming anomaly segmentation corresponds to rotation, scaling, deformation, and / or displacement of anomaly segmentation.

[0032] According to a second embodiment, a method for automatically segmenting medical images is proposed. This automatic segmentation method includes the steps of training a segmentation neural network according to any one of the previous embodiments, and then segmenting the medical images using the trained segmentation neural network.

[0033] According to a third aspect, a computer-readable storage medium is proposed that includes a segmentation neural network trained according to any one of the previous embodiments.

[0034] Description of the drawing The present invention will be better understood by reading the following description. The following description is provided with reference to Figures 1 to 12 as non-limiting examples. [Brief explanation of the drawing]

[0035] [Figure 1] This is a schematic representation of the main steps of an embodiment of the method for training a segmentation neural network according to the present invention. [Figure 2] This is a schematic representation of training a generative neural network to generate a composite image based on a segmentation mask. [Figure 3] This is a schematic representation of the method for training a segmentation neural network according to the present invention. [Figure 4] This is a schematic representation of the calculation of a supervised cost function that shows the difference between the predictions of a segmentation neural network and the segmentation mask associated with the image on which the predictions are performed. [Figure 5] This is a schematic representation of the calculation of a semi-supervised cost function that shows the difference between the predictions of a teacher network and the predictions of a segmentation neural network. [Figure 6]This is a schematic representation of the updating of weights in a segmentation neural network and a teacher network, based on an overall cost function calculated as a function of supervised and semi-supervised cost functions. [Figure 7] This is a schematic representation of the steps involved in obtaining actual medical images and their associated segmentation masks. [Figure 8] This is a schematic representation of the steps involved in obtaining an artificial segmentation mask based on a majority segmentation mask and a minority segmentation mask. [Figure 9] This is a schematic representation of the steps involved in obtaining an artificial segmentation mask along with the transformation of anomaly segmentation. [Figure 10] This is an example of obtaining an artificial segmentation mask based on a majority segmentation mask and a minority segmentation mask. [Figure 11] Examples of actual medical images, their associated segmentation masks, and synthetic medical images generated by a generative neural network based on the segmentation masks are shown for the majority case (top of the figure) and the minority case (bottom of the figure). [Figure 12] Examples of three different majority real medical images include the majority real medical image, its associated majority segmentation mask, an artificial segmentation mask generated based on the majority segmentation mask, and a synthetic medical image generated by a generative neural network based on the artificial segmentation mask. [Modes for carrying out the invention]

[0036] Throughout these figures, the same reference number from one figure to another indicates the same or similar element. For clarity, unless otherwise specified, the elements depicted are not necessarily at the same scale.

[0037] Detailed description of the invention Figures 1 and 3 schematically illustrate the main steps of an embodiment of a method 100 for training a segmentation neural network 30 to segment anatomical structures on medical images representing target anatomical sites in a patient.

[0038] In this application, "neural network" corresponds to a computer implementation model whose operation is suggested by the operation of neurons in the human brain. It is accompanied by various deep learning techniques and itself forms part of a machine learning algorithm. Machine learning algorithms form a category in the field of artificial intelligence.

[0039] The target anatomical site may correspond to an organ (e.g., liver, pancreas, gallbladder, lung, or kidney) or another anatomical structure (e.g., bone or blood vessel). Abnormalities present within the target anatomical site generally correspond to lesions such as tumors, cysts, ablation zones, and aneurysms. Ablation zones correspond to lesions that have undergone ablation treatment using known methods (e.g., microwave, laser, radiofrequency), and are necrotic areas. Abnormalities may also correspond to surgical artifacts, such as parts of medical devices.

[0040] The segmentation neural network 30 is trained to take a real medical image as input and provide a segmentation mask associated with this image as output. The segmentation mask is an image in which each voxel provides specific information about the element indicated at the location of the voxel on the real medical image. Thus, a voxel can take a specific numerical value associated with the element indicated at the location of the voxel on the real medical image. For example, different unique numerical values ​​are defined for healthy parts of the target anatomical site, abnormalities within the target anatomical site, other anatomical structures (e.g., bones or blood vessels), the background of the image, etc. Thus, different numerical values ​​represent different classes of elements visible on the real medical image. Note that the term "voxel" is generally used to define a specific zone of an image (a voxel identifies the location of the above zone on the image and takes a value that represents what is indicated in the above zone on the image). It may be a two-dimensional or three-dimensional image. In the case of a two-dimensional image, the term "voxel" has the same meaning as the term "pixel".

[0041] "Actual medical images" are understood to mean medical images of a patient obtained using medical imaging devices, for example, by computed tomography (CT), positron emission tomography (PET), magnetic resonance imaging (MRI), ultrasound, or X-ray.

[0042] The segmentation neural network 30 may take the form of a U-Net type convolutional neural network, for example. The segmentation neural network 30 may also be based on a model derived from the U-Net architecture, such as the RITnet model, which combines U-Net and DenseNet models (see "RITnet: Real-time Semantic Segmentation of the Eye for Gaze Tracking" by AK Chaudhary et al.). However, the segmentation neural network 30 may also be based on other models. It may be a transformer neural network, such as a "Swin Transformer" network (see, for example, "Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images" by A. Hatamizadeh et al.).

[0043] As shown in Figures 1 and 3, Method 100 includes step 101 of obtaining segmented real medical images 11, 12, each associated with a segmentation mask 21, 22. These real medical images are 〇 A majority medical image 11, associated with the majority segmentation mask 21, representing the target anatomical site of a patient without abnormalities. Includes a minority medical image 12 associated with a minority segmentation mask 22, representing a target anatomical site 41 of a patient with an abnormality.

[0044] The terms "majority" and "minority" are generally used because the number of medical images representing normal anatomical sites is significantly greater than the number of medical images representing abnormal sites.

[0045] Figure 7 schematically shows step 101, which involves obtaining actual medical images 11, 12 and their associated segmentation masks 21, 22.

[0046] The left side of Figure 7 shows the generation of a majority segmentation mask 21 based on the majority medical image 11. The target anatomical site 41 is visible on the majority medical image 11. In the medical image referred to as the "majority" medical image, there are no abnormalities within the target anatomical site. The majority segmentation mask 21 includes the segmentation of the target anatomical site.

[0047] The left side of Figure 7 shows the generation of a minority segmentation mask 22 based on a minority clinical image 12. On the minority clinical image 12, the target anatomical site 41 and the abnormalities 42 present within the target anatomical site 41 are visible. The minority segmentation mask 22 includes segmentation of the target anatomical site and segmentation of the abnormalities 44.

[0048] Segmentation can be performed manually by a medical professional. In this case, the medical professional uses a graphical interface (mouse, stylus, touchscreen, etc.) on the electronic device (computer, tablet, etc.) displaying the image to define the contours of various anatomical regions on the medical image. Segmentation can also be performed semi-automatically using conventional image segmentation algorithms. For example, bone visible on a medical image can be segmented by a method of thresholding the intensity of voxels in an intensity window having a width of 1800 HU and a center of 400 HU (HU is an acronym for "Hounsfield Unit," which is a quantitative measure of radiation density, i.e., a unit of measurement representing the opacity of a substance to X-rays).

[0049] As shown in Figures 1 and 3, Method 100 is, - Step 102 involves training a generative neural network 31 to generate a composite image 13 based on a segmentation mask, - Step 103 to obtain an artificial segmentation mask 23, - The process includes step 104, which involves using a trained generative neural network 31 to obtain a composite image 13 based on an artificial segmentation mask 23.

[0050] Figure 2 shows an embodiment of step 102 for training the generative neural network 31. The generative neural network 31 is trained to generate a composite image based on segmentation masks associated with real medical images. The majority real medical image 11 and the minority real medical image 12, along with their associated segmentation masks 21 and 22, may be used to train the generative neural network 31. However, it should be noted that this does not prevent the generative neural network 31 from being trained with segmentation masks other than the majority segmentation mask 21 and the minority segmentation mask 22.

[0051] Various types of neural networks can be considered to generate a composite image based on a segmentation mask. For example, the use of an autoencoder ("Variational Autoencoder" or VAE) type neural network may be considered. However, it is preferable to use a generator to produce a pair of generative adversarial neural networks (GANs), as shown in Figure 2. These types of neural networks actually enable the generation of images with a high degree of realism. A GAN is a generative model in which two neural networks compete in a zero-sum game scenario. The first network, i.e., the generator 31, generates an image, and its competitor, the discriminator 32, attempts to detect whether the generated image is realistic or whether it is a composite image generated by the generator 31.

[0052] Firstly, the generator 31 is trained to generate a composite image based on a segmentation mask. Secondly, the composite image generated by the generator is analyzed by the discriminator 32, which is pre-trained to recognize whether the pair formed by the image and its associated segmentation mask is real, using the image and its associated segmentation mask as input. Therefore, the output from the discriminator 32 is a "true" or "false" decision, depending on whether the pair formed by the image and segmentation mask is considered real or not. Through a backpropagation loop, the parameters of the generator 31 are modified as a function of the truthfulness of the decision made by the discriminator 32 until the discriminator 32 considers the generated composite image to be real.

[0053] The generator 31 is a neural network for converting one image to another. This is a "pix2pix" type convolutional neural network, such as the one described in the paper titled "Image-to-image translation with conditional adversarial networks" by Isola, P et al. In the example considered, the neural network includes a first part of encoder type consisting of "batch normalized leaky ReLU" convolutional layers using 4x4 convolutional filters of sizes 64, 128, 256, 512, 512, 512, and 512; and a second part of decoder type consisting of "batch normalized dropout ReLU" convolutional layers using 4x4 convolutional filters of sizes 512, 512, and 512, followed by "batch normalized ReLU" convolutional layers using 4x4 convolutional filters of sizes 256, 128, and 64. The reduction in image size in the encoder section is caused by two pixel strides, while the increase in image size in the decoder section is caused by a 2x2 2D expansion layer ("upsampling 2D," nearest neighbor method). The output is produced by a hyperbolic tangent (Tanh) type activation layer.

[0054] The discriminator 32 is a neural network. This is a "PatchGAN" type convolutional neural network, such as the one described in the paper titled "Image-to-image translation with conditional adversarial networks" by Isola, P et al., modified to accept, for example, two images concatenated into a single image as input. The rest of the neural network consists of a "leak ReLU" convolutional layer using 64 size 4x4 convolutional filters, followed by four "batch normalized leak ReLU" convolutional layers using 128, 256, 512, and 512 size 4x4 convolutional filters. The output is produced using a sigmoid activation layer.

[0055] A generative neural network 31 ("pix2pix") for converting one image to another is coupled with a PatchGAN-type classifier 32 such that the output prediction (synthesized medical image) of the generator 31 forms the second input of the classifier 32. The first input of the classifier corresponds to the medical image associated with the segmentation mask provided in the input of the generator 31. The output is a 70x70 probability matrix. The neuronal weights of the classifier 32 cannot be modified during training of the generator 31. The weights of the generator 31 can be updated during training. The cost calculation function consists of cross-entropy and a norm of 1 for ratios between 1 and 100.

[0056] The classifier 32 and generator 31 are, as an alternative, trained on a training game. The output of classifier 32 is optimized using an Adam-type stochastic gradient algorithm (adaptive moment estimation, beta1:0.9, beta02:0.999, epsilon:1e-08) on a 70x70 matrix where the value is 1 if the input is a "true" pair (i.e., a pair containing a segmentation mask and its associated real medical image), and 0 if the input is a "false" pair (i.e., a pair containing a segmentation mask and a synthetic medical image created by generator 31 based on the segmentation mask). The output of generator 31 is optimized using an Adam-type stochastic gradient algorithm on a 70x70 matrix with the value 1, so that if classifier 32 detects that the input pair is not close enough to a "true" pair already encountered, the weights of the neurons in generator 31 are updated, but the weights of classifier 32 are not. Alternatively, updating the weights of the classifier 32 allows the classifier 32 to precede the generator 31, thereby forcing the generator 31 to update.

[0057] Method 100 also includes step 103 of obtaining an artificial segmentation mask 23 based on a majority segmentation mask 21 and a minority segmentation mask 22. As shown in Figure 8, this step 103 is - Select the majority segmentation mask 21, - Selecting a minority segmentation mask 22, - Identify the set of voxels on the selected minority segmentation mask 22 in which the numerical values ​​encode anomalies, - This may include replacing the numerical values ​​of the identified voxels on the selected majority segmentation mask 21 with numerical values ​​that encode the anomalies.

[0058] The majority segmentation mask 21 and the minority segmentation mask 22 may be selected randomly. Therefore, the number of combinations of artificial segmentation masks may be equivalent to the product of the number of majority segmentation masks 21 and the number of minority segmentation masks 22.

[0059] Several minority segmentation masks 21 with different semantic values ​​(i.e., different types of abnormalities, tumors, cysts, ablation areas, artifacts, etc.) may be combined with a majority segmentation mask 22, thereby increasing the number of potential combinations. By manipulating the ratios of the various minority masks 21 used in the combination of artificial segmentation masks 23, it becomes possible to control the features of the resulting composite image. Similarly, several majority masks 22 with different semantic values ​​(liver, lungs, pancreas, gallbladder, etc.) may be combined with a minority segmentation mask 21. These different majority masks 22 may be used to control which organs or structures the minority segmentation masks 21 may appear in by Boolean logic rules (e.g., "AND", "OR", "NOT" type rules) between the different majority masks 22 and minority masks 21 for each pixel of the artificial segmentation mask 23. This feature can be used, for example, to control for organs where the random distribution of the minority segmentation mask 21 of the tumor may appear ("AND") or for structures or organs where this minority segmentation mask 21 does not appear ("NOT").

[0060] A combination of a more target majority segmentation mask 21 and a minority segmentation mask 22 may be selected as a function of the estimated distance between the abnormal segmentation 25 on the minority segmentation mask 22 and the target segmentation of the majority segmentation mask 21, such as the gallbladder, blood vessels, hepatic hilum, or even the liver capsule.

[0061] As shown in Figure 9, step 103 to obtain the artificial segmentation mask 23 may also include transforming the anomaly segmentation 44 of the selected minority segmentation mask 22. Such a configuration makes it possible to generate a large number of different artificial segmentation masks. Transformations of the anomaly segmentation 44 correspond to, for example, rotation (shown for artificial segmentation mask 23-1), displacement (shown for artificial segmentation mask 23-2), scaling (shown for artificial segmentation mask 23-3), scaling (shown for artificial segmentation mask 23-4), deformation (shown for artificial segmentation mask 23-5), or various possible combinations of these transformations (shown for artificial segmentation mask 23-6 where the anomaly segmentation 44 is equally displaced, scaled, and rotated).

[0062] Figure 10 shows the generation of an artificial segmentation mask 23 based on a majority segmentation mask 21 and a minority segmentation mask 22. The majority segmentation mask 21 shows the segmentation 43 of the target anatomical site. The minority segmentation mask 22 shows both the segmentation 43 of the target anatomical site and the segmentation 44 of abnormalities. The generated artificial segmentation mask 23 shows the segmentation 43 of the target anatomical site of the majority segmentation mask 21 combined with the segmentation 44 of abnormalities from the minority segmentation mask 22 (in the example examined, the segmentation 44 of abnormalities is also displaced).

[0063] Step 103, which generates this artificial segmentation mask 23, is implemented by a computer. The computer includes memory containing a set of program code instructions that constitute one or more processors of the computer in order to generate the artificial segmentation mask 23 based on the majority segmentation mask 21 and the minority segmentation mask 22, as described above with reference to Figures 8 to 10, when the program is executed.

[0064] As shown in Figures 1 and 3, Method 100 includes step 104 of obtaining a synthetic image 13 based on an artificial segmentation mask 23 using a trained generative neural network 31.

[0065] Figure 11 shows an example of a real medical image 11 (12 in the case of a minority with abnormalities) for the majority with no abnormalities, its associated segmentation mask 21 (22 in the case of a minority with abnormalities), and a synthetic medical image 13 generated by a trained neural network 31 based on the segmentation mask 21 (22 in the case of a minority with abnormalities).

[0066] Figure 12 shows, as an example, the majority real medical image 11, its associated majority segmentation mask 21, the artificial segmentation mask 23 generated based on the majority segmentation mask 21, and the synthetic medical image 13 generated by a trained generative neural network 31 based on the artificial segmentation mask 23, for three different majority real medical images 11.

[0067] In this way, it is possible to generate a wide variety of artificial segmentation masks 23. This then makes it possible to generate a wide variety of synthetic medical images 13 with anatomical abnormalities.

[0068] The segmentation neural network 30 can be trained on supervised training data 25, which on the one hand includes segmented real medical images 11, 12 and their associated segmentation masks 21, 22, and on the other hand includes a composite image 13 and its associated artificial segmentation mask 23. Supervised training of the segmentation neural network 30 corresponds to step 107 in Figures 1 and 3.

[0069] The solution proposed in this application goes further, proposing to further enhance the supervised training data 25.

[0070] For this purpose, method 100 includes step 105 of obtaining unsegmented medical images 14, each representing a target anatomical site 41 of the patient and each associated with a weak annotation. The unsegmented medical images 14 may include images with abnormalities and / or images without abnormalities.

[0071] Weak annotation can correspond to approximate segmentation (having lower accuracy compared to the majority or minority segmentation described above) or to a simple location of the structure being segmented. This approximate location can be embodied by one or more points on the anatomical structure being segmented, by a geometry that encloses a region within the anatomical structure being segmented, or by a geometry (bounding box) that surrounds or encloses the anatomical structure being segmented. It is particularly advantageous when the weak annotation is embodied by a geometry that encloses a region within the anatomical structure being segmented. Advantageously, weak annotation can be generated, or at least verified, by medical professionals.

[0072] Unsegmented real medical images 14 and their associated weak annotations form semi-supervised training data 26.

[0073] As shown in Figures 1 and 3, method 100 includes creating identical copies 106 (cloning) of the segmentation neural network 30 to form the teacher network 30'. Thus, the teacher network 30' is initialized with weights equal to the weights of the segmentation neural network 30 during the creation of the copy 106.

[0074] As shown below, the use of weak annotations means that the segmentation neural network 30 does not need to be trained before cloning the segmentation neural network 30 in order to form the teacher network 30'. In fact, weak annotations make it possible to start segmentation even from a native teacher network (which is little to no training).

[0075] The weights of the segmentation neural network 30 (which behaves like the student network) and the teacher network 30' are updated iteratively. Each iteration corresponds to a training stage based on using a batch of images from the training data.

[0076] In each iteration, the weights of the segmentation neural network 30 are updated using gradient descent. Meanwhile, the weights of the training network 30' are updated using an exponential moving average as a function of the training network 30' weights calculated during the previous iteration and the segmentation neural network 30 weights calculated during the current iteration.

[0077] The teacher network 30' can then be used to generate a pseudo-segmentation mask 24 corresponding to the predictions of the teacher network 30' obtained based on the semi-supervised training data 26. The pseudo-segmentation mask 24 thus generated can then be used as supervised training data to continue training the segmentation neural network 107 during subsequent iterations.

[0078] Therefore, each iteration is, - Supervised training step 107 of a segmentation neural network 30 based on supervised training data 25, - Step 108 updates the weights of the segmentation neural network 30 using gradient descent, - Step 109 updates the weights of the training network 30' using exponential moving average, - Step 110 generates a pseudo-segmentation mask 24 using the teacher network 30', - The process includes the step 111 of adding the generated pseudo-segmentation mask 24 and the unsegmented real medical image 14 that enabled the generation of the pseudo-segmentation mask 24 110 to supervised training data 25.

[0079] It should be noted that the above steps can be performed in a different order (in other words, the steps listed above do not have to be performed in the proposed order). For example, this does not prevent the weights of the training network 30' from being updated using an exponential moving average before updating the weights of the segmentation neural network 30 using gradient descent for the current iteration (in this case, updating the weights of the training network 30' using an exponential moving average is performed as a function of the weights of the segmentation neural network 30 updated during the previous iteration).

[0080] As shown in Figure 4, supervised training 107 is performed on the predictions of the segmentation neural network 30 based on images belonging to the supervised training data 25.

number

[0081] Supervised cost function C S This is calculated, for example, according to the Sorensen-Dice index.

number

[0082] Simultaneously, as shown in Figure 5, method 100 also predicts the teacher network 30'.

number

number

[0083] Semi-supervised cost function C SS This can be calculated, for example, by binary cross-entropy.

number

[0084] The predictions of the supervising network 30' are a reliable source of information for calculating the semi-supervised cost function (similarly the segmentation masks 21, 22 for calculating the supervised cost function).

[0085] As shown in Figure 6, the overall cost function C is then supervised cost function C S and semi-supervised cost function C SS It can be calculated as a function of .

[0086] The overall cost function C is, in particular, the supervised cost function C S and semi-supervised cost function C SS It can be calculated in the form of a weighted sum of . For example, the overall cost function is C=(1-γ)C S +γC SS It could be written as follows.

[0087] Supervised cost function C S semi-supervised cost function C SS Combining this with other methods contributes to the superior segmentation performance capabilities achieved upon completion of training.

[0088] It has been empirically observed that the optimal value for the coefficient y is in the range of 0.05 to 0.15.

[0089] Next, gradient descent is used to determine the weights of the segmentation neural network 30 w S Updating 108 can be done as a function of the result of the overall cost function C.

[0090] On the other hand, the weight of the teacher network 30' w T This is the weight before the teacher network 30'.

number

[0091] As shown in Figure 6, the weights w of the teacher network 30' T For example,

number

[0092] In this equation, the term δ is a parameter with a predetermined value. It has been empirically observed that the optimal value for parameter δ is in the range of 0.85 to 0.95.

[0093] Using exponential moving averages, the weights of the training network are calculated as w T By updating 109, the weights of different versions of the student network can be accumulated over time. Thus, the conservative predictions of the teacher network remain slightly different, which has a regularization effect for training the student network (i.e., for the intended segmentation neural network 30 to be trained), thereby limiting overfitting.

[0094] It is also important to note that, in order to limit overfitting, the supervised training data 25 is not consumed by the teacher network 30'. Therefore, the weights of the teacher network 30' are updated based on a different set of training data than the data consumed by the teacher network 30' to generate the pseudo-annotations 24. This limits overfitting.

[0095] Advantageously, as shown in Figure 5, the predictions of the training network 30' performed on images belonging to the semi-supervised training data 26 are enhanced by the weak annotations associated with the above images (the resulting predictions are merged with the weak annotations).

[0096] Enhanced predictions

number

number

[0097] In this formula,

number

number

number

[0098] As shown above, the weak annotations correspond to the location information of the structure being segmented, and this information is generated by medical professionals. By combining the predictions of the training network 30' with the weak annotations in this way, it becomes possible to guide the segmentation training on an effective structure. This makes it possible to refine the generation of pseudo-segmentation masks by the training network 30'.

[0099] In the example examined, the prediction of the teacher network 30'

number

number

[0100] The supervised training 107 of the student network 30 allows the teacher network 30' to be updated, which in turn can produce a more precise pseudo-segmentation mask 24 thanks to weak annotations.

[0101] Supervised cost function C S It can be calculated in the form of a linear combination of the following elements. - Actual cost function C calculated based on segmented medical images 11, 12 and their associated segmentation masks 21, 22 Re - Composite cost function C calculated based on composite images 13 and their associated artificial segmentation masks 23 Syn

[0102] In that case, the overall cost function C is: C=(1-γ)(C Re +αρC Syn )+γC ss It could be written as follows.

[0103] In this equation, γ is the semi-supervised cost function C SS This is the contribution rate, where α is the composite cost function C Syn The contribution rate α is calculated using a discriminator neural network 32, which forms a pair of adversarial generative networks with the generative neural network 31, and ρ is the reality score of the synthesized image 13. The higher the discriminator 32's confidence in the reality of the synthesized images produced by the generator 31, the greater the contribution of these synthesized images to the overall cost function. The optimal value of the contribution rate α has been observed to be in the range of 0.35 to 0.45. The optimal value of the contribution rate γ has also been observed to be in the range of 0.05 to 0.15.

[0104] Subsequently, training of the segmentation neural network 30 can continue until the model converges through cross-validation. The predictions of the segmentation neural network 30 are binarized using a threshold to create a segmentation mask for the image to be segmented.

[0105] Each of the different artificial intelligence algorithms (segmentation neural network 30, teacher network 30', generative neural network 31, and discriminator neural network 32) is implemented by a computer. It may be considered to use the same computer to implement all of these algorithms. Alternatively, some of these algorithms may be implemented on separate computers.

[0106] Once the segmentation neural network 30 is sufficiently trained, it can be used in a method for automatically segmenting medical images. The trained segmentation neural network 30 can be stored on a computer-readable storage medium. One or more processors of a computer may be configured to use the medical images to be segmented as input and to use the trained segmentation neural network 30 to produce segmentation masks associated with the medical images as output.

[0107] The above description clearly demonstrates that the present invention achieves its defined objectives through its various features and their advantages.

[0108] In particular, the construction of artificial segmentation masks 23 and the generation of associated synthetic images 13 make it possible to limit overfitting of the segmentation neural network 30. In fact, an infinite number of different artificial segmentation masks 23 can be constructed by randomly combining the majority segmentation mask 21 and the minority segmentation mask 22, and by transforming the anomalous segmentation 44 of the minority segmentation mask 22 before it is combined with the majority segmentation mask 21. Thus, each synthetic image 13 is generated to be used only once when training the segmentation algorithm.

[0109] The majority segmentation mask 21 and minority segmentation mask 22, as well as the weak annotations, are preferably obtained by a medical professional. This ensures a sufficient level of involvement of medical expertise in the learning of the segmentation neural network.

[0110] Weak annotation is significantly less expensive to produce than segmentation masks and allows for more precise predictions of pseudo-segmentation masks by the training network.

[0111] The pseudo-segmentation mask generated by the training network 30' allows the supervised training dataset 25 to be enhanced. Thus, the supervised training data 25 may contain a very large number of different images.

[0112] The fact that the teacher network 30' does not consume the supervised training data 25 also allows overfitting to be limited.

[0113] By combining segmented real medical images 11 and 12, a composite image 13 containing rare abnormalities, and an unsegmented real medical image 14 from which pseudo-segmentation has been generated, in supervised training data, it becomes possible to obtain extremely excellent segmentation performance capabilities upon completion of training.

Claims

1. A method (100) for training a segmentation neural network (30) to segment anatomical structures on a medical image representing a target anatomical region (41) of a patient, wherein the method (100) is Step (101) of acquiring segmented medical images (11, 12), each associated with a segmentation mask (21, 22), wherein the medical images (11, 12) are A majority medical image (11) associated with a majority segmentation mask (21) and representing the target anatomical site (41) of a patient without abnormalities, and A step (101) to acquire a minority medical image (12) associated with a minority segmentation mask (22) and representing the target anatomical site (41) of a patient with an abnormality (42), Step (102) involves training a generative neural network (31) to generate a composite image based on a segmentation mask, Step (103) of obtaining an artificial segmentation mask (23), wherein each artificial segmentation mask (23) is obtained by combining the majority segmentation mask (11) with the anomalous segmentation (44) of the minority segmentation mask (12), The steps include: (104) using the trained generative neural network (31) to obtain a composite image (13) based on the artificial segmentation mask (23); The steps include (105) acquiring unsegmented medical images (14), each representing a target anatomical region (41) of the patient and each associated with weak annotation, Step (106) of identically duplicating the segmentation neural network (30) in order to form a teacher network (30'), Includes, The aforementioned method, A supervised training step (107) of the segmentation neural network (30) based on supervised training data (25) including the segmented real medical images (11, 12) and associated segmentation masks (21, 22), and the composite image (13) and associated artificial segmentation mask (23), The weights of the segmentation neural network (30) are calculated using gradient descent. Ｓ Step (108) to update, The weights before the aforementioned teacher network (30') [Math 1] and the updated weights w of the segmentation neural network (30) Ｓ The weights of the training network (30') are calculated using an exponential moving average calculated as a function of w Ｔ Step (109) to update, Step (110) of generating a pseudo-segmentation mask (24) corresponding to the prediction of the training network (30') obtained based on the non-segmented medical images (14) and associated weak annotations, The steps include adding the generated pseudo-segmentation mask (24) and the unsegmented medical image (14) that enabled the generation of the pseudo-segmentation mask (24) (110) to the supervised training data (25) (111), Method (100), further comprising multiple iterations of the method.

2. With each iteration, A supervised cost function C represents the difference between the prediction made by the segmentation neural network (30) based on images belonging to the supervised training data (25) and the segmentation mask associated with the images. Ｓ Calculating and A semi-supervised cost function C represents the difference between the prediction of the teacher network (30') and the prediction of the segmentation neural network (30) performed on images belonging to the semi-supervised training data (26). ＳＳ Calculating and The aforementioned supervised cost function C Ｓ and the aforementioned semi-supervised cost function C ＳＳ The overall cost function C is calculated as a function of, The weights w of the segmentation neural network (30) using gradient descent, including the weights w Ｓ The method according to claim 1 (100), wherein the update (108) is performed as a function of the overall cost function.

3. The predictions of the training network (30') performed on images belonging to the semi-supervised training data (26) are enhanced with the weak annotations associated with the images, and the enhanced predictions are [Math 2] It was shown that, [Math 3] It can be written as, [Math 4] is the rough prediction of the teacher network (30'), y Ｆ corresponds to the weak annotation, and β is the contribution rate to the weak annotation, the method (100) according to claim 2.

4. The method according to claim 3 (100), wherein the contribution rate β of the weak annotation is in the range of 0.45 to 0.

55.

5. The method according to any one of claims 2 to 4 (100), wherein each weak annotation includes an approximate location of the anatomical structure to be segmented, the approximate location being embodied by one or more points of the anatomical structure to be segmented, a geometric shape encompassing a region within the anatomical structure to be segmented, or a geometric shape surrounding or encompassing the anatomical structure to be segmented.

6. The overall cost function C is the same as the supervised cost function C Ｓ and the aforementioned semi-supervised cost function C ＳＳ The method according to any one of claims 2 to 5 (100), which is calculated in the form of a weighted sum of the following:

7. The overall cost function is C = (1 - γ)C Ｓ +γC ＳＳ The method according to claim 6 (100), wherein γ is in the range of 0.05 to 0.

15.

8. The aforementioned supervised cost function C Ｓ This is the actual cost function C calculated based on the segmented medical images (11, 12) and associated segmentation masks (21, 22). Ｒｅ , as well as the composite cost function C calculated based on the composite image (13) and the associated artificial segmentation mask (23). Ｓｙｎ The method according to any one of claims 2 to 7 (100), which is calculated in the form of a linear combination of the following.

9. The overall cost function C is given by C = (1 - γ)(C Ｒｅ +αρC Ｓｙｎ ) + γC ｓｓ It is written that equation γ is the semi-supervised cost function C ＳＳ This is the contribution rate, where α is the composite cost function C Ｓｙｎ The method according to claim 8 (100), wherein ρ is the contribution rate of the composite image (13) calculated using a discriminator neural network (32) that forms a pair of adversarial generative networks together with the generative neural network (31).

10. The method according to claim 9 (100), wherein the contribution rate α is in the range of 0.35 to 0.45, and the contribution rate γ is in the range of 0.05 to 0.

15.

11. The method according to any one of claims 1 to 10 (100), wherein obtaining an artificial segmentation mask (23) (103) includes converting the anomalous segmentation (44) of the minority segmentation mask.

12. The method according to claim 11 (100), wherein transforming the abnormal segmentation (44) corresponds to rotation, enlargement, reduction, deformation, and / or displacement of the abnormal segmentation (44).

13. A method for automatically segmenting medical images, Training the segmentation neural network (30) according to any one of claims 1 to 12, The medical image is segmented using the aforementioned trained segmentation neural network (30), Methods that include...

14. A computer-readable storage medium comprising a trained segmentation neural network (30) according to any one of claims 1 to 12.

Images