A semi-supervised dynamic self-learning segmentation method for cardiac MRI

Abstract

The present invention discloses a semi-supervised dynamic self-learning segmentation method for cardiac MRI. This method uses a generator to generate corresponding labels for unlabeled data, generates an average pseudo-label with the assistance of an exponential dynamic network and dynamically selects it through entropy constraint. At the same time, a discriminator is used to determine the confidence of these pseudo-labels, and different annotation sources are considered during the training process. The present invention uses dynamically allocated weights to allocate weights to the training at different times, and uses feature matching constraints to reduce the inconsistency of data distribution between the generated data and the real data. Therefore, the method of the present invention can, to a certain extent, solve the dependence of traditional fully supervised methods on the amount of annotation, so as to better assist intelligent medical analysis and diagnosis in the case of a small amount of annotation.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image segmentation, and particularly relates to a semi-supervised dynamic self-learning segmentation method for cardiac MRI. Background Art

[0002] Most of the image segmentation methods based on deep learning require a large number of labeled images for model fitting training. However, due to the particularity of cardiac images, it is difficult to obtain a large amount of training data. At the same time, annotating these images is also a time-consuming and laborious task, which makes the fully supervised cardiac image segmentation task always restricted by the annotation quality and quantity of the images. Due to these factors, semi-supervised methods have gradually become popular.

[0003] Semi-supervised learning allows us to use a small amount of annotations as training labels, and at the same time, we can use a large amount of unlabeled data as additional supplementary information to enhance the performance of the model. This semi-supervised learning method can be summarized into three categories: graph-based methods, co-training methods, and self-learning methods. Among them, using a graph as a data-related prior is a relatively traditional method. This method designs a non-parametric Bayesian Gaussian random field to segment images and can better estimate the data distribution of graph-based methods. This method usually regards samples and their similarities as nodes and edges respectively, and the percentage of samples and their similarities as nodes and edges of the graph respectively. Connected nodes may have the same label. However, performing operations such as multiple loop matching to match the labels of all graph nodes will lead to the complexity of the operations.

[0004] With the popularity of deep learning, co-training and self-training methods have gradually become the mainstream ways of semi-supervised learning. Co-training generally uses data from different data domains to assist each other's training. Different views or supplementary structures of image features can be used as the content for assistance. However, this method still cannot overcome the problem of data volume limitation. A large amount of different data domains is still the key to model training.

[0005] Different from co-learning that requires a large number of different data domains for assistance, self-learning only requires data from its own data domain, thus greatly reducing the data volume requirement. The classifier first uses labeled data for conventional supervised learning, and then the remaining unlabeled data can be used to assist training through traditional algorithms, teacher-student mutual learning networks, reconstruction assistance, or adversarial generative learning. Among them, the currently more advanced method is to use an attention structure-based generative network. Unlabeled data will be input into a confidence network to obtain feature maps, and the qualified parts on a feature map will be extracted for the self-learning process. However, since this method only selects a part of the feature maps and does not consider the global information, the learning of the network is likely to be misled.

[0006] All the above-mentioned methods directly fuse the generated pseudo-labels with the real labels, ignoring the fact that they come from different source domains. The key to the success of semi-supervised learning lies in generating pseudo-labels of high enough quality for auxiliary training of the model. Poor-quality pseudo-labels may cause fluctuations in training. Therefore, how to reasonably fuse labels from different domains becomes a research focus worthy of consideration. Summary of the Invention

[0007] In view of the above, the present invention provides a semi-supervised dynamic self-learning segmentation method for cardiac MRI, which can, to a certain extent, solve the dependence of traditional fully supervised methods on the amount of annotation, thereby better assisting intelligent medical analysis and diagnosis with a small amount of annotation.

[0008] A semi-supervised dynamic self-learning segmentation method for cardiac MRI includes the following steps:

[0009] (1) Classify the labeled and unlabeled cardiac MRI images and normalize these images to the same size.

[0010] (2) Input the labeled cardiac MRI images into a generator based on a deep learning neural network to extract feature information and output corresponding feature probability maps.

[0011] (3) Use a cross-entropy function based on dynamic weight assignment to constrain the training of the neural network of the generator, and continuously iterate until convergence.

[0012] (4) Input the unlabeled cardiac MRI images into the generator to generate corresponding feature probability maps, and after passing the feature probability maps through the argmax function, fuse them with the original cardiac MRI images and input them into the discriminator. The discriminator will output an indication value for determination and a feature vector for loss constraint.

[0013] (5) If the indication value is greater than a set threshold, use a dynamic label screening network, that is, extract the feature probability maps of the same images from the Exponential Moving Average model, and screen them under entropy constraint with the feature probability maps generated in step (4) to select the feature probability map with the lowest entropy value as the pseudo-label of the unlabeled cardiac MRI image, and then perform self-learning supervision training on the neural network of the generator.

[0014] (6) Input the labeled cardiac MRI image x a and the unlabeled cardiac MRI image x u into the generator respectively to generate corresponding feature probability maps P a and P u , and at the same time input the unlabeled cardiac MRI image xu Input the dynamic exponential average network to generate the feature probability map P ema ; Combine P a with x a to form f a , Combine P u with x u to form f u ;

[0015] (7) Input f a and f u into the discriminator respectively. The discriminator will output two indication values I a and I u as well as two feature vectors F a and F u . Use the L1 norm to constrain the two feature vectors F a and F u . Use binary cross-entropy constraint for I a with 1 and binary cross-entropy constraint for I u with 0 to complete the training of the discriminator and finally use it for image segmentation.

[0016] Furthermore, the generator adopts a Deeplab generator based on Resnet-101.

[0017] Furthermore, the definition expression of the feature probability map is as follows:

[0018]

[0019] where: P(i, j, n) is the feature probability value of the element in the i-th row and j-th column of the feature probability map corresponding to the n-th class of images, f(i, j, n) is the feature information of the element in the i-th row and j-th column of the n-th class of images, and N is the total number of categories of cardiac MRI images.

[0020] Furthermore, the expression of the cross-entropy function based on dynamic weight allocation is as follows:

[0021]

[0022]

[0023]

[0024] where: L s is the dynamic weight cross-entropy function of the generator for labeled images, is the dynamic weight cross-entropy of the generator for unlabeled images, For the dynamic weight cross-entropy of the dynamic exponential average network for unannotated images, P(i, j, n) is the feature probability value of the element at the i-th row and j-th column in the feature probability map corresponding to the n-th class of images, W and H are the width and height of the image respectively, and y a (i, j, n) is the annotation information corresponding to the element at the i-th row and j-th column in the n-th class of annotated images, and y u (i, j, n) is the feature information extracted by the generator for the element at the i-th row and j-th column in the n-th class of unannotated images, which is P u The pseudo-annotation map after argmax, and y ema (i, j, n) is the feature information extracted by the dynamic exponential average network for the element at the i-th row and j-th column in the n-th class of unannotated images, which is P ema The pseudo-annotation map after argmax, is the dynamic weight.

[0025] Furthermore, the dynamic weight has the following expression:

[0026]

[0027]

[0028]

[0029] where: W s is the weight based on the training time (its value increases continuously with time for unannotated images and decreases continuously with time for annotated images), I current is the current training cycle, and I total is the total number of training cycles, is the weight based on the inverse ratio of regions, R b is the area occupied by background pixels, and R N is the pixel area occupied by a certain class.

[0030] Furthermore, the discriminator adopts a four-layer neural network structure.

[0031] Furthermore, in steps (3) and (5), the cross-entropy function based on dynamic weight allocation uses the stochastic gradient descent method to train the neural network of the generator.

[0032] Furthermore, in step (7), the neural network of the discriminator is trained according to the binary cross-entropy using the Adam gradient descent algorithm.

[0033] The present invention combines a deep learning segmentation network with a generative adversarial network. The deep learning segmentation network uses labeled images to perform conventional segmentation tasks. At the same time, the segmentation network can also be used as a generator of the discriminator to generate pseudo-labels and provide inputs for the discriminator. The overall network uses an exponential moving average network for dynamic label screening and uses dynamic weight allocation to reasonably fuse labels from different domains. The entire training process is an end-to-end closed loop, and the unlabeled data is continuously iterated in the generator and the discriminator in a dynamic self-learning manner, thereby optimizing the structures of both parts simultaneously. Under this self-learning loop operation that considers annotation information from different domains, the network will be able to generate high-quality pseudo-labels to supervise learning, so that the final test results of the network can achieve an effect similar to that of full supervision, thus effectively solving the problem of insufficient training labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic structural diagram of the semi-supervised model with dynamic learning of the present invention.

[0035] Figure 2 It is a schematic flow chart of the steps of the image segmentation method of the present invention.

[0036] Figure 3 (a)(1) are semi-supervised, fully supervised, and self-labeled edge contour images (axial slice, epicardium) generated using half of the training data.

[0037] Figure 3 (a)(2) are semi-supervised, fully supervised, and self-labeled edge contour images (axial slice, epicardium) generated using one-fourth of the training data.

[0038] Figure 3 (a)(3) are semi-supervised, fully supervised, and self-labeled edge contour images (axial slice, epicardium) generated using one-eighth of the training data.

[0039] Figure 3 (b)(1) are semi-supervised, fully supervised, and self-labeled edge contour images (axial slice, endocardium) generated using half of the training data.

[0040] Figure 3 (b)(2) are semi-supervised, fully supervised, and self-labeled edge contour images (axial slice, endocardium) generated using one-fourth of the training data.

[0041] Figure 3 (b)(3) are semi-supervised, fully supervised, and self-labeled edge contour images (axial slice, endocardium) generated using one-eighth of the training data. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] To describe the present invention more specifically, the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0043] The semi-supervised model of the present invention is as Figure 1 shown. The labeled image x a is input into the Deeplab generator based on Resnet101 to extract the feature information f, and the corresponding output prediction feature map P is obtained. The prediction map will be calculated with the label using the cross-entropy loss function. At the same time, the unlabeled data x u will also be input into the generator to generate a probability prediction map P. The definition of P is as follows:

[0044]

[0045] where: (i, j) is the coordinate position of each corresponding point of the feature map f, and N is the total number of classification categories.

[0046] The unlabeled image x u is input into the Deeplab generator based on Resnet101 to extract the feature information f u , and the corresponding output prediction feature map P is obtained u . At the same time, this image will also enter the exponential moving average network to obtain f ema , and the corresponding output prediction feature map P is obtained ema .

[0047] To determine the reliability of the prediction map, the prediction map will be fused with x u and input into the discriminator (the discriminator is a four-layer neural network). The discriminator will output an indicator p d . If the value of the indicator is greater than the set threshold, this probability map will be passed through argmax and used as the pseudo-label obtained from the exponential moving average network for entropy screening to obtain the most suitable pseudo-label for self-learning training. The loss function for training is:

[0048]

[0049]

[0050]

[0051] In each iteration cycle, the labeled image will pass through the generator to generate a probability distribution map P a , and the unlabeled image will also pass through the generator to generate a probability distribution map P u . To make the output data distribution unified, P a will be fused with x aConcatenation operations will be performed to form f a ; P u will be concatenated with x u to form f u , f a and f u will be input into the discriminator, and the discriminator will output two feature vectors F a and F u . We will use the L1 norm constraint on the two vectors, i.e., L fm = ‖F a - F u ‖ to increase the consistency of the data distribution, where F a and F u are expressed as follows:

[0052] F a = D(C(y a , x a ))

[0053] F u = D(C(p, x u ))

[0054] Meanwhile, the fused f a and f u will generate indicators I a and I u through the discriminant network. I a will be subject to binary cross-entropy constraint with 1, and I u will be subject to binary cross-entropy constraint with 0. The discriminator constraint L d is as follows; under this constraint, the discriminator will be able to better distinguish the difference between the pseudo-labels generated by the generator and the real labels.

[0055]

[0056] where: Dis a , Dis u are the data distributions of the labeled set and the unlabeled set respectively.

[0057] The process of the method of the present invention is as Figure 2 shown. First, a labeled data set is randomly divided as:

[0058] (x a , y a ) = random_select(x, ratio)

[0059] The remaining part will be the unlabeled data set, where ratio is

[0060] Then, construct the generator G and the discriminator D:

[0061] For the generator, for the labeled part, the SGD gradient descent algorithm is used to iteratively solve the cross-entropy of the objective function, and the following iterative steps are obtained:

[0062]

[0063]

[0064] where: α is the learning rate, w is the matrix parameter of the network, is the gradient operator, N is the total number of classification categories, and the stochastic gradient descent algorithm is used overall to accelerate the convergence speed.

[0065] The iterative convergence conditions are as follows:

[0066] |L′ s -L s ″|≤∈

[0067] where: L′ s and L″ s are the arithmetic functions before and after, ∈ is the convergence threshold, and is set to 10 -4 .

[0068] Next, we use right ventricular MRI images for experiments to verify the effectiveness of this embodiment. While using the method of the present invention for segmentation, we also use the fully supervised method under the same dataset division for segmentation, compare the segmentation results of the two, and divide the data at a ratio of respectively, and normalize the images for experiments. Table 1 shows the specific settings.

[0069] Table 1

[0070]

[0071] For the unlabeled part, for the segmentation network, the SGD algorithm is also used for gradient descent to converge the model, and for the discriminator, we use the Adam algorithm for gradient descent.

[0072]

[0073] m t =β1*m t-1 +(1-β1*g t )

[0074]

[0075]

[0076]

[0077]

[0078] Among them: β1, β2, α, and ∈ are hyperparameters with values of 0.9, 0.999, 0.001, and 10 respectively -8 , where m and v are momentum.

[0079] Our experiments used an NVIDIA 1080Ti GPU for accelerated computing. The total number of sets for overall training was 4. The images were resized to 216×216. The initial learning rates for the generation network and the discriminator network were 0.00025 and 0.0001 respectively. The threshold of the indicator was 0.6. The entire training process required 13,000 iterations; the fully supervised training dataset was consistent with its corresponding semi-supervised labeled data, and the same generative segmentation network was used. Figure 3 (a)(1)~ Figure 3 (a)(3) are the right ventricular epicardium result diagrams obtained using labels with proportions of 1 / 2, 1 / 4, and 1 / 8 respectively. Figure 3 (b)(1)~ Figure 3 (b)(3) are the right ventricular endocardium segmentation contour lines obtained using labels with proportions of 1 / 2, 1 / 4, and 1 / 8 respectively. The light curves represent our semi-supervised contours, and the dark curves represent the fully supervised contours. Each row represents the same image. From Figure 3 (a)(1)~ Figure 3 (a)(3) and Figure 3 (b)(1)~ Figure 3 (b)(3), it can be found that as the amount of labeled data decreases, the overall segmentation result deteriorates. However, from Figure 3 (a)(1)~ Figure 3 (a)(3), it can be seen that although the overall segmentation result deteriorates as the amount of labeled data decreases, compared with the fully supervised method, the results of the present invention are more continuous and accurate. This shows that the semi-supervised segmentation method of the present invention can make the most of unlabeled data to achieve better segmentation accuracy with a small amount of labels; we not only used a dynamic label screening mechanism to screen higher-quality pseudo-labels to assist the convergence of the network, but also used a dynamic weight allocation constraint to combine labels from different source domains to maximize the use of label information, thereby obtaining more accurate results than the fully supervised method. Thus, we can alleviate the shortage of labels to a certain extent and better assist in the analysis of intelligent medical images with a small amount of labels.

[0080] The above description of the embodiments is provided to enable those of ordinary skill in the art to understand and apply the present invention. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative efforts. Therefore, the present invention is not limited to the above embodiments, and all improvements and modifications made by those skilled in the art based on the disclosure of the present invention should fall within the protection scope of the present invention.

Claims

1. A semi-supervised dynamic self-learning segmentation method for cardiac MRI, comprising the following steps: (1) Classify the labeled and unlabeled cardiac MRI images and normalize these images to the same size. (2) Input the labeled cardiac MRI images into a generator based on a deep learning neural network to extract feature information and output corresponding feature probability maps; the generator adopts a Deeplab generator based on Resnet-101; the definition expression of the feature probability maps is as follows: Wherein: P (i,j,n) is the feature probability value of the element in the i-th row and j-th column of the corresponding feature probability map for the n-th class of images, f(i,j,n) is the feature information of the element in the i-th row and j-th column of the n-th class of images, and N is the total number of classes of cardiac MRI images. (3) Use a cross-entropy function based on dynamic weight allocation to constrain and train the neural network of the generator, and continuously iterate until convergence; the expression of the cross-entropy function based on dynamic weight allocation is as follows: Among them: L s is the dynamic weight cross-entropy function of the generator for labeled images, is the dynamic weight cross-entropy of the generator for unlabeled images, is the dynamic weight cross-entropy of the dynamic exponential moving average network for unlabeled images. P(i, j, n) is the feature probability value of the element in the i-th row and j-th column of the feature probability map corresponding to the n-th class of images. W and H are the width and height of the image respectively, and y a (i, j, n) is the annotation information corresponding to the element in the i-th row and j-th column of the n-th class of labeled images, and y u (i, j, n) is the feature information extracted by the generator for the element in the i-th row and j-th column of the n-th class of unlabeled images, which is P u the pseudo-annotation map after argmax, and y ema (i, j, n) is the feature information extracted by the dynamic exponential moving average network for the element in the i-th row and j-th column of the n-th class of unlabeled images, which is P ema the pseudo-annotation map after argmax, is the dynamic weight, and W s is the weight based on the training time, and I current is the current training epoch, and I total is the total number of training epochs, is the weight based on the inverse ratio of regions, and R b is the area occupied by background pixels, and R N is the pixel area occupied by a certain class; (4) Input the unlabeled cardiac MRI images into the generator to generate corresponding feature probability maps, and after passing the feature probability maps through the argmax function, fuse them with the original cardiac MRI images and input them into the discriminator, and the discriminator will output an indication value for determination and a feature vector for loss constraint; the discriminator adopts a four-layer neural network structure. (5) If the indication value is greater than the set threshold, use a dynamic label screening network, that is, extract the feature probability maps of the same images from the dynamic exponential moving average network, and screen them under entropy constraint with the feature probability maps generated in step (4), and select the feature probability map with the lowest entropy value as the pseudo-label of the unlabeled cardiac MRI images, and then perform self-learning supervision training on the neural network of the generator. (6) Input the labeled cardiac MRI image x a and the unlabeled cardiac MRI image x u into the generator respectively, and correspondingly generate the feature probability maps P a and P u . At the same time, input the unlabeled cardiac MRI image x u into the dynamic exponential moving average network to generate the feature probability map P ema ; fuse P a with x a to form f a , and fuse P u with x u to form f u ; (7) Input f a and f u into the discriminator respectively. The discriminator will correspondingly output two indication values I a and I u as well as two feature vectors F a and F u . Constrain the two feature vectors F a and F u using the L1 norm, constrain I a with 1 by binary cross-entropy, and constrain I u with 0 by binary cross-entropy, complete the training of the discriminator and finally use it for image segmentation.

2. The semi-supervised dynamic self-learning segmentation method according to claim 1, wherein: In steps (3) and (5), the cross-entropy function based on dynamic weight allocation uses the stochastic gradient descent method to train the neural network of the generator.

3. The semi-supervised dynamic self-learning segmentation method according to claim 1, characterized in that: In step (7), the Adam gradient descent algorithm is used to train the neural network of the discriminator according to binary cross-entropy.

Images