Semi-supervised segmentation method based on multi-mode teacher-student consistency learning

Abstract

The invention is suitable for the technical field of medical image segmentation, and provides a semi-supervised segmentation method based on multi-modal teacher-student consistency learning, which comprises the following steps: constructing a teacher-double student model framework which comprises a multi-modal teacher model and two independent single-modal student models with different learning conditions, the two student models use segmentation networks of the same structure to respectively process images of two different modalities; in the data input and training stage, data with labels train a student model through real labels, and data without labels are trained through false labels generated by a teacher model; in the training process, the consistency prediction between the teacher model and the student model and between the two student models is ensured. According to the method, the problem of overfitting caused by insufficient annotation data is effectively relieved. In an actual medical scene with limited annotation data, the method can achieve a reliable segmentation result only by means of fewer labels, so as to relieve the pressure of medical conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image segmentation, and in particular relates to a semi-supervised segmentation method based on multimodal teacher-student consistency learning. Background Art

[0002] Multimodal medical image segmentation is crucial to improving diagnostic accuracy in various clinical settings. In order to achieve the best learning performance, it is usually necessary to accumulate a large amount of annotated data that can effectively guide how deep models generate outputs. However, the process of obtaining these annotated data requires the participation of domain experts, who must perform tedious annotation tasks and conduct multiple rounds of consensus building among peers. The preparation of datasets is not only time-consuming but can also be quite expensive. In addition, unlabeled data cannot provide effective supervision signals for the model. Therefore, how to train deep models for multimodal medical image segmentation with limited annotated data has become a pressing problem to be solved. For the problem of lack of segmentation labels, semi-supervised learning methods are usually used, that is, training with labeled and unlabeled data to help segmentation models learn rich label information from unlabeled images, especially when only a limited amount of annotated data is usually available in medical image segmentation.

[0003] Currently, the most advanced semi-supervised multimodal medical image segmentation methods mainly rely on two frameworks: the teacher-student framework and the cross-pseudo-supervision (CPS) framework. In the teacher-student framework, the student model is first trained using labeled data, and then the teacher model is generated by the exponential moving average (EMA) of the student model. Next, the pseudo-labels generated from the teacher model are used together with the labeled data to further train the student model, and its performance is enhanced through this collaborative feedback loop. In the cross-pseudo-supervision framework, the consistency between the two perturbed models is exploited, and the pseudo-labels generated by one network are used to train the other network. However, the generation of pseudo-labels in the semi-supervised learning process usually relies on the model's own predictions or some heuristic methods. If the model is biased in the initial stage, or the heuristic method cannot accurately capture the intrinsic characteristics of the data, the generated pseudo-labels may contain errors, which will affect the training process and the performance of the final model.

[0004] To solve the above problems, the present invention proposes a semi-supervised segmentation method based on multimodal teacher-student consistency learning. Summary of the invention

[0005] The purpose of the present invention is to provide a semi-supervised segmentation method based on multimodal teacher-student consistency learning, aiming to solve the problems raised in the above background technology.

[0006] The purpose of the present invention is achieved through the following technical solutions: The semi-supervised segmentation method based on multimodal teacher-student consistency learning includes the following steps: Step 1: Construct a teacher-dual student model framework: The teacher-dual student model framework includes a multimodal teacher model and two independent unimodal student models with different learning conditions; the two unimodal student models use the same segmentation network to process images of two different modalities respectively; Step 2, data input and training: The images of two different modalities in the multimodal teacher model are input into the two unimodal student models respectively; for labeled data, the unimodal student model is trained by the corresponding real labels; for unlabeled data, the multimodal teacher model guides the training of the unimodal student model by generating pseudo labels; during the training process, the consistency between the multimodal teacher model and the unimodal student model is maintained, and consistent predictions are generated between the two unimodal student models.

[0007] Furthermore, in step 2, the two unimodal student models use a hybrid segmentation loss combining cross entropy loss and Dice loss in the medical image segmentation task, which is defined as follows:

[0008] in, Indicates the first mode The segmentation loss of the student model; Indicates the second mode The segmentation loss of the student model; Indicates the corresponding first mode m 1 student model; Indicates the corresponding second mode m 2 student models; and Represents the first modality dataset from the labeled And the second modality dataset Extract samples from and expectations; represents the cross entropy loss; represents Dice loss, which is used to measure the degree of match between the model output and the true label; and Represent the true labels of the first modality and the second modality respectively; Represents the first modality student model for the input image The predicted output of Represents the second modality student model for the input image The predicted output.

[0009] Furthermore, in step 2, when the images of two different modalities are respectively input into two single-modal student models, consistent predictions are generated, and a consistent learning strategy is used to reduce the prediction deviation between the two images:

[0010] in, represents the consistency loss between two student models; Indicates the unmarked first mode Datasets; Indicates the unlabeled second mode Datasets; express and The symmetric KL divergence between Represents the first modality student model for the input image The predicted output of Represents the second modality student model for the input image The predicted output.

[0011] Furthermore, in step 2, the specific implementation steps for maintaining the consistency between the multimodal teacher model and the unimodal student model are as follows: The multimodal teacher model is configured with the same architecture as the unimodal student model, so that the multimodal teacher model contains four modalities at the same time, and consistency learning is performed by aligning the outputs of the multimodal teacher model with the two unimodal student models;

[0012] in, represents the consistency loss between the teacher and student models; T represents the teacher model; Represents a multimodal input data sample composed of four modalities; m 4 represents the spliced ​​image of the four modalities of the BraTS dataset.

[0013] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a semi-supervised segmentation method based on multimodal teacher-student consistency learning, which effectively alleviates the overfitting problem caused by insufficient labeled data by ensuring semantic consistency and utilizing unlabeled data. In addition, the method also introduces multimodal knowledge distillation technology to promote consistent learning of pseudo labels. In actual medical scenarios with limited labeled data, this method can achieve reliable segmentation results with only a few labels, thus helping to alleviate the pressure of medical conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1It is a flow chart of the present invention.

[0015] Figure 2 It is a framework diagram of the present invention.

[0016] Figure 3 The following are the results of multimodal medical image segmentation under different models. DETAILED DESCRIPTION

[0017] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below, but it should not be understood as limiting the scope of the present invention. The experimental methods described in the following examples are conventional methods unless otherwise specified; the reagents and materials can be obtained from commercial sources unless otherwise specified.

[0018] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0019] like Figure 1 and Figure 2 As shown, a semi-supervised segmentation method based on multimodal teacher-student consistency learning provided by an embodiment of the present invention includes the following steps: Step 1: Construct a teacher-dual student model framework: The teacher-dual student model framework includes a multimodal teacher model and two independent unimodal student models with different learning conditions; the two unimodal student models use the same segmentation network to process images of two different modalities respectively; Step 2, data input and training: Input the images of two different modalities in the multimodal teacher model into the two unimodal student models respectively; for labeled data, the unimodal student model is trained with the corresponding real labels to ensure accurate segmentation of the target area. For unlabeled data, the multimodal teacher model generates pseudo labels to guide the training of the unimodal student model, ensuring that high-quality pseudo labels provide sufficient supervision information for the student model to make up for the lack of unlabeled data, so that the network can continuously improve the segmentation performance.

[0020] During the training process, the consistency between the multimodal teacher model and the unimodal student model is maintained. Specifically, the multimodal teacher model is configured with the same architecture as the unimodal student model. Through multimodal information fusion, the prediction results generated by the multimodal teacher model are aligned with the outputs of the two unimodal student models, thereby achieving cross-modal consistency learning. This consistency mechanism not only improves the robustness of the model, but also makes more full use of the complementary information between multimodal images, thereby improving the accuracy of segmentation.

[0021] At the same time, in order to ensure information sharing between multiple modalities and enhance the accuracy of segmentation, the two single-modal student models should produce consistent prediction outputs when processing images of their respective modalities. By using a consistent learning strategy, the prediction deviation between the two groups of images can be effectively reduced.

[0022] The scheme of step 2 includes: (1) Dual semi-supervised student segmentation network: The two unimodal student models widely use a hybrid segmentation loss that combines cross entropy loss and Dice loss. The loss is defined as follows: The two unimodal student models use a hybrid segmentation loss that combines cross entropy loss and Dice loss in the medical image segmentation task, defined as follows:

[0023] in, Indicates the first mode The segmentation loss of the student model; Indicates the second mode The segmentation loss of the student model; Indicates the corresponding first mode m 1 student model; Indicates the corresponding second mode m 2 student models; and Represents the first modality dataset from the labeled And the second modality dataset Extract samples from and expectations; represents the cross entropy loss; represents Dice loss, which is used to measure the degree of match between the model output and the true label; and Represent the true labels of the first modality and the second modality respectively; Represents the first modality student model for the input image The predicted output of Represents the second modality student model for the input image The predicted output.

[0024] In addition, in order to maximize the use of unlabeled data, it is assumed that images from both modalities should produce consistent predictions when fed into two unimodal student models. At the same time, a consistency learning strategy is used to reduce the prediction bias between the two sets of images:

[0025] in, represents the consistency loss between two student models; Indicates the unmarked first mode Datasets; Indicates the unlabeled second mode Datasets; express and The symmetric KL divergence between Represents the first modality student model for the input image The predicted output of Represents the second modality student model for the input image The predicted output.

[0026] Therefore, the two unimodal student models can not only learn the semantic consistency between modalities, but also utilize unlabeled data for semi-supervised training to make up for the lack of supervisory signals and alleviate the overfitting problem caused by insufficient labeling.

[0027] (2) Multimodal knowledge distillation for pseudo-label consistency learning: The multimodal teacher model is configured with the same architecture as the unimodal student model, the input of the multimodal teacher model is changed to include four modalities at the same time, and then consistency learning is performed by aligning the outputs of the multimodal teacher model with the two unimodal student models;

[0028] in, represents the consistency loss between the teacher and student models; T represents the teacher model; Represents a multimodal input data sample composed of four modalities; m 4 represents the spliced ​​image of the four modalities of the BraTS dataset.

[0029] Example 1: The present invention achieves better segmentation results on the registered BraTS dataset, outperforming other competing methods; Specifically, the present invention (consistency learning model) is compared with the baseline model (using the Unet method), the Y-shaped model, the X-shaped model, the Ummkd model, the SSUML model, the DSAN model, the LE-UDA model, and the ASTCMSeg model in multimodal medical image segmentation on MRI (magnetic resonance imaging) and CT (computed tomography) imaging. The results are as follows Figure 3 As shown, compared with the segmentation results of other models, the segmentation results of the method of the present invention are obviously closer to the true labels in terms of edges and more segmentation details.

[0030] In the segmentation tasks of the whole tumor (WT), enhanced tumor (ET) and tumor core (TC) regions, the model of the present invention has achieved significant improvements in evaluation indicators such as Dice Similarity Coefficient (DSC), sensitivity (Sensitivity) and precision (Precision). Specifically: compared with the baseline model, the DSC of the model of the present invention in the whole tumor region increased by 11.4% to 88.7%; the sensitivity in the enhanced tumor region increased by 7.5%; the precision in the tumor core region increased by 8.9%, and the specific results are shown in Table 1.

[0031]

[0032] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A semi-supervised segmentation method based on multimodal teacher-student consistency learning, characterized by: The following steps are involved: Step 1: Construct a teacher-dual student model framework: The teacher-dual student model framework includes a multimodal teacher model and two independent unimodal student models with different learning conditions; the two unimodal student models use the same segmentation network to process images of two different modalities respectively; Step 2, data input and training: The images of two different modalities in the multimodal teacher model are input into the two unimodal student models respectively; for labeled data, the unimodal student model is trained by the corresponding real labels; for unlabeled data, the multimodal teacher model guides the training of the unimodal student model by generating pseudo labels; during the training process, the consistency between the multimodal teacher model and the unimodal student model is maintained, and consistent predictions are generated between the two unimodal student models.

2. The semi-supervised segmentation method based on multimodal teacher-student consistency learning according to claim 1 is characterized in that: In step 2, the two unimodal student models use a hybrid segmentation loss that combines cross entropy loss and Dice loss in the medical image segmentation task, which is defined as follows: ； ； in, Indicates the first mode The segmentation loss of the student model; Indicates the second mode The segmentation loss of the student model; Indicates the corresponding first mode m 1 student model; Indicates the corresponding second mode m 2 student models; and Represents the first modality dataset from the labeled And the second modality dataset Extract samples from and expectations; represents the cross entropy loss; represents Dice loss, which is used to measure the degree of match between the model output and the true label; and Represent the true labels of the first modality and the second modality respectively; Represents the first modality student model for the input image The predicted output of Represents the second modality student model for the input image The predicted output.

3. The semi-supervised segmentation method based on multimodal teacher-student consistency learning according to claim 2 is characterized in that: In step 2, when two images of different modalities are respectively input into two single-modal student models, consistent predictions are generated, and a consistent learning strategy is used to reduce the prediction deviation between the two images: ； in, represents the consistency loss between two student models; Indicates the unmarked first mode Datasets; Indicates the unlabeled second mode Datasets; express and The symmetric KL divergence between Represents the first modality student model for the input image The predicted output of Represents the second modality student model for the input image The predicted output.

4. The semi-supervised segmentation method based on multimodal teacher-student consistency learning according to claim 1, characterized in that: In step 2, the specific implementation steps for maintaining the consistency between the multimodal teacher model and the unimodal student model are as follows: The multimodal teacher model is configured with the same architecture as the unimodal student model, so that the multimodal teacher model contains four modalities at the same time, and consistency learning is performed by aligning the outputs of the multimodal teacher model with the two unimodal student models; ； in, represents the consistency loss between the teacher and student models; T represents the teacher model; Represents a multimodal input data sample composed of four modalities; m 4 represents the spliced ​​image of the four modalities of the BraTS dataset.

Images