Semi-supervised medical image segmentation method based on knowledge distillation
By adopting the Self-Training framework of knowledge distillation and feature perturbation in medical image segmentation, using teacher assistants and deterministic knowledge transfer, the problem of unlabeled data utilization in semi-supervised learning is solved, and high-performance segmentation of medical images is achieved, which significantly improves the The learning efficiency and generalization ability of the model are achieved.
Patent Information
- Application Number
- CN202411249023.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-12-13
AI Technical Summary
In the field of medical image segmentation, existing semi-supervised learning methods are difficult to effectively utilize unlabeled data, especially under the high cost and complexity of manual labeling. How to effectively mine and utilize unlabeled data resources is a core problem.
Using Self-Training training framework based on knowledge distillation and feature perturbation, by introducing teacher assistants and deterministic knowledge transfer mechanisms, the feature perturbation from weak to strong is achieved through label-free data, and the self-training of the model is realized through multiple decoders to ensure that The correctness of knowledge transmission and the ability gap.
The model's learning efficiency and generalization ability on unlabeled data is significantly improved, achieving high-performance segmentation of medical images under semi-supervised, especially in distance metrics.
Smart Images
Figure CN119130978A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of semi-supervised medical image segmentation, and in particular to a semi-supervised medical image segmentation method based on knowledge distillation. Background Art
[0002] Semantic segmentation is an accurate pixel-level classification technology, which is crucial in medical image analysis. Especially when processing 3D CT and MRI data, fully supervised learning methods are limited by the high cost and complexity of manual annotation. In order to solve the bottleneck problem of the widespread application of deep learning in the field of medical image segmentation, semi-supervised medical image segmentation research has emerged and quickly become a hot topic. The key is to use a small amount of labeled data combined with a large amount of unlabeled data for effective training to overcome the high-cost annotation problem.
[0003] The core challenge of semi-supervised learning is how to effectively mine and utilize unlabeled data resources. The current research frontier has gradually shifted from adversarial training mechanisms driven by generative adversarial networks (GANs) to a broader consistency regularization framework, and self-training strategies are also showing strong development momentum. In this field, co-teaching and mutual learning paradigms have proven to be promising methods, usually involving simultaneous parallel training of two models. DMD further introduces this concept of collaborative training into the scope of knowledge distillation, verifying the fact that optimizing simplified model structures through knowledge distillation can also achieve excellent performance.
[0004] In recent years, advanced methods have further explored image-level perturbations and feature-level perturbations. When processing unlabeled images, applying various types of perturbations is a common strategy, and many methods are committed to exploring a wider range of perturbation strategies. In the field of image segmentation, the potential for feature perturbations is huge and has not been fully explored. It perturbs the feature information extracted by the Encoder to a certain extent and sends the perturbed features to the Decoder. Similar to consistency regularization, this process guides and constrains training under heavy feature perturbations by using the prediction results produced by mild feature perturbations, thereby ensuring that the model maintains consistent performance on the same image segmentation task under different levels of perturbations.
[0005] Inspired by knowledge distillation and feature perturbation technology, the present invention innovatively designs DistillMatch, which deepens and expands the consistency regularization framework in self-training. DistillMatch perturbs the high-dimensional feature graph of unlabeled data from weak to strong, and inputs the perturbed features into the Decoder for decoding, and uses the prediction consistency of the model under different perturbations to perform internal knowledge transfer: the weak perturbation model (teacher) guides the learning process of the strong perturbation model (teacher), thereby optimizing the learning efficiency and generalization ability of the model on unlabeled data. In this process, the role of teacher assistant (TA) is set in the Self-KD structure in order to bridge the performance gap. At the same time, in order to ensure the effectiveness and accuracy of knowledge transfer, the present invention also proposes a deterministic knowledge transfer strategy (DKT), which uses a threshold method to filter uncertain knowledge. Summary of the invention
[0006] The present invention proposes a semi-supervised medical image segmentation method based on knowledge distillation. The method proposes a self-training training framework based on knowledge distillation and feature perturbation, which focuses on knowledge transfer for unlabeled data to meet the challenges of medical image segmentation. The correctness of knowledge transfer and the reduction of ability gap are ensured by introducing deterministic knowledge transfer and teacher assistants. High-performance computing is achieved through equivalent multi-decoders. Experimental results on two benchmark data sets show that compared with the previous best results, there is a significant performance improvement, especially in the distance metric. The technical solution adopted by this method includes the following steps:
[0007] Step 1: Design a Self-Training framework to add different degrees of feature perturbations between the encoder and the decoder;
[0008] Step 2: Apply the teacher assistant between the decoders in 1) for self-model parameter training;
[0009] Step 3: Introduce a knowledge transfer mechanism during model training for semi-supervised learning;
[0010] Step 4: Use efficient multi-decoding strategy to complete the model training process;
[0011] The semi-supervised medical image segmentation method based on knowledge distillation proposed in this invention has the following advantages over the traditional semi-supervised medical image segmentation method:
[0012] (1) This paper proposes a self-training framework based on knowledge distillation and feature perturbation, which focuses on knowledge transfer for unlabeled data to meet the challenge of medical image segmentation and can effectively segment medical images under semi-supervision.
[0013] (2) The present invention ensures the correctness of knowledge transfer and reduces the ability gap by introducing deterministic knowledge transfer and teacher assistants, and achieves high-performance computing through equivalent multi-decoders. Experimental results on two benchmark datasets show that there is a significant performance improvement compared with the previous best results. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 Schematic diagram of the semi-supervised medical image segmentation method based on knowledge distillation proposed in the present invention. DETAILED DESCRIPTION
[0015] The method of the present invention is described in detail with reference to the accompanying drawings and embodiments.
[0016] Schematic diagram of the semi-supervised medical image segmentation method based on knowledge distillation Figure 1 The overall process of this method is as follows: first, the samples with image-level perturbations are sent to the Self-Training training framework, then different degrees of perturbations are performed during feature transfer between the encoder and the decoder, and the teacher assistant and the determined knowledge transfer strategy are used at the output of the decoder to implement the self-training process of the model.
[0017] Step 1: Design a Self-Training framework to add different degrees of feature perturbations between the encoder and the decoder;
[0018] The present invention designs a Self-Training training framework, which uses the following formula to calculate the loss:
[0019]
[0020] Step 2: Apply the teacher assistant between the decoders in 1) for self-model parameter training;
[0021] Decoders with different perturbations are regarded as teachers, teaching assistants, and students. With the help of multiple teacher assistant networks, the performance gap between teachers and students can be narrowed. For the convenience of description, in the pairwise relationship, the stronger Decoder (teacher) provides guidance to the weaker (student), with the goal of imparting knowledge and thus improving the overall network capability.
[0022] Step 3: Introduce a knowledge transfer mechanism during model training for semi-supervised learning;
[0023] In knowledge distillation and semi-supervised learning techniques, temperature scaling is a widely used entropy control strategy. This paper proposes deterministic knowledge transfer as an entropy control strategy. Specifically, this paper abandons the traditional temperature constant T (or sets it to 1) and instead uses a threshold parameter τ to filter deterministic knowledge: only when , this part is regarded as deterministic knowledge and participates in the gradient calculation; the rest is ignored.
[0024] Step 4: Complete the model training process using an efficient multi-decoding strategy.
[0025] K perturbed results are stacked up in the Batch dimension at one time, thereby achieving one-time forward propagation and ensuring the independence of perturbation results with different features.
Claims
1. A semi-supervised medical image segmentation method based on knowledge distillation, characterized in that: A self-training framework based on knowledge distillation and feature perturbation is proposed, which focuses on knowledge transfer for unlabeled data, introduces deterministic knowledge transfer and teacher assistants to ensure the correctness of knowledge transfer and reduce the ability gap, and achieves high-performance computing through a multi-effect decoder. Specifically, it includes the following steps: 1) Design a Self-Training framework to add different degrees of feature perturbations between the encoder and decoder; 2) Applying a teacher assistant between the decoders in 1) for training its own model parameters; 3) Introducing a deterministic knowledge transfer mechanism during model training for semi-supervised learning; 4) Use efficient multi-decoding strategy to complete the model training process.
2. The semi-supervised medical image segmentation method based on knowledge distillation according to claim 1, characterized in that: By applying multi-feature perturbations, a series of self-training processes are generated without relying on EMA.
3. The semi-supervised medical image segmentation method based on knowledge distillation according to claim 1, characterized in that: The performance gap between the teacher model and the student model can be narrowed with the help of multiple teacher assistant networks, where decoders with different perturbation levels are regarded as teacher models and teaching assistant models.
4. The semi-supervised medical image segmentation method based on knowledge distillation according to claim 1, characterized in that: Deterministic knowledge transfer is proposed as a control strategy, and threshold parameters are used to screen the deterministic knowledge.
5. The semi-supervised medical image segmentation method based on knowledge distillation according to claim 1, characterized in that: An equivalent multi-decoder implementation is introduced, that is, multiple perturbation results are stacked in the Batch dimension at one time, so as to achieve one-time forward propagation and ensure the independence between different feature perturbation results.