Data distribution optimization method based on data fusion and knowledge distillation

Abstract

The invention discloses a data distribution optimization method based on data fusion and knowledge distillation. Comprising the following steps: designing a semi-supervised learning training framework containing three neural networks; performing shallow feature fusion on the input image by using image-level bidirectional copying and pasting; frequency copying and pasting are used in the frequency domain, and the data information of the frequency domain is integrated by exchanging the amplitude spectrum of the image; a knowledge distillation design loss function is introduced for model training. According to the method, data distribution is effectively reduced, the precision of a semi-supervised medical segmentation task is greatly improved, and experimental results on a plurality of public data sets show that compared with a previous best result, the performance is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of semi-supervised medical image segmentation, and in particular to a data distribution optimization method based on data fusion and knowledge distillation. Background Art

[0002] In recent years, deep neural networks have proven that they can be effectively trained using sufficient labeled data. However, creating sufficient data requires a lot of manual annotation, especially in medical image processing, where medical image annotation requires expertise and specialized equipment, which results in a shortage of fully labeled datasets. Fortunately, semi-supervised learning (SSL) is an effective way to solve this problem. Semi-supervised learning technology makes full use of a large amount of unlabeled data, thereby reducing the reliance on fully labeled data, which has a significant impact on the accuracy of clinical imaging diagnosis and treatment planning.

[0003] There has been a lot of research activity in the field of semi-supervised segmentation of medical images. Many recent studies have focused on consistency regularization, which enables the model to learn more robust and stable feature representations by requiring it to produce consistent outputs under different transformations of the input data. For example, FixMatch proposed a concise consistency regularization framework that uses a weak enhancement branch to supervise a strong enhancement branch. This strategy has subsequently been widely used.

[0004] In addition, knowledge distillation (KD) has been widely proven to effectively supplement semi-supervised learning and improve the performance of semi-supervised models. Knowledge distillation uses the output of the teacher model to train the student model, thereby generalizing the model and transferring knowledge. Knowledge distillation has great potential in the semi-supervised field. It can promote deeper feature learning and enhance the stability of model behavior by strengthening predictions. This feature of knowledge distillation enhances its effectiveness in improving model robustness and accuracy, especially when dealing with highly variable and diverse medical images.

[0005] Semi-supervised models usually focus on the empirical match between labeled and unlabeled data. When the empirical match between labeled and unlabeled data is poor, existing methods often have difficulty in effectively utilizing the knowledge in the labeled data. Some methods use distribution alignment, which aligns the predicted marginal distribution of unlabeled data with the marginal distribution of true labels, thereby forcing the predicted aggregation of unlabeled data to match the distribution of the provided labeled data. Other methods incorporate labeled and unlabeled data into the network by mixing labeled and unlabeled data (such as Cutmix and Bidirectional Copy Paste (BCP)) to minimize the empirical distribution gap and keep the distribution of labeled and unlabeled data consistent.

[0006] Inspired by previous studies, the present invention aims to address the limitations of existing SSL methods in medical image segmentation. Specifically, the present invention attempts to develop a robust method to enhance the model's ability to learn from labeled data and improve its generalization ability in the presence of an imperfect empirical match between labeled and unlabeled data. The present invention utilizes frequency copy-paste (FCP) at the frequency domain level and bidirectional copy-paste (IBCP) at the image level to effectively bridge the distribution gap between labeled and unlabeled data. In addition, the present invention also synergistically combines knowledge distillation and semi-supervised learning to promote deeper feature learning and enhance the stability of the model. Summary of the invention

[0007] The present invention proposes a data distribution optimization method based on data fusion and knowledge distillation. The method proposes to use a method based on frequency copy and paste and image-level bidirectional copy and paste to reduce the distribution gap between labeled data and unlabeled data, introduce knowledge distillation into semi-supervised learning, improve feature learning depth and model stability, and help improve the accuracy of medical image segmentation. The technical solution adopted by the method includes the following steps:

[0008] Step 1: Design a semi-supervised learning training framework containing three neural networks;

[0009] Step 2: Use image-level bidirectional copy and paste to perform shallow feature fusion on the input image;

[0010] Step 3: Use frequency copy and paste in the frequency domain to integrate the data information in the frequency domain by exchanging the amplitude spectrum of the image;

[0011] Step 4: Introduce knowledge distillation to design loss function for model training.

[0012] The data distribution optimization method based on data fusion and knowledge distillation proposed in this invention has the following advantages over the traditional semi-supervised medical image segmentation method:

[0013] (1) The present invention proposes to use Fourier transform and inverse Fourier transform to transition data enhancement from the traditional time domain to the frequency domain through a frequency copy and paste method, thereby narrowing the distribution gap between labeled data and unlabeled data.

[0014] (2) The present invention introduces consistency regularization in image-level bidirectional copy-paste to reduce the empirical matching between labeled data and unlabeled data, and uses integral distance to quantify their distribution gap. In addition, the present invention combines semi-supervised learning with knowledge distillation to improve the model's ability to extract data features. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of the data distribution optimization method based on data fusion and knowledge distillation proposed in the present invention. DETAILED DESCRIPTION

[0016] The method of the present invention is described in detail with reference to the accompanying drawings and embodiments.

[0017] The schematic diagram of the data distribution optimization method based on data fusion and knowledge distillation is as follows: Figure 1 The overall process of this method is as follows: first, the teacher network generates pseudo annotations for unlabeled data, the student network integrates unlabeled data and labeled data, and the assistant network integrates data enhanced by frequency copy and paste. It is best to calculate the loss function of pseudo annotation data and predicted data with labeled data respectively, and update the network parameters.

[0018] Step 1: Design a semi-supervised learning training framework containing three neural networks;

[0019] The teacher network generates pseudo labels for unlabeled data, the student network integrates the fusion data of labeled data and unlabeled data, and the assistant network integrates the data enhanced by frequency copy and paste. Parameters are transferred between the student and teacher networks through exponential moving average (EMA), the network parameters are shared between the assistant network and the student network, and the teacher network outputs pseudo annotations.

[0020] Step 2: Use image-level bidirectional copy and paste to perform shallow feature fusion on the input image;

[0021] Bidirectional copy and paste is used for unlabeled data and labeled data to align their distributions. The calculation process is shown in formula (1):

[0022]

[0023] Similarly, image-level bidirectional copy and paste is also applied to pseudo-annotation labels and real labels. The calculation process is shown in formula (2):

[0024]

[0025] Step 3: Use frequency copy and paste in the frequency domain to integrate the data information in the frequency domain by exchanging the amplitude spectrum of the image;

[0026] Apply multidimensional Fourier transform to the input images to obtain their frequency domain representation, and then exchange their amplitude spectrum information in a certain area. The calculation process is shown in formula (3):

[0027]

[0028] Step 4: Introduce knowledge distillation to design loss function for model training.

[0029] Combine Dice with knowledge distillation and use the following loss function for the newly added distillation branch:

[0030] L Dice = Dice(σ(O T / T),σ(O s / T)) (4).

Claims

1. A data distribution optimization method based on data fusion and knowledge distillation, characterized in that: A method based on frequency copy and paste and image-level bidirectional copy and paste is proposed to reduce the distribution gap between labeled data and unlabeled data, introduce knowledge distillation into semi-supervised learning, improve feature learning depth and model stability, and help improve the accuracy of medical image segmentation. The specific steps include: 1) Design a semi-supervised learning training framework containing three neural networks; 2) Use image-level bidirectional copy and paste to perform shallow feature fusion on the input image; 3) Use frequency copy and paste in the frequency domain to integrate the data information in the frequency domain by exchanging the amplitude spectrum of the image; 4) Introduce knowledge distillation to design loss function for model training.

2. The data distribution optimization method based on data fusion and knowledge distillation according to claim 1, characterized in that: The teacher network generates pseudo labels for unlabeled data, the student network integrates the fusion data of labeled and unlabeled data, and the assistant network integrates the data enhanced by frequency copy-paste.

3. The data distribution optimization method based on data fusion and knowledge distillation according to claim 1, characterized in that: The unlabeled data uses the pseudo labels output by the teacher network, while the labeled data retains its original labels, and a bidirectional copy-paste operation is applied to both images and their corresponding labels.

4. The data distribution optimization method based on data fusion and knowledge distillation according to claim 1, characterized in that: Multidimensional Fourier transform is applied to the input images to obtain their frequency domain representation, and then the amplitude spectrum information of a certain region of them is exchanged.

5. The data distribution optimization method based on data fusion and knowledge distillation according to claim 1, characterized in that: Combine Dice with knowledge distillation and use the following loss function for the newly added distillation branch: L Dice =Dice(σ(O T / T),σ(O s / T)) (1).

Images