Unsupervised domain adaptation medical image segmentation method and device based on bezier mix data mixing

Abstract

The application discloses a medical image segmentation method and device based on BezierMix data mixing and unsupervised domain adaptation, and relates to the field of medical image segmentation. The method comprises the following steps: S1, obtaining target domain samples and source domain samples in medical image data and mixing the target domain samples and the source domain samples to generate mixed samples; S2, constructing an unsupervised domain adaptation segmentation model comprising a teacher model, a student model and a discriminator; S3, performing first-stage training on a self-training framework formed by the teacher model and the student model; performing second-stage adversarial training on the student model by using the discriminator; and S4, performing medical image segmentation by using the trained model. The mixed samples are generated by using a cross-domain data mixing method BezierMix, so that the model generalization capability is improved; the style of the source domain is migrated to the target domain, and the domain difference between the source domain and the target domain in the image level and the output space is reduced by combining adversarial learning and a self-training framework.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image segmentation, and in particular to an unsupervised domain adaptation medical image segmentation method and device based on BezierMix data mixing. BACKGROUND

[0002] In the field of medical imaging, different imaging modalities (such as MRI, CT and X-ray) often provide different image information, which is of great significance for clinical diagnosis. With the development of deep learning technology, deep neural networks have achieved outstanding performance in medical image segmentation. However, when a well-trained model is applied to an unseen modality image, its performance will often decrease significantly. However, retraining a new modality medical image segmentation model is very expensive, because the pixel-level annotation of structural segmentation medical images is both expensive and time-consuming, and is usually completed by experienced radiologists. Unsupervised domain adaptation (UDA) is an effective solution that learns the correlation and difference between labeled source domain data and unlabeled target domain data to accurately predict unlabeled target domain data.

[0003] In recent years, self-training methods have shown significant effectiveness in unsupervised domain adaptation. It usually mixes the information of the target domain and the source domain together to construct cross-domain augmented samples for consistency training. Cross-domain data mixing strategies usually have Class-based methods and Patch-based methods. These methods have achieved outstanding results in unsupervised domain adaptation of natural images. In the experiment, these two strategies were tested on medical images. The AdvEnt model was used to perform unsupervised domain adaptation training on the abdominal dataset, with 8K iterations. Then various cross-domain augmented methods were used to generate cross-domain mixed samples on the test set of the dataset, and then these mixed samples were input into the model to obtain the entropy map and the prediction map. For medical images, it was found in the experiment that these cross-domain data mixing strategies have the following problems:

[0004] (1) The Class-based method is to cut the source domain half the number of class organ to the target domain to construct enhanced samples. Due to the structural characteristics of organs in medical images, that is, an organ of a certain class almost only exists in a certain fixed area of an image (for example, the liver organ is usually in the upper left corner of the image). This spatial distribution property causes the corresponding class organ area in the target domain to be easily covered by the corresponding class in the source domain when constructing cross-domain mixed samples, and there are distribution differences between different modalities, which causes the model to usually only identify the pixels in the entire class copied from the source domain, ignoring the corresponding class pixels in the target domain, thereby causing the model to not fully learn the target domain features. At the same time, due to the limited organ classes of medical images, and the small morphological difference of the same class, which makes the Class-based method easily duplicate a certain class of organ, causing the model to overfit the distribution of part of the organ, thereby causing the performance of the model to degrade.

[0005] (2) The Patch-based method is to cut through a random area cutting strategy, which can contain more background class information, and at the same time allows the model to contain more scene or structural features, so that the model can better adapt to the difference in domain distribution while keeping the content consistent. However, the Patch-based method is to cut in the form of Patch blocks, which is very single in shape, and the distribution of the generated enhanced samples changes relatively "rigidly", which is significantly different from the flexible boundary in the actual scene (such as the organ outline in medical images, the object edge in natural scenes), and does not conform to the actual distribution change, thereby causing the stability of the mixed area to decrease, so that the model does not focus on learning in the mixed area, and therefore may not be conducive to the iterative learning of the model. SUMMARY

[0006] To solve the above problems, the present application provides an unsupervised domain adaptation medical image segmentation method and device based on BezierMix data mixing. BezierMix data mixing uses the characteristics of Bezier curves to construct more diverse and complex mixed enhanced samples, construct more diverse and complex cross-domain mixed enhanced samples, thereby adding stronger perturbations to enable the trained model to have better generalization ability. In addition, BezierMix uses a parametric curve to fit the flexible boundary, which is more consistent with the morphological outline of the organ in the medical image than the rigid boundary of the rectangle, thereby improving the stability of the mixed area of the image. The style of the source domain is migrated to the target domain, which reduces the domain difference between the mixed source domain and the target domain in the image level, and also increases the diversity of the source domain samples. The entropy adversarial learning and self-training framework are combined to reduce the domain difference between the two fields in the output space.

[0007] On the one hand, the unsupervised domain adaptation medical image segmentation method based on BezierMix data mixing has the following specific steps:

[0008] S1, a data processing step, obtaining target domain samples and source domain samples in medical image data, and mixing the target domain samples and the source domain samples to generate mixed samples;

[0009] S2, an unsupervised domain adaptation segmentation model construction step, constructing an unsupervised domain adaptation segmentation model comprising a teacher model, a student model and a discriminator;

[0010] The teacher model takes the target domain samples as input to generate pseudo labels;

[0011] The student model takes the source domain samples, the target domain samples or the mixed samples as input to output source domain predictions, target domain predictions or mixed predictions, respectively;

[0012] The total loss function of the unsupervised domain adaptation segmentation model comprises a supervision loss and a mixed loss, and the total loss function is a weighted sum of the supervision loss and the mixed loss; the supervision loss is constructed using the source domain predictions and the true labels; the mixed loss is constructed using the mixed labels generated by the true labels and the pseudo labels and the mixed predictions;

[0013] S3, an unsupervised domain adaptation segmentation model training step,

[0014] The source domain samples and the mixed samples are used as the input of the student model, and the target domain samples are used as the input of the teacher model, and the student model is trained using the total loss function to obtain a self-trained student model;

[0015] The source domain samples and the target domain samples are input into the self-trained student model, and the target domain predictions and the source domain predictions obtained respectively are input into the discriminator for adversarial training to obtain a trained student model;

[0016] S4, a medical image segmentation step, using the trained student model to perform medical image segmentation.

[0017] Preferably, in the process of training the student model using the total loss function, the teacher model is iteratively updated in weight based on the weight of the student model by exponential moving average (EMA).

[0018] Preferably, the target domain samples and the source domain samples are mixed by BezierMix data to generate the mixed samples, and the specific implementation steps are as follows:

[0019] A Bezier mask (BezierMask) is generated using a Bezier curve;

[0020] The mixed samples are generated by the BezierMask; represented as:

[0021] ;

[0022] wherein, represents a mixed sample; represents element multiplication; represents a BezierMask; represents a source domain sample; represents a target domain sample.

[0023] Preferably, based on the BezierMask, a mixed label is generated using a real label and a pseudo label, represented as:

[0024] ;

[0025] wherein, represents a mixed label; represents element multiplication; represents a BezierMask; represents a source domain sample; represents a target domain sample, represents a real label of the source domain, represents a pseudo label of the target domain obtained by a teacher model.

[0026] Preferably, after obtaining the target domain sample and the source domain sample in the medical image data, a style transfer operation is used to style the source domain sample to the target domain sample.

[0027] Preferably, the supervision loss includes a cross-entropy loss and a Dice loss, represented as:

[0028] ;

[0029] wherein, represents a supervision loss; represents a cross-entropy loss; represents a Dice loss; represents a predicted value of an i-th source domain sample; represents a real label of an i-th source domain sample; represents a number of source domain samples.

[0030] Preferably, the total loss model further includes an entropy loss, and the total loss function is a weighted sum of the supervision loss, the mixed loss and the entropy loss; the entropy loss is constructed based on the mixed prediction, represented as:

[0031] ;

[0032] wherein, represents an entropy loss; represents a mixed prediction probability of an i-th pixel of a mixed sample.

[0033] ​​​Preferably, characterized in that the mixed loss is represented as:

[0034] ;

[0035] Wherein, The mixed loss is represented; The number of mixed samples is represented; The mixed prediction of the th mixed sample is represented; The mixed label of the th mixed sample is represented.

[0036] Preferably, in the adversarial training process, the optimization objective of the discriminator is represented as:

[0037] ;

[0038] Wherein, The optimization objective of the discriminator is represented; The th target domain sample is represented, The student model is represented; The entropy map is represented; The discriminator is represented; The number of target domain samples is represented; The number of source domain samples is represented; The th source domain sample is represented.

[0039] On the other hand, the unsupervised domain adaptation medical image segmentation device based on BezierMix data mixing includes the following:

[0040] The data processing module is used to obtain the target domain samples and the source domain samples in the medical image data, and to mix the target domain samples and the source domain samples to generate mixed samples;

[0041] The unsupervised domain adaptation segmentation model construction module is used to construct an unsupervised domain adaptation segmentation model including a teacher model, a student model and a discriminator;

[0042] The teacher model takes the target domain sample as input to generate a pseudo label;

[0043] The student model takes the source domain sample, the target domain sample or the mixed sample as input to output the source domain prediction, the target domain prediction and the mixed prediction respectively;

[0044] The total loss function of the unsupervised domain adaptation segmentation model includes a supervision loss and a mixed loss, and the total loss function is a weighted sum of the supervision loss and the mixed loss; the supervision loss is constructed using the source domain prediction and the real label; the mixed loss is constructed using the mixed label generated by the real label and the pseudo label and the mixed prediction.

[0045] The unsupervised domain adaptation segmentation model training module is used for taking the source domain sample and the mixed sample as the input of the student model, taking the target domain sample as the input of the teacher model, training the student model by using a total loss function, and obtaining the student model after self-training.

[0046] The medical image segmentation module is used for performing medical image segmentation by using the trained student model.

[0047] Compared with the prior art, the present application has the following beneficial effects:

[0048] (1) The present application proposes a new cross-domain data mixing method BezierMix based on Bezier curve, which can construct more diverse and complex mixed augmented samples, thereby introducing more complex disturbance information, so that the unsupervised domain adaptation segmentation model has better generalization ability; in addition, BezierMix uses a parameterized curve to fit a flexible boundary, which is more consistent with the morphological profile of organs in medical images than a rigid boundary of a rectangle, thereby improving the stability of the periphery of the mixed image area.

[0049] (2) The unsupervised domain adaptation segmentation model of the present application combines style transfer and adversarial learning into the self-training framework, further reduces the domain difference from the image level and the output level; the style of the source domain is transferred to the target domain, which reduces the domain difference between the two domains at the image level and increases the diversity of the source domain samples; meanwhile, the output level is considered, and entropy adversarial learning is combined with the self-training framework to reduce the domain difference between the two domains in the output space. BRIEF DESCRIPTION OF DRAWINGS

[0050] The present application will be further described in detail below with reference to the accompanying drawings;

[0051] Figure 1 The flowchart of the unsupervised domain adaptation medical image segmentation method based on BezierMix data mixing of the embodiment of the present application;

[0052] Figure 2 The visualization comparison chart of various cross-domain mixed augmented methods of the unsupervised domain adaptation medical image segmentation method based on BezierMix data mixing of the embodiment of the present application;

[0053] Figure 3 The model structure diagram of the unsupervised domain adaptation medical image segmentation method based on BezierMix data mixing of the embodiment of the present application;

[0054] Figure 4BezierMask acquisition process schematic diagram of the BezierMix data mixing based unsupervised domain adaptation medical image segmentation method of the embodiment of the present application;

[0055] Figure 5 Control point quantity change influence on curve shape schematic diagram of the BezierMix data mixing based unsupervised domain adaptation medical image segmentation method of the embodiment of the present application;

[0056] Figure 6 Different method segmentation result visualization comparison diagram of the BezierMix data mixing based unsupervised domain adaptation medical image segmentation method of the embodiment of the present application;

[0057] Figure 7 Different data mixing method in training process Dice coefficient change curve diagram of the BezierMix data mixing based unsupervised domain adaptation medical image segmentation method of the embodiment of the present application;

[0058] Figure 8 Different teacher model update strategy self-training loss curve diagram of the BezierMix data mixing based unsupervised domain adaptation medical image segmentation method of the embodiment of the present application;

[0059] Figure 9 Structure block diagram of the BezierMix data mixing based unsupervised domain adaptation medical image segmentation device of the embodiment of the present application. DETAILED DESCRIPTION

[0060] The present application is further described below through specific embodiments.

[0061] As shown in the following, Figure 1 The BezierMix data mixing based unsupervised domain adaptation medical image segmentation method includes the following specific steps:

[0062] The existing cross-domain data mixing strategies include Class-based method and Patch-based method. The Class-based method is to cut the source domain half number of class organs to the target domain to construct enhanced samples. Due to the structural characteristics of organs in medical images, that is, an organ of a certain class almost only exists in a certain fixed area of an image (for example, the liver organ is usually in the upper left corner of the image). This spatial distribution property causes the corresponding class organ area in the target domain to be easily covered by the corresponding class in the source domain when constructing cross-domain mixed samples, and there are distribution differences between different modalities, which causes the model to usually only identify the pixels in the entire class copied from the source domain, while ignoring the corresponding class pixels in the target domain (for example, the liver in the CT image is usually in the upper left corner of the image, while the liver in the MRI image is usually in the lower right corner of the image). Figure 2ClassMix (as shown in the row of ClassMix in Table 1), so that the model cannot fully learn the characteristics of the target domain. At the same time, due to the limited organ categories of medical images and the small morphological differences of the same category, the Class-based method is prone to repeat and copy some organs, which leads to overfitting of the distribution of some organs by the model, thereby causing the performance of the model to degrade. The Patch-based method is through a random region cropping strategy, which can contain more background class information, and at the same time allows the model to contain more scene or structural features, so that the model can better adapt to the differences in domain distribution while keeping the content consistent. However, the Patch-based method is cropped in the form of a Patch block, which is very single in shape, and the distribution of the generated augmented samples changes relatively rigidly, which is significantly different from the flexible boundary in the actual scene (such as the organ outline in medical images and the object edge in natural scenes), and does not conform to the actual distribution change, thereby causing the stability of the mixed area to decrease (such as Figure 2 as shown in the row of CutMix in Table 1), so that the model does not focus on learning the mixed area, and therefore may not be conducive to the iterative learning of the model. To solve the above problems, an unsupervised domain adaptation medical image segmentation method based on BezierMix data mixing is proposed.

[0063] S1, a data processing step, obtains target domain samples and source domain samples in medical image data, and mixes the target domain samples and the source domain samples to generate mixed samples.

[0064] First, a labeled data set and an unlabeled data set are defined, and the task of domain adaptation is to transfer the knowledge learned by to . Figure 3 The overall framework diagram of the unsupervised domain adaptation medical image segmentation method proposed in the embodiment is outlined. The basic framework is composed of two segmentation models and with the same architecture, and a discriminator . The two segmentation models and The teacher model and the student model are named respectively, and their difference is only in the way of updating the weights. Among them, the student model needs to go through two stages of learning. In the first stage, the student model is first trained according to the source domain data, and then the mixed samples and pseudo labels generated by BezierMix are used for self-training while minimizing the prediction entropy of the mixed samples. In the second stage, the student model is trained on the source domain and the target domain to realize the consistency alignment of the source domain distribution and the target domain distribution. The weights of the teacher model are updated according to the exponential moving average (EMA) of the weights of the student model trained in the first stage. As shown in Figure 3 , the framework aims to solve two basic problems in unsupervised domain adaptation: (1) content consistency learning: use the BezierMix method to construct more diverse and complex cross-domain mixed augmented samples, add stronger perturbations, and force the target domain images to be consistent through the prediction of the teacher model and the prediction of the cross-domain mixed augmented samples, so that the trained model has better generalization ability and enhances the content learning ability. (2) Domain distribution learning: first align the domain distribution at the image level, and then combine the entropy adversarial strategy to further align the structural distribution features in the output space.

[0065] BezierMix data mixing method.

[0066] In order to design a more diverse and complex cross-domain augmentation method, the positions and numbers of control points in a full 0 mask are set to control the shape of the Bezier curve using the properties of the Bezier curve. The values of the region surrounded by the curve in the mask are set to 1 to form the BezierMask. Then the source domain and target domain data are mixed through the BezierMask. For the selection of control point positions, first evenly grid the mask with a value of full 0, then randomly select grid from all grids, and for each selected grid, randomly sample a coordinate point inside it to form the control point set . The number of control points in the final set is . In this way, the control points in the set are sampled in different grids, which can disperse the positions of the control points in the mask and avoid the problem of too small clipping shape caused by too concentrated control point positions in the set . For the number of control points, Figure 3 shows the influence of the number of control points on the curve shape, as shown in the leftmost graph in Figure 3 , the number of control points is in the range of 3-5, and the curve shape is relatively simple. Figure 3The number of control points in the left four figures is in the range of 15-20, and the curve shape is relatively complex. Generally, the more control points, the more complex the curve shape. Next, the detailed algorithm flow is introduced.

[0067] The linear interpolation is performed on the line segment formed by the connection, and new intermediate vertices are gradually generated. Then, the recursive interpolation is performed, and the linear interpolation operation with the same parameters is repeated on the line segment formed by the connection of the newly generated vertices of each layer. After layer-by-layer recursion, a vertex is finally obtained, which is the point on the Bezier curve corresponding to the parameter t. Finally, all points on the Bezier curve are calculated by traversing all values from 0 to 1, and the complete Bezier curve is obtained. This process can be defined using Bernstein polynomials, which is represented as:

[0068] ;

[0069] Among them, is the binomial coefficient, is the i-th control point of the Bezier curve, is the order of the Bezier curve, An order of Bezier curve corresponds to control points. Actually, since it is impossible to completely traverse all values in the range of [0, 1], the embodiment selects to use a limited set for drawing. However, if a limited value is used for drawing, irregular areas and obvious jagged edges may be generated. Therefore, after completing the curve drawing, connection and filling, Gaussian filtering needs to be applied to smooth the jagged edges. The specific steps of obtaining the BezierMask are shown in . Figure 4

[0070] Finally, the cross-domain mixed enhanced sample and the corresponding cross-domain mixed pseudo label can be represented as:

[0071] ;

[0072] ;

[0073] Among them, represents element-wise multiplication, and respectively refer to the source domain sample and the target domain sample, represents the real label of the source domain, ​​This represents the pseudo-labels obtained from the teacher model for the target domain.

[0074] Style transfer.

[0075] Before model training, histogram matching was used to transfer the style from the source domain to the target domain. This method reduces the domain differences between the two domains at the image level and also increases the diversity of the source domain samples. A histogram is the frequency of occurrence of each pixel value in an image. For example, for a grayscale image with a grayscale range of 0-255, the histogram represents the number of pixels at each grayscale level. Histogram matching is an image processing technique used to adjust the histogram of one image to match the histogram distribution of another target image. It is a fundamental tool in traditional image processing. Its core idea is to use a pixel value mapping function to match the statistical distribution of the source image with that of the target image, thereby achieving consistency enhancement or cross-domain data alignment at the image level. In the experiments, other style transfer methods were also explored, such as CycleGAN, FDA, Style Randomization, and Histogram Matching. Specific experiments are detailed in the subsequent "Experiments and Analysis".

[0076] S2, the steps for constructing an unsupervised domain adaptive segmentation model.

[0077] An unsupervised domain-adaptive segmentation model is constructed, comprising a teacher model, a student model, and a discriminator. The teacher model takes target domain samples as input and generates pseudo-labels; the student model takes source domain samples, target domain samples, or mixed samples as input and outputs source domain predictions, target domain predictions, or mixed predictions, respectively. The discriminator takes target domain predictions and source domain predictions as input.

[0078] S3, Unsupervised Domain Adaptive Segmentation Model Training Steps.

[0079] Self-trained adversarial strategy.

[0080] This embodiment combines a self-training framework with entropy adversarial learning to effectively mitigate the domain shift problem. The model training is divided into two phases: a self-training phase and an adversarial discrimination phase. The self-training adversarial strategy will be described in detail below.

[0081] In the first stage, the student model mainly undertakes three tasks: (1) ensuring its basic structural segmentation ability by learning from the source domain data; (2) enhancing its content learning and domain adaptation capabilities by maintaining perturbation invariance through consistent prediction of mixed samples; and (3) reducing the uncertainty of the model's prediction of mixed samples by minimizing entropy. Therefore, the total loss in this stage can be expressed as:

[0082] ;

[0083] in, This is a supervised loss based on labeled source images. This embodiment uses cross-entropy loss and Dice loss for joint training. Cross-entropy loss focuses the model on pixel-level segmentation accuracy, while Dice loss addresses class imbalance by optimizing overlapping regions. The formula for the supervised loss is as follows:

[0084] ;

[0085] in, Indicates the first Predicted values ​​for each source domain sample. Indicates the first The true labels of the source domain samples.

[0086] For cross-domain hybrid augmentation samples, this embodiment forces the target domain image to be consistent with the predictions of the teacher model and the cross-domain hybrid augmentation samples. The loss can be expressed as:

[0087] ;

[0088] in, Indicates the first Prediction of a mixed sample, Indicates the first Pseudo-labels for a mixed sample.

[0089] Prediction entropy typically represents the uncertainty of prediction results. In this embodiment, entropy minimization directly reduces the uncertainty of the model's prediction of mixed samples, constrains the structural rationality of mixed sample prediction, and thus helps the model better understand the edge structures and features in mixed samples.

[0090] ;

[0091] in, It is a mixed sample. The predicted probability for each pixel is calculated. Considering the low predictive power of the teacher model in the early stages of domain adaptation, a weight coefficient adjustment mechanism is designed. and The weighting coefficient It employs the exponential ramp-up technique. The weights increase with each training step, eventually reaching a maximum of 1. After the student model completes the first phase of learning, the weights of the teacher model are updated using an exponential moving average (EMA). In the next iteration, the updated formula is as follows:

[0092] ;

[0093] wherein, is a smoothing coefficient parameter, usually set between 0.99 to 0.999. is the weight of the first stage student model learning.

[0094] The source domain samples and the target domain samples are input into the self-trained student model, and the target domain prediction and the source domain prediction obtained respectively are input into the discriminator for adversarial training to obtain the trained student model.

[0095] After updating the weight of the teacher model, the student model also needs the second stage of learning. In the first stage of learning, the student model only encounters mixed samples, and does not contact complete and real target domain samples. Therefore, the student model is adversarially trained on the source domain and the target domain in this embodiment to ensure that it realizes the consistency alignment of the source domain distribution and the target domain distribution while contacting complete target domain data. Given a discriminator , the discriminator learns to distinguish the entropy distribution of the source domain and the target domain through adversarial training. The prediction entropy map of the source domain usually only has high entropy values at the edges of the category object, and generally presents low entropy values, while the entropy map of the target domain presents high entropy values due to noise and distribution difference. The discriminator forces the generator (segmentation network) to adjust the prediction result of the target domain so that its entropy distribution is close to the source domain, thereby reducing the domain difference. At the same time, the low-dimensional space of the entropy map contains rich semantic structure information, and the discriminator directly acts on the entropy map for adversarial training, retaining the structure features shared between the domains.

[0096] The adversarial loss of the second stage can be expressed as:

[0097] ;

[0098] wherein, denotes the th target domain sample, is the student model, and the output of the generator (student model) is input into the discriminator through to obtain the entropy prediction . The generator tricks the discriminator through this adversarial loss. At the same time, the discriminator also needs to be trained so that it can accurately distinguish the source domain and target domain entropy prediction. Therefore, the optimization objective of the discriminator can be expressed as:

[0099] ;

[0100] Through the alternating optimization training of the generator and the discriminator, the distribution consistency alignment of the source domain and the target domain in the output space is finally realized.

[0101] S4, a medical image segmentation step, using the trained student model to perform medical image segmentation.

[0102] using the trained student model to perform medical image segmentation.

[0103] Experiments and analysis.

[0104] To fully verify the effectiveness of the proposed unsupervised domain adaptation medical image segmentation framework, a large number of comparative experiments and ablation experiments are conducted in this paper. First, the basic experimental details are introduced. Second, comparison is made with the current most advanced unsupervised domain adaptation method under the same experimental setting. Finally, ablation experiments are conducted on four aspects, namely BezierMix, style transfer, teacher model update strategy and different modules, to prove the effectiveness of the proposed method.

[0105] Dataset and evaluation index.

[0106] This embodiment verifies the proposed method on two different datasets, namely the Multi-Modality Whole Heart Segmentation Challenge 2017 (MMWHS) and the Abdominal Multi-Organ Dataset.

[0107] This embodiment uses two evaluation indexes in total, namely the Dice similarity coefficient (Dice) and the Average Symmetric Surface Distance (ASD).

[0108] Experimental details setting.

[0109] To facilitate comparison with other existing methods, this embodiment uses a segmentation model based on the DeepLabV2 architecture, with ResNet-101 as the backbone network, i.e., the feature extractor module. The discriminator configuration is the same as DCGAN, consisting of four convolutional layers and one classifier layer, with each convolutional layer connected to a Leaky-ReLU activation layer. This embodiment implements the proposed method using the PyTorch library on an NVIDIA Tesla A30. First, the student model is pre-trained for 8000 epochs using an entropy adversarial approach, and then the pre-trained weights are used as the initial weights for both the teacher and student models. In the BezierMix data mixing method, the number of control points on the Bezier curve is set to a range of 3-20. The batch size is 4. The number of iterations is 30000. The optimizer used is the SGD optimizer, with an initial learning rate of 0.001, momentum of 0.9, and weight decay of 1 × 10⁻⁶. −5 .

[0110] Comparative experiments with other domain adaptation methods.

[0111] This embodiment compares the method with seven state-of-the-art methods, including two unsupervised domain adaptation methods for natural images: AdvEnt and DACS, and five unsupervised domain adaptation methods for the medical field: SIFA, SSC, SE_ASA, MPSCL, and FSUDA. AdvEnt, SIFA, DACS, and MPSCL were replicated in the experiments, while the other results are from their original papers. The method in this embodiment first trains the model using only source domain data without any domain adaptation method, and then tests the model with target domain data, thus obtaining a lower bound for unsupervised learning. In addition, supervised training is performed using target domain data and labels to obtain an upper bound for supervised learning.

[0112] Table 1 and Table 2 show the performance comparison of different unsupervised domain adaptation methods in the heart segmentation task, wherein the symbol * indicates the results reproduced in the experiment. It can be found that, whether in the MRI modality to CT modality or in the CT modality to MRI modality task, the results on the domain adaptation model show that there is a serious domain shift problem between the MRI data and the CT data. At the same time, the method proposed in the embodiment has achieved significant improvement in the two evaluation indexes of Dice and ASD. In the heart CT to MRI task, the average Dice of the model proposed by the method of the embodiment first breaks through 80%, which is 10.25 percentage points higher than the current most advanced FSUDA method, which shows the great advantage of the embodiment. At the same time, for the MRI to CT task, the average Dice of the four heart structures is only 1.77 percentage points lower than the supervised learning upper limit value, and the average ASD only differs by 0.6. This highlights the great potential of the method proposed in the embodiment in various clinical applications.

[0113] Table 1: Performance comparison of different unsupervised domain adaptation methods in the segmentation task of the heart dataset MRI modality to CT modality.

[0114]

[0115] Table 2: Performance comparison of different unsupervised domain adaptation methods in the segmentation task of the heart dataset CT modality to MRI modality.

[0116]

[0117] Table 3 and Table 4 show the performance comparison of different unsupervised domain adaptation methods in the abdominal segmentation task, wherein the symbol * indicates the results reproduced in the experiment. It can be seen that the method proposed in the embodiment also achieves good results on the abdominal dataset: for CT images, the average Dice coefficient is only 1.74 percentage points lower than the supervised upper limit, and the average ASD only differs by 0.06, which are significantly better than the current optimal method. For MRI images, the average Dice coefficient is only 1 percentage point lower than the supervised upper limit, and the average ASD differs by 0.2. However, it is worth noting that on the MRI image, the segmentation results of the left kidney and the spleen two categories of the method proposed in the embodiment are slightly lower than FSUDA, which may be due to the fact that FSUDA adopts a U-Net architecture that can better capture image details, which is more advantageous for organ segmentation with fewer categories.

[0118] Table 3: Performance comparison of different unsupervised domain adaptation methods in the segmentation task of the abdominal dataset MRI modality to CT modality.

[0119]

[0120] Table 4: Performance comparison of different unsupervised domain adaptation methods in the segmentation task of abdominal dataset CT modality to MRI modality.

[0121]

[0122] Figure 5 The different domain adaptation medical image segmentation methods are shown to visualize the segmentation results of cardiac MRI images (first row), cardiac CT images (second row), abdominal multi-organ MRI images (third row) and abdominal multi-organ CT images (fourth row). The cardiac structures of the ascending aorta (AA), left atrial blood cavity (LAC), left ventricular blood cavity (LVC) and left ventricular myocardium (MYO) are represented in blue, green, yellow and red, respectively. The liver, right kidney, left kidney and spleen are represented in red, green, yellow and blue, respectively. It can be seen that due to the domain shift problem, the no-domain adaptation method is difficult to accurately segment the cardiac structure. Compared with other domain adaptation methods, the segmentation results obtained by the method proposed in the embodiment are closer to the true labels.

[0123] Ablation experiments of BezierMix.

[0124] The ablation experiments of BezierMix verify the effectiveness of the proposed BezierMix. All experiments of the ablation experiments of BezierMix are performed on the MRI modality to CT modality domain adaptation task in the MMWHS cardiac dataset. In order to compare fairly, all experiments use the same network architecture (DeepLabV2 as the segmentation model and ResNet-101 as the encoder). First, the Dice coefficient curve changes in the training process of BezierMix and the two mainstream cross-domain mixing methods ClassMix and CutMix are compared. As shown in FIG. 6, the Dice coefficient curve of BezierMix is higher than that of ClassMix and CutMix, which shows that the proposed BezierMix has better performance than ClassMix and CutMix. Figure 6As shown, the performance of the Class-based method (ClassMix) starts to decline at 10K iterations and the decline is particularly significant. It can be considered that in the Class-based method, the data distribution of some organs in the target domain is easily covered by the corresponding classes in the source domain, and the model is easy to ignore the corresponding class pixels in the target domain. At the same time, due to the limited organ classes, the model is prone to overfitting the distribution of some organs. In contrast, the Pach-based method retains more background class information and contains different scene features through random region cropping, so it does not decay prematurely compared to ClassMix. However, due to its single rectangular cropping method, CutMix starts to decline in performance after 20K iterations. The BezierMix proposed in this embodiment can construct more diverse and complex cross-domain mixed augmented samples, thereby adding stronger perturbations and allowing the trained model to have better generalization ability, so it does not decay prematurely.

[0125] In addition, the performance of ClassMix, CutMix and BezierMix in the MRI modality to CT modality domain adaptation task in the cardiac data set is also compared, and the detailed results are shown in Table 5. It can be seen that BezierMix achieves the highest Dice score on all cardiac structures, indicating the effectiveness of the BezierMix method.

[0126] Table 5: Performance comparison of different data mixing methods.

[0127]

[0128] Ablation experiment on the complexity of BezierMix. The BezierMix method controls the shape of the cut by setting the number of control points. When the number of control points is small, the shape of the cut is relatively simple. The more the number of control points, the more complex the shape of the cut. Therefore, four intervals are divided in the range of 3-20 control points for experiments. As shown in Table 6, representing the left end point of the control point number interval, representing the right end point of the control point number interval. The experimental results show that when the number of control points is in the 10-15 interval, the domain adaptation performance is best. If the number of control points is outside this interval (whether increased or decreased), the model performance shows a downward trend. Finally, considering that the model needs more diversity and complexity, experiments are conducted in the range of 3-20 control points, and the performance is best.

[0129] Table 6: Ablation experiment on the complexity of BezierMix.

[0130]

[0131] Ablation experiment on style transfer.

[0132] Ablation experiments of style transfer verify the effectiveness of the style transfer proposed in this embodiment. All ablation experiments are performed on the cardiac dataset MRI modality to CT modality domain adaptation task. In order to make the model adapt to the distribution difference between different modalities of medical images faster, the style transfer method is used to transfer the style of the source domain to the target domain, in this way to reduce the domain difference between the two domains at the image level. Table 7 shows the changes of Dice scores before and after using style transfer for the three data mixing methods. It can be observed that after introducing style transfer, the model's adaptation ability to the target domain data is significantly improved.

[0133] Table 7: Effectiveness of style transfer in four data mixing methods.

[0134]

[0135] In addition, as shown in Table 8, ablation studies are also conducted on different style transfer methods. Four different style transfer methods are tried in the experiment, namely CycleGAN, FDA, style randomization (Style Randomization) and histogram matching (Match Histograms). Among them, the implementation details of histogram matching and the implementation details of style randomization are consistent with the literature [Ma H, Lin X, Wu Z, et al. Coarse-to-fine domain adaptive semantic segmentation with photometricalignment and category-center regularization [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 4051-4060.]. The experimental results show that no matter which style transfer method is used, the domain adaptation performance of the model can be effectively improved. The histogram matching method with the best effect is finally selected in this embodiment.

[0136] Table 8: Effectiveness analysis of different style transfer methods in BezierMix.

[0137]

[0138] Ablation experiments of teacher model update strategy.

[0139] The teacher model updates parameters by exponential moving average (EMA), and the training of the model is divided into two stages: self-training stage and adversarial discrimination stage. The ablation experiment of the teacher model update strategy aims to explore the influence of the timing of the teacher model parameter update on the stability and accuracy of the segmentation model. All ablation experiments are performed on the MRI modality to CT modality domain adaptation task in the MMWHS cardiac dataset. Figure 7 The self-training stage loss curve is shown in Table 9. The mean, standard deviation and variance of the self-training loss are further calculated. Baseline represents the model trained only in the self-training stage without the adversarial discrimination stage. Phase1 represents the update strategy of the teacher model in the first stage, and Phase2 represents the update strategy of the teacher model in the second stage. By analyzing the loss curve and the mean, standard deviation and variance of the loss value, it can be found that the first stage update scheme exhibits higher stability in the training process compared to the second stage teacher model update strategy. Specifically, the loss standard deviation and variance of Phase1 are lower, indicating that the training process tends to converge. In addition, the average Dice coefficient of Phase1 strategy (88.42%) is significantly better than that of Phase2 (87.63%), indicating that the segmentation performance is also improved. In addition, as can be seen from Table 9, the self-training loss does not fluctuate significantly after the introduction of adversarial learning using the first stage update scheme, and the standard deviation and variance are only slightly different from those without the use of adversarial learning strategy.

[0140] Table 9: Comparison of different teacher model update strategies and no adversarial learning methods.

[0141]

[0142] Ablation experiment of different modules.

[0143] In order to verify the effectiveness of the key modules of the method proposed in this embodiment, sufficient ablation experiments are performed on the MRI modality to CT modality domain adaptation task in the MMWHS cardiac dataset, and the experimental results are shown in Table 10. First, BezierMix is used as the baseline model, and the average Dice score is 86.01%. After introducing the entropy minimization loss , the performance is improved by 0.80 percentage points compared with the baseline; and the addition of style transfer improves the performance by 1.35 percentage points compared with the baseline. If the entropy minimization and style transfer methods are combined, the model improves the overall performance by 1.80 percentage points compared with the baseline. Finally, after adding the entropy adversarial strategy, the Dice score of the model reaches 88.83%. In summary, each module has independent contribution to the model performance, but their combination seems to be the most effective, which shows that these modules play a complementary role in the self-training framework.

[0144] Table 10: Ablation experiments of different modules.

[0145]

[0146] In summary, the embodiment proposes a new, simple unsupervised domain adaptation medical image segmentation framework based on BezierMix data mixing. The characteristics of Bezier curves are used to construct more diverse and complex mixed augmented samples, add stronger perturbations, and force the target domain image to be consistent through the prediction of the teacher model and the prediction of the cross-domain mixed augmented sample, so that the trained model has better generalization ability and enhances the content learning ability. At the same time, style transfer and adversarial learning are combined into the self-training framework to further narrow the domain difference from the image level and the output level. The experimental results on two cross-modality medical image segmentation datasets (heart and abdomen) show that the proposed method is superior to other unsupervised domain adaptation methods and approaches the results of supervised learning.

[0147] As shown in Figure 9 The application further discloses an unsupervised domain adaptation medical image segmentation device based on BezierMix data mixing, which comprises:

[0148] The data processing module 901 is configured to obtain target domain samples and source domain samples in medical image data, and mix the target domain samples and the source domain samples to generate mixed samples.

[0149] The unsupervised domain adaptation segmentation model construction module 902 is configured to construct an unsupervised domain adaptation segmentation model comprising a teacher model, a student model and a discriminator.

[0150] The teacher model takes the target domain samples as input to generate pseudo labels.

[0151] The student model takes the source domain samples, the target domain samples or the mixed samples as input to respectively output source domain predictions, target domain predictions and mixed predictions.

[0152] The total loss function of the unsupervised domain adaptation segmentation model comprises a supervision loss and a mixed loss, and the total loss function is a weighted sum of the supervision loss and the mixed loss; the supervision loss is constructed by using the source domain predictions and real labels; and the mixed loss is constructed by using mixed labels generated by the real labels and the pseudo labels and the mixed predictions.

[0153] The unsupervised domain adaptation segmentation model training module 903 is configured to take the source domain samples and the mixed samples as input of the student model, take the target domain samples as input of the teacher model, train the student model by using the total loss function, obtain a self-trained student model, input the source domain samples and the target domain samples into the self-trained student model, input the obtained target domain predictions and source domain predictions into the discriminator for adversarial training, and obtain a trained student model.

[0154] The medical image segmentation module 904 is configured to perform medical image segmentation using the trained student model.

[0155] The specific implementation of the unsupervised domain adaptation medical image segmentation device based on BezierMix data mixing is the same as the unsupervised domain adaptation medical image segmentation method based on BezierMix data mixing, and will not be repeated here.

[0156] The above is only a specific embodiment of the present application, but the design concept of the present application is not limited thereto, and any non-essential modification of the present application using this concept shall be deemed to be an infringement of the protection scope of the present application.

Claims

1. An unsupervised domain-adaptive medical image segmentation method based on BezierMix data fusion, characterized in that, Includes the following steps: S1, data processing step: obtain target domain samples and source domain samples from medical image data, and mix the target domain samples and source domain samples to generate mixed samples; S2, Unsupervised domain adaptive segmentation model construction steps: Construct an unsupervised domain adaptive segmentation model including a teacher model, a student model, and a discriminator; The teacher model takes the target domain samples as input and generates pseudo-labels; The student model takes source domain samples, target domain samples, or mixed samples as input and outputs source domain prediction, target domain prediction, or mixed prediction, respectively. The total loss function of the unsupervised domain adaptive segmentation model includes supervised loss and hybrid loss, and the total loss function is a weighted sum of supervised loss and hybrid loss; the supervised loss is constructed using source domain prediction and real labels; the hybrid loss is constructed using hybrid labels generated from real labels and pseudo labels and hybrid predictions; S3, Training steps for an unsupervised domain adaptive segmentation model. Using source domain samples and mixed samples as inputs to the student model and target domain samples as inputs to the teacher model, the student model is trained using the total loss function to obtain a self-trained student model. Input source domain samples and target domain samples into the self-trained student model, and input the obtained target domain predictions and source domain predictions into the discriminator for adversarial training to obtain the trained student model. S4, Medical Image Segmentation Step, uses a trained student model to segment medical images; The target domain samples and source domain samples are mixed using BezierMix data mixing to generate the mixed samples. The specific implementation steps are as follows: Use Bézier curves to generate a Bézier mask (BezierMask); Generate mixed samples using BezierMask; represented as: ； in, Indicates a mixed sample; Indicates element-wise multiplication; This refers to BezierMask; Represents source domain samples; Represents the target domain sample; Based on BezierMask, hybrid tags are generated using real and pseudo tags, represented as follows: ； in, Indicates mixed tags; Indicates element-wise multiplication; This refers to BezierMask; Represents the true label of the source domain. This represents the pseudo-labels obtained from the teacher model for the target domain.

2. The unsupervised domain-adaptive medical image segmentation method based on BezierMix data fusion according to claim 1, characterized in that, During the training of the student model using the total loss function, the teacher model's weights are iteratively updated using the exponential moving average (EMA) based on the student model's weights.

3. The unsupervised domain-adaptive medical image segmentation method based on BezierMix data fusion according to claim 1, characterized in that, After acquiring target domain samples and source domain samples from medical image data, the method also includes using style transfer operations to transfer the style of the source domain samples to the target domain samples.

4. The unsupervised domain-adaptive medical image segmentation method based on BezierMix data fusion according to claim 1, characterized in that, The supervision loss includes cross-entropy loss and Dice loss, expressed as: ； in, Indicates monitoring losses; Represents cross-entropy loss; Indicates Dice loss; Indicates the first Predicted values ​​for each source domain sample; Indicates the first The true labels of the source domain samples; This indicates the number of samples in the source domain.

5. The unsupervised domain-adaptive medical image segmentation method based on BezierMix data fusion according to claim 1, characterized in that, The total loss model also includes entropy loss, and the total loss function is a weighted sum of the supervision loss, the mixed loss, and the entropy loss; the entropy loss is constructed based on the mixed prediction and is expressed as: ； in, Indicates entropy loss; Indicates the mixed sample number The mixed prediction probability of each pixel.

6. The unsupervised domain-adaptive medical image segmentation method based on BezierMix data fusion according to claim 1, characterized in that, The hybrid loss is expressed as: ； in, Indicates mixed loss; Indicates the number of samples in the mixture; Indicates the first Mixed prediction of a mixed sample; Indicates the first A mixed label.

7. The unsupervised domain-adaptive medical image segmentation method based on BezierMix data fusion according to claim 1, characterized in that, During the adversarial training process, the optimization objective of the discriminator is expressed as: ； in, This represents the optimization objective of the discriminator; Indicates the first One target domain sample, Representing the student model; This represents the transformation into an entropy graph; Indicates the discriminator; Indicates the number of samples in the target domain; Indicates the number of samples in the source domain; Indicates the first One source domain sample.

8. An unsupervised domain-adaptive medical image segmentation device based on BezierMix data fusion, characterized in that, Including the following: The data processing module is used to acquire target domain samples and source domain samples from medical image data, and to mix the target domain samples and source domain samples to generate mixed samples. The module for building an unsupervised domain adaptive segmentation model is used to construct an unsupervised domain adaptive segmentation model that includes a teacher model, a student model, and a discriminator. The teacher model takes the target domain samples as input and generates pseudo-labels; The student model takes source domain samples, target domain samples, or mixed samples as input and outputs source domain prediction, target domain prediction, and mixed prediction, respectively. The total loss function of the unsupervised domain adaptive segmentation model includes supervised loss and hybrid loss, and the total loss function is a weighted sum of supervised loss and hybrid loss; the supervised loss is constructed using source domain prediction and real labels; the hybrid loss is constructed using hybrid labels generated from real labels and pseudo labels and hybrid predictions; The unsupervised domain adaptation segmentation model training module is used to train the student model with source domain samples and mixed samples as inputs and target domain samples as inputs to the teacher model, using the total loss function to obtain the self-trained student model; the source domain samples and target domain samples are input into the self-trained student model, and the obtained target domain predictions and source domain predictions are input into the discriminator for adversarial training to obtain the trained student model; The medical image segmentation module is used to segment medical images using a pre-trained student model. The target domain samples and source domain samples are mixed using BezierMix data mixing to generate the mixed samples. The specific implementation steps are as follows: Use Bézier curves to generate a Bézier mask (BezierMask); Generate mixed samples using BezierMask; represented as: ； in, Indicates a mixed sample; Indicates element-wise multiplication; This refers to BezierMask; Represents source domain samples; Represents the target domain sample; Based on BezierMask, hybrid tags are generated using real and pseudo tags, represented as follows: ； in, Indicates mixed tags; Indicates element-wise multiplication; This refers to BezierMask; Represents the true label of the source domain. This represents the pseudo-labels obtained from the teacher model for the target domain.

Images