Medical image unsupervised domain adaptive segmentation method based on semantic perception and orthogonal decomposition
By combining feature alignment and unentangled representation, unsupervised domain adaptation segmentation of medical images is achieved, and the problems of prior art sensitivity to label data dependence and domain transfer are solved, and the performance and robustness of medical image segmentation are improved.
Patent Information
- Application Number
- CN202411236429.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-12-27
AI Technical Summary
Existing medical image segmentation methods rely on a large amount of labeled data and are sensitive to domain transfer, making it difficult to effectively adapt between different domains, resulting in performance degradation.
An unsupervised domain adaptation medical image segmentation method combining feature alignment and detangled representation is proposed. By introducing semantic perception standardization and whitening modules, the center alignment and distributed alignment of input features are realized, and the feature extractor and loss function are redesigned. Decompose into domain-invariant features and domain-specific features to improve the generalization ability of the model.
It significantly improves the performance of unsupervised domain adaptation networks, especially in medical image segmentation tasks, and improves segmentation accuracy and robustness, surpassing the indicator performance of many classic networks.
Smart Images
Figure CN119205808A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an unsupervised domain adaptation medical image segmentation method combining feature alignment and disentanglement representation, and belongs to the technical field of unsupervised domain adaptation segmentation of medical images. Background Art
[0002] In the field of medical image segmentation, deep learning technology has made remarkable progress, especially in the image segmentation task based on supervised learning. However, the limitation of supervised learning methods is their dependence on a large amount of labeled data, which is particularly difficult and expensive in the field of medical image segmentation. In addition, existing models mainly rely on the oversimplified assumption that the training data and test data are independent and identically distributed (iid). When there is a difference between the data distribution used to train the deep learning model and the data distribution encountered during testing, domain shift occurs, resulting in a sharp drop in performance. To overcome the limitations of supervised learning, unsupervised domain adaptation (UDA) has become a crucial solution.
[0003] Unsupervised domain adaptation uses images from the source domain and their labeled data and unlabeled images from the target domain to train the model and achieve the process of predicting the target domain images. In unsupervised domain adaptation medical image segmentation, a variety of methods are used to address the distribution differences between different domains, thereby improving the generalization ability and performance of the model in the target domain.
[0004] Feature alignment is one of the key methods of unsupervised domain adaptation methods, which aims to map or adjust the feature spaces of different domains so that the image features of different domains are closer or overlapped in the shared feature space, thereby achieving cross-domain knowledge transfer. Han et al. proposed a new deep symmetric architecture of UDA for medical image segmentation, introducing a bidirectional alignment scheme to align features simultaneously through a shared encoder and two private decoders. Xian et al. proposed a dual adaptation guided network DAG-Net, which combines Fourier-based contrast style enhancement and residual space alignment in training to collaboratively adjust the segmentation model from the labeled source domain to the unlabeled target domain.
[0005] Disentangled representation methods play an important role in the field of unsupervised domain adaptation. They are used to learn shared representations between source and target domains, thereby achieving cross-domain feature alignment and knowledge transfer. The core idea of disentangled representation methods is to decompose the feature space of data into common parts and specific parts, so that in the shared feature space, data from different domains can be represented as shared factors and domain-specific factors, so that the representation of cross-domain images can be closer, thereby improving the generalization ability of the model. Wang et al. proposed an unsupervised domain adaptation method based on feature disentanglement, which learns disentangled features through two parts: content-related features, which are responsible for segmentation tasks and cross-domain invariance, and style-related features, which clarify the differences between different domains. Pei et al. proposed a UDA framework for cross-modal image segmentation, which decomposes each domain into DIFs and DSFs, uses a self-attention module to enhance the representation of DIFs, and introduces zero loss to enhance the characteristics of DSFs.
[0006] The main contents of the present invention can be summarized as follows: 1) a semantic-aware orthogonal decomposition unsupervised domain adaptation segmentation framework SAODUDA combining feature alignment and disentangled representation is proposed; 2) semantic-aware normalization (SAN) and semantic-aware whitening (SAW) modules are introduced to achieve center alignment and distributed alignment of input features; 3) the feature extractor DIR is redesigned, and the loss function is redesigned to promote center alignment and distributed alignment of input features, and decompose them into domain-invariant features (DIFs) and domain-specific features (DSFs), thereby improving the performance of unsupervised domain adaptation networks. Summary of the invention
[0007] In view of the above-mentioned defects in the prior art, the present invention proposes an unsupervised domain adaptation medical image segmentation method combining feature alignment and disentanglement representation.
[0008] The unsupervised domain adaptation medical image segmentation method combining feature alignment and disentanglement representation of the present invention comprises the following steps:
[0009] S1: Propose an unsupervised domain adaptation segmentation task for medical images, including the following steps:
[0010] S11: Establish a domain adaptation segmentation network SAODUDA based on semantic perception and orthogonal decomposition: ResNet50 is used as the backbone network of SAODUDA, and DeepLab network is used as the segmentation network of SAODUDA;
[0011] S12: Introducing the semantic-aware normalization module: Introducing the SAN (Semantic-Aware Normalization) module in the first and second layers of the ResNet network. The SAN module converts the input features into category-centric features to achieve center alignment of the input features. Then, introducing the SAW (Semantic-Aware Whitening) module after the SAN module, the SAW module converts the center-aligned features of the SAN module into distributed-aligned features to align the edge distribution of joint features from different domains, thereby achieving semantic-aware center alignment and distributed alignment.
[0012] S13: Design feature extractor DIR: Input features are activated by CELU (Continuously Differentiable Exponential Linear Unit) activation function after convolution layer and BN layer, and randomly inactivated by Dropout. The above operation is repeated once, and finally the features are extracted by convolution layer and BN layer.
[0013] S14: Introduce the input feature orthogonal decomposition module: feature F after center alignment and distributed alignment ss The feature extractor DIR with large convolution kernel extracts the domain invariant features F from the input features. ssdi , feature F ss With domain invariant feature F ssdi Input to the difference module Diff to obtain domain-specific features F ssds ; In order to strengthen the domain invariant feature F ssdi With the domain-specific feature F ssds Independence of input features F ss At the same time, it is input into the feature extractor DIR with a small convolution kernel to obtain another set of domain invariant features F ssdi With the domain-specific feature F ssds , calculate the loss optimization model; then the gradient reversal layer GRL and the domain classifier learn the domain invariant feature F ssdi With the domain-specific feature F ssds ;
[0014] S15: Design loss function: introduce a two-step optimization strategy. The first step is to learn the segmentation feature F in the source domain. ss At this time, only LogNLL Loss is involved in training. The second step is to learn the segmentation feature F in the source domain and the target domain. ss , and decompose it into domain invariant features F ssdi With domain-specific features F ssds, at this time, LogNLL Loss, Orthogonal Loss, SAW Loss, and DiceBCE Loss all participate in the training; the loss function redesigned by the present invention is defined as follows:
[0015]
[0016] L DB =α·L Dice +(1-α)·L BCE (4)
[0017]
[0018] L st =L s +L t (7)
[0019] Among them, L Dice For Dice Loss, L BCE is BCE Loss, C is the domain classifier, Generate the mean loss for the first and second SAW modules, L ⊥1 ,L ⊥2 F is generated by the first and second feature extractors DIR respectively ssdi ,F ssds Calculated loss, L s is the loss calculated in the source domain, L t is the loss calculated in the target domain, L st is the total loss in this paper, α, β, χ are parameters, which are 0.9, 0.4, and 0.9 respectively in this invention;
[0020] S2: Establish a data set for model evaluation, including the following steps:
[0021] S21: Selection of evaluation index: DSC (Dice Similarity Coefficient), ASSD (Average Surface Distance), HD (Hausdorff Distance), Sensitivity, Specificity, Jaccard are selected as the evaluation indexes of the present invention, wherein:
[0022]
[0023] ASSD(X,Y)={ASD(X,Y)+ASD(Y,X)} / 2 (10)
[0024]
[0025] Where: TP is the number of pixels properly classified into the foreground area;
[0026] TN represents the number of pixels correctly classified as background area;
[0027] FP represents the number of pixels that are incorrectly classified as foreground areas;
[0028] FN represents the number of pixels that are incorrectly classified as background areas;
[0029] L and P represent the foreground area sets of real data and predicted data, respectively;
[0030] S22: Verification of the effectiveness of the evaluation model: This paper compares the SAODUDA model with seven classic UDA networks: DANN, ADDA, CycleGAN, CyCADA, SynSeg-Net, AdaOutput, and PnP-AdaNet, three advanced UDA networks: ODADA, Shen et al, and MPSCL, and a fully supervised network. SAODUDA has a significant improvement in the DSC indicator, even surpassing the DSC indicator of supervised learning. It is also better than the other 10 networks in the HD indicator. It is slightly lower than the most advanced network in the three indicators of ASSD, Sensitivity, and Specificity, but still better than most classic networks.
[0031] Preferably, in step S11, the input is an image and label of the source domain, and an image of the target domain; the input image passes through the first layer of ResNet, SAN and SAW modules and the second layer of ResNet, SAN and SAW modules to obtain center-aligned and distributed-aligned features; then after two downsamplings, it is input into the feature extractor, the difference module, the gradient reversal layer, and the domain classifier to obtain DIFs (Domain-Invariant Features) and DSFs (Domain-Specific Features), and finally, the input features pass through DeepLab Head to obtain the segmentation result.
[0032] Preferably, in step S12, the intermediate small batch features are input into SAN, and the features are passed through a classifier to generate a segmentation mask to highlight the foreground area while suppressing the background area. In view of the particularity of medical images, the present invention cancels the original strategy of inputting the real segmentation mask optimization model into the SAN module during the training process, and adopts a two-step optimization strategy during the training process to more effectively learn the center alignment features of the segmentation representation. The normalized features in SAN are sent to SAW for further processing to further enhance channel decorrelation and align the global edge distribution with the conditional distribution.
[0033] Preferably, in step S13, since ReLU (Rectified Linear Unit) is not differentiable at zero and sets all negative values to zero, but CELU (Continuously Differentiable Exponential Linear Unit) is a continuously differentiable activation function and smoothes the negative value part, the CELU activation function is used instead of the ReLU activation function to perform gradient optimization more stably and retain more information and gradient signals.
[0034] Preferably, in step S14, a large convolution kernel can capture features in a wider range, and thus can better understand the spatial information in the input image or feature map, but some detail information will be lost; introducing a small convolution kernel can capture local features more finely, have a more sensitive response to detail information, and optimize feature extraction.
[0035] The beneficial effects of the present invention are:
[0036] Based on the unsupervised domain adaptation medical image segmentation method combining feature alignment and disentangled representation, the present invention designs an unsupervised domain adaptation segmentation network SAODUDA based on semantic perception and orthogonal decomposition for the unsupervised domain adaptation segmentation task of medical images. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is the overall block diagram of the SAODUDA network of the present invention;
[0038] Figure 2 It is the overall structure diagram of the SAODUDA network of the present invention;
[0039] Figure 3 is a structural diagram of the SAN module of the present invention;
[0040] Figure 4 is a structural diagram of the SAW module of the present invention;
[0041] Figure 5 It is a structural diagram of the feature extractor DIR of the present invention;
[0042] Figure 6 It is a visualization comparison diagram of the segmentation results of the SAODUDA model of the present invention on the cross-site dataset;
[0043] Figure 7 It is a visualization comparison diagram of the segmentation results of the SAODUDA model of the present invention on a cross-modal dataset; DETAILED DESCRIPTION
[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0045] Embodiment 1:
[0046] like Figures 1 to 7 As shown, the unsupervised domain adaptation medical image segmentation method combining feature alignment and disentanglement representation described in the present invention aims to establish an unsupervised domain adaptation segmentation model for medical images. In view of the excellent performance of feature alignment method and disentanglement representation method in the field of unsupervised domain adaptation, the present invention combines feature alignment method and disentanglement representation method, first introduces SAN and SAW modules to achieve semantically aware center alignment and distributed alignment; then decomposes the aligned features into domain invariant features and domain specific features; finally, redesigns the loss function to promote the adaptation of the source domain to the target domain.
[0047] The overall block diagram of SAODUDA of the present invention is as follows Figure 1 As shown in the figure: first, the source domain image and label as well as the target domain image are input into the feature alignment network to realize the center alignment and distributed alignment of the input features; then the aligned features are input into the disentanglement representation network to decompose the input features into domain invariant features and domain specific features; finally, segmentation is realized through the segmentation network.
[0048] The overall network structure diagram of SAODUDA of the present invention is as follows Figure 2 As shown in the figure: the input image passes through the first layer of ResNet, SAN and SAW modules, and the second layer of ResNet, SAN and SAW modules to obtain center-aligned and distributed-aligned features; then after two downsamplings, it is input into the feature extractor, differential module, gradient reversal layer, and domain classifier to obtain DIFs (Domain-Invariant Features) and DSFs (Domain-Specific Features). Finally, the input features pass through DeepLab Head to obtain the segmentation result.
[0049] It should be noted that: SAODUDA of the present invention inputs the intermediate small batch features into SAN, and the features are processed by a classifier to generate a segmentation mask to highlight the foreground area while suppressing the background area; in view of the particularity of medical images, the present invention cancels the original strategy of the SAN module to input the real segmentation mask optimization model during the training process, and adopts a two-step optimization strategy during the training process to more effectively learn the center alignment features of the segmentation representation; the normalized features in SAN are sent to SAW for further processing to further enhance channel decorrelation and align the global edge distribution with the conditional distribution.
[0050] Embodiment 2:
[0051] The effectiveness of the present invention is evaluated below with reference to specific cases.
[0052] This paper tests the effectiveness of our proposed method in the cross-site prostate segmentation task and the cross-modal brain tumor segmentation task.
[0053] The cross-site prostate segmentation dataset is a collection of T2-weighted MRI images from two different medical institutions. The source domain contains 261 prostate images and segmentation masks from BIDMC, of which 239 are used as training sets and 22 are used as test sets. The target domain contains 158 prostate images and segmentation masks from HK, of which 147 are used as training sets and 11 are used as test sets. The data is preprocessed to keep only the slices containing the prostate.
[0054] The cross-modal brain tumor segmentation dataset is the 2018 Multimodal Brain Tumor Segmentation Challenge (BraTS) dataset. The source domain is T2 modality images and the target domain is FLAIR modality images. The dataset contains data from 75 patients, of which 60 patients' data are randomly selected as training sets and 15 patients' data are used as test sets. We focus on the segmentation of the entire tumor, conduct experiments in an unpaired manner, preprocess the data, and retain only slices containing brain tumors.
[0055] In order to verify the performance of the model proposed in this paper, this paper compares the SAODUDA model with seven classic UDA networks: DANN, ADDA, CycleGAN, CyCADA, SynSeg-Net, AdaOutput, and PnP-AdaNet, and three advanced UDA networks: ODADA, Shen et al, and MPSCL.
[0056] Table 1 Segmentation metrics of SAODUDA on the cross-site prostate segmentation dataset
[0057]
[0058] Table 1 is a comparison between the SAODUDA network proposed in this paper and the other 10 networks. It can be seen from the table that the SAODUDA proposed in this paper has a significant improvement in the DSC index, even surpassing the DSC index of supervised learning, and is also better than the other 10 networks in the HD index. It is slightly lower than the most advanced network in the three indicators of ASSD, Sensitivity, and Specificity, but still better than most classic networks. The SAODUDA network proposed in this paper has a significant improvement in both the DSC and HD indicators that are of great concern in medicine, which fully demonstrates the effectiveness of the network proposed in this paper in the cross-site domain adaptation segmentation task. Figure 6 Visualization of the segmentation results of the present invention on the cross-site prostate segmentation dataset.
[0059] Table 2 Segmentation indicators of SAODUDA in cross-modal brain tumor segmentation dataset
[0060]
[0061] It can be seen that the model proposed in the present invention is higher than other models in both DSC and Jaccard indexes, especially the Jaccard index, which is improved by nearly 4% compared with other models. Figure 7 Visualization of the segmentation results of the present invention on the cross-modal brain tumor segmentation dataset.
[0062] The present invention can be widely used in the unsupervised domain adaptation segmentation of medical images.
[0063] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0064] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. An unsupervised domain adaptation medical image segmentation method combining feature alignment and disentangled representation, characterized in that: The steps include: S1: Propose an unsupervised domain adaptation segmentation task for medical images, including the following steps: S11: Establish a domain adaptation segmentation network SAODUDA based on semantic perception and orthogonal decomposition: ResNet50 is used as the backbone network of SAODUDA, and DeepLab network is used as the segmentation network of SAODUDA; S12: Introducing the semantic-aware normalization module: Introducing the SAN (Semantic-Aware Normalization) module in the first and second layers of the ResNet network. The SAN module converts the input features into category-centric features to achieve center alignment of the input features. Then, the SAW (Semantic-Aware Whitening) module is introduced after the SAN module. The SAW module converts the features that have been centrally aligned by the SAN module into distributedly aligned features, aligning the edge distribution of joint features from different domains, thereby achieving semantically aware central alignment and distributed alignment. S13： Design feature extractor DIR: Input features are passed through the convolution layer, and then activated using the CELU (Continuously Differentiable Exponential Linear Unit) activation function after the BN layer, and randomly inactivated using Dropout. The above operation is repeated once, and finally passed through the convolution layer and the BN layer to obtain the extracted features. S14: Introduce the input feature orthogonal decomposition module: feature F after center alignment and distributed alignment ss The feature extractor DIR with large convolution kernel extracts the domain invariant features F from the input features. ssdi , feature F ss With domain invariant feature F ssdi Input to the difference module Diff to obtain domain-specific features F ssds ; In order to strengthen the domain invariant feature F ssdi With the domain-specific feature F ssds Independence of input features F ss At the same time, it is input into the feature extractor DIR with a small convolution kernel to obtain another set of domain invariant features F ssdi With the domain-specific feature F ssds , calculate the loss optimization model; then the gradient reversal layer GRL and the domain classifier learn the domain invariant feature F ssdi With the domain-specific feature F ssds ; S15: Design loss function: introduce a two-step optimization strategy. The first step is to learn the segmentation feature F in the source domain. ss At this time, only LogNLL Loss is involved in training. The second step is to learn the segmentation feature F in the source domain and the target domain. ss , and decompose it into domain invariant features F ssdi With domain-specific features F ssds , at this time, LogNLL Loss, Orthogonal Loss, SAW Loss, and DiceBCE Loss all participate in the training; the loss function redesigned by the present invention is defined as follows: L DB =α·L Dice +(1-a)·L BCE (4) Among them, L Dice For Dice Loss, L BCE is BCE Loss, C is the domain classifier, Generate the mean loss for the first and second SAW modules, L ⊥1 ,L ⊥2 F is generated by the first and second feature extractors DIR respectively ssdi ,F ssds Calculated loss, L s is the loss calculated in the source domain, L t is the loss calculated in the target domain, L st is the total loss in this paper, α, β, χ are parameters, which are 0.9, 0.4, and 0.9 respectively in this invention; S2: Establish a data set for model evaluation, including the following steps: S21: Selection of evaluation index: DSC (Dice Similarity Coefficient), ASSD (Average Surface Distance), HD (Hausdorff Distance), Sensitivity, Specificity, Jaccard are selected as the evaluation indexes of the present invention, wherein: ASSD(X,Y)={ASD(X,Y)+ASD(Y,X)} / 2 (10) Where: TP is the number of pixels properly classified into the foreground area; TN represents the number of pixels correctly classified as background area; FP represents the number of pixels that are incorrectly classified as foreground areas; FN represents the number of pixels that are incorrectly classified as background areas; L and P represent the foreground area sets of real data and predicted data, respectively; S22: Verification of the effectiveness of the evaluation model: This paper compares the SAODUDA model with seven classic UDA networks: DANN, ADDA, CycleGAN, CyCADA, SynSeg-Net, AdaOutput, and PnP-AdaNet, three advanced UDA networks: ODADA, Shen et al, and MPSCL, and a fully supervised network. SAODUDA has a significant improvement in the DSC indicator, even surpassing the DSC indicator of supervised learning. It is also better than the other 10 networks in the HD indicator. It is slightly lower than the most advanced network in the three indicators of ASSD, Sensitivity, and Specificity, but still better than most classic networks.
2. The unsupervised domain adaptation segmentation method for medical images according to claim 1, characterized in that: In the step S11, the input is the image and label of the source domain, and the image of the target domain; the input image passes through the first layer of ResNet, SAN and SAW modules and the second layer of ResNet, SAN and SAW modules to obtain the features of center alignment and distributed alignment; then after two downsampling, it is input to the feature extractor, the difference module, the gradient reversal layer, and the domain classifier to obtain DIFs (Domain-Invariant Features) and DSFs (Domain-Specific Features), and finally, the input features pass through DeepLabHead to obtain the segmentation result.
3. The unsupervised domain adaptation segmentation method for medical images according to claim 1, characterized in that: In the step S12, the intermediate small batch features are input into the SAN, and the features are passed through the classifier to generate a segmentation mask to highlight the foreground area and suppress the background area. In view of the particularity of medical images, the present invention cancels the original strategy of the SAN module to input the real segmentation mask optimization model during the training process, and adopts a two-step optimization strategy during the training process to more effectively learn the center alignment features of the segmentation representation; The normalized features in SAN are sent to SAW for further processing to further enhance channel decorrelation and align the global marginal distribution with the conditional distribution.
4. The unsupervised domain adaptation segmentation method for medical images according to claim 1, characterized in that: In step S13, since ReLU (Rectified Linear Unit) is not differentiable at zero and sets all negative values to zero, but CELU (Continuously Differentiable Exponential Linear Unit) is a continuously differentiable activation function and smoothes the negative value part, the CELU activation function is used instead of the ReLU activation function to perform gradient optimization more stably and retain more information and gradient signals.
5. The unsupervised domain adaptation segmentation method for medical images according to claim 1, characterized in that: In step S14, a large convolution kernel can capture features in a wider range, so that the spatial information in the input image or feature map can be better understood, but some detail information will be lost; introducing a small convolution kernel can capture local features more finely, have a more sensitive response to detail information, and optimize feature extraction.