Unsupervised medical image segmentation methods based on collaborative appearance and semantic adaptation

Abstract

This invention relates to an unsupervised medical image segmentation method based on collaborative appearance and semantic adaptation. Addressing the issues of high annotation costs and domain mismatch in medical datasets, this invention proposes an unsupervised domain adaptation framework that uses collaborative appearance and semantic adaptation for medical image segmentation. The method first captures pixel information of the target domain and the appearance of the region of interest (ROI) in the source domain through a feature transfer module. Then, it learns image representations between the two domains through a representation transfer module. Finally, it learns domain-invariant knowledge between the source and target domains through appearance-level and semantic-level knowledge alignment to segment the medical image. Compared to existing domain-adaptive medical image segmentation methods, this method utilizes appearance and semantic adaptation to overcome the domain transfer problem, enabling more accurate and effective extraction of ROIs in unlabeled target medical images.

Description

Technical Field

[0001] This invention relates to medical image segmentation methods, specifically an unsupervised medical image segmentation method based on collaborative appearance and semantic adaptation. It is primarily used in fields such as medical imaging, computer vision, and digital image processing. Background Technology

[0002] Deep neural networks have achieved great success when large amounts of labeled data are available and training and testing data come from the same distribution. However, because medical images acquired using different acquisition parameters or modes have very different features and their visual appearance varies significantly, cross-modal transfer will cause a severe degrade in the performance of deep networks. Therefore, well-trained models often fail when deployed to real-world clinical scenarios.

[0003] To mitigate the performance degradation caused by domain transfer, research on domain adaptation of deep models is emerging, aiming to efficiently transfer knowledge learned from the source domain to the target domain. A straightforward approach is to fine-tune a model pre-trained on the source data using additionally labeled target data. However, labeling data for each new domain is prohibitively expensive, sometimes even infeasible, especially in medical fields requiring specialized knowledge. Therefore, unsupervised domain adaptation is more attractive because it does not require labels from the target domain. Existing unsupervised domain adaptation methods typically employ adversarial learning to address domain transfer from different angles, including image-level alignment, feature-level alignment, and their hybrids. With the widespread success of CycleGAN in unpaired image-to-image translation, many image alignment methods are built upon this framework with additional constraints to further regulate the image translation process. For cross-modal adaptation, Jiang et al. first used CycleGAN with a tumor-sensing loss to convert CT images into MRI-like appearances, then combined the generated MRI images with some real MRI data for semi-supervised tumor segmentation. Zhang et al. combined CycleGAN with a segmentation network to create an end-to-end framework. Meanwhile, another category of work focuses on feature alignment, extracting domain-invariant features from deep neural networks in adversarial learning scenarios. For medical applications, the cross-protocol MRI segmentation proposed by Kamnitsas et al. was the first attempt to align feature distributions with adversarial losses. Subsequently, adversarial training was combined with other regularizations to achieve better adaptive performance on ultrasound datasets, histopathological images, and cardiac MRI.

[0004] Image and feature alignment are transfers from different perspectives to deep neural networks, which are actually complementary. Combining these two adaptive strategies to achieve stronger domain adaptation techniques is under exploration. As a recent approach to semantic segmentation adaptation, CyCADA and Zhang et al. have achieved leading performance in adapting between synthetic and real-world driving scene domains. However, their image and feature alignments are sequentially connected and trained at different stages without interaction. Due to the severe domain shift in cross-modal medical images, feature alignment or image alignment alone may not be sufficient to accomplish this challenging task, and simultaneous alignment from both perspectives has not been fully explored. Summary of the Invention

[0005] To address the shortcomings of the aforementioned technologies, the present invention aims to provide an unsupervised medical image segmentation method based on collaborative appearance and semantic adaptation. This method utilizes appearance and semantic adaptation to overcome the domain transfer problem, enabling more accurate and effective extraction of regions of interest in unlabeled target medical images.

[0006] The technical solution adopted by this invention to solve its technical problem is:

[0007] An unsupervised medical image segmentation method based on collaborative appearance and semantic adaptation includes the following steps:

[0008] Establish a feature transfer module to capture pixel information in the target domain and features of the region of interest in the source domain;

[0009] A representation transfer module is established to learn image representations between two domains. By aligning appearance-level and semantic-level knowledge, domain-invariant knowledge between the source and target domains is learned to segment and extract regions of interest in medical images.

[0010] The feature transmission module captures pixel information of the target domain and features of the region of interest in the source domain, including:

[0011] Based on the Convolutional Neural Network (CNN), an optimization formula L is established. CTM Optimize the feature transfer module to obtain source adaptive samples with domain-invariant appearance transfer.

[0012]

[0013] Where μ is a hyperparameter.

[0014] This includes: Step 1) Extracting high-level contextual information about objects and semantic relationships from the source image:

[0015] Given a source domain and a target domain Where N s Indicates the number of samples in the source domain. This represents the i-th source domain sample. N represents the annotation label of the i-th source domain sample. t Indicates the number of samples in the target domain. This represents the j-th target domain sample;

[0016] Samples with source domain The semantic context and target domain style of the white noise image are iteratively forwarded to a pre-trained convolutional neural network (CNN) to generate source-adapted samples.

[0017] This involves using the last three convolutional layers of a CNN to train and learn from the source samples. The semantic context information is used to establish a function that assigns different weights to different convolutional layers to better represent the source samples. Semantic context:

[0018]

[0019] This function encourages the source to adapt to the sample. Learning source domain samples High-level semantic information; among which, and These are the i-th source-adapted samples. and source domain samples Feature map of the l-th convolutional layer is the balancing parameter, Dist(·) is the Euclidean distance measurement function, and L1 is the last three convolutional layers in a CNN.

[0020] include:

[0021] Step 2) Extract the low-level features of all target samples and treat them as the style of the target domain:

[0022] a. In the l-th convolutional layer, there is C l The response map, therefore, the j-th target domain sample Styles on the l-th convolutional layer It can be defined as:

[0023]

[0024] Where ⊙ represents the inner product, C represents the connection of the vectorized response graph. l H l and W l These represent the channel dimension, height, and weight of the corresponding feature map, respectively;

[0025] b. Similarly, obtain the i-th source sample. Styles on the l-th convolutional layer

[0026] c. By analyzing the style of all target samples on the l-th convolutional layer Average the target samples The style is extended to the entire target domain, resulting in the style pattern of the target domain of the l-th convolutional layer.

[0027] Step 3) Use the first two convolutional layers in the CNN to obtain samples from the target domain. The style of the target domain is integrated into the domain-adapted sample images:

[0028]

[0029] This function is used for source-adapted samples. Learning target domain samples The style; among them, L2 represents the balancing parameter, and L2 is the first two convolutional layers in a CNN.

[0030] The representation transmission module learns the image representation between two domains, including the following steps:

[0031] Construct a condition generator G(·) to process the source domain samples. Learning to generate output feature maps

[0032] Construct a feature extractor E(·) to obtain source-adapted samples. Feature map

[0033] Construct the discriminator D(·), and establish the discriminant optimization function L. GAN This is used to distinguish between target domain images and source domain images;

[0034] Construct a pixel classifier C(·), and establish a classification optimization function L. SEG Used for source sample feature maps Perform semantic segmentation and extract regions of interest;

[0035] Establish a comprehensive optimization function L representing the transmission module. RTM ; used to obtain feature maps using the generator G(·) with the function minimized as the final result. The region of interest is extracted using a pixel classifier c(·);

[0036]

[0037] Where σ is a hyperparameter.

[0038] The construction condition generator G(·) includes:

[0039] Combine the noise vector z and the source adaptive sample The feature map is used as input, and the source-adapted samples are used as input. Feature map and noise mapping Convert to source domain sample Feature map

[0040]

[0041] The condition generator G(·) consists of a feature extractor E(·) and a noise encoder. The feature extractor E(·) is used to obtain source-adapted samples. Feature map noise encoder Source-adapted samples are obtained by encoding the noise vector z. noise mapping θ G and It is G(·) and Network parameters.

[0042] It is the source sample feature map The feature maps of the target samples are fed into the discriminator D(·) to distinguish between the target domain image and the source domain image; the optimization function L GAN for:

[0043]

[0044] Where, θ D These are the network weights of the discriminator D(·). This represents the expectation when all samples are samples from the target domain. This indicates that all samples are the expected values ​​when the generator generates samples. Used to generate target domain samples Feature map, Used to generate source domain samples The feature map.

[0045] It is the source sample feature map Forwarded to pixel classifier C(·), used for semantic segmentation of medical images in a fully supervised manner;

[0046] Given the target representation of the j-th sample Input it into a pixel classifier C(·) to mine confidence pseudo-labels in a self-supervised manner. Furthermore, the entire segmentation architecture was optimized; the formulas L for C(·) and G(·) were optimized. SEG The definition is as follows:

[0047]

[0048] in, Let I(·) represent the confidence pseudo-label of the j-th target sample, I(·) represent the cross-entropy loss, Z(·) represent the Dice loss function, and θ C These are the network parameters of C(·).

[0049] The present invention has the following beneficial effects and advantages:

[0050] 1. The method of the present invention is aimed at unsupervised domain adaptation medical image segmentation. It considers using appearance and semantic adaptation to overcome the domain transfer problem. Compared with existing domain adaptation medical image segmentation methods, this method can extract regions of interest more accurately and effectively in unlabeled target medical images.

[0051] 2. The method of this invention is a segmentation method for medical images. It mainly alleviates the severe domain offset problem in cross-modal medical images from two perspectives: image alignment and feature alignment. It focuses on reducing the distribution differences between domains to improve the generalization performance of unlabeled target domains. A feature transfer module is used to learn low-level features of target samples and high-level semantic information of source samples to generate source adaptive samples with the style of the target samples. Then, a generative adversarial network is used to increase the robustness of the network. The Dice loss function is used to solve the class imbalance problem in medical image segmentation, achieving more accurate segmentation. Attached Figure Description

[0052] Figure 1 This is a flowchart of the method of the present invention;

[0053] Figure 2 These are the medical images used in this invention; where a and b are CT images, and c and d are MR images;

[0054] Figure 3 These are two processing examples of CT images with MR style;

[0055] Figure 4 These are two processing examples of MR images with CT style;

[0056] Figure 5 These are two instance segmentation results from a CT image;

[0057] Figure 6 This is a segmentation result of two instances of an MR image. Detailed Implementation

[0058] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0060] In medical image segmentation, the visual appearance varies significantly due to the different physical principles of imaging methods, leading to a severe performance degradation of deep networks due to cross-modal domain transfer. To address the challenging cross-modal adaptation problem in segmentation tasks, this invention collaboratively merges image and feature alignment adaptive processes into a unified network to fully leverage their mutual benefits in unsupervised domain adaptation. First, a feature transfer module captures pixel information from the target domain and the appearance of the region of interest (ROI) from the source domain. Then, a representation transfer module learns image representations between the two domains. Finally, appearance-level and semantic-level knowledge alignment learns domain-invariant knowledge between the source and target domains to segment the medical image, thereby extracting the ROI. The flowchart of the unsupervised medical image segmentation method based on collaborative appearance and semantic adaptation proposed in this chapter is attached. Figure 1 As shown.

[0061] This invention uses CT scans as the source domain image (the CT image has been labeled with regions of interest by a doctor), learns the features of the regions of interest in the source domain, processes images with MR as the target domain, extracts and segments the regions of interest on the target domain-style image, and labels them with classification tags (for machine learning classification labeling, replacing manual labeling by doctors). The reverse is also true.

[0062] 1. The feature transfer module captures pixel information of the target domain and the appearance of the region of interest in the source domain.

[0063] To make samples from different fields visually similar, this invention develops a feature transfer module. Its goal is to make the source region of interest appear as if it were extracted from the target dataset. CT and MR images are common medical images. Figure 2Some CT and MR images are listed, with CT images used as source images and MR images as target images, and vice versa; where a and b are CT images, and c and d are MR images. In the feature transfer module, high-level contextual information about objects and semantic relationships is first extracted from the source images. Then, low-level features of all target samples are extracted and treated as the style of the target domain, combined with the high-level contextual information of objects and semantic relationships, and finally, source region of interest appearance adaptation samples with the target domain style are generated. (Appendix) Figure 3 Displayed MR images with CT style, attached Figure 4 The image displayed is a CT image with an MR style.

[0064] 1) Extract high-level contextual information about objects and semantic relationships from the source image. Given a source domain... and a target domain Where N s Indicates the number of samples in the source domain. This represents the i-th source domain sample. N represents the annotation label of the i-th source domain sample (the classification label of the region of interest given by the doctor). t Indicates the number of samples in the target domain. This represents the j-th target domain sample. It will have samples from the source domain. The semantic context and target domain style of the white noise image are iteratively forwarded to a pre-trained convolutional neural network (CNN) to generate source-adapted samples. In CNNs, higher-level convolutional layers can explore richer image contextual information; therefore, the last three convolutional layers in a CNN are used to explore source-adapted samples. The semantic context information is used to establish a function that assigns different weights to different convolutional layers to better represent the source samples. Semantic context:

[0065]

[0066] This function encourages the source to adapt to the sample. Learning source domain samples High-level semantic information. Among them, and These are the i-th source-adapted samples. and source domain samples Feature map of the l-th convolutional layer is the balancing parameter, Dist(·) is the Euclidean distance measurement function, and L1 is the last three convolutional layers in a CNN.

[0067] 2) Extract low-level features from all target samples and treat them as the style of the target domain. The style of the target domain is considered as statistical information and related to semantic relationships between different response maps. In the l-th convolutional layer, there is C l The response map, therefore, the j-th target domain sample Styles on the l-th convolutional layer It can be defined as:

[0068]

[0069] Where ⊙ represents the inner product, C represents the connection of the vectorized response graph. l H l and W l These represent the channel dimension, height, and weight of the corresponding feature map, respectively. Similarly, this method can be used to obtain the i-th source-adapted sample. Styles on the l-th convolutional layer

[0070] By analyzing the style of all target samples on the l-th convolutional layer By averaging, the target sample can be... The style is extended to the entire target domain, resulting in the style pattern of the target domain of the l-th convolutional layer.

[0071] 3) In CNNs, lower-level convolutional layers can explore the appearance information of an image; therefore, the first two convolutional layers in a CNN are used to explore samples in the target domain. The style of the target domain is integrated into the domain-adapted sample images:

[0072]

[0073] This function encourages the source to adapt to the sample. Learning target domain samples The style. Among them, L2 represents the balancing parameter, and L2 is the first two convolutional layers in a CNN.

[0074] 4) Using the optimization formula L CTM Optimize the feature transmission module:

[0075]

[0076] The feature transfer module can use this formula to obtain source adaptive samples with domain-invariant appearance transfer. Where μ is a hyperparameter.

[0077] 2. Learning the image representation between two domains through the representation transmission module includes the following steps:

[0078] 1) Using the noise vector z as input and adapting the source samples A conditional generator G(·) is constructed from the feature maps to achieve cross-domain representation adaptation. The conditional generator G(·) consists of a feature extractor E(·) and a noise encoder. The feature extractor E(·) is used to obtain source-adapted samples. Feature map noise encoder Source-adapted samples are obtained by encoding the noise vector z. noise mapping The condition generator G(·) adapts the source to the sample. Feature map and noise mapping Convert to source domain sample Feature map

[0079]

[0080] Where, θ G and It is G(·) and Network parameters.

[0081] 2) This indicates that the transmission module encourages the source sample feature map. Preserving source adaptive samples The semantic context and style of the target domain are considered, so the source sample feature map is... The feature maps of the target samples are fed into the discriminator D(·) to distinguish between the target domain image and the source domain image. An optimization function L is established. GAN :

[0082]

[0083] Where, θ D These are the network weights of the discriminator D(·). This represents the expectation when all samples are samples from the target domain. This indicates that all samples are the expected values ​​when the generator generates samples. Used to generate target domain samples Feature map, Used to generate source domain samples The feature map.

[0084] The method of segmenting medical images by learning domain-invariant knowledge between the source and target domains through alignment of appearance-level and semantic-level knowledge, thereby extracting regions of interest, includes:

[0085] Source Sample Feature Map The data is forwarded to a pixel classifier C(·) for semantic segmentation of medical images in a fully supervised manner. Furthermore, given the target representation of the j-th sample... Input it into a pixel classifier C(·) to mine confidence pseudo-labels in a self-supervised manner. And further optimize the entire partitioning architecture. Optimize the formulas L for C(·) and G(·). sEG The definition is as follows:

[0086]

[0087] in, Let I(·) represent the confidence pseudo-label of the j-th target sample, I(·) represent the cross-entropy loss, Z(·) represent the Dice loss function, and θ C These are the network parameters of C(·).

[0088] In summary, the optimization function L of the transmission module is... RTM Represented as:

[0089]

[0090] Where σ is a hyperparameter.

[0091] Figure 5 This is a segmentation result image of a CT image, which is a... Figure 2 The result of processing a and b is shown in the figure; Figure 6 This is a segmentation result image of an MR image, which is a... Figure 2 The result of processing c and d is shown in the figure.

[0092] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should be considered within the scope of protection of the present invention.

Claims

1. An unsupervised medical image segmentation method based on collaborative appearance and semantic adaptation, characterized in that, Includes the following steps: Establish a feature transfer module to capture pixel information in the target domain and features of the region of interest in the source domain; 1) Extract high-level contextual information about objects and semantic relationships from the source image; given a source domain and a target domain Where N s Indicates the number of samples in the source domain. This represents the i-th source domain sample. N represents the classification label of the i-th source domain sample. t Indicates the number of samples in the target domain. Represents the j-th target domain sample; and includes samples from the source domain. The semantic context and target domain style of the white noise image are iteratively forwarded to a pre-trained convolutional neural network to generate source-adapted samples. 2) Extract the low-level features of all target samples and treat them as the style of the target domain. The style of the target domain is considered as statistical information and related to the semantic relationships between different response maps; in the l-th convolutional layer, there is C l The response map, therefore, the j-th target domain sample Styles on the l-th convolutional layer Defined as: Where ⊙ represents the inner product, The connection represents the vectorized response graph; The same method is used to obtain the i-th source adaptation sample. Styles on the l-th convolutional layer 3) In CNNs, lower-level convolutional layers can explore the appearance information of an image; therefore, the first two convolutional layers in a CNN are used to explore samples in the target domain. The style of the target domain is integrated into the domain-adapted sample images: This function encourages the source to adapt to the sample. Learning target domain samples The style; among them, The L2 parameter represents the balancing parameter; L2 is the first two convolutional layers in a CNN. The style of the target domain of the l-th convolutional layer; For the i-th source sample Style patterns on the l-th convolutional layer; 4) Using the optimization formula L CTM Optimize the feature transmission module: in, and These are the i-th source-adapted samples. and source domain samples Feature map of the l-th convolutional layer; The feature transfer module uses this formula to obtain source adaptive samples with domain-invariant appearance transfer. Where μ is a hyperparameter; A representation transfer module is established to learn image representations between two domains. By aligning appearance-level and semantic-level knowledge, domain-invariant knowledge between the source and target domains is learned to segment and extract regions of interest in medical images. The representation transmission module learns the image representation between two domains, including the following steps: Construct a condition generator G(·) to process the source domain samples. Learning to generate output feature maps Construct a feature extractor E(·) to obtain source-adapted samples. Feature map Construct the discriminator D(·), and establish the discriminant optimization function L. GAN Used to distinguish source-adapted samples Feature map and output feature map Construct a pixel classifier C(·), and establish a classification optimization function L. SEG Used for source sample feature maps Perform semantic segmentation and extract regions of interest; Establish a comprehensive optimization function L representing the transmission module. RTM ; used to obtain feature maps using the generator G(·) with the function minimized as the final result. The region of interest is extracted using a pixel classifier C(·); Where σ is a hyperparameter.

2. The unsupervised medical image segmentation method based on collaborative appearance and semantic adaptation according to claim 1, characterized in that, The construction condition generator G(·) includes: Combine the noise vector z and the source adaptive sample The feature map is used as input, and the source-adapted samples are used as input. Feature map and noise mapping Convert to source domain sample Feature map The condition generator G(·) consists of a feature extractor E(·) and a noise encoder. The feature extractor E(·) is used to obtain source-adapted samples. Feature map noise encoder Source-adapted samples are obtained by encoding the noise vector z. noise mapping θ G and It is G(·) and Network parameters.

3. The unsupervised medical image segmentation method based on collaborative appearance and semantic adaptation according to claim 1, characterized in that, It is the source sample feature map The feature maps of the target samples are fed into the discriminator D(·) to distinguish between the target domain image and the source domain image; the optimization function L GAN for: Where, θ D These are the network weights of the discriminator D(·). This represents the expectation when all samples are samples from the target domain. This indicates that all samples are the expected values ​​generated by the generator when the samples were generated. Used to generate target domain samples Feature map, Used to generate source domain samples The feature map.

4. The unsupervised medical image segmentation method based on collaborative appearance and semantic adaptation according to claim 1, characterized in that, It is the source sample feature map Forwarded to pixel classifier C(·), used for semantic segmentation of medical images in a fully supervised manner; Given the target representation of the j-th sample Input it into a pixel classifier C(·) to mine confidence pseudo-labels in a self-supervised manner. Furthermore, the entire segmentation architecture was optimized; the formulas L for C(·) and G(·) were optimized. SEG The definition is as follows: in, Let I(·) represent the confidence pseudo-label of the j-th target sample, I(·) represent the cross-entropy loss, Z(·) represent the Dice loss function, and θ C These are the network parameters of C(·).

Images