Bidirectional cross-modal unsupervised image segmentation domain adaptation method based on wavelet spectrum migration

Abstract

The application discloses a bidirectional cross-modal unsupervised image segmentation domain adaptation method based on wavelet spectrum migration, comprising the following steps: constructing source domain data and target domain data, obtaining migration data based on the source domain data and the target domain data; constructing a domain segmentation model based on the migration data; optimizing the domain segmentation model based on a self-training strategy to obtain an optimal segmentation model; and performing image domain segmentation based on the optimal segmentation model. The bidirectional cross-modal unsupervised domain adaptation framework proposed by the application discards the traditional adversarial network training mode, explicitly reduces the data difference between the source domain and the target domain, and thus effectively improves the segmentation performance in the target domain.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image processing and application, and particularly relates to a bidirectional cross-modal unsupervised image segmentation domain adaptation method based on wavelet spectrum migration. BACKGROUND

[0002] At present, multi-modal medical image segmentation based on deep learning has achieved great success, which is of great significance for clinical evaluation, diagnosis and treatment. We often have a basic assumption that in the process of model training, the training set (source domain) and the test set (target domain) come from the same domain, and the data distribution is unbiased. However, in practical application, this assumption is too strong. The cardiac region images under different modalities have completely different visual characteristics due to physical imaging characteristics. Therefore, when the trained model is applied to the actual medical scene, the performance of the model will often decline dramatically due to the cross-domain bias of the medical image data collected from different scanning machines, configurations and modalities. In order to solve the problem of domain bias, domain adaptation technology has attracted widespread attention of scholars in recent years, which tries to train a model that can perform well in the target domain under the data distribution of the source domain. When labels are available, the most direct way is to fine-tune the pre-trained model in the source domain in the target domain. However, unlike natural images, there is no large-scale labeled dataset similar to ImageNet in medical images. Firstly, it is difficult to obtain a reliable and effective pre-trained model. In addition, due to the extremely expensive medical image labeling, not only expert knowledge is needed, but also time-consuming and laborious labeling is needed, so it is not practical to additionally label a new domain. On the contrary, unsupervised domain adaptation, as a powerful solution, is widely used in clinical practice because it does not require target domain labeled data.

[0003] Currently, advanced unsupervised domain adaptation methods in medical image analysis mainly include feature-level alignment, image-level alignment and feature and image combined alignment, and some works are based on decoupled representation and ensemble learning. The method based on feature-level alignment learns domain-invariant features by forcing the network to minimize the feature distribution between the two domains through adversarial learning, but this method is usually not sensitive to pixel-level domain shift, so it is not conducive to dense medical image segmentation tasks. In addition, the method based on image-level alignment is mainly to transform the source domain image at the pixel level into an image similar to the target domain through image migration, and then reduce the domain shift through adversarial means, but it cannot completely guarantee the consistency of semantics. Combining the above two methods, a few works combine the two methods to further solve the domain adaptation in cross-modal medical images. The methods currently proposed have achieved good results, but all have a problem: the training is complex. The adversarial network is prone to collapse due to the training itself, and the model optimization process is too complex. Therefore, it is urgent to seek a simple and effective way to explicitly learn the shift between domains, thereby directly improving the performance of the model on the target domain. SUMMARY

[0004] The purpose of the present application is to provide a bidirectional cross-modal unsupervised image segmentation domain adaptation method based on wavelet spectrum migration to solve the problems existing in the prior art.

[0005] To achieve the above purpose, the present application provides a bidirectional cross-modal unsupervised image segmentation domain adaptation method based on wavelet spectrum migration, comprising:

[0006] Constructing source domain data and target domain data, and obtaining migration data based on the source domain data and the target domain data;

[0007] Constructing a domain segmentation model based on the migration data;

[0008] Optimizing the domain segmentation model based on a self-training strategy to obtain an optimal segmentation model;

[0009] Performing image domain segmentation based on the optimal segmentation model.

[0010] Optionally, the source domain data includes source domain images, and the target domain data includes target domain images.

[0011] Optionally, the migration data is data with texture detail information of the target domain data; the process of obtaining migration data based on the source domain data and the target domain data comprises:

[0012] Performing spectrum migration training on the source domain data to obtain the migration data.

[0013] Optionally, the process of performing spectrum migration training on the source domain data comprises:

[0014] Performing stationary wavelet transform on the source domain data and the target domain data, obtaining a source domain high-frequency wavelet component and a target domain high-frequency wavelet component, and replacing the source domain high-frequency wavelet component with the target domain high-frequency wavelet component.

[0015] Optionally, the process of constructing the domain segmentation model based on the migration data comprises:

[0016] Passing the migration data through the encoder and training the migration data to obtain the domain segmentation model.

[0017] Optionally, in the process of passing the migration data through the encoder and training the migration data, an entropy-based regularization constraint is adopted on the migration data.

[0018] Optionally, the entropy-based regularization constraint comprises:

[0019]

[0020] wherein ρ(x)=(x 2 +0.001 2 ) η , is target domain data, E w is a shared encoder;

[0021] The total loss function is

[0022] Optionally, the process of optimizing the domain segmentation model based on a self-training strategy comprises:

[0023] Obtaining three segmentation models based on horizontal, vertical and diagonal high-frequency components of the wavelet domain of the target domain data respectively, obtaining three prediction results based on the three segmentation models, and optimizing the domain segmentation model based on the average value of the three prediction results as the pseudo label of the target domain data.

[0024] The technical effects of the present application are:

[0025] 1. The bidirectional cross-modal unsupervised domain adaptation framework proposed in the present application discards the traditional adversarial network training method, explicitly reduces the data difference between the source domain and the target domain, and thus effectively improves the segmentation performance in the target domain.

[0026] 2.The application proposes a spectrum migration method based on stationary wavelet transform, which can obtain source domain data with target domain detail characteristics, so as to train a model suitable for the target domain. For medical cross-modal images, due to the inherent property difference of the scanning device, there is a large difference in the texture detail information between different modal images. Therefore, in the process of model training, this texture detail difference is what the model needs to learn, but this low-dimensional information is relatively difficult for the model to learn. Therefore, if we migrate this information from the target domain to the source domain at the beginning, it will make the model optimization process simpler, so as to focus on learning other features.

[0027] 3.In order to further improve the domain adaptation ability of the model and achieve higher performance in the target domain, the application adopts an entropy-based loss function to constrain the decision boundary of the target domain, so as to optimize the model to achieve higher segmentation performance in the target domain.

[0028] 4.The application introduces a self-training technique, which generates unlabeled data by using a model learned in the labeled source domain data. We use the average teacher training strategy to optimize the model. Specifically, we migrate the horizontal, vertical and diagonal high-frequency components of the wavelet domain of the target domain data to train three different models, and then use the average of the prediction results of the three models as the pseudo-label of the target domain data to optimize the model. BRIEF DESCRIPTION OF DRAWINGS

[0029] The drawings constituting a part of this application are used to provide a further understanding of the application, the illustrative embodiments of the application and the description thereof are used to explain the application, and do not constitute an improper limitation on the application. In the drawings:

[0030] Figure 1 Fig. 1 is a diagram of a wavelet spectrum migration-based bidirectional cross-modal unsupervised image segmentation domain adaptation model training framework in the embodiments of the application, wherein (a) is a diagram of wavelet transform decomposition and high-frequency spectrum component migration, and (b) is a diagram of a wavelet spectrum migration-based unsupervised domain adaptation framework. DETAILED DESCRIPTION

[0031] It should be noted that the embodiments and features in the embodiments in the application can be combined with each other without conflict. The application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0032] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] Embodiment one

[0034] like Figure 1 As shown, this embodiment provides a bidirectional cross-modal unsupervised image segmentation domain adaptation method based on wavelet spectral migration.

[0035] For unsupervised cross-modal medical image segmentation domain adaptation, we provide source domain data. in It is a source domain image. yes The segmentation label, n c This represents the number of categories to be segmented. Target domain data. Source domain data and target domain data There is a domain offset. To enable the model to learn the aforementioned texture detail differences, such as... Figure 1 As shown in (b) above, the source domain data is first... Data with the target domain is obtained through wavelet-based spectral transfer (WST) module. x texture detail information s→t Then through the shared encoder E w Then through S w The trained model is suitable for the target domain. Furthermore, to better constrain the network's decision boundary and thus better suit the target domain, we introduce an entropy-based regularization constraint, i.e. After passing through the shared encoder and then through the mapper P w Constraints were then applied. Furthermore, a self-training strategy was introduced to further improve model performance.

[0036] To reduce domain bias, considering the characteristics of medical imaging images, different modalities exhibit significant differences in the presentation of texture details. Therefore, to mitigate this difference, we introduce stationary wavelet transform and propose a conceptually simple yet effective spectral transfer domain adaptation method. Stationary wavelet transform possesses translation invariance and, compared to wavelet transform, offers better representation of the image's frequency domain information due to the absence of a downsampling process. The process of stationary wavelet transform is as follows: Figure 1 As shown in (a) in the figure, where ψ0 and ψ0 represent the low-pass and high-pass filters of the standard 1-level 2D stationary wavelet decomposition, respectively.

[0037] Specifically, a random image from the source domain is given. and an image of the target domain. (like Figure 1 (a) in (For example). We first perform a stationary wavelet transform on it. The frequency subbands at the j-th layer are obtained as follows: LL, LH, HL, HH.

[0038]

[0039] After M-th layer decomposition, we get (3M+1) wavelet subbands of the input source image Similarly sized wavelet subbands, i.e. LL M , where LL M represents the low frequency component of the M-th layer, which mainly represents the shape semantic approximation information of the original image, while LH j , HL j , HH j represent the high frequency detail texture information in horizontal, vertical and diagonal directions respectively.

[0040] Specifically, for cross-modal medical image domain adaptation, we randomly give a source domain image and a target domain image First, we perform stationary wavelet transform to get their corresponding high frequency wavelet components, then we directly replace the high frequency components of the source domain image with the high frequency components of the target domain image, and the migrated image has the detail information of the target domain image while still retaining the shape information of the source domain, and then the segmentation model is trained.

[0041] After getting the migrated image, we train a segmentation network φ w = E w · S w , and we use the following cross-entropy loss and Dice loss

[0042]

[0043] Through the WST migration module, we can align the detail information of the source domain and the target domain, but since there is no label in the target domain, in order to better constrain the decision boundary of the network and thus better adapt to the target domain, we introduce an entropy-based regularization constraint to optimize E w ,

[0044]

[0045] where ρ(x) = (x 2 + 0.001 2 ) η , and the overall loss function is:

[0046]

[0047] Further, we introduce self-training as a way to improve the performance. It generates pseudo labels for the unlabeled data by using the model learned on the labeled data. We optimize the model by using the average teacher training strategy. Specifically, we migrate the horizontal, vertical and diagonal high frequency components of the target domain data to train three different models, and then use the average of the predicted results of the three models as the pseudo label of the target domain data to optimize the model.

[0048]

[0049] It is shown that low-dimensional difficult-to-handle changes, such as contours and edge details, are difficult for the model to learn relevant features for domain adaptation if the source domain does not have corresponding changes. However, if we migrate such changes to the source domain at the beginning, the domain adaptation process will become easier. To achieve this migration, we introduce stationary wavelet transform. The high frequency spectrum components in the wavelet domain represent different directional texture information, and the low frequency spectrum components represent high-level semantic shape approximation information. We directly migrate the high frequency spectrum components of the target domain data to the source domain, and align the low-dimensional details and texture features of the source and target domains through simple processing. The principle of the method is shown in Figure 1 We respectively perform stationary wavelet transform on the source and target domain images to obtain low and high frequency spectrum (which includes horizontal, vertical and diagonal spectrum components, representing horizontal, vertical and diagonal direction details and contour information, respectively). We directly replace the high frequency spectrum in the source domain with the high frequency spectrum of the target domain while keeping the low frequency spectrum unchanged, and then reconstruct the migrated image through inverse stationary wavelet transform. Therefore, the migrated image has the details of the target domain while maintaining the high-level semantic information of the source domain. We use the migrated labeled source domain data to train a model that can perform well on unlabeled target domain data. Since there are no labels in the target domain, we introduce entropy-based regularization constraints to better constrain the model's decision boundary and make it more suitable for the target domain. To further improve the learning ability of the model, we also introduce self-training strategy to generate pseudo labels for unlabeled data by using the model learned on the labeled data, and use the average teacher training strategy to further optimize the model. Experiments show that the adaptation method of simple high frequency spectrum migration is superior to the relatively complex adversarial learning method.

[0050] The method of the present application proposes a simple and effective spectrum migration strategy based on stationary wavelet transform, and combines an entropy-based regularization constraint and a self-training strategy to propose an effective cross-modal unsupervised domain adaptation technology. With the development of the domain adaptation method based on adversarial learning, great progress has been made in cross-modal unsupervised medical image segmentation domain adaptation, but because the training of adversarial learning itself is easy to collapse, and the model optimization process is too complex. Therefore, we hope to propose an easy-to-train and simple and effective domain adaptation method. A large number of experiments in the public whole heart segmentation dataset show that our technology is superior to many advanced methods at present. The present application can effectively alleviate the performance degradation caused by the difference between the training set and the test set in the actual cross-modal medical image segmentation application, and is easy to implement, and has high application value.

[0051] The above description is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A bidirectional cross-modal unsupervised image segmentation domain adaptation method based on wavelet spectral transfer, characterized in that, Includes the following steps: Construct source domain data and target domain data, and obtain migration data based on the source domain data and target domain data; The migration data is data containing the target domain data texture details; The process of obtaining migration data based on the source domain data and the target domain data includes: performing spectral transfer training on the source domain data to obtain the migration data; The process of performing spectral transfer training on the source domain data includes: performing stationary wavelet transform on the source domain data and the target domain data to obtain the source domain high-frequency wavelet components and the target domain high-frequency wavelet components, and replacing the source domain high-frequency wavelet components with the target domain high-frequency wavelet components. A domain segmentation model is constructed based on the migration data. The process of constructing the domain segmentation model based on the migration data includes: passing the migration data through an encoder and training it to obtain the domain segmentation model; during the process of passing the migration data through the encoder and training it, entropy-based regularization constraints are applied to the migration data. The entropy-based regularization constraints include: Where ρ(x)=(x 2 +0.001 2 ) η , For target domain data, E w For shared encoders; The overall loss function is The domain segmentation model is optimized based on a self-training strategy to obtain the optimal segmentation model; The process of optimizing the domain segmentation model based on the self-training strategy includes: obtaining three segmentation models based on the horizontal, vertical and diagonal high-frequency components of the wavelet domain of the target domain data respectively; obtaining three prediction results based on the three segmentation models; and optimizing the domain segmentation model based on the average of the three prediction results as the pseudo-label of the target domain data. Image domain segmentation is performed based on the optimal segmentation model.

2. The bidirectional cross-modal unsupervised image segmentation domain adaptation method based on wavelet spectral transfer according to claim 1, characterized in that, The source domain data includes a source domain image, and the target domain data includes a target domain image.

Images