Unsupervised Domain Adaptation Medical Image Segmentation Method and Device Based on Multi-Scale Features

Abstract

The present invention discloses an unsupervised domain adaptation medical image segmentation method and device based on multi-scale features, which relates to the field of computer technology. The method includes: constructing a multi-scale feature fusion segmentation network based on dual attention; using the multi-scale feature fusion segmentation network as a generator, and constructing a first discriminator and a second discriminator; training based on the generator, the first discriminator and the second discriminator to obtain a trained multi-scale feature fusion segmentation network; using the trained multi-scale feature fusion segmentation network to segment the medical image to be processed. The present invention introduces multi-scale features to consider feature representations at different levels, and through an adversarial feature alignment method, realizes the alignment of source domain and target domain features through adversarial learning, thereby improving the performance and generalization ability of the segmentation model in the target domain; moreover, the present invention does not need to use target domain labels, and can well alleviate the unsupervised domain adaptation problem of the medical image segmentation task.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to an unsupervised domain adaptation medical image segmentation method and device based on multi-scale features. Background Art

[0002] Domain Adaptation (DA) is of great significance in the segmentation of optic cup and optic disc in fundus images. Because in different fundus image datasets (source domain and target domain), there may be differences in shooting devices, environmental conditions, image quality, etc., resulting in a decline in the performance of the model trained in the source domain in the target domain. This performance decline is often caused by the data distribution difference between the source domain and the target domain. For example, differences in the morphology, color, texture, etc. of the optic cup and optic disc. A common DA solution is to reduce the distribution difference between the source domain and the target domain through feature alignment to improve the generalization ability of the model in the target domain. However, in Unsupervised Domain Adaptation (UDA), the target domain usually lacks label information, which makes the traditional feature alignment methods that require target domain label information inapplicable. Therefore, for the task of optic cup and optic disc segmentation in fundus images, a feature alignment method applicable to UDA needs to be proposed.

[0003] In a neural network, low-level features cover rich details such as color, illumination, and resolution, which help capture local information of an image. In the optic cup and optic disc segmentation task, low-level features may include color information of the image (such as different gray levels), texture information (such as spots and stripes), and edge information (such as the boundaries of the optic cup and optic disc). These features help the network identify subtle changes and local structures in different regions of the image. On the contrary, high-level features involve more abstract semantic information such as semantic structures and deformations. In the optic cup and optic disc segmentation task, high-level features may include the shape, size, and relative position of the optic cup and optic disc, as well as the spatial relationship between them. These high-level features help the network understand the association between different regions in the image and convert it into semantically meaningful information, thus better completing the segmentation task. However, there are also some redundant features caused by external factors in fundus images, such as shadows under illumination conditions, reflections on the surface of the eyeball, and different image noises. To comprehensively consider low-level features and high-level features, this chapter adopts the method of multi-scale features to consider feature representations at different scales. Multi-scale features can comprehensively consider the information of the image at different scales. However, due to the existence of domain shift, the feature differences between the source domain and the target domain are large, so it is necessary to align the features of the source domain and the target domain. But due to the lack of label information in the target domain, it is impossible to directly align the features of the source domain and the target domain. To solve this problem, this chapter proposes an adversarial feature alignment method, which realizes the effective alignment of feature representations between the source domain and the target domain without the need for target domain labels by introducing adversarial learning techniques based on generative adversarial networks, thereby improving the performance and generalization ability of the model in the target domain. Summary of the Invention

[0004] The purpose of this application is to propose an unsupervised domain adaptation medical image segmentation method and device based on multi-scale features for the above-mentioned technical problems. By introducing multi-scale features to consider feature representations at different levels, and through the adversarial feature alignment method, the features of the source domain and the target domain are aligned through adversarial learning, thereby improving the performance and generalization ability of the segmentation model in the target domain; and the present invention does not require the use of target domain labels, and can well alleviate the unsupervised domain adaptation problem of medical image segmentation tasks.

[0005] On the one hand, an unsupervised domain adaptation medical image segmentation method based on multi-scale features includes:

[0006] Steps for constructing a multi-scale feature fusion segmentation network: Construct a multi-scale feature fusion segmentation network based on dual attention. The multi-scale feature fusion segmentation network includes a number of sequentially connected residual modules, dual attention mechanism modules (DAMMs) respectively connected to each residual module, and a multi-scale feature fusion module (MFFM). The convolutional layer performs initial feature extraction on the input source domain / target domain medical image, and then extracts high-level image features through each residual module to generate feature maps of different sizes. Each feature map is fed into a DAMM module to generate compressed feature maps of different sizes after fine-grained compression of the features. The compressed feature maps of different sizes are aligned to the same size through upsampling. After all the aligned compressed feature maps are input into an MFFM module for feature fusion, a probability map for per-pixel prediction is obtained through a 1x1 convolution and a softmax layer to obtain semantic features. Some of the aligned compressed feature maps are input into another MFFM module for feature fusion to obtain aggregated features.

[0007] Steps for constructing the first discriminator and the second discriminator based on the generator: Use the multi-scale feature fusion segmentation network as the generator, and construct the first discriminator and the second discriminator. The first discriminator is used to align the aggregated features of the source domain medical image and the target domain medical image, and the second discriminator is used to align the semantic features of the source domain medical image and the target domain medical image.

[0008] Steps for training the generator and the discriminators: Based on the generator, the first discriminator, and the second discriminator, train to obtain a trained multi-scale feature fusion segmentation network. During the training process, use the source domain data to optimize the hybrid segmentation loss of the generator. Transfer the source domain aggregated features and the target domain aggregated features to the first discriminator to optimize the first discriminator loss and the adversarial loss between the first discriminator and the generator. Transfer the source domain semantic features and the target domain semantic features to the second discriminator to optimize the second discriminator loss and the adversarial loss between the second discriminator and the generator.

[0009] Segmentation step: Use the trained multi-scale feature fusion segmentation network to segment the medical image to be processed.

[0010] Preferably, the multi-scale feature fusion segmentation network includes four residual modules, and the corresponding DAMMs include four. The partial compressed feature maps are the compressed feature maps output by the corresponding DAMMs of the last three residual modules.

[0011] Preferably, each DAMM includes a first channel attention module, a first spatial attention module, and a 1x1 convolution for compressing the number of channels. The first channel attention module is connected to the input feature map, and after processing, outputs a feature map with refined channels to the spatial attention module. The spatial attention module outputs a feature map with refined spatial attention mechanism to the 1x1 convolution for channel number compression to output the compressed feature map.

[0012] Preferably, in the first channel attention module, first, the input feature map F ∈ R C×H×W aggregates spatial information through average pooling and max pooling operations, and outputs two different feature maps containing spatial information and represents the average channel pooling feature, represents the max channel pooling feature; then and are fed into the MLP network, and the outputs and are and element-wise added to obtain the final channel attention map Mc is normalized by the sigmoid function to obtain the attention weight ω for each channel c ; the input feature map is weighted for each channel through the attention weight of each channel, and finally the feature map with refined channels is obtained wherein, represents element-wise addition; R represents real numbers, C represents the number of channel categories; H represents height; W represents width.

[0013] Preferably, in the first spatial attention module, for the feature map F c output by the first channel attention module, first, average pooling and max pooling operations are respectively performed on F c to obtain the average spatial pooling feature map and the max spatial pooling feature map Then, these two feature maps are concatenated () in the channel dimension to obtain a fused feature map For a 5x5 convolution is used to reduce the number of channels to 1, thereby obtaining the spatial attention map Ms; the spatial attention map Ms is normalized by the sigmoid function to obtain the attention weight ω for each space s ; through ω s the feature map F c is weighted for each space, and the feature map with refined channel spatial attention mechanism is obtained F cs is compressed through a 1x1 convolution to obtain a compressed feature map; wherein, represents element-wise addition.

[0014] Preferably, the MFFM includes a second channel attention module and a second spatial attention module; let the two features to be fused be F X and F Y , F X∈R C×H×W , F Y ∈R C×H×W ; First, perform element-wise addition on F X and F Y to obtain Then, input F Z into the second-channel attention module to obtain the channel attention weight ω c , and the feature map after channel refinement where represents element-wise multiplication; then is input into the second spatial attention module to obtain the spatial attention weight ω s , and through the spatial attention weight ω s obtain the fused feature map of F X and F Y Fuse the newly obtained fused feature map with the next feature map until all feature maps are fused.

[0015] Preferably, the hybrid segmentation loss L seg is expressed as follows:

[0016]

[0017] where L ce represents the cross-entropy loss; L Dice represents the Dice loss; N represents the total number of pixels of the sample X; C represents the number of channel categories; for the sample X = {(x 1 , y 1 ), (x 2 , y 2 ), …, (x N , y N )}, x i represents the value of the sample X at the i-th pixel, y i represents the label of the sample X at the i-th pixel, y i,c represents the binary label of the sample X at the i-th pixel with respect to the channel category c, represents the predicted binary label of the sample X at the i-th pixel with respect to the channel category c, represents the predicted probability of the sample X at the i-th pixel with respect to the channel category c; ω c is the weight factor of the channel category, W is the model weight, is the L2 norm of the model weight; λ 1 and λ 2 are empirical parameters.

[0018] Preferably, the first discriminator loss and the adversarial loss between the first discriminator and the generator ​It is expressed as follows:

[0019]

[0020] The second discriminator loss and the adversarial loss between the second discriminator and the generator

[0021]

[0022] where E represents the mathematical expectation; z s represents generating the aggregated feature x using the source domain data s →f s ,z t represents generating the aggregated feature x using the target domain data t →f t ,u s represents generating the semantic feature x using the source domain data s →o s ,u t represents generating the predicted probability map x using the target domain data t →o t ,c 1 represents inputting the source domain aggregated feature f s or the target domain aggregated feature f t into the first discriminator, c 2 represents inputting the source domain semantic feature o s or the target domain semantic feature o t into the second discriminator, and respectively represent the distribution sets of f s 、f t 、o s and o t .

[0023] Preferably, the overall loss L total during the training process is expressed as follows:

[0024]

[0025] where L seg represents the mixed segmentation loss; represents the first discriminator loss; represents the second discriminator loss; represents the adversarial loss between the first discriminator and the generator; represents the adversarial loss between the second discriminator and the generator; λ 3 、λ 4 and λ 5 are empirical parameters.

[0026] On the other hand, an unsupervised domain adaptation medical image segmentation device based on multi-scale features includes:

[0027] A multi-scale feature fusion segmentation network construction module for constructing a multi-scale feature fusion segmentation network based on dual attention; the multi-scale feature fusion segmentation network includes a number of sequentially connected residual modules, dual attention mechanism modules DAMM respectively connected to each residual module, and multi-scale feature fusion modules MFFM; the convolutional layer performs initial feature extraction on the input source domain / target domain medical image, and then extracts high-level image features through each residual module to generate feature maps of different sizes; each feature map is fed into a DAMM module to generate compressed feature maps of different sizes after fine-grained compression of the features; the compressed feature maps of different sizes are aligned to the same size through upsampling; all the aligned compressed feature maps are input into an MFFM module for feature fusion, and then a probability map for per-pixel prediction is obtained through a 1x1 convolution and a softmax layer to obtain semantic features; a part of the aligned compressed feature maps are input into another MFFM module for feature fusion to obtain aggregated features;

[0028] A discriminator construction module for using the multi-scale feature fusion segmentation network as a generator to construct a first discriminator and a second discriminator; using the first discriminator to align the aggregated features of the source domain medical image and the target domain medical image, and using the second discriminator to align the semantic features of the source domain medical image and the target domain medical image;

[0029] A training module for training based on the generator, the first discriminator, and the second discriminator to obtain a trained multi-scale feature fusion segmentation network; during the training process, the hybrid segmentation loss of the generator is optimized using source domain data; the source domain aggregated features and the target domain aggregated features are passed to the first discriminator to optimize the first discriminator loss and the adversarial loss between the first discriminator and the generator; the source domain semantic features and the target domain semantic features are passed to the second discriminator to optimize the second discriminator loss and the adversarial loss between the second discriminator and the generator;

[0030] A segmentation module for segmenting the medical image to be processed using the trained multi-scale feature fusion segmentation network.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) The multi-scale feature fusion strategy based on the dual attention mechanism of the present invention improves the segmentation accuracy by combining low-level features and high-level features;

[0033] (2) The adversarial learning method for shortening domain shift of the present invention aligns the target domain to the source domain by aligning the aggregated features and the semantic features, further improving the generalization ability of the model in the target domain;

[0034] (3) Through unsupervised domain adaptation experiments conducted on fundus image datasets and prostate image datasets, compare with some existing advanced unsupervised domain adaptation methods in terms of performance and visualization images to verify the effectiveness of the method proposed in the present invention;

[0035] (4) Through ablation experiments and specific alignment strategy comparison experiments, further verify that the method of the present invention can effectively solve the problem of the unavailability of target domain labels during the model migration process in medical image segmentation. Description of the Drawings

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0037] Figure 1 Schematic flowchart of the unsupervised domain adaptation medical image segmentation method based on multi-scale features for the embodiments of the present application;

[0038] Figure 2 Schematic diagram of the overall model structure of the medical image segmentation model of the unsupervised domain adaptation medical image segmentation method based on multi-scale features for the embodiments of the present application;

[0039] Figure 3 Schematic diagram of the structure of the multi-scale feature fusion segmentation network based on dual attention for the embodiments of the present application;

[0040] Figure 4 Schematic diagram of the structure of the dual attention mechanism module for the embodiments of the present application;

[0041] Figure 5 Schematic diagram of the structure of the multi-scale feature fusion module for the embodiments of the present application;

[0042] Figure 6 Segmentation results of the method of the embodiments of the present application and the method of the prior art in fundus images;

[0043] Figure 7 Segmentation results of the method of the embodiments of the present application and the method of the prior art in prostate images;

[0044] Figure 8 Schematic diagram of the unsupervised domain adaptation medical image segmentation device based on multi-scale features for the embodiments of the present application. Detailed Embodiments

[0045] To make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Apparently, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0046] As Figure 1 shown, an unsupervised domain adaptation medical image segmentation method based on multi-scale features (Unsupervised Domain Adaptation Medical Image Segmentation Method Based on Multi-scale Feature Adversarial Learning, MFAD) provided in this embodiment includes the following steps:

[0047] S101. Step of constructing a multi-scale feature fusion segmentation network: construct a multi-scale feature fusion segmentation network based on dual attention; the multi-scale feature fusion segmentation network includes a number of sequentially connected residual modules, a dual attention mechanism module DAMM and a multi-scale feature fusion module MFFM respectively connected to each residual module; the convolutional layer performs initial feature extraction on the input source domain / target domain medical image, and then extracts high-level image features through each residual module to generate feature maps of different sizes; each feature map is fed into a DAMM module to generate compressed feature maps of different sizes after fine-grained compression of the features; the compressed feature maps of different sizes are aligned to the same size through upsampling; all the aligned compressed feature maps are input into an MFFM module for feature fusion, and then a probability map for per-pixel prediction is obtained through a 1x1 convolution and a softmax layer to obtain semantic features; a part of the aligned compressed feature maps are input into another MFFM module for feature fusion to obtain aggregated features;

[0048] S102. Step of constructing a first discriminator and a second discriminator based on a generator: use the multi-scale feature fusion segmentation network as a generator, and construct a first discriminator and a second discriminator; use the first discriminator to align the aggregated features of the source domain medical image and the target domain medical image, and use the second discriminator to align the semantic features of the source domain medical image and the target domain medical image;

[0049] S103. Generator and discriminator training steps: Based on the generator, the first discriminator, and the second discriminator, train to obtain a trained multi-scale feature fusion segmentation network. During the training process, use the source domain data to optimize the hybrid segmentation loss of the generator. Transmit the source domain aggregated features and the target domain aggregated features to the first discriminator to optimize the first discriminator loss and the adversarial loss between the first discriminator and the generator. Transmit the source domain semantic features and the target domain semantic features to the second discriminator to optimize the second discriminator loss and the adversarial loss between the second discriminator and the generator.

[0050] S104. Segmentation step: Use the trained multi-scale feature fusion segmentation network to segment the medical image to be processed.

[0051] As Figure 2 shown, the multi-scale feature fusion segmentation network based on dual attention (Multi-scale feature fusion strategy based on dual attention, MFFDA) in this embodiment removes the first max pooling layer, the last average pooling layer, and the fully connected layer on the basis of the original ResNet34, only retains the first 7×7 convolutional layer, and four designed residual modules, including residual blocks with the numbers of 4, 6, 3, and 3 respectively. Each residual block consists of two 3x3 convolutional layers, and the downsampling rate is designed to be 16. A total of 4 times of downsampling are performed, and each time the width and height of the input feature map are changed to half of the original. For the input medical image, four-scale feature maps are generated. However, the generated feature maps may not be able to effectively extract all features of the medical image. Therefore, this embodiment introduces a dual attention mechanism module (Dual Attention Mechanism Module, DAMM) to perform refined compression processing on each extracted feature. Finally, the four refined feature maps are upsampled to the same size, and then they are fed into the multi-scale feature fusion module (Multi-scale Feature Fusion Module, MFFM), fused from low layer to high layer into an aggregated feature, and finally a segmentation result (medical image segmentation mask) is generated through a softmax layer.

[0052] Specifically, as Figure 3As shown, in the encoding stage, the input medical image is first forwarded to the first convolutional layer for initial feature extraction to capture low-level image features. The convolutional layer is designed as a 7x7 convolution with a stride of 2. For a medical image input of size 3x512x512, a feature map of size 64x256x256 is generated. Then, through four residual modules, high-level image features are further extracted, and finally, four feature maps of sizes 64x256x256, 128x128x128, 256x64x64, and 512x32x32 are generated. Each feature map is fed into the DAMM module, and after the features are refined and compressed to 16 channels, compressed feature maps of sizes 16x256x256, 16x128x128, 16x64x64, and 16x32x32 are generated. In the decoding stage, the feature maps of different sizes are aligned to the same size of 16x512x512 through upsampling, and then the compressed feature maps of these four scales are input into the MFFM module. Finally, a refined aggregated feature map after analyzing and fusing the feature information of each scale is obtained. Subsequently, a 1x1 convolution is used to generate a vector of Cx512x512, where the number of channels is the number of segmentation categories C (C is defined as the number of channel categories). Finally, a softmax layer is used to obtain the probability of per-pixel prediction.

[0053] As Figure 4 shown, the DAMM module contains a channel attention module, a spatial attention module, and a 1x1 convolution for compressing the number of channels. In the channel attention module, for the input feature map F ∈ R C×H×W , first, it is respectively passed through average pooling and max pooling operations to aggregate spatial information. At this time, two different feature maps containing spatial information are output and denotes the average channel pooling feature, denotes the max channel pooling feature. Then, through and , a channel attention map M c is generated. Specifically, a multi-layer perceptron (MLP) network is designed. This network consists of two fully connected (FC) layers and a hidden layer. and are fed into the MLP network. The MLP processes and independently and shares weights at the same time. The output and obtained through the MLP are added element-wise to obtain the final channel attention map After the above operations, the channel attention can consider both the average value and the maximum value information in the channel, thereby generating a more detailed and comprehensive channel attention signal. The channel attention map Mc After normalization by the sigmoid function, the attention weight ω of each channel is obtained. c After obtaining ω c After that, the weighted operation of each channel is performed on the original feature map through the attention weight of each channel, and finally the feature map with refined channels is obtained. Among them, denotes the element-wise multiplication of ω c broadcast to the size of R C×H×W and F.

[0054] After obtaining the feature map F with refined channels c then F c needs to be fed into the spatial attention module to obtain a more refined feature. For the input feature F c , first, average pooling and max pooling operations are respectively performed on F c along the channel dimension C to obtain the average spatial pooling feature map and the max spatial pooling feature map Then, these two feature maps are concatenated () in the channel dimension to obtain a fused feature map To capture the spatial context information and obtain the spatial attention map M s , for a 5x5 convolution is used, which reduces the number of channels to 1, thereby obtaining the spatial attention map M s , M s ∈R 1×H×W . Similarly, the spatial attention map M s is normalized by the sigmoid function to obtain the attention weight ω of each space s . ω s can enable the model to focus on more important spatial regions when processing features. Finally, through ω s the feature map F adjusted by the channel attention module c is weighted for each space to obtain the feature map refined by the channel-spatial attention mechanism Then, F cs is compressed to 16 channels through a 1x1 convolution to obtain the final compressed and refined feature map. This step is to align the channels of feature maps of different sizes to 16 channels, facilitating subsequent multi-scale feature fusion operations.

[0055] After obtaining the feature maps of each scale through the DAMM module, it is necessary to fuse these feature maps of different scales through the MFFM module.

[0056] Such as Figure 5As shown, the MFFM module fuses two features through a channel attention module (the second channel attention module) and a spatial attention module (the second spatial attention module) such as in the DAMM module. The way the MFFM module obtains the aggregated feature that fuses all-scale feature information is to repeatedly fuse the newly obtained fused feature map with the next feature map until all feature maps are fused. Let the two features to be fused be F X and F Y , F X , F Y ∈R C×H×W . First, perform an element-wise addition on F X and F T to obtain Then input F Z into the channel attention module to obtain the channel attention weight ω c . Different from the DAMM module, the feature map after channel refinement where represents element-wise multiplication. Then input into the spatial attention module to obtain the spatial attention weight ω s . Through the spatial attention weight ω s , obtain the final fused feature map of F X and F Y This enables the fused feature F to perform a weighted average between F Z and F X and F Y . The fused feature map is fused with the feature map to be fused until there is no feature map to be fused, and the final aggregated feature map is obtained.

[0057] It should be noted that the MFFDA in this embodiment includes two MFFM modules. One MFFM module analyzes and fuses the compressed feature maps of four scales and generates a vector of Cx512x512 through 1x1 convolution. Finally, a softmax layer is used to obtain the per-pixel prediction probability map to obtain the semantic feature. The other MFFM module obtains the aggregated feature through analyzing and fusing the compressed feature maps of three scales of the last three aligned layers.

[0058] Through the proposed MFFDA, supervised learning of multi-scale feature fusion is performed using source domain data and labels to establish a mapping from the input image to the label space. To optimize the classification accuracy of each pixel of the model and improve the segmentation quality, especially for the case of unbalanced classes. In this embodiment, a hybrid segmentation loss L ce combining the cross-entropy loss L Dice and the Dice loss L seg is used to optimize the MFFDA as follows.

[0059]

[0060] Among them, N represents the total number of pixels of the sample X; for the sample X = {(x 1 , y 1 ), (x 2 , y 2 ), …, (x N , y N )}, x i represents the value of the sample X at the i-th pixel, y i represents the label of the sample X at the i-th pixel, and y i,c represents the binary label of the sample X at the i-th pixel with respect to the channel category c, represents the predicted binary label of the sample X at the i-th pixel with respect to the channel category c, represents the predicted probability of the sample X at the i-th pixel with respect to the channel category c; ω c is the weight factor of the channel category, which is inversely proportional to the sample frequency of the category; W is the model weight, is the L2 norm of the model weight, which is used for regularization to prevent overfitting; λ 1 and λ 2 are empirical parameters. The segmentation network of the source domain is optimized by minimizing the mixed segmentation loss L seg composed of multi-class cross-entropy loss and Dice coefficient loss.

[0061] The aggregation-semantic feature alignment method based on adversarial learning will be described as follows.

[0062] In the medical image segmentation task, the segmentation network based on multi-scale feature fusion comprehensively utilizes low-level and high-level features to ensure that the model can fully capture the subtle information and abstract features in the image, thereby achieving accurate segmentation. However, due to cross-domain, there are significant differences between the features of the target domain and the source domain. Therefore, domain-invariant features of the source domain and the target domain (i.e., the segmentation structures of different domains share features) are learned through feature alignment. Based on this, this embodiment proposes an aggregation-semantic feature alignment method based on adversarial learning (Adversarial Global Local Feature Alignment strategy, AGLA) starting from the idea of GAN.

[0063] In medical image segmentation tasks, although the images generated by different devices may have significant differences in low-level features such as gray-scale distribution and texture, the high-level semantic information, such as the shape, size, and relative position of the segmentation structure, should be similar. Therefore, the main cause of domain shift is low-level features, and the methods commonly used in existing technologies are to align features at all levels. The problem with this all-feature alignment strategy is that if all levels of features, whether high-level or low-level, are uniformly aligned without discrimination, it may lead to overemphasis on the changes in low-level features that are not highly relevant to the task, resulting in the model overfitting to these irrelevant changes and ignoring the high-level semantic information that should actually be concerned. Therefore, in this embodiment, the high-level features of the source domain and the target domain are aligned while ignoring the low-level feature alignment, reducing the visual differences caused by different imaging conditions (such as different imaging devices or settings) to promote the consistency of the high-level feature distributions between the source domain and the target domain. However, considering the hierarchical structure of Resnet34, the features of a certain layer depend on the activation of its previous layer and will affect the subsequent layers. If only the features of a certain layer obtained from MFDA are aligned, the potential spatial alignment may be unstable. Therefore, this problem is solved by aligning the aggregated features of the last three layers of compressed feature fusion of MFDA. In this way, compared with aligning the features of all layers, it is not only easier to transfer and adapt between different domains, but also requires less computational resources to process these features because of the low dimension of the alignment. Moreover, the medical image prediction probability map generated by MFDA not only contains rich semantic knowledge but also has a low dimension. Aligning such high-level semantic features of the prediction probability map not only has a small overhead but also can perform domain alignment based on the geometric and semantic attributes of the segmentation structure. Therefore, the idea of feature alignment in this embodiment is to align the aggregated features and semantic features of the source domain and the target domain respectively.

[0064] The AGLA method proposed in this embodiment aligns the features of the source domain and the target domain by designing two discriminators and a generator. The generator G is the previously proposed MFFDA model, which is responsible for generating aggregated and semantic features. The first discriminator D 1 is used to align the distributions of the aggregated features (f s and f t ), and the second discriminator D 2 is used to align the distributions of the semantic features (o s and o t ). Among them, f s is the aggregated feature of the source domain data, f t is the aggregated feature of the target domain data, o s is the semantic feature (prediction probability map) of the source domain data, and o t is the semantic feature (prediction probability map) of the target domain data. The discriminators D 1 and D 2Designed based on CGAN, it is implemented through 5 convolutional blocks. Each convolutional block contains a convolutional layer, a Leaky-ReLU activation function, a batch normalization layer, and a Sigmoid function as the last layer. Leaky-ReLU is used as the activation function, and convolution is used for downsampling inside the discriminator instead of the common practice of using ReLU and max pooling to avoid instability caused by sparse gradients. During the training process of GAN, it is usually necessary to handle the dynamic confrontation between the generator and the discriminator. In the traditional GAN framework, this confrontation is implemented by maximizing and minimizing the Jensen-Shannon divergence. However, this method may lead to training instability, especially when the discriminator is too excellent, resulting in the vanishing gradient of the generator. To overcome these problems, this embodiment uses the Wasserstein distance as the loss function to improve the training process. The Wasserstein distance provides a method to measure the difference between two probability distributions, with better geometric intuition and mathematical properties. It quantifies the difference between these two distributions by calculating the "least amount of work" required to "transfer" one distribution into another.

[0065] First, use the discriminator D 1 to align the aggregated feature distributions, and input the aggregated features f s and f t into D 1 to estimate the Wasserstein distance between the distributions of the aggregated features f s and f t to determine whether the aggregated features are aligned. The task of the discriminator D 1 is to estimate from which domain these features come. Then input the predicted probability maps o s and o t obtained through G into the discriminator D 2 to align the semantic feature distributions, and estimate the Wasserstein distance between the distributions of the predicted probability maps o s and o t to determine whether the semantic features are aligned. The task of the discriminator D 2 is to estimate from which domain these predicted probability maps come. The purpose in this process is to align the aggregated features and semantic features by reducing the Wasserstein distance, making the target domain detection results gradually become more similar to the source domain. The task of the generator G is to make the feature / predicted probability map distributions close to the source domain, and the tasks of the discriminators D 1 and D 2 are to minimize the Wasserstein distance of the aggregated / semantic feature distributions of the generator G between the source domain and the target domain, so as to discriminate whether the features come from the source domain or the target domain.

[0066] Since in AGLA, it is desired that the distribution generated by the generator G is as close as possible to the data distribution of the source domain, which is achieved by minimizing the Wasserstein distance, the optimization objective of the discriminator is to minimize its ability to distinguish between source domain data and target domain data, thus guiding the generator to improve the generated target domain data distribution to make it closer to the source domain data distribution. Therefore, the task of the discriminator is to minimize the Wasserstein distance, rather than maximizing the distance as in traditional GANs. In Aggregated Feature Alignment, the first discriminator loss and the adversarial loss between the first discriminator and the generator are expressed as follows:

[0067]

[0068] In Semantic Feature Alignment, the second discriminator loss and the adversarial loss between the second discriminator and the generator

[0069]

[0070] where E represents the mathematical expectation; z s denotes generating aggregated feature x from source domain data s →f s ,z t denotes generating aggregated feature x from target domain data t →f t ,u s denotes generating semantic feature x from source domain data S →o s ,u t denotes generating predicted probability map x from target domain data t →o t ,c 1 denotes inputting the source domain aggregated feature f s or the target domain aggregated feature f t into the first discriminator, c 2 denotes inputting the source domain semantic feature o s or the target domain semantic feature o t into the second discriminator, and respectively represent the distribution sets of f s 、f t 、o s and o t .

[0071] Therefore, the overall loss L total of MFAD proposed in this embodiment is obtained by summing up all the following losses, as shown in the following formula:

[0072]

[0073] During the entire training process, first, the segmentation loss L of the detection network G is optimized using the source domain data seg , and then G is used to predict the target domain data. Then, the aggregated features (f s and f t ) and the predicted probability maps (o s and o t ) are passed to the discriminators D 1 and D 2 to optimize the discriminator losses and Finally, the adversarial losses and are calculated. As G is continuously updated, the Wasserstein distance becomes smaller and smaller, making the target domain detection results gradually become more similar to the source domain, and the domain gap also shrinks more and more.

[0074] The unsupervised domain adaptation medical image segmentation method based on multi-scale features of this embodiment will be verified through experiments as follows, including the following steps.

[0075] (1) Dataset.

[0076] In the field of medical image segmentation, the medical image database is the cornerstone of medical image segmentation. Due to the difficulty of data acquisition and the high requirements for data privacy, it is impossible to provide sufficient samples for medical image segmentation research. Therefore, the research and development of medical image segmentation are challenged by small data samples. This embodiment mainly uses publicly available fundus image datasets and prostate image datasets for experiments. There are mainly three fundus image datasets, namely Drishti-GS, RIM-ONE_r3, and REFUGE challenge training sets, and three prostate image datasets, namely NCI, I2CVB, and PROMISE12.

[0077] (2) Data preprocessing

[0078] Specifically, if the entire medical image is directly used for training, there are the following problems: The background information of the original image, such as the edge region and non-target structures, may introduce noise, affecting the model's ability to identify key features and resulting in a decrease in the model's resolution. Therefore, this paper introduces the concept of ROI (region of interest). The ROI image reduces background interference by focusing on the segmentation region, enabling the model to more attentively learn the details of these key regions, thereby improving the accuracy of segmentation.

[0079] Due to the limitation of the small data volume of the medical image dataset, the model directly trained with the medical image dataset is prone to problems such as overfitting and poor generalization ability. To improve the model training effect, this embodiment adopts a series of data augmentation strategies to simulate different visual effects and expand the training dataset. By image inversion, rotation, and adjustment of brightness and contrast, the diversity of samples can be artificially increased, which helps the model to have better generalization ability in actual applications. For the images after these enhancement processes, for fundus images, they are fixed and scaled to a size of 512×512, and for the prostate image dataset, each sample is adjusted to a size of 384×384 in the axial plane, ensuring the consistency of data during the training process. At the same time, random flipping and normalization processing are performed on the images, so that the pixel values of each channel have a unified average value and standard deviation, further reducing the risk of overfitting during the model training process. Through these preprocessing steps, the number of images in the training dataset increases and the quality improves, ultimately solving the limitation of the small data volume of the medical image dataset that leads to difficulties in training the model.

[0080] (3) Evaluation Metrics

[0081] To objectively evaluate the segmentation performance of the target domain model on the medical image dataset in the experiment, this embodiment selects two widely used standard metrics in image segmentation evaluation: Dice coefficient (Dice) and average surface distance (ASD). These metrics jointly provide a comprehensive evaluation for the quantification of the segmentation effect. The Dice coefficient (Dice) compares the similarity between the predicted segmentation result and the true segmentation label. It calculates the ratio of twice the intersection area of the predicted and true segmentations to the sum of their respective areas. The value range of the Dice coefficient is from 0 to 1, and the higher the value, the closer the segmentation result is to the true situation, that is, the better the segmentation effect. The calculation formula is as follows.

[0082]

[0083] Where A is the set of predicted segmentation results, B is the set of true segmentation results, |A∩B| represents the number of elements in the intersection of sets A and B, |A| represents the number of elements in set A, and |B| represents the number of elements in set B.

[0084] The average surface distance (ASD) measures the average distance between the predicted segmentation and the true segmentation surface. It calculates the average shortest distance from all points on the predicted segmentation to the true segmentation surface, and the average shortest distance from all points on the true segmentation to the predicted segmentation surface. The smaller the ASD value, the closer the segmentation result is to the true situation, that is, the higher the segmentation quality. The calculation formula is as follows.

[0085]

[0086] Among them, X and Y respectively represent the point sets on the predicted segmentation surface and the ground truth segmentation surface, and ||x - y|| represents the Euclidean distance between points x and y. ASD calculates the average shortest distance between all pairs of points from X to Y and from Y to X.

[0087] (4) Experimental environment and training configuration information

[0088] The software and hardware experimental environment in this experiment is shown in Table 1.

[0089] The detailed training parameters of this experiment are as follows:

[0090] Use the Adam optimizer to optimize the training of the MFDA model, with momentum set to 0.9 and 0.99, the learning rate set to 0.001, batch set to 10, λ 1 set to 0.8, λ 2 set to 0.2. In the unsupervised domain adaptation adversarial total loss, λ 3 set to 1, λ 4 set to 0.5, λ 5 set to. The slope of the Leaky - RrLU of discriminator D 1 and D 2 is set to 0.25. Use to optimize the discriminator, with a learning rate of 0.0001.

[0091] Table 1 Configuration of the software and hardware experimental environment;

[0092]

[0093] (5) Experimental design

[0094] The experiments in this embodiment are mainly to explore the following questions:

[0095] First, in the UDA medical image segmentation scenario, what is the accuracy of the MFAD method proposed by the present invention, and whether it has advantages compared with other medical image segmentation methods, and the results can be intuitively seen through the visualized segmentation results.

[0096] Second, in the UDA medical image segmentation scenario, the impact of the MFDA and AGLA modules proposed by the present invention on the model performance. In this embodiment, under the condition of using the same dataset and training process, the effectiveness of each module is verified through four groups of ablation experiments.

[0097] Third, in the medical image segmentation scenario, explore the effectiveness of the alignment method. In this embodiment, under the condition of using the same dataset and training process, four groups of experiments, namely full-feature alignment, fusion feature alignment of the first two layers, fusion feature alignment of the first three layers, fusion feature alignment of the last two layers, and fusion feature alignment of the last three layers proposed in this embodiment, are designed to explore the effectiveness of the feature alignment strategy. Then, a set of comparative experiments are designed to explore the superiority of the aggregation-semantic feature alignment proposed in this embodiment compared with the methods of only aligning aggregation features and only aligning semantic features.

[0098] (6) Experimental Results and Analysis

[0099] (6.1) Experimental Analysis of Fundus Image Segmentation

[0100] Specifically, publicly available fundus image datasets are mainly used for experiments, including Drishti-GS, RIM-ONE_r3, and REFUGE challenge training sets, three fundus image datasets. To verify the segmentation performance of the model on the target domain dataset when no target domain labels are available, two domain transfer situations are designed, including (1) setting the challenge training set as the source domain dataset and the dataset as the target domain dataset; (2) setting the challenge training set as the source domain dataset and the dataset as the target domain dataset. To verify the effectiveness of the model, an algorithm comparison experiment is conducted, comparing with other unsupervised domain adaptation methods, to test whether the MFAD method proposed in this embodiment achieves the best results compared with similar methods. The BEAL method, AdvEnt method, ISFA method, pOSAL method, and SIFA method are used as comparison algorithms. Among them, BEAL is a model that solves source-unlabeled domain adaptation through adversarial learning and boundary prediction, AdvEnt is a model that solves source-unlabeled domain adaptation through source domain and target domain entropy consistency, the ISFA method is a method that uses GAN to generate synthetic images through feature alignment, the method is a method that aligns features with PatchGAN, and the pOSAL method is a method that aligns images and features collaboratively. Among them, adding Baseline represents a direct transfer method that directly predicts the target domain results using the baseline Resnet34 model trained from the source domain data. For fairness, based on the evaluation criteria for measuring the segmentation effect mentioned above, the average value of the five running results of each method is taken.

[0101] Table 2 shows the experimental results of different algorithms on the REFUGE challenge training set → Drishti-GS dataset. As can be seen from Table 2, the experimental results of different algorithms on the transfer from the REFUGE challenge training set to the Drishti-GS dataset are presented. Although the gap with several other sub-optimal models is small, the MFAD proposed in this embodiment performs the best. The method not only obtains the highest Dice coefficients for the optic cup and optic disc, but also has the lowest ASD distance. The method achieves a coefficient of 86.82 in the detection of the optic cup, which is only 0.07 percentage points higher than the sub-optimal ISFA method (86.75), but 4.07 percentage points higher than the weakest-performing AdvEnt method (82.75). The ASD for the optic cup is 10.81, which is not only lower than the score of the best-performing BEAL method (11.32) among other methods, a decrease of 0.51, but also significantly better than the worst-performing pOSAL method (14.18), with a gap of 3.37. For the Dice coefficient of the optic disc, the MFAD method also reaches the highest score of 96.81, 0.13 percentage points higher than the sub-optimal ISFA method (96.68), and significantly exceeds the worst-performing pOSAL method (94.51) among all algorithms, with a gap of 2.3 percentage points. In the ASD of the optic disc, the MFAD method also has the lowest score of 4.08, 0.2 lower than the closest BEAL method (4.28), and 2.04 lower than the worst method pOSAL (6.12).

[0102] Table 2 Comparison of the results of different algorithms on the REFUGE challenge training set → Drishti-GS dataset;

[0103]

[0104] Table 3 shows the experimental results of different algorithms on the REFUGE challenge training set → RIM-ONE_r3 dataset. It can be seen from Table 3 the experimental results of different algorithms when migrating from the REFUGE challenge training set to the RIM-ONE_r3 dataset. It can be seen from Table 3 and Table 2 that the overall effect of the model when migrating to the RIM-ONE_r3 dataset is worse than that when migrating to the Drishti-GS dataset. This is because the domain shift of the model on the RIM-ONE_r3 dataset is larger than that on the Drishti-GS dataset, resulting in a worse segmentation effect. However, the overall effect of the MFAD method proposed in this embodiment is still the highest in the experiment of migrating the REFUGE challenge training set to the RIM-ONE_r3 dataset. The Dice coefficient of the MFAD method for the optic cup reaches the highest of 81.58, which is 0.31 percentage points higher than the ISFA method and 3.59 percentage points higher than the worst-performing AdvEnt method. In the Dice coefficient of the optic disc, the MFAD method also reaches the highest score of 91.16, exceeding the second-best ISFA method by 0.43 percentage points and being 4.64 percentage points higher than the worst-performing pOSAL method. For the evaluation of ASD, the MFAD method also performs the best in terms of the prediction boundary precision of the optic cup and optic disc, reaching 7.12 and 9.32 respectively. In the prediction of the optic cup, its performance is 0.45 lower than that of the AdvEnt method, and in the prediction of the optic disc boundary, it is also 0.2 lower than the score of the ISFA method.

[0105] Table 3 Comparison of the results of different algorithms on the REFUGE challenge training set → RIM-ONE_r3 dataset;

[0106]

[0107] (6.2) Experimental analysis of prostate image segmentation

[0108] To further verify the performance, this experiment verifies on three prostate image datasets of NCI-ISBI 2013, I2CVB, and PROMISE12. To verify the segmentation performance of the model on the target domain dataset when no target domain labels are available, 2 domain transfer situations are designed, including (1) the I2CVB dataset as the source domain and the NCI-ISBI 2013 as the target domain, and (2) the I2CVB dataset as the source domain and the PROMISE12 as the target domain. The algorithm comparison is still the same as before, taking the BEAL method, AdvEnt method, ISFA method, pOSAL method, and SIFA method as the comparison algorithms. Among them, Baseline is also added to represent the result of direct migration. Combining the evaluation criteria for measuring the segmentation effect mentioned above, the average value of the five running results of each method is taken.

[0109] Table 4 shows the experimental results of different algorithms on the I2CVB dataset → NCI-ISBI 2013 dataset. It can be seen that both the Dice score and the ASD distance of the MFAD method proposed in this embodiment are significantly better than other methods. The Dice score is 1.18 higher than the sub-optimal BEAL method and 5.78 higher than the worst AdvEnt method. In terms of distance, it is 1.05 lower than the sub-optimal ISFA method and 3.76 lower than the worst AdvEnt method.

[0110] Table 4 Comparison of the results of different algorithms on the I2CVB dataset → NCI-ISBI 2013 dataset;

[0111] Method Dice ASD Baseline 78.91 9.14 BEAL 90.51 4.35 AdvEnt 85.91 6.90 ISFA 88.90 4.19 pOSAL 86.10 5.92 SIFA 87.21 6.14 MFAD 91.69 3.14

[0112] Table 5 shows the experimental results of different algorithms on the I2CVB dataset → PROMISE12 dataset. The scores and distances of the method proposed in this embodiment are also significantly better than other methods. The Dice score is 1.6 higher than the sub-optimal ISFA method, and the distance is 1.08 lower than the sub-optimal ISFA method.

[0113] Table 5 Comparison of the results of different algorithms on the I2CVB dataset → ROMISE12 dataset;

[0114] Method Dice ASD Baseline 75.81 13.91 BEAL 85.14 7.15 AdvEnt 84.52 5.13 ISFA 85.58 4.92 pOSAL 82.16 8.16 SIFA 83.01 7.26 Method Dice ASD MFAD 87.18 3.84

[0115] It can be seen that in the case of domain adaptation from the I2CVB dataset to the PROMISE12 dataset, the segmentation performance of all methods, including the MFAD method, is worse than that in the case of domain adaptation from the I2CVB dataset to the NCI-ISBI 2013 dataset. It can be seen from the Baseline that the scores and distances of the model trained on the I2CVB dataset directly transferred to the NCI-ISBI 2013 dataset are lower than those directly transferred to the ROMISE12 dataset, which indicates that the domain shift from the I2CVB dataset to the ROMISE12 dataset is larger than that to the NCI-ISBI 2013 dataset. However, the scores and distances of the method proposed in this embodiment when transferred to the ROMISE12 dataset are close to those when transferred to the NCI-ISBI 2013 dataset. This indicates that the method proposed in this embodiment can effectively align features and shorten the domain shift.

[0116] Figure 6 and Figure 7The experimental results of different algorithms in the case of SFUDA for fundus images and prostate images are shown respectively. It can be seen from the figure that the segmentation results obtained by the directly migrated Baseline method are very poor in terms of shape due to domain shift, the boundaries are rough, and there are even small areas of incorrect segmentation. The other comparison methods have significant improvements in terms of shape and boundaries, but there are still some deficiencies. For example, the segmentation boundary of the SIFA method is still very rough, and the shape segmentation effect of the AdvEnt method is not good. The MFAD method proposed in this embodiment not only has smooth segmentation edges but also is very close to the true label in terms of shape. At the same time, it can be seen that the segmentation effect of the optic disc is better than that of the optic cup, because the segmentation of the optic cup is often more difficult than that of the optic disc, and the segmentation effect of the MFAD method on the optic cup is also very close to the true label. This is because the MFAD method can fully align the shape semantic features and low-level boundary features of the source domain and the target domain, thus ensuring that the segmentation results are close to the true label in terms of shape and boundary.

[0117] Based on the above comparative experiments, the MFAD method of this embodiment performs well in the unsupervised domain adaptation scenario on both the fundus image dataset and the prostate image dataset, and has a certain improvement in terms of the Dice score and the ASD distance compared with other methods. It can also be seen from the visualized segmentation images that the MFAD method has superiority in segmentation results. It shows that the MFAD method makes full use of the advantages of multi-scale features and adversarial learning, and effectively solves the UDA problem in medical image segmentation.

[0118] (6.3) Ablation experiment

[0119] In this experiment, ablation experiments were conducted in the UDA case of the REFUGE challenge training set → Drishti-GS dataset to verify the effectiveness of the MFDA strategy (multi-scale feature fusion strategy based on dual attention) and the AGLA method (aggregation-semantic feature alignment method based on adversarial learning) proposed in this embodiment.

[0120] Table 6 Comparison of results in the ablation experiment for the REFUGE challenge training set → Drishti-GS dataset;

[0121]

[0122] Table 6 shows the segmentation results on the REFUGE challenge training set → Drishti-GS dataset under different strategies. It can be seen from the table that compared with the directly transferred Baseline method, both the MFDA strategy and the AGLA method have greatly improved the segmentation effect. This is because the MFDA strategy improves the segmentation effect by integrating low-level features and high-level features, and AGLA improves the segmentation effect by aligning the features of the source domain and the target domain. It can be seen that the AGLA method brings a greater improvement than the MFDA strategy. This is because in the case of domain adaptation, the segmentation effect mainly decreases due to domain shift, and the MFDA method shortens the domain difference through feature alignment, so it can better improve the segmentation effect. The MFAD method proposed in the embodiment (i.e., Baseline+MFDA+AGLA) combines the advantages of MFDA and AGLA, greatly improves the Baseline, and achieves good segmentation performance. MFDA mainly improves the model's ability to recognize segmentation boundaries, while AGLA improves the model's ability to recognize segmentation shapes. The MFAD method that combines the advantages of MFDA and AGLA can accurately recognize the segmentation shapes and boundaries.

[0123] (6.4) Other experiments

[0124] The effectiveness of the three-layer aggregated feature alignment of the feature alignment strategy proposed in this embodiment is experimentally verified as follows. Five groups of experiments are respectively set up: full feature fusion alignment, first two-layer feature fusion alignment, first three-layer feature fusion alignment, last two-layer feature fusion alignment, and last three-layer feature fusion alignment. Only the aggregated features are aligned and the semantic features are not aligned in these experiments. Table 7 shows the results under different feature fusion strategies.

[0125] Table 7 Comparison of results of different fusion alignment strategies on the REFUGE challenge training set → Drishti-GS dataset;

[0126]

[0127] It can be seen that full feature fusion can achieve good results. However, if the last two layers of features are discarded and only the aggregated features of the first two layers of feature fusion are aligned, the model will focus too much on the low-level features of the data and ignore the alignment of high-level shape semantics, resulting in a lower segmentation effect. However, the strategy of fusing the first three layers of features with the addition of the third layer of features can more effectively capture high-level semantic features, but still cannot achieve the effect of full feature alignment. This is because the alignment of high-level shape semantics of the last layer of features is very important for domain adaptation. The model focuses on aligning high-level semantic features and ignores the low-level features that may affect the segmentation effect, achieving a better segmentation result than the full feature fusion alignment strategy. However, only aligning the last two layers of features ignores low-level features such as segmentation boundaries. Therefore, the proposed strategy of fusing and aligning the last three layers of features in this embodiment comprehensively considers low-level and high-level features, ignores the first layer of features that will affect segmentation, and makes full use of the boundary and semantic information of the other three layers of features to achieve the best segmentation result.

[0128] The following verifies the effectiveness of the proposed aggregation-semantic feature alignment strategy in this embodiment. Taking Baseline+MFDA as the comparative benchmark model, a discriminator D is added respectively 1 Align the aggregated features and the discriminator D 2 Align the semantic features and add D 1 and D 2 Four groups of experiments are conducted on aligning the aggregated semantic features. Table 8 shows the results under different feature fusion strategies. It can be seen that adding D 1 Aligning the aggregated features brings a greater improvement in effect than adding D 2 Aligning the semantic features. This is because D 1 By aligning low-level and high-level features, it can better reduce the domain shift, while D 2 Aligning the semantic features mainly ensures that the segmented shape is close to the true segmented shape by aligning the shape, and it is more difficult to align regions with a larger domain shift such as boundaries.

[0129] Table 8 Comparison of results of different feature alignment strategies on the REFUGE challenge training set → Drishti-GS dataset;

[0130]

[0131] As Figure 8 shown, this embodiment also discloses an unsupervised domain adaptation medical image segmentation device based on multi-scale features, including:

[0132] The multi-scale feature fusion segmentation network construction module 801 is used to construct a multi-scale feature fusion segmentation network based on dual attention. The multi-scale feature fusion segmentation network includes a number of sequentially connected residual modules, dual attention mechanism modules DAMM and multi-scale feature fusion modules MFFM respectively connected to each residual module. The convolutional layer performs initial feature extraction on the input source domain / target domain medical image, and then extracts high-level image features through each residual module to generate feature maps of different sizes. Each feature map is fed into a DAMM module to generate compressed feature maps of different sizes after fine-grained compression of the features. The compressed feature maps of different sizes are aligned to the same size through upsampling. All the aligned compressed feature maps are input into an MFFM module for feature fusion, and then a probability map for per-pixel prediction is obtained through a 1x1 convolution and a softmax layer to obtain semantic features. Some of the aligned compressed feature maps are input into another MFFM module for feature fusion to obtain aggregated features.

[0133] The discriminator construction module 802 is used to use the multi-scale feature fusion segmentation network as a generator to construct a first discriminator and a second discriminator. The first discriminator is used to align the aggregated features of the source domain medical image and the target domain medical image, and the second discriminator is used to align the semantic features of the source domain medical image and the target domain medical image.

[0134] The training module 803 is used to train based on the generator, the first discriminator and the second discriminator to obtain a trained multi-scale feature fusion segmentation network. During the training process, the mixed segmentation loss of the generator is optimized using the source domain data. The source domain aggregated features and the target domain aggregated features are passed to the first discriminator to optimize the first discriminator loss and the adversarial loss between the first discriminator and the generator. The source domain semantic features and the target domain semantic features are passed to the second discriminator to optimize the second discriminator loss and the adversarial loss between the second discriminator and the generator.

[0135] The segmentation module 804 is used to segment the medical image to be processed using the trained multi-scale feature fusion segmentation network.

[0136] The specific implementation of each module of an unsupervised domain adaptation medical image segmentation device based on multi-scale features is the same as that of an unsupervised domain adaptation medical image segmentation method based on multi-scale features, and will not be repeated in this embodiment.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unsupervised domain adaptation medical image segmentation method based on multi-scale features, characterized in that: include: A multi-scale feature fusion segmentation network construction step, constructing a multi-scale feature fusion segmentation network based on dual attention; the multi-scale feature fusion segmentation network includes a plurality of residual modules connected in sequence, a dual attention mechanism module DAMM connected to each residual module respectively, and a multi-scale feature fusion module MFFM; the convolution layer performs initial feature extraction on the input source domain / target domain medical image, and then extracts high-level image features through each residual module to generate feature maps of different sizes; Each feature map is fed into a DAMM module, and the features are finely compressed to generate compressed feature maps of different sizes; the compressed feature maps of different sizes are aligned to the same size through upsampling; all the aligned compressed feature maps are input into an MFFM module for feature fusion, and then a 1x1 convolution and softmax layer are used to obtain the pixel-by-pixel prediction probability map to obtain the semantic features; The aligned partial compressed feature maps are input into another MFFM module for feature fusion to obtain aggregated features; A first discriminator and a second discriminator construction step based on the generator, using the multi-scale feature fusion segmentation network as a generator to construct the first discriminator and the second discriminator; using the first discriminator to align the aggregate features of the source domain medical image and the target domain medical image, and using the second discriminator to align the semantic features of the source domain medical image and the target domain medical image; Generator and discriminator training steps, training is performed based on the generator, the first discriminator and the second discriminator to obtain a trained multi-scale feature fusion segmentation network; during the training process, the source domain data is used to optimize the hybrid segmentation loss of the generator; the source domain aggregated features and the target domain aggregated features are passed to the first discriminator to optimize the first discriminator loss and the adversarial loss between the first discriminator and the generator; the source domain semantic features and the target domain semantic features are passed to the second discriminator to optimize the second discriminator loss and the adversarial loss between the second discriminator and the generator; In the segmentation step, the trained multi-scale feature fusion segmentation network is used to segment the medical image to be processed.

2. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 1, characterized in that: The multi-scale feature fusion segmentation network includes four layers of residual modules, and the corresponding DAMMs include four; the partial compressed feature map is the compressed feature map output by the corresponding DAMMs of the last three layers of residual modules.

3. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 1, characterized in that: Each of the DAMMs includes a first channel attention module, a first spatial attention module, and a 1x1 convolution for compressing the number of channels; the first channel attention module is connected to the input feature map, and after processing, outputs the channel-refined feature map to the spatial attention module; After processing by the spatial attention module, the feature map refined by the spatial attention mechanism is output to 1x1 convolution for channel compression and output compressed feature map.

4. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 3, characterized in that: In the first channel attention module, the input feature map F∈R C×H×W Through average pooling and maximum pooling operations to aggregate spatial information, two different features containing spatial information are output. and represents the average channel pooling feature, Represents the maximum channel pooling feature; Then and Feed into the MLP network and output and Will and Element-wise addition to get the final channel attention map Mc is normalized by the sigmoid function to obtain the attention weight ω of each channel c ; Perform a weighted operation on each channel of the input feature map through the attention weight of each channel, and finally obtain the feature map after channel refinement in, represents element-wise addition; R represents a real number, C represents the number of channel categories; H represents height; and W represents width.

5. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 3, characterized in that: In the first spatial attention module, for the feature map F output by the first channel attention module c First, for F c Perform average pooling and maximum pooling operations respectively to obtain the average spatial pooling feature map And the maximum spatial pooling feature map Then concat() these two feature maps in the channel dimension to get a fused feature map right A 5x5 convolution is used to reduce the number of channels to 1 to obtain the spatial attention map Ms; the spatial attention map Ms is normalized by the sigmoid function to obtain the attention weight ω of each space. s ; Through ω s F c Perform weighted operations on each space to obtain the feature map after the channel space attention mechanism is refined F cs After a 1x1 convolution, the compressed feature map is obtained; Represents element-wise addition.

6. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 1, characterized in that: The MFFM includes a second channel attention module and a second spatial attention module; let the two features to be fused be F X and F Y , F X ∈R C×H×W , F Y ∈R C×H×W ; First, F X and F Y Performing element-wise addition, we get Then F Z Input the second channel attention module to get the channel attention weight ω c , feature map after channel refinement in, represents element-wise multiplication; then Input the second spatial attention module to get the spatial attention weight ω s , through the spatial attention weight ω s Get F X and F Y Fusion feature map after fusion The newly obtained fused feature map is fused with the next feature map until all feature maps are fused.

7. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 1, characterized in that: Hybrid segmentation loss L seg , which is expressed as follows: Among them, L ce represents the cross entropy loss; L Dice represents Dice loss; N represents the total number of pixels of sample X; C represents the number of channel categories; for sample X = {(x1, y1), (x2, y2), …, (x N ,y N )},x i Represents the value of sample X at the i-th pixel, y i Indicates the label of sample X at the i-th pixel, y i,c represents the binary label of the channel category c at the ith pixel of sample X, represents the predicted binary label of the sample X at the i-th pixel with respect to the channel category c, Represents the predicted probability of sample X at the i-th pixel about channel category c; ω c is the weight factor of the channel category, W is the model weight, is the L2 norm of the model weight; λ1 and λ2 are empirical parameters.

8. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 1, characterized in that: First discriminator loss and the adversarial loss between the first discriminator and the generator It is expressed as follows: Second discriminator loss and the adversarial loss between the second discriminator and the generator Where E represents mathematical expectation; z s Indicates the use of source domain data to generate aggregate features x s →f s , z t Indicates the use of target domain data to generate aggregate features x t →f t ,u s Indicates the use of source domain data to generate semantic features x s → s ,u t Indicates the use of target domain data to generate a predicted probability map x t → t , c1 represents the aggregation of source domain features f s Or the target domain aggregate feature f t Input the first discriminator, c2 represents the source domain semantic feature o s Or the target domain semantic features o t Input the second discriminator, and Respectively represent f s 、f t , o s and t The distribution set of .

9. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 1, characterized in that: The overall loss during training is L total , which is expressed as follows: Among them, L seg represents the hybrid segmentation loss; represents the first discriminator loss; represents the second discriminator loss; represents the adversarial loss of the first discriminator and the generator; represents the adversarial loss of the second discriminator and the generator; λ3, λ4, and λ5 are empirical parameters.

10. An unsupervised domain adaptation medical image segmentation device based on multi-scale features, characterized in that: include: A multi-scale feature fusion segmentation network construction module is used to construct a multi-scale feature fusion segmentation network based on dual attention; the multi-scale feature fusion segmentation network includes a plurality of residual modules connected in sequence, a dual attention mechanism module DAMM connected to each residual module, and a multi-scale feature fusion module MFFM; the convolution layer performs initial feature extraction on the input source domain / target domain medical image, and then extracts high-level image features through each residual module to generate feature maps of different sizes; Each feature map is fed into a DAMM module, and the features are finely compressed to generate compressed feature maps of different sizes; the compressed feature maps of different sizes are aligned to the same size through upsampling; all the aligned compressed feature maps are input into an MFFM module for feature fusion, and then a 1x1 convolution and softmax layer are used to obtain the pixel-by-pixel prediction probability map to obtain the semantic features; The aligned partial compressed feature maps are input into another MFFM module for feature fusion to obtain aggregated features; A discriminator construction module is used to use the multi-scale feature fusion segmentation network as a generator to construct a first discriminator and a second discriminator; use the first discriminator to align the aggregate features of the source domain medical image and the target domain medical image, and use the second discriminator to align the semantic features of the source domain medical image and the target domain medical image; A training module is used to perform training based on the generator, the first discriminator and the second discriminator to obtain a trained multi-scale feature fusion segmentation network; during the training process, the source domain data is used to optimize the mixed segmentation loss of the generator; the source domain aggregated features and the target domain aggregated features are passed to the first discriminator to optimize the first discriminator loss and the adversarial loss between the first discriminator and the generator; the source domain semantic features and the target domain semantic features are passed to the second discriminator to optimize the second discriminator loss and the adversarial loss between the second discriminator and the generator; The segmentation module is used to segment the medical image to be processed using the trained multi-scale feature fusion segmentation network.

Images