Unsupervised domain adaptive medical image segmentation method and device based on multi-scale features

Abstract

The invention discloses an unsupervised domain adaptive medical image segmentation method and device based on multi-scale features, and relates to the technical field of computers, and the method comprises the steps: constructing a multi-scale feature fusion segmentation network based on double attention; taking the multi-scale feature fusion segmentation network as a generator, and constructing a first discriminator and a second discriminator; training based on the generator, the first discriminator and the second discriminator to obtain a trained multi-scale feature fusion segmentation network; and segmenting a medical image to be processed by using the trained multi-scale feature fusion segmentation network. According to the method, multi-scale features are introduced, feature representation of different levels is considered, and alignment of source domain and target domain features is realized through an adversarial feature alignment method and adversarial learning, so that the performance and generalization ability of a segmentation model on a target domain are improved; moreover, the method does not need to use a target domain label, and can well solve a problem of unsupervised domain adaptation of a medical image segmentation task.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an unsupervised domain adaptation medical image segmentation method and device based on multi-scale features. Background Art

[0002] Domain adaptation (DA) is of great significance in the segmentation of optic cup and optic disc in fundus images, because in different fundus image datasets (source domain and target domain), there may be different shooting equipment, environmental conditions, and image quality, which may lead to the performance degradation of the model trained in the source domain in the target domain. This performance degradation is often caused by the difference in data distribution between the source domain and the target domain, such as the differences in the morphology, color, texture, etc. of the optic cup and optic disc. A common DA solution is to reduce the distribution difference between the source domain and the target domain through feature alignment to improve the generalization ability of the model in the target domain. However, in unsupervised domain adaptation (UDA), the target domain usually lacks label information, which makes the traditional feature alignment method that requires target domain label information inapplicable. Therefore, for the task of optic cup and optic disc segmentation in fundus images, it is necessary to propose a feature alignment method suitable for UDA.

[0003] In neural networks, low-level features cover rich details, such as color, illumination, and resolution, which help capture local information of the image. In the optic cup and optic disc segmentation task, low-level features may include image color information (such as different gray levels), texture information (such as spots and stripes), and edge information (such as the boundaries of the optic cup and optic disc), which help the network identify subtle changes and local structures in different regions of the image. In contrast, high-level features involve more abstract semantic information, such as semantic structure and deformation. In the optic cup and optic disc segmentation task, high-level features may include the shape, size, and relative position of the optic cup and optic disc, as well as the spatial relationship between them. These high-level features help the network understand the correlation between different regions in the image and convert it into semantically meaningful information, thereby better completing the segmentation task. However, there are also some redundant features in fundus images caused by external factors, such as shadows under lighting conditions, reflections on the surface of the eyeball, and different noise in the image. In order to comprehensively consider low-level and high-level features, this chapter adopts a multi-scale feature approach to consider feature representations at different scales. Multi-scale features can fully consider the information of images at different scales, but due to the existence of domain offset, the features between the source domain and the target domain are very different, so the features of the source domain and the target domain need to be aligned. However, due to the lack of label information in the target domain, it is impossible to directly align the features of the source domain and the target domain. To solve this problem, this chapter proposes an adversarial feature alignment method. By introducing adversarial learning technology based on generative adversarial networks, it achieves effective alignment of feature representations between the source domain and the target domain without the need for target domain labels, thereby improving the performance and generalization ability of the model on the target domain. Summary of the invention

[0004] The purpose of this application is to propose an unsupervised domain adaptation medical image segmentation method and device based on multi-scale features to address the above-mentioned technical problems, introduce multi-scale features to consider feature representations at different levels, and use adversarial feature alignment methods and adversarial learning to achieve alignment of source domain and target domain features, thereby improving the performance and generalization ability of the segmentation model in the target domain; and the present invention does not require the use of target domain labels, which can well alleviate the unsupervised domain adaptation problem of medical image segmentation tasks.

[0005] On the one hand, an unsupervised domain adaptation medical image segmentation method based on multi-scale features, comprising:

[0006] The multi-scale feature fusion segmentation network construction step constructs a multi-scale feature fusion segmentation network based on dual attention; the multi-scale feature fusion segmentation network includes a plurality of residual modules connected in sequence, a dual attention mechanism module DAMM connected to each residual module and a multi-scale feature fusion module MFFM; the convolution layer performs initial feature extraction on the input source domain / target domain medical image, and then extracts high-level image features through each residual module to generate feature maps of different sizes; each feature map is fed into a DAMM module, and the features are refined and compressed to generate compressed feature maps of different sizes; the compressed feature maps of different sizes are aligned to the same size by upsampling; all the aligned compressed feature maps are input into an MFFM module for feature fusion, and a probability map of pixel-by-pixel prediction is obtained through a 1x1 convolution and a softmax layer to obtain semantic features; the aligned partial compressed feature maps are input into another MFFM module for feature fusion to obtain aggregated features;

[0007] A first discriminator and a second discriminator construction step based on the generator, using the multi-scale feature fusion segmentation network as a generator to construct the first discriminator and the second discriminator; using the first discriminator to align the aggregate features of the source domain medical image and the target domain medical image, and using the second discriminator to align the semantic features of the source domain medical image and the target domain medical image;

[0008] Generator and discriminator training steps, training is performed based on the generator, the first discriminator and the second discriminator to obtain a trained multi-scale feature fusion segmentation network; during the training process, the source domain data is used to optimize the hybrid segmentation loss of the generator; the source domain aggregated features and the target domain aggregated features are passed to the first discriminator to optimize the first discriminator loss and the adversarial loss between the first discriminator and the generator; the source domain semantic features and the target domain semantic features are passed to the second discriminator to optimize the second discriminator loss and the adversarial loss between the second discriminator and the generator;

[0009] In the segmentation step, the trained multi-scale feature fusion segmentation network is used to segment the medical image to be processed.

[0010] Preferably, the multi-scale feature fusion segmentation network includes four layers of residual modules, and the corresponding DAMMs include four; the partially compressed feature map is a compressed feature map output by the corresponding DAMMs of the last three layers of residual modules.

[0011] Preferably, each of the DAMMs includes a first channel attention module, a first spatial attention module and a 1x1 convolution for compressing the number of channels; the first channel attention module is connected to the input feature map, and after processing, outputs the feature map after channel refinement to the spatial attention module; after processing, the spatial attention module outputs the feature map after spatial attention mechanism refinement to the 1x1 convolution for channel number compression and outputs the compressed feature map.

[0012] Preferably, in the first channel attention module, the input feature map is first Through average pooling and maximum pooling operations to aggregate spatial information, two different features containing spatial information are output. and , represents the average channel pooling feature, represents the maximum channel pooling feature; then and Feed into the MLP network, output and ,Will and Element-wise addition to get the final channel attention map ; The attention weight of each channel is obtained after normalization by the sigmoid function ; Perform a weighted operation on each channel of the input feature map through the attention weight of each channel, and finally obtain the feature map after channel refinement ;in, represents element-wise addition; represents a real number, Indicates the number of channel categories; Indicates height; Indicates width.

[0013] Preferably, in the first spatial attention module, for the feature map output by the first channel attention module , first of all Perform average pooling and maximum pooling operations respectively to obtain the average spatial pooling feature map And the maximum spatial pooling feature map ; Then concat() these two feature maps in the channel dimension to get a fused feature map ,right Use a 5x5 convolution to reduce the number of channels to 1 to obtain the spatial attention map ; Spatial Attention Map The attention weight of each space is obtained after normalization by the sigmoid function ;pass right Perform weighted operations on each space to obtain the feature map after the channel space attention mechanism is refined ;Will After a 1x1 convolution, the compressed feature map is obtained; Represents element-wise addition.

[0014] Preferably, the MFFM includes a second channel attention module and a second spatial attention module; assuming that the two features to be fused are and , , ; First of all, and Performing element-wise addition, we get ; then Input the second channel attention module to get the channel attention weight , feature map after channel refinement ,in, represents element-wise multiplication; then Input the second spatial attention module to get the spatial attention weight , through the spatial attention weight get and Fusion feature map after fusion ; Fuse the newly obtained fused feature map with the next feature map until all feature maps are fused.

[0015] Preferably, the hybrid segmentation loss , which is expressed as follows:

[0016] ;

[0017] ;

[0018] ;

[0019] in, represents the cross entropy loss; represents Dice loss; Representation sample The total number of pixels; Indicates the number of channel categories; for samples , Representation sample At the value of the i-th pixel, Representation sample The label at the ith pixel, Representation sample The binary label of channel category c at the ith pixel, Representation sample Predict the binary label for channel category c at the i-th pixel, Representation sample The predicted probability of channel category c at the i-th pixel; is the weight factor of the channel category, is the model weight, is the L2 norm of the model weight; and is an empirical parameter.

[0020] Preferably, the first discriminator loss and the adversarial loss between the first discriminator and the generator , which is expressed as follows:

[0021] ;

[0022] ;

[0023] Second discriminator loss and the adversarial loss between the second discriminator and the generator :

[0024] ;

[0025] ;

[0026] in, represents mathematical expectation; Indicates the use of source domain data to generate aggregate features , Indicates the use of target domain data to generate aggregate features , Indicates the use of source domain data to generate semantic features , Indicates the use of target domain data to generate a predicted probability map , Indicates that the source domain features are aggregated Or the target domain aggregate features Input the first discriminator, Indicates that the source domain semantic features Or the target domain semantic features Input the second discriminator, Respectively The distribution set of .

[0027] Preferably, the overall loss during training is , which is expressed as follows:

[0028] ;

[0029] in, represents the hybrid segmentation loss; represents the first discriminator loss; represents the second discriminator loss; represents the adversarial loss of the first discriminator and the generator; represents the adversarial loss between the second discriminator and the generator; , and is an empirical parameter.

[0030] On the other hand, an unsupervised domain adaptation medical image segmentation device based on multi-scale features comprises:

[0031] A multi-scale feature fusion segmentation network construction module is used to construct a multi-scale feature fusion segmentation network based on dual attention; the multi-scale feature fusion segmentation network includes a plurality of residual modules connected in sequence, a dual attention mechanism module DAMM connected to each residual module and a multi-scale feature fusion module MFFM; the convolution layer performs initial feature extraction on the input source domain / target domain medical image, and then extracts high-level image features through each residual module to generate feature maps of different sizes; each feature map is fed to a DAMM module, and the features are refined and compressed to generate compressed feature maps of different sizes; compressed feature maps of different sizes are aligned to the same size by upsampling; all aligned compressed feature maps are input into an MFFM module for feature fusion, and a probability map of pixel-by-pixel prediction is obtained through a 1x1 convolution and a softmax layer to obtain semantic features; the aligned partial compressed feature maps are input into another MFFM module for feature fusion to obtain aggregated features;

[0032] A discriminator construction module is used to use the multi-scale feature fusion segmentation network as a generator to construct a first discriminator and a second discriminator; use the first discriminator to align the aggregate features of the source domain medical image and the target domain medical image, and use the second discriminator to align the semantic features of the source domain medical image and the target domain medical image;

[0033] A training module is used to perform training based on the generator, the first discriminator and the second discriminator to obtain a trained multi-scale feature fusion segmentation network; during the training process, the source domain data is used to optimize the mixed segmentation loss of the generator; the source domain aggregated features and the target domain aggregated features are passed to the first discriminator to optimize the first discriminator loss and the adversarial loss between the first discriminator and the generator; the source domain semantic features and the target domain semantic features are passed to the second discriminator to optimize the second discriminator loss and the adversarial loss between the second discriminator and the generator;

[0034] The segmentation module is used to segment the medical image to be processed using the trained multi-scale feature fusion segmentation network.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) The multi-scale feature fusion strategy based on the dual attention mechanism of the present invention improves the segmentation accuracy by combining low-level features and high-level features;

[0037] (2) The adversarial learning method for shortening domain shift of the present invention aligns the target domain to the source domain by aligning aggregate features and semantic features, further improving the generalization ability of the model in the target domain;

[0038] (3) Through unsupervised domain adaptation experiments conducted on fundus image datasets and prostate image datasets, the effectiveness of the method proposed in this invention is verified by comparing it with some existing advanced unsupervised domain adaptation methods in terms of performance and visualized images;

[0039] (4) Through ablation experiments and specific alignment strategy comparison experiments, it is further verified that the proposed method can effectively solve the problem of unavailability of target domain labels in the process of model migration in medical image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A schematic diagram of a flow chart of an unsupervised domain adaptation medical image segmentation method based on multi-scale features according to an embodiment of the present application;

[0042] Figure 2 A schematic diagram of the overall model structure of a medical image segmentation model of an unsupervised domain adaptation medical image segmentation method based on multi-scale features according to an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the structure of a multi-scale feature fusion segmentation network based on dual attention according to an embodiment of the present application;

[0044] Figure 4 This is a schematic diagram of the structure of a dual attention mechanism module of an embodiment of the present application;

[0045] Figure 5 A schematic diagram of the structure of a multi-scale feature fusion module according to an embodiment of the present application;

[0046] Figure 6 The segmentation results of the embodiment method of the present application and the prior art method in the fundus image;

[0047] Figure 7The segmentation results of the embodiment method of the present application and the prior art method in the prostate image;

[0048] Figure 8 Schematic diagram of an unsupervised domain adaptation medical image segmentation device based on multi-scale features according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] like Figure 1 As shown, this embodiment provides an unsupervised domain adaptation medical image segmentation method based on multi-scale features (Unsupervised Domain Adaptation Medical Image Segmentation Method Based on Multi-scale Feature Adversarial Learning, MFAD), including the following steps:

[0051] S101, a multi-scale feature fusion segmentation network construction step, constructing a multi-scale feature fusion segmentation network based on dual attention; the multi-scale feature fusion segmentation network includes a plurality of residual modules connected in sequence, a dual attention mechanism module DAMM connected to each residual module and a multi-scale feature fusion module MFFM; the convolution layer performs initial feature extraction on the input source domain / target domain medical image, and then extracts high-level image features through each residual module to generate feature maps of different sizes; each feature map is fed to a DAMM module, and the features are refined and compressed to generate compressed feature maps of different sizes; the compressed feature maps of different sizes are aligned to the same size by upsampling; all the aligned compressed feature maps are input into an MFFM module for feature fusion, and a probability map of pixel-by-pixel prediction is obtained through a 1x1 convolution and a softmax layer to obtain semantic features; the aligned partial compressed feature maps are input into another MFFM module for feature fusion to obtain aggregated features;

[0052] S102, based on the first discriminator and the second discriminator construction step of the generator, the multi-scale feature fusion segmentation network is used as the generator to construct the first discriminator and the second discriminator; the first discriminator is used to align the aggregate features of the source domain medical image and the target domain medical image, and the second discriminator is used to align the semantic features of the source domain medical image and the target domain medical image;

[0053] S103, generator and discriminator training step, training is performed based on the generator, the first discriminator and the second discriminator to obtain a trained multi-scale feature fusion segmentation network; during the training process, the source domain data is used to optimize the hybrid segmentation loss of the generator; the source domain aggregated features and the target domain aggregated features are passed to the first discriminator, the first discriminator loss and the adversarial loss of the first discriminator and the generator are optimized; the source domain semantic features and the target domain semantic features are passed to the second discriminator, the second discriminator loss and the adversarial loss of the second discriminator and the generator are optimized;

[0054] S104, a segmentation step, using the trained multi-scale feature fusion segmentation network to segment the medical image to be processed.

[0055] like Figure 2 As shown, the multi-scale feature fusion strategy based on dual attention (MFFDA) of this embodiment removes the first maximum pooling layer, the last average pooling layer and the fully connected layer on the basis of the original ResNet34, and only retains the first 7×7 convolution layer, and the designed four groups of residual modules, including residual blocks of 4, 6, 3, and 3, respectively. Each residual block consists of two 3x3 convolution layers, and the downsampling rate is designed to be 16. A total of 4 downsamplings are performed, and each time the width and height of the input feature map are changed to half of the original. For the input medical image, four scale feature maps are generated. However, the generated feature map may not be able to effectively extract all the features of the medical image. This embodiment introduces a dual attention mechanism module (Dual Attention Mechanism Module, DAMM) to perform refined compression processing on each extracted feature. Finally, the four refined feature maps are upsampled to the same size and then fed into the Multi-scale Feature Fusion Module (MFFM) to be fused from low to high layers into an aggregated feature, and finally a softmax layer is used to generate the segmentation result (medical image segmentation mask).

[0056] Specifically, Figure 3As shown in the figure, in the encoding stage, the input medical image is first forwarded to the first convolutional layer for initial feature extraction to capture low-level image features. The convolutional layer is designed as a 7x7 convolution with a step size of 2. For a medical image input of size 3x512x512, a feature map of size 64x256x256 is generated. Then, four residual modules are used to further extract high-level image features, and finally feature maps of sizes 64x256x256, 128x128x128, 256x64x64, and 512x32x32 are generated. Each feature map is fed to the DAMM module, and the features are refined and compressed to 16 channels to generate compressed feature maps of sizes 16x256x256, 16x128x128, 16x64x64, and 16x32x32. In the decoding stage, feature maps of different sizes are aligned to the same size of 16x512x512 by upsampling, and then the compressed feature maps of these four scales are input into the MFFM module, and finally the refined and aggregated feature maps are obtained after analyzing and fusing the feature information of each scale. Then a 1x1 convolution is used to generate a vector of Cx512x512, and the number of channels is the number of segmentation categories C (C is defined as the number of channel categories). Finally, a softmax layer is used to obtain the probability of pixel-by-pixel prediction.

[0057] like Figure 4 As shown in Figure 1, the DAMM module contains a channel attention module, a spatial attention module, and a 1x1 convolution for compressing the number of channels. In the channel attention module, for the input feature map First, it is subjected to average pooling and maximum pooling operations to aggregate spatial information, and then two different features containing spatial information are output and , , represents the average channel pooling feature, represents the maximum channel pooling feature. Then through and Generate channel attention map Specifically, a multi-layer perceptron (MLP) network is designed, which consists of two fully connected (FC) layers and one hidden layer. and Feed into the MLP network, which processes each independently and The output obtained by MLP is and Element-wise addition to get the final channel attention map After the above operations, channel attention can simultaneously consider the average and maximum information in the channel, thereby generating a more detailed and comprehensive channel attention signal. Channel Attention Map The attention weight of each channel is obtained after normalization by the sigmoid function .get Finally, the original feature map is weighted by the attention weight of each channel, and finally the feature map after channel refinement is obtained. .in, Indicates that Broadcast to Element-wise multiplication with F after resizing.

[0058] Get the feature map after channel refinement After that, you need to Feed it into the spatial attention module to obtain more refined features. , first along the channel dimension C Perform average pooling and maximum pooling operations respectively to obtain the average spatial pooling feature map And the maximum spatial pooling feature map , Then concat() these two feature maps in the channel dimension to get a fused feature map In order to capture the spatial context information, we get the spatial attention map ,right Use a 5x5 convolution, which reduces the number of channels to 1 to obtain a spatial attention map Similarly, the spatial attention map The attention weight of each space is obtained after normalization by the sigmoid function . This allows the model to focus on more important spatial regions when processing features. Finally, by Feature map after adjustment of the channel attention module Perform weighted operations on each space to obtain the feature map after the channel space attention mechanism is refined Then, After a 1x1 convolution, the channels are compressed to 16 to obtain the final compressed and refined feature map. This step is to align the feature map channels of different sizes to 16 channels to facilitate subsequent multi-scale feature fusion operations.

[0059] After obtaining the feature maps of each scale through the DAMM module, these feature maps of different scales need to be fused through the MFFM module.

[0060] like Figure 5As shown in , the MFFM module fuses the two features through the channel attention module (second channel attention module) and the spatial attention module (second spatial attention module) in the DAMM module. The MFFM module obtains the aggregated feature that fuses all scale feature information by repeatedly fusing the newly obtained fused feature map with the next feature map until all feature maps are fused. Suppose the two features to be fused are and , First of all, and Performing element-wise addition, we get Then Input channel attention module to get channel attention weight , unlike the DAMM module, the feature map after channel refinement ,in, represents element-wise multiplication. Then Input the spatial attention module to get the spatial attention weight , through the spatial attention weight Get the final Fusion feature map after fusion This makes the fusion feature Can and The fused feature map is then fused with the feature map to be fused until there is no feature map to be fused, and the final aggregated feature map is obtained.

[0061] It should be noted that the MFFDA of this embodiment includes two MFFM modules, one of which analyzes and fuses the compressed feature maps of four scales and generates a Cx512x512 vector through 1x1 convolution. Finally, a softmax layer is used to obtain a pixel-by-pixel prediction probability map to obtain semantic features. The other MFFM module obtains aggregate features by analyzing and fusing the compressed feature maps of three scales of the three aligned layers.

[0062] The MFFDA proposed above uses source domain data and labels to perform supervised learning of multi-scale feature fusion to establish a mapping from the input image to the label space. In order to optimize the classification accuracy of the model for each pixel and improve the segmentation quality, especially for unbalanced categories, this embodiment uses cross entropy loss and Dice loss Combined hybrid segmentation loss To optimize MFFDA, as shown below.

[0063] ;

[0064] ;

[0065] ;

[0066] in, Representation sample The total number of pixels; for the sample , Representation sample At the value of the i-th pixel, Representation sample The label at the ith pixel, Representation sample The binary label of channel category c at the ith pixel, Representation sample Predict the binary label for channel category c at the i-th pixel, Representation sample The predicted probability of channel category c at the i-th pixel; is the weight factor of the channel category, which is inversely proportional to the sample frequency of the category; is the model weight, is the L2 norm of the model weights, used for regularization to prevent overfitting; and is an empirical parameter. By minimizing the hybrid segmentation loss composed of multi-class cross entropy loss and Dice coefficient loss To optimize the segmentation network of the source domain.

[0067] The aggregation-semantic feature alignment method based on adversarial learning is explained as follows.

[0068] In the medical image segmentation task, the segmentation network based on multi-scale feature fusion makes comprehensive use of low-level and high-level features to ensure that the model can fully capture the subtle information and abstract features in the image, thereby achieving accurate segmentation. However, due to the large differences between the features of the target domain and the source domain, the domain-invariant features of the source domain and the target domain are learned through feature alignment (i.e., the segmentation structures of different domains share features). Based on this, this embodiment proposes an aggregation-semantic feature alignment method (Adversarial Global Local Feature Alignments strategy, AGLA) based on adversarial learning from the idea of ​​GAN.

[0069] In the medical image segmentation task, although the images generated by different devices may have significant differences in low-level features such as grayscale distribution and texture, the high-level semantic information, such as the shape, size and relative position of the segmentation structure, should be similar. Therefore, the low-level features are the main cause of domain shift, and the method often used in the prior art is to align features of all levels. The problem with this full feature alignment strategy is that if all levels of features, whether high-level or low-level, are aligned indiscriminately, it may cause those low-level feature changes that are not closely related to the task to be overemphasized, causing the model to over-adapt to these irrelevant changes and ignore the high-level semantic information that should actually be paid attention to. Therefore, this embodiment aligns the high-level features of the source domain and the target domain while ignoring the low-level feature alignment, reducing the visual differences caused by different imaging conditions (such as different imaging devices or settings) to promote the consistency of high-level feature distribution between the source domain and the target domain. However, considering the hierarchical structure of Resnet34, the features of a certain layer depend on the activation of its previous layer and will affect the subsequent layers. If only a certain layer of features obtained from MFDA is aligned, the potential spatial alignment may be unstable. Therefore, this problem is solved by aligning the aggregated features of the fused compressed features of the last three layers of MFDA. Compared with aligning the features of all layers, this is not only easier to transfer and adapt between different domains, but also because the dimension of the alignment is low, fewer computing resources are required to process these features. The medical image prediction probability map generated by MFDA not only contains rich semantic knowledge, but also has a low dimension. Aligning high-level semantic features such as the prediction probability map not only has low overhead, but also can perform domain alignment based on the geometric and semantic properties of the segmentation structure. Therefore, the idea of ​​feature alignment in this embodiment is to align the aggregated features and semantic features of the source domain and the target domain respectively.

[0070] The AGLA method proposed in this embodiment aligns the features of the source domain and the target domain by designing two discriminators and one generator. The generator G is the MFFDA model proposed above, which is responsible for generating aggregation and semantic features. For aligning aggregate features ( and ) distribution, the second discriminator For aligning semantic features ( and ) distribution. is the aggregated feature of the source domain data, is the aggregated feature of the target domain data, is the semantic feature of the source domain data (prediction probability map), is the semantic feature of the target domain data (prediction probability map). Discriminator and Based on the CGAN design, it is implemented by 5 convolution blocks, each of which contains a convolution layer, a Leaky-ReLU activation function, a batch normalization layer, and a Sigmoid function as the last layer. The common practice of using Leaky-ReLU as the activation function and using convolution inside the discriminator for downsampling instead of ReLU and maximum pooling is to avoid instability caused by sparse gradients. In the training process of GAN, it is usually necessary to deal with the dynamic confrontation between the generator and the discriminator. In the traditional GAN ​​framework, this confrontation is implemented by maximizing and minimizing the Jensen-Shannon divergence. However, This method may lead to instability in training, especially when the discriminator is too good, causing the generator gradient to disappear. In order to overcome these problems, this embodiment uses Wasserstein distance as a loss function to improve the training process. Wasserstein distance provides a method to measure the difference between two probability distributions, with better geometric intuition and mathematical properties. It quantifies the difference between the two distributions by calculating the "minimum amount of work" required to "transfer" one distribution to another.

[0071] First, use the discriminator Align the distribution of aggregated features and and enter Estimating Aggregate Features and Wasserstein distance between distributions to determine whether the aggregated features are aligned. The task is to estimate which domain these features come from. Then the predicted probability map obtained by G and Input Discriminator Align semantic feature distribution and estimate prediction probability map and The Wasserstein distance between the distributions to determine whether the semantic features are aligned. The task of the generator G is to estimate which domain these predicted probability maps come from. The purpose of this process is to align the aggregated features and semantic features by reducing the Wasserstein distance, so that the target domain detection results gradually become more similar to the source domain. The task of the generator G is to make the feature / prediction probability map distribution close to the source domain, and the discriminator and The task of is to minimize the Wasserstein distance of the aggregate / semantic feature distribution of the generator G between the source domain and the target domain, thereby discriminating whether the feature comes from the source domain or the target domain.

[0072] Because in AGLA, we hope that the distribution generated by the generator G is as close to the data distribution of the source domain as possible, which is achieved by minimizing the Wasserstein distance. Therefore, the optimization goal of the discriminator is to minimize its ability to distinguish between source domain data and target domain data, thereby guiding the generator to improve the target domain data distribution it generates to make it closer to the source domain data distribution. Therefore, the task of the discriminator is to minimize the Wasserstein distance, not to maximize the distance like traditional GAN. In the aggregate feature alignment, the first discriminator loss and the adversarial loss between the first discriminator and the generator , which is expressed as follows:

[0073] ;

[0074] ;

[0075] In semantic feature alignment, the second discriminator loss and the adversarial loss between the second discriminator and the generator :

[0076] ;

[0077] ;

[0078] in, represents mathematical expectation; Indicates the use of source domain data to generate aggregate features , Indicates the use of target domain data to generate aggregate features , Indicates the use of source domain data to generate semantic features , Indicates the use of target domain data to generate a predicted probability map , Indicates that the source domain features are aggregated Or the target domain aggregate features Input the first discriminator, Indicates that the source domain semantic features Or the target domain semantic features Input the second discriminator, Respectively The distribution set of .

[0079] Therefore, the overall loss of the MFAD proposed in this embodiment is It is obtained by summing up all the following losses, as follows:

[0080] ;

[0081] During the entire training process, the segmentation loss of the detection network G is first optimized using the source domain data. Then use G to predict the target domain data. Then, the aggregate features ( and ) and predicted probability plots ( and ) is passed to the discriminator and , to optimize the discriminator loss and , and finally calculate the adversarial loss and As G is continuously updated, the Wasserstein distance becomes smaller and smaller, making the target domain detection results gradually become more similar to the source domain, and the domain gap becomes smaller and smaller.

[0082] The unsupervised domain adaptation medical image segmentation method based on multi-scale features of this embodiment is verified by experiments as follows, including the following steps.

[0083] (1) Dataset.

[0084] In the field of medical image segmentation, medical image database is the cornerstone of medical image segmentation. Due to the difficulty of data collection and high data privacy requirements, it is impossible to provide sufficient samples for the realization of medical image segmentation research. Therefore, the research and development of medical image segmentation is challenged by the small data samples. This embodiment mainly uses public fundus image datasets and prostate image datasets for experiments. There are mainly three fundus image datasets, namely Drishti-GS, RIM-ONE_r3 and REFUGE challenge training set, and three prostate image datasets, namely NCI, I2CVB and PROMISE12. These datasets are introduced below.

[0085] Drishti-GS dataset: The dataset was developed by a team of researchers from India to provide high-quality fundus images to support automatic segmentation of the optic disc and optic cup. It provides 101 2049 color fundus images and corresponding annotations (the training set includes 50 images and the test set includes 51 images).

[0086] RIM-ONE_r3 dataset: is a public fundus image dataset focusing on dry eye research, aiming to promote the development of glaucoma detection and optic disc and optic cup segmentation technology. This dataset collects 159 images from several European hospitals. The training set includes 99 images and the test set includes 60 images. Each image has been accurately annotated by ophthalmologists, including the boundaries of the optic disc and optic cup.

[0087] REFUGE Challenge Dataset: The 2018 REFUGE Challenge provides a set of annotated fundus images for training and validating computer vision and machine learning algorithms. The training set includes 400 The REFUGE dataset contains high-quality color fundus images that are commonly used for the diagnosis and monitoring of glaucoma. These images can show the detailed structure of the retina, optic disc, and macula, which is crucial for the diagnosis of glaucoma. The images in the dataset are accompanied by expert annotations, including the boundary positioning of the optic disc and optic cup, which are key indicators for evaluating optic nerve head damage and glaucoma. By calculating the cup-to-disc ratio, it can assist in the diagnosis of glaucoma.

[0088] NCI-ISBI2013 Dataset: The NCI-ISBI2013 dataset is a prostate magnetic resonance imaging (MRI) data challenge jointly initiated by the Cancer Imaging Program of the National Cancer Institute (NCI) of the United States and the International Society of Biomedical Imaging (ISBI) in 2013. The training set of the challenge dataset includes 60 cases, half of which are from Boston Medical Center (BMC), with a magnetic field strength of 1.5T, a resolution of 0.4 mm in the transverse plane and 3 mm in the sagittal plane. An endoscopic coil is used, and imaging is performed by Philips equipment. The other half of the cases are provided by Radboud University Medical Center (RUNMC) in Nijmegen, the Netherlands, with a magnetic field strength of 3T, a resolution of 0.6-0.625 mm in the transverse plane and 3.6-4 mm in the sagittal plane. A surface coil is used, and imaging is performed by Siemens equipment.

[0089] I2CVB dataset: This dataset comes from the Initiative for Collaborative Computer Visior Benchmarking (I2CVB), provided by the HCRUDB institution, and contains 19 cases. The magnetic field strength is 3T, the resolution is 0.67-0.79 mm in the transverse plane and 1.25 mm in the sagittal plane. No endoscopic coil is used, and imaging is performed by Siemens equipment.

[0090] PROMISE1 dataset: This dataset is derived from ProstateMRImageSegmentation2012 (PROMISE12), provided by UCL, BIDMC and HK, with 13, 12 and 12 cases respectively. The dataset used in the experiment of this embodiment is the data provided by HK, which contains 12 cases, with a magnetic field strength of 1.5T, a resolution of 0.625 mm in the transverse plane and 3.6 mm in the sagittal plane, using an endoscopic coil and imaging by Siemens equipment.

[0091] (2) Data preprocessing

[0092] Specifically, if the entire medical image is used directly for training, there are the following problems: the background information of the original image, such as edge areas and non-target structures, may introduce noise, affecting the model's ability to recognize key features, resulting in a decrease in the model's resolution. To this end, this paper introduces the concept of ROI (region of interest). The ROI image focuses on the segmented area and reduces background interference, allowing the model to learn the details of these key areas more attentively, thereby improving the accuracy of segmentation.

[0093] Due to the limitation of the small amount of medical image datasets, the model trained directly with medical image datasets is prone to overfitting and poor generalization. In order to improve the model training effect, this embodiment adopts a series of data enhancement strategies to simulate different visual effects and expand the training dataset. By inverting and rotating the image, adjusting the brightness and contrast, the diversity of samples can be artificially increased, which helps the model have better generalization ability in practical applications. After these enhanced images, for fundus images, by fixing the zoom to For the prostate image dataset, each sample is resized to The size of the image is 1000×10, ensuring the consistency of the data during the training process. At the same time, the image is randomly flipped and standardized so that the pixel values ​​of each channel have a uniform mean and standard deviation, further reducing the risk of overfitting during model training. Through these preprocessing steps, the number of images in the training dataset is increased and the quality is improved, ultimately solving the limitation of the difficulty of training models caused by the small amount of data in the medical image dataset.

[0094] (3) Evaluation indicators

[0095] In order to objectively evaluate the segmentation performance of the target domain model on the medical image dataset in the experiment, this embodiment selects two widely used standard indicators in image segmentation evaluation: Dice coefficient (Dice) and average surface distance (ASD). These indicators together provide a comprehensive evaluation for the quantification of segmentation effect. The Dice coefficient (Dice) compares the similarity between the predicted segmentation result and the true segmentation label. It calculates the ratio of twice the intersection area of ​​the predicted and true segmentation to the sum of their respective areas. The value range of the Dice coefficient is from 0 to 1. The higher the value, the closer the segmentation result is to the actual situation, that is, the better the segmentation effect. The calculation formula is as follows.

[0096] ;

[0097] Among them, A is the predicted segmentation result set, B is the actual segmentation result set, Represents the number of elements in the intersection of set A and set B. represents the number of elements in set A. Represents the number of elements in set B.

[0098] The average surface distance (ASD) measures the average distance between the predicted segmentation and the true segmentation surface. It calculates the average shortest distance from the predicted segmentation to all points on the true segmentation surface, and the average shortest distance from the true segmentation to all points on the predicted segmentation surface. The smaller the ASD value, the closer the segmentation result is to the true situation, that is, the higher the segmentation quality. The calculation formula is as follows.

[0099] ;

[0100] Among them, X and Y represent the point sets on the predicted segmentation surface and the true segmentation surface, respectively. Represents the Euclidean distance between point x and point y. ASD calculates the average shortest distance between all pairs of points from X to Y and from Y to X.

[0101] (4) Experimental environment and training configuration information

[0102] The software and hardware experimental environment in this experiment is shown in Table 1.

[0103] The training parameters of this experiment are detailed as follows:

[0104] The MFDA model was trained using the Adam optimizer with momentum of 0.9 and 0.99, learning rate of 0.001, and batch size of 10. Set to 0.8, Set to 0.2. In the unsupervised domain adaptation adversarial total loss, Set to 1, Set to 0.5, Let be the discriminator and The slope of the Leaky-RrLU is set to 0.25. The discriminator is optimized using , and the learning rate is 0.0001.

[0105] Table 1 Experimental software and hardware environment configuration;

[0106]

[0107] (5) Experimental design

[0108] The experiment of this embodiment is mainly to explore the following questions:

[0109] First, in the UDA medical image segmentation scenario, what is the accuracy of the MFAD method proposed in the present invention, and whether it has any advantages compared with other medical image segmentation methods, and the results can be intuitively seen through visual segmentation results.

[0110] Second, in the UDA medical image segmentation scenario, the impact of the MFDA and AGLA modules proposed in the present invention on the model performance. In this embodiment, four sets of ablation experiments are designed to verify the effectiveness of each module using the same data set and training process.

[0111] Third, in the medical image segmentation scenario, the effectiveness of the alignment method is explored. In this embodiment, using the same data set and training process, four groups of experiments are designed to explore the effectiveness of the feature alignment strategy: full feature alignment, first two layers of fusion feature alignment, first three layers of fusion feature alignment, last two layers of fusion feature alignment, and last three layers of fusion feature alignment proposed in this embodiment. Then a group of comparative experiments are designed to explore the superiority of the aggregation-semantic feature alignment proposed in this embodiment compared to the method of aligning only the aggregation features and only the semantic features.

[0112] (6) Experimental results and analysis

[0113] (6.1) Fundus image segmentation experiment analysis

[0114] Specifically, experiments were conducted using public fundus image datasets, including Drishti-GS, RIM-ONE_r3, and REFUGE challenge training sets. In order to verify the segmentation performance of the model on the target domain dataset when no target domain labels are available, two domain transfer situations were designed, including (1) setting the challenge training set as the source domain dataset and the dataset as the target domain dataset; (2) setting the challenge training set as the source domain dataset and the dataset as the target domain dataset. In order to verify the effectiveness of the model, algorithm comparison experiments were conducted to compare it with other unsupervised domain adaptation methods to test whether the MFAD method proposed in this embodiment achieves the best results compared with similar methods. The BEAL method, AdvEnt method, ISFA method, pOSAL method, and SIFA method were used as comparison algorithms. Among them, BEAL is a model that solves active unsupervised domain adaptation through adversarial learning and boundary prediction, AdvEnt is a model that solves active unsupervised domain adaptation through entropy consistency between source and target domains, ISFA is a method that uses GAN to generate synthetic images through feature alignment, PatchGAN is a method that aligns features, and pOSAL is a method that uses collaborative image and feature alignment. Among them, adding Baseline represents a method of direct migration of the target domain results predicted by the baseline Resnet34 model trained directly from the source domain data. For fairness, the evaluation criteria for measuring the segmentation effect mentioned above are combined to take the average of the five running results for each method.

[0115] Table 2 shows the experimental results of different algorithms on the REFUGE challenge training set → Drishti-GS dataset. From Table 2, we can see the experimental results of different algorithms on the migration of the REFUGE challenge training set to the Drishti-GS dataset. Although the gap with several other suboptimal models is small, the MFAD proposed in this embodiment is the best performer. The method not only obtains the highest Dice coefficients for the optic cup and optic disc, but also has the lowest ASD distance. The method achieved a coefficient of 86.82 in the detection of the optic cup, which is only 0.07 percentage points higher than the 86.75 of the suboptimal ISFA method, but 4.07 percentage points higher than the weakest performing AdvEnt method (82.75). The score of the optic cup is 10.81, which is not only lower than the score of the BEAL method (11.32), which performs best among other methods, but also decreases by 0.51, and is significantly better than the worst performing pOSAL method (14.18), with a gap of 3.37. For the coefficient of the optic disc, the MFAD method also achieved the highest score with 96.81, which is 0.13 percentage points higher than the second-best ISFA method (96.68) and significantly exceeds the worst performing pOSAL method (94.51) among all algorithms by 2.3 percentage points. In the ASD of the optic disc, the MFAD method also achieved the lowest score with 4.08, which is 0.2 lower than the closest BEAL method (4.28) and 2.04 lower than the worst method pOSAL (6.12).

[0116] Table 2 Comparison of results of different algorithms on REFUGE challenge training set → Drishti-GS dataset;

[0117]

[0118] Table 3 shows the experimental results of different algorithms on the REFUGE challenge training set → RIM-ONE_r3 dataset. From Table 3, we can see the experimental results of different algorithms on the migration of the REFUGE challenge training set to the RIM-ONE_r3 dataset. As can be seen from Table 3 and Table 2, the overall effect of the model in migrating to the RIM-ONE_r3 dataset is worse than that in migrating to the Drishti-GS dataset. This is because the domain offset of the model on the RIM-ONE_r3 dataset is larger than that on the Drishti-GS dataset, resulting in worse segmentation effect. However, the MFAD method proposed in this embodiment still has the highest overall effect in the experiment of migrating the REFUGE challenge training set to the RIM-ONE_r3 dataset. The MFAD method achieved the highest Dice coefficient of 81.58 in the optic cup, which is 0.31 percentage points higher than the ISFA method, and the worst-performing AdvEnt method is 3.59 percentage points higher. In the Dice coefficient of the optic disc, the MFAD method also achieved the highest score of 91.16, which was 0.43 percentage points higher than the second-best ISFA method and 4.64 percentage points higher than the worst pOSAL method. For the evaluation of ASD, the MFAD method also performed best in the prediction accuracy of the optic cup and optic disc, reaching 7.12 and 9.32 respectively. In the prediction of the optic cup, its performance was 0.45 lower than the AdvEnt method, and in the prediction of the optic disc boundary, it was 0.2 lower than the ISFA method.

[0119] Table 3 Comparison of results of different algorithms on REFUGE challenge training set → RIM-ONE_r3 dataset;

[0120]

[0121] (6.2) Experimental analysis of prostate image segmentation

[0122] To further verify the performance, this experiment was conducted on three prostate image datasets: NCI-ISBI 2013, I2CVB, and PROMISE12. In order to verify the segmentation performance of the model on the target domain dataset when no target domain labels are available, two domain transfer cases were designed, including (1) I2CVB dataset as the source domain and NCI-ISBI 2013 as the target domain, and (2) I2CVB dataset as the source domain and PROMISE12 as the target domain. The algorithm comparison is still the same as before, and the BEAL method, AdvEnt method, ISFA method, pOSAL method, and SIFA method are used as comparison algorithms. Among them, Baseline is also added to represent the results of direct transfer. Based on the evaluation criteria for measuring the segmentation effect mentioned above, the average of the five running results is taken for each method.

[0123] Table 4 shows the experimental results of different algorithms on the I2CVB dataset → NCI-ISBI 2013 dataset. It can be seen that the Dice score and ASD distance of the MFAD method proposed in this embodiment are significantly better than those of other methods. The Dice score is 1.18 higher than the second-best BEAL method and 5.78 higher than the worst AdvEnt method. The distance is 1.05 lower than the second-best ISFA method and 3.76 lower than the worst AdvEnt method.

[0124] Table 4 Comparison of results of different algorithms on I2CVB dataset → NCI-ISBI 2013 dataset;

[0125]

[0126] Table 5 shows the experimental results of different algorithms on the I2CVB dataset → PROMISE12 dataset. The scores and distances of the method proposed in this embodiment are also significantly better than those of other methods. The Dice score is 1.6 higher than that of the second-best ISFA method, and the distance is 1.08 lower than that of the second-best ISFA method.

[0127] Table 5 Comparison of results of different algorithms on I2CVB dataset → ROMISE12 dataset;

[0128]

[0129] It can be seen that in the case of domain adaptation from I2CVB dataset to PROMISE12 dataset, the segmentation performance of all methods, including the MFAD method, is worse than that of the domain adaptation from I2CVB dataset to NCI-ISBI 2013. It can be seen from the Baseline that the score and distance of the model trained from the I2CVB dataset directly migrated to the NCI-ISBI 2013 dataset are lower than those directly migrated to the ROMISE12 dataset, which indicates that the domain shift of the I2CVB dataset to the ROMISE12 dataset is larger than that of the NCI-ISBI 2013 dataset. However, the score and distance of the method proposed in this embodiment in migrating to the ROMISE12 data are close to the effect of migrating to the NCI-ISBI 2013 dataset. This shows that the method proposed in this embodiment can effectively align features and shorten domain shift.

[0130] Figure 6 and Figure 7The experimental results of different algorithms in the fundus image and prostate image SFUDA cases are shown respectively. It can be seen from the figure that the segmentation results obtained by the Baseline method of direct migration not only have poor shape segmentation effects due to domain offset, but also have rough boundaries, and even have small erroneous segmentation areas. Other comparative methods have significant improvements in shape and boundaries, but they all have some shortcomings. For example, the segmentation boundary of the SIFA method is still very rough, and the shape segmentation effect of the AdvEnt method is not good. The MFAD method proposed in this embodiment not only has smooth segmentation edges, but also is very close to the true label in shape. At the same time, it can be seen that the segmentation effect of the optic disc will be better than that of the optic cup, because the segmentation of the optic cup is often more difficult than that of the optic disc, and the segmentation effect of the optic cup by the MFAD method is also very close to the true label. This is because the MFAD method can fully align the shape semantic features and low-level boundary features of the source domain and the target domain, thereby ensuring that the segmentation results are close to the true label in shape and boundary.

[0131] Based on the above comparative experiments, the MFAD method of this embodiment performs well in the unsupervised domain adaptation scenario on the fundus image dataset and the prostate image dataset, and has a certain improvement in Dice score and ASD distance compared with other methods. The superiority of the segmentation results of the MFAD method can also be seen from the visualized segmentation image. This shows that the MFAD method makes full use of the advantages of multi-scale features and adversarial learning to effectively solve the UDA problem in medical image segmentation.

[0132] (6.3) Ablation experiment

[0133] In this experiment, an ablation experiment was conducted on the UDA case of the REFUGE challenge training set → Drishti-GS dataset to verify the effectiveness of the MFDA strategy (multi-scale feature fusion strategy based on dual attention) and the AGLA method (aggregation-semantic feature alignment method based on adversarial learning) proposed in this embodiment.

[0134] Table 6 Comparison of results in ablation experiments on the REFUGE challenge training set → Drishti-GS dataset;

[0135]

[0136] Table 6 shows the segmentation results on the REFUGE challenge training set → Drishti-GS dataset under different strategies. As can be seen from the table, compared with the Baseline method of direct migration, the MFDA strategy and the AGLA method have greatly improved the segmentation effect. This is because the MFDA strategy improves the segmentation effect by combining low-level features and high-level features, and the AGLA improves the segmentation effect by aligning the source domain and target domain features. It can be seen that the AGLA method brings a greater improvement than the MFDA strategy. This is because the segmentation effect is mainly reduced due to domain offset in the case of domain adaptation, while the MFDA method shortens the domain difference through feature alignment, so it can improve the segmentation effect. The MFAD method (i.e., Baseline+MFDA+AGLA) proposed in the embodiment combines the advantages of MFDA and AGLA, greatly improves the Baseline, and achieves good segmentation performance. MFDA mainly improves the model's ability to recognize segmentation boundaries, while AGLA improves the model's ability to recognize segmentation shapes. The MFAD method that combines the advantages of MFDA and AGLA can accurately recognize segmentation shapes and boundaries.

[0137] (6.4) Other experiments

[0138] The effectiveness of the last three layers of aggregated feature alignment of the feature alignment strategy proposed in this embodiment is tested as follows. Five groups of experiments are set up, namely, full feature fusion alignment, first two layers of feature fusion alignment, first three layers of feature fusion alignment, last two layers of feature fusion alignment, and last three layers of feature fusion alignment. In these experiments, only aggregated features are aligned without aligning semantic features. Table 7 shows the results under different feature fusion strategies.

[0139] Table 7 Comparison of results of different fusion alignment strategies on the REFUGE challenge training set → Drishti-GS dataset;

[0140]

[0141] It can be seen that full feature fusion can achieve good results, but if the last two layers of features are discarded and only the aggregated features of the first two layers of features are aligned, the model will pay too much attention to the low-level features of the data and ignore the alignment of high-level shape semantics, resulting in lower segmentation effect. However, the fusion strategy of the first three layers of features plus the third layer of features can more effectively grasp the high-level semantic features, but it still cannot achieve the effect of full feature alignment. This is because the high-level shape semantic alignment of the last layer of features is very important for domain adaptation. This is because the model focuses on aligning high-level semantic features and ignores low-level features that may affect the segmentation effect, achieving better segmentation results than the full feature fusion alignment strategy. However, aligning only the last two layers of features ignores low-level features such as segmentation boundaries. Therefore, the fusion alignment strategy of the last three layers of features proposed in this embodiment comprehensively considers low-level features and high-level features, ignores the first layer of features that will affect segmentation, and makes full use of the boundaries and semantic information of the other three layers of features to achieve the best segmentation results.

[0142] The effectiveness of the aggregation-semantic feature alignment strategy proposed in this embodiment is verified as follows: Baseline+MFDA is used as the comparison benchmark model, and the discriminator is added respectively Aligning aggregated features and discriminators Aligning semantic features and adding and Four sets of experiments were conducted to align and aggregate semantic features. Table 8 shows the results under different feature fusion strategies. Aligning aggregate features than joining The effect of aligning semantic features is even greater because By aligning low-level features with high-level features, the domain shift can be better shortened. Aligning semantic features is more about ensuring that the segmented shape is close to the actual segmented shape by aligning the shapes. It is more difficult to align areas with larger domain offsets such as boundaries.

[0143] Table 8 Comparison of results of different feature alignment strategies on the REFUGE challenge training set → Drishti-GS dataset;

[0144]

[0145] like Figure 8 As shown, this embodiment also discloses an unsupervised domain adaptation medical image segmentation device based on multi-scale features, including:

[0146] The multi-scale feature fusion segmentation network construction module 801 is used to construct a multi-scale feature fusion segmentation network based on dual attention; the multi-scale feature fusion segmentation network includes a plurality of residual modules connected in sequence, a dual attention mechanism module DAMM connected to each residual module and a multi-scale feature fusion module MFFM; the convolution layer performs initial feature extraction on the input source domain / target domain medical image, and then extracts high-level image features through each residual module to generate feature maps of different sizes; each feature map is fed to a DAMM module, and the features are refined and compressed to generate compressed feature maps of different sizes; the compressed feature maps of different sizes are aligned to the same size by upsampling; all the aligned compressed feature maps are input into an MFFM module for feature fusion, and a probability map of pixel-by-pixel prediction is obtained through a 1x1 convolution and a softmax layer to obtain semantic features; the aligned partial compressed feature maps are input into another MFFM module for feature fusion to obtain aggregated features;

[0147] The discriminator construction module 802 is used to use the multi-scale feature fusion segmentation network as a generator to construct a first discriminator and a second discriminator; use the first discriminator to align the aggregate features of the source domain medical image and the target domain medical image, and use the second discriminator to align the semantic features of the source domain medical image and the target domain medical image;

[0148] The training module 803 is used to perform training based on the generator, the first discriminator and the second discriminator to obtain a trained multi-scale feature fusion segmentation network; during the training process, the source domain data is used to optimize the mixed segmentation loss of the generator; the source domain aggregated features and the target domain aggregated features are passed to the first discriminator to optimize the first discriminator loss and the adversarial loss between the first discriminator and the generator; the source domain semantic features and the target domain semantic features are passed to the second discriminator to optimize the second discriminator loss and the adversarial loss between the second discriminator and the generator;

[0149] The segmentation module 804 is used to segment the medical image to be processed using the trained multi-scale feature fusion segmentation network.

[0150] The specific implementation of each module of an unsupervised domain adaptation medical image segmentation device based on multi-scale features is the same as the unsupervised domain adaptation medical image segmentation method based on multi-scale features, and this embodiment will not repeat the description.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unsupervised domain adaptation medical image segmentation method based on multi-scale features, characterized in that: include: A multi-scale feature fusion segmentation network construction step, constructing a multi-scale feature fusion segmentation network based on dual attention; the multi-scale feature fusion segmentation network includes a plurality of residual modules connected in sequence, a dual attention mechanism module DAMM connected to each residual module respectively, and a multi-scale feature fusion module MFFM; the convolution layer performs initial feature extraction on the input source domain / target domain medical image, and then extracts high-level image features through each residual module to generate feature maps of different sizes; Each feature map is fed into a DAMM module, and the features are finely compressed to generate compressed feature maps of different sizes; the compressed feature maps of different sizes are aligned to the same size through upsampling; all the aligned compressed feature maps are input into an MFFM module for feature fusion, and then a 1x1 convolution and softmax layer are used to obtain the pixel-by-pixel prediction probability map to obtain the semantic features; The aligned partial compressed feature maps are input into another MFFM module for feature fusion to obtain aggregated features; A first discriminator and a second discriminator construction step based on the generator, using the multi-scale feature fusion segmentation network as a generator to construct the first discriminator and the second discriminator; using the first discriminator to align the aggregate features of the source domain medical image and the target domain medical image, and using the second discriminator to align the semantic features of the source domain medical image and the target domain medical image; Generator and discriminator training steps, training is performed based on the generator, the first discriminator and the second discriminator to obtain a trained multi-scale feature fusion segmentation network; during the training process, the source domain data is used to optimize the hybrid segmentation loss of the generator; the source domain aggregated features and the target domain aggregated features are passed to the first discriminator to optimize the first discriminator loss and the adversarial loss between the first discriminator and the generator; the source domain semantic features and the target domain semantic features are passed to the second discriminator to optimize the second discriminator loss and the adversarial loss between the second discriminator and the generator; In the segmentation step, the trained multi-scale feature fusion segmentation network is used to segment the medical image to be processed.

2. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 1, characterized in that: The multi-scale feature fusion segmentation network includes four layers of residual modules, and the corresponding DAMMs include four; the partial compressed feature map is the compressed feature map output by the corresponding DAMMs of the last three layers of residual modules.

3. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 1, characterized in that: Each of the DAMMs includes a first channel attention module, a first spatial attention module, and a 1x1 convolution for compressing the number of channels; the first channel attention module is connected to the input feature map, and after processing, outputs the channel-refined feature map to the spatial attention module; After processing by the spatial attention module, the feature map refined by the spatial attention mechanism is output to 1x1 convolution for channel compression and output compressed feature map.

4. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 3, characterized in that: In the first channel attention module, the input feature map is first Through average pooling and maximum pooling operations to aggregate spatial information, two different features containing spatial information are output. and , represents the average channel pooling feature, represents the maximum channel pooling feature; Then and Feed into the MLP network and output and ,Will and Element-wise addition to get the final channel attention map ; The attention weight of each channel is obtained after normalization by the sigmoid function ; Perform a weighted operation on each channel of the input feature map through the attention weight of each channel, and finally obtain the feature map after channel refinement ;in, represents element-wise addition; represents a real number, Indicates the number of channel categories; Indicates height; Indicates width.

5. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 3, characterized in that: In the first spatial attention module, for the feature map output by the first channel attention module , first of all Perform average pooling and maximum pooling operations respectively to obtain the average spatial pooling feature map And the maximum spatial pooling feature map ; Then concat() these two feature maps in the channel dimension to get a fused feature map ,right Use a 5x5 convolution to reduce the number of channels to 1 to obtain the spatial attention map ; Spatial Attention Map The attention weight of each space is obtained after normalization by the sigmoid function ;pass right Perform weighted operations on each space to obtain the feature map after the channel space attention mechanism is refined ;Will After a 1x1 convolution, the compressed feature map is obtained; Represents element-wise addition.

6. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 1, characterized in that: The MFFM includes a second channel attention module and a second spatial attention module; let the two features to be fused be and , , ; First of all, and Performing element-wise addition, we get ; then Input the second channel attention module to get the channel attention weight , feature map after channel refinement ,in, represents element-wise multiplication; then Input the second spatial attention module to get the spatial attention weight , through the spatial attention weight get and Fusion feature map after fusion ; Fuse the newly obtained fused feature map with the next feature map until all feature maps are fused.

7. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 1, characterized in that: Hybrid Segmentation Loss , which is expressed as follows: ； ； ； in, represents the cross entropy loss; represents Dice loss; Representation sample The total number of pixels; Indicates the number of channel categories; for samples , Representation sample At the value of the i-th pixel, Representation sample The label at the ith pixel, Representation sample The binary label of channel category c at the ith pixel, Representation sample Predict the binary label for channel category c at the i-th pixel, Representation sample The predicted probability of channel category c at the i-th pixel; is the weight factor of the channel category, is the model weight, is the L2 norm of the model weight; and is an empirical parameter.

8. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 1, characterized in that: First discriminator loss and the adversarial loss between the first discriminator and the generator , which is expressed as follows: ； ； Second discriminator loss and the adversarial loss between the second discriminator and the generator : ； ； in, represents mathematical expectation; Indicates the use of source domain data to generate aggregate features , Indicates the use of target domain data to generate aggregate features , Indicates the use of source domain data to generate semantic features , Indicates the use of target domain data to generate a predicted probability map , Indicates that the source domain features are aggregated Or the target domain aggregate features Input the first discriminator, Indicates that the source domain semantic features Or the target domain semantic features Input the second discriminator, Respectively The distribution set of .

9. The unsupervised domain adaptation medical image segmentation method based on multi-scale features according to claim 1, characterized in that: The overall loss during training , which is expressed as follows: ； in, represents the hybrid segmentation loss; represents the first discriminator loss; represents the second discriminator loss; represents the adversarial loss of the first discriminator and the generator; represents the adversarial loss between the second discriminator and the generator; , and is an empirical parameter.

10. An unsupervised domain adaptation medical image segmentation device based on multi-scale features, characterized in that: include: A multi-scale feature fusion segmentation network construction module is used to construct a multi-scale feature fusion segmentation network based on dual attention; the multi-scale feature fusion segmentation network includes a plurality of residual modules connected in sequence, a dual attention mechanism module DAMM connected to each residual module, and a multi-scale feature fusion module MFFM; the convolution layer performs initial feature extraction on the input source domain / target domain medical image, and then extracts high-level image features through each residual module to generate feature maps of different sizes; Each feature map is fed into a DAMM module, and the features are finely compressed to generate compressed feature maps of different sizes; the compressed feature maps of different sizes are aligned to the same size through upsampling; all the aligned compressed feature maps are input into an MFFM module for feature fusion, and then a 1x1 convolution and softmax layer are used to obtain the pixel-by-pixel prediction probability map to obtain the semantic features; The aligned partial compressed feature maps are input into another MFFM module for feature fusion to obtain aggregated features; A discriminator construction module is used to use the multi-scale feature fusion segmentation network as a generator to construct a first discriminator and a second discriminator; use the first discriminator to align the aggregate features of the source domain medical image and the target domain medical image, and use the second discriminator to align the semantic features of the source domain medical image and the target domain medical image; A training module is used to perform training based on the generator, the first discriminator and the second discriminator to obtain a trained multi-scale feature fusion segmentation network; during the training process, the source domain data is used to optimize the mixed segmentation loss of the generator; the source domain aggregated features and the target domain aggregated features are passed to the first discriminator to optimize the first discriminator loss and the adversarial loss between the first discriminator and the generator; the source domain semantic features and the target domain semantic features are passed to the second discriminator to optimize the second discriminator loss and the adversarial loss between the second discriminator and the generator; The segmentation module is used to segment the medical image to be processed using the trained multi-scale feature fusion segmentation network.

Images