Cross-modal brain tumor image segmentation method with finite source domain label

Abstract

The invention relates to a cross-modal brain tumor image segmentation method with a finite source domain label, which is used for carrying out cross-modal image segmentation by effectively utilizing unmarked data of a source domain and a target domain under the condition that source domain labels are scarce aiming at a multi-modal brain tumor data set. The method comprises the following specific steps: constructing a source domain segmentation network, carrying out multi-scale consistency regular constraint, constructing a cross-modal adaptive network, carrying out double-circulation feature alignment through two sub-networks, then integrating the source domain segmentation network into the cross-modal adaptive network, and carrying out multi-scale consistency regular constraint on the cross-modal adaptive network. And forming a joint segmentation framework of a semi-supervised source domain and an unsupervised target domain, performing cross-modal adaptive segmentation model training optimization by using an end-to-end training mode, and testing a new target domain brain tumor image to obtain a segmentation result. The method can effectively reduce the domain difference, alleviates the problem of source domain label scarcity, and can be applied to the field of brain tumor cross-modal image segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer image processing, relates to medical image processing, and is a cross-modal brain tumor image segmentation method with limited source domain labels. Background Art

[0002] Artificial intelligence methods have been widely used in various medical image analysis tasks, and deep convolutional neural networks have shown significant performance improvements in image segmentation. Brain tumor segmentation is an important task in medical image processing. Accurately segmenting the tumor area is the key to treating brain tumors, and early diagnosis of brain tumors plays an important role in improving treatment possibilities and improving patient survival rates. However, manually segmenting brain tumors for cancer diagnosis from a large number of images of different modalities generated in clinical routine is a difficult and time-consuming task.

[0003] There are two limitations in brain tumor segmentation. On the one hand, medical image annotation is expensive and requires a lot of tedious annotation by experienced biomedical experts. On the other hand, considering that clinical diagnosis often has medical images of multiple modalities, images of different modalities are obtained through different imaging principles or acquisition parameters, and there are certain differences in appearance and data distribution. When trained on one modality and applied to another modality, the performance of the network will drop sharply due to the existence of domain transfer.

[0004] In terms of labeling costs, in order to reduce the burden of manual labeling, some methods use partial labeling information to improve the labeling efficiency of medical imaging, including self-supervised learning, weakly supervised learning, and semi-supervised learning. These methods can perform label-efficient learning on a single partially labeled dataset, but they cannot make good use of cross-domain information when using datasets from multiple different fields.

[0005] In terms of cross-domain adaptation, some deep unsupervised domain adaptation methods use adversarial learning through image-level adaptation, feature-level adaptation, and their hybrid methods to gradually narrow the domain differences between source domain images and target domain images. These unsupervised domain adaptation methods can utilize source domain data with complete annotations and a large amount of unlabeled target domain data to promote cross-modal image segmentation and alleviate the label scarcity problem on the target domain. However, they often rely on rich source domain annotations and suffer severe performance degradation when the source domain annotations are limited.

[0006] None of the above methods can simultaneously meet the needs of making full use of limited source domain annotations and making full use of shared features between different modalities, and cannot effectively overcome the two limitations of brain tumor segmentation mentioned above. Summary of the invention

[0007] The problem to be solved by the present invention is that the existing deep learning transfer learning scheme cannot meet the needs of medical image recognition scenarios and is limited by the annotation cost and data modality.

[0008] The technical solution of the present invention is: a cross-modal brain tumor image segmentation method with limited source domain labels, comprising the following steps:

[0009] 1) Obtain a multimodal brain tumor medical image dataset, including CT and MR images, as the source domain and target domain respectively, of which 20% of the source domain data are manually annotated; pre-process the labeled source domain data, unlabeled source domain data, and unlabeled target domain data given in the dataset, use all source domain data as training sets, and divide the target domain data into training sets, validation sets, and test sets according to a 6:2:2 ratio;

[0010] 2) Constructing a source domain segmentation network, which consists of an encoder E and a pixel-level classifier. The pixel-level classifier includes a decoder corresponding to the encoder. An auxiliary layer is added after each block of the decoder to form a multi-scale prediction component to obtain multi-scale predictions. The input of the source domain segmentation network training includes labeled source domain data and unlabeled source domain data. The multi-scale consistency regularization constraint is used to make the predictions of different scales consistent for a given input, and generate pseudo labels for medical images in the unlabeled source domain data.

[0011] 3) Construct a cross-modal adaptive network. The input of the cross-modal adaptive network is the source domain data and the target domain data. Through the two conversion sub-networks of the source domain and the target domain, feature alignment is performed in adversarial learning to extract domain-invariant features and solve the domain transfer problem.

[0012] 4) Integrate the source domain segmentation network in step 2) into the cross-modal adaptive network, use encoder E as a shared encoder, and then segment the source domain data and the target domain data respectively to form a joint segmentation framework of semi-supervised source domain and unsupervised target domain;

[0013] 5) Use end-to-end training to optimize the training of the joint segmentation framework and save model parameters;

[0014] 6) Use the optimized trained model to test the brain tumor images in the target domain to obtain the final segmentation results.

[0015] Furthermore, the preprocessing implementation process described in step 1) is as follows: three-dimensional brain CT and MR data are obtained as a multimodal brain tumor medical image dataset, and the brain tumor medical images are resampled, data normalized, and image expansion operations are performed, including random cropping and random flipping.

[0016] Furthermore, in the source domain segmentation network described in step 2), the labeled source domain data is used to predict features of different layers, and the features are optimized with the real labels through supervised segmentation loss, wherein the supervised segmentation loss includes cross entropy loss and Dice loss, and the supervised components of different scales are weighted;

[0017] Unlabeled source domain images do not have real labels. Multi-scale features are used to encourage predictions at different layers to remain consistent. Consistency loss is used to minimize the difference between predictions at different scales. First, the average prediction representation of different scales is calculated. Multi-scale consistency loss encourages the L2 distance between the prediction of each scale and the average prediction to be minimized, as shown below:

[0018]

[0019] in, is the original unlabeled source domain image, P avg is the average prediction of different scales, E is the shared encoder, represents the segmentation prediction of the multi-scale prediction component at scale k, α k is the weight of scale k.

[0020] Furthermore, the cross-modal adaptive network described in step 3) is composed of an encoder E with shared weights, a decoder U for both the source domain and the target domain. s and U t , image-level discriminator D for two domains s and D t , two pixel-level classifiers C s and C t , two mask discriminators and Composition: Based on a shared encoder, two conversion sub-networks are constructed. In adversarial learning, dual-loop adaptive feature alignment is performed to reduce the distribution difference between the source domain and the target domain, and domain-invariant features are extracted.

[0021] Among them, the encoder E and the target domain decoder U t Construct the target domain generator, encoder E and source domain decoder U s Construct the source domain generator, the original source domain image x s Synthesize target-like images x through the target domain generator s→t , and then reconstruct the source domain image x through the source domain generator s→t→s ,Similarly, in order to make the reconstructed source domain image and the reconstructed target domain image retain the information of the original image, a pixel-by-pixel cycle consistency loss constraint is imposed on them as the reconstruction loss;

[0022] Image-level discriminator D for source and target domains s and Dt It is used to correctly distinguish the original image, the generated cross-domain image and the reconstructed image, and perform adversarial learning with the generator, and is trained through image-level discriminant loss.

[0023] Furthermore, in the joint segmentation framework of step 4), the encoder E and the pixel-level classifier C of the source domain s Construct the source domain segmentation network, encoder E and pixel-level classifier C of the target domain t Construct the target domain segmentation network;

[0024] The segmentation loss involves labeled source domain data, unlabeled source domain data, and unlabeled target domain data. Based on the multi-scale prediction component in step 2), for labeled source domain images, the segmentation loss is a multi-scale supervised segmentation loss; for unlabeled source domain images, a multi-scale consistency loss is used;

[0025] For cross-domain images, since the source domain image x s And its synthesized class target domain image x s→t have the same segmentation mask, y s =y s→t Therefore, in the target domain, for the target-like images synthesized from the labeled source domain images, the segmentation loss is the supervision loss between the real labels; for the target-like images synthesized from the unlabeled source domain images, in order to enhance the consistency of features, hierarchical regularization is performed in the hidden feature space, and the cross-domain consistency loss is expressed as:

[0026]

[0027] in, is the original unlabeled source domain image, is the target domain image it synthesizes, K is the number of scales, α k is the weight of scale k;

[0028] Setting up the mask discriminator To correspond to x s→t or x t The output of the mask discriminator is classified To correspond to x t→s or x s The output of x is classified so that s→t The features extracted from x t The extracted features are aligned to minimize the gap between the feature distributions of the two, thereby correctly predicting the segmentation mask. The adversarial discriminant loss of the mask is also an adversarial loss at the semantic level. The mask discriminator Loss and mask discriminator Loss They are:

[0029]

[0030]

[0031] in, represents the expectation of f(x) for samples x from distribution X.

[0032] Furthermore, the end-to-end training method of step 5) is: the overall training goal of the model is to minimize the image reconstruction loss L cyc , segmentation loss L seg , image discrimination loss L adv and semantic discriminative loss The weighted sum of the weighted sum is used to determine the weight coefficient, and the optimization method and hyperparameters used in the training process are determined, including the optimizer, initial learning rate, and number of training iterations. When training the model, the model and visualization results are saved every fixed number of iterations, and the hyperparameters and other settings are adjusted according to the results to iteratively optimize the model.

[0033] Furthermore, the optimized trained model described in step 6) is used to test the brain tumor image in the target domain to obtain the final segmentation result, and the evaluation indicators such as Dice score and average surface distance ASD are output to evaluate the performance of the joint segmentation framework.

[0034] This paper proposes a cross-modal brain tumor image segmentation method with limited source domain labels, integrating semi-supervised learning into unsupervised domain adaptation when source domain labels are scarce. The source domain segmentation network is used to generate pseudo labels for unlabeled source domain images, and the distribution difference between the source domain and the target domain is reduced through an unsupervised domain adaptation model based on multiple constraints. The source domain knowledge is effectively transferred to the target domain image for segmentation. At the same time, the segmentation information of the target domain is used to constrain the generation of pseudo labels for unlabeled source domain data, thereby realizing unsupervised domain adaptive cross-modal brain tumor segmentation under the problem of limited source domain labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the process of the present invention.

[0036] Figure 2 Schematic diagram of the source domain segmentation method in the present invention.

[0037] Figure 3 Schematic diagram of cross-modal image conversion in the present invention.

[0038] Figure 4 Schematic diagram of the cross-modal adaptive segmentation network module in the present invention. DETAILED DESCRIPTION

[0039] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings.

[0040] The present invention designs a cross-modal brain tumor image segmentation method with limited source domain labels, such as Figure 1 As shown, the specific steps include:

[0041] Step (1): Preprocessing brain tumor data and constructing a dataset

[0042] Three-dimensional brain tumor CT and MR data were selected to obtain a multimodal brain tumor dataset. The target area of ​​the acquired medical image was intercepted and resampled to 96×96×96. Then the data was normalized and the pixel value of the image was scaled to the range of [0,1]. The source domain and target domain data were obtained respectively, and the training set, validation set, and test set were divided into 3:1:1. In order to improve the generalization ability of the model and avoid overfitting, the training images were randomly cropped and randomly flipped for image expansion.

[0043] Construct a dataset for formal representation, given a labeled and no label The source domain image and the unlabeled target image in is the input 3D image, x i ∈{0,1} H×W×D is the real annotation of 3D. L, N and M represent the number of annotated source images, all source images and all target images respectively. In the traditional unsupervised domain adaptation setting, rich source annotations are given, i.e., L = M. In order to reduce the annotation cost and make the method more practical and challenging, we use very limited source annotations, i.e., L < <M。

[0044] Step (2): Build a source domain segmentation network

[0045] The source domain segmentation network uses multi-scale consistency regularization constraints to encourage predictions of different scales to remain consistent for a given input. The network framework uses 3D V-Net as the backbone, consisting of an encoder E and a pixel-level classifier C of the source domain. s In order to extract hidden feature representations hierarchically, an auxiliary layer is added after each block of the decoder to form a multi-scale prediction component. Specifically, the auxiliary layer consists of an upsampling layer, a 1×1×1 convolution, and a softmax layer to obtain multi-scale predictions. It is incorporated into the optimization of the source domain segmentation model, and the losses between the prediction results and the true labels of different scales are calculated separately to expand the receptive field and fuse multi-scale features for deep supervision, such as Figure 2 As shown. For labeled source domain images Using multi-scale supervised segmentation loss for optimization, the function is defined as follows:

[0046]

[0047] Among them, L ce and L dice They are cross entropy loss and Dice loss, represents the segmentation prediction of the multi-scale prediction component at scale k, α k is the weight of scale k.

[0048] For unlabeled source domain images, multi-scale predictions are encouraged to be consistent, and the consistency loss is used to minimize the difference between predictions of different scales. The average predictions of different scales are expressed as:

[0049]

[0050] The multi-scale consistency loss encourages minimization of the L2 distance between the prediction at each scale and the average prediction, as follows:

[0051]

[0052] The network generates pseudo labels for unlabeled source domain images as follows:

[0053]

[0054] Step (3): Build a cross-modal adaptive network

[0055] The purpose of the cross-modal adaptation network is to leverage the labeled information from the source domain and the unlabeled samples from the target domain to improve the model performance on the target domain.

[0056] A dual-loop adaptive alignment network is constructed, and through two conversion sub-networks, effective feature alignment is performed in adversarial learning to extract domain-invariant features. Specifically, the network mainly consists of a shared-weight encoder E, two domain-specific decoders U s and U t , two domain-specific image-level discriminators D s and D t , two pixel-level classifiers C s and C t , two mask discriminators and composition.

[0057] The generator and the discriminator generate adversarial networks to achieve image-to-image conversion. s ) is converted to the target domain image (x s→t ), and then reconstruct the source domain image (x s→t→s ), the same is true for the original target domain image, such as Figure 3 Specifically, the encoder E and the target domain decoder U t The target domain generator is constructed, which can transform the source domain input into the target domain distribution, generate source domain-like images and reconstruct target domain images in the target domain, namely U t (E(x s ))=x s →t , U t (E(x t→s ))=x t→s→t Similarly, the encoder E and the source domain decoder U s The source domain generator is constructed to generate target domain images and reconstruct source domain images on the source domain, namely U s (E(x t ))=x t→s , U s (E(x s→t ))=x s→t→s .

[0058] In order to encourage the reconstructed source domain image and the reconstructed target domain image to retain the information of the original image, a pixel-by-pixel cycle consistency loss constraint is imposed on them. The reconstruction loss is as follows:

[0059]

[0060]

[0061] The discriminator and the generator play a game, the goal is to correctly distinguish the original image, the generated image and the reconstructed image, and optimize through adversarial learning. t Aims to distinguish whether the image is the real target domain image x t , class target domain image x s→t Or reconstruct the target domain image x t→s→t , distinguish between the two. Discriminator D s Aims to transform the original image x s , class source domain image x t→s and reconstruct the source domain image x s→t→s Perform pairwise discrimination. The image-level discrimination loss is as follows:

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068] Step (4): Integrate the source domain segmentation network into the cross-modal adaptation network

[0069] The source domain segmentation network is integrated into the cross-modal adaptive network in an end-to-end manner to form a joint segmentation framework for the semi-supervised source domain and the unsupervised target domain. s It constitutes the source domain segmentation network, encoder E and pixel-level classifier C of the target domain t The target domain segmentation network is constructed, such as Figure 4 As shown in Figure 2. Segmentation loss includes source domain image segmentation loss and cross-domain (i.e., target domain) image segmentation loss. Since the source domain involves labeled and unlabeled data, the segmentation loss on the target domain is also divided into labeled and unlabeled data. The labeled image uses supervised segmentation loss, and the unlabeled image uses multi-scale consistency regularization constraints.

[0070] First, for labeled source images, the source domain segmentation loss is the multi-scale supervised segmentation loss; for unlabeled source images, the source domain consistency loss is the multi-scale loss, that is:

[0071]

[0072]

[0073] Secondly, taking the source domain image as an example, the converted image obtained from the generator can be brought into the data distribution of the target domain. Although there are obvious differences in the appearance of the image across domains, it should have the same structural information as the original source domain image. In other words, the source domain image x s And its synthesized class target domain image x s→t should have the same segmentation mask, i.e., y s =y s→t Therefore, in the target domain, the cross-domain supervised segmentation loss of the target-like image synthesized using the labeled source domain image is:

[0074]

[0075] in, represents the segmentation prediction of the multi-scale prediction component at scale k, α k is the weight of scale k.

[0076] The target domain classifier, like the source domain classifier, has multi-scale information and can extract hidden feature representations. For target domain images synthesized using unlabeled source domain images, in order to enhance feature consistency, hierarchical regularization is encouraged in the hidden feature space. The cross-domain consistency loss is expressed as:

[0077]

[0078] Furthermore, in order to make s→t The features extracted from x t The extracted features are aligned to minimize the gap between the feature distributions of the two, thereby correctly predicting the segmentation mask and constructing a mask discriminator To correspond to x s→t or x t Similarly, construct To correspond to x t→s or x s The output of is classified, imposing semantic discriminative constraints. The adversarial discriminative loss of the mask is as follows:

[0079]

[0080]

[0081] Step (5): Model optimization training

[0082] The cross-modal adaptive segmentation model is trained and optimized using an end-to-end training method, and the model parameters are saved. The SGD optimizer is used during training, and the initial learning rate is set to 2×10 -4 The network was trained iteratively for 20,000 cycles.

[0083] Image reconstruction loss, including source domain reconstruction loss and target domain reconstruction loss, is expressed as:

[0084]

[0085] The segmentation loss can be expressed as the weighted sum of source domain supervision loss, source domain consistency loss, cross-domain supervision loss, and cross-domain consistency loss, as shown below:

[0086]

[0087] where λ(t) is a time-dependent Gaussian weighting function that controls the supervision loss term and the consistency regularization term, defined as where t and t max Represent the current and maximum number of training times respectively.

[0088] Image discrimination loss can be expressed as:

[0089]

[0090] The semantic discriminative loss, i.e., the adversarial discriminative loss of the mask, is expressed as:

[0091]

[0092] The overall training objective loss is defined as follows:

[0093]

[0094] Among them, the weight coefficients of segmentation loss and generation loss in image space are both 1.0, and the weight coefficients of image discrimination and semantic space discrimination loss are 0.1, that is, λ cyc =λ seg =1,

[0095] A multi-scale hierarchical classifier mechanism is used in the segmentation network, and K is set to 4. The scale weight {α k ,k=0,…,3} is used for supervised loss and unsupervised consistency loss, specified as {0.5, 0.4, 0.05, 0.05}.

[0096] Step (6): Conduct cross-domain image segmentation test on brain tumors

[0097] Using the optimized trained model, the brain tumor images in the target domain are tested to obtain the final segmentation results, and the Dice score and average surface distance ASD and other evaluation indicators are output. Dice is used to measure the overlap ratio between the true label and the predicted segmentation results. The average surface distance ASD is the average distance between all the edge points of the predicted segmentation results and all the edge points of the true label. The higher the Dice and the lower the ASD, the higher the segmentation accuracy.

[0098] The above is a detailed introduction to a cross-modal brain tumor image segmentation method with limited source domain labels provided by the present invention. It is worth noting that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, they can still modify the technical details recorded in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. Any equivalent replacements, modifications, etc. made within the core ideas and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cross-modal brain tumor image segmentation method with limited source domain labels, characterized by The steps include: 1) Obtain a multimodal brain tumor medical image dataset, including CT and MR images, as the source domain and target domain respectively, of which 20% of the source domain data are manually annotated; pre-process the labeled source domain data, unlabeled source domain data, and unlabeled target domain data given in the dataset, use all source domain data as training sets, and divide the target domain data into training sets, validation sets, and test sets according to a 6:2:2 ratio; 2) Constructing a source domain segmentation network, which consists of an encoder E and a pixel-level classifier. The pixel-level classifier includes a decoder corresponding to the encoder. An auxiliary layer is added after each block of the decoder to form a multi-scale prediction component to obtain multi-scale predictions. The input of the source domain segmentation network training includes labeled source domain data and unlabeled source domain data. The multi-scale consistency regularization constraint is used to make the predictions of different scales consistent for a given input, and generate pseudo labels for medical images in the unlabeled source domain data. 3) Construct a cross-modal adaptive network. The input of the cross-modal adaptive network is the source domain data and the target domain data. Through the two conversion sub-networks of the source domain and the target domain, feature alignment is performed in adversarial learning to extract domain-invariant features and solve the domain transfer problem. 4) Integrate the source domain segmentation network in step 2) into the cross-modal adaptive network, use encoder E as a shared encoder, and then segment the source domain data and the target domain data respectively to form a joint segmentation framework of semi-supervised source domain and unsupervised target domain; 5) Use end-to-end training to optimize the training of the joint segmentation framework and save model parameters; 6) Use the optimized trained model to test the brain tumor images in the target domain to obtain the final segmentation results.

2. According to the cross-modal brain tumor segmentation method with limited source domain labels according to claim 1, it is characterized in that the preprocessing implementation process described in step 1) is as follows: three-dimensional brain CT and MR data are obtained as a multimodal brain tumor medical image dataset, and the brain tumor medical images are resampled, data normalized, and image expansion operations are performed, including random cropping and random flipping.

3. The cross-modal brain tumor segmentation method with limited source domain labels according to claim 1, characterized in that In the source domain segmentation network described in step 2), the labeled source domain data is used to predict the features of different layers, and the predicted features are optimized with the real labels through the supervised segmentation loss, which includes the cross entropy loss and the Dice loss, and the supervised components of different scales are weighted; Unlabeled source domain images do not have real labels. Multi-scale features are used to encourage predictions at different layers to remain consistent. Consistency loss is used to minimize the difference between predictions at different scales. First, the average prediction representation of different scales is calculated. Multi-scale consistency loss encourages the L2 distance between the prediction of each scale and the average prediction to be minimized, as shown below: in, is the original unlabeled source domain image, P avg is the average prediction of different scales, E is the shared encoder, represents the segmentation prediction of the multi-scale prediction component at scale k, α k is the weight of scale k.

4. The cross-modal brain tumor segmentation method with limited source domain labels according to claim 1, characterized in that The cross-modal adaptive network described in step 3) consists of an encoder E with shared weights, a decoder U for both the source domain and the target domain s and U t , image-level discriminator D for two domains s and D t , two pixel-level classifiers C s and C t , two mask discriminators and Composition: Based on a shared encoder, two conversion sub-networks are constructed. In adversarial learning, dual-loop adaptive feature alignment is performed to reduce the distribution difference between the source domain and the target domain, and domain-invariant features are extracted. Among them, the encoder E and the target domain decoder U t Construct the target domain generator, encoder E and source domain decoder U s Construct the source domain generator, the original source domain image x s Synthesize target-like images x through the target domain generator s→t , and then reconstruct the source domain image x through the source domain generator s→t→s ,Similarly, in order to make the reconstructed source domain image and the reconstructed target domain image retain the information of the original image, a pixel-by-pixel cycle consistency loss constraint is imposed on them as the reconstruction loss; Image-level discriminator D for source and target domains s and D t It is used to correctly distinguish the original image, the generated cross-domain image and the reconstructed image, and perform adversarial learning with the generator, and is trained through image-level discriminant loss.

5. The cross-modal brain tumor segmentation method with limited source domain labels according to claim 1, The feature is that in the joint segmentation framework of step 4), the encoder E and the pixel-level classifier C of the source domain s Construct the source domain segmentation network, encoder E and pixel-level classifier C of the target domain t Construct the target domain segmentation network; The segmentation loss involves labeled source domain data, unlabeled source domain data, and unlabeled target domain data. Based on the multi-scale prediction component in step 2), for labeled source domain images, the segmentation loss is a multi-scale supervised segmentation loss; for unlabeled source domain images, a multi-scale consistency loss is used; For cross-domain images, since the source domain image x s And its synthesized target domain image x s→t have the same segmentation mask, y s =y s→t Therefore, in the target domain, for the target-like images synthesized from the labeled source domain images, the segmentation loss is the supervision loss between the real labels; for the target-like images synthesized from the unlabeled source domain images, in order to enhance the consistency of features, hierarchical regularization is performed in the hidden feature space, and the cross-domain consistency loss is expressed as: in, is the original unlabeled source domain image, is the target domain image it synthesizes, K is the number of scales, α k is the weight of scale k; Setting up the mask discriminator To correspond to x s→t or x t The output of the mask discriminator is classified To correspond to x t →s or x s The output of x is classified so that s→t The features extracted from x t The extracted features are aligned to minimize the gap between the feature distributions of the two, thereby correctly predicting the segmentation mask. The adversarial discriminant loss of the mask is also an adversarial loss at the semantic level. The mask discriminator Loss and mask discriminator Loss They are: in, represents the expectation of f(x) for samples x from distribution X.

6. The cross-modal brain tumor segmentation method with limited source domain labels according to claim 1, characterized in that The end-to-end training method of step 5) is: the overall training goal of the model is to minimize the image reconstruction loss L cyc , segmentation loss L seg , image discrimination loss L adv and semantic discriminative loss The weighted sum of , determines the weight coefficient, and determines the optimization method and hyperparameters used in the training process, including the optimizer, initial learning rate, and number of training iterations; When training a model, save the model and visualize the results every fixed number of iterations, adjust hyperparameters and other settings based on the results, and iteratively optimize the model.

7. The cross-modal brain tumor segmentation method with limited source domain labels according to claim 1, characterized in that In step 6), the optimized trained model is used to test the brain tumor image in the target domain to obtain the final segmentation result, and the evaluation indicators such as Dice score and average surface distance ASD are output to evaluate the performance of the joint segmentation framework.

Images