Cross-modality medical image segmentation method based on soft sharing collaborative adaptation

Abstract

The application discloses a cross-modal domain adaptation medical image segmentation method based on soft sharing cooperative adaptation, and belongs to the technical field of medical image segmentation. The application comprises the following steps: collecting multi-modal medical image data of a patient, performing data labeling on source domain data, pre-processing the medical image data, customizing a modal correlation model based on a soft sharing strategy, aligning the data distribution between modes based on a cooperative adaptation strategy, training a medical image segmentation model by using the source domain data label, and segmenting target domain modal medical images. The application improves the extraction capability of the model for the homogeneous features of similar structures under different modes through the soft sharing strategy; the data distribution alignment is realized by introducing a field reconstructor to reconstruct the source domain and target domain data at the data level. The feature alignment is realized by minimizing the distribution difference of the source domain and target domain features, so that the data distribution of the source domain and the target domain is closer, and the generalization capability of the model on the target domain is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical image segmentation, and particularly relates to a cross-modal domain adaptation medical image segmentation method based on soft shared synergistic adaptation. BACKGROUND

[0002] In order to more efficiently perform disease diagnosis and treatment planning, modern medicine uses different imaging technology principles to obtain multiple types of medical images to obtain comprehensive patient pathological information. The pathological information reflected by these medical images has different emphases, for example, CT images can provide information about anatomical structures, and MRI (magnetic resonance imaging) images can provide information about soft tissues. Since doctors often need to use multiple types of medical images to diagnose and treat diseases and plan treatments, and due to the limitation of human cost, it is not possible to timely annotate the data, so cross-domain segmentation for different modal medical images has strong practical significance.

[0003] Medical image segmentation algorithms based on traditional unsupervised domain adaptation can effectively solve the distribution shift problem in general cross-domain scenarios using labeled source domain data and unlabeled target domain data. These methods can effectively overcome the difference in data distribution, largely because the source domain data and the target domain data belong to the same modality and have similar information structures. In this case, traditional unsupervised domain adaptation methods can extract homogeneous features between domains at zero cost without changing the model structure, and then implement cross-domain knowledge transfer. However, there are significant differences and heterogeneity between different types of medical images, making it difficult for traditional unsupervised domain adaptation methods to effectively extract shared features between modalities, and unable to solve the distribution shift problem, resulting in a serious performance decline.

[0004] Currently, cross-modal domain adaptation medical image segmentation research mainly attempts from two aspects of style transfer and feature and image based synergistic adaptation. The style transfer approach mainly converts source domain images into target domain modal image styles through technologies such as Cycle Generative Adversarial Network (CycleGAN), and then uses a unified feature extraction module to extract deep features; the synergistic adaptation integrates style transfer and feature alignment strategies, i.e., aligning the data distribution of the source domain and the target domain in the feature space and the image space at the same time, to alleviate the distribution shift problem caused by different modalities. Most existing researches use the feature and image synergistic adaptation method to implement the cross-modal medical image segmentation model.

[0005] For style transfer based cross-modality medical image segmentation algorithms, Pei et al. proposed a framework based on disentangle domain features (DDFseg). The framework first decomposes each domain into domain-invariant features and domain-specific features, enhances the domain-invariant feature representation using a self-attention module in the encoder, minimizes the loss to approach zero the domain-specific information in the source and target domain images, then iteratively encodes and decodes these features to maintain the consistency of the anatomical structure, and introduces multiple discriminators to improve the quality of the generated images and segmentation results through adversarial learning, ultimately achieving cardiac structure segmentation. Dou et al. proposed a plug-and-play adversarial network (PnP-AdaNet) framework. The framework learns a domain adaptation module through adversarial loss to replace the early encoder layers of the source network flexibly, and the higher layers are shared between the two domains, aligning the feature spaces of the source and target domains at multiple scales to address the significant inter-domain distribution shift. However, due to the instability of the generative adversarial network, semantic information may be destroyed when generating cross-domain images.

[0006] Collaborative adaptation based cross-modality domain adaptation methods usually use image reconstruction to convert image styles and extract shared features. For example, Chen et al. proposed a synergistic image and feature adaptation (SIFA) framework. The framework uses a cross-domain conversion method to enhance image appearance and improve the domain invariance of features extracted from cross-domain images in the segmentation task. In addition, the framework shares the feature encoder layers from two angles to jointly master their advantages and uses adversarial loss to guide the learning of the unified model, achieving cross-modality segmentation of the heart. Han et al. proposed a deep symmetric adaptation network (DSAN) framework for cross-modality segmentation of cardiac structures and brain tumors. The framework consists of a segmentation subnetwork and two symmetric source and target domain conversion subnetworks, achieving feature alignment from the source domain to the target domain and from the target domain to the source domain, and training a pixel-level classifier using the original source image and the reconstructed target image to fully utilize the semantic information from different style images. However, since existing methods use hard sharing (Hard Shared) to extract homogeneous features from both modalities, modal information loss inevitably occurs, and modal-related noise is introduced, leading to ineffective alignment of inter-modal data distribution and degradation of model performance. SUMMARY

[0007] The application aims to provide a cross-modal medical image segmentation method based on soft sharing cooperative adaptation, which can extract inter-modal homogeneous features, align data distribution of source domain and target domain through bidirectional domain adaptation, and obtain a precise and efficient cross-modal medical image segmentation model.

[0008] The technical scheme adopted by the application is as follows:

[0009] A cross-modal medical image segmentation method based on soft sharing cooperative adaptation, comprising the following steps:

[0010] Step 1, collecting multi-modal medical images of patients, and performing data labeling on the source domain images in the multi-modal medical images to obtain image segmentation labels of the source domain images;

[0011] Step 2, pre-processing the medical images;

[0012] Step 3, constructing a cross-modal medical image segmentation model;

[0013] The cross-modal medical image segmentation model comprises a source domain encoder E s , a target domain encoder E t , a shared encoding layer E z , a shared layer R z , a source domain reconstructor R s , a target domain reconstructor R t , a source domain discriminator D s , a target domain discriminator D t , and a segmentation subnetwork.

[0014] Among them,

[0015] The source domain encoder E s is used to extract image features of the source domain images, the target domain encoder E t is used to extract image features of the target domain images, and parameters of the source domain encoder E s and the target domain encoder E t meet L2 norm constraints.

[0016] The input of the shared encoding layer E z is the output of the source domain encoder E s and the target domain encoder E t , which is used to extract shallow shared features in the source domain and target domain images.

[0017] The input of the shared layer R z is the output of the shared encoding layer E z , which is used to extract deep homogeneous features in the source domain and target domain images.

[0018] The source domain reconstructor R s , the target domain reconstructor Rt The inputs to both the split subnetwork and the shared layer R are... z The output of, where the source domain reconstructor R s The target domain reconstructor R is used to map the input feature map to the source domain to generate the corresponding reconstructed image. t The segmentation subnetwork is used to map the input feature map to the target domain to generate the corresponding reconstructed image, and is used to output the pixel-level segmentation results of the image.

[0019] Source Domain Discriminator D s With source domain reconstructor R s Connected, used to determine R s The authenticity of the output reconstructed image compared to the corresponding source image;

[0020] Target Domain Discriminator D t With the target domain reconstructor R t Connected, used to determine R t The authenticity of the output reconstructed image compared to the corresponding target domain image;

[0021] Step 4: Train a cross-modal medical image segmentation model;

[0022] The segmentation sub-network is pre-trained based on the source domain image and its data annotations so that the model can initially acquire the ability to segment medical images.

[0023] During training, the source domain image is input into the source domain encoder E. s Then through the shared coding layer E z and shared layer R z After mapping, the input is fed into the segmentation sub-network, which is pre-trained based on the semantic segmentation loss and the soft-shared loss composed of the parameter constraints of the two encoders; wherein, the semantic segmentation loss can be obtained by combining Dice loss (a type of semantic segmentation loss) and cross-entropy loss;

[0024] The cross-modal medical image segmentation model is then trained based on soft-shared loss, reconstruction loss, and adversarial loss. When the preset training convergence condition is met (the number of training iterations reaches a specified limit or the total loss composed of soft-shared loss, reconstruction loss, and adversarial loss converges), the target domain encoder E is then trained. t Shared coding layer E z Shared layer R z The convergent target domain model is obtained by dividing the subnetwork;

[0025] Step 5: Obtain the medical image segmentation results of the target domain modality based on the target domain model.

[0026] The technical solution provided by this invention brings at least the following beneficial effects:

[0027] (1) The application introduces a soft sharing strategy, promotes inter-modal feature extraction by adding a sharing module to the model, improves the extraction ability of the model for homogeneous features of similar structures under different modalities, and improves the alignment of data distribution.

[0028] (2) The application also uses a cooperative adaptation technology to align the data distribution in the data space and the feature space. In the data space, the alignment of the data distribution is realized by introducing a domain reconstructor to reconstruct the source domain and the target domain data at the data level. In the feature space, the alignment of the features is realized by minimizing the distribution difference of the source domain and the target domain features, so that the data distribution of the source domain and the target domain is closer, and the generalization ability of the model in the target domain is improved. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0030] Figure 1 is a structural schematic diagram of a medical image segmentation model used in the embodiments of the application.

[0031] Figure 2 is a flowchart of a cross-modal medical image segmentation method based on soft sharing and cooperative adaptation provided by the embodiments of the application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the application more clear, the embodiments of the application will be further described in detail with reference to the drawings.

[0033] Recently, soft shared technology is often used to solve the problem of inter-modal information capture. The idea is to consider that the difference between modal data is too large, and the traditional hard shared model cannot perfectly capture the shared information between modal according to the characteristics of the modal, and is easy to be disturbed by modal related noise, so the soft shared way is adopted to customize the modal related feature extraction model, and the inter-modal homogeneous feature extraction ability of the model is enhanced. For example, Wang et al. proposed a transferable couple network (TCN) to solve the problem of image retrieval based on zero sample sketch. In this framework, the model parameter soft sharing method is adopted to extract the deep features of hand-drawn sketches and real images respectively. Based on this technology, the model can better adapt to the difference between modal, and can also maintain better stability and robustness in the presence of modal related noise. Therefore, based on the soft shared technology, the homogeneous feature extraction between different modal medical images is realized, which is the key to realize high-precision cross-modal medical image segmentation.

[0034] The embodiment of the application provides a cross-modal medical image segmentation method based on soft shared collaborative adaptation, which is based on soft shared technology and collaborative adaptation strategy, and can extract inter-modal homogeneous features under the premise of aligning the data distribution of the source domain and the target domain through bidirectional domain adaptation, so as to obtain a precise and efficient cross-modal medical image segmentation model.

[0035] The embodiment of the application adopts a model soft sharing based strategy to deeply mine inter-modal homogeneous information. Specifically, given the source domain data and target domain data of different modal, their deep features can be represented as:

[0036] F S =E S (S;θ S ),

[0037] F T =E T (T;θ T ),

[0038] Wherein, E S is a source domain feature extractor, and its parameter is represented as θ S , E T is a target domain feature extractor, and its parameter is represented as θ T , F S and F T represent the feature representation of the source domain data and the target domain data extracted in the last convolution stage.

[0039] The soft shared homogeneous feature encoder of the source domain and the target domain can be described as follows:

[0040]

[0041] where, and are the parameters of the l-th layer of the source domain encoder E S and target domain encoder E T respectively. By comparing their Euclidean distance to promote the target domain encoder to model the target domain data guided by the source domain parameters. Meanwhile, in order to separate the modality general information and modality specific information, an independent BatchNorm layer is adopted for cross-modal retrieval. If the l-th layer of the source domain or target domain feature encoder is not a BatchNorm layer, the indicator function equals 1, otherwise 0.

[0042] The reason for this setting is that in the convolutional neural network model, the design of the convolutional layer aims to mimic the primary visual cortex in the human brain by utilizing the ideas of sparse interaction, parameter sharing and equivalent representation; the pooling layer aims to gradually capture the edges, textures and parts of objects, so as to discover the patterns of the input image data. The soft sharing of these two parameters helps to extract shallow shared features. The BatchNorm layer contains the calculation and memory of batch statistics, such as mean and variance, which can describe the relevance of the input modalities, and the parameter hard sharing can effectively mine the deep homogeneous information. Given the parameter sets θ s and θ t of the source domain and target domain feature encoders, the data label y s on the source domain, the soft sharing loss coefficient λ, combined with the medical image segmentation task, the optimization objective can be rewritten as:

[0043]

[0044] where, denotes the objective function obtained by combining the Dice loss and the cross-entropy loss, which aims to optimize the model using the source domain medical image and its corresponding data label to enable the model to have the ability of medical image segmentation.

[0045] The following compares and analyzes the parameter soft sharing and hard sharing methods.

[0046] For the hard sharing method, there is θ = θ s = θ t , i.e. Therefore, the loss for θ can be expressed as:

[0047]

[0048] This will cause the feature encoder of the target domain to be dominated by the source domain modality, because there is no target domain information in the constraint equation, resulting in an unbalanced training process relative to the two domains.

[0049] For the soft sharing mode, the θ s related gradient can be described as: t

[0050]

[0051]

[0052] The parameter-based soft sharing mode can achieve a better trade-off between the source domain and the target domain, thereby effectively alleviating the learning imbalance problem.

[0053] Specifically, while the source domain model tends to learn parameters suitable for the source domain modality at each optimization step, the target domain model has a certain degree of freedom to learn parameters suitable for the target domain modality. Therefore, the soft sharing loss generated by the parameter difference will prevent the source domain model from learning too many parameters from the source domain modality. Based on the above soft sharing strategy, homogeneous information between different modalities can be effectively extracted, the cross-modality data information heterogeneity problem can be solved, more effective deep features can be generated, and the understanding and application ability of multi-modality data can be improved, which provides a technical basis for solving the distribution shift problem subsequently.

[0054] On the basis of having the ability to extract homogeneous features between modalities, the embodiment of the application adopts a collaborative adaptation strategy to align the distribution of the feature space, and further enhances the homogeneous feature extraction capability of the model. Collaborative adaptation has two meanings, one is the collaborative alignment of the feature space and the data space, that is, aligning the data distribution in the feature space and the data space at the same time; the other is the collaborative alignment of the source domain and the target domain, that is, simultaneously performing bidirectional alignment from the source domain to the target domain and from the target domain to the source domain.

[0055] Based on the adversarial loss, the embodiment of the application performs bidirectional reconstruction of the source domain and the target domain image through two sub-networks in the framework, and additionally introduces an image reconstruction loss to better maintain semantic information. Among them, the encoders E s 、E t based on the soft sharing strategy, the image reconstructor R s 、R t belongs to the domain private. Through the image reconstruction task, the independent decoders of the source domain and the target domain mainly focus on the modality shared information. Specifically, the encoder E s takes the source domain image x s as input, and after encoding, it is sent to the shared encoding layer E z , and then after mapping through the shared layer R z , it is sent to the reconstructor R s 、R t ​followed by adversarial learning on both source and target domains, so that the reconstructor generates images similar to the opposite domain to fool the domain discriminators D s , D t As shown in FIG. 3. The domain discriminators try to distinguish the reconstructed images x s→t from the target images x t . The process is optimized by the adversarial loss as follows: Figure 1

[0056]

[0057] where L denotes the adversarial loss on the target domain, D t () denotes the output of the domain discriminator of the target domain, denote the expectations of the target domain, source domain, respectively, R t () denotes the output of the image reconstructor of the target domain, E t () denotes the encoder of the target domain. x s→t denotes the reconstructed image of the source domain mapped to the target domain.

[0058] The domain discriminators maximize the objective function during the optimization process. Meanwhile, the encoder and the image reconstructor cause confusion between the source domain images and the target domain images, so that the objective function is constantly reduced.

[0059] Similarly, when the target domain image x t is input, the encoder and the image reconstructor are also optimized by the corresponding adversarial loss, which can be represented as follows:

[0060]

[0061] where L denotes the adversarial loss on the source domain, D s () denotes the output of the domain discriminator of the source domain, s () denotes the output of the image reconstructor of the source domain, E s () denotes the encoder of the source domain. x t→s denotes the reconstructed image of the target domain mapped back to the source domain.

[0062] To ensure that the reconstructed image x s→t retains the content information of the original image and learns the style of another domain image, the embodiments of the present application optimize the encoder and the image reconstructor using the target domain reconstruction loss. For the reconstruction loss on the target domain, it can be represented as:

[0063]

[0064] ​Similarly, the source domain also optimizes its encoder and image reconstructor based on the reconstruction loss to ensure that the reconstructed image x t→s retains the necessary information, and for the reconstruction loss on the source domain, it can be expressed as:

[0065]

[0066] Based on this, the bidirectional data distribution alignment in the feature space and the data space, as well as the source domain and the target domain can be realized, and the optimization target can be expressed as:

[0067]

[0068] where λ' and φ are the weight factors of the adversarial loss and the reconstruction loss, respectively. The embodiments of the present application can effectively capture the inter-modal homogeneous features while ignoring the differences in the inter-modal information structure, and overcome the data distribution difference to align the data distribution of the source domain and the target domain.

[0069] Next, the reconstructed image x s→t and the original source domain image x s are needed to train the segmentation subnetwork of the cross-modal segmentation model. Since the reconstructed image x s→t is aligned with the target domain in the data space and obtains homogeneous features after being encoded by the encoder, the supervised training based on this data can effectively enable the model to obtain cross-modal segmentation capability:

[0070]

[0071] wherein, represents the image segmentation loss (i.e., the segmentation loss shown in Figure 1 ), , represents the image segmentation loss on the source domain, represents the image segmentation loss when the input is the reconstructed image x s→t , both of which use the same segmentation subnetwork, and the difference is only in the input data; represents that the source domain image x s is input into the encoder E s , and then sequentially passes through the shared encoding layer E z and the shared layer R z to obtain the segmentation result p s of the source domain image x s ; E s→t () represents the output of the cross-domain encoder from the source domain to the target domain, i.e., the encoded features obtained by inputting the reconstructed image x s→t into the encoder E t , and then sequentially passing through the shared encoding layer E z and the shared layer Rz After mapping, the reconstructed image x is obtained by inputting into a segmentation sub-network S s→t The segmentation result p of the image s→t As shown in the following formula. Figure 1

[0072] In summary, the overall optimization goal is:

[0073]

[0074] Based on the above optimization goal, the problem of aligning the distribution of inter-modal data can be effectively solved, and the precision of the model in the cross-modal medical image segmentation task is improved.

[0075] The training process of the model can be divided into two steps, which are alternately iterated for training. First, a soft sharing strategy is used to customize the source domain and target domain encoders based on the characteristics of the modalities; then, a collaborative adaptation strategy is used for bidirectional alignment of the source domain and the target domain to train the cross-modal segmentation model.

[0076] According to the parameter set of the source domain and the target domain encoder, the soft sharing loss is calculated The model is optimized according to the loss. The purpose is to effectively extract the homogeneous features between the modalities, and to avoid the problem that existing work cannot align the distribution of inter-modal data due to ignoring the heterogeneity of modal information.

[0077] The embodiment of the application aligns the data distribution of the source domain and the target domain through adversarial training and image reconstruction. The image reconstruction module converts the image style from the data level, and is implemented by optimizing the loss function and After adversarial training, the encoder is used to align the feature distribution of the source domain and the target domain in the feature space, and is implemented by optimizing the loss function and The purpose is to further enhance the homogeneous feature extraction capability of the encoder, and to align the data distribution of the source domain and the target domain.

[0078] As a possible implementation manner, as shown in Figure 2 The cross-modal medical image segmentation method based on soft sharing and collaborative adaptation provided by the embodiment of the application specifically includes the following steps:

[0079] Step S1, collecting patient multi-modal medical image data, and performing data labeling on the source domain data;

[0080] Based on the medical imaging device, the multi-modal medical image data of a plurality of patients is collected, and the collected multi-modal medical image is manually labeled at the pixel level to obtain the labeled modal data as the source domain D S ;

[0081] ​Collecting medical image data of another modality of the same patient as the target domain D T (the data does not need to be manually labeled);

[0082] Step S2, pre-processing the medical image data;

[0083] Data pre-processing is required before sending the data into the model training to eliminate the influence of random factors on the model training;

[0084] Wherein, the source domain is denoted as D s ={x s ∈X s ,y s ∈Y s},x s represents the source domain image, y s represents the label (i.e. data annotation) of x s , and Y s ∈R H×W×C ; the target domain is denoted as D t ={x t ∈X t}, and C represents the number of classes of the source domain samples. X s represents the source domain image data space, Y s represents the source domain label space, X t represents the target domain image data space, x t represents the target domain image, and HxW represents the image size. In cross-modality medical image segmentation, the difference between the marginal probability distribution and the conditional probability distribution of the source domain and the target domain is much larger than that in the traditional unsupervised domain adaptation problem.

[0085] In this embodiment, the data pre-processing is to slice the collected medical image data of multiple modalities (source domain and target domain) to obtain two-dimensional plane images, then scale each image to a specified image size, for example, 224x224 pixel size, and map the pixel value to [0, 255] to obtain a grayscale image, use Gaussian filtering to eliminate random noise interference, and use the processed data for model training.

[0086] Step S3, customizing the modality-related model based on the soft sharing strategy;

[0087] Customizing the modality-specific feature extractor parameters based on the model soft sharing strategy, wherein the source domain feature extractor is denoted as E S , and the parameters thereof are denoted as θ S ; the target domain feature extractor is denoted as E T , and the parameters thereof are denoted as θ T , and the parameters of the two satisfy the L2 norm constraint.

[0088] Step S301: Construct a modal-related feature extraction module based on the combination of shared layers and dedicated layers. The shared layers are used to extract shallow shared features in source domain and target domain images. These features have certain similarities for images in both domains, so they can help the model capture homogeneous deep features between different modalities. Among them, the source domain deep features can be represented as F S , and the target domain deep features can be represented as F T . The dedicated layers are introduced to better mine deep homogeneous features. The dedicated layers are designed according to the characteristics of each modality, which can better learn and represent the features of the modality. Through the dedicated layer, the model can more fully utilize the unique information of each modality and conduct more in-depth analysis and modeling.

[0089] Step S302: Combine the shared layers and dedicated layers to obtain the complete source domain feature extractor E S and the target domain feature extractor E T , so that the model can consider both the sharing and differences between the source domain and the target domain when extracting features. The shallow shared features extracted by the shared layers can help the model establish a certain correspondence between the two domains, while the dedicated layers can better capture the unique features of each modality. Such a combined architecture aims to align the data distribution between modalities, so that the data of different modalities are closer in the feature space, thereby improving the performance and generalization ability of the model.

[0090] Through the above steps, the characteristics of multi-modal medical image data can be fully utilized to extract inter-modal homogeneous features and enhance the subsequent data distribution alignment effect.

[0091] Based on the characteristics of convolutional neural networks and combined with the medical image segmentation task, the model is trained by combining the target function obtained by the Dice loss and the cross-entropy loss, and a pixel-level classifier H (i.e., the segmentation subnetwork S) is obtained. The purpose is to optimize the model using source domain medical images and their corresponding data annotations to make the model initially acquire the ability of medical image segmentation and enhance the generalization performance of the model before domain adaptation.

[0092] Step S4, align the data distribution between modalities based on the collaborative adaptation strategy;

[0093] Based on the collaborative adaptation strategy, the distribution in the feature space is aligned, and the homogeneous feature extraction capability of the model is further enhanced. Collaborative adaptation aligns the data distribution from two aspects: first, the collaborative alignment of feature space and data space, that is, aligning the data distribution in both feature space and data space; second, the collaborative alignment of source domain and target domain, that is, simultaneously performing bidirectional alignment of source domain to target domain S→T and target domain to source domain T→S.

[0094] Step S401: Based on the adversarial loss, the bidirectional reconstruction of the source domain and target domain images is performed through two sub-networks, and an image reconstruction loss is additionally introduced to better maintain the semantic information. The image reconstructor R s t Belongs to the domain private. Through the image reconstruction task, the independent decoder of the source domain and the target domain mainly focuses on the modal shared information.

[0095] Specifically, the encoder E s takes the source domain image x s as input, and after encoding, it is sent to the shared encoding layer E z , and then mapped to the shared layer R z , and then sent to the reconstructor R s of the source domain and the target domain. t Then, the adversarial learning is performed in the source domain and the target domain at the same time, so that the reconstructor generates images similar to the opposite domain to deceive the domain discriminator D s , D t . While the domain discriminator tries to distinguish the reconstructed image x s→t from the target image x t .

[0096] Step S402: The source domain also optimizes its encoder and image reconstructor using the reconstruction loss to ensure that the reconstructed image x t→s retains the necessary information. Based on this, bidirectional data distribution alignment can be achieved in the feature space and the data space, as well as in the source domain and the target domain.

[0097] Step S5, train the medical image segmentation model using the source domain data label;

[0098] Based on the reconstructed image x s→t and the original source domain image x s , the segmentation sub-network (medical image segmentation model) is trained. Since the reconstructed image x s→t is aligned with the target domain in the data space, and homogeneous features are obtained after encoding by the encoder, supervised training based on this data can effectively enable the model to obtain cross-modal segmentation capability.

[0099] The data involved in the training includes: source domain medical images and target domain medical images The training goal is to obtain a target domain model (denoted as M t ).

[0100] When the cross-modal segmentation model (cross-modal medical image segmentation model) does not converge, the following calculations are performed during the training process:

[0101] Calculate the soft shared loss between the source domain image segmentation model and the target domain image segmentation model​

[0102] reconstruct the source domain and target domain images, calculate reconstruction loss and

[0103] discriminate the reconstructed images from which domain to calculate adversarial loss and

[0104] based on soft sharing loss reconstruction loss on the source domain and target domain and and adversarial loss on the source domain and target domain and the overall loss of the model can be obtained

[0105] based on the overall loss, back propagation is performed on the cross-modal segmentation model (including source domain encoder E s , target domain encoder E t , shared encoding layer E z , shared layer R z , source domain reconstructor R s , target domain reconstructor R t , source domain discriminator D s , target domain discriminator D t and segmentation subnetwork S) to update the model parameters. Wherein, E s , E t , E z , R z , R s and R t may be a network structure based on convolutional neural network. The segmentation subnetwork S can be pre-trained, i.e. pre-training based on source domain images and perform pre-training. Then, based on the overall loss of the model train the cross-modal segmentation model.

[0106] When the cross-modal medical image segmentation model converges, based on the currently trained target domain encoder E t , shared encoding layer E z , shared layer R z and segmentation subnetwork S, the converged target domain model M t is obtained.

[0107] Step S6, segmenting the target domain modality medical image, i.e. segmenting the target domain modality medical image based on the target domain model M t obtained by training in step S5.

[0108] The embodiment of the present application provides a cross-modal medical image segmentation method based on soft sharing cooperative adaptation.

[0109] Specifically, the present application introduces a soft sharing strategy to promote inter-modal feature extraction by adding a shared module to the model. This can improve the model's ability to extract homogeneous features of similar structures under different modalities, thereby improving the alignment of data distribution. In addition, the present application also uses a cooperative adaptation technology to perform bidirectional distribution alignment in data space and feature space. In the data space, a domain reconstructor is introduced to reconstruct the source domain and target domain data at the data level to achieve data distribution alignment. In the feature space, the distribution difference between the source domain and the target domain features is minimized to achieve feature alignment. This can make the data distribution of the source domain and the target domain closer, and improve the model's generalization ability in the target domain.

[0110] Compared with existing cross-modal domain adaptation medical image segmentation algorithms, the method of the present application can effectively solve the problem of data distribution alignment caused by information heterogeneity between modalities. Through the combination of soft sharing strategy and cooperative adaptation technology, the method of the present application can improve the generalization ability and segmentation accuracy of the model.

[0111] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0112] The above only describes some embodiments of the present application. For those skilled in the art, without departing from the spirit and scope of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application.

Claims

1. A cross-modal medical image segmentation method based on soft-shared collaborative adaptation, characterized in that, Includes the following steps: Step 1: Acquire multimodal medical images of the patient and perform data annotation on the source domain images to obtain image segmentation labels for the source domain images; Step 2: Preprocess the medical images; Step 3: Construct a cross-modal medical image segmentation model; The cross-modal medical image segmentation model includes: a source domain encoder. Target domain encoder Shared coding layer Shared layer Source domain reconfigurator Target domain reconstructor Source domain discriminator Target Domain Discriminator and segmented subnetworks; in, Source domain encoder Image features extracted from source domain images; target domain encoder Used to extract image features from the target domain image; and the source domain encoder and target domain encoder The parameters satisfy Normalization constraints: ； in, This represents the parameter constraints between the encoders in the source and target domains. This indicates the layer number of the encoder, where BN represents the normalization layer. and They represent the first Layer source domain encoder and target domain encoder The parameters, if the source encoder or target domain encoder The If the layer is not a batch normalized layer, then the indicator function... equal Otherwise ; Shared coding layer The input is the source domain encoder. and target domain encoder The output is used to extract shallow shared features in the source and target domain images; Shared layer The input is a shared coding layer The output is used to extract deep homogeneous features from source and target domain images; Source domain reconfigurator Target domain reconstructor The inputs to both the split subnetwork and the shared layer are shared layers. The output, where the source domain reconstructor The target domain reconstructor is used to map the input feature map to the source domain to generate the corresponding reconstructed image. The segmentation subnetwork is used to map the input feature map to the target domain to generate the corresponding reconstructed image, and is used to output the pixel-level segmentation results of the image. Source Domain Discriminator With source domain reconstructor Connected, used for discrimination The authenticity of the output reconstructed image compared to the corresponding source image; Target Domain Discriminator With target domain reconstructor Connected, used for discrimination The authenticity of the output reconstructed image compared to the corresponding target domain image; Step 4: Train a cross-modal medical image segmentation model; Input the source domain image into the source domain encoder Then through the shared coding layer and shared layer After mapping, the input is fed into the segmentation sub-network, and the segmentation sub-network is pre-trained based on the semantic segmentation loss and the soft-shared loss composed of the parameter constraints of the two encoders; The cross-modal medical image segmentation model is then trained based on the total loss function composed of soft-shared loss, reconstruction loss, and adversarial loss. When the preset training convergence condition is met, the model is then trained based on the currently trained target domain encoder. Shared coding layer Shared layer The convergent target domain model is obtained by dividing the subnetwork; Step 5: Obtain the medical image segmentation results of the target domain modality based on the target domain model.

2. The method as described in claim 1, characterized in that, In step 4, the soft-shared loss during pre-training of the segmentation subnetwork is specifically as follows: ,in, Represents semantic segmentation loss. This represents the soft-share loss coefficient.

3. The method as described in claim 1, characterized in that, The semantic segmentation loss is obtained by combining Dice loss and cross-entropy loss.

4. The method as described in claim 1, characterized in that, Step 4, after pre-training the segmentation sub-network, also includes the following when training the cross-modal medical image segmentation model: Target domain refactor Reconstructed image generated by mapping the source domain to the target domain Input target domain encoder Then through the shared coding layer and shared layer After mapping, the image is fed into the segmentation sub-network, and the reconstructed image is obtained based on its output. semantic segmentation loss; Based on source domain image semantic segmentation loss Source domain image The corresponding reconstructed image semantic segmentation loss And the parameter constraints between the encoder representing the source and target domains. Constitutes the second soft-share loss ; The total loss function when training the cross-modal medical image segmentation model is: ； in, Represents the total loss function. The loss, comprised of reconstruction loss and adversarial loss, is expressed as: ; in, Indicates adversarial losses in the target domain. Indicates the adversarial loss in the source domain. Represents the reconstruction loss over the target domain. Represents the reconstruction loss over the source domain. and These are the weighting factors for adversarial loss and reconstruction loss, respectively.

Images