Incomplete multi-mode brain tumor segmentation method based on key mode oriented contrast learning

Abstract

The invention discloses an incomplete multi-modal brain tumor segmentation method based on key modal guided contrast learning, which comprises the following steps: 1, inputting four different modal MRI images, extracting features through two groups of encoders respectively, and simulating a missing modal through modal codes to obtain FLAIR modal guided group fusion features and T1c modal guided group fusion features; 2, constructing a mode-oriented comparison branch, and shortening the distance between other modes and the key mode through comparison learning; 3, constructing a multi-modal segmentation branch, alternately training the multi-modal segmentation branch and a modal-oriented comparison branch, and optimizing a segmentation result; and 4, segmenting the input multi-modal MRI image data by using the trained incomplete multi-modal brain tumor segmentation model and discarding the modal-oriented contrast branch. According to the method, the distance between other modals and the key modals can be reduced in the scene of medical MRI image modality deficiency, and key source information is supervised and reserved through the segmentation result, so that the medical image segmentation performance can be improved.

Description

Technical Field

[0001] The present invention relates to medical image processing technology, in particular to a brain tumor segmentation method based on modality-guided contrastive learning, which is used to improve the accuracy and reliability of multimodal image segmentation of brain tumors in the absence of modalities. It belongs to the fields of computer vision, deep learning and medical image analysis. Background Art

[0002] With the rapid development of medical imaging technology, the diagnosis and treatment planning of brain tumors are increasingly dependent on high-precision brain tumor segmentation techniques. Magnetic resonance imaging (MRI), a commonly used medical imaging technique, can provide a variety of tissue contrast information and is widely used for brain tumor detection and segmentation. However, the frequent lack of different modalities in clinical scenarios (such as FLAIR, T1, T1c, and T2 modalities) poses a challenge to traditional brain tumor segmentation methods based on multimodal imaging.

[0003] Existing multimodal brain tumor segmentation methods can generally be divided into three categories:

[0004] 1. Train multiple models for different modality combinations:

[0005] This approach trains separate models for each modality combination and integrates these models to handle the missing modality problem. Although this strategy can address the missing modalities of different modalities, training multiple models is not only time-consuming and labor-intensive, but also increases the demand for computing resources and is difficult to handle the missing modality situations commonly seen in clinical practice.

[0006] 2. Generate missing modalities and perform fusion:

[0007] Another class of methods attempts to restore complete multimodal data by generating missing modalities. These methods typically employ generative models (such as generative adversarial networks) to synthesize the missing modalities. However, this approach often introduces generation errors, which can lead to spurious modal information and affect segmentation accuracy. This generation error can significantly impact the accuracy of results, particularly in tasks requiring high precision, such as brain tumor segmentation.

[0008] 3. Interaction and decoupling between modals:

[0009] In recent years, some research in the field of deep learning has begun to focus on the interaction or decoupling between modalities, aiming to improve segmentation performance by extracting modality-specific features. However, most of these methods focus on direct interactions between modalities, while ignoring the guiding role of certain modalities in tumor segmentation. For example, the FLAIR modality provides a clear contrast across the entire tumor boundary, while the T1c modality clearly reveals the tumor core and enhancing areas. Lacking information that fully exploits these modality characteristics, existing methods often fail to fully improve segmentation performance in the absence of a modality.

[0010] Although the above methods address the problem of missing modalities to some extent, they generally suffer from the following problems:

[0011] High computational complexity: Training multiple models for different modal combinations requires a lot of computing resources and cannot effectively address the missing modality problem in real-time diagnosis.

[0012] Generation error affects segmentation accuracy: The method of restoring complete data by synthesizing missing modes is prone to introducing noise. Especially for tasks such as brain tumors that require fine anatomy, generation error will lead to a decrease in segmentation accuracy.

[0013] Ignoring the guiding role of modalities: Existing methods mostly focus on the interaction between modalities, while ignoring the important guiding role of certain modalities in tumor segmentation. For example, the performance of FLAIR and T1c modalities varies significantly in different tumor regions (such as the tumor core, enhancing tumor, and surrounding edema). As a result, the model cannot fully utilize the discriminative ability of these modalities for key regions during learning. This may lead to uneven recognition of different tumor regions and reduce overall segmentation accuracy. In fact, the differences between these modalities not only reveal the characteristics of different tissue structures but also provide effective guidance for feature learning of other modalities. Summary of the Invention

[0014] In order to address the deficiencies of the above-mentioned prior art, the present invention proposes an incomplete multimodal brain tumor segmentation method based on key modality-guided contrastive learning, in order to effectively utilize the tumor-specific features of FLAIR and T1c modalities to guide the feature learning of other modalities, thereby narrowing the distance between other modalities and the FLAIR (T1c) modality and retaining key source information, thereby maintaining high segmentation accuracy even in the absence of a modality.

[0015] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0016] The incomplete multimodal brain tumor segmentation method based on key modality-guided contrastive learning of the present invention is characterized by being performed according to the following steps:

[0017] Step 1: Construct a medical image dataset ,in, Indicates the Multimodal MRI images; express The true segmentation mask of represents the number of MRI images, Indicates the modality number of MRI images, including: FLAIR modality, T1 modality, T1c modality, T2 modality, 、 、 Represent the height, width and depth of the MRI image respectively;

[0018] Step 2: Construct a modality-oriented efficient module, including two sets of convolutional encoders and modality fusion modules, and Processing is performed to obtain the FLAIR modality-guided group fusion features and T1c modality-guided group fusion features ;

[0019] Step 3: Construct a modality-guided contrast branch, including key feature modeling units and unsupervised contrastive learning units, and and Process and construct the total contrast loss function ;

[0020] Step 4: Construct a multimodal segmentation branch, including encoder, bottleneck layer, decoder and segmentation prediction head, and and Process and obtain The predicted segmentation labels , thereby constructing the dice loss function ;

[0021] Step 5: Based on dice loss function , warm up the multimodal segmentation branch. After a period of warm-up training, add the total contrast loss function The modality-guided comparison branch is used to alternately train the multimodal segmentation branch and the modality-guided comparison branch. When the number of training iterations reaches the set number, the training stops and an incomplete multimodal brain tumor segmentation model is obtained after training.

[0022] Step 6: In the inference stage, all modalities of MRI images are input into the trained multimodal segmentation branch for inference to obtain the final brain tumor segmentation results.

[0023] The incomplete multimodal brain tumor segmentation method based on key modality-guided contrastive learning of the present invention is also characterized in that step 2 is performed as follows:

[0024] Step 2.1: The first set of convolutional encoders consists of 1 key encoder and 3 query encoders, and is guided by the FLAIR modality and uses the key encoder to extract The FLAIR modal features are extracted using three query encoders. After the remaining modal features are obtained, they are spliced into the initial features of the FLAIR modal guide group through channels. ;

[0025] The second set of convolutional encoders also contains 1 key encoder and 3 query encoders, which are guided by the T1c mode and use the key encoder to extract The T1c modal features are obtained by using three query encoders. After the remaining modal features are obtained, they are spliced into the initial features of the T1c modality-guided group through channels. ;

[0026] Step 2.2: The modality fusion module is composed of vector Modal code , and thus use formula (1) to obtain the FLAIR modality-guided group fusion feature and T1c modality-guided group fusion features ,in, The module code representing the FLAIR modality, The module code representing the T1 mode, The module code representing the T1c mode, Indicates the module code of T2 mode, and the module code of any mode , 0 indicates that the corresponding MRI imaging modality is missing, and 1 indicates that the corresponding MRI imaging modality exists:

[0027] (1).

[0028] Furthermore, step 3 is performed as follows:

[0029] Step 3.1: Key feature modeling unit pairs and Process it and get Key features of the FLAIR modality-guided group Hedi Key features of the T1c modality-guided group ;

[0030] Step 3.2: The unsupervised contrastive learning unit guides the group of key features from the FLAIR modality The four modal features are separated and the separated T1 modal features, T1c modal features and T2 modal features are taken as the first FLAIR modality-guided group query features, denoted as , the separated FLAIR modal features are used as the first FLAIR modal-oriented group bond features, denoted as ; From any other Key features of the FLAIR modality-guided group The four modal features are also separated and used as negative sample features, denoted as , thereby constructing a positive sample pair And negative sample pairs ,in, ;

[0031] Key features of the group guided by T1c modality The four modal features are separated and the separated FLAIR modal features, T1 modal features and T2 modal features are taken together as the first T1c modality-guided group query features, denoted as , the separated T1c modal features are used as the first T1c modality-oriented group key features, denoted as ; From any other Key features of the T1c modality-guided group The four modal features are also separated and used as negative sample features, denoted as , thereby constructing a positive sample pair And negative sample pairs ;

[0032] Step 3.3: Use formula (2) to construct the total contrast loss function :

[0033] (2)

[0034] In formula (2), and represent the contrast loss of the FLAIR modality-guided group and the T1c modality-guided group, respectively, and are obtained from Equations (3) and (4):

[0035] (3)

[0036] (4)

[0037] In formula (3) and formula (4), is the cosine similarity;

[0038] Step 3.4: Use formula (5) to construct the parameter update formula of the key encoder in the modal-guided efficient module:

[0039] (5)

[0040] In formula (5), and They are the parameters before query encoder and key encoder update, represents a hyperparameter constant; The updated parameters for the key encoder.

[0041] Furthermore, step 3.1 is performed as follows:

[0042] Step 3.1.1: Use Equation (9) to obtain the FLAIR modality-guided group adaptive sampling feature and T1c modality-guided group adaptive sampling features :

[0043] (9)

[0044] In formula (9), Before selection pixels;

[0045] Step 3.1.2: Use formula (10) to obtain the FLAIR modality-guided group global average pooling feature and T1c modality-guided group global average pooling features :

[0046] (10)

[0047] In formula (10), represents global average pooling;

[0048] Step 3.1.3: Use Equation (11) to obtain the FLAIR modality-guided group contrast features mapped to the contrast space Comparison of characteristics with the T1c modality-guided group :

[0049] (11)

[0050] In formula (10), yes The fully connected weights of the first layer in the projector, yes The fully connected bias of the first layer in the projector; yes The fully connected weights of the second layer in the projector, yes The fully connected bias of the second layer in the projector; represents the activation function;

[0051] Step 3.1.4: Use formula (12) to obtain the FLAIR modality-guided group contrast normalized feature Comparison of normalized features with the T1c modality-oriented group :

[0052] (12)

[0053] In formula (12), represents the L2 norm.

[0054] Furthermore, step 4 is performed as follows:

[0055] Step 4.1: Use formula (6) to get Fusion features of FLAIR modality-guided group and T1c modality-guided group :

[0056] (6)

[0057] Step 4.2: Divided into of image patches, and use convolution to perform linear embedding operations on all image patches, resulting in a dimension of After linear embedding image patches and form the Initial embedding features ;in, represents the side length of the image partition patch, represents the dimensionality of the embedding space;

[0058] Step 4.3: The encoder is composed of W stages, and each stage contains several SwinTransformer blocks and a Patch Merging layer, and The features are processed to obtain the Wth stage Encoded features :

[0059] When w=1, After the w-th stage of the encoder, the w-th stage is obtained. Encoded features ;

[0060] When w=2,3,…,W, the first Encoded features After the wth stage of processing, the wth stage of Encoded features Thus, the Wth stage outputs the Wth stage Encoded features ;

[0061] Step 4.4: The bottleneck layer includes: several Swin Transformer blocks, and After processing, the output features are obtained ;

[0062] Step 4.5: The decoder is composed of M stages, each stage contains several 3D convolutional layers and 1 deconvolution layer, and Processing is performed to obtain the first Decoding features ;Where, M=W;

[0063] When m=1, After upsampling in the mth stage of the decoder, it is then combined with the Wmth stage Encoded features Fusion is performed to obtain the mth stage Decoding features ;

[0064] When m=2,3,…,M-1, the first Decoding features After upsampling at the mth stage of the decoder, it is then combined with the Wmth stage Encoded features The features are fused to obtain the mth stage Decoding features ; Thus, the M-1 stage outputs the M-1 stage Decoding features ;

[0065] When m=M, the m-1th stage Decoding features After upsampling in the mth stage of the decoder, it is combined with the initial embedding feature Fusion is performed to obtain the first Decoding features ;

[0066] Step 4.6: Segmentation prediction head pair After upsampling, Fusion, get Fusion output features , and thus using formula (7) we can get The predicted segmentation labels :

[0067] (7)

[0068] In formula (7), Indicates the number of target categories for segmentation, The convolution kernel is 3D convolution mapping operation;

[0069] Step 4.7: Use formula (8) to construct the dice loss function :

[0070] (8).

[0071] The electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the incomplete multimodal brain tumor segmentation method based on key modality-guided contrast learning, and the processor is configured to execute the program stored in the memory.

[0072] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the incomplete multimodal brain tumor segmentation method based on key modality-guided contrastive learning are executed.

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] 1. By adopting a modality-guided comparative learning method, the present invention can effectively utilize the tumor-specific feature guidance in FLAIR and T1c modalities to guide the similarity between the features of each modality to optimize the feature learning of other modalities, thereby solving the problems of generation errors affecting accuracy and failure to effectively utilize key modality features in the existing technology. It can also significantly improve segmentation accuracy in the absence of a modality.

[0075] 2. By employing a modality-guided, efficient modular dynamic convolution kernel and modal encoding, this invention automatically adjusts the feature extraction process based on the input modality, achieving efficient modal feature fusion. This eliminates the need to train multiple independent models for each possible modality combination and eliminates the need to generate missing modalities, thereby reducing training and inference costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 The framework diagram of the entire model of the present invention includes input, modality-guided efficiency module, contrastive learning module and multimodal segmentation module;

[0077] Figure 2 Schematic diagram of the modalities of the present invention, wherein the FLAIR modality shows the boundary of the entire tumor most clearly, while the T1c modality has the highest correspondence with the tumor core and the texture of the enhancing tumor;

[0078] Figure 3 A schematic diagram of the modal code of the present invention simulating a missing mode;

[0079] Figure 4 This is a schematic diagram of the present invention showing that other modes are clustered towards the key mode after being guided by the key mode;

[0080] Figure 5 The following are the segmentation results of the model of the present invention under different missing modes, and are compared with other models. DETAILED DESCRIPTION

[0081] In this embodiment, an incomplete multimodal brain tumor segmentation method based on key modality guided contrastive learning is described as follows: Figure 1 As shown, the following steps are included:

[0082] Step 1: Construct a medical image dataset ,in, Indicates the Multimodal MRI images; express The true segmentation mask of represents the number of MRI images, Indicates the modality number of MRI images, including: FLAIR modality, T1 modality, T1c modality, T2 modality, 、 、 Represent the height, width, and depth of the MRI image respectively; in this example, the BraTS2020 and BraTS2018 datasets are used. 、 、 The sizes are 240×240×155 respectively; the two datasets contain 369 and 285 annotated MRI sequences, respectively, covering four modalities. Tumor labels are divided into three tumor sub-regions: enhancing tumor (ET), tumor core (TC), and whole tumor (WT).

[0083] Step 2: Construct a modality-oriented efficient module, including two sets of convolutional encoders and modality fusion modules, and Processing is performed to obtain the FLAIR modality-guided group fusion features and T1c modality-guided group fusion features ;

[0084] Step 2.1: If Figure 2As shown, the FLAIR modality most clearly displays the boundaries of the entire tumor, while the T1c modality has the best texture correspondence with the tumor core and enhancing tumors. Therefore, the FLAIR modality performs best for segmenting the entire tumor. Similarly, the T1c modality is beneficial for segmenting the tumor core and enhancing tumors. This observation supports the use of the FLAIR and T1c modalities as guiding modalities for segmentation methods.

[0085] The first set of convolutional encoders consists of 1 key encoder and 3 query encoders, and is guided by the FLAIR modality. The key encoder is used to extract The FLAIR modal features are extracted using three query encoders. After the remaining modal features are obtained, they are spliced into the initial features of the FLAIR modal guide group through channels. , the key encoder and query encoder are 3D convolutions, and their 3D convolution kernels are ;

[0086] The second set of convolutional encoders also contains 1 key encoder and 3 query encoders, which are guided by the T1c mode and use the key encoder to extract The T1c modal features are obtained by using three query encoders. After the remaining modal features are obtained, they are spliced into the initial features of the T1c modality-guided group through channels. ;

[0087] In addition, the modality-guided efficient module also has a low computational cost, which can improve efficiency while maintaining multimodal modeling capabilities. Assuming there are n encoders, its computational complexity is shown in formula (13):

[0088] (13)

[0089] In formula (13), is the input / output channel dimension, is the kernel size in each dimension. In contrast, if encoder fusion is not performed, the computational cost will be as shown in Equation (14):

[0090] (14)

[0091] Obviously, the computational cost grows slowly as the number of encoders n increases, which indicates that the modality-guided efficient module is a lightweight module even when more modality-specific encoders are included.

[0092] Step 2.2: The modal fusion module is composed of vector Modal code , and thus use formula (1) to obtain the FLAIR modality-guided group fusion feature and T1c modality-guided group fusion features ,in, The module code representing the FLAIR modality, The module code representing the T1 mode, The module code representing the T1c mode, Indicates the module code of T2 mode, and the module code of any mode ,like Figure 3 As shown, 0 indicates that the corresponding MRI imaging modality is missing, and 1 indicates that the corresponding MRI imaging modality is present:

[0093] (1)

[0094] Step 3: Construct a modality-guided contrast branch, including key feature modeling units and unsupervised contrastive learning units, and and Process and construct the total contrast loss function ;

[0095] Step 3.1: Key feature modeling unit pairs and Process it and get Key features of the FLAIR modality-guided group Hedi Key features of the T1c modality-guided group ;

[0096] Step 3.1.1: Use Equation (9) to obtain the FLAIR modality-guided group adaptive sampling feature and T1c modality-guided group adaptive sampling features In this example, we evaluated the performance of different numbers of sampling points (including 2400, 4800, and 9600). The best performance was achieved using 9600 sampling points. However, considering the computational complexity, we finally decided to use K = 4800:

[0097] (9)

[0098] In formula (9), Before selection pixels.

[0099] Step 3.1.2: Use formula (10) to obtain the FLAIR modality-guided group global average pooling feature and T1c modality-guided group global average pooling features :

[0100] (10)

[0101] In formula (10), Represents global average pooling.

[0102] Step 3.1.3: Use Equation (11) to obtain the FLAIR modality-guided group contrast features mapped to the contrast space Comparison of characteristics with the T1c modality-guided group :

[0103] (11)

[0104] In formula (10), yes The fully connected weights of the first layer in the projector, yes The fully connected bias of the first layer in the projector; yes The fully connected weights of the second layer in the projector, yes The fully connected bias of the second layer in the projector; Represents the activation function.

[0105] Step 3.1.4: Use formula (12) to obtain the FLAIR modality-guided group contrast normalized feature Comparison of normalized features with the T1c modality-oriented group :

[0106] (12)

[0107] In formula (12), represents the L2 norm.

[0108] Step 3.2: Unsupervised contrastive learning unit guides group key features from FLAIR modality The four modal features are separated and the separated T1 modal features, T1c modal features and T2 modal features are taken as the first FLAIR modality-guided group query features, denoted as , the separated FLAIR modal features are used as the first FLAIR mode-guided group bond features, denoted as ; From any other Key features of the FLAIR modality-guided group The four modal features are also separated and used as negative sample features, denoted as , thereby constructing a positive sample pair And negative sample pairs ,in, .

[0109] Corresponding key features of the T1c modality-guided group The four modal features are separated and the separated FLAIR modal features, T1 modal features and T2 modal features are taken together as the first T1c modality-guided group query features, denoted as , the separated T1c modal features are used as the first T1c modality-oriented group key features, denoted as ; From any other Key features of the T1c modality-guided group The four modal features are also separated and used as negative sample features, denoted as , thereby constructing a positive sample pair And negative sample pairs .

[0110] Step 3.3: Use formula (2) to construct the total contrast loss function :

[0111] (2)

[0112] In formula (2), and represent the contrast loss of the FLAIR modality-guided group and the T1c modality-guided group, respectively, and are obtained from Equations (3) and (4):

[0113] (3)

[0114] (4)

[0115] In formula (3) and formula (4), is the cosine similarity;

[0116] Step 3.4: Use formula (5) to construct the parameter update formula of the key encoder in the modal-guided efficient module:

[0117] (5)

[0118] In formula (5), and They are the parameters before query encoder and key encoder update, represents a hyperparameter constant; is the updated parameter of the key encoder. In this example, is 0.999.

[0119] Step 4: Construct a multimodal segmentation branch, including encoder, bottleneck layer, decoder and segmentation prediction head, and and Process and obtain The predicted segmentation labels , thereby constructing the dice loss function ;

[0120] Step 4.1: Use formula (6) to get Fusion features of FLAIR modality-guided group and T1c modality-guided group :

[0121] (6)

[0122] Step 4.2: Divided into of image patches, and use convolution to perform linear embedding operations on all image patches, resulting in a dimension of After linear embedding image patches and form the Initial embedding features ,in, represents the side length of the image partition patch, represents the dimension of the embedding space. In this example, a and C are 2 and 48 respectively.

[0123] Step 4.3: The encoder is composed of W stages, and each stage contains several SwinTransformer blocks and a Patch Merging layer, and The features are processed to obtain the Wth stage Encoded features In this example, W is 4 and the number of Swin Transformer blocks is 2:

[0124] When w=1, After the w-th stage of the encoder, it passes through two Swin Transformer blocks and then one Patch Merging layer to obtain the w-th stage. Encoded features ;

[0125] When w=2,3,…,W, the first Encoded features After the wth stage of processing, it first passes through 2 Swin Transformer blocks and then passes through 1 Patch Merging layer to obtain the wth stage of Encoded features Thus, the Wth stage outputs the Wth stage Encoded features ;

[0126] Step 4.4: The bottleneck layer includes: several Swin Transformer blocks, and After processing, the output features are obtained ,In this example, there are 2 Swin Transformer blocks;

[0127] Step 4.5: The decoder is composed of M stages, each stage contains several 3D convolutional layers and 1 deconvolution layer, and Processing is performed to obtain the first Decoding features ;Where M=W, in this example, M is 4 and there are 2 3D convolutional layers;

[0128] When m=1, After upsampling by the deconvolution layer of the mth stage of the decoder, it is then combined with the Encoded features After fusion, and then through two 3D convolution layers, we get the mth stage Decoding features ;

[0129] When m=2,3,…,M-1, the first Decoding features After upsampling by the deconvolution layer of the mth stage of the decoder, it is then combined with the Encoded features The features are fused and then passed through two 3D convolution layers to obtain the mth stage Decoding features ; Thus, the M-1 stage outputs the M-1 stage Decoding features ;

[0130] When m=M, the m-1th stage Decoding features After upsampling in the deconvolution layer of the decoder at stage m, it is combined with the initial embedding feature After fusion, and then through two 3D convolution layers, we get the first Decoding features ;

[0131] Step 4.6: Segmentation prediction head pair After upsampling, Fusion, get Fusion output features , and thus using formula (7) we can get The predicted segmentation labels :

[0132] (7)

[0133] In formula (7), Indicates the number of target categories for segmentation. In this example, there are 4 categories. The convolution kernel is 3D convolution mapping operation.

[0134] Step 4.7: Use formula (8) to construct the dice loss function :

[0135] (8)

[0136] Step 5: In the training phase, a dual-branch approach is used to enhance the capabilities of the model. Training begins with the dice loss function. , warm-up training is performed on the multimodal segmentation branch, because the representations initially obtained from the FLAIR modality and T1c modality may not be sufficient to effectively guide the representation of other modalities, and in this branch, the missing modality scenario is simulated by randomly setting some modalities to zero mapping. After a period of warm-up training, a total contrast loss function is added. The modality-guided comparison branch is used to improve the discriminative power of the learning features and the correlation between modalities. The complete modality is retained in this branch, and the representation of other modalities is guided by the FLAIR modality and the T1c modality respectively. Then the multimodal segmentation branch and the modality-guided comparison branch are trained alternately. When the number of training iterations reaches the set number, the training stops and the trained incomplete multimodal brain tumor segmentation model is obtained. In this example, the warm-up training is 100 iterations, and the set number of training iterations is 1000 iterations. The modality-guided comparison branch is used to shorten the distance between other modalities and the key modality. Figure 4 As shown;

[0137] Because both optimizers contribute to guiding the optimization module towards efficient modules under different modal inputs, this embodiment employs two optimizers, each aligned with a corresponding loss function, to alternately optimize the comprehensive framework of the present invention. This alternating training strategy, particularly the targeted optimization of the segmentation branch, reduces the risk of over-aligning other modalities to the key modality. Therefore, it implicitly ensures the preservation of unique source information within each modality.

[0138] Step 6: In the inference phase, the modality-guided contrast branch is discarded, and all modal MRI images are fed into the trained multimodal segmentation branch. The entire volume is inferred using a sliding window method with an overlap ratio of 0.5 to obtain the final brain tumor segmentation result. This approach can fully utilize the trained model to accurately segment brain tumors in practical applications, providing a reliable basis for clinical diagnosis. Figure 5 The segmentation results of this model under different missing modes are shown and compared with other segmentation models.

[0139] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0140] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

Claims

1. An incomplete multimodal brain tumor segmentation method based on key modality-guided contrastive learning, characterized by: The steps are as follows: Step 1: Construct a medical image dataset ,in, Indicates the Multimodal MRI images; express The true segmentation mask of represents the number of MRI images, Indicates the modality number of MRI images, including: FLAIR modality, T1 modality, T1c modality, T2 modality, 、 、 Represent the height, width and depth of the MRI image respectively; Step 2: Construct a modality-oriented efficient module, including two sets of convolutional encoders and modality fusion modules, and Processing is performed to obtain the FLAIR modality-guided group fusion features and T1c modality-guided group fusion features ; Step 3: Construct a modality-guided contrast branch, including key feature modeling units and unsupervised contrastive learning units, and and Process and construct the total contrast loss function ; Step 4: Construct a multimodal segmentation branch, including encoder, bottleneck layer, decoder and segmentation prediction head, and and Process and obtain The predicted segmentation labels , thereby constructing the dice loss function ; Step 5: Based on dice loss function , warm up the multimodal segmentation branch. After a period of warm-up training, add the total contrast loss function The modality-guided comparison branch is used to alternately train the multimodal segmentation branch and the modality-guided comparison branch. When the number of training iterations reaches the set number, the training stops and an incomplete multimodal brain tumor segmentation model is obtained after training. Step 6: In the inference stage, all modalities of MRI images are input into the trained multimodal segmentation branch for inference to obtain the final brain tumor segmentation results.

2. The incomplete multimodal brain tumor segmentation method based on key modality-guided contrastive learning according to claim 1, characterized in that: Described step 2 is carried out as follows: Step 2.1: The first set of convolutional encoders consists of 1 key encoder and 3 query encoders, and is guided by the FLAIR modality and uses the key encoder to extract The FLAIR modal features are extracted using three query encoders. After the remaining modal features are obtained, they are spliced into the initial features of the FLAIR modal guide group through channels. ; The second set of convolutional encoders also contains 1 key encoder and 3 query encoders, which are guided by the T1c mode and use the key encoder to extract The T1c modal features are obtained by using three query encoders. After the remaining modal features are obtained, they are spliced into the initial features of the T1c modality-guided group through channels. ; Step 2.2: The modality fusion module is composed of vector Modal code , and thus use formula (1) to obtain the FLAIR modality-guided group fusion feature and T1c modality-guided group fusion features ,in, The module code representing the FLAIR modality, The module code representing the T1 mode, The module code representing the T1c mode, Indicates the module code of T2 mode, and the module code of any mode , 0 indicates that the corresponding MRI imaging modality is missing, and 1 indicates that the corresponding MRI imaging modality exists: (1)。 3. The incomplete multimodal brain tumor segmentation method based on key modality-guided contrastive learning according to claim 2, characterized in that: Described step 3 is carried out as follows: Step 3.1: Key feature modeling unit pairs and Process it and get Key features of the FLAIR modality-guided group Hedi Key features of the T1c modality-guided group ; Step 3.2: The unsupervised contrastive learning unit guides the group of key features from the FLAIR modality The four modal features are separated and the separated T1 modal features, T1c modal features and T2 modal features are taken as the first FLAIR modality-guided group query features, denoted as , the separated FLAIR modal features are used as the first FLAIR modal-oriented group bond features, denoted as ; From any other Key features of the FLAIR modality-guided group The four modal features are also separated and used as negative sample features, denoted as , thereby constructing a positive sample pair And negative sample pairs ,in, ; Key features of the group guided by T1c modality The four modal features are separated and the separated FLAIR modal features, T1 modal features and T2 modal features are taken together as the first T1c modality-guided group query features, denoted as , the separated T1c modal features are used as the first T1c modality-oriented group key features, denoted as ; From any other Key features of the T1c modality-guided group The four modal features are also separated and used as negative sample features, denoted as , thereby constructing a positive sample pair And negative sample pairs ; Step 3.3: Use formula (2) to construct the total contrast loss function : (2) In formula (2), and represent the contrast loss of the FLAIR modality-guided group and the T1c modality-guided group, respectively, and are obtained from Equations (3) and (4): (3) (4) In formula (3) and formula (4), is the cosine similarity; Step 3.4: Use formula (5) to construct the parameter update formula of the key encoder in the modal-guided efficient module: (5) In formula (5), and They are the parameters before query encoder and key encoder update, represents a hyperparameter constant; The updated parameters for the key encoder.

4. The incomplete multimodal brain tumor segmentation method based on key modality-guided contrastive learning according to claim 3, characterized in that: The step 3.1 is carried out as follows: Step 3.1.1: Use Equation (9) to obtain the FLAIR modality-guided group adaptive sampling feature and T1c modality-guided group adaptive sampling features : (9) In formula (9), Before selection pixels; Step 3.1.2: Use formula (10) to obtain the FLAIR modality-guided group global average pooling feature and T1c modality-guided group global average pooling features : (10) In formula (10), represents global average pooling; Step 3.1.3: Use Equation (11) to obtain the FLAIR modality-guided group contrast features mapped to the contrast space Comparison of characteristics with the T1c modality-guided group : (11) In formula (10), yes The fully connected weights of the first layer in the projector, yes The fully connected bias of the first layer in the projector; yes The fully connected weights of the second layer in the projector, yes The fully connected bias of the second layer in the projector; represents the activation function; Step 3.1.4: Use formula (12) to obtain the FLAIR modality-guided group contrast normalized feature Comparison of normalized features with the T1c modality-oriented group : (12) In formula (12), represents the L2 norm.

5. The incomplete multimodal brain tumor segmentation method based on key modality-guided contrastive learning according to claim 4, characterized in that: Described step 4 is carried out as follows: Step 4.1: Use formula (6) to get Fusion features of FLAIR modality-guided group and T1c modality-guided group : (6) Step 4.2: Divided into of image patches, and use convolution to perform linear embedding operations on all image patches, resulting in a dimension of After linear embedding image patches and form the Initial embedding features ;in, represents the side length of the image partition patch, represents the dimensionality of the embedding space; Step 4.3: The encoder is composed of W stages, and each stage contains several Swin Transformer blocks and a Patch Merging layer, and The features are processed to obtain the Wth stage Encoded features : When w=1, After the w-th stage of the encoder, the w-th stage is obtained. Encoded features ; When w=2,3,…,W, the first Encoded features After the wth stage of processing, the wth stage of Encoded features Thus, the Wth stage outputs the Wth stage Encoded features ; Step 4.4: The bottleneck layer includes: several Swin Transformer blocks, and After processing, the output features are obtained ; Step 4.5: The decoder is composed of M stages, each stage contains several 3D convolutional layers and 1 deconvolution layer, and Processing is performed to obtain the first Decoding features ;Where, M=W; When m=1, After upsampling in the mth stage of the decoder, it is then combined with the Wmth stage Encoded features Fusion is performed to obtain the mth stage Decoding features ; When m=2,3,…,M-1, the first Decoding features After upsampling at the mth stage of the decoder, it is then combined with the Wmth stage Encoded features The features are fused to obtain the mth stage Decoding features ; Thus, the M-1 stage outputs the M-1 stage Decoding features ; When m=M, the m-1th stage Decoding features After upsampling in the mth stage of the decoder, it is combined with the initial embedding feature Fusion is performed to obtain the first Decoding features ; Step 4.6: Segmentation prediction head pair After upsampling, Fusion, get Fusion output features , and thus using formula (7) we can get The predicted segmentation labels : (7) In formula (7), Indicates the number of target categories for segmentation, The convolution kernel is 3D convolution mapping operation; Step 4.7: Use formula (8) to construct the dice loss function : (8)。 6. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the incomplete multimodal brain tumor segmentation method based on key modality-guided contrastive learning according to any one of claims 1 to 5, and the processor is configured to execute the program stored in the memory.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the incomplete multimodal brain tumor segmentation method based on key modality-guided contrastive learning according to any one of claims 1 to 5 are performed.

Images