Glioma segmentation model based on feature decoupling and method for establishing glioma segmentation model

Abstract

The application discloses a feature decoupling-based brain glioma segmentation model establishment method and brain glioma segmentation method, and belongs to the medical image segmentation field.The established model comprises a feature decoupling module and a feature compensation module.The feature decoupling module comprises four branches corresponding to four modes;after four groups of features are extracted from the branch corresponding to any mode m, the features are spliced into features X m according to channels, the attention scores of the channels are calculated, the channels are ranked according to the scores from high to low, and the features X m are added to obtain CSSA(X m ), the features CSSA(X m ) are equally divided into four groups of features along the channels, the first group of features is the self-feature of the mode m, and the other three groups of features are the mutual features of the mode m and the other three modes;the feature compensation module is used for constructing full-mode features by using the self-feature of the non-missing mode and the mutual features of the non-missing mode and the missing mode.The application can effectively improve the brain glioma segmentation performance under the condition of missing modes.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical image segmentation, and more particularly relates to a missing modality glioma segmentation model establishment method and a glioma segmentation method based on feature decoupling. BACKGROUND

[0002] Glioma is a common malignant tumor in the central nervous system with high mortality, and its image segmentation is a key step for clinical diagnosis and treatment. Clinically, four common magnetic resonance imaging (MRI) modalities are mainly used for image diagnosis of glioma, including T1 weighted (T1), contrast-enhanced T1 weighted (T1ce), T2 weighted (T2) and fluid attenuated inversion recovery (FLAIR). T1 modality and T1ce modality are usually used to evaluate tumor core, while T2 modality and FLAIR modality are used to detect edema.

[0003] In order to assist clinicians to diagnose more accurately and efficiently, researchers have proposed many deep learning glioma segmentation methods using four MRI modalities. These methods improve the segmentation effect by designing more effective modality feature fusion, edge information extraction and other strategies. However, due to factors such as image damage, scanning scheme and patient condition, some MRI modalities are often missing in clinical practice, which seriously affects the segmentation performance of the above methods. Therefore, the missing modality glioma segmentation method has important significance in clinical practice.

[0004] The goal of missing modality segmentation is to segment glioma using the existing MRI modalities and obtain a segmentation result close to that of the full modality. For the clinical scenario of up to 14 missing modalities, existing research mainly uses modality adaptation methods to enable the network to handle various missing modalities, which can be divided into explicit modality adaptation and implicit modality adaptation. Explicit modality adaptation usually uses image generation methods to try to complete the missing modality, but when multiple modalities are missing, the generation effect is often poor, and the final segmentation result is too dependent on the quality of the generated image. Some researches use knowledge transfer strategies to transfer the model knowledge of the full modality to the missing modality model, which usually requires training a full modality teacher model and multiple missing modality student models, and the training and deployment cost is too high.

[0005] The implicit modality adaptation fuses features from different modalities and processes the fused features using a shared decoder to obtain the final segmentation result. Such methods train the network using a modality perturbation strategy, i.e., randomly discarding some modalities during the training process, so that the shared decoder can adapt to different missing modalities. In addition, auxiliary regularizers are often used to help the model extract more rich modality features. Such methods are currently a research hotspot, such as RFNet (Region-aware Fusion Network), mmFormer (Multimodal Medical Transformer), MAVP (Modal-aware Visual Prompting), and GSS (Group Self-Support Learning), etc.

[0006] The above-mentioned explicit modality adaptation and implicit modality adaptation methods have a common goal when applied to brain glioma segmentation in the missing modality scenario, i.e., they all expect the features extracted by each modality to represent the features of the current modality and the potential features of other modalities. However, it is very difficult for a set of features to achieve self-expression and mutual expression, which are actually two different abilities. Most existing methods entangle them in the same feature space. This entanglement of self-modality and other modality information puts a heavy burden on the deep learning network and reduces the effectiveness of modality adaptation, and ultimately the segmentation performance of the trained model cannot be guaranteed. SUMMARY

[0007] In view of the defects of the prior art and the need for improvement, the present application provides a feature decoupling-based missing modality brain glioma segmentation model establishment method and a brain glioma segmentation method, which aims to improve the performance of brain glioma segmentation under the condition of missing modalities.

[0008] To achieve the above-mentioned purpose, according to one aspect of the present application, a feature decoupling-based missing modality brain glioma segmentation model establishment method is provided, comprising:

[0009] Collecting brain tissue MRI images with labeled brain glioma segmentation labels, normalizing the MRI images of the same brain tissue under different modalities after registering them to the same modality to obtain a brain glioma segmentation dataset;

[0010] Initializing a 3D U-Net network, replacing the first layer thereof with a feature decoupling module and a feature compensation module connected to each other to obtain an improved 3D U-Net network, and training the same using the brain glioma segmentation dataset, and obtaining a brain glioma segmentation model after the training is completed;

[0011] The feature decoupling module includes four branches corresponding to four modalities; the branch corresponding to any modality m includes: four parallel feature extraction units for performing feature extraction on the brain tissue images under the modality m to obtain four groups of intermediate decoupled features; and a channel sparse self-attention unit for concatenating the four groups of intermediate decoupled features by channels to obtain the feature X m , and then scoring the attention of each channel in the feature X m , and sorting each channel in the feature X m in descending order of attention score and adding the sorted channel to the feature X m to obtain the feature CSSA(X m ), and equally dividing the feature CSSA(X m ) into four groups of features, the first group of features being the self-feature of the modality m, and the remaining three groups of features being the mutual features of the modality m and other modalities.

[0012] The feature compensation module is configured to construct full-modality features, wherein the features of the non-missing modalities are self-features of the modalities, and the features of the missing modalities are mutual features of the non-missing modalities and the modalities.

[0013] Further, when the feature compensation module constructs the full-modality features, the mutual features used to compensate for the missing modalities are determined in the following priority order:

[0014] When the missing modality is the T1 modality, if the T1ce modality is not missing, the mutual features of the T1ce modality and the T1 modality are selected to compensate for the missing modality; if the T1ce modality is missing and the T2 modality is not missing, the mutual features of the T2 modality and the T1 modality are selected to compensate for the missing modality; and if both the T1ce modality and the T2 modality are missing, the mutual features of the FLAIR modality and the T1 modality are selected to compensate for the missing modality.

[0015] When the missing modality is the T1ce modality, if the T1 modality is not missing, the mutual features of the T1 modality and the T1ce modality are selected to compensate for the missing modality; if the T1 modality is missing and the FLAIR modality is not missing, the mutual features of the FLAIR modality and the T1ce modality are selected to compensate for the missing modality; and if both the T1 modality and the FLAIR modality are missing, the mutual features of the T2 modality and the T1ce modality are selected to compensate for the missing modality.

[0016] When the missing modality is the T2 modality, if the FLAIR modality is not missing, the mutual features of the FLAIR modality and the T2 modality are selected to compensate for the missing modality; if the FLAIR modality is missing and the T1 modality is not missing, the mutual features of the T1 modality and the T2 modality are selected to compensate for the missing modality; and if both the FLAIR modality and the T1 modality are missing, the mutual features of the T1ce modality and the T2 modality are selected to compensate for the missing modality.

[0017] When the missing mode is the FLAIR mode, if the T2 mode is not missing, the mutual features between the T2 mode and the FLAIR mode are selected for compensation; if the T2 mode is missing but the T1ce mode is not missing, the mutual features between the T1ce mode and the FLAIR mode are selected for compensation; if both the T2 mode and the T1ce mode are missing, the features between the T1 mode and the FLAIR mode are selected for compensation.

[0018] Furthermore, in the feature decoupling module, among the four sets of features ultimately obtained from the branch corresponding to the T1 mode, the first set of features is the self-feature s of the T1 mode. t1 The second set of features consists of the mutual features u between the T1 mode and the T1ce mode. t1→tc The third set of features consists of the mutual features between the T1 and T2 modes. t1→t2 The fourth set of features consists of the mutual features between the T1 mode and the FLAIR mode. t1→fl ;

[0019] Of the four sets of features obtained from the branch corresponding to the T1ce mode, the first set of features is the self-feature s of the T1ce mode. tc The second set of features consists of the mutual features between the T1ce mode and the T1 mode. tc→t1 The third set of features consists of the mutual features between the T1ce mode and the FLAIR mode. tc→fl The fourth set of features consists of the mutual features between the T1ce mode and the T2 mode. tc→t2 ;

[0020] Of the four sets of features obtained from the branch corresponding to the T2 mode, the first set of features is the self-feature s of the T2 mode. t2 The second set of features consists of the mutual features between the T2 mode and the FLAIR mode. t2→fl The third set of features consists of the mutual features between the T2 and T1 modes. t2→t1 The fourth set of features consists of the mutual features between the T2 mode and the T1ce mode. t2→tc ;

[0021] Of the four sets of features obtained from the branches corresponding to the FLAIR mode, the first set of features is the self-feature s of the FLAIR mode. fl The second set of features consists of the mutual features between the FLAIR mode and the T2 mode. fl→t2 The third set of features consists of the mutual features between the FLAIR mode and the T1ce mode. fl→tc The fourth set of features consists of the mutual features between the FLAIR mode and the T1 mode. fl→t1 .

[0022] Furthermore, in the feature decoupling module, each feature extraction unit is a convolutional module.

[0023] Further, in the improved 3D U-Net network, a segmentation head is further arranged on each feature of the decoder part, to obtain an output probability atlas at a corresponding scale.

[0024] When the improved 3D U-Net network is trained by using a brain glioma segmentation dataset, the training loss function comprises a sum of segmentation losses corresponding to the output probability atlas at each scale; and the segmentation loss corresponding to the output probability atlas at each scale is a loss of the output probability atlas at the scale relative to a brain glioma segmentation label down-sampled to the same scale.

[0025] Further, the segmentation loss corresponding to the output probability atlas at each scale comprises a Dice loss and a cross-entropy loss.

[0026] According to still another aspect of the present application, a brain glioma segmentation method is provided, comprising:

[0027] After the MRI image of the brain tissue to be segmented is registered to the same modality, the MRI image is input into the brain glioma segmentation model established by the above-mentioned brain glioma segmentation model establishment method based on feature decoupling provided by the present application, to obtain a brain glioma segmentation result.

[0028] According to still another aspect of the present application, a computer readable storage medium is provided, comprising a stored computer program; when the computer program is executed by a processor, the computer readable storage medium controls a device where the computer readable storage medium is located to execute the above-mentioned brain glioma segmentation model establishment method based on feature decoupling provided by the present application, and / or the above-mentioned brain glioma segmentation method provided by the present application.

[0029] According to still another aspect of the present application, an electronic device is provided, comprising:

[0030] a computer readable storage medium for storing a computer program;

[0031] and a processor for reading the computer program stored in the computer readable storage medium, to execute the above-mentioned brain glioma segmentation model establishment method based on feature decoupling provided by the present application, and / or the above-mentioned brain glioma segmentation method provided by the present application.

[0032] Overall, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0033] (1) The original segmentation model 3D U-Net network is improved, so that the improved network can extract four groups of features corresponding to four modalities through four groups of parallel feature extraction units for each modality, realize feature decoupling, then splice the four groups of features, and perform channel association according to the channel attention score, thereby realizing the bidirectional mapping between channels, so that the intermediate state decoupling features can effectively interact, and each channel is only associated with a unique channel, realizing a sparse channel self-attention mechanism, thereby avoiding the features from being entangled again, and finally for each modality, the self-features and mutual features can be decoupled, thereby effectively reducing the burden of feature learning, and for the missing modality, the corresponding mutual features are used for compensation, compared with using a zero matrix to replace the missing modality, more knowledge can be provided for feature learning, and the segmentation performance of the model obtained by training under the condition of missing modality is effectively improved.

[0034] (2) In the preferred scheme of the present application, based on clinical prior knowledge, the selection method of mutual features for compensating for missing modalities and the determination method of mutual features of each modality and other modalities are determined, thereby effectively constructing a pseudo full modality feature closer to the real scene under the condition of missing modalities, which is more conducive to the training of the model.

[0035] (3) In the preferred scheme of the present application, a parameter-light and portable convolution module is specifically selected as the feature extraction unit, thereby further reducing the burden of feature learning, and also enabling the model to pay more attention to the subsequent brain glioma segmentation task during training.

[0036] (4) In the preferred scheme of the present application, a segmentation head is set at each decoder feature scale during training to obtain output probability maps of different scales, and the sum of the calculated segmentation losses is taken as the overall training loss after calculating the corresponding segmentation losses, thereby being able to supervise the intermediate output results of the model and guide the model to produce more accurate brain glioma segmentation results. In the further preferred scheme thereof, the Dice loss and the cross-entropy loss are simultaneously calculated for each scale of the output probability map, which can provide more effective guidance for model training. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The feature decoupling-based missing modality brain glioma segmentation model establishment method provided by the embodiment of the present application is shown in the schematic diagram.

[0038] Figure 2 The structure schematic diagram of the feature decoupling module provided by the embodiment of the present application is shown in the schematic diagram.

[0039] Figure 3 The channel sparse self-attention mechanism provided by the embodiment of the present application is shown in the schematic diagram.

[0040] Figure 4 A feature compensation mechanism schematic diagram provided for an embodiment of the present application;

[0041] Figure 5 A quantitative result comparison schematic diagram of a brain glioma segmentation method provided for an embodiment of the present application and other missing modality brain glioma segmentation methods;

[0042] Figure 6 And Figure 7 A segmentation result schematic diagram of a brain glioma segmentation method provided for an embodiment of the present application for two brain glioma patients;

[0043] Figure 8 And Figure 9 A segmentation result comparison schematic diagram of a brain glioma segmentation method provided for an embodiment of the present application and other missing modality brain glioma segmentation methods. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0045] In the present application, the terms "first", "second", etc. (if any) in the present application and the drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0046] In order to solve the problem of high coupling of modal features in the current deep learning brain glioma segmentation method for missing modality scene, which brings burden to the learning of features and the training of model, and further leads to poor segmentation effect, the present application provides a missing modality brain glioma segmentation model establishment method and a brain glioma segmentation method based on feature decoupling, the overall idea of which is to improve the segmentation model structure so that it can realize the decoupling of self-features and mutual-features of the modalities, and based on the mutual-features obtained by decoupling, a pseudo full modality feature more consistent with the real scene is constructed, realizing efficient and accurate segmentation of brain glioma in a missing modality scene. On this basis, the authenticity of the pseudo full modality feature can be further combined with clinical prior knowledge information.

[0047] For ease of description, in the following embodiments, the T1 modality, the T1ce modality, the T2 modality and the FLAIR modality are represented by symbols t1, tc, t2 and fl respectively, and the full modality set composed of the four modalities is represented as u m represents the self-feature of modality m, and the mutual-feature um→m′ denotes the cross-feature of modality m and modality m', and m'≠m.

[0048] The following is an embodiment.

[0049] Embodiment 1:

[0050] A feature decoupling-based missing modality brain glioma segmentation model establishment method includes three processes of data preprocessing, model construction and model training.

[0051] In this embodiment, the purpose of data preprocessing is to construct a brain glioma segmentation dataset that can be directly used for model training. To achieve this purpose, in this embodiment, data preprocessing includes:

[0052] First, collect brain tissue MRI images with labeled brain glioma segmentation labels. In actual application, head MRI images under four modalities can be obtained first, skull stripping is performed to obtain the corresponding brain tissue MRI images. To adapt to the input size requirements of the model, the obtained brain tissue MRI images are further resampled to 1mmx1mmx1mm size, and the brain glioma region therein is labeled as a brain glioma segmentation label.

[0053] Then, register the MRI images of the same brain tissue under different modalities to the same modality to avoid the influence of the offset existing in different modality imaging on the subsequent segmentation result.

[0054] Next, crop the excess background part and only keep the brain tissue region to reduce the image size and remove invalid information.

[0055] Finally, normalize the image matrix of the four modalities to zero mean and unit variance within the brain region to avoid the negative impact of the data distribution difference of different patients on subsequent model training. In this embodiment, for the image I m , the normalization can be expressed as follows:

[0056]

[0057] where I' m denotes the image I m after normalization, and are the mean and standard deviation of the modality m brain region voxel matrix, respectively.

[0058] In this embodiment, the purpose of model construction is to propose a new glioma segmentation model which can decouple the feature expression of each modality to reduce the learning burden of modality features and enhance the robustness of the model to different missing modality conditions. To achieve this purpose, the structure of the traditional segmentation model 3D U-Net network is improved, which specifically includes:

[0059] After initializing the 3D U-Net network, the first layer is replaced by the interconnection of the feature decoupling module and the feature compensation module to obtain the improved 3D U-Net network, the structure of which is shown in Figure 1 .

[0060] The structure of the feature decoupling module is shown in Figure 2 , which includes four branches corresponding to four modalities.

[0061] The branch corresponding to any modality m includes four parallel feature extraction units and a channel sparse self-attention unit.

[0062] The four parallel feature extraction units do not share weights with each other and extract features from the brain tissue image under modality m to obtain four groups of intermediate decoupling features. In order to further reduce the model parameters, in this embodiment, a light and portable convolution module is used as the feature extraction unit, each convolution module is composed of two consecutive convolution units (convolution-normalization-activation function), the convolution kernel size is 3x3x3, the kernel number is 8, the step is 1, the padding is 1, the normalization is three-dimensional instance normalization, and the activation function is LeakyReLU with a negative coefficient of 0.01.

[0063] The channel sparse self-attention unit is used to concatenate the four groups of intermediate decoupling features by channel to obtain feature X m , then score the attention of each channel in feature X m , and sort each channel in feature X m in descending order of attention score, and then add feature X m to obtain feature CSSA(X m ), divide feature CSSA(X m ) into four groups of features along the channel, the first group of features is the self-feature of modality m, and the remaining three groups of features are the mutual features of modality m and other modalities.

[0064] In this embodiment, the channel sparse self-attention unit implements a channel sparse self-attention mechanism (CSSA), which acts to enable effective interaction of intermediate state decoupled features, while avoiding features from being entangled again, to better guide feature learning. The channel sparse self-attention mechanism enables each channel to be associated with only a unique channel, i.e., the first channel is associated with the channel with the highest attention score, the second channel is associated with the channel with the second highest attention score, and so on, thereby enabling a bijective relationship between channels. Since each channel is associated with only a unique target channel, and vice versa, this means that the attention matrix generated by the CSSA is sparse and full rank.

[0065] In this embodiment, the channel sparse self-attention mechanism is as shown in Figure 3 , where the concatenated features where C = 32 is the number of feature channels, and D, H and W are the depth, height and width of the features, respectively. A Multi-Layer Perceptron (MLP) module is used to score X m , and obtain the score of each channel to form a score matrix The scores of each channel are arranged in descending order to obtain the correspondence Q between scores and channels:

[0066] Q(i) = j, 0 ≤ i, j < C

[0067] This means that the jth channel has the ith highest score, and i = 0 indicates the highest score. Next, Q(i) is converted into a one-hot encoding vector e Q(i) , and a transpose matrix P m is constructed:

[0068]

[0069] The final output of the CSSA mechanism is:

[0070] CSSA(x m ) = X m + X m * P m

[0071] The features CSSA(x m ) are equally divided into four groups of features along the channel, with the first group of features being the self-features of the modality m, and the remaining three groups of features being the mutual features of the modality m and other modalities. Thus, for the modality m, the decoupled features can be divided into two categories, self-features s m and mutual features u m→m′ , the self-features s m are used to express the intrinsic features of the modality m, and the mutual features u m→m′The features for compensating for missing modalities are constructed to construct pseudo full-modal features.

[0072] In this embodiment, a feature decoupling module is introduced into the segmentation model, and the features of each modality are decoupled into features expressing themselves and features expressing other modalities, thereby greatly reducing the burden of feature learning.

[0073] To further improve the expression ability of the mutual features, in the decoupling of the modality features, the correlation between the modalities is determined in combination with clinical prior knowledge, and the mutual features are further determined. For any modality m, the closer the mutual features are to the self-features of the modality m, the stronger the correlation of the modality. Specifically, in clinical practice, T1 modality and T1ce modality are usually combined to diagnose tumor cores, and T2 modality and FLAIR modality are usually combined to diagnose peritumoral edema. Therefore, the correlation between T1 modality and T1ce modality is relatively strong, and the correlation between T2 modality and FLAIR modality is relatively strong. In addition, T1ce modality and FLAIR modality perform well in single-modality segmentation, and T1 modality and T2 modality exhibit similar physical properties, so these two groups of modalities have certain correlation. Based on this, as shown in Figure 2 , the determination method of the mutual features obtained by decoupling each modality is as follows:

[0074] Among the four groups of features finally obtained by the branch corresponding to the T1 modality, the first group of features is the self-features s t1 of the T1 modality, the second group of features is the mutual features u t1→tc between the T1 modality and the T1ce modality, the third group of features is the mutual features u t1→t2 between the T1 modality and the T2 modality, and the fourth group of features is the mutual features u t1→fl between the T1 modality and the FLAIR modality.

[0075] Among the four groups of features finally obtained by the branch corresponding to the T1ce modality, the first group of features is the self-features s tc of the T1ce modality, the second group of features is the mutual features u tc→t1 between the T1ce modality and the T1 modality, the third group of features is the mutual features u tc→fl between the T1ce modality and the FLAIR modality, and the fourth group of features is the mutual features u tc→t2 between the T1ce modality and the T2 modality.

[0076] Among the four groups of features finally obtained by the branch corresponding to the T2 modality, the first group of features is the self-features s t2 of the T2 modality, the second group of features is the mutual features u t2→fl between the T2 modality and the FLAIR modality, the third group of features is the mutual features u t2→t1 between the T2 modality and the T1 modality, and the fourth group of features is the mutual features u t2→tc between the T2 modality and the T1ce modality.

[0077] Among the four groups of features finally obtained by the branches corresponding to the FLAIR modality, the first group of features is the self-feature s of the FLAIR modality fl , the second group of features is the mutual feature u of the FLAIR modality and the T2 modality fl→t2 , the third group of features is the mutual feature u of the FLAIR modality and the T1ce modality fl→tc , and the fourth group of features is the mutual feature u of the FLAIR modality and the T1 modality fl→t1 .

[0078] In this embodiment, the purpose of the feature compensation module is to use the results of feature decoupling to construct real full-modal features for the full modality:

[0079] F fused = [s t1 ·s tc ·s t2 ·s fl ]

[0080] Where [·] represents feature concatenation along the channel dimension.

[0081] In the case of missing modality, the mutual feature of the non-missing modality is used to compensate for the related feature of the missing modality. Compared with directly replacing the related feature of the missing modality with a zero matrix, the present embodiment can construct a pseudo full-modal feature that is closer to the scene. In order to specifically handle different missing modality cases, the present embodiment combines the above clinical prior knowledge to design a modality feature compensation mechanism (RCR), as shown in Figure 4 , where δ = [δ t1 , δ tc , δ t2 , δ fl ] is an MRI modality indicator vector, indicating whether the T1 modality, the T1ce modality, the T2 modality and the FLAIR modality exist, and when a modality is missing, the mutual feature used to compensate for the missing modality is determined according to the priority as shown in Figure 4 , and the details are as follows:

[0082] When the missing modality is the T1 modality, if the T1ce modality is not missing, the mutual feature of the T1ce modality and the T1 modality is selected to compensate for it; if the T1ce modality is missing and the T2 modality is not missing, the mutual feature of the T2 modality and the T1 modality is selected to compensate for it; if both the T1ce modality and the T2 modality are missing, the mutual feature of the FLAIR modality and the T1 modality is selected to compensate for it;

[0083] When the missing modality is the T1ce modality, if the T1 modality is not missing, the cross features of the T1 modality and the T1ce modality are selected to compensate for the missing T1 modality; if the T1 modality is missing and the FLAIR modality is not missing, the cross features of the FLAIR modality and the T1ce modality are selected to compensate for the missing T1 modality; if the T1 modality and the FLAIR modality are both missing, the cross features of the T2 modality and the T1ce modality are selected to compensate for the missing T1 modality.

[0084] When the missing modality is the T2 modality, if the FLAIR modality is not missing, the cross features of the FLAIR modality and the T2 modality are selected to compensate for the missing T2 modality; if the FLAIR modality is missing and the T1 modality is not missing, the cross features of the T1 modality and the T2 modality are selected to compensate for the missing T2 modality; if the FLAIR modality and the T1 modality are both missing, the cross features of the T1ce modality and the T2 modality are selected to compensate for the missing T2 modality.

[0085] When the missing modality is the FLAIR modality, if the T2 modality is not missing, the cross features of the T2 modality and the FLAIR modality are selected to compensate for the missing FLAIR modality; if the T2 modality is missing and the T1ce modality is not missing, the cross features of the T1ce modality and the FLAIR modality are selected to compensate for the missing FLAIR modality; if the T2 modality and the T1ce modality are both missing, the features of the T1 modality and the FLAIR modality are selected to compensate for the missing FLAIR modality.

[0086] For example, when the T1 modality and the T1ce modality are missing, based on the above modality feature compensation mechanism, the corresponding cross features of the T2 modality and the FLAIR modality will compensate for the T1 and the T1ce respectively, and the pseudo full modality features at this time are:

[0087] F fused = [u T2→T1 ·u FLAIR→T1ce ·s T2 ·s FLAIR ]

[0088] In addition to the feature decoupling module and the feature compensation module, the improved 3DU-Net network constructed in this embodiment uses 2 convolution units to encode the features at each feature scale in the encoder, and the convolution unit structure is the same as that used in step 2, except that the convolution step of the first convolution unit here is 2 to realize the down sampling of the features. The convolution kernel numbers of the different scales of the encoder are 64, 128, 256, 320, and 320. In the decoder, transposed convolution is used to up sample the features, and the convolution kernel size is 2x2x2 and the step is 2. When decoding the features, the features of the same scale of the encoder and the decoder are first spliced in the channel layer, and then pass through 2 convolution units, and the parameters are the same as those used in step 2. Finally, the output probability atlas is obtained by using convolution, and the convolution kernel size is 1x1x1 and the step is 1.

[0089] In this embodiment, the model training includes: training the established improved 3D U-Net network by using the brain glioma segmentation dataset, and obtaining a brain glioma segmentation model after the training is completed.

[0090] In order to guide the model to produce more accurate brain glioma segmentation results, the present application uses a deep supervision strategy to improve the model segmentation performance. Specifically, the decoder part is further provided with a segmentation head on each feature, which is used to obtain the output probability atlas at the corresponding scale. When training the improved 3D U-Net network by using the brain glioma segmentation dataset, the training loss function includes the sum of the segmentation losses corresponding to the output probability atlas at each scale. The segmentation loss corresponding to the output probability atlas at each scale is the loss of the output probability atlas at the scale with respect to the brain glioma segmentation label after being down-sampled to the same scale.

[0091] The segmentation loss corresponding to the output probability atlas at each scale specifically includes a Dice loss and a cross-entropy loss, which can be expressed by the following formula:

[0092]

[0093] wherein K represents the total number of classes, N represents the total number of voxels, represents the true label value and the predicted value of the nth voxel of class k.

[0094] Optionally, a stochastic gradient descent (SGD) optimizer is used to update the network model weight during the training, the initial learning rate is 0.01, and the total iteration training is 1000 rounds. After the training is completed, the brain glioma segmentation model can be obtained. The brain glioma segmentation model can process brain glioma segmentation under the full modal and 14 different missing modal scenarios, which has important significance for clinical application. It is easy to understand that the smaller scale of the decoder in the brain glioma segmentation model does not contain the segmentation head when actually performing brain glioma segmentation.

[0095] In summary, the brain glioma segmentation model established in this embodiment can decouple the self-features and mutual-features of each modality, extract modality features with only a small amount of parameters, reduce the feature learning burden, achieve effective modality adaptation, and further improve the model segmentation performance. At the same time, a new attention mechanism, i.e. a channel-scale sparse self-attention mechanism (CSSA), is introduced, which can realize effective cross-guiding between the modality self-features and mutual-features, and prevent them from being coupled again. In addition, a feature compensation mechanism based on clinical prior knowledge is introduced, which can effectively construct pseudo full modal features close to the real scene in the case of missing modalities, which is more conducive to model training and improves the segmentation performance.

[0096] Embodiment 2:

[0097] A brain glioma segmentation method, comprising:

[0098] After registering the MRI image of the brain tissue to be segmented to the same modality, inputting the brain glioma segmentation model established by the feature decoupling-based missing modality brain glioma segmentation model establishment method provided in the above embodiment 1 to obtain a brain glioma segmentation result.

[0099] Since the brain glioma segmentation model used in the embodiment can effectively decouple the self-features and mutual-features of the modalities and effectively compensate for the features of the missing modalities, the embodiment can effectively improve the segmentation performance under the condition of missing modalities.

[0100] The embodiment is experimentally verified on multi-modality MRI images of real brain glioma patients, and experiments are conducted by inputting full modalities and simulating 14 missing modalities. The experimental results show that the brain glioma segmentation method provided in the embodiment can produce accurate brain glioma segmentation results under full modalities and various missing modalities, which reflects the robustness of the model to missing modalities. In addition, comparisons are made with other advanced missing modality brain glioma segmentation algorithms, and the quantitative and qualitative experimental results further illustrate the effectiveness and superiority of the method.

[0101] In order to quantify the comparison results, the Dice Similarity Coefficient (DSC) which is the most commonly used index in segmentation is used to measure the brain glioma segmentation results in the experiment, which is defined as follows:

[0102]

[0103] Where X, Y represent the prediction results and the true labels. The DSC value range is [0, 1], and the larger the value, the higher the degree of coincidence of the two sets. The quantitative comparison results are shown in Figure 5 As shown in the table, four latest and most advanced missing modality brain glioma segmentation methods are compared in the experiment, which are RFNet, mmFormer, MAVP and GSS. The brain glioma segmentation results output by the model are converted into three sub-regions for comparison, i.e. Whole Tumor (WT), Tumor Core (TC) and Enhancing Tumor (ET). Figure 5 The results in the table show that the full modality and different missing modality brain glioma segmentation results produced by the embodiment all obtain the highest average DSC in the three sub-regions, which is higher than the current most advanced method GSS. In addition, when two or three modalities are missing, it is usually difficult to obtain good segmentation results due to the very limited information that can be provided by the available modalities. Figure 5The results of the first 10 rows show that for these difficult-to-handle scenarios, the embodiment still achieves significantly better segmentation results than GSS. These quantitative results fully demonstrate that the decoupled self-modal feature expression, channel sparse self-attention mechanism and feature compensation mechanism based on clinical prior knowledge proposed by the embodiment can enhance the ability of the model to extract features from the existing modalities, thereby maximizing the use of these features, and then achieving high-quality brain glioma segmentation in different missing modal scenarios.

[0104] Figure 6 and Figure 7 are two brain glioma patient segmentation results generated using the present application, which show the segmentation results in the full modal and 14 missing modal cases, respectively. The first column of the figure shows the 4 MRI modalities of the patient's head, and the second to fifth columns are the segmentation results in different missing modal scenarios, t1, tc, t2, fl and GT represent T1, T1ce, T2, FLAIR and the true label, respectively. The segmentation results of the embodiment in different scenarios are very close to the true label, and high-quality brain glioma segmentation results can be achieved even when multiple modalities are missing or only one modality is present.

[0105] Figure 8 and Figure 9 shows a comparison of the visualization results of the method of the present application with other advanced missing modal brain glioma segmentation algorithms. Figure 8 and Figure 9 shows two test results, which correspond to Figure 6 and Figure 7 , which shows the visualization results of the four cases of only FLAIR modality, T1ce modality and FLAIR modality, missing T1 modality and full modality. The first column in the figure is the input 4 MRI modalities, the second column is the segmentation result of the RFNet method, the third column is the segmentation result of the GSS method, the fourth column is the segmentation result of the embodiment, and the last column is the true label. Figure 8 The results show that compared with other methods, the embodiment can achieve more accurate segmentation results when the missing modalities are severe, and can obtain segmentation results closer to the true label when the full modalities and the missing modalities are less. Figure 9 The arrow indicates the part of the tumor core shape that is complex. It can be seen that in the full modal and most missing modal cases, the segmentation results of other methods differ greatly from the true label, while the segmentation results obtained by the method of the present application are closer in shape to the true label, fully demonstrating that the embodiment has superior segmentation performance in full modal and missing modal.

[0106] In summary, the embodiment does not require multiple large encoders, complex feature fusion modules or auxiliary regularizers to achieve optimal brain glioma segmentation results.

[0107] Embodiment 3:

[0108] A computer readable storage medium, comprising a stored computer program; the computer program is executed by a processor to control a device where the computer readable storage medium is located to perform the feature decoupling based missing modality brain glioma segmentation model establishing method provided in the above embodiment 1, and / or the brain glioma segmentation method provided in the above embodiment 2.

[0109] Embodiment 4:

[0110] An electronic device, comprising:

[0111] A computer readable storage medium for storing a computer program;

[0112] and a processor for reading the computer program stored in the computer readable storage medium, performing the feature decoupling based missing modality brain glioma segmentation model establishing method provided in the above embodiment 1, and / or the brain glioma segmentation method provided in the above embodiment 2.

[0113] Those skilled in the art will easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for establishing a segmentation model for gliomas with a missing modality based on feature decoupling, characterized in that, include: We collected MRI images of brain tissue labeled with glioma segmentation tags, registered MRI images of the same brain tissue in different modalities to the same modality, and then normalized them to obtain a glioma segmentation dataset. After initializing the 3D U-Net network, its first layer is replaced with interconnected feature decoupling modules and feature compensation modules to obtain an improved 3D U-Net network. The network is then trained using the glioma segmentation dataset. After training, a glioma segmentation model is obtained. The feature decoupling module includes four branches, corresponding to four modes; arbitrary modes m The corresponding branch includes: four parallel feature extraction units, each for modality... m Feature extraction was performed on brain tissue images to obtain four sets of intermediate decoupling features; and channel-sparse self-attention units were used to concatenate the four sets of intermediate decoupling features by channel to obtain the feature set. Then, the features Attention scores are assigned to each channel, and features are analyzed in descending order of attention scores. After sorting each channel, it is compared with the features Add them together to obtain the features. , will feature The features are divided into four groups along the channel. The first group of features is the modality. m The self-features, the other three sets of features are modalities. m Mutual characteristics with other modalities; The feature compensation module is used to construct full-modal features, wherein the features of the non-missing modal are the self-features of the modality, and the features of the missing modal are the mutual features of the non-missing modal and the missing modal.

2. The method for establishing a missing modality glioma segmentation model based on feature decoupling as described in claim 1, characterized in that, When constructing full-modal features, the feature compensation module determines the mutual features used to compensate for missing modalities according to the following priority order: When the missing mode is the T1 mode, if the T1ce mode is not missing, the mutual features between the T1ce mode and the T1 mode are selected to compensate for it. If the T1ce mode is missing but the T2 mode is not missing, then the mutual features of the T2 mode and the T1 mode are selected for compensation. If both T1ce and T2 modes are missing, the mutual features between the FLAIR mode and the T1 mode are selected to compensate for them. When the missing mode is the T1ce mode, if the T1 mode is not missing, the mutual features between the T1 mode and the T1ce mode are selected to compensate for it. If the T1 mode is missing but the FLAIR mode is not missing, then the mutual features between the FLAIR mode and the T1ce mode are selected to compensate for it. If both the T1 mode and the FLAIR mode are missing, the mutual features of the T2 mode and the T1ce mode are selected to compensate for them. When the missing mode is the T2 mode, if the FLAIR mode is not missing, the mutual features between the FLAIR mode and the T2 mode are selected to compensate for it. If the FLAIR mode is missing but the T1 mode is not missing, then the mutual features of the T1 and T2 modes are selected for compensation. If both the FLAIR mode and the T1 mode are missing, the mutual features of the T1ce mode and the T2 mode are selected to compensate for them. When the missing mode is the FLAIR mode, if the T2 mode is not missing, the mutual features between the T2 mode and the FLAIR mode are selected to compensate for it. If the T2 mode is missing but the T1ce mode is not missing, the mutual features of the T1ce mode and the FLAIR mode are selected for compensation; if both the T2 mode and the T1ce mode are missing, the features of the T1 mode and the FLAIR mode are selected for compensation.

3. The method for establishing a missing modality glioma segmentation model based on feature decoupling as described in claim 2, characterized in that, In the feature decoupling module, among the four sets of features finally obtained from the branch corresponding to the T1 mode, the first set of features is the self-feature of the T1 mode. The second set of features consists of the mutual features between the T1 mode and the T1ce mode. The third set of features consists of the mutual features between the T1 and T2 modes. The fourth set of features consists of the mutual features between the T1 mode and the FLAIR mode. ; Of the four sets of features obtained from the branch corresponding to the T1ce mode, the first set of features is the self-feature of the T1ce mode. The second set of features consists of the mutual features between the T1ce mode and the T1 mode. The third set of features consists of the mutual features between the T1ce mode and the FLAIR mode. The fourth set of features consists of the mutual features between the T1ce mode and the T2 mode. ; Of the four sets of features obtained from the branch corresponding to the T2 mode, the first set of features is the self-feature of the T2 mode. The second set of features consists of the mutual features between the T2 mode and the FLAIR mode. The third set of features consists of the mutual features between the T2 and T1 modes. The fourth set of features consists of the mutual features between the T2 mode and the T1ce mode. ; Of the four sets of features obtained from the branches corresponding to the FLAIR mode, the first set of features is the self-feature of the FLAIR mode. The second set of features consists of the mutual features between the FLAIR mode and the T2 mode. The third set of features consists of the mutual features between the FLAIR mode and the T1ce mode. The fourth set of features consists of the mutual features between the FLAIR mode and the T1 mode. .

4. The method for establishing a missing modality glioma segmentation model based on feature decoupling as described in any one of claims 1 to 3, characterized in that, In the feature decoupling module, each feature extraction unit is a convolutional module.

5. The method for establishing a missing modality glioma segmentation model based on feature decoupling as described in any one of claims 1 to 3, characterized in that, In the improved 3D U-Net network, the decoder part also has a segmentation head set on each feature to obtain the output probability map at the corresponding scale; Furthermore, when training the brain glioma segmentation dataset, the training loss function includes the sum of segmentation losses corresponding to the output probability map at each scale; The segmentation loss corresponding to the output probability map at each scale is the loss of the output probability map at that scale relative to the glioma segmentation label after downsampling to the same scale.

6. The method for establishing a missing modality glioma segmentation model based on feature decoupling as described in claim 5, characterized in that, The segmentation loss corresponding to the output probability map at each scale includes Dice loss and cross-entropy loss.

7. A method for segmenting a glioma, characterized in that, include: After registering the MRI images of the brain tissue to be segmented to the same modality, the glioma segmentation model established by the feature decoupling-based missing modality glioma segmentation model establishment method described in any one of claims 1 to 6 is input to obtain the glioma segmentation result.

8. A computer-readable storage medium, characterized in that, The computer program includes a stored computer program; when the computer program is executed by a processor, it controls the device containing the computer-readable storage medium to execute the method for establishing a missing modality glioma segmentation model based on feature decoupling as described in any one of claims 1 to 6, or the glioma segmentation method as described in claim 7.

9. An electronic device, characterized in that, include: A computer-readable storage medium for storing computer programs; And a processor, configured to read a computer program stored in the computer-readable storage medium and execute the method for establishing a missing modality glioma segmentation model based on feature decoupling as described in any one of claims 1 to 6, or the glioma segmentation method as described in claim 7.

Images