Methods, devices, equipment, media, and products for segmenting brain tumor images with missing modalities

Abstract

This invention provides a method, apparatus, device, medium, and product for segmenting brain tumor images with missing modalities, belonging to the field of image segmentation technology. It constructs a knowledge distillation model architecture, including a teacher network and a student network. The student network includes at least one feature reconstruction module, which is used to reconstruct the features of brain tumor images with missing modalities based on the semantic correlations between brain tumor images of different modalities. The student network is trained based on the trained teacher network, and an image segmentation model is obtained from the trained student network. The image segmentation model is used to segment brain tumor images with missing modalities. The student network of this invention can reconstruct the features of brain tumor images with missing modalities based on the semantic correlations between brain tumor images of different modalities, thus better representing the features of brain tumor images with missing modalities and effectively improving the segmentation accuracy of brain tumor images with missing modalities.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to a method, apparatus, device, medium, and product for segmenting brain tumor images with missing modalities. Background Technology

[0002] Multimodal brain tumor magnetic resonance imaging (MRI) images include four MRI modalities: T1-weighted imaging, T2-weighted imaging, T1 contrast-enhanced weighted imaging, and Fluid Attenuated Inversion Recovery (Flair). Missing modal brain tumor images refer to multimodal MRI images lacking any one or more modalities. In practical applications, due to various reasons such as equipment limitations, poor imaging conditions, and data acquisition errors, brain tumor image data with missing modalities often occurs, making it impossible to obtain complete multi-view brain tumor information and reducing the accuracy of brain tumor image segmentation.

[0003] In recent years, knowledge distillation, as an emerging technique, has been widely applied to solve problems involving incomplete modal data processing. The core principle of knowledge distillation is to transfer the knowledge held by a complex teacher model to a student model, allowing the student model to achieve performance levels similar to the teacher model while maintaining lower computational complexity. In the field of multimodal data processing, cross-modal knowledge distillation aims to transfer the knowledge of a teacher model trained on full-modal data to a student model trained on incomplete modal data. However, teacher models, focusing on full-modal data, struggle to effectively capture and learn the key features of missing modalities, resulting in student models being unable to accurately segment data using existing information. Therefore, the accuracy of segmenting brain tumor images with missing modalities still needs improvement. Summary of the Invention

[0004] This invention provides a method, apparatus, device, medium, and product for segmenting brain tumor images with missing modalities. It addresses the problem that teacher models, which focus on full-modal data, struggle to effectively capture and learn key features of missing modalities, resulting in student models being unable to accurately segment using existing information, and the accuracy of brain tumor image segmentation with missing modalities still needing improvement.

[0005] This invention provides a method for segmenting brain tumor images with missing modalities, comprising:

[0006] Acquire images of brain tumors with missing modalities to be segmented;

[0007] The brain tumor image of the missing modality to be segmented is input into the image segmentation model to obtain the image segmentation result output by the image segmentation model;

[0008] The training steps of the image segmentation model include:

[0009] A knowledge distillation model architecture is constructed, which includes a teacher network and a student network. The student network includes at least one feature reconstruction module, which is used to reconstruct brain tumor image features of missing modalities based on the semantic correlation between brain tumor images of different modalities.

[0010] The student network is trained based on the trained teacher network, and the image segmentation model is obtained based on the trained student network.

[0011] As one embodiment, the feature reconstruction module includes:

[0012] The feature labeling unit is used to map the features of at least one modality of the input brain tumor image to a preset labeling space to obtain a first modality label, and based on the first modality label, obtain a second modality label corresponding to the brain tumor image with missing modality;

[0013] The feature reconstruction unit is used to add random spatial location codes to the merged first modality labels and second modality labels and input them into the feature reconstruction network to obtain the reconstructed brain tumor image features of each modality.

[0014] As one embodiment, mapping the input brain tumor image features of at least one modality to a preset labeling space to obtain a first modality label includes:

[0015] The features of at least one modality of the input brain tumor image are flattened into one-dimensional feature vectors;

[0016] The one-dimensional feature vector is mapped to the label space based on a preset projection function to obtain the first modality label;

[0017] Correspondingly, obtaining the second modality marker corresponding to the brain tumor image with missing modality based on the first modality marker includes:

[0018] Based on the average feature representation of the first modality label, the second modality label corresponding to the brain tumor image with missing modality is obtained.

[0019] As one embodiment, training the student network based on the trained teacher network includes:

[0020] Construct a dataset and a loss function, wherein the loss function includes at least a knowledge distillation loss function and a feature reconstruction loss function. The knowledge distillation loss function is used to characterize the degree of difference between the predicted probability distribution of the trained teacher network and the predicted probability distribution of the student network, and the feature reconstruction loss function is used to characterize the difference between the image features extracted by the trained teacher network and the image features reconstructed by the student network.

[0021] Configure the initial parameters of the student network based on the parameters of the trained teacher network;

[0022] Based on the dataset and the loss function, the initial parameters are optimized to complete the training of the student network.

[0023] As an example, the loss function further includes a segmentation loss sub-function, which is used to characterize the degree of overlap between the predicted result and the true label, as well as the sum of the differences between the predicted probability distribution and the true label distribution.

[0024] As an example, the dataset includes a subset of full-modality brain tumor image data and a subset of missing-modality brain tumor image data. The subset of full-modality brain tumor image data is used to train a teacher network based on full-modality brain tumor images, and the subset of missing-modality brain tumor image data is used to train a student network based on random missing-modality brain tumor images.

[0025] The present invention also provides a brain tumor image segmentation device with missing modality, comprising:

[0026] A training module is used to construct a knowledge distillation model architecture, which includes a teacher network and a student network. The student network includes at least one feature reconstruction module, which is used to reconstruct brain tumor image features of missing modalities based on the semantic correlation between brain tumor images of different modalities. The student network is trained based on the trained teacher network, and an image segmentation model is obtained based on the trained student network.

[0027] The acquisition module is used to acquire brain tumor images of missing modalities to be segmented;

[0028] The segmentation module is used to input the brain tumor image of the missing modality to be segmented into the image segmentation model to obtain the image segmentation result output by the image segmentation model.

[0029] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the missing modality brain tumor image segmentation method as described above.

[0030] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the missing modality brain tumor image segmentation method as described above.

[0031] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the missing modality brain tumor image segmentation method as described above.

[0032] The missing modality brain tumor image segmentation method, apparatus, device, medium and product provided by the present invention, by setting at least one feature reconstruction module in the student network, can reconstruct the features of missing modality brain tumor images based on the semantic correlation between brain tumor images of different modalities, which can better characterize the features of missing modality brain tumor images for accurate image segmentation and effectively improve the accuracy of missing modality brain tumor image segmentation. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating the missing modality brain tumor image segmentation method provided by the present invention.

[0035] Figure 2 This is a schematic diagram of the training process of the knowledge distillation model architecture provided by the present invention.

[0036] Figure 3 This is a schematic diagram of the student network structure provided by the present invention.

[0037] Figure 4 This is a schematic diagram of the feature reconstruction module provided by the present invention.

[0038] Figure 5 This is a schematic diagram of the structure of the missing modality brain tumor image segmentation device provided by the present invention.

[0039] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0041] Currently, there are two main types of methods in the field of brain tumor image segmentation with missing modalities: targeted methods and comprehensive methods.

[0042] The key to targeted methods lies in training specialized models for different missing modalities. This approach primarily includes two strategies: co-training and data generation. The co-training strategy accurately transfers key information from the full-modality network to the missing modality network, allowing the model to fully leverage the rich features learned from the full-modality network when processing missing modality data. This results in a more detailed and accurate understanding of the missing modality data, effectively improving segmentation accuracy. Meanwhile, the data generation strategy based on generative adversarial networks supplements the missing modality information at the data level, providing a more complete data foundation for subsequent segmentation tasks, enhancing the model's adaptability to complex data distributions, and further improving the accuracy and stability of segmentation.

[0043] Targeted approaches have certain limitations. On the one hand, methods based on co-training require training a separate network for each modality combination, significantly increasing the cost of model deployment. On the other hand, data generation strategies based on Generative Adversarial Networks (GANs) face training difficulties when used for 3D image generation, and incur additional resource consumption during training and practical applications. Furthermore, generative models struggle to fully reproduce images with true contrast; for example, gadolinium contrast agents are indispensable in practice, but generative models often fail to reproduce their effects.

[0044] Comprehensive approaches attempt to address all missing modalities using a single, general model. These methods typically employ modality-specific encoders to embed data from different modalities into a shared latent space, followed by feature fusion and subsequent processing for segmentation. Multimodal variational autoencoders reconstruct modalities from common latent variables, fostering truly shared latent feature representations. However, employing complex architectures with multiple encoders (and sometimes decoders) not only increases the difficulty and computational cost of model training but may also lead to decreased model interpretability.

[0045] This invention provides a method, apparatus, device, medium, and product for segmenting brain tumor images with missing modalities. It improves the network structure of the student model, enabling it to effectively capture and learn the key features of missing modalities, achieve accurate image segmentation, and improve the overall accuracy of segmenting brain tumor images with missing modalities.

[0046] Figure 1 This is a flowchart illustrating the missing modality brain tumor image segmentation method provided by the present invention, as shown below. Figure 1 As shown, the present invention provides a method for segmenting brain tumor images with missing modalities, including steps S100-S200, where steps S100-S200 are the application stages of the present invention.

[0047] S100, acquire the brain tumor image with missing modalities to be segmented. The brain tumor image with missing modalities to be segmented refers to a multimodal MRI image acquired by the MRI equipment that is missing one or more modalities of any of the four MRI modalities.

[0048] S200, the brain tumor image of the missing modality to be segmented is input into the image segmentation model to obtain the image segmentation result output by the image segmentation model. The image segmentation model is used to segment the brain tumor region in the image. Correspondingly, the image segmentation result includes one or more of binary segmentation results, multi-class segmentation results, and boundaries or contours. The binary segmentation result is used to characterize tumor and non-tumor regions, the multi-class segmentation result is used to mark multiple different structures or tissues, and the boundaries or contours are used to describe the boundary position and shape of the structure.

[0049] The training steps of the image segmentation model include steps S010-S020, which are performed before step S100.

[0050] S010, Construct a knowledge distillation model architecture, which includes a teacher network and a student network. The student network includes at least one feature reconstruction module, which is used to reconstruct the features of brain tumor images with missing modalities based on the semantic correlation between brain tumor images of different modalities. The teacher network and student network are constructed using the same network structure to facilitate knowledge transfer.

[0051] S020, the student network is trained based on the trained teacher network, and the image segmentation model is obtained based on the trained student network.

[0052] In one embodiment, the image segmentation model includes a trained student network for segmenting brain tumor images with missing modalities. In a preferred embodiment, the image segmentation model includes a trained teacher network and a trained student network; the trained teacher network can be used for segmenting brain tumor images with all modalities, and the trained student network can be used for segmenting brain tumor images with missing modalities.

[0053] It is understood that by setting at least one feature reconstruction module in the student network, the present invention can reconstruct the features of brain tumor images with missing modalities based on the semantic correlation between brain tumor images of different modalities, which can better characterize the features of brain tumor images with missing modalities for accurate image segmentation and effectively improve the segmentation accuracy of brain tumor images with missing modalities.

[0054] Based on the above embodiments, as an optional embodiment, the training of the student network based on the trained teacher network includes steps S021-S023.

[0055] Step S021: Construct a dataset and a loss function. The loss function includes at least a knowledge distillation loss function and a feature reconstruction loss function. The knowledge distillation loss function is used to characterize the degree of difference between the predicted probability distribution of the trained teacher network and the predicted probability distribution of the student network. The feature reconstruction loss function is used to characterize the difference between the image features extracted by the trained teacher network and the image features reconstructed by the student network.

[0056] Step S022: Configure the initial parameters of the student network based on the parameters of the trained teacher network.

[0057] Step S023: Based on the dataset and the loss function, optimize the initial parameters to complete the training of the student network.

[0058] As an optional embodiment, the dataset includes a subset of full-modality brain tumor image data and a subset of missing-modality brain tumor image data. The subset of full-modality brain tumor image data is used to train a teacher network based on full-modality brain tumor images, and the subset of missing-modality brain tumor image data is used to train a student network based on random missing-modality brain tumor images.

[0059] Suppose there exists a set of brain tumor image data. Each of the images Represented as a set of three-dimensional matrices ,in This represents the number of modes in each matrix group. x, y, z These correspond to the number of voxels in the three spatial dimensions of the image. Each image It contains four modes, which can be expanded along the modality dimension, and are denoted as follows: and ,Right now .gather This can be denoted as a subset of full-modality brain tumor image data.

[0060] In real-world scenarios, modality loss may occur for various reasons, meaning one or more modalities of data may be randomly missing. From a combinatorial perspective, there are 15 possible modality loss scenarios. For example, if an image... The j-th mode is missing (j takes a value of 1 to represent...) 2 represents 3 represents 4 represents ), can be represented as . Set The 15 missing cases of any image are combined and denoted as the missing modality brain tumor image data subset.

[0061] The core task of multimodal brain tumor segmentation in the absence of modalities is to find a segmentation function. S This makes it possible for any missing case All . Is with Segmentation mask matrices of the same size Represents a label. , where 0 represents the background and 1 represents the tumor area.

[0062] This invention trains a teacher network using a subset of full-modality brain tumor image data, rather than training a teacher network for a single modality. This reduces the complexity of the knowledge distillation model architecture. The full-modality brain tumor image data subset corresponds to the missing modality brain tumor image data subset, which facilitates the teacher network in guiding the reconstruction features of the student network during the training or application phases, thereby improving the accuracy of the reconstruction features.

[0063] The image segmentation model based on knowledge distillation and feature reconstruction provided by this invention mainly includes two stages in the training process: teacher network pre-training and student network training, such as... Figure 2 As shown, Figure 2 A one-way arrow is used to represent the direction of data flow, while a double arrow represents the loss function.

[0064] Both the teacher network and the student network are based on image segmentation networks. This embodiment of the invention uses the Vnet image segmentation network as an example. Vnet is a deep learning convolutional neural network structure for medical image segmentation, specifically designed for automatic segmentation tasks of 3D medical images. The teacher network is directly constructed using the Vnet network, while the student network improves upon the Vnet network by adding at least one feature reconstruction module. For example... Figure 3 In the preferred embodiment shown, the student network uses Vnet as the backbone network. A feature reconstruction module is introduced in the last three stages of the Vnet network. The feature reconstruction module utilizes the semantic correlation between brain tumor images of different modalities to realize feature reconstruction of brain tumor images with missing modalities.

[0065] The teacher's online pre-training uses full-modality brain tumor image samples from a subset of the full-modality brain tumor image data for training. The knowledge learned by the teacher's network refers to the knowledge of segmenting full-modality brain tumor images, which can provide a reference for students' online training.

[0066] After the teacher network pre-training is completed, the student network training phase begins. During the student network training phase, the teacher network parameters are kept frozen, and knowledge distillation technology is used to transfer the knowledge of the teacher network to the student network. That is, the parameters of the trained teacher network are transferred to the student network to obtain the initial parameters of the student network.

[0067] Considering that the segmentation problem can essentially be reduced to a pixel-level classification problem, this embodiment of the invention adopts the common sense distillation method, which guides the student network to learn the knowledge of the teacher network by minimizing the Kullback-Leibler (KL) divergence between the predicted probability distribution of the student network and the predicted probability distribution of the teacher network.

[0068] Assume the teacher network processes the input image I The predicted probability distribution is The predicted probability distribution of the student network is as follows: The general formula for calculating the KL divergence is as follows:

[0069]

[0070] in, Let KL divergence be denoted as KL divergence.

[0071] Suppose a full-modal teacher network is used for input image data I The predicted probability distribution is The predicted probability distribution of the missing modality student model is as follows: The formula for calculating the pixel-by-pixel knowledge distillation loss function is as follows:

[0072]

[0073] in, This represents the softmax operation, which transforms the original predicted values ​​into a probability distribution. T This indicates that the temperature parameter is out of range; adjust accordingly. T The value controls the degree of softening in the softmax operation output, optimizing the knowledge distillation effect; KL represents the KL divergence operation, used to measure the degree of difference between two predicted probability distributions.

[0074] Teacher networks are used not only to guide student networks in predicting probability distributions, but also to guide student networks in reconstructing accurate brain tumor image features with missing modalities.

[0075] set up The full-modality brain tumor image features extracted by the teacher network during the training phase of the student network. Let the brain tumor image features representing the missing modalities reconstructed during the student's network training phase be represented by the following formula for calculating the feature reconstruction loss function:

[0076]

[0077] in, This represents the mean squared error function, used to calculate the mean squared error between the image features extracted by the teacher network and the image features reconstructed by the student network. The feature reconstruction loss function can be used to guide the student network in optimizing the reconstruction process, improving the reconstruction quality of brain tumor image features with missing modalities.

[0078] Based on the dataset and the loss function, optimizing the initial parameters to complete the training of the student network specifically refers to randomly extracting full-modality image samples and corresponding missing modality image samples from the dataset, inputting the full-modality image samples into the trained teacher network and the missing modality image samples into the student network, calculating the knowledge distillation loss function value and the feature reconstruction loss function value based on the above formulas (1)-(3), determining whether the knowledge distillation loss function value and the feature reconstruction loss function value are within the preset loss range, if so, determining that the student network training is complete, otherwise, adjusting the parameters of the Vnet network of the student network based on the knowledge distillation loss function value, adjusting the parameter value of the feature reconstruction module based on the feature reconstruction loss function value, and returning to execute the step of randomly extracting full-modality image samples and corresponding missing modality image samples from the dataset.

[0079] Specifically, during the student network training process, in order to fully simulate the modality missing situations that may occur in real-world scenarios, 15 modality combinations in the dataset are sampled to obtain the missing modality combinations for the input. The existence of each mode is determined by... It means that, among them ∈{0,1}, =0 indicates that the mode is missing. =1 indicates that the mode exists. For each missing mode... The input part is replaced with a matrix of zeros.

[0080] In this embodiment of the invention, a Feature Reconstruction Module (FRM module) is added to the last three stages of the Vnet network to obtain a student network. The FRM module reconstructs the features of the missing modalities based on existing modal features and learnable modality-specific labels. The FRM module has learnable modality-specific labels and continuously optimizes its parameters through a large amount of training data, thereby accurately capturing the potential relationships between different modalities and the general semantic features of each modality, and realizing the reconstruction of high-level semantic information of the missing modalities.

[0081] The loss function of this invention includes at least a knowledge distillation loss sub-function and a feature reconstruction loss sub-function, which can improve the accuracy of the predicted probability distribution of the student network and enhance the reconstruction quality of brain tumor image features with missing modalities.

[0082] As an optional embodiment, the loss function further includes a segmentation loss sub-function, which is used to characterize the degree of overlap between the predicted result and the true label, as well as the sum of the differences between the predicted probability distribution and the true label distribution.

[0083] The segmentation loss function includes Dice loss and cross-entropy loss. Dice loss can effectively measure the degree of overlap between the student network's prediction results and the sample's true labels, and it has a good effect on image segmentation tasks where the foreground and background are clearly distinguishable. Cross-entropy loss focuses on evaluating the difference between the student network's predicted probability distribution and the true label distribution. Combining the two can optimize the model's segmentation performance from different perspectives.

[0084] The formula for calculating the segmentation loss function is shown below:

[0085]

[0086] in, This represents the segmentation loss function. Represents cross-entropy loss, Indicates dice loss. This indicates the actual label.

[0087] Optionally, the learning process of the student network is jointly guided by the teacher network and the segmentation mask, and the corresponding formula for calculating the loss function is as follows:

[0088]

[0089] in, and This is a hyperparameter used to balance the relationship between different loss sub-functions. Used to adjust the influence of the feature reconstruction loss function on the total loss function. Used to adjust the degree of influence of the knowledge distillation loss sub-function on the total loss function.

[0090] It is understood that this invention constructs the total loss function by segmenting the loss function, the knowledge distillation loss function, and the feature reconstruction loss function. Furthermore, the proportions of the knowledge distillation loss function and the feature reconstruction loss function can be adjusted by hyperparameters, which can effectively improve the performance of the student network.

[0091] As an optional embodiment, the feature reconstruction module includes a feature labeling unit and a feature reconstruction unit.

[0092] The feature labeling unit is used to map the features of at least one modality of the input brain tumor image to a preset labeling space to obtain a first modality label, and based on the first modality label, obtain a second modality label corresponding to the brain tumor image with missing modality.

[0093] The feature reconstruction unit adds random spatial location encoding to the merged first modality label and second modality label and inputs it into the feature reconstruction network to obtain the reconstructed brain tumor image features of each modality. Optionally, the feature reconstruction network consists of four stacked deep learning Transformer layers, each Transformer layer corresponding to a feature channel.

[0094] Optionally, mapping the input brain tumor image features of at least one modality to a preset labeling space to obtain a first modality label includes:

[0095] The features of at least one modality of the input brain tumor image are flattened into one-dimensional feature vectors;

[0096] The one-dimensional feature vector is mapped to the label space based on a preset projection function to obtain the first modality label;

[0097] Correspondingly, obtaining the second modality marker corresponding to the brain tumor image with missing modality based on the first modality marker includes:

[0098] Based on the average feature representation of the first modality label, the second modality label corresponding to the brain tumor image with missing modality is obtained.

[0099] The feature reconstruction module can be used for feature reconstruction in the training phase and feature reconstruction in the application phase. The feature reconstruction process is the same in both phases. This embodiment of the invention will use feature reconstruction in the training phase as an example for illustration.

[0100] like Figure 4As shown, for the Vnet network, the first... Characteristics of each stage In terms of channel dimension, it is divided into four modal features, namely ,in Vnet Networks The feature reconstruction module in each stage first performs modal feature labeling processing on the brain tumor image features of at least one modality input.

[0101] For actual existing modal features, i.e., satisfying The modal features are mapped to first modal labels using the following formula by the feature reconstruction module. :

[0102]

[0103] The flatten operation flattens the three-dimensional feature map into a one-dimensional vector. Here, is the projection function, which maps the flattened features to a projection of dimension . The tag space, Indicates the length of the marker space. The hidden dimension of the labeled space.

[0104] for The modal features are used, and the feature reconstruction module employs learnable modal labels. (i.e., second modality labels) are used for representation. Learnable modality labels, on the one hand, convey specific modality feature information that needs to be reconstructed to the feature reconstruction network, and on the other hand, interact with existing modality features to learn shared modality information, thereby better reconstructing missing modality features. Specifically, It has initialization parameters that simulate the average features of the missing modalities. The feature reconstruction module will comprehensively utilize... Based on existing modal features, various modal features are obtained through inference using an attention mechanism; during the backpropagation phase of the model, these parameters are updated as the learning process progresses. In other words, It includes coarse feature simulation of missing modalities, which can be compared to the model's memory of features of a certain type of modality, similar to a doctor's prior knowledge of a specific modality.

[0105] After the tokenization process is completed, the feature reconstruction module merges all modality tokens along the channel dimension to obtain the input sequence of the feature reconstruction network. The specific formula is as follows:

[0106]

[0107] To more effectively characterize the spatial relationship of features, the embodiments of the present invention are as follows: Add randomly initialized spatial location coding Input to feature reconstruction network To obtain the modal features of the reconstruction. The feature reconstruction network uses a self-attention mechanism to learn potential shared features across different modalities and to complete missing modal features.

[0108] The reconstructed modal features are connected to the original modal features using residual connections to achieve efficient feature learning. The calculation formula is as follows:

[0109]

[0110] Finally, The shape is adjusted to the original size to complete the reconstruction of each missing modality and the learning process of modality-shared features.

[0111] It is understandable that the training samples have randomly missing modalities. Correspondingly, this invention constructs an input sequence through modal labels, which can obtain the average feature representation of the missing modalities during the training process. The feature reconstruction module adopts an attention mechanism, which can effectively utilize existing information and missing information from the missing label sequence and existing modal features to better represent the features of the missing modalities.

[0112] This invention constructs an innovative teacher-student network architecture, transferring full-modal knowledge to the student network through a knowledge transfer mechanism. It also designs a student network suitable for missing modalities, introducing a Feature Reconstruction Module (FRM) to enable the teacher network to guide the student network in learning features of the missing modalities. This invention outperforms existing methods in average segmentation performance across all modality combinations, and exhibits particularly superior performance in the extreme case where only a single modality exists.

[0113] Next, we will describe in detail the experimental hardware and software environment, the dataset used for the experiment, the experimental setup, the experimental evaluation indicators, and the experimental results of one embodiment of the present invention.

[0114] This invention was conducted on the BraTS2018 dataset. The BraTS2018 dataset contains 285 cases, all of which are multimodal MRI data, covering four modalities: T1, T2, T1ce, and Flair. The multimodal images of each case were manually annotated by experienced radiologists. In the segmentation task, it is necessary to distinguish the whole tumor (WT), enhancing tumor (ET), and tumor core (TC) from the background.

[0115] During the data preprocessing stage, to ensure the data met the requirements for model training, the data for each case was standardized. First, the data was normalized to zero mean and unit variance to eliminate intensity differences between samples caused by factors such as scanning equipment and parameters, allowing the model to better learn data features. Simultaneously, considering GPU memory limitations, each voxel was randomly cropped to a size of 128×128×128 before being input into the network during model training. To comprehensively evaluate model performance, the dataset was randomly divided into training, validation, and test sets. The training set consisted of 200 cases, the validation set of 28 cases, and the test set of 57 cases.

[0116] This invention is based on the PyTorch 1.12.1 deep learning framework and runs on a computing platform equipped with eight NVIDIA GeForce RTX-3090-24GB GPUs. For data preprocessing, data augmentation is performed on all input model data, including random pruning, random flipping, and random scaling. For model optimization, the AdamW algorithm is used as the optimizer, and the weight decay coefficient is set to... This is to prevent the model from overfitting. The initial learning rate is set empirically. The batch size for all training phases is set to 4. The teacher network pre-training phase consists of 1000 rounds, and the student network learning phase consists of 5000 rounds. Hyperparameters in the knowledge distillation process. Based on experience, the scaling factor is set to 10 in the total loss. Set to 5, scaling factor The value is set to 10. To increase attention to missing combinations with poor segmentation results, this invention also employs a dynamic sampling rule. After every 100 training epochs, the DICE value of each missing case on the validation set is evaluated. Subsequent training epochs then sample different missing cases according to a weight coefficient of 1-DICE. This invention uses the DICE metric, most widely used in medical image segmentation, as the evaluation metric for assessing segmentation results.

[0117] The method proposed in this invention is a comprehensive approach, applicable to every missing condition. To evaluate the effectiveness of this invention, it is compared with four other methods under all missing conditions: Unified Heterogeneous Variational Encoder-Decoder (U-HVED), Adversarial Co-training Network (ACN), Saliency-Guided Morphological Aware U-Shaped Neural Network Model (SMU-Net), and Region-aware Fusion Network (RFNet). ACN and SMU-Net are specific methods, while the other two are comprehensive approaches. Furthermore, to evaluate the performance of this invention in the case of extreme data loss (data containing only one modality), it was compared with several image segmentation methods specifically designed for this situation, such as the baseline method (a single-modal Vnet trained in a fully supervised mode), Knowledge Distillation Network (KD-Net), Privileged Multimodal Knowledge Learning (PMKL), and Prototype Knowledge Distillation Loss (ProtoKL). KD-Net, PMKL, and ProtoKL are all based on the principle of knowledge distillation.

[0118] Table 1. Comparative Experiment of Combinations with Complete WT Region Deletion

[0119]

[0120] Table 2 Comparative Experiment of Combinations with Complete TC Region Deletion

[0121]

[0122] Table 3 Comparison Experiment of Combinations with All Missing ET Regions

[0123]

[0124] In a brain tumor image segmentation experiment with missing modalities, this invention compared the results for all missing modalities in the WT, TC, and ET regions. In the WT and TC regions, the average scores of this invention were 83.5 and 74.6, respectively, second only to RFNet. In the ET region, the average score of this invention was 62.3, significantly exceeding other comparative methods, demonstrating a clear advantage in this region. The advantages of this invention lie in its innovative knowledge distillation architecture from all modalities to the missing modalities and the student network introducing a feature reconstruction module, which enables knowledge transfer, effectively utilizes existing data, reduces information loss, and enhances adaptability to incomplete data. In extreme cases of trimodal missing modalities, this invention shows significant advantages compared to existing comprehensive methods, leading in 7 out of 12 indicators across four scenarios, fully demonstrating its suitability for extreme situations.

[0125] Table 4 Comparison Experiment of Single-Mode Missing Combinations - T1 and T2 Data

[0126]

[0127] Table 5 Comparison Experiment of Single-Mode Missing Combinations - T1c and Flair Data

[0128]

[0129] As shown in Tables 4 and 5, in the case of a single modality, this invention is compared with the baseline method (a Vnet trained under full supervision in a single modality) and three knowledge distillation-based methods specifically designed for this scenario. The teacher network is trained on full modality data, achieving an average score of 80.93 in the WT, TC, and ET regions, providing a performance benchmark for other methods. When only T1 data is available, this invention ranks second in average dice score among all methods; when only T2 data is available, this invention achieves the highest scores in the TC and ET regions, with an average dice score significantly outperforming all other methods; when only T1ce data is available, this invention achieves the highest dice score in the WT region; and when only Flair data is available, the dice scores in the CO and EC regions are the highest, with an average dice score also leading. Overall, this invention has advantages in different single modalities, performs outstandingly in the T2 and Flair modalities, and achieves excellent results in some key regions in the T1 and T1c modalities, demonstrating strong competitiveness under single modality conditions.

[0130] The missing modality brain tumor image segmentation device provided by the present invention is described below. The missing modality brain tumor image segmentation device described below can be referred to in correspondence with the missing modality brain tumor image segmentation method described above.

[0131] Figure 5 This is a schematic diagram of the structure of the missing modality brain tumor image segmentation device provided by the present invention, as shown below. Figure 5 As shown, the present invention also provides a brain tumor image segmentation device with missing modalities, comprising the following modules.

[0132] Training module 510 is used to construct a knowledge distillation model architecture, which includes a teacher network and a student network. The student network includes at least one feature reconstruction module, which is used to reconstruct the features of brain tumor images with missing modalities based on the semantic correlation between brain tumor images of different modalities. The student network is trained based on the trained teacher network, and an image segmentation model is obtained based on the trained student network.

[0133] The acquisition module 520 is used to acquire brain tumor images of missing modalities to be segmented;

[0134] The segmentation module 530 is used to input the brain tumor image of the missing modality to be segmented into the image segmentation model to obtain the image segmentation result output by the image segmentation model.

[0135] As one embodiment, the feature reconstruction module includes:

[0136] The feature labeling unit is used to map the features of at least one modality of the input brain tumor image to a preset labeling space to obtain a first modality label, and based on the first modality label, obtain a second modality label corresponding to the brain tumor image with missing modality;

[0137] The feature reconstruction unit is used to add random spatial location codes to the merged first modality labels and second modality labels and input them into the feature reconstruction network to obtain the reconstructed brain tumor image features of each modality.

[0138] As one embodiment, mapping the input brain tumor image features of at least one modality to a preset labeling space to obtain a first modality label includes:

[0139] The features of at least one modality of the input brain tumor image are flattened into one-dimensional feature vectors;

[0140] The one-dimensional feature vector is mapped to the label space based on a preset projection function to obtain the first modality label;

[0141] Correspondingly, obtaining the second modality marker corresponding to the brain tumor image with missing modality based on the first modality marker includes:

[0142] Based on the average feature representation of the first modality label, the second modality label corresponding to the brain tumor image with missing modality is obtained.

[0143] As one embodiment, the training module 510 is further configured to:

[0144] Construct a dataset and a loss function, wherein the loss function includes at least a knowledge distillation loss function and a feature reconstruction loss function. The knowledge distillation loss function is used to characterize the degree of difference between the predicted probability distribution of the trained teacher network and the predicted probability distribution of the student network, and the feature reconstruction loss function is used to characterize the difference between the image features extracted by the trained teacher network and the image features reconstructed by the student network.

[0145] Configure the initial parameters of the student network based on the parameters of the trained teacher network;

[0146] Based on the dataset and the loss function, the initial parameters are optimized to complete the training of the student network.

[0147] As an example, the loss function further includes a segmentation loss sub-function, which is used to characterize the degree of overlap between the predicted result and the true label, as well as the sum of the differences between the predicted probability distribution and the true label distribution.

[0148] As an example, the dataset includes a subset of full-modality brain tumor image data and a subset of missing-modality brain tumor image data. The subset of full-modality brain tumor image data is used to train a teacher network based on full-modality brain tumor images, and the subset of missing-modality brain tumor image data is used to train a student network based on random missing-modality brain tumor images.

[0149] The missing modality brain tumor image segmentation device provided by the present invention is used to perform the missing modality brain tumor image segmentation method described in any of the above embodiments, and has the technical effects corresponding to the missing modality brain tumor image segmentation method, which will not be described in detail here.

[0150] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the missing modality brain tumor image segmentation method, the method including:

[0151] Acquire images of brain tumors with missing modalities to be segmented;

[0152] The brain tumor image of the missing modality to be segmented is input into the image segmentation model to obtain the image segmentation result output by the image segmentation model;

[0153] The training steps of the image segmentation model include:

[0154] A knowledge distillation model architecture is constructed, which includes a teacher network and a student network. The student network includes at least one feature reconstruction module, which is used to reconstruct brain tumor image features of missing modalities based on the semantic correlation between brain tumor images of different modalities.

[0155] The student network is trained based on the trained teacher network, and the image segmentation model is obtained based on the trained student network.

[0156] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the missing modality brain tumor image segmentation method provided by the above methods, the method comprising:

[0158] Acquire images of brain tumors with missing modalities to be segmented;

[0159] The brain tumor image of the missing modality to be segmented is input into the image segmentation model to obtain the image segmentation result output by the image segmentation model;

[0160] The training steps of the image segmentation model include:

[0161] A knowledge distillation model architecture is constructed, which includes a teacher network and a student network. The student network includes at least one feature reconstruction module, which is used to reconstruct brain tumor image features of missing modalities based on the semantic correlation between brain tumor images of different modalities.

[0162] The student network is trained based on the trained teacher network, and the image segmentation model is obtained based on the trained student network.

[0163] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the missing modality brain tumor image segmentation method provided by the methods described above, the method comprising:

[0164] Acquire images of brain tumors with missing modalities to be segmented;

[0165] The brain tumor image of the missing modality to be segmented is input into the image segmentation model to obtain the image segmentation result output by the image segmentation model;

[0166] The training steps of the image segmentation model include:

[0167] A knowledge distillation model architecture is constructed, which includes a teacher network and a student network. The student network includes at least one feature reconstruction module, which is used to reconstruct brain tumor image features of missing modalities based on the semantic correlation between brain tumor images of different modalities.

[0168] The student network is trained based on the trained teacher network, and the image segmentation model is obtained based on the trained student network.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for segmenting brain tumor images with missing modalities, characterized in that, include: Acquire images of brain tumors with missing modalities to be segmented; The brain tumor image of the missing modality to be segmented is input into the image segmentation model to obtain the image segmentation result output by the image segmentation model; The training steps of the image segmentation model include: A knowledge distillation model architecture is constructed, which includes a teacher network and a student network. The student network includes at least one feature reconstruction module, which is used to reconstruct brain tumor image features of missing modalities based on the semantic correlation between brain tumor images of different modalities. A dataset and a loss function are constructed. The loss function includes at least a knowledge distillation loss function and a feature reconstruction loss function. The knowledge distillation loss function characterizes the difference between the predicted probability distribution of the trained teacher network and the predicted probability distribution of the student network. The feature reconstruction loss function characterizes the difference between the image features extracted by the trained teacher network and the image features reconstructed by the student network. Initial parameters of the student network are configured based on the parameters of the trained teacher network. Based on the dataset and the loss function, the initial parameters are optimized to complete the training of the student network. The image segmentation model is obtained based on the trained student network. The feature reconstruction module includes: The feature labeling unit is used to flatten the features of at least one modality of the input brain tumor image into a one-dimensional feature vector; map the one-dimensional feature vector to the labeling space based on a preset projection function to obtain a first modality label; and obtain a second modality label corresponding to the brain tumor image with missing modality based on the average feature representation of the first modality label. The feature reconstruction unit is used to add random spatial location codes to the merged first modality labels and second modality labels and input them into the feature reconstruction network to obtain the reconstructed brain tumor image features of each modality.

2. The method for segmenting brain tumor images with missing modalities according to claim 1, characterized in that, The loss function further includes a segmentation loss sub-function, which is used to characterize the degree of overlap between the prediction result and the true label, as well as the sum of the differences between the prediction probability distribution and the true label distribution.

3. The method for segmenting brain tumor images with missing modalities according to claim 1, characterized in that, The dataset includes a subset of full-modality brain tumor image data and a subset of missing-modality brain tumor image data. The subset of full-modality brain tumor image data is used to train a teacher network based on full-modality brain tumor images, and the subset of missing-modality brain tumor image data is used to train a student network based on random missing-modality brain tumor images.

4. A brain tumor image segmentation device with missing modalities, characterized in that, A method for segmenting brain tumor images with missing modalities as described in any one of claims 1-3 includes: A training module is used to construct a knowledge distillation model architecture, which includes a teacher network and a student network. The student network includes at least one feature reconstruction module, which is used to reconstruct brain tumor image features of missing modalities based on the semantic correlation between brain tumor images of different modalities. The student network is trained based on the trained teacher network, and an image segmentation model is obtained based on the trained student network. The acquisition module is used to acquire brain tumor images of missing modalities to be segmented; The segmentation module is used to input the brain tumor image of the missing modality to be segmented into the image segmentation model to obtain the image segmentation result output by the image segmentation model.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the missing modality brain tumor image segmentation method as described in any one of claims 1-3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the missing modality brain tumor image segmentation method as described in any one of claims 1-3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the missing modality brain tumor image segmentation method as described in any one of claims 1-3.

Images