Mode-deficient brain tumor image segmentation method, device, equipment, medium and product

Abstract

The invention provides a missing-modal brain tumor image segmentation method, device, equipment, medium and product, and belongs to the technical field of image segmentation, a knowledge distillation model architecture is constructed, the knowledge distillation model architecture comprises a teacher network and a student network, the student network comprises at least one feature reconstruction module, and the teacher network comprises at least one feature reconstruction module; the feature reconstruction module is used for reconstructing brain tumor image features of a missing mode based on semantic relevance between brain tumor images of different modes; the student network is trained based on the trained teacher network, an image segmentation model is obtained according to the trained student network, and the image segmentation model is used for performing image segmentation on the to-be-segmented missing modal brain tumor image. The student network can reconstruct the brain tumor image features of the missing mode based on the semantic relevance between the brain tumor images of different modes, can better represent the brain tumor image features of the missing mode, and effectively improves the segmentation accuracy of the brain tumor images of the missing mode.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation technology, and in particular to a method, device, equipment, medium and product for missing modality brain tumor image segmentation. Background Art

[0002] Multimodal brain tumor Magnetic Resonance Imaging (MRI) images include data from four MRI modalities: T1-weighted imaging, T2-weighted imaging, T1 contrast-enhanced weighted imaging, and Fluid Attenuated Inversion Recovery (Flair). Missing modality brain tumor images are multimodal MRI images that are missing any one or more of these modalities. In practical applications, missing modality brain tumor image data often occurs due to various reasons, such as equipment limitations, poor imaging conditions, and data acquisition errors. This makes it impossible to obtain complete multi-view brain tumor information, reducing the accuracy of brain tumor image segmentation.

[0003] In recent years, knowledge distillation strategies, as an emerging technical means, have been widely used to solve the problem of incomplete modal data processing. The core of knowledge distillation is to transfer the knowledge mastered by the complex teacher model to the student model, so that the student model can achieve a performance level similar to that of the teacher model while maintaining a low computational complexity. In the field of multimodal data processing, cross-modal knowledge distillation aims to transfer the knowledge of the teacher model trained on full modal data to the student model trained on incomplete modal data. However, the teacher model focuses on full modal data and it is difficult to effectively capture and learn the key features of the missing modality, resulting in the student model being unable to use the existing information for accurate segmentation. The accuracy of missing modality brain tumor image segmentation still needs to be improved. Summary of the Invention

[0004] The present invention provides a missing modality brain tumor image segmentation method, device, equipment, medium and product to solve the problem that the teacher model focuses on full modality data and has difficulty in effectively capturing and learning the key features of the missing modality, resulting in the student model being unable to use the existing information for accurate segmentation, and the accuracy of missing modality brain tumor image segmentation still needs to be improved.

[0005] The present invention provides a missing modality brain tumor image segmentation method, comprising: Acquire a brain tumor image with a missing modality to be segmented; Inputting the missing modality brain tumor image to be segmented into an image segmentation model to obtain an image segmentation result output by the image segmentation model; The image segmentation model training step includes: Constructing a knowledge distillation model architecture, the knowledge distillation model architecture comprising a teacher network and a student network, the student network comprising at least one feature reconstruction module, the feature reconstruction module being configured to reconstruct features of brain tumor images in a missing modality based on semantic correlations between brain tumor images in different modalities; The student network is trained based on the trained teacher network, and the image segmentation model is obtained according to the trained student network.

[0006] As an embodiment, the feature reconstruction module includes: a feature labeling unit, configured to map input features of at least one modality of brain tumor images to a preset labeling space to obtain a first modality label, and based on the first modality label, obtain a second modality label corresponding to the brain tumor image of the missing modality; The feature reconstruction unit is used to add random spatial position coding to the merged first modality label and the second modality label and input the coding into the feature reconstruction network to obtain reconstructed brain tumor image features of each modality.

[0007] As an embodiment, mapping the input brain tumor image features of at least one modality to a preset labeling space to obtain a first modality label includes: Flattening the input brain tumor image features of at least one modality into a one-dimensional feature vector; Mapping the one-dimensional feature vector to the label space based on a preset projection function to obtain the first modal label; Correspondingly, obtaining the second modality label corresponding to the brain tumor image of the missing modality based on the first modality label includes: Based on the average feature representation of the first modality marker, a second modality marker corresponding to the brain tumor image of the missing modality is obtained.

[0008] As an embodiment, the training of the student network based on the trained teacher network includes: Constructing a data set and a loss function, wherein the loss function includes at least a knowledge distillation loss sub-function and a feature reconstruction loss sub-function, wherein the knowledge distillation loss sub-function is used to characterize the degree of difference between the predicted probability distribution of the trained teacher network and the predicted probability distribution of the student network, and the feature reconstruction loss sub-function is used to characterize the difference between the image features extracted by the trained teacher network and the image features reconstructed by the student network; Configure the initial parameters of the student network based on the parameters of the trained teacher network; Based on the data set and the loss function, the initial parameters are optimized to complete the training of the student network.

[0009] As an embodiment, the loss function further includes a segmentation loss sub-function, which is used to characterize the degree of overlap between the prediction result and the true label and the sum of the differences between the predicted probability distribution and the true label distribution.

[0010] As an embodiment, the dataset includes a full-modality brain tumor image data subset and a missing-modality brain tumor image data subset, wherein the full-modality brain tumor image data subset is used to train a teacher network based on full-modality brain tumor images, and the missing-modality brain tumor image data subset is used to train a student network based on random missing-modality brain tumor images.

[0011] The present invention also provides a missing modality brain tumor image segmentation device, comprising: A training module is configured to construct a knowledge distillation model architecture, the knowledge distillation model architecture comprising a teacher network and a student network, the student network comprising at least one feature reconstruction module configured to reconstruct features of brain tumor images in missing modalities based on semantic associations between brain tumor images in different modalities; the student network is trained based on the trained teacher network, and an image segmentation model is obtained based on the trained student network; An acquisition module, used for acquiring a brain tumor image with a missing modality to be segmented; The segmentation module is used to input the brain tumor image with missing modality to be segmented into the image segmentation model to obtain the image segmentation result output by the image segmentation model.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for segmenting brain tumor images with missing modalities as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for segmenting brain tumor images with missing modalities.

[0014] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned missing modality brain tumor image segmentation methods.

[0015] The missing modality brain tumor image segmentation method, device, equipment, medium and product provided by the present invention can reconstruct the missing modality brain tumor image features based on the semantic correlation between brain tumor images of different modalities by setting at least one feature reconstruction module in the student network, and can better characterize the missing modality brain tumor image features for accurate image segmentation, thereby effectively improving the accuracy of missing modality brain tumor image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 1 is a flow chart of the missing modality brain tumor image segmentation method provided by the present invention.

[0018] Figure 2 This is a schematic diagram of the training process of the knowledge distillation model architecture provided by the present invention.

[0019] Figure 3 It is a structural diagram of the student network provided by the present invention.

[0020] Figure 4 It is a structural diagram of the feature reconstruction module provided by the present invention.

[0021] Figure 5 It is a schematic structural diagram of the missing modality brain tumor image segmentation device provided by the present invention.

[0022] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0024] Currently, there are two main types of methods in the research field of missing modality brain tumor image segmentation: targeted methods and comprehensive methods.

[0025] The key to the targeted approach is to train specialized models for different missing modalities. This approach mainly includes two strategies: collaborative training and data generation. The collaborative training strategy accurately transfers the key information of the full-modal network to the missing-modal network, allowing the model to fully draw on the rich features learned by the full-modal network when processing missing modal data, thereby achieving a more detailed and accurate understanding of the missing modal data and effectively improving the segmentation accuracy. The data generation strategy based on the generative adversarial network supplements the missing modal information at the data level, providing a more complete data foundation for subsequent segmentation tasks, enhancing the model's adaptability to complex data distributions, and further improving the accuracy and stability of segmentation.

[0026] Targeted approaches have limitations. For one thing, collaborative training methods require separate network training for each modality combination, significantly increasing the cost of model deployment. Furthermore, data generation strategies based on generative adversarial networks (GANs) are difficult to train for 3D image generation, resulting in additional resource consumption during training and practical applications. Furthermore, generative models struggle to fully reproduce images with realistic contrast. For example, gadolinium contrast agents are indispensable in practice, but generative models struggle to replicate their effects.

[0027] Comprehensive approaches attempt to address all missing modalities using a single, universal model. These approaches typically embed data from different modalities into a shared latent space using modality-specific encoders, followed by feature fusion and subsequent processing to achieve segmentation. Multimodal variational autoencoders are used to reconstruct modalities from common latent variables, leading to a truly shared latent feature representation. The use of multiple encoders (and sometimes multiple decoders) with complex interactions not only increases the difficulty and computational cost of model training but can also lead to poor model interpretability.

[0028] The present invention provides a method, device, equipment, medium and product for missing modality brain tumor image segmentation, which improves the network structure of the student model so that it can effectively capture and learn the key features of the missing modality, achieve accurate image segmentation, and improve the overall accuracy of missing modality brain tumor image segmentation.

[0029] Figure 1 FIG. 1 is a flow chart of the missing modality brain tumor image segmentation method provided by the present invention, as shown in FIG. Figure 1 As shown, the present invention provides a missing modality brain tumor image segmentation method, including steps S100-S200, and steps S100-S200 are the application stage of the present invention.

[0030] S100: Acquire a brain tumor image with missing modalities to be segmented. The brain tumor image with missing modalities to be segmented refers to a multimodal MRI image acquired by an MRI device that is missing any one or more modalities among four MRI modality data.

[0031] S200: Inputting the missing modality brain tumor image to be segmented into an image segmentation model to obtain an image segmentation result output by the image segmentation model. The image segmentation model is used to segment the brain tumor region in the image. Correspondingly, the image segmentation result includes one or more of a binary segmentation result, a multi-class segmentation result, and a boundary or contour. The binary segmentation result is used to characterize tumor and non-tumor regions, the multi-class segmentation result is used to mark multiple different structures or tissues, and the boundary or contour is used to describe the boundary position and shape of the structure.

[0032] The training steps of the image segmentation model include steps S010 to S020, and steps S010 to S020 are performed before step S100.

[0033] S010: Construct a knowledge distillation model architecture. The knowledge distillation model architecture includes a teacher network and a student network. The student network includes at least one feature reconstruction module, which is used to reconstruct features of brain tumor images in missing modalities based on semantic correlations between brain tumor images of different modalities. The teacher network and the student network are constructed using the same model structure to facilitate knowledge transfer.

[0034] S020: Training the student network based on the trained teacher network, and obtaining the image segmentation model according to the trained student network.

[0035] In one embodiment, the image segmentation model includes a trained student network to perform image segmentation on brain tumor images in the missing modality. In a preferred embodiment, the image segmentation model includes a trained teacher network and a trained student network. The trained teacher network can be used to perform image segmentation on brain tumor images in all modalities, and the trained student network can be used to perform image segmentation on brain tumor images in the missing modality.

[0036] It can be understood that the present invention, by setting at least one feature reconstruction module in the student network, can reconstruct the features of the brain tumor image of the missing modality based on the semantic correlation between brain tumor images of different modalities, and can better characterize the features of the brain tumor image of the missing modality for accurate image segmentation, thereby effectively improving the accuracy of the segmentation of the brain tumor image of the missing modality.

[0037] Based on the above embodiment, as an optional embodiment, the training of the student network based on the trained teacher network includes steps S021 to S023.

[0038] Step S021, constructing a data set and a loss function, wherein the loss function includes at least a knowledge distillation loss sub-function and a feature reconstruction loss sub-function, wherein the knowledge distillation loss sub-function is used to characterize the degree of difference between the predicted probability distribution of the trained teacher network and the predicted probability distribution of the student network, and the feature reconstruction loss sub-function is used to characterize the difference between the image features extracted by the trained teacher network and the image features reconstructed by the student network.

[0039] Step S022: configuring the initial parameters of the student network based on the parameters of the trained teacher network.

[0040] Step S023: Based on the data set and the loss function, optimize the initial parameters to complete the training of the student network.

[0041] As an optional embodiment, the dataset includes a full-modality brain tumor image data subset and a missing-modality brain tumor image data subset, wherein the full-modality brain tumor image data subset is used to train a teacher network based on full-modality brain tumor images, and the missing-modality brain tumor image data subset is used to train a student network based on random missing-modality brain tumor images.

[0042] Suppose there is a set of brain tumor image data , where each image Represented as a set of three-dimensional matrices ,in represents the number of modes of each matrix, x, y, z Corresponding to the number of voxels in the three spatial dimensions of the image. Contains four modes, which can be expanded in the modal dimension and recorded as and ,Right now .gather It can be considered as a subset of full-modality brain tumor image data.

[0043] In actual scenarios, modality loss may occur due to various reasons, that is, random loss of one or more modal data. From a combination perspective, there are 15 possibilities for modality loss. For example, if the image Missing the jth mode (j value is 1 represents , 2 represents , 3 represents , 4 represents ), which can be expressed as . Will gather The 15 missing conditions of any image in are combined and recorded as the missing modality brain tumor image data subset.

[0044] The core task of multimodal brain tumor segmentation in the absence of modality is to find a segmentation function S , so that for any missing Both . is with Segmentation mask matrices of the same size, Represents a tag, , where 0 represents the background and 1 represents the tumor area.

[0045] The present invention trains a teacher network by using a subset of full-modality brain tumor image data instead of training a teacher network of a single modality, which can reduce the complexity of the knowledge distillation model architecture. The full-modality brain tumor image data subset corresponds to the missing modality brain tumor image data subset, which makes it easier for the teacher network to guide the student network's reconstruction features during the training stage or application stage, thereby improving the accuracy of the reconstructed features.

[0046] The image segmentation model based on knowledge distillation and feature reconstruction provided by the present invention mainly includes two stages in the training process: teacher network pre-training and student network training. Figure 2 As shown, Figure 2 The unidirectional arrows in the figure represent the data flow direction, and the double arrows represent the loss sub-function.

[0047] Both the teacher network and the student network are determined based on the image segmentation network. The embodiment of the present invention is described using the image segmentation network Vnet as an example. Vnet is a deep learning convolutional neural network structure for medical image segmentation, specifically designed for the automatic segmentation of three-dimensional medical images. The teacher network is directly constructed using the Vnet network, and the student network improves the Vnet network by adding at least one feature reconstruction module. Figure 3 In the preferred embodiment shown, the student network uses Vnet as the backbone network, and introduces a feature reconstruction module in the last three stages of the Vnet network. The feature reconstruction module uses the semantic correlation between brain tumor images of different modalities to realize feature reconstruction of brain tumor images of missing modalities.

[0048] The teacher network pre-training uses full-modality brain tumor image samples from the full-modality brain tumor image data subset for training. The knowledge learned by the teacher network refers to the knowledge of segmenting full-modality brain tumor images, which can provide a reference for the student network training.

[0049] After the teacher network pre-training is completed, the student network training phase begins. During the student network training phase, the teacher network parameters are kept frozen, and the knowledge of the teacher network is transferred to the student network using knowledge distillation technology. That is, the parameters of the trained teacher network are transferred to the student network to obtain the initial parameters of the student network.

[0050] Considering that the segmentation problem can essentially be reduced to a pixel-level classification problem, an embodiment of the present invention adopts a common sense distillation method to guide the student network to learn the knowledge of the teacher network by minimizing the Kullback-Leibler (KL) divergence between the predicted probability distribution of the student network and the predicted probability distribution of the teacher network.

[0051] Assume that the teacher network has a I The predicted probability distribution of , the predicted probability distribution of the student network is , then the general calculation formula of KL divergence is as follows:

[0052] in, is the KL divergence.

[0053] Assume that the full-modal teacher network is trained on the input image data I The predicted probability distribution of , the predicted probability distribution of the missing mode student model is , then the calculation formula of the pixel-by-pixel knowledge distillation loss sub-function is as follows:

[0054] in, Represents the softmax operation, which converts the original prediction value into a probability distribution; T Represents the temperature hyperparameter, adjust T The value of can control the softening degree of the softmax operation output and optimize the knowledge distillation effect; KL represents the KL divergence operation, which is used to measure the degree of difference between two predicted probability distributions.

[0055] The teacher network is used not only to guide the student network to predict probability distribution knowledge, but also to guide the student network to reconstruct accurate brain tumor image features of missing modalities.

[0056] set up represents the full-modality brain tumor image features extracted by the teacher network during the training phase of the student network, represents the brain tumor image features of the missing modality reconstructed during the student network training phase. The calculation formula of the feature reconstruction loss subfunction is as follows:

[0057] in, represents the mean squared error (MSE) function, which is used to calculate the mean squared error between the image features extracted by the teacher network and the image features reconstructed by the student network. The feature reconstruction loss subfunction can be used to guide the student network to optimize the reconstruction process and improve the reconstruction quality of brain tumor image features in the missing modality.

[0058] Based on the data set and the loss function, optimizing the initial parameters to complete the training of the student network specifically refers to randomly extracting full-modality image samples and corresponding missing-modality image samples from the data set, inputting the full-modality image samples into the trained teacher network and inputting the missing-modality image samples into the student network, and calculating the knowledge distillation loss sub-function value and the feature reconstruction loss sub-function value based on the above formulas (1)-(3), judging whether the knowledge distillation loss sub-function value and the feature reconstruction loss sub-function value are within the preset loss range, and if so, determining that the student network training is completed, otherwise, adjusting the parameters of the Vnet network of the student network based on the knowledge distillation loss sub-function value, adjusting the parameter values ​​of the feature reconstruction module based on the feature reconstruction loss sub-function value, and returning to execute the step of randomly extracting full-modality image samples and corresponding missing-modality image samples from the data set.

[0059] Specifically, during the student network training process, in order to fully simulate the possible mode loss in real scenarios, 15 mode combinations in the data set are sampled to obtain the input missing mode combination. The existence of each mode is determined by Indicates that ∈{0,1}, =0 means the mode is missing, =1 means the mode exists. For each missing mode , replace it with a 0 matrix in the input part.

[0060] In this embodiment of the present invention, a feature reconstruction module (FRM) is added to the final three stages of the Vnet network to create a student network. The FRM module infers and reconstructs features of the missing modalities based on existing modality features and learnable modality-specific markers. The FRM module, equipped with learnable modality-specific markers, continuously optimizes its parameters using extensive training data to accurately capture the underlying connections between modalities and the general semantic features of each modality, thereby reconstructing high-level semantic information from the missing modalities.

[0061] The loss function of the present invention includes at least a knowledge distillation loss sub-function and a feature reconstruction loss sub-function, which can improve the accuracy of the predicted probability distribution of the student network and enhance the reconstruction quality of brain tumor image features with missing modalities.

[0062] As an optional embodiment, the loss function further includes a segmentation loss sub-function, which is used to characterize the degree of overlap between the prediction result and the true label and the sum of the differences between the predicted probability distribution and the true label distribution.

[0063] The segmentation loss sub-functions include dice loss and cross entropy loss. Dice loss can effectively measure the degree of overlap between the prediction results of the student network and the true label of the sample, and has a good effect on image segmentation tasks with clear distinction between foreground and background; cross entropy loss focuses on evaluating the difference between the predicted probability distribution of the student network and the true label distribution. The combination of the two can optimize the segmentation performance of the model from different angles.

[0064] The calculation formula of the segmentation loss sub-function is as follows:

[0065] in, represents the segmentation loss sub-function, represents the cross entropy loss, represents dice loss, represents the true label.

[0066] Optionally, the learning process of the student network is guided by the teacher network and the segmentation mask. Correspondingly, the loss function is calculated as follows:

[0067] in, and is a hyperparameter used to balance the relationship between different loss sub-functions. Used to adjust the influence of the feature reconstruction loss sub-function on the total loss function. Used to adjust the influence of the knowledge distillation loss sub-function on the total loss function.

[0068] It can be understood that the present invention constructs a total loss function through the segmentation loss sub-function, the knowledge distillation loss sub-function and the feature reconstruction loss sub-function, and can also adjust the respective proportions of the knowledge distillation loss sub-function and the feature reconstruction loss sub-function through hyperparameters, which can effectively improve the performance of the student network.

[0069] As an optional embodiment, the feature reconstruction module includes a feature marking unit and a feature reconstruction unit.

[0070] The feature labeling unit is used to map the input brain tumor image features of at least one modality to a preset labeling space to obtain a first modality label, and based on the first modality label, obtain a second modality label corresponding to the brain tumor image of the missing modality.

[0071] A feature reconstruction unit is configured to add random spatial position codes to the combined first and second modality markers and input the codes into a feature reconstruction network to obtain reconstructed brain tumor image features for each modality. Optionally, the feature reconstruction network is composed of four stacked deep learning transformer layers, each transformer layer corresponding to a feature channel.

[0072] Optionally, mapping the input brain tumor image features of at least one modality to a preset label space to obtain a first modality label includes: Flattening the input brain tumor image features of at least one modality into a one-dimensional feature vector; Mapping the one-dimensional feature vector to the label space based on a preset projection function to obtain the first modal label; Correspondingly, obtaining the second modality label corresponding to the brain tumor image of the missing modality based on the first modality label includes: Based on the average feature representation of the first modality marker, a second modality marker corresponding to the brain tumor image of the missing modality is obtained.

[0073] The feature reconstruction module can be used for feature reconstruction in the training phase and feature reconstruction in the application phase. The feature reconstruction processes in the two phases are the same. The embodiment of the present invention takes the feature reconstruction in the training phase as an example for explanation.

[0074] like Figure 4 As shown, for the Vnet network Characteristics of each stage , it is divided into four modal features in the channel dimension, namely ,in . Vnet network The feature reconstruction module of each stage first performs modality feature tokenization processing on the input brain tumor image features of at least one modality.

[0075] For the actual modal characteristics, that is, The feature reconstruction module maps the modal features of the first modality into the first modality mark using the following formula: :

[0076] Among them, the flatten operation flattens the three-dimensional feature map into a one-dimensional vector; Is the projection function, which is used to map the flattened features to a dimension of The mark space, Indicates the length of the mark space, is the hidden dimension of the label space.

[0077] for The feature reconstruction module uses learnable modality tags (i.e., the second modality tag) to represent. The learnable modality tag, on the one hand, conveys the specific modality feature information that needs to be reconstructed to the feature reconstruction network, and on the other hand, interacts with the existing modality features to learn modality sharing information, thereby better reconstructing the missing modality features. Specifically, Has initialization parameters that are used to simulate the average features of the missing modalities. The feature reconstruction module will make comprehensive use of and existing modal features, and use the attention mechanism to infer the various modal features; in the back propagation phase of the model, these parameters will be updated as the learning process progresses. In other words, It includes coarse feature simulation of missing modalities, and this part of the features can be compared to the model's memory of a certain type of modal features, similar to a doctor's prior knowledge of a specific modality.

[0078] After the tokenization process is completed, the feature reconstruction module merges all modal tags in the channel dimension to obtain the input sequence of the feature reconstruction network. , the specific formula is as follows:

[0079] In order to more effectively characterize the spatial position relationship of features, the embodiment of the present invention is Add randomly initialized spatial position encoding , input to the feature reconstruction network , to obtain the reconstructed modal features The feature reconstruction network uses the self-attention mechanism to learn the potential shared features of each modality and complete the missing modality features.

[0080] The reconstructed modal features are connected with the original modal features using residual connections to achieve efficient feature learning. The calculation formula is as follows:

[0081] Finally, The shape of is adjusted to the original size, completing the reconstruction of each missing mode and the learning process of the modal shared features.

[0082] It can be understood that the training samples randomly lack modalities. Correspondingly, the present invention constructs an input sequence through modal labels, and can obtain the average feature representation of the missing modality during the training process; the feature reconstruction module adopts an attention mechanism, which can effectively utilize the existing information and missing information from the missing label sequence and the existing modal features through the attention mechanism to better characterize the characteristics of the missing modality.

[0083] This paper constructs an innovative teacher-student network architecture, transferring all-modal knowledge to the student network through a knowledge transfer mechanism. It also designs a student network suitable for missing modalities and introduces a feature reconstruction module (FRM) into the student network, allowing the teacher network to guide the student network in learning the features of the missing modality. This paper outperforms existing methods in average segmentation performance across all modal combinations, and performs particularly well in the extreme case where only a single modality is present.

[0084] Next, the experimental hardware and software environment, the data set used for the experiment, the experimental settings, the experimental evaluation indicators and the experimental results of an embodiment of the present invention are introduced in detail.

[0085] This paper conducts experiments on the BraTS2018 dataset. The BraTS2018 dataset contains 285 cases, all of which are multimodal MRI data covering four modalities: T1, T2, T1ce, and Flair. Each multimodal image is manually annotated by experienced radiologists. The segmentation task requires distinguishing the whole tumor (WT), enhancing tumor (ET), and tumor core (TC) from the background.

[0086] During the data preprocessing phase, the data for each case was standardized to ensure that it met the requirements for model training. First, the data was normalized to zero mean and unit variance to eliminate intensity differences between samples caused by factors such as scanning equipment and parameters, allowing the model to better learn data features. At the same time, considering GPU memory limitations, each voxel was randomly cropped to a size of 128×128×128 during model training and then input into the network. To comprehensively evaluate model performance, the dataset was randomly divided into training, validation, and test sets. The training set consisted of 200 cases, the validation set consisted of 28 cases, and the test set consisted of 57 cases.

[0087] This paper is based on the PyTorch1.12.1 deep learning framework and is run on a computing platform equipped with 8 NVIDIA GeForce RTX-3090-24GB GPUs. In terms of data preprocessing, all input model data are enhanced, including random cropping, random inversion, and random scaling. In terms of model optimization, the AdamW algorithm is used as the optimizer, and the weight decay coefficient is set to , to prevent the model from overfitting. The initial learning rate is set to The batch size of all training stages is set to 4, the teacher network pre-training stage is trained for 1000 epochs, and the student network is trained for 5000 epochs. Hyperparameters in the knowledge distillation process According to experience, it is set to 10, the scaling factor in the total loss Set to 5, the scaling factor The DICE metric is set to 10. To increase attention to missing combinations that produce poor segmentation results, the present invention also adopts a dynamic sampling rule. After every 100 training rounds, the DICE situation of each missing condition in the validation set is evaluated. In subsequent training rounds, different missing conditions are sampled according to a weight coefficient of 1-DICE. The present invention adopts the DICE metric, which is the most widely used in medical image segmentation, as the evaluation metric for evaluating segmentation results.

[0088] The proposed method is comprehensive and applicable to all missing cases. To evaluate its effectiveness, it was compared with four methods: a unified heterogeneous variational encoder-decoder (U-HVED), an adversarial co-training network (ACN), a saliency-guided morphology-aware U-shaped neural network (SMU-Net), and a region-aware fusion network (RFNet) for all missing cases. ACN and SMU-Net are targeted methods, while the other two are comprehensive. In addition, in order to evaluate the performance of the present invention in extreme missing data (data contains only one modality), it is compared with several image segmentation methods specifically for this situation, such as the baseline method (single-modal Vnet trained in fully supervised mode), Knowledge Distillation Network (KD-Net), Privileged Multimodal Knowledge Learning (PMKL) and Prototype Knowledge Distillation Loss (ProtoKL). KD-Net, PMKL and ProtoKL are also based on the principle of knowledge distillation.

[0089] Table 1 Comparative experiment of WT region complete deletion combination

[0090] Table 2 Comparative experiment of TC region complete deletion combination

[0091] Table 3 Comparative experiment of all missing ET regions

[0092] In the missing modality brain tumor image segmentation experiment, the present invention conducted a complete missing combination comparison for the three regions of WT, TC and ET. In the WT and TC regions, the average scores of the present invention were 83.5 and 74.6, respectively, second only to RFNet; in the ET region, the average score of the present invention was 62.3, significantly exceeding other comparison methods, and its advantage in this region was obvious. The advantages of the present invention lie in the innovative knowledge distillation architecture from all modalities to missing modalities and the student network that introduces the feature reconstruction module, which can realize knowledge transfer, effectively utilize existing data, reduce information loss, and enhance adaptability to incomplete data. In the extreme case of three-modality missing, the present invention has significant advantages over the existing comprehensive methods. In the four cases, 7 of the 12 indicators are in the leading position, which fully proves that the present invention is particularly suitable for extreme situations.

[0093] Table 4 Single modality missing combination comparison experiment - T1 and T2 data

[0094] Table 5 Comparative experiment of single modality loss combination - T1c and Flair data

[0095] As shown in Tables 4 and 5, the proposed method is compared with a baseline method (a Vnet trained with unimodal full supervision) and three knowledge distillation-based methods specifically designed for this scenario, when only a single modality is available. The teacher network, trained on all-modal data, achieved an average score of 80.93 in the WT, TC, and ET regions, providing a performance benchmark for other methods. When only T1 data is available, the proposed method's average dice score ranks second among all methods. When only T2 data is available, the proposed method achieves the highest scores in the TC and ET regions, significantly outperforming all other methods in average dice score. When only T1ce data is available, the proposed method achieves the highest dice score in the WT region. When only Flair data is available, the proposed method achieves the highest dice score in the CO and EC regions, also leading in average dice score. Overall, the proposed method exhibits advantages in different unimodal conditions, with outstanding overall performance in the T2 and Flair modalities and excellent performance in key areas in the T1 and T1c modalities, demonstrating strong competitiveness in unimodal conditions.

[0096] The missing modality brain tumor image segmentation device provided by the present invention is described below. The missing modality brain tumor image segmentation device described below and the missing modality brain tumor image segmentation method described above can be referenced to each other.

[0097] Figure 5 FIG. 1 is a schematic diagram of the structure of the missing modality brain tumor image segmentation device provided by the present invention, as shown in FIG. Figure 5As shown, the present invention also provides a missing modality brain tumor image segmentation device, which includes the following modules.

[0098] A training module 510 is configured to construct a knowledge distillation model architecture, the knowledge distillation model architecture comprising a teacher network and a student network, the student network comprising at least one feature reconstruction module configured to reconstruct features of brain tumor images in a missing modality based on semantic associations between brain tumor images of different modalities; the student network is trained based on the trained teacher network, and an image segmentation model is obtained based on the trained student network; An acquisition module 520 is used to acquire a brain tumor image with a missing modality to be segmented; The segmentation module 530 is configured to input the missing modality brain tumor image to be segmented into the image segmentation model to obtain an image segmentation result output by the image segmentation model.

[0099] As an embodiment, the feature reconstruction module includes: a feature labeling unit, configured to map input features of at least one modality of brain tumor images to a preset labeling space to obtain a first modality label, and based on the first modality label, obtain a second modality label corresponding to the brain tumor image of the missing modality; The feature reconstruction unit is used to add random spatial position coding to the merged first modality label and the second modality label and input the coding into the feature reconstruction network to obtain reconstructed brain tumor image features of each modality.

[0100] As an embodiment, mapping the input brain tumor image features of at least one modality to a preset labeling space to obtain a first modality label includes: Flattening the input brain tumor image features of at least one modality into a one-dimensional feature vector; Mapping the one-dimensional feature vector to the label space based on a preset projection function to obtain the first modal label; Correspondingly, obtaining the second modality label corresponding to the brain tumor image of the missing modality based on the first modality label includes: Based on the average feature representation of the first modality marker, a second modality marker corresponding to the brain tumor image of the missing modality is obtained.

[0101] As an embodiment, the training module 510 is further configured to: Constructing a data set and a loss function, wherein the loss function includes at least a knowledge distillation loss sub-function and a feature reconstruction loss sub-function, wherein the knowledge distillation loss sub-function is used to characterize the degree of difference between the predicted probability distribution of the trained teacher network and the predicted probability distribution of the student network, and the feature reconstruction loss sub-function is used to characterize the difference between the image features extracted by the trained teacher network and the image features reconstructed by the student network; Configure the initial parameters of the student network based on the parameters of the trained teacher network; Based on the data set and the loss function, the initial parameters are optimized to complete the training of the student network.

[0102] As an embodiment, the loss function further includes a segmentation loss sub-function, which is used to characterize the degree of overlap between the prediction result and the true label and the sum of the differences between the predicted probability distribution and the true label distribution.

[0103] As an embodiment, the dataset includes a full-modality brain tumor image data subset and a missing-modality brain tumor image data subset, wherein the full-modality brain tumor image data subset is used to train a teacher network based on full-modality brain tumor images, and the missing-modality brain tumor image data subset is used to train a student network based on random missing-modality brain tumor images.

[0104] The missing modality brain tumor image segmentation device provided by the present invention is used to execute the missing modality brain tumor image segmentation method described in any of the above embodiments, and has technical effects corresponding to the missing modality brain tumor image segmentation method, which will not be repeated here.

[0105] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the missing modality brain tumor image segmentation method, which includes: Acquire a brain tumor image with a missing modality to be segmented; Inputting the missing modality brain tumor image to be segmented into an image segmentation model to obtain an image segmentation result output by the image segmentation model; The image segmentation model training step includes: Constructing a knowledge distillation model architecture, the knowledge distillation model architecture comprising a teacher network and a student network, the student network comprising at least one feature reconstruction module, the feature reconstruction module being configured to reconstruct features of brain tumor images in a missing modality based on semantic correlations between brain tumor images in different modalities; The student network is trained based on the trained teacher network, and the image segmentation model is obtained according to the trained student network.

[0106] Furthermore, the logic instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0107] In another aspect, the present invention further provides a computer program product, comprising a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the missing modality brain tumor image segmentation method provided by each of the above methods, the method comprising: Acquire a brain tumor image with a missing modality to be segmented; Inputting the missing modality brain tumor image to be segmented into an image segmentation model to obtain an image segmentation result output by the image segmentation model; The image segmentation model training step includes: Constructing a knowledge distillation model architecture, the knowledge distillation model architecture comprising a teacher network and a student network, the student network comprising at least one feature reconstruction module, the feature reconstruction module being configured to reconstruct features of brain tumor images in a missing modality based on semantic correlations between brain tumor images in different modalities; The student network is trained based on the trained teacher network, and the image segmentation model is obtained according to the trained student network.

[0108] In yet another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the missing modality brain tumor image segmentation method provided by each of the above methods, the method comprising: Acquire a brain tumor image with a missing modality to be segmented; Inputting the missing modality brain tumor image to be segmented into an image segmentation model to obtain an image segmentation result output by the image segmentation model; The image segmentation model training step includes: Constructing a knowledge distillation model architecture, the knowledge distillation model architecture comprising a teacher network and a student network, the student network comprising at least one feature reconstruction module, the feature reconstruction module being configured to reconstruct features of brain tumor images in a missing modality based on semantic correlations between brain tumor images in different modalities; The student network is trained based on the trained teacher network, and the image segmentation model is obtained according to the trained student network.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0110] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A missing modality brain tumor image segmentation method, characterized in that: include: Acquire a brain tumor image with a missing modality to be segmented; Inputting the missing modality brain tumor image to be segmented into an image segmentation model to obtain an image segmentation result output by the image segmentation model; The image segmentation model training step includes: Constructing a knowledge distillation model architecture, the knowledge distillation model architecture comprising a teacher network and a student network, the student network comprising at least one feature reconstruction module, the feature reconstruction module being configured to reconstruct features of brain tumor images in a missing modality based on semantic correlations between brain tumor images in different modalities; The student network is trained based on the trained teacher network, and the image segmentation model is obtained according to the trained student network.

2. The missing modality brain tumor image segmentation method according to claim 1, characterized in that: The feature reconstruction module includes: a feature labeling unit, configured to map input features of at least one modality of brain tumor images to a preset labeling space to obtain a first modality label, and based on the first modality label, obtain a second modality label corresponding to the brain tumor image of the missing modality; The feature reconstruction unit is used to add random spatial position coding to the merged first modality label and the second modality label and input the coding into the feature reconstruction network to obtain reconstructed brain tumor image features of each modality.

3. The missing modality brain tumor image segmentation method according to claim 2, characterized in that: Mapping the input brain tumor image features of at least one modality to a preset label space to obtain a first modality label includes: Flattening the input brain tumor image features of at least one modality into a one-dimensional feature vector; Mapping the one-dimensional feature vector to the label space based on a preset projection function to obtain the first modal label; Correspondingly, obtaining the second modality label corresponding to the brain tumor image of the missing modality based on the first modality label includes: Based on the average feature representation of the first modality marker, a second modality marker corresponding to the brain tumor image of the missing modality is obtained.

4. The method for segmenting brain tumor images with missing modality according to any one of claims 1 to 3, characterized in that: The training of the student network based on the trained teacher network includes: Constructing a data set and a loss function, wherein the loss function includes at least a knowledge distillation loss sub-function and a feature reconstruction loss sub-function, wherein the knowledge distillation loss sub-function is used to characterize the degree of difference between the predicted probability distribution of the trained teacher network and the predicted probability distribution of the student network, and the feature reconstruction loss sub-function is used to characterize the difference between the image features extracted by the trained teacher network and the image features reconstructed by the student network; Configure the initial parameters of the student network based on the parameters of the trained teacher network; Based on the data set and the loss function, the initial parameters are optimized to complete the training of the student network.

5. The missing modality brain tumor image segmentation method according to claim 4, characterized in that: The loss function also includes a segmentation loss sub-function, which is used to characterize the degree of overlap between the prediction result and the true label and the sum of the differences between the predicted probability distribution and the true label distribution.

6. The missing modality brain tumor image segmentation method according to claim 4, characterized in that: The dataset includes a full-modality brain tumor image data subset and a missing-modality brain tumor image data subset. The full-modality brain tumor image data subset is used to train a teacher network based on full-modality brain tumor images, and the missing-modality brain tumor image data subset is used to train a student network based on random missing-modality brain tumor images.

7. A missing modality brain tumor image segmentation device, characterized in that: include: A training module for constructing a knowledge distillation model architecture, the knowledge distillation model architecture comprising a teacher network and a student network, the student network comprising at least one feature reconstruction module for reconstructing features of brain tumor images in a missing modality based on semantic correlations between brain tumor images in different modalities; The student network is trained based on the trained teacher network, and an image segmentation model is obtained based on the trained student network; An acquisition module, used for acquiring a brain tumor image with a missing modality to be segmented; The segmentation module is used to input the brain tumor image with missing modality to be segmented into the image segmentation model to obtain the image segmentation result output by the image segmentation model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the missing modality brain tumor image segmentation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the missing modality brain tumor image segmentation method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the missing modality brain tumor image segmentation method according to any one of claims 1 to 6 is implemented.

Images