Twin self-distillation method and system for automatic segmentation of modality-missing brain tumor images

Abstract

The present invention discloses a twin self-distillation method and system for automatic segmentation of modality-missing brain tumor images, wherein the twin self-distillation method includes: using a twin U-shaped network to establish a twin model, using labeled brain tumor image data to perform supervised training and generative adversarial training on the twin model to obtain the model's intermediate layer features and output layer prediction map; modally decoupling the model's intermediate layer features according to random modality missing signals to obtain missing modality features and non-missing modality features, and distilling and aligning the missing modality features and non-missing modality features; randomly adding noise and reconstructing the missing modality prediction map, and distilling and aligning the reconstructed missing modality prediction map and the full modality prediction map; using missing modality data to perform inference and model performance verification on the trained twin model to obtain the segmentation accuracy of the twin model. The technical solution of the present invention can solve the problems of resource waste and low segmentation accuracy in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a twin self-distillation method and system for automatic segmentation of modality-missing brain tumor images. Background Art

[0002] Brain tumors are common and potentially harmful tumors. The segmentation of gliomas is crucial for the clinical diagnosis and treatment planning of primary brain tumors. To achieve automatic brain tumor segmentation, numerous CNN-based brain tumor segmentation methods have been proposed. These methods extract robust features from complementary pathological information from full sequences and combine deep and shallow semantic information to achieve accurate brain tumor segmentation. In daily clinical practice, limited availability of physicians and imaging equipment is a major challenge. Furthermore, the visibility of some tumor subregions relies on contrast agents, which not only increases the overall scan time but also places patients at higher risk. These factors often result in the loss of some modalities. In the absence of multiple modalities, the performance of currently used full-modality sequence brain tumor segmentation methods is severely degraded.

[0003] To address the severe performance degradation of brain tumor segmentation caused by the absence of key modalities, a large body of work has been proposed. Generally speaking, these works can be roughly divided into three categories: methods based on missing modality synthesis, methods based on latent shared representations, and methods based on knowledge distillation. Algorithms based on modality synthesis aim to learn a generator from available modalities that can be used in complex missing modalities. This generator can generate missing modality sequences by leveraging redundant information in the available modalities, and then use the complete sequences for subsequent tumor segmentation. However, most sequences generated by these methods still differ significantly from actual clinical data, and these discrepancies can severely impact subsequent optimization. Algorithms based on latent shared representations extract shared representations for tumor segmentation by learning correlations between multimodal features or statistical information between multimodal features. These shared representations form a shared latent feature space. When some modal information is missing, the network can extract relevant information from this shared feature space to assist in segmentation. While methods based on latent shared spaces can improve the performance of neural networks for brain tumor segmentation in modality-missing scenarios, the quality of the shared latent feature representations learned by the model is difficult to guarantee, and the feature information provided by this feature space is very limited when some key modalities are missing.

[0004] The knowledge distillation-based method aims to use the supervisory signal from the full modality sequence to guide the feature optimization of the missing modality through distillation learning. In essence, it achieves feature alignment between the full modality features and the missing modality features through knowledge distillation. Compared with the previous two methods, this type of method is well compatible with other methods and can significantly improve the performance of the model in complex missing scenarios after distillation. However, there are two problems with the existing knowledge distillation work for the modality-missing brain tumor segmentation task: First, an additional teacher model always needs to be trained to guide the student model for distillation learning, and in the inference stage, the teacher model with excellent segmentation performance for the full modality is discarded, which results in a waste of resources. Second, the segmentation accuracy of the model still needs to be improved when some key modalities are missing. Summary of the Invention

[0005] The present invention provides a twin self-distillation solution for automatic segmentation of brain tumor images with missing modalities, aiming to solve the problem that existing distillation methods always require training additional teacher models for distillation, and the segmentation performance of the model significantly degrades when key modalities are missing.

[0006] To achieve the above objectives, according to a first aspect of the present invention, a twin self-distillation method for automatic segmentation of modality-missing brain tumor images is proposed, comprising:

[0007] A twin model was established using a twin U-shaped network. The twin model was supervised and trained using labeled brain tumor imaging data. This model obtained intermediate layer features and output layer prediction maps. The output layer prediction maps included missing modality prediction maps and full modality prediction maps.

[0008] According to the random modal missing signal of the twin model, the model's intermediate layer features are modally decoupled to obtain the missing modal features and the non-missing modal features, which are then distilled and aligned.

[0009] The missing modal prediction map is randomly denoised and reconstructed in sequence, and the reconstructed missing modal prediction map and the full modal prediction map in the output layer prediction map are distilled and aligned;

[0010] The missing modal data is used to perform inference and model performance verification on the trained twin model to obtain the segmentation accuracy of the twin model.

[0011] Preferably, in the above-mentioned twin self-distillation method, the steps of using labeled brain tumor image data to perform supervised training and generative adversarial training on the twin model include:

[0012] Input the full modality imaging data and missing modality imaging data of the same sample into the twin model;

[0013] Use the twin model encoder to extract intermediate layer features of full-modality image data and missing-modality image data;

[0014] Use the decoder of the twin model to decode the intermediate layer features and obtain the output layer prediction map;

[0015] Calculate the loss function value of the twin model based on the label and output layer prediction graph, and use the loss function value to optimize the twin model;

[0016] Use the discriminator of the twin model to supervise the encoder and optimize the discriminator, encoder and decoder simultaneously.

[0017] Preferably, in the above-mentioned twin self-distillation method, the model intermediate layer features include missing modal semantic features and full modal semantic features, and the step of distilling and aligning the missing modal features and the non-missing modal features includes:

[0018] Use convolutional blocks to re-encode the missing modal semantic features to obtain the encoded missing semantic features;

[0019] According to the order of modal input, the encoded missing semantic features and full modal semantic features are divided to obtain a variety of basic modal features, among which the basic modal features include missing modal features and non-missing modal features;

[0020] Distillation aligns the missing modal features of the encoding missing semantic features and full-modal semantic features, and distillation aligns the non-missing modal features of the encoding missing semantic features and full-modal semantic features.

[0021] Preferably, in the above-mentioned twin self-distillation method, the steps of distilling the missing modal features of the aligned encoding missing semantic features and the full-modal semantic features, and distilling the non-missing modal features of the aligned encoding missing semantic features and the full-modal semantic features include:

[0022] According to the missing mode distillation loss function:

[0023]

[0024] The missing mode distillation loss is calculated, where represents the kth missing modal feature, represents the kth non-missing modal feature;

[0025] According to the non-missing mode distillation loss function:

[0026]

[0027] The non-missing mode distillation loss is calculated, where represents the kth non-missing modal feature, N is the number of all modes, and n is the number of missing modes;

[0028] The ratio of missing mode distillation loss to non-missing mode distillation loss is controlled by hyperparameters to obtain the mode decoupling distillation loss.

[0029] Preferably, in the above-mentioned twin self-distillation method, the steps of sequentially performing random noise addition and reconstruction on the missing modal prediction graph include:

[0030] Generate a random mask matrix of the same size as the label according to the preset mask ratio;

[0031] Use a random mask matrix to generate masks of tumor areas and non-tumor areas in the missing modality prediction map according to the labels;

[0032] Traverse the masks of the tumor area and the non-tumor area to obtain the pixels of the tumor area and the non-tumor area;

[0033] A random mask matrix is used to add noise to the tumor area and non-tumor area in the missing mode prediction map, respectively, to obtain a noisy missing mode prediction map;

[0034] The noisy missing mode prediction map is convolved using a convolution block to obtain the reconstructed missing mode prediction map.

[0035] Preferably, in the above-mentioned twin self-distillation method, the step of performing distillation alignment on the reconstructed missing modal prediction graph and the full modal prediction graph includes:

[0036] Using the noisy missing modality prediction map and the full modality prediction map, the distillation loss function for distillation is generated according to the label-guided mask:

[0037]

[0038] The distillation loss of the label-guided mask generation distillation is calculated, where Y f Represents the full modal prediction diagram, Y noisy Represents the missing mode prediction map after noise addition, Rec represents the convolution block, and k represents the tumor area with different degrees of lesions.

[0039] Preferably, in the above-mentioned twin self-distillation method, the steps of using missing modal data to infer the trained twin model and verify the model performance include:

[0040] When the twin model training is completed, the missing modal data is input into the twin model as the sample to be inferred, and the output layer prediction graph of the sample to be inferred is obtained;

[0041] Use the output layer prediction graph and label of the sample to be inferred, and score the function according to the model performance:

[0042]

[0043] Calculate the model performance score of the twin model, where predmap represents the output layer prediction map of the sample to be inferred, and label represents the label of the sample to be inferred;

[0044] The highest weight corresponding to the model performance score is selected as the inference weight of the twin model.

[0045] According to a second aspect of the present invention, the present invention provides a twin self-distillation system for automatic segmentation of modality-missing brain tumor images, comprising:

[0046] Model building module, used to build a twin model using the twin U-shaped network;

[0047] A model training module is used to supervise the training of the twin model using labeled brain tumor imaging data and generate adversarial training to obtain the model's intermediate layer features and output layer prediction maps, where the output layer prediction maps include missing modality prediction maps and full modality prediction maps;

[0048] The modal decoupling module is used to perform modal decoupling on the model's intermediate layer features based on the random modal missing signal of the twin model to obtain the missing modal features and the non-missing modal features;

[0049] A first distillation alignment module is used to perform distillation alignment on the missing modal features and the non-missing modal features;

[0050] The noise reconstruction module is used to randomly add noise and reconstruct the missing modal prediction map in the output layer prediction map;

[0051] The second distillation alignment module is used to perform distillation alignment on the reconstructed missing modality prediction map and the full modality prediction map;

[0052] Model inference module, used to infer the trained twin model using missing modality data;

[0053] The performance verification module is used to verify the model performance of the trained twin model and obtain the segmentation accuracy of the twin model.

[0054] Preferably, in the above twin self-distillation system, the model training module includes:

[0055] The data input submodule is used to input the full-modality image data and missing-modality image data of the same sample into the twin model;

[0056] The feature extraction submodule is used to extract the intermediate layer features of the full-modality image data and the missing-modality image data using the encoder of the twin model;

[0057] The feature decoding submodule is used to decode the intermediate layer features using the decoder of the twin model to obtain the output layer prediction map;

[0058] The loss calculation submodule is used to calculate the loss function value of the twin model based on the label and output layer prediction graph, and use the loss function value to optimize the twin model;

[0059] The adversarial training submodule is used to supervise the encoder using the discriminator of the twin model and optimize the discriminator, encoder, and decoder simultaneously.

[0060] Preferably, in the above twin self-distillation system, the noise addition and reconstruction module includes:

[0061] The matrix generation submodule is used to generate a random mask matrix of the same size as the label according to the preset mask ratio;

[0062] A mask generation submodule is used to generate masks of tumor regions and non-tumor regions in the missing modality prediction image according to the labels using a random mask matrix;

[0063] The mask traversal submodule is used to traverse the masks of the tumor area and the non-tumor area to obtain the pixels of the tumor area and the non-tumor area;

[0064] A prediction image noise submodule is used to add noise to the tumor region and non-tumor region in the missing mode prediction image using a random mask matrix to obtain a noisy missing mode prediction image;

[0065] The prediction graph convolution submodule is used to reconstruct the noisy missing mode prediction graph using a convolution block to obtain a reconstructed missing mode prediction graph.

[0066] In summary, the twin self-distillation scheme for automatic segmentation of modality-missing brain tumor images provided by the present invention establishes a twin model by using a twin U-shaped network. The twin model serves as a basic segmentation network (i.e., a generator) and can simultaneously train brain tumor image data with random modality missing and brain tumor image data with full modality. In the training stage, the random modality missing signal is used as the student signal, and the full modality signal is used as the teacher signal, so that two types of data can be trained on one model; the twin model is trained to obtain the model intermediate layer features and the output layer prediction map, so that the teacher signal can be used in the intermediate layer and output layer of the twin model to guide the distillation alignment of the modality-missing student signal. In addition, the twin model can also perform generative adversarial training, so that the trained model does not require an additional teacher model in the training stage, and can handle complex modality missing problems in the inference stage. In addition, the technical solution of the present application improves the existing mask generation distillation method. Through the label-guided mask generation distillation method, the missing modal prediction map in the output layer prediction map is randomly noised and reconstructed through label guidance, and then the reconstructed student signal is distilled and aligned with the teacher signal. In this way, the distillation method of the model can have better regularization constraints and generalization capabilities, and can enable the model to learn robust feature expressions under complex modal missing conditions, thereby improving the segmentation performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0068] Figure 1 1 is a flow chart of a twin self-distillation method for automatic segmentation of modality-missing brain tumor images provided by an embodiment of the present invention;

[0069] Figure 2 yes Figure 1 A schematic flow chart of a twin model training method provided in the illustrated embodiment;

[0070] Figure 3 yes Figure 1 A schematic flow chart of a modal decoupling and distillation method for model intermediate layer features provided by the illustrated embodiment;

[0071] Figure 4 yes Figure 3 A schematic flow chart of a distillation and alignment method for modal features provided by the illustrated embodiment;

[0072] Figure 5 yes Figure 1 A schematic flow chart of a mask generation distillation method using an output layer prediction graph guided by labels provided by the illustrated embodiment;

[0073] Figure 6 yes Figure 1 A schematic flow chart of a model reasoning and verification method provided by the illustrated embodiment;

[0074] Figure 7 Schematic diagram of the structure of a twin model provided by an embodiment of the present invention;

[0075] Figure 8 1 is a flow chart of a modal decoupling distillation method provided by an embodiment of the present invention;

[0076] Figure 9 1 is a flow chart of a label-guided mask generation and distillation method provided by an embodiment of the present invention;

[0077] Figure 10 1 is a flow chart of a twin self-distillation method for automatic segmentation of modality-missing brain tumor images provided by an embodiment of the present invention;

[0078] Figure 11 yes Figure 10 A schematic structural diagram of a model training module provided by the illustrated embodiment;

[0079] Figure 12 yes Figure 10 The illustrated embodiment provides a structural diagram of a noise addition and reconstruction module.

[0080] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0081] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0082] The main technical problems solved by the embodiments of the present invention are:

[0083] Existing automatic brain tumor segmentation methods can be roughly divided into three categories: methods based on missing modality synthesis, methods based on latent shared representations, and methods based on knowledge distillation. Among them, the knowledge distillation-based method aims to use the supervisory signal from the full modality sequence to guide the feature optimization of the missing modality through distillation learning. In essence, it achieves feature alignment between full modality features and missing modality features through knowledge distillation. Compared with the first two methods, this type of method is well compatible with other methods and can significantly improve the performance of the model in complex missing scenarios after distillation. However, there are two problems with the existing knowledge distillation work for the task of brain tumor segmentation with missing modalities: first, an additional teacher model must always be trained to guide the student model for distillation learning, and in the inference stage, the teacher model with excellent segmentation performance for the full modality is discarded, resulting in a waste of resources. Second, the segmentation performance of the model still needs to be improved when some key modalities are missing.

[0084] In order to solve the above problems, the following embodiments of the present invention provide a twin self-distillation scheme for automatic segmentation of modality-missing brain tumor images, so as to realize the training and reasoning of the end-to-end missing modality brain tumor automatic segmentation method. Specifically, the following embodiments of the present application use a twin U-shaped network as the basic segmentation network (i.e., generator), and the U-shaped network trains both random modality missing data and full modality data. In the training stage, we use the signal from the random modality missing data as the student signal and the signal from the full modality data as the teacher signal, and then use the complete teacher signal to guide the distillation alignment of the missing student signal in the middle layer and output layer of the U-shaped network, so that the model middle layer features and output layer prediction map can be obtained. In order to better help the model distinguish different twin features, the following embodiments of the present application use a very small convolution block as a discriminator to supervise the above-mentioned segmentation network (i.e., generator) for generative adversarial training. The twin model trained by this method does not require an additional teacher model in the training stage, and can handle complex modality missing situations in the reasoning stage.

[0085] In addition, in order to improve the distillation performance of the twin model, the embodiments of this application explore a variety of commonly used distillation methods. Existing distillation methods do not play an ideal role under the twin training framework of the embodiments of this application. So far, we have proposed two new distillation methods, modality decoupling distillation MDD and label-guided mask generation distillation L-MGD. These two distillation methods are used for the distillation of the intermediate layer features and output layer distillation of the model respectively. Among them, modality decoupling distillation MDD divides the intermediate layer features of the model of the missing modality into missing modality features and non-missing modality features according to the actual missing situation. Then, they are distilled and aligned with the corresponding non-missing modality features. This distillation method has better regularization constraints and generalization capabilities. In addition, in order to enable the model to learn robust feature expressions under complex modality missing conditions, the following embodiments of this application improve the existing mask generation distillation method and propose a label-guided mask generation distillation L-MGD. This method adds background noise and tumor noise to the tumor area and non-tumor area respectively through label guidance, and then uses a small convolution block to reconstruct the noisy signal, and distills and aligns the reconstructed signal with the teacher signal.

[0086] To achieve the above purpose, please refer to Figure 1 , Figure 1 A flowchart of a twin self-distillation method for automatic segmentation of modality-missing brain tumor images provided by an embodiment of the present invention. Figure 1 As shown in Figure 2, the twin self-distillation method for automatic segmentation of modality-missing brain tumor images includes:

[0087] S110: Use the twin U-shaped network to build a twin model. Use labeled brain tumor imaging data to perform supervised training and generative adversarial training on the twin model to obtain the model's intermediate layer features and output layer prediction maps. The output layer prediction maps include missing modality prediction maps and full modality prediction maps.

[0088] The purpose of supervised training and generative adversarial training for the twin model is to eliminate the twin model's dependence on the independent teacher model and improve the twin model's ability to judge similar samples. The twin model is an encoder-decoder model.

[0089] The process is as follows: first, the basic encoder-decoder model is supervised and trained using labeled brain tumor image data, and then an additional discriminator is used to supervise the output layer prediction graph of the twin model. Figure 2 Specifically, the steps of using labeled brain tumor imaging data to perform supervised training and generative adversarial training on the twin model include:

[0090] S111: Input the full modality image data and the missing modality image data of the same sample into the twin model.

[0091] S112: Use the encoder of the twin model to extract the intermediate layer features of the full modality image data and the missing modality image data. Figure 7 As shown in the twin model, the twin model uses the full modality image X of the same sample full and missing modality image X missing As input. The two inputs are respectively passed through the encoder E to obtain the intermediate layer features, which include the full-modal semantic features C f and randomly missing modal semantic features C m Here the middle layer feature C f and C m It is extracted by encoder E, where C f It expresses the full-modal semantic features from full-modal image data, C m It expresses the missing modality semantic features from randomly missing modality image data. The above type of features are common features extracted by the encoder's 3D convolutional neural network.

[0092] S113: Use the decoder of the twin model to decode the intermediate layer features and obtain the output layer prediction map.

[0093] like Figure 7 As shown, the two intermediate layer features C obtained by the twin model f and C m Then, the output layer feature map is obtained through the decoder D of the twin model, that is, the full-modal output feature map Y f And the missing modality output feature map Y m , the output layer feature map obtained here is the full-modal output feature map Y f And the missing modality output feature map Y m In fact, it is the prediction map of the entire segmentation model, which is the segmentation result of the segmentation model on the input brain tumor image. m and Y f are the segmentation results of the missing modality and full modality images respectively.

[0094] S114: Calculate the loss function value of the twin model based on the label and the output layer prediction graph, and use the loss function value to optimize the twin model.

[0095] In the supervised training of the twin model, the label GT of the brain tumor image data and the output layer prediction map Y of the corresponding model are used. m and Y f (The output layer prediction graph Y m and Y f It can be obtained by forward reasoning through the segmentation model, and then the intermediate layer feature C is obtained. f and C m, and then input the obtained intermediate layer features into the decoder to obtain the corresponding prediction graph Y m and Y f ) Calculate the loss value of the twin model's loss function and use this loss value to optimize the model. The loss function used by the twin model is generally Dice loss, a common process in brain tumor segmentation tasks. The reason behind this is to address data imbalance. Data imbalance refers to the huge difference in the proportion of tumor pixels to background pixels (tumors only account for a small part of the entire human brain structure). Therefore, to prevent the network from learning bias, Dice loss is used here.

[0096] S115: Use the discriminator of the twin model to supervise the encoder and optimize the discriminator, encoder and decoder simultaneously.

[0097] In the generation adversarial training of the twin model, such as Figure 7 As shown, the twin model uses the discriminator d to supervise the generator (encoder E and decoder D) for adversarial training, thereby optimizing the discriminator and generator (encoder and decoder) of the above twin model.

[0098] The discriminator d used in the embodiment of the present application is a module composed of multiple convolutional layers, normalization layers, and activation layers. The discriminator d here is actually a common small network. Like the encoder and decoder, they are both composed of multiple convolutional layers, normalization layers, and activation layers in this order. The implementation method can be more flexible. Specifically, in the implementation of this application, the discriminator is composed of four such modules stacked together.

[0099] The specific supervision process is as follows: Specifically, the generator's generation results are optimized through adversarial training between the discriminator d and the generator (the adversarial training used here will update the discriminator and the generator at the same time, and the specific method of adversarial training here is to input the full modality and the missing modality images into the generator to obtain the corresponding prediction map; then input the prediction map of the decoder in the generator into the discriminator, and optimize the generator and discriminator by minimizing the loss function of the adversarial sample discrimination result.

[0100] Finally, when training the discriminator, it learns the random modality missing signal and the full modality signal separately, so that only the discriminator is updated. The specific process is to use the missing modality and full modality prediction graphs to train the discriminator, in order to improve the discriminator's ability to distinguish target samples.

[0101] Figure 1 The twin self-distillation method for automatic segmentation of modality-missing brain tumor images provided by the illustrated embodiment further includes, after obtaining the model intermediate layer features and output layer prediction graph:

[0102] S120: Based on the random modal missing signal of the twin model, modal decoupling is performed on the intermediate layer features of the model to obtain the missing modal features and the non-missing modal features, and distillation alignment training is performed on the missing modal features and the non-missing modal features.

[0103] The modal decoupling and distillation alignment process of the model's intermediate layer features in this step is collectively referred to as modal decoupling distillation. Its purpose is to align the intermediate layer features of the teacher signal and the student signal (where the teacher signal is a full-modal signal and the student signal is a random modal missing signal) to improve the distillation effect of the model.

[0104] For the intermediate layer features of the student signal and the teacher signal, that is, the missing modal semantic feature C m and the omnimodal semantic feature C f , according to the missing condition of the student signal (random mode missing signal) input, the intermediate layer feature C f and C m Decoupled into missing modal features (miss) and non-missing modal features (non-miss); then the missing modal semantic features C in the intermediate layer features are respectively m and the omnimodal semantic feature C f The missing modal features (miss) and non-missing modal features (non-miss) are aligned.

[0105] Specifically, as a preferred embodiment, Figure 3 As shown, the model intermediate layer features include missing modality semantic features and full modality semantic features. The above step S120: performing distillation alignment training on the missing modality features and the non-missing modality features specifically includes:

[0106] S121: Use convolutional blocks to perform secondary encoding on the missing modal semantic features to obtain encoded missing semantic features. Figure 2 As shown in the modal decoupling distillation process, the technical solution provided by the embodiment of the present application needs to use a convolution block (Generation Layer) to align the missing modal semantic features C before aligning the missing modal features with the non-missing modal features. m Perform secondary encoding to obtain the encoding missing semantic feature C mg , using this encoding missing semantic feature C mg Perform subsequent modal feature alignment.

[0107] S122: Divide the encoded missing semantic features and full modality semantic features according to the modality input order to obtain multiple basic modality features, wherein the basic modality features include missing modality features and non-missing modality features.

[0108] S123: Distill the missing modal features in the aligned encoding missing semantic features and the missing modal features of the full-modal semantic features, and distill the non-missing modal features of the aligned encoding missing semantic features and the non-missing modal features of the full-modal semantic features.

[0109] See Figure 8 For example, if the basic modal feature m3 of the student signal input is missing, the embodiment of the present application will replace C mg and the omnimodal semantic feature C f They are divided into four corresponding basic modal features, and then the missing modal features m3 and the non-missing modal features m1, m2 and m4 are aligned (i.e. distilled) separately. In fact, it is based on the missing condition of the input random missing modal image (in Figure 8 The missing situation in the C mg and C f They are divided into four modes, namely m1-m4, and then the missing modal features (m3) and the non-missing modal features (m1, m2 and m4) in the missing cases are aligned respectively.

[0110] Among them, as a preferred embodiment, Figure 4 As shown, the above step S123: distilling the missing modal features of the aligned encoded missing semantic features and the full modal semantic features, and distilling the non-missing modal features of the aligned encoded missing semantic features and the full modal semantic features, specifically includes:

[0111] S1231: Distillation loss function based on missing modalities:

[0112]

[0113] The missing mode distillation loss is calculated, where represents the kth missing modal feature, represents the kth non-missing modal feature;

[0114] S1232: Distillation loss function based on non-missing modalities:

[0115]

[0116] The non-missing mode distillation loss is calculated, where Represents the kth non-missing modal feature; wherein, the alignment loss function is added to the above missing modal distillation loss function and the non-missing modal distillation loss function Where n is the number of missing modes, k is the corresponding mode number, and N is the total number of modes; C, H, and W represent the resolution, dimension, and size of the feature.

[0117] S1233: Use hyperparameters to control the ratio of missing modal distillation loss and non-missing modal distillation loss to obtain modal decoupling distillation loss.

[0118] In this way, after obtaining the missing mode distillation loss function and the non-missing mode distillation loss function, the modal decoupling distillation loss function can be obtained as follows:

[0119] The total modal decoupling distillation loss L MDD is calculated as follows:

[0120] L MMD =(1-α)×L non-miss +α×L miss

[0121] Here, α is a hyperparameter.

[0122] In the technical solution provided in the embodiment of the present application, an additional hyperparameter α is used to control the ratio of the two distillation losses. Distill the missing modal features and the non-missing modal features to obtain the missing modal distillation loss (L miss ) and the non-missing modal distillation loss (L non-miss ).

[0123] Figure 1 The twin self-distillation method for automatic segmentation of modality-missing brain tumor images provided by the illustrated embodiment, after performing distillation alignment training on the missing modality features and the non-missing modality features, further includes:

[0124] S130: randomly adding noise and reconstructing the missing modal prediction map in sequence, and performing distillation alignment on the reconstructed missing modal prediction map and the full modal prediction map.

[0125] The above process is collectively referred to as label-guided mask generation distillation. Label-guided mask generation distillation is performed on the output layer prediction graph of the twin model. Its purpose is to align the output layer prediction graphs of the teacher signal and the student signal, thereby improving the distillation effect of the model.

[0126] The specific process is as follows: first generate a mask matrix according to the preset rules, and then use the generated mask matrix to predict the missing mode (Y m ) is used to add noise, and then an additional convolution block is used to reconstruct the noised features, and the reconstructed features are compared with the full-modal prediction map (Y f ) for alignment (i.e. distillation).

[0127] Among them, as a preferred embodiment, Figure 5 As shown, the steps of randomly adding noise and reconstructing the missing modal prediction graph in the output layer prediction graph in sequence include:

[0128] S131: Generate a random mask matrix of the same size as the label according to a preset mask ratio.

[0129] S132: Using a random mask matrix, respectively generate masks of the tumor region and the non-tumor region in the missing modality prediction image according to the labels.

[0130] A random (0, 1) mask matrix Randommask, equal in size to the label, is generated according to a preset mask ratio β (β is between 0, 1). The mask ratio β is the percentage of 1s in the random matrix. For example, when β = 0.6, 60% of the pixels are 1, and the rest are 0. In our implementation, we set β = 0.5. We then generate masks for the tumor and non-tumor regions in the missing modality prediction image based on the label. The tumor region mask only has a random mask (this random mask is the tumor region of Randommask) in the tumor region and is 0 in the background region. In contrast, the non-tumor region only has a random mask (this random mask is the non-tumor region of Randommask) in the background region and is 0 in the tumor region.

[0131] S133: Traverse the masks of the tumor area and the non-tumor area to obtain pixels of the tumor area and the non-tumor area.

[0132] S134: Using a random mask matrix, noise is added to the tumor region and the non-tumor region in the missing mode prediction image respectively to obtain a noisy missing mode prediction image.

[0133] S135: reconstructing the noisy missing mode prediction map using a convolution block to obtain a reconstructed missing mode prediction map.

[0134] See Figure 9 , the embodiment of the present application adds noise to the missing modality prediction map by traversing the masks of the tumor area and the non-tumor area. First, add noise to the tumor area in the missing modality prediction map: traverse the tumor area mask, if the pixel corresponding to the tumor area mask is 0, then set the corresponding pixel of the missing modality prediction map to the background noise point 0; if the pixel corresponding to the tumor area mask is 1, then it remains unchanged. Then, add noise to the background area of the missing modality prediction map after the initial noise addition: traverse the non-tumor area mask, if the pixel corresponding to the non-tumor area mask is 1, then it remains unchanged; if the pixel is 0, it is set to the foreground noise point (mean_pred_map). After the above noise addition process, the noisy prediction map Y can be obtained. noisy Finally, an additional convolution block Rec is used to reconstruct the noisy missing mode prediction map. The foreground noise point mean_pred_map is the missing mode prediction map Y m The mean of .

[0135] Preferably, the step of performing distillation alignment on the reconstructed missing modal prediction graph and the full modal prediction graph comprises:

[0136] Using the reconstructed missing modality prediction map and the full modality prediction map, the distillation loss function for distillation is generated according to the label-guided mask:

[0137]

[0138] The distillation loss of the label-guided mask generation distillation is calculated, where Y f Represents the full modal prediction diagram, Y noisy Represents the missing mode prediction map after noise addition, Rec represents the convolution block, and k represents the tumor area with different degrees of lesions.

[0139] Distillation loss for label-guided mask generation distillation (L L-MGD ) can be achieved through the above distillation loss function: Calculated; where Rec is a convolution block and k represents the corresponding tumor area.

[0140] in addition, Figure 1 The twin self-distillation method for automatic segmentation of modality-missing brain tumor images provided by the illustrated embodiment further includes, after label-guided mask generation distillation:

[0141] S140: Use the missing modal data to perform inference and model performance verification on the trained twin model to obtain the segmentation accuracy of the twin model. During this model inference and performance verification process, after the twin model is trained, the learned weights can be used for inference, and the segmentation accuracy of the model can be evaluated using relevant performance indicators.

[0142] Specifically as a preferred embodiment, Figure 6 As shown in the figure, the steps of using missing modal data to infer the trained twin model and verify the model performance include:

[0143] S141: When the twin model training is completed, the missing modal data is input into the twin model as the sample to be inferred, and the output layer prediction graph of the sample to be inferred is obtained;

[0144] S142: Use the output layer prediction graph and label of the sample to be inferred, and score the model performance function according to the following:

[0145]

[0146] Calculate the model performance score of the twin model, where predmap represents the output layer prediction map of the sample to be inferred, and label represents the label of the sample to be inferred;

[0147] S143: Select the highest weight corresponding to the model performance score as the inference weight of the twin model.

[0148] The technical solution provided by the embodiment of the present application is that during the training process of the twin model, the twin model accepts both full-modality image and missing-modality image data as input; and during the model inference process, only the required missing-modality image data needs to be input. Among them, the missing modality image data can be the missing of any preset modality (the full modality can also be used as input). And the inference process only needs to forward propagate the image data in the end-to-end encoder-decoder model (without the need for the distillation process in training).

[0149] To verify and evaluate model performance, the Dice Score (Dice Score) is calculated based on the predicted graph and labels of the inference samples. The performance of the twin model is judged based on the Dice Score. A higher Dice Score indicates higher segmentation accuracy, while a lower Dice Score indicates lower accuracy. The specific calculation formula is shown above. The score for this metric is calculated using the predicted graph and labels of the twin model. A higher Dice Score indicates better segmentation accuracy, while a lower Dice Score indicates worse segmentation. After evaluating the model's performance, the weight with the highest Dice Score is selected as the weight for the final inference application.

[0150] In summary, the twin self-distillation method for automatic segmentation of modality-missing brain tumor images provided by the present invention establishes a twin model by using a twin U-shaped network. The twin model serves as a basic segmentation network (i.e., a generator) and can simultaneously train brain tumor image data with random modality missing and brain tumor image data with full modality. In the training stage, the random modality missing signal is used as the student signal, and the full modality signal is used as the teacher signal, so that two types of data can be trained on one model; the twin model is trained to obtain the model intermediate layer features and the output layer prediction map, so that the teacher signal can be used in the intermediate layer and output layer of the twin model to guide the distillation alignment of the modality-missing student signal. In addition, the twin model can also perform generative adversarial training, so that the trained model does not require an additional teacher model in the training stage, and can handle complex modality missing problems in the inference stage. In addition, the technical solution of the present application improves the existing mask generation distillation method. Through the label-guided mask generation distillation method, the missing modal prediction map in the output layer prediction map is randomly noised and reconstructed through label guidance, and then the reconstructed student signal is distilled and aligned with the teacher signal. In this way, the distillation method of the model can have better regularization constraints and generalization capabilities, and can enable the model to learn robust feature expressions under complex modal missing conditions, thereby improving the segmentation performance of the model.

[0151] In addition, based on the same concept of the above-mentioned method embodiment, the embodiment of the present invention also provides a twin self-distillation system for automatic segmentation of modality-missing brain tumor images, which is used to implement the above-mentioned method of the present invention. Since the principles and methods of solving the problems in this system embodiment are similar, it has at least all the beneficial effects brought by the technical solutions of the above-mentioned embodiments, and will not be repeated here one by one.

[0152] See also Figure 10 , Figure 10 A schematic diagram of the structure of a twin self-distillation system for automatic segmentation of modality-missing brain tumor images provided by an embodiment of the present invention. Figure 11 As shown in Figure 2, the twin self-distillation system for automatic segmentation of modality-missing brain tumor images includes:

[0153] A model building module 110 is used to build a twin model using a twin U-shaped network;

[0154] The model training module 120 is used to perform supervised training and generative adversarial training on the twin model using labeled brain tumor imaging data to obtain the model's intermediate layer features and output layer prediction maps; wherein the output layer prediction maps include missing modality prediction maps and full modality prediction maps.

[0155] The modal decoupling module 130 is used to perform modal decoupling on the model's intermediate layer features based on the random modal missing signal of the twin model to obtain missing modal features and non-missing modal features;

[0156] A first distillation alignment module 140 is configured to perform distillation alignment on the missing modal features and the non-missing modal features;

[0157] Noise adding and reconstruction module 150, used for randomly adding noise and reconstructing the missing modal prediction map in the output layer prediction map;

[0158] a second distillation alignment module 160 , configured to perform distillation alignment on the reconstructed missing modality prediction graph and the full modality prediction graph;

[0159] A model inference module 170 is used to infer the trained twin model using missing modal data;

[0160] The performance verification module 180 is used to verify the model performance of the trained twin model and obtain the segmentation accuracy of the twin model.

[0161] In summary, the twin self-distillation system for automatic segmentation of modality-missing brain tumor images provided by the present invention establishes a twin model by using a twin U-shaped network. The twin model serves as a basic segmentation network (i.e., a generator) and can simultaneously train brain tumor image data with random modality missing and brain tumor image data with full modality. In the training stage, the random modality missing signal is used as the student signal, and the full modality signal is used as the teacher signal, so that two types of data can be trained on one model; the twin model is trained to obtain the model intermediate layer features and the output layer prediction map, so that the teacher signal can be used in the intermediate layer and output layer of the twin model to guide the distillation alignment of the modality-missing student signal. In addition, the twin model can also perform generative adversarial training, so that the trained model does not require an additional teacher model in the training stage, and can handle complex modality missing problems in the inference stage. In addition, the technical solution of the present application improves the existing mask generation distillation method. Through the label-guided mask generation distillation method, the missing modal prediction map in the output layer prediction map is randomly noised and reconstructed through label guidance, and then the reconstructed student signal is distilled and aligned with the teacher signal. In this way, the distillation method of the model can have better regularization constraints and generalization capabilities, and can enable the model to learn robust feature expressions under complex modal missing conditions, thereby improving the segmentation performance of the model.

[0162] Among them, as a preferred embodiment, Figure 11 As shown, the model training module 120 includes:

[0163] The data input submodule 121 is used to input the full modality image data and the missing modality image data of the same sample into the twin model;

[0164] A feature extraction submodule 122 is used to extract intermediate layer features of the full modality image data and the missing modality image data using the encoder of the twin model;

[0165] A feature decoding submodule 123 is used to decode the intermediate layer features using the decoder of the twin model to obtain an output layer prediction map;

[0166] A loss calculation submodule 124 is used to calculate the loss function value of the twin model based on the label and the output layer prediction graph, and use the loss function value to optimize the twin model;

[0167] The adversarial training submodule 125 is used to supervise the encoder using the discriminator of the twin model and optimize the discriminator, encoder and decoder.

[0168] Among them, as a preferred embodiment, Figure 12 As shown, in the above twin self-distillation system, the noise addition and reconstruction module 150 includes:

[0169] The matrix generation submodule 151 is used to generate a random mask matrix of the same size as the label according to a preset mask ratio;

[0170] a mask generation submodule 152 for generating masks of the tumor region and the non-tumor region in the missing modality prediction image according to the labels using a random mask matrix;

[0171] The mask traversal submodule 153 is used to traverse the masks of the tumor area and the non-tumor area to obtain pixels of the tumor area and the non-tumor area;

[0172] The prediction image noise adding submodule 154 is used to add noise to the tumor area and the non-tumor area in the missing mode prediction image using a random mask matrix to obtain a noisy missing mode prediction image;

[0173] The prediction graph convolution submodule 155 is configured to reconstruct the noisy missing mode prediction graph using a convolution block to obtain a reconstructed missing mode prediction graph.

[0174] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0178] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third etc. does not indicate any order. These words may be interpreted as names.

[0179] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0180] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A twin self-distillation method for automatic segmentation of modality-missing brain tumor images, characterized by: include: A twin model is established using a twin U-shaped network, and the twin model is supervised trained and generative adversarial trained using labeled brain tumor imaging data to obtain the model's intermediate layer features and output layer prediction maps, wherein the output layer prediction maps include missing modality prediction maps and full modality prediction maps; According to the random modal missing signal of the twin model, modal decoupling is performed on the intermediate layer features of the model to obtain missing modal features and non-missing modal features, and distillation alignment is performed on the missing modal features and non-missing modal features; Randomly adding noise and reconstructing the missing modal prediction map in sequence, and performing distillation alignment on the reconstructed missing modal prediction map and the full modal prediction map; Using the missing modality data to perform reasoning and model performance verification on the trained twin model to obtain the segmentation accuracy of the twin model; The model intermediate layer features include missing modality semantic features and full modality semantic features, and the step of performing distillation alignment on the missing modality features and the non-missing modality features includes: Using a convolution block to perform secondary encoding on the missing modal semantic feature to obtain an encoded missing semantic feature; The encoded missing semantic features and the full modality semantic features are divided according to the modality input order to obtain a plurality of basic modality features, wherein the basic modality features include missing modality features and non-missing modality features; Distilling and aligning the missing modal features of the encoded missing semantic features and the full-modal semantic features, and distilling and aligning the non-missing modal features of the encoded missing semantic features and the full-modal semantic features; The step of sequentially performing random noise addition and reconstruction on the missing modal prediction graph comprises: Generate a random mask matrix of the same size as the label according to a preset mask ratio; Using the random mask matrix, respectively generating masks of the tumor region and the non-tumor region in the missing modality prediction image according to the labels; Traversing the masks of the tumor area and the non-tumor area to obtain pixels of the tumor area and the non-tumor area; Using the random mask matrix to add noise to the tumor area and the non-tumor area in the missing mode prediction map, respectively, to obtain a noisy missing mode prediction map; Reconstructing the noisy missing mode prediction map using a convolution block to obtain a reconstructed missing mode prediction map; The step of performing distillation alignment on the reconstructed missing modality prediction map and the full modality prediction map comprises: Using the noisy missing modality prediction map and the full modality prediction map, the distillation loss function for distillation is generated according to the label-guided mask: ； The distillation loss of the label-guided mask generation distillation is calculated, where Yf represents the full-modal prediction map, Represents the missing mode prediction map after noise addition, Rec represents the convolution block, and k represents the tumor area with different degrees of lesions.

2. The twin self-distillation method according to claim 1, characterized in that The step of using labeled brain tumor image data to perform supervised training and generative adversarial training on the twin model includes: Inputting full modality image data and missing modality image data of the same sample into the twin model; Extracting intermediate layer features of the full modality image data and the missing modality image data using the encoder of the twin model; Decoding the intermediate layer features using the decoder of the twin model to obtain the output layer prediction graph; Calculating a loss function value of the twin model according to the label and the output layer prediction graph, and optimizing the twin model using the loss function value; The encoder is supervised using the discriminator of the twin model, and the discriminator, encoder, and decoder are optimized simultaneously.

3. The twin self-distillation method according to claim 1, characterized in that: The steps of distilling and aligning the missing modal features of the encoded missing semantic features and the missing modal features of the full modal semantic features, and distilling and aligning the missing modal features of the encoded missing semantic features and the non-missing modal features of the full modal semantic features, include: According to the missing mode distillation loss function: ; The missing mode distillation loss is calculated, where represents the kth missing modal feature, represents the non-missing modal features of the kth full modality; According to the non-missing mode distillation loss function: ; The non-missing mode distillation loss is calculated, where represents the kth non-missing modal feature, N is the number of all modes, and n is the number of missing modes; represents the kth non-missing modal feature; A hyperparameter is used to control the ratio of the missing mode distillation loss and the non-missing mode distillation loss to obtain the mode decoupling distillation loss.

4. The twin self-distillation method according to claim 1, characterized in that The step of using the missing modal data to perform reasoning and model performance verification on the trained twin model includes: When the twin model training is completed, the missing modal data is input into the twin model as a sample to be inferred, and an output layer prediction graph of the sample to be inferred is obtained; Use the output layer prediction graph and label of the sample to be inferred, and the model performance scoring function: ； Calculate the model performance score of the twin model, where predmap represents the output layer prediction map of the sample to be inferred, and label represents the label of the sample to be inferred; The highest weight corresponding to the model performance score is selected as the inference weight of the twin model.

5. A twin self-distillation system for automatic segmentation of modality-missing brain tumor images, used to implement the twin self-distillation method according to any one of claims 1 to 4, characterized in that: Said include: Model building module, used to build a twin model using the twin U-shaped network; A model training module, configured to perform supervised training and generative adversarial training on the twin model using labeled brain tumor imaging data to obtain model intermediate layer features and output layer prediction maps, wherein the output layer prediction maps include missing modality prediction maps and full modality prediction maps; A modal decoupling module is used to perform modal decoupling on the intermediate layer features of the model according to the random modal missing signal of the twin model to obtain missing modal features and non-missing modal features; A first distillation alignment module, configured to perform distillation alignment on the missing modal features and the non-missing modal features; A noise adding and reconstruction module, used for randomly adding noise and reconstructing the missing modal prediction map in the output layer prediction map; a second distillation alignment module, configured to perform distillation alignment on the reconstructed missing modality prediction graph and the full modality prediction graph; A model inference module, configured to use missing modal data to infer the trained twin model; The performance verification module is used to verify the model performance of the trained twin model and obtain the segmentation accuracy of the twin model.

6. The twin self-distillation system according to claim 5, characterized in that: The model training module includes: A data input submodule, configured to input full-modality image data and missing-modality image data of the same sample into the twin model; A feature extraction submodule, configured to extract intermediate layer features of the full modality image data and the missing modality image data using the encoder of the twin model; A feature decoding submodule, configured to decode the intermediate layer features using the decoder of the twin model to obtain the output layer prediction map; A loss calculation submodule, configured to calculate a loss function value of the twin model based on the label and the output layer prediction graph, and optimize the twin model using the loss function value; An adversarial training submodule is used to supervise the encoder using the discriminator of the twin model and simultaneously optimize the discriminator, encoder and decoder.

7. The twin self-distillation system according to claim 5, characterized in that: The noise adding and reconstruction module comprises: A matrix generation submodule, configured to generate a random mask matrix of the same size as the label according to a preset mask ratio; a mask generation submodule, configured to use the random mask matrix to generate masks of the tumor region and the non-tumor region in the missing modality prediction image according to the labels; A mask traversal submodule, configured to traverse the masks of the tumor area and the non-tumor area to obtain pixels of the tumor area and the non-tumor area; a prediction image noise adding submodule, configured to use the random mask matrix to add noise to the tumor region and the non-tumor region in the missing mode prediction image respectively, to obtain a noisy missing mode prediction image; The prediction graph convolution submodule is used to reconstruct the noisy missing mode prediction graph using a convolution block to obtain a reconstructed missing mode prediction graph.

Images