T1 brain glioma discrimination method and system based on multi-modal information distillation
Through multimodal information distillation technology, the characteristics of multimodal MR images are distilled to singlemodal T1 images, and a brain glioma discrimination model is constructed, which solves the problem of complex resource consumption in the existing system and lacks multi-dimensional characteristics in the singlemodal system, and achieves efficient and accurate brain glioma discrimination.
Patent Information
- Application Number
- CN202411691186.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-03-14
AI Technical Summary
The existing multimodal glioma discrimination system is complex and resource-consuming, prone to overfitting, reducing generalization ability; the single-modal system lacks multidimensional characteristics, making it difficult to accurately distinguish different levels and types of gliomas.
The T1 brain glioma discrimination method based on multimodal information distillation is adopted to generate multimodal image fusion characteristics through feature extraction and fusion of multimodal multiparameter MR images, and distillate them into singlemodal T1 images through knowledge distillation to construct a brain glioma discrimination model.
During the testing phase, only single-modal T1 images are needed to make the discrimination, which improves the scanning efficiency and economic benefits of patients, and improves the accuracy of brain glioma discrimination.
Smart Images
Figure CN119625397A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent diagnosis and treatment technology, and in particular to a T1 brain glioma identification method and system based on multimodal information distillation. Background Art
[0002] Gliomas are the most common primary tumors in the central nervous system, with high morbidity and mortality. Accurate diagnosis and classification are crucial for developing effective treatment strategies and assessing patient prognosis. Magnetic resonance imaging (MRI) has become an important tool for the diagnosis and evaluation of gliomas because of its ability to provide high-resolution soft tissue contrast. MRI can obtain imaging data in multiple modalities, such as T1-weighted images (T1), T2-weighted images (T2), fluid-attenuated inversion recovery sequences (Flair), and diffusion-weighted imaging (DWI).
[0003] MRI images of different modalities reflect different characteristics of brain gliomas. T1-weighted images mainly display anatomical information, which helps to determine the location and boundaries of the tumor; T2-weighted images and Flair sequences are sensitive to changes in water content in brain tissue and can highlight the difference between tumor tissue and surrounding normal tissue; diffusion-weighted imaging can reflect the diffusion characteristics of water molecules in tumor tissue, which is of great significance for judging the cell density and tissue structure of the tumor.
[0004] The identification of brain gliomas based on these multimodal MRI image data is a current research hotspot in the field of medical image analysis. Traditional methods mainly rely on the doctor's experience and subjective judgment to determine the presence and characteristics of the tumor through visual observation and analysis of different modal images. However, this method has great limitations. It is not only time-consuming and labor-intensive, but also easily affected by the doctor's experience and subjective factors, making it difficult to ensure the accuracy and consistency of the diagnostic results.
[0005] With the development of computer technology and artificial intelligence, methods based on machine learning and deep learning have been widely used in the identification of brain gliomas. These methods use neural network models to automatically learn characteristic patterns in MRI images, and classify and diagnose based on the learned features. Multimodal deep learning methods can integrate the information advantages of different modalities and improve the accuracy of brain glioma identification by processing image data of multiple modalities at the same time. For example, some studies have constructed a complex convolutional neural network (CNN) architecture, taking MRI images of different modalities as input, extracting features through multi-layer convolution and pooling operations, and then classifying through a fully connected layer.
[0006] However, the current brain glioma identification system based on multimodal MRI images still faces some challenges and problems.
[0007] (II) Technical issues
[0008] 1. Problems with multimodal glioma discrimination systems: In order to process multimodal imaging data and fully learn its features, existing multimodal discrimination models are usually designed to be more complex, containing a large number of network layers and parameters. This requires a large amount of computing resources and time to train the model, and is prone to overfitting, which reduces the generalization ability and practical application value of the model. Especially in the clinical verification stage, the multimodal glioma discrimination model also needs to scan multi-parameter MR images, which is not conducive to improving clinical efficiency.
[0009] 2. Problems with single-modality brain glioma discrimination systems: Single-modality MRI images only contain information on one aspect. For example, T1-weighted images mainly focus on anatomical structure information. Compared with multimodal images, their reflection of the pathological characteristics and internal structure of tumor tissue is relatively limited. This makes it difficult for single-modality discrimination systems to distinguish between different levels and types of brain gliomas, and the discrimination accuracy is difficult to compare with that of multimodal systems. Due to the lack of comprehensive utilization of multimodal information, single-modality discrimination systems find it difficult to capture the characteristic changes of tumor tissue in multiple dimensions, and their discrimination capabilities are severely limited. Especially for some complex tumor cases, the information provided by single-modality images may not be sufficient to support accurate diagnosis and classification. Summary of the invention
[0010] Based on the problems existing in the above-mentioned traditional multimodal and single-modal glioma discrimination systems, the purpose of the present invention is to propose a T1 glioma discrimination technology based on multimodal information distillation, aiming to solve these problems and improve the accuracy and efficiency of glioma discrimination.
[0011] In order to achieve the above technical objectives, the present application provides a T1 brain glioma discrimination method based on multimodal information distillation, comprising the following steps:
[0012] Collect multi-modal and multi-parameter MR images, extract and fuse features for each modality, and use a classifier to identify gliomas, generating multi-modal image fusion features for identifying gliomas;
[0013] Acquire T1 images and extract T1 features. Use Gaussian noise resampling technology to form supplementary features, which are then fused with T1 features to form the final fusion features.
[0014] Through knowledge distillation, the multimodal image fusion features are distilled into the best fusion features, and then a glioma discrimination model is constructed to obtain glioma discrimination results based on T1 images.
[0015] Preferably, in the process of acquiring multi-modality multi-parameter MR images, the multi-modality includes but is not limited to T1, T2, Flair and DWI modalities.
[0016] Preferably, in the process of extracting features for each modality, based on each modality, an encoder corresponding to the modality is obtained, and feature extraction is performed through the encoder, wherein the encoder includes but is not limited to Resnet18, VGG16, Swin-Transformer, etc.
[0017] Preferably, in the process of feature extraction and feature fusion for each modality, feature fusion is achieved through a feature fusion module, wherein the feature fusion module includes but is not limited to: feature concatenation, addition, or utilizing the self-attention mechanism in Transformer.
[0018] Preferably, in the process of acquiring the supplementary features, the supplementary features are generated based on the T1 features through a noise encoder, wherein the noise encoder includes but is not limited to: Resnet18, VGG16, Swin-Transformer.
[0019] Preferably, in the process of knowledge distillation, KL divergence is used as the loss function of the model to distill the multimodal image fusion features into new fusion features.
[0020] The present invention discloses a T1 brain glioma discrimination system based on multimodal information distillation, comprising:
[0021] The multimodal feature fusion module is used to collect multimodal and multiparameter MR images, extract and fuse features for each modality, and then use the classifier to identify gliomas, thereby generating multimodal image fusion features for identifying gliomas.
[0022] The T1 feature fusion module is used to obtain T1 images and extract T1 features. After forming supplementary features through Gaussian noise resampling technology, it is fused with T1 features to form the final fusion features.
[0023] The glioma discrimination module is used to distill the multimodal image fusion features into new fusion features through knowledge distillation, and then build a glioma discrimination model to obtain glioma discrimination results based on T1 images.
[0024] Preferably, the multimodal feature fusion module is also used to obtain the encoder corresponding to each modality based on the modality, and perform feature extraction through the encoder, wherein the encoder includes but is not limited to Resnet18, VGG16, Swin-Transformer; the multimodality includes but is not limited to T1, T2, Flair and DWI modalities.
[0025] Preferably, the system performs feature fusion by feature concatenation, addition, or fusion operation using the self-attention mechanism in Transformer.
[0026] Preferably, the glioma discrimination module is also used to use KL divergence as the loss function of the model to distill the multimodal image fusion features into new fusion features.
[0027] The present invention discloses the following technical effects:
[0028] The present invention is consistent with it in the training stage and requires MR data of multiple modalities, but in the testing stage, the present invention only needs to scan and collect single-modality T1 images, thereby improving the patient's scanning efficiency and reducing the patient's economic burden.
[0029] The present invention distills multimodal information into a single-modal brain glioma discrimination system by means of knowledge distillation, thereby providing additional multimodal information for the single-modal brain glioma discrimination method and improving the accuracy of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0031] Figure 1 : is a flow chart of the model described in the present invention, in which part (A) is a brain glioma discrimination network based on multi-parameter MR; part (B) is a brain glioma discrimination system based on single-modality T1;
[0032] Figure 2 This is the model clinical testing process described in the present invention. DETAILED DESCRIPTION
[0033] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0034] The present invention provides a T1 brain glioma discrimination method based on multimodal information distillation, comprising the following steps:
[0035] Collect multi-modal and multi-parameter MR images, extract and fuse features for each modality, and use a classifier to identify gliomas, generating multi-modal image fusion features for identifying gliomas;
[0036] Acquire T1 images and extract T1 features. Use Gaussian noise resampling technology to form supplementary features, which are then fused with T1 features to form the final fusion features.
[0037] Through knowledge distillation, the multimodal image fusion features are distilled into the best fusion features, and then a glioma discrimination model is constructed to obtain glioma discrimination results based on T1 images.
[0038] Further preferably, in the T1 glioma discrimination method based on multimodal information distillation provided by the present invention, in the process of acquiring multimodal multi-parameter MR images, the multimodality includes but is not limited to T1, T2, Flair and DWI modalities.
[0039] Further preferably, in the T1 glioma discrimination method based on multimodal information distillation provided by the present invention, in the process of feature extraction for each modality, an encoder corresponding to the modality is obtained based on each modality, and feature extraction is performed through the encoder, wherein the encoder includes but is not limited to Resnet18, VGG16, Swin-Transformer, etc.
[0040] Further preferably, in the T1 glioma discrimination method based on multimodal information distillation provided by the present invention, in the process of feature extraction and feature fusion for each modality, feature fusion is achieved through a feature fusion module, wherein the feature fusion module includes but is not limited to: feature concatenation, addition, or utilizing the self-attention mechanism in Transformer, etc.
[0041] Further preferably, in the T1 glioma discrimination method based on multimodal information distillation provided by the present invention, in the process of obtaining supplementary features, supplementary features are generated based on T1 features through a noise encoder, wherein the noise encoder includes but is not limited to: Resnet18, VGG16, Swin-Transformer, etc.
[0042] Further preferably, the present invention provides a T1 glioma discrimination method based on multimodal information distillation, which uses KL divergence as the loss function of the model during the knowledge distillation process to distill the multimodal image fusion features into new fusion features.
[0043] The present invention discloses a T1 brain glioma discrimination system based on multimodal information distillation, comprising:
[0044] The multimodal feature fusion module is used to collect multimodal and multiparameter MR images, extract and fuse features for each modality, and then use the classifier to identify gliomas, thereby generating multimodal image fusion features for identifying gliomas.
[0045] The T1 feature fusion module is used to obtain T1 images and extract T1 features. After forming the T1 features through Gaussian noise resampling technology, they are fused with the T1 features to form the final fusion features.
[0046] The glioma discrimination module is used to distill the multimodal image fusion features into the best fusion features through knowledge distillation, and then build a glioma discrimination model to obtain glioma discrimination results based on T1 images.
[0047] Further preferably, the multimodal feature fusion module of the T1 glioma discrimination system based on multimodal information distillation provided by the present invention is also used to obtain the encoder corresponding to each modality based on the modality, and perform feature extraction through the encoder, wherein the encoder includes but is not limited to Resnet18, VGG16, Swin-Transformer, etc.; the multimodality includes but is not limited to T1, T2, Flair and DWI modalities, etc.
[0048] Further preferably, the system performs feature fusion through fusion operations such as feature concatenation, addition, or utilizing the self-attention mechanism in Transformer.
[0049] Further preferably, the glioma discrimination module of the T1 glioma discrimination system based on multimodal information distillation provided by the present invention is also used to use KL divergence as the loss function of the model to distill the multimodal image fusion features into new fusion features.
[0050] Example: Figure 1-2 As shown, the present invention provides a T1 glioma discrimination method based on multimodal information distillation, which specifically includes the following contents: a glioma discrimination network based on multi-parameter MR (first step, A) and a glioma discrimination system based on single-modality T1 (second step, B).
[0051] Step 1: Construction of brain glioma discrimination network (A) based on multi-parameter MR:
[0052] Input layer: This process takes multi-modal and multi-parameter MR images as input, including but not limited to T1, T2, Flair, DWI and other modalities.
[0053] Encoder layer: Each modality has an encoder (including but not limited to Resnet18, VGG16, Swin-Transformer, etc.) to extract features of the corresponding modality. For example, there is a T1 encoder for the T1 modality and a T2 encoder for the T2 modality.
[0054] Feature fusion: The features extracted by each encoder are fused through a feature fusion module (including but not limited to feature concatenation, addition, or using the self-attention mechanism in Transformer) to obtain the fused features.
[0055] Classifier layer: The fused features are input into the classifier to identify brain gliomas. The fused features are used as feature information of multimodal images to facilitate knowledge distillation in the subsequent single-modal training process.
[0056] Step 2: Construction of a brain glioma discrimination system based on single-modality T1 (B):
[0057] Input layer: Input T1 image.
[0058] Shared encoder layer: The shared T1 encoder obtains the T1 feature, but the feature only contains single-modal features.
[0059] Feature generation branch: In order to obtain multimodal fusion features as close as possible, a feature generation branch is designed. New features are obtained by resampling from Gaussian noise and passing through a noise encoder (including but not limited to Resnet18, VGG16, Swin-Transformer, etc.).
[0060] Feature fusion and distillation layer: The new features are fused with the T1 features (including but not limited to feature concatenation, addition, or using the self-attention mechanism in Transformer, etc.), and then the multimodal image features are distilled into the new features through knowledge distillation technology and KL divergence as the loss function, and the brain glioma discrimination results are output (the loss function includes but is not limited to various binary classification loss functions, such as cross entropy, etc.).
[0061] After the model is trained, Figure 2As shown, during testing, the present invention does not require multi-parameter MR images as input, but only needs to use single-modality T1 images as image input to identify brain gliomas, thereby reducing the economic burden on patients and improving clinical scanning efficiency.
[0062] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0063] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A T1 glioma discrimination method based on multimodal information distillation, characterized in that: The following steps are involved: Acquiring multi-modal multi-parameter MR images, extracting and fusing features for each modality, and then distinguishing gliomas through a classifier to generate multi-modal image fusion features for distinguishing the gliomas; Acquire T1 images and extract T1 features, and use Gaussian noise resampling technology to form supplementary T1 features, which are then fused with the T1 features to form final fused features; Through knowledge distillation, the multimodal image fusion features are distilled to the T1 fusion features, and a glioma discrimination model is constructed to obtain glioma discrimination results based on T1 images.
2. According to claim 1, a T1 glioma discrimination method based on multimodal information distillation is characterized by: In the process of acquiring multi-modality and multi-parameter MR images, the multi-modality includes but is not limited to T1, T2, Flair and DWI modalities.
3. According to claim 2, a T1 glioma discrimination method based on multimodal information distillation is characterized by: In the process of extracting features for each modality, based on each modality, an encoder corresponding to the modality is obtained, and feature extraction is performed through the encoder, wherein the encoder includes but is not limited to Resnet18, VGG16, and Swin-Transformer.
4. According to claim 3, a T1 brain glioma discrimination method based on multimodal information distillation is characterized by: In the process of feature extraction and feature fusion for each modality, feature fusion is achieved through a feature fusion module, wherein the feature fusion module includes but is not limited to: feature concatenation, addition, or utilizing the self-attention mechanism in Transformer.
5. The T1 glioma identification method based on multimodal information distillation according to claim 4 is characterized by: In the process of acquiring the supplementary features, the supplementary features are generated based on the T1 features through a noise encoder, wherein the noise encoder includes but is not limited to: Resnet18, VGG16, Swin-Transformer.
6. The T1 glioma identification method based on multimodal information distillation according to claim 5, characterized in that: In the process of knowledge distillation, KL divergence is used as the loss function of the model to distill the multimodal image fusion features into the new fusion features.
7. A T1 glioma discrimination system based on multimodal information distillation, characterized in that: include: A multimodal feature fusion module is used to collect multimodal and multiparameter MR images, extract and fuse features for each modality, and then identify gliomas through a classifier to generate multimodal image fusion features for identifying the gliomas. A T1 feature fusion module is used to obtain T1 images and extract T1 features, and after forming supplementary features through Gaussian noise resampling technology, fuse them with the T1 features to form the final fusion features; The glioma discrimination module is used to distill the multimodal image fusion features into the T1 fusion features through knowledge distillation, and then construct a glioma discrimination model to obtain glioma discrimination results based on T1 images.
8. The T1 glioma discrimination system based on multimodal information distillation according to claim 7, characterized in that: The multimodal feature fusion module is also used to obtain an encoder corresponding to each modality based on the modality, and perform feature extraction through the encoder, wherein the encoder includes but is not limited to Resnet18, VGG16, Swin-Transformer; the multimodality includes but is not limited to T1, T2, Flair and DWI modalities.
9. A T1 glioma discrimination system based on multimodal information distillation according to claim 8, characterized in that: The system performs feature fusion by concatenating and adding features or by using the self-attention mechanism in Transformer.
10. A T1 glioma discrimination system based on multimodal information distillation according to claim 9, characterized in that: The brain glioma discrimination module is also used to use KL divergence as the loss function of the model to distill the multimodal image fusion features into the new fusion features.