Tumor classification method and system based on multimodal completion and knowledge distillation strategy

Abstract

The present invention belongs to the technical field of tumor classification, and provides a tumor classification method and system based on multimodal completion and knowledge distillation strategy, including constructing a first multimodal diagnosis model based on acquired MRI and tissue pathology images as a teacher network; using MRI to generate pseudo pathology images, and constructing a second multimodal diagnosis model based on MRI and pseudo pathology images as a student network; performing knowledge distillation between the teacher network and the student network, defining the objective function of the dual tasks of generation and distillation, jointly optimizing the model parameters, and obtaining a trained tumor classification model; and using the trained tumor classification model to perform tumor recognition and classification. The present invention generates pseudo pathology images based on magnetic resonance images, and combines knowledge distillation methods to transfer the multimodal diagnostic performance of magnetic resonance images and real pathology images to magnetic resonance images and pseudo pathology image models, thereby improving the accuracy of preoperative tumor molecular diagnosis and the interpretability of results.

Description

Technical Field

[0001] The present disclosure relates to the technical field of tumor classification, and in particular to a tumor classification method and system based on multimodal completion and knowledge distillation strategy. Background Art

[0002] As a major health problem worldwide, tumors seriously threaten human life. They not only bring a heavy medical burden, but also pose a huge challenge to social and economic development. Accurate classification of tumors is the basis for tumor diagnosis and subsequent treatment. According to the classification results of tumors, individualized treatment plans can be provided to patients. Accurate classification and individualized treatment of tumors depend on the results of molecular testing. For example, molecular testing can evaluate the PD-L1 expression level of lung cancer patients, assist in distinguishing patients who are effective and ineffective in immunotherapy, and avoid ineffective treatment; molecular testing is the basis for the classification of gliomas. However, molecular testing usually produces results about a week after surgery and cannot be used for preoperative planning of treatment plans. In clinical practice, there is an urgent need for a tool for non-invasive preoperative assessment of tumor molecular characteristics.

[0003] Medical imaging helps to evaluate the molecular characteristics of tumors before surgery. For example, magnetic resonance imaging (MRI) can present the overall structure of the tumor non-invasively and is an important tool for the initial diagnosis of tumors. The macroscopic structural information on MRI can predict the molecular characteristics and histological grade of the tumor. A "Radiology" study extracted the tumor morphological characteristics and apparent diffusion coefficient of MRI to construct a machine learning model, achieving an 86% accuracy rate in the molecular diagnosis of gliomas. However, molecular diagnosis based solely on information from a single MRI modality lacks reliability and clinical interpretability.

[0004] Histopathological images can present the morphology and microenvironment of cells, and are an important basis for the histological grading of tumors. Combined with deep learning, they can improve the accuracy of molecular diagnosis. Previous studies have identified molecular subtypes of brain tumors based on histopathology, achieving accuracies of 93.9% and 82.0% on internal and external validation sets, respectively. A Nature Communications study also constructed a histopathology deep learning model, with an area under the receiver operating curve of higher than 0.9 for molecular classification of gliomas. In addition, previous studies have confirmed that the diagnostic performance of a multimodal model constructed by combining MRI and histopathology images is better than that of a single modality model. Therefore, histopathology has great potential in predicting tumor molecular subtypes, but it usually still needs to be obtained during surgery or early after surgery. How to complete the missing pathological modality information before surgery is a key issue in improving the accuracy of molecular diagnosis of tumors.

[0005] Recent studies have used knowledge distillation to solve the problem of missing modalities, transferring the knowledge of a multimodal teacher network to a single-modal student network, so that the student network has multimodal diagnostic performance. However, the current tumor molecular diagnosis model based on knowledge distillation still has limitations: (1) Using tissue pathology and gene maps to distinguish molecular subtypes cannot achieve preoperative non-invasive diagnosis; (2) Compared with the teacher network, the performance of the student network will decline; (3) The knowledge propagation method between the teacher network and the student network is unknown and lacks clinical interpretability. Preoperative diagnosis of tumors requires a more intuitive modality completion strategy to improve diagnostic accuracy and interpretability.

[0006] Generative AI has a strong ability to create content. It can achieve the conversion and completion of multimodal data by learning the mapping relationship between modalities. It performs well in MRI super-resolution, sequence completion, cross-modal conversion, and pathological image generation and completion. Through generative AI technology, microstructure images of tissue pathology are generated on the macrostructure of MRI to characterize key MRI information related to tumor molecular typing, which is expected to build a bridge between the macrostructure and microstructure of tumors and improve the accuracy and interpretability of tumor molecular diagnosis. Summary of the invention

[0007] In response to the above-mentioned defects, the present invention proposes a tumor classification method and system based on multimodal completion and knowledge distillation strategy. By learning the mapping relationship between MRI macrostructure and tissue pathology micromorphology, pseudo-pathological images with the same diagnostic ability as real pathological images are generated based on MRI. In combination with the knowledge distillation method, the multimodal diagnostic performance of MRI and real pathological images is transferred to the MRI and pseudo-pathological image model, thereby improving the accuracy of preoperative molecular diagnosis and the interpretability of the results.

[0008] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0009] The first aspect of the present disclosure provides a tumor classification method based on multimodal completion and knowledge distillation strategy, comprising the following steps:

[0010] The first multimodal diagnostic model is constructed based on the acquired MRI and histopathological images as a teacher network;

[0011] Generate pseudo-pathological images using MRI, and build a second multimodal diagnosis model based on MRI and pseudo-pathological images as a student network;

[0012] Perform knowledge distillation between the teacher network and the student network, define the objective functions of the dual tasks of generation and distillation, jointly optimize the model parameters, and obtain a trained tumor classification model;

[0013] The trained tumor classification model is used for tumor identification and classification.

[0014] As a further implementation method, the acquired MRI and tissue pathology images are also preprocessed.

[0015] As a further implementation method, the tissue pathology image is preprocessed, specifically:

[0016] The tissue pathology image is divided into several pathology sub-images, and the pathology sub-images with a high probability of correct diagnosis are screened out as the gold standard for generating pseudo-pathology images by MRI.

[0017] As a further implementation method, MRI is preprocessed, specifically:

[0018] The tumor area outlined by the doctor on the MRI is evenly cropped into several sub-regions for subsequent sub-region association analysis, and different sequences are registered and aligned to achieve regional correspondence between sequences.

[0019] As a further implementation method, a first multimodal diagnosis model is constructed, specifically:

[0020] Self-attention is used to identify the structural interaction information between MRI sequences, between MRI sub-regions, and between sub-regions of tissue pathology images, and cross-attention is used to perform information joint training to distinguish the molecular typing and grading of tumors.

[0021] As a further implementation method, MRI is used to generate pseudo pathological images, specifically:

[0022] The structural interactions between MRI sequences and MRI sub-regions are identified and fused through the hybrid attention module, and the structural fusion information is integrated and refined through the scale transformation module, and gradually transformed into pseudo pathological images of the target resolution.

[0023] As a further implementation, the total loss of the objective function includes the loss of the distillation task, the loss of the generation task, and the classification loss of the student network.

[0024] A second aspect of the present disclosure provides a tumor classification system based on multimodal completion and knowledge distillation strategy, comprising:

[0025] The teacher network construction module is configured to: construct a first multimodal diagnosis model based on the acquired MRI and tissue pathology images as a teacher network;

[0026] The student network construction module is configured to: generate pseudo pathological images using MRI, and construct a second multimodal diagnosis model based on the MRI and the pseudo pathological images as a student network;

[0027] The model training module is configured to: perform knowledge distillation between the teacher network and the student network, define the objective functions of the dual tasks of generation and distillation, jointly optimize the model parameters, and obtain a trained tumor classification model;

[0028] The recognition and classification module is configured to: use the trained tumor classification model to perform tumor recognition and classification.

[0029] The third aspect of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps in the tumor classification method based on multimodal completion and knowledge distillation strategy described in the first aspect of the present disclosure are implemented.

[0030] The fourth aspect of the present disclosure provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of a tumor classification method based on multimodal completion and knowledge distillation strategy described in the first aspect of the present disclosure are implemented.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The tumor classification method and system based on multimodal completion and knowledge distillation strategy proposed in the present invention aims at the difficulty of obtaining pathological modalities in preoperative non-invasive diagnosis of tumors, and innovatively proposes a generative multimodal completion strategy to explore the mapping relationship between the tumor MRI macrostructure and the tissue pathology microstructure. By learning the mapping relationship between the MRI macrostructure and the tissue pathology micromorphology, pseudo-pathological images are generated based on preoperative MRI, and a multimodal molecular diagnosis model is constructed by combining MRI and pseudo-pathological images. Through knowledge distillation, the multimodal diagnostic performance of MRI and real pathological images is transferred to the multimodal molecular diagnosis model of MRI and pseudo-pathological images, so that the performance of the multimodal molecular diagnosis model of MRI and real pathological images is close to that of the multimodal molecular diagnosis model of MRI and real pathological images, so as to improve the preoperative diagnostic accuracy of tumor molecular typing. At the same time, through the generation and visualization of pathological information, the key information of molecular diagnosis is clarified to improve the interpretability of the results.

[0033] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0035] Figure 1 is a schematic diagram of the framework structure of the multimodal diagnostic model disclosed in the present invention;

[0036] Figure 2It is a schematic diagram of the framework structure of the dual-task model of MRI generating pseudo-pathological images and knowledge distillation disclosed in the present invention, wherein (a) is a schematic diagram of the dual-task network structure of image generation and knowledge distillation, and (b) is a schematic diagram of the scale transformation module structure. DETAILED DESCRIPTION

[0037] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0038] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0039] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0040] Embodiment 1

[0041] like Figure 1-2 As shown, the first embodiment of the present disclosure provides a tumor classification method based on multimodal completion and knowledge distillation strategy, which mainly includes the following steps:

[0042] S1 builds the first multimodal diagnosis model based on the acquired MRI and histopathological images as a teacher network;

[0043] S2 uses MRI to generate pseudo-pathological images and builds a second multimodal diagnosis model based on MRI and pseudo-pathological images as a student network;

[0044] S3 performs knowledge distillation between the teacher network and the student network, defines the objective functions of the dual tasks of generation and distillation, jointly optimizes the model parameters, and obtains a trained tumor classification model;

[0045] S4 uses the trained tumor classification model to perform tumor identification and classification.

[0046] The present invention collects MRI and pathological data of tumor patients, builds a molecular pathology diagnosis model based on tissue pathology images, and selects diagnostic pathology thumbnails as the gold standard for generation tasks. Then, a multimodal diagnosis model is constructed based on MRI and real pathology images as a teacher network for knowledge distillation. The MRI feature matrix is ​​then input into the TransformerGAN dual-task model to generate pseudo-pathology thumbnails, and a multimodal molecular diagnosis student network for generation and completion is jointly constructed. Knowledge distillation is performed with the teacher network, and the image generation effect and the diagnostic performance of the student network are improved through joint training of the dual tasks of image generation and knowledge distillation. Finally, the effectiveness of the generative multimodal completion strategy is verified by comparing the diagnostic performance of the student network with the teacher network and the student network of traditional knowledge distillation. The specific research contents are as follows:

[0047] (1) Construction of multimodal dataset.

[0048] The molecular pathology test results, histopathology images and MRI sequences of tumor patients were collected, and the molecular pathology test results were the gold standard in the present invention. The histopathology images were divided into several pathology sub-images, and the pathology diagnosis model was trained based on weak supervision to predict molecular typing and grading. The model was used to screen out pathology sub-images with a high probability of correct diagnosis as the gold standard for MRI to generate pathology images. The tumor area outlined by the doctor on the MRI was evenly cut into several sub-regions for subsequent sub-region association analysis, and different sequences were registered and aligned to achieve regional correspondence between sequences.

[0049] (2) Construction and verification of a multimodal diagnostic model based on MRI and real pathological images.

[0050] like Figure 1 As shown in the figure, a multimodal diagnosis model is constructed based on the mixed attention module of MRI and pathology based on Transformer. Self-attention is used to identify the structural interaction information between MRI sequences, between MRI sub-regions, and between sub-regions of tissue pathology images, and cross-attention is used to perform information joint training to distinguish the molecular typing and histological grading of tumors. During the training process, pathology images are randomly extracted to enhance data and improve model performance. The model performance is verified by molecular diagnosis and grading accuracy in the validation set. The multimodal diagnosis model constructed based on MRI and real pathology images is used as the teacher network for subsequent knowledge distillation tasks. When the TransformerGAN model generates pseudo pathology images, another multimodal diagnosis model based on MRI and pseudo pathology images is constructed as the student network.

[0051] First, the self-attention (SA) module is used to calculate the MRI sequence feature matrix according to formula (1): MRI sub-region feature matrix Inter- and pathological image patch feature matrices The structural feature fusion relationship between them:

[0052] (1)

[0053] in, They are , and , corresponding They are , and . Used for Perform a scale transformation to prevent When the value is too large, softmax The gradient of the function vanishes. , is the query matrix; , is the keyword matrix; , is a numerical matrix; , and are hyperparameters that are automatically learned during training.

[0054] Then, the MRI sequence fusion information and the MRI sub-region fusion information are cross-combined through the cross-attention (CA) module:

[0055] (2)

[0056] (3)

[0057] in, It is the structural feature fusion relationship of one set of features and the concatenation of another set of features.

[0058] and On the one hand, respectively and Combine and re-enter the self-attention module for a new round of feature fusion training. On the other hand, combine and combine with pathological information Further fusion, the fusion method is the same as formula (2)-(3). The fusion of MRI and pathological information is repeated N After the linear layer and Softmax The function outputs the prediction result. During the training process, pathology images are randomly extracted to enhance the data and improve the model performance. The model performance is verified by molecular diagnosis and grading accuracy in the validation set.

[0059] (3) Construction and verification of the TransformerGAN dual-task model

[0060] like Figure 2As shown in the figure, a TransformerGAN dual-task model is constructed based on Transformer hybrid attention to simultaneously implement the MRI pathology image generation task and the knowledge distillation task between the teacher network and the student network. The generation task uses the hybrid attention module to identify and fuse the structural interaction between MRI sequences / sub-regions, and integrates and refines the structural fusion information through the scale transformation module, and gradually transforms it into a pathology image of the target resolution. The distillation task is performed by offline distillation to reduce the number of parameters. The student network uses the same network structure as the teacher network to prevent performance degradation. The gradient is updated jointly by the distillation loss, the student network classification loss and the generation loss to optimize the generation and distillation effects. The pathology generation effect is verified by the molecular diagnosis accuracy and the pathology expert Turing test respectively; at the same time, the effectiveness of the generative multimodal completion strategy is verified by comparing the diagnostic performance with the teacher network and the student network of traditional knowledge distillation.

[0061] The student network in the distillation task has the same network structure as the teacher network. The difference from the teacher network is that the input data of the student network is MRI and the generated pseudo pathology map. L KD Calculate the difference in output predictions between the teacher network and the student network, and the classification loss function of the student network L S , the loss function of the generation task L G,D Superimposed together, as the total loss function L:

[0062] (4)

[0063] in, and are the coefficients of each loss function, which are adjusted during the training process. L KD The calculation formula is:

[0064] (5)

[0065] in, and Respectively i The output probabilities of the sample teacher network and student network are calculated as follows:

[0066] (6)

[0067] in, is the output of the model, T is the distillation temperature coefficient, by adjusting TThe distillation probability distribution can be changed to suppress overfitting, reduce the impact of noise labels, and improve robustness.

[0068] The calculation method is:

[0069] (7)

[0070] in, For the i The labels of samples.

[0071] L G,D The calculation method is:

[0072] (8)

[0073] in, For the real pathological image, is the distribution of real pathological images, is the input MRI feature, is the distribution of MRI features, is the expected value of the distribution function, G For the generator, D For the discriminator.

[0074] pass L Gradient updates are performed, and the generation task and distillation task are jointly supervised and optimized, ultimately resulting in a higher-performance student network model and a generator that can generate high-quality pathological images.

[0075] The trained student network model (tumor classification model) is used for tumor identification and classification.

[0076] Embodiment 2

[0077] Embodiment 2 of the present disclosure provides a tumor classification system based on multimodal completion and knowledge distillation strategy, including:

[0078] The teacher network construction module is configured to: construct a first multimodal diagnosis model based on the acquired MRI and tissue pathology images as a teacher network;

[0079] The student network construction module is configured to: generate pseudo pathological images using MRI, and construct a second multimodal diagnosis model based on the MRI and the pseudo pathological images as a student network;

[0080] The model training module is configured to: perform knowledge distillation between the teacher network and the student network, define the objective functions of the dual tasks of generation and distillation, jointly optimize the model parameters, and obtain a trained tumor classification model;

[0081] The recognition and classification module is configured to: use the trained tumor classification model to perform tumor recognition and classification.

[0082] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0083] Embodiment 3

[0084] Embodiment 3 of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps in the tumor classification method based on multimodal completion and knowledge distillation strategy as described in Embodiment 1 of the present disclosure are implemented.

[0085] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0086] Embodiment 4

[0087] Embodiment 4 of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a tumor classification method based on multimodal completion and knowledge distillation strategy as described in Embodiment 1 of the present disclosure are implemented.

[0088] The more detailed steps are the same as those in the first embodiment and will not be repeated here.

[0089] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A tumor classification method based on multimodal completion and knowledge distillation strategy, characterized in that: The following steps are involved: The first multimodal diagnostic model is constructed based on the acquired MRI and histopathological images as a teacher network; Construct the first multimodal diagnostic model, specifically: Self-attention is used to identify the structural interaction information between MRI sequences, MRI sub-regions, and histopathology sub-regions, and cross-attention is used to perform information joint training to distinguish the molecular typing and grading of tumors. Generate pseudo-pathological images using MRI, and build a second multimodal diagnosis model based on MRI and pseudo-pathological images as a student network; Generate pseudo-pathological images using MRI, specifically: The hybrid attention module is used to identify and fuse the structural interactions between MRI sequences and MRI sub-regions, and the scale transformation module is used to integrate and refine the structural fusion information, gradually transforming it into a pseudo-pathological image of the target resolution. Perform knowledge distillation between the teacher network and the student network, define the objective functions of the dual tasks of generation and distillation, jointly optimize the model parameters, and obtain a trained tumor classification model; The trained tumor classification model is used for tumor identification and classification.

2. The tumor classification method based on multimodal completion and knowledge distillation strategy as described in claim 1, characterized in that: It also includes preprocessing of acquired MRI and histopathology images.

3. The tumor classification method based on multimodal completion and knowledge distillation strategy as described in claim 2, characterized in that: Preprocess the tissue pathology images, specifically: The tissue pathology image is divided into several pathology sub-images, and the pathology sub-images with a high probability of correct diagnosis are screened out as the gold standard for generating pseudo-pathology images by MRI.

4. The tumor classification method based on multimodal completion and knowledge distillation strategy as described in claim 2, characterized in that: MRI was preprocessed as follows: The tumor area outlined by the doctor on the MRI is evenly cropped into several sub-regions for subsequent sub-region association analysis, and different sequences are registered and aligned to achieve regional correspondence between sequences.

5. The tumor classification method based on multimodal completion and knowledge distillation strategy as described in claim 1, characterized in that: The total loss of the objective function includes the loss of the distillation task, the loss of the generation task, and the classification loss of the student network.

6. A tumor classification system based on multimodal completion and knowledge distillation strategy, characterized in that: include: The teacher network construction module is configured to: construct a first multimodal diagnosis model based on the acquired MRI and tissue pathology images as a teacher network; Construct the first multimodal diagnostic model, specifically: Self-attention is used to identify the structural interaction information between MRI sequences, MRI sub-regions, and histopathology sub-regions, and cross-attention is used to perform information joint training to distinguish the molecular typing and grading of tumors. The student network construction module is configured to: generate pseudo pathological images using MRI, and construct a second multimodal diagnosis model based on the MRI and the pseudo pathological images as a student network; Generate pseudo-pathological images using MRI, specifically: The hybrid attention module is used to identify and fuse the structural interactions between MRI sequences and MRI sub-regions, and the scale transformation module is used to integrate and refine the structural fusion information, gradually transforming it into a pseudo-pathological image of the target resolution. The model training module is configured to: perform knowledge distillation between the teacher network and the student network, define the objective functions of the dual tasks of generation and distillation, jointly optimize the model parameters, and obtain a trained tumor classification model; The recognition and classification module is configured to: use the trained tumor classification model to perform tumor recognition and classification.

7. A storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the tumor classification method based on multimodal completion and knowledge distillation strategy as described in any one of claims 1 to 5 are implemented.

8. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the steps in the tumor classification method based on multimodal completion and knowledge distillation strategy as described in any one of claims 1-5.

Images