Multimodal intracavitary tumor imaging staging prediction method, system, device and medium

Abstract

The present invention is a staging prediction method, system and device for multimodal intracavitary tumor images, which relates to the field of medical imaging technology. First, a multimodal MRI image information disentanglement module is designed to effectively extract common representations in different MRI modalities; then, the common representations between modalities are used to complete missing modal information filling and knowledge distillation in the Teacher-Student network layer; finally, the prediction information of different modal networks is coupled to realize intelligent staging prediction of intracavitary tumors. The present invention innovatively introduces multimodal information disentanglement and domain knowledge deep distillation strategies to effectively separate common and specific information in multimodal MRI images, and use common information to fill missing modal information, effectively realizing accurate staging and non-invasive biopsy of intracavitary tumors when any imaging modality or modality combination is input, which is expected to provide strong support for the best clinical treatment decision.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and in particular to a staging prediction method for multi-modal intracavitary tumor imaging. Background Art

[0002] Intracavitary tumors mainly include colorectal cancer, gastric cancer, bladder cancer, cervical cancer, esophageal cancer, etc., accounting for about 60% of human tumors. The clinical manifestations are hidden, and most patients have progressed to the late stage when diagnosed, resulting in an abnormally high mortality rate.

[0003] The TNM staging of tumors is one of the most important bases for formulating patient stratification and clinical strategies. Intracavitary tumors mostly originate from the intracavitary mucosal layer. While growing into the cavity, they gradually infiltrate and grow into the submucosal layer, muscular layer, and the outside of the cavity wall. Therefore, the T stage of the tumor mainly depends on the degree of tumor infiltration into the cavity wall, and the N stage, that is, whether there is lymph node metastasis, is also an important factor in determining the diagnosis and treatment of intracavitary tumors. Once lymph node metastasis occurs, the patient's 5-year survival rate will be greatly reduced (15%–31% vs.>60%) (4), and the probability of recurrence in cases with lymph node metastasis is much higher than that in negative cases (80% vs. 30%). The National Comprehensive Cancer Network (NCCN) guidelines point out that different TNM stages (degree of muscle layer invasion and metastasis) and recurrence risks lead to different patient stratification, and the corresponding surgical, treatment, and follow-up strategies are also different. Therefore, accurate preoperative prediction of the T and N stages of intracavitary tumors is of great significance for patient stratification, formulation of treatment strategies, and prognosis evaluation.

[0004] Optical endoscopy is the gold standard for clinical examination and diagnosis of intracavitary tumors, but its invasive detection method not only involves wounds, but also small-field sampling biopsies can only reflect the local characteristics of tumor lesions, which can easily lead to underestimation or misjudgment of tumor infiltration depth or stage. Taking bladder cancer as an example, multiple clinical studies have shown that the stage and grade of 9%-49% of bladder cancer patients are underestimated or misjudged, which affects the prognosis of patients and easily leads to the risk of intracavitary tumor recurrence, significantly reducing the survival benefit of patients.

[0005] As an important clinical examination method, medical imaging can comprehensively evaluate tumors and surrounding tissues, as well as lymph node metastasis, and plays an increasingly important role in the precise diagnosis and treatment of tumors. In recent years, in response to the clinical needs of diagnosis and treatment of malignant tumors, the preoperative diagnosis and treatment of tumors and individualized treatment decision-making technologies using multimodal imaging combined with artificial intelligence-assisted diagnosis methods have continued to develop, bringing hope for preoperative non-invasive, large-field-of-view, and precise biopsy of intracavitary tumors.

[0006] Although multimodal imaging intelligent analysis technology has greater potential in non-invasive tumor biopsy than single-modal imaging analysis, there are the following problems in practical applications: ① Modality loss, that is, not all patients have all the imaging modalities required by the intelligent analysis model effectively collected before surgery, and there are different degrees of modality loss; ② Modality feature entanglement leads to information redundancy, which reduces the predictive efficiency and generalization ability of intelligent imaging analysis technology. There is a problem that multimodal imaging intelligent diagnosis methods are difficult to apply when faced with modality loss and incomplete modality information. Summary of the invention

[0007] In order to overcome the shortcomings of the above-mentioned prior art multimodal imaging intelligent diagnosis method in the face of modality loss and incomplete modality information, the main purpose of the present invention is to provide a multimodal intracavitary tumor imaging staging prediction method, system, device and medium.

[0008] To achieve the above object, the present invention adopts the following technical solution, a multi-modal intracavitary tumor imaging staging prediction method, comprising the following steps:

[0009] A multimodal image standardization strategy was used to process the intracavitary tumor images, and the bounding boxes were manually constructed for the images to obtain processed multimodal tumor images, including main modality images and submodality images.

[0010] Extracting partial and complete original features of the main modality image and the sub-modality image respectively, intertwining the original features of the main modality image with the original features of the corresponding sub-modality image respectively, and obtaining the commonality representation matrix between multiple main image modalities and corresponding modalities;

[0011] The Encoder-Decoder module is introduced to extract features from multiple common representation matrices, and then merged with sub-modal image features to obtain multiple fusion features.

[0012] Multiple fusion features are distilled through the Teacher network to obtain the prediction results of intracavitary tumor staging;

[0013] At the same time, the Student network is designed, and the Teacher-Student feature loss function is constructed by using the original features of the subject modality image and the distilled-coupled features of the Teacher network to obtain the intracavitary tumor staging prediction results of the Student network;

[0014] Different staging prediction results of intracavitary tumors are obtained according to the Teacher and Student networks respectively. The distillation prediction loss function of KL divergence is introduced, and soft voting is used to fuse the staging prediction results of different intracavitary tumors to obtain the staging prediction results of intracavitary tumor images.

[0015] The partial and complete original features of the main modality image and the sub-modality image are extracted by an encoder at the network head;

[0016] The information entanglement is a feature untangling module based on the regional mutual information analysis strategy to extract the image features of the subject's nuclear magnetic resonance modality, obtain the mutual information through the image sub-region features with the most disease prediction ability corresponding to other modalities, and use the loss function of the feature untangling module to maximize the mutual information, thereby obtaining the commonality representation matrix between the subject's image modality and the corresponding modality.

[0017] The introduction of the Encoder-Decoder module to extract features from multiple common representation matrices and to merge and merge them with sub-modal image features includes:

[0018] The Encoder-Decoder module is introduced to extract features from the common representation matrix. The original features extracted by the head encoder in the submodal image are concatenated with the features extracted by the Encoder-Decoder module of the common representation matrix, and a 1×1×4 convolution is performed to obtain the fused features.

[0019] After the multiple fused features, Maxpooling operation is used to perform feature dimensionality reduction, and ReLU is used for feature activation. After multiple layers of convolution-pooling-activation operations, the network distills the key information and domain knowledge in the multimodal images that can effectively characterize the staging characteristics of intracavitary tumors.

[0020] The multimodal image standardization strategy includes data noise reduction, filtering, and grayscale standardization processing, so that images of different modalities show better consistency.

[0021] Multimodal intracavitary tumor imaging staging prediction system, including:

[0022] The feature extraction module uses a multimodal image standardization strategy to process the intracavitary tumor image and manually constructs a bounding box for the image to obtain a processed multimodal tumor image, including a main modal image and a submodal image; extracts local and complete original features of the main modal image and the submodal image respectively, and entangles the original features of the main modal image with the original features of the corresponding submodal image to obtain a common representation matrix between multiple main image modalities and corresponding modalities;

[0023] The feature fusion module introduces the Encoder-Decoder module to extract features from multiple common representation matrices, and then merges and fuses them with submodal image features to obtain multiple fused features. Multiple fused features are distilled through the Teacher network to obtain key information and domain knowledge of intracavitary tumor staging characteristics.

[0024] Construct the Teacher-Student feature loss function module, design the Student network, and use the original features of the subject modality image and the distilled-coupled features of the Teacher network to jointly construct the Teacher-Student feature loss function;

[0025] In the staging prediction module, the Teacher and Student networks obtain different staging prediction results of intracavitary tumors respectively, introduce the KL divergence distillation prediction loss function, and use soft voting to fuse the staging prediction results of different intracavitary tumors to obtain the staging prediction results of intracavitary tumor images.

[0026] A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the following steps:

[0027] A multimodal image standardization strategy was used to process the intracavitary tumor images, and the bounding boxes were manually constructed for the images to obtain processed multimodal tumor images, including main modality images and submodality images.

[0028] Extracting partial and complete original features of the main modality image and the sub-modality image respectively, intertwining the original features of the main modality image with the original features of the corresponding sub-modality image respectively, and obtaining the commonality representation matrix between multiple main image modalities and corresponding modalities;

[0029] The Encoder-Decoder module is introduced to extract features from multiple common representation matrices, and then merged with sub-modal image features to obtain multiple fusion features.

[0030] Multiple fusion features are distilled through the Teacher network to obtain the prediction results of intracavitary tumor staging;

[0031] At the same time, the Student network is designed, and the Teacher-Student feature loss function is constructed by using the original features of the subject modality image and the distilled-coupled features of the Teacher network to obtain the intracavitary tumor staging prediction results of the Student network;

[0032] Different staging prediction results of intracavitary tumors are obtained according to the Teacher and Student networks respectively. The distillation prediction loss function of KL divergence is introduced, and soft voting is used to fuse the staging prediction results of different intracavitary tumors to obtain the staging prediction results of intracavitary tumor images.

[0033] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the following steps: using a multimodal image standardization strategy to process an intracavitary tumor image, and manually constructing a bounding box for the image to obtain a processed multimodal tumor image, including a main modality image and a submodality image;

[0034] Extracting partial and complete original features of the main modality image and the sub-modality image respectively, intertwining the original features of the main modality image with the original features of the corresponding sub-modality image respectively, and obtaining the commonality representation matrix between multiple main image modalities and corresponding modalities;

[0035] The Encoder-Decoder module is introduced to extract features from multiple common representation matrices, and then merged with sub-modal image features to obtain multiple fusion features.

[0036] Multiple fusion features are distilled through the Teacher network to obtain the prediction results of intracavitary tumor staging;

[0037] At the same time, the Student network is designed, and the Teacher-Student feature loss function is constructed by using the original features of the subject modality image and the distillation-coupled features of the Teacher network to obtain the intracavitary tumor staging prediction results of the Student network;

[0038] Different staging prediction results of intracavitary tumors are obtained according to the Teacher and Student networks respectively. The distillation prediction loss function of KL divergence is introduced, and soft voting is used to fuse the staging prediction results of different intracavitary tumors to obtain the staging prediction results of intracavitary tumor images.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows: the present application further obtains the staging prediction results of the tumor by extracting and fusing multimodal features of tumor images, constructing a Teacher-Student feature loss function, and performing further processing to obtain the staging prediction results of the tumor images. Through the staging prediction of tumor images, a multimodal intracavitary tumor image intelligent analysis and prediction method is disclosed, which can be used for accurate preoperative staging diagnosis and non-invasive biopsy of luminal organ tumors such as bladder cancer, colorectal cancer, cervical cancer, and gastric cancer. It effectively solves the field pain points of the existing multimodal image intelligent diagnosis methods that are difficult to apply when faced with modality missing and incomplete modal information, as well as the technical pain points of the model overfitting caused by the entanglement and overlap of modal features resulting in a large amount of information redundancy.

[0040] The present invention innovatively introduces multimodal information disentanglement and domain knowledge deep distillation strategies to effectively separate common and specific information in multimodal MRI images, and use common information to complete missing modal information, effectively realizing accurate staging and non-invasive biopsy of intracavitary tumors when any MRI image modality or multimodal combination input is used, which is expected to provide strong support for the best clinical treatment decision. In general, the integration of multimodal feature disentanglement, information filling, knowledge distillation and other technologies is expected to realize accurate and non-invasive preoperative staging biopsy of intracavitary tumors under any modality or modality combination conditions, which not only helps clinical decision-making and treatment planning, but also has important positive impacts on scientific research, resource management, and patient support. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the multimodal feature disentanglement-information filling-knowledge distillation network framework of the present invention;

[0042] Figure 2 It is a schematic diagram of the overall technical process of the present invention;

[0043] Figure 3 It is a schematic diagram of the process structure of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0045] Embodiment 1:

[0046] See also Figure 1-2 , a multi-modality intracavitary tumor imaging staging prediction method comprises the following steps:

[0047] A multimodal image standardization strategy was used to process the intracavitary tumor images, and the bounding boxes were manually constructed for the images to obtain processed multimodal tumor images, including main modality images and submodality images.

[0048] Extracting partial and complete original features of the main modality image and the sub-modality image respectively, intertwining the original features of the main modality image with the original features of the corresponding sub-modality image respectively, and obtaining the commonality representation matrix between multiple main image modalities and corresponding modalities;

[0049] The Encoder-Decoder module is introduced to extract features from multiple common representation matrices, and then merged with sub-modal image features to obtain multiple fusion features.

[0050] Multiple fusion features are distilled through the Teacher network to obtain the prediction results of intracavitary tumor staging;

[0051] At the same time, the Student network is designed, and the Teacher-Student feature loss function is constructed by using the original features of the subject modality image and the distilled-coupled features of the Teacher network to obtain the intracavitary tumor staging prediction results of the Student network;

[0052] Different staging prediction results of intracavitary tumors are obtained according to the Teacher and Student networks respectively. The distillation prediction loss function of KL divergence is introduced, and soft voting is used to fuse the staging prediction results of different intracavitary tumors to obtain the staging prediction results of intracavitary tumor images.

[0053] The partial and complete original features of the main modality image and the sub-modality image are extracted by an encoder at the network head;

[0054] The information entanglement is a feature untangling module based on the regional mutual information analysis strategy to extract the image features of the subject's nuclear magnetic resonance modality, obtain the mutual information through the image sub-region features with the most disease prediction ability corresponding to other modalities, and use the loss function of the feature untangling module to maximize the mutual information, thereby obtaining the commonality representation matrix between the subject's image modality and the corresponding modality.

[0055] The introduction of the Encoder-Decoder module to extract features from multiple common representation matrices and to merge and merge them with sub-modal image features includes:

[0056] The Encoder-Decoder module is introduced to extract features from the common representation matrix. The original features extracted by the head encoder in the submodal image are concatenated with the features extracted by the Encoder-Decoder module of the common representation matrix, and a 1×1×4 convolution is performed to obtain the fused features.

[0057] After the multiple fused features, Maxpooling operation is used to perform feature dimensionality reduction, and ReLU is used for feature activation. After multiple layers of convolution-pooling-activation operations, the network distills the key information and domain knowledge in the multimodal images that can effectively characterize the staging characteristics of intracavitary tumors.

[0058] The multimodal image standardization strategy includes data noise reduction, filtering, and grayscale standardization processing, so that images of different modalities show better consistency.

[0059] Multimodal intracavitary tumor imaging staging prediction system, including:

[0060] The feature extraction module uses a multimodal image standardization strategy to process the intracavitary tumor image and manually constructs a bounding box for the image to obtain a processed multimodal tumor image, including a main modal image and a submodal image; extracts local and complete original features of the main modal image and the submodal image respectively, and entangles the original features of the main modal image with the original features of the corresponding submodal image to obtain a common representation matrix between multiple main image modalities and corresponding modalities;

[0061] The feature fusion module introduces the Encoder-Decoder module to extract features from multiple common representation matrices, and then merges and fuses them with submodal image features to obtain multiple fused features. Multiple fused features are distilled through the Teacher network to obtain key information and domain knowledge of intracavitary tumor staging characteristics.

[0062] Construct the Teacher-Student feature loss function module, design the Student network, and use the original features of the subject modality image and the distilled-coupled features of the Teacher network to jointly construct the Teacher-Student feature loss function;

[0063] In the staging prediction module, the Teacher and Student networks obtain different staging prediction results of intracavitary tumors respectively, introduce the KL divergence distillation prediction loss function, and use soft voting to fuse the staging prediction results of different intracavitary tumors to obtain the staging prediction results of intracavitary tumor images.

[0064] A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the following steps:

[0065] A multimodal image standardization strategy was used to process the intracavitary tumor images, and the bounding boxes were manually constructed for the images to obtain processed multimodal tumor images, including main modality images and submodality images.

[0066] Extracting partial and complete original features of the main modality image and the sub-modality image respectively, intertwining the original features of the main modality image with the original features of the corresponding sub-modality image respectively, and obtaining the commonality representation matrix between multiple main image modalities and corresponding modalities;

[0067] The Encoder-Decoder module is introduced to extract features from multiple common representation matrices, and then merged with sub-modal image features to obtain multiple fusion features.

[0068] Multiple fusion features are distilled through the Teacher network to obtain the prediction results of intracavitary tumor staging;

[0069] At the same time, the Student network is designed, and the Teacher-Student feature loss function is constructed by using the original features of the subject modality image and the distilled-coupled features of the Teacher network to obtain the intracavitary tumor staging prediction results of the Student network;

[0070] Different staging prediction results of intracavitary tumors are obtained according to the Teacher and Student networks respectively. The distillation prediction loss function of KL divergence is introduced, and soft voting is used to fuse the staging prediction results of different intracavitary tumors to obtain the staging prediction results of intracavitary tumor images.

[0071] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the following steps: using a multimodal image standardization strategy to process an intracavitary tumor image, and manually constructing a bounding box for the image to obtain a processed multimodal tumor image, including a main modality image and a submodality image;

[0072] Extracting partial and complete original features of the main modality image and the sub-modality image respectively, intertwining the original features of the main modality image with the original features of the corresponding sub-modality image respectively, and obtaining the commonality representation matrix between multiple main image modalities and corresponding modalities;

[0073] The Encoder-Decoder module is introduced to extract features from multiple common representation matrices, and then merged with sub-modal image features to obtain multiple fusion features.

[0074] Multiple fusion features are distilled through the Teacher network to obtain the intracavitary tumor staging prediction results. At the same time, the Student network is designed to use the original features of the subject modality image and the distilled-coupled features of the Teacher network to jointly construct the Teacher-Student feature loss function to obtain the intracavitary tumor staging prediction results of the Student network.

[0075] Different staging prediction results of intracavitary tumors are obtained according to the Teacher and Student networks respectively. The distillation prediction loss function of KL divergence is introduced, and soft voting is used to fuse the staging prediction results of different intracavitary tumors to obtain the staging prediction results of intracavitary tumor images.

[0076] Embodiment 2:

[0077] See also Figure 1-2 Based on Example 1, the multi-modality intracavitary tumor imaging staging prediction method provided in this embodiment includes:

[0078] 1. Multimodal image preprocessing

[0079] Taking into account the influence of multimodal imaging parameters, grayscale range, noise and other factors, the present invention first uses a multimodal image standardization strategy during implementation, including data denoising, filtering, and grayscale standardization processing, so that images of different modalities show better consistency.

[0080] 2.BoundingBox

[0081] Considering that there may be a large amount of useless or even interfering information in the complex background area outside the luminal organs, which dilutes or weakens the information representation ability of the tumor area, thus having a serious impact on the staging prediction performance, the present invention manually constructs a rectangular bounding box BoundingBox based on the middle layer of the bladder image and automatically extends it to other layers to effectively remove the complex background area outside the lumen and reduce its interference with the feature extraction and staging prediction of luminal tumors.

[0082] 3. Multimodal Image Feature Disentanglement

[0083] First, after completing the multimodal image preprocessing, the MRI images of each modality are input into the HeadEncoder of the network head for local and global feature extraction. Considering that multimodal image features are entangled when jointly characterizing disease typing, it is expected to provide important information support for missing modal filling. This paper proposes a feature disentanglement module based on the regional mutual information analysis strategy, such as Figure 2 This module uses the image features extracted from the subject MRI modality to obtain the mutual information (MI) through the image sub-region features with the best disease prediction ability corresponding to other modalities, and uses the feature disentanglement module loss function LMI (as shown in formula (2)) to maximize MI during learning and training, thereby obtaining the common representation matrix between the subject image modality and the corresponding modality, as shown in formulas (3) and (4).

[0084] 4. Missing image modality information filling, knowledge distillation, and coupled prediction

[0085] 1. Missing Modality Information Imputation:

[0086] In multimodal data, each modality may contain different information, such as image data and text descriptions. Sometimes, due to various reasons, the information of a certain modality may be missing or incomplete. In order to make full use of the information of other modalities, filling techniques can be used to predict the data of the missing modality through the information learned from other modalities. This can be achieved by using various machine learning or deep learning methods, including autoencoders, generative adversarial networks (GANs), variational autoencoders (VAEs), etc. This helps to improve the completeness and availability of multimodal data.

[0087] 2. Knowledge Distillation:

[0088] Knowledge distillation is a model compression technique in which the knowledge of a complex model (teacher model) is transferred to a simplified model (student model). This is mainly achieved by using the relationship between the output and labels of the teacher model during the training process. The goal is to let the student model learn the "knowledge" of the teacher model, so that the student model is more lightweight while maintaining performance. Knowledge distillation is often used to deploy deep learning models in resource-constrained environments, such as on mobile devices.

[0089] 3. Coupled Prediction:

[0090] Coupled prediction involves processing two or more related prediction tasks at the same time. The two tasks may be related to each other, or their results may affect each other. By coupling these prediction tasks, the model can better exploit the relationship between them to improve the overall performance. For example, in image and text data, the prediction performance can be improved by predicting the image label and the text description of the image at the same time. This coupling can be achieved by designing the model structure of multi-task learning, allowing some layers or parameters to be shared, thereby prompting the model to learn related tasks together.

[0091] Considering that in practical applications, except for the main mode (taking mode 1 as an example), other modes (taking mode 2 as an example) may be missing, the present invention first further extracts features from the common representation matrix based on the Encoder-Decoder (ED) module to fill the information gap caused by the missing mode 2, such as Figure 2 shown.

[0092] Then, in the Teacher branch of modality 2, the original features extracted by HeadEncoder from all multimodal images are concatenated with the features extracted by the common representation matrix through the ED module, and a 1×1×4 convolution is performed to achieve feature fusion. After that, the Maxpooling operation is used for feature dimensionality reduction, and ReLU is used for feature activation. After multiple layers of convolution-pooling-activation operations, the network fully distills the key information and domain knowledge in multimodal images that can effectively characterize the staging characteristics of intracavitary tumors for prediction model construction.

[0093] At the same time, when the input only has the subject image modality (modality 1), a Student network is designed, and the original features extracted from the subject modality by the HeadEncoder are combined with the features distilled and coupled by the Teacher network to construct the Teacher-Student feature loss function (as shown in Formula (5)), thus realizing the interactive migration of multimodal information.

[0094] After continuous distillation, coupling, re-distillation, and re-coupling, the staging prediction of intracavitary tumors is finally achieved in the Teacher and Student networks. In order to effectively combine the classification advantages of different networks, the present invention proposes a distillation prediction loss based on KL divergence, as shown in formula (6). Finally, the prediction information of different networks is fused by soft voting to achieve accurate staging of intracavitary tumors.

[0095] In summary, compared with the prior art, the present invention has the following advantages:

[0096] Based on the existing technology, the present invention discloses a multimodal intracavitary tumor image intelligent analysis and prediction method, which is used for accurate preoperative staging diagnosis and non-invasive biopsy of luminal organ tumors such as bladder cancer, colorectal cancer, cervical cancer, and gastric cancer. It effectively solves the field pain points of existing multimodal image intelligent diagnosis methods that are difficult to apply when faced with modality missing and incomplete modality information, as well as the technical pain points of a large amount of information redundancy caused by the entanglement and overlap of modal features, thereby causing model overfitting.

[0097] The present invention innovatively introduces multimodal information disentanglement and domain knowledge deep distillation strategies to effectively separate common and specific information in multimodal MRI images, and use the common information to complete the missing modal information, effectively realizing the precise staging and non-invasive biopsy of intracavitary 2 tumors when any MRI image modality or multimodal combination input is used, which is expected to provide strong support for the best clinical treatment decisions.

[0098] It should be noted that, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or includes elements inherent to such process, method, article or device.

[0099] The above embodiments are merely examples of the present invention and do not limit the protection scope of the present invention. All designs that are the same or similar to the present invention fall within the protection scope of the present invention.

Claims

1. A multimodal intracavitary tumor imaging staging prediction method, characterized in that: The following steps are involved: A multimodal image standardization strategy was used to process the intracavitary tumor images, and the bounding boxes were manually constructed for the images to obtain processed multimodal tumor images, including main modality images and submodality images. Extracting local and complete original features of the main modality image and the sub-modality image respectively, wherein the local and complete original features of the main modality image and the sub-modality image are extracted by an encoder at the head of the network; information entanglement is performed on the original features of the main modality image and the original features of the corresponding sub-modality image respectively, wherein the information entanglement is a feature untangling module based on a regional mutual information analysis strategy to extract the image features of the main magnetic resonance imaging modality, and obtain mutual information through the image sub-region features with the most disease prediction ability corresponding to other modalities, and the loss function of the feature untangling module is used to maximize the mutual information, thereby obtaining a commonality representation matrix between multiple main image modalities and corresponding modalities; The Encoder-Decoder module is introduced to extract features from multiple common representation matrices, and then merged with sub-modal image features to obtain multiple fusion features. Multiple fusion features are distilled through the Teacher network to obtain the prediction results of intracavitary tumor staging; At the same time, the Student network is designed, and the Teacher-Student feature loss function is constructed by using the original features of the subject modality image and the distillation-coupled features of the Teacher network to obtain the intracavitary tumor staging prediction results of the Student network; Different staging prediction results of intracavitary tumors are obtained according to the Teacher and Student networks respectively. The distillation prediction loss function of KL divergence is introduced, and soft voting is used to fuse the staging prediction results of different intracavitary tumors to obtain the staging prediction results of intracavitary tumor images.

2. The multimodal intracavitary tumor imaging staging prediction method according to claim 1, characterized in that: The introduction of the Encoder-Decoder module to extract features from multiple common representation matrices and to merge and merge them with sub-modal image features includes: The Encoder-Decoder module is introduced to extract features from the common representation matrix. The original features extracted by the head encoder in the submodal image are concatenated with the features extracted by the Encoder-Decoder module of the common representation matrix, and a 1×1×4 convolution is performed to obtain the fused features.

3. The multimodal intracavitary tumor imaging staging prediction method according to claim 1, characterized in that: After the multiple fused features, Maxpooling operation is used to perform feature dimensionality reduction, and ReLU is used for feature activation. After multiple layers of convolution-pooling-activation operations, the network distills the key information and domain knowledge in the multimodal images that can effectively characterize the staging characteristics of intracavitary tumors.

4. The multimodal intracavitary tumor imaging staging prediction method according to claim 1, characterized in that: The multimodal image standardization strategy includes data noise reduction, filtering, and grayscale standardization processing.

5. A multi-modality intracavitary tumor imaging staging prediction system, characterized in that: include: The feature extraction module uses a multimodal image standardization strategy to process the intracavitary tumor image, and manually constructs a bounding box for the image to obtain a processed multimodal tumor image, including a main modality image and a submodality image; extracts local and complete original features of the main modality image and the submodality image respectively, and the local and complete original features of the main modality image and the submodality image are extracted by an encoder at the head of the network; the original features of the main modality image are information entangled with the original features of the corresponding submodality image respectively, and the information entanglement is based on the regional mutual information analysis strategy. The feature untangling module extracts the image features of the main magnetic resonance modality, and obtains the mutual information through the image sub-region features with the most disease prediction ability corresponding to other modalities, and the loss function of the feature untangling module is used to maximize the mutual information, thereby obtaining a commonality representation matrix between multiple main image modalities and corresponding modalities; The feature fusion module introduces the Encoder-Decoder module to extract features from multiple common representation matrices, and then merges and fuses them with submodal image features to obtain multiple fused features. Multiple fused features are distilled through the Teacher network to obtain key information and domain knowledge of intracavitary tumor staging characteristics. Construct a Teacher-Student feature loss function module to design the Student network. Use the original features of the subject modality image and the distilled-coupled features of the Teacher network to jointly construct the Teacher-Student feature loss function to obtain the intracavitary tumor staging prediction results of the Student network. The staging prediction module is used to obtain different staging prediction results of intracavitary tumors according to the Teacher and Student networks, introduce the KL divergence distillation prediction loss function, and use soft voting to fuse the staging prediction results of different intracavitary tumors to obtain the staging prediction results of intracavitary tumor images.

6. A computer device, characterized in that: The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps: A multimodal image standardization strategy was used to process the intracavitary tumor images, and the bounding boxes were manually constructed for the images to obtain processed multimodal tumor images, including main modality images and submodality images. Extracting local and complete original features of the main modality image and the sub-modality image respectively, wherein the local and complete original features of the main modality image and the sub-modality image are extracted by an encoder at the head of the network; information entanglement is performed on the original features of the main modality image and the original features of the corresponding sub-modality image respectively, wherein the information entanglement is a feature untangling module based on a regional mutual information analysis strategy to extract the image features of the main magnetic resonance imaging modality, and obtain mutual information through the image sub-region features with the most disease prediction ability corresponding to other modalities, and the loss function of the feature untangling module is used to maximize the mutual information, thereby obtaining a commonality representation matrix between multiple main image modalities and corresponding modalities; The Encoder-Decoder module is introduced to extract features from multiple common representation matrices, and then merged with sub-modal image features to obtain multiple fusion features. Multiple fusion features are distilled through the Teacher network to obtain the prediction results of intracavitary tumor staging; At the same time, the Student network is designed, and the Teacher-Student feature loss function is constructed by using the original features of the subject modality image and the distillation-coupled features of the Teacher network to obtain the intracavitary tumor staging prediction results of the Student network; Different staging prediction results of intracavitary tumors are obtained according to the Teacher and Student networks respectively. The distillation prediction loss function of KL divergence is introduced, and soft voting is used to fuse the staging prediction results of different intracavitary tumors to obtain the staging prediction results of intracavitary tumor images.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps: using a multimodal image standardization strategy to process the intracavitary tumor image, and manually constructing a bounding box for the image to obtain a processed multimodal tumor image, including a main modality image and a submodality image; Extracting local and complete original features of the main modality image and the sub-modality image respectively, wherein the local and complete original features of the main modality image and the sub-modality image are extracted by an encoder at the head of the network; information entanglement is performed on the original features of the main modality image and the original features of the corresponding sub-modality image respectively, wherein the information entanglement is a feature untangling module based on a regional mutual information analysis strategy to extract the image features of the main magnetic resonance imaging modality, and obtain mutual information through the image sub-region features with the most disease prediction ability corresponding to other modalities, and the loss function of the feature untangling module is used to maximize the mutual information, thereby obtaining a commonality representation matrix between multiple main image modalities and corresponding modalities; The Encoder-Decoder module is introduced to extract features from multiple common representation matrices, and then merged with sub-modal image features to obtain multiple fusion features. Multiple fusion features are distilled through the Teacher network to obtain the prediction results of intracavitary tumor staging; At the same time, the Student network is designed, and the Teacher-Student feature loss function is constructed by using the original features of the subject modality image and the distillation-coupled features of the Teacher network to obtain the intracavitary tumor staging prediction results of the Student network; Different staging prediction results of intracavitary tumors are obtained according to the Teacher and Student networks respectively. The distillation prediction loss function of KL divergence is introduced, and soft voting is used to fuse the staging prediction results of different intracavitary tumors to obtain the staging prediction results of intracavitary tumor images.

Images