Endometrial cancer molecular typing deep learning prediction algorithm based on multi-modal data fusion

Abstract

The invention discloses an endometrial cancer molecular typing deep learning prediction algorithm based on multi-modal data fusion, and relates to the technical field of endometrial cancer prediction. According to the method, depth features of pathological images and preoperative MRI are respectively extracted through improved ResNet 18 and 3D ResNet, space and channel features are dynamically fused by using a cross-modal attention module, and the problem of space-time dislocation is eliminated to a certain extent. And inputting the fused features into a causal gating time sequence convolutional network, extracting high-order semantic information through a cascaded residual block, finally outputting a molecular typing probability and an uncertainty score, and optimizing a model through joint loss. According to the method, the cross-modal deep learning model is provided, and the endometrial cancer typing precision is improved in combination with the attention mechanism and the causal sequential network. Pathology-image space-time dislocation is eliminated through dynamic soft alignment, molecular subtyping causal association is modeled through expansion convolution, and the defect that a traditional method has insufficient attention to tumor heterogeneity is overcome. According to the design, multi-modal characteristic representation is enhanced, and technical support is provided for precise diagnosis and treatment.

Description

Technical Field

[0001] The present invention belongs to the technical field of endometrial cancer prediction, and in particular relates to a deep learning prediction algorithm for molecular typing of endometrial cancer based on multimodal data fusion. Background Art

[0002] Molecular classification and prediction of endometrial cancer is a technology that divides the disease into different molecular subtypes by detecting specific gene mutations and molecular marker status in tumor tissues. Its core is to use molecular characteristics to analyze differences in tumor biological behavior, provide a basis for clinical risk stratification, and assist in the formulation of individualized diagnosis and treatment strategies.

[0003] Currently, molecular subtyping prediction for endometrial cancer relies primarily on multi-platform molecular detection technologies and biomarker combinations, based on the 2013 TCGA (The Cancer Genome Atlas) classification and the 2020 ProMisE (Proactive Molecular Risk Classifier for Endometrial Cancer) clinical applicability improvement plan. However, single-modality data cannot fully reflect the molecular characteristics of tumors. Recent attempts have been made to integrate time-series data, such as preoperative serological data. However, the time interval between intraoperative pathological examinations and preoperative sampling leads to temporal misalignment of multimodal data. Existing fusion methods, such as direct feature concatenation or attention mechanisms, cannot effectively align spatiotemporal features. While traditional dynamic time warping (DTW) algorithms can align single-modal time-series signals, their hard alignment constraints are non-differentiable and they do not consider the spatial texture characteristics of pathological images, resulting in insufficient cross-modal alignment accuracy. Furthermore, existing deep learning models lack a targeted focus on heterogeneous regions of the tumor, making it difficult to capture the causal relationship between temporal gradient changes in serological indicators and pathological morphology, impacting the clinical applicability of subtyping prediction.

[0004] To this end, we provide a deep learning prediction algorithm for molecular classification of endometrial cancer based on multimodal data fusion to solve the above problems. Summary of the Invention

[0005] The purpose of the present invention is to provide a deep learning prediction algorithm for molecular typing of endometrial cancer based on multimodal data fusion. Through a cross-modal attention fusion module and a causal gated temporal convolutional network, it solves the problems of insufficient spatiotemporal alignment accuracy of multimodal data and insufficient capture of tumor heterogeneity characteristics in the existing technology.

[0006] To solve the above technical problems, the present invention is implemented through the following technical solutions.

[0007] The present invention is a deep learning prediction algorithm for molecular typing of endometrial cancer based on multimodal data fusion, comprising the following steps:

[0008] Step a: Obtain intraoperative frozen section pathological images and preoperative single imaging examination (MRI) data. The pathological images are segmented into tumor interstitial regions and then the spatial feature matrix is ​​extracted by convolutional neural network. The imaging data is extracted by 3D convolutional network. The spatial feature matrix F of the pathological image is obtained. s ∈R 512×7×7 Generated by the modified ResNet 18 network, the ResNet 18 removes the fully connected layer and adjusts the last layer of convolution output channels to 512, the imaging feature matrix F img ∈R 512×7×7×7 Generated by 3D ResNet network;

[0009] Step b: Construct a cross-modal attention fusion module, and dynamically weight the fusion of pathological image features and imaging features through the spatial attention map. The calculation formula is:

[0010]

[0011] in represents channel splicing, SEBlock is the channel attention module;

[0012] Step c: Input the fused features into the cross-modal gated temporal convolutional network (CausalTCN). The network contains 4 layers of causal expansion convolution layers with expansion coefficients of 1, 2, 4, and 8, respectively. The convolution kernel size of each layer is 5, the number of output channels is 256, and the calculation formula of the cross-modal gated unit is:

[0013] G = ReLU(Conv1D(F fusion ))⊙σ(Conv1D(F s ))

[0014] Where σ is the Sigmoid function, ⊙ represents element-by-element multiplication;

[0015] Step d: Extract high-level semantic features through cascaded residual blocks and output molecular typing probability distribution P∈R 4 (corresponding to the four categories of TCGA) and uncertainty score U = Var(h final ), where h final ∈R 1024 is the final hidden layer feature;

[0016] Step e: Model training uses a joint loss function:

[0017] L=L CE (P,y)+0.1·U.

[0018] The present invention is further configured such that the pathological image preprocessing in step a includes using the Otsu threshold segmentation algorithm to locate the tumor interstitial area, cropping the region of interest with a size of 224×224 pixels, adding a channel attention module to the ResNet 18 network after the first convolution layer, and dynamically adjusting the weight parameters according to the imaging data.

[0019] The present invention is further configured such that, in the cross-modal attention fusion module, the spatial attention weight is generated by interactive learning of pathological image features and imaging features, and the calculation formula is:

[0020]

[0021] in Represents element-wise multiplication.

[0022] The present invention is further configured to: in step c, the spatial feature F s ∈R 512×49 Reshape into R 512×7×7 , the imaging feature F img ∈R 512×7×7×7 Dimensionality reduction to R 512×49 Then it is spliced ​​with the pathological features to obtain F fusion ∈R 1024×7×7 .

[0023] The present invention is further configured such that the calculation of the gate control unit in step c further includes the calculation of the spatial feature F s Perform channel compression, use 1×1 convolution to reduce the number of channels from 512 to 64, fuse the compressed spatial features with the imaging features, and generate the spatial modulation coefficient G∈R 256×49 .

[0024] The present invention is further configured such that the uncertainty quantification in step d is specifically implemented as the final hidden layer feature h final Perform 100 Monte Carlo samplings to calculate the variance of the prediction results, and trigger the manual review mechanism when U>0.15.

[0025] The present invention is further configured such that the imaging data is collected for a single examination within 7 days before the operation, and the time interval between the imaging data and the intraoperative pathological image does not exceed 72 hours.

[0026] The present invention is further configured such that the hole rate of the causal expansion convolution layer is dynamically adjusted according to the sampling interval, and the adjustment formula is:

[0027]

[0028] where t interval = 24 hours is the time interval between the association of imaging data and pathological images, and d is the base of the expansion coefficient.

[0029] The present invention is further configured such that the model training adopts a progressive alignment strategy, fixes the imaging feature encoding network in the first 10 epochs, trains only the pathological image feature extraction module, and jointly optimizes the multimodal fusion network in subsequent epochs.

[0030] The present invention is further configured such that if imaging data is missing during the intraoperative rapid diagnosis, imaging features are synthesized from the pathological image through a generative adversarial network (GAN) to complete the fusion input.

[0031] The present invention has the following beneficial effects.

[0032] 1. The present invention significantly improves the spatiotemporal alignment capability and feature representation accuracy of multimodal data through the collaborative design of the cross-modal attention fusion module and the causal gated temporal convolutional network. To address the spatiotemporal misalignment problem between intraoperative pathological images and preoperative imaging data, the cross-modal attention module dynamically learns the spatial correlation weights of pathological features and imaging features, and eliminates the dimensional conflict caused by hard splicing through a soft alignment mechanism. At the same time, the causal gated temporal convolutional network uses the temporal modeling capability of dilated convolution to capture the causal relationship between molecular typing and multimodal data, overcoming the defect of traditional methods that pay insufficient attention to tumor heterogeneous regions. The accuracy is improved in the TCGA four-classification task, providing highly reliable decision support for rapid intraoperative diagnosis.

[0033] 2. The present invention enhances the robustness and practicality of the algorithm in complex clinical scenarios through uncertainty quantification and adaptive completion mechanisms. The uncertainty score based on Monte Carlo sampling evaluates the confidence of the prediction results in real time. When the score exceeds the threshold, the manual review process is automatically triggered, effectively reducing the risk of misdiagnosis. In addition, to address the common problem of missing intraoperative imaging data, the generative adversarial network (GAN) is used to synthesize and complete imaging features from pathological images to ensure the integrity of the multimodal fusion input. This mechanism improves the prediction stability of the algorithm in scenarios with missing data or noise interference, significantly expanding its application potential in primary medical institutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments.

[0035] Figure 1 This is the main flow chart of the deep learning prediction algorithm for molecular classification of endometrial cancer based on multimodal data fusion.

[0036] Figure 2 This is a flowchart of feature extraction in the deep learning prediction algorithm for molecular classification of endometrial cancer based on multimodal data fusion.

[0037] Figure 3This is a flowchart of cross-modal fusion in the deep learning prediction algorithm for molecular classification of endometrial cancer based on multimodal data fusion.

[0038] Figure 4 This is a flowchart for model training and optimization in the deep learning prediction algorithm for molecular classification of endometrial cancer based on multimodal data fusion. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present invention will be described below in conjunction with the drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0040] Example 1

[0041] See also Figure 1-4 The present invention is a deep learning prediction algorithm for molecular typing of endometrial cancer based on multimodal data fusion, comprising the following steps:

[0042] Step a: Obtain intraoperative frozen section pathological images and preoperative single imaging examination (MRI) data. After segmenting the pathological images into tumor interstitial regions, the spatial feature matrix is ​​extracted through a convolutional neural network. The imaging data is extracted through a 3D convolutional network. The spatial feature matrix F of the pathological image is s ∈R 512×7×7 Generated by the modified ResNet 18 network, ResNet 18 removes the fully connected layer and adjusts the last layer of convolution output channels to 512, the imaging feature matrix F img ∈R 512×7×7×7 Generated by 3D ResNet network;

[0043] Step b: Construct a cross-modal attention fusion module, and dynamically weight the fusion of pathological image features and imaging features through the spatial attention map. The calculation formula is:

[0044]

[0045] in represents channel splicing, SEBlock is the channel attention module;

[0046] Step c: Input the fused features into the cross-modal gated temporal convolutional network (CausalTCN). The network contains 4 layers of causal expansion convolution layers, with expansion coefficients of 1, 2, 4, and 8, respectively. The convolution kernel size of each layer is 5, and the number of output channels is 256. The calculation formula of the cross-modal gated unit is:

[0047] G = ReLU(Conv1D(F fusion ))⊙σ(Conv1D(F s ))

[0048] Where σ is the Sigmoid function, ⊙ represents element-by-element multiplication;

[0049] Step d: Extract high-level semantic features through cascaded residual blocks and output molecular typing probability distribution P∈R 4 (corresponding to the four categories of TCGA) and uncertainty score U = Var(h final ), where h final ∈R 1024 is the final hidden layer feature;

[0050] Step e: Model training uses a joint loss function:

[0051] L=L CE (P,y)+0.1·U

[0052] The pathology image preprocessing in step a includes locating the tumor interstitial region using the Otsu threshold segmentation algorithm, cropping the region of interest with a size of 224×224 pixels, and adding a channel attention module after the first convolution layer of the ResNet 18 network. The weight parameters are dynamically adjusted based on the imaging data. In the cross-modal attention fusion module, the spatial attention weight is generated through the interactive learning of pathology image features and imaging features. The calculation formula is: in Indicates element-by-element multiplication. In step c, the spatial feature F s ∈R 512×49 Reshape into R 512×7×7 , the imaging feature F img ∈R 512×7×7×7 Dimensionality reduction to R 512×49 Then it is spliced ​​with the pathological features to obtain F fusion ∈R 1024×7×7 , the calculation of the gate control unit in step c further includes the spatial feature F s Perform channel compression, use 1×1 convolution to reduce the number of channels from 512 to 64, fuse the compressed spatial features with the imaging features, and generate the spatial modulation coefficient G∈R 256×49 The specific implementation of uncertainty quantification in step d is to calculate the final hidden layer feature h final Monte Carlo sampling was performed 100 times to calculate the variance of the prediction results. When U > 0.15, the manual review mechanism was triggered. The imaging data were collected for a single examination within 7 days before surgery, and the time interval with the intraoperative pathological images was no more than 72 hours. The dilation rate of the causal expansion convolution layer was dynamically adjusted according to the sampling interval. The adjustment formula is: where t interval=24 hours is the association time interval between imaging data and pathological images, d is the expansion coefficient base, and the model training adopts a progressive alignment strategy. In the first 10 epochs, the imaging feature encoding network is fixed, and only the pathological image feature extraction module is trained. In the subsequent epochs, the multimodal fusion network is jointly optimized. If the imaging data is missing during rapid intraoperative diagnosis, the imaging features are synthesized from the pathological image through the generative adversarial network (GAN) to complete the fusion input.

[0053] Specifically: Through the collaborative design of the cross-modal attention fusion module and the causal gated temporal convolutional network, the spatiotemporal alignment capability and feature representation accuracy of multimodal data are significantly improved. To address the spatiotemporal misalignment problem between intraoperative pathological images and preoperative imaging data, the cross-modal attention module dynamically learns the spatial correlation weights of pathological features and imaging features, and eliminates the dimensional conflict caused by hard splicing through a soft alignment mechanism. At the same time, the causal gated temporal convolutional network uses the temporal modeling capability of dilated convolution to capture the causal relationship between molecular typing and multimodal data, overcoming the defect of traditional methods that pay insufficient attention to tumor heterogeneity areas. The accuracy rate is improved in the TCGA four-classification task, which provides a fast solution for intraoperative diagnosis and treatment. Rapid Diagnosis provides highly reliable decision support. Through uncertainty quantification and adaptive completion mechanisms, the algorithm enhances its robustness and practicality in complex clinical scenarios. The uncertainty score based on Monte Carlo sampling evaluates the confidence of the prediction results in real time. When the score exceeds the threshold, the manual review process is automatically triggered, effectively reducing the risk of misdiagnosis. In addition, to address the common problem of missing intraoperative imaging data, the generative adversarial network (GAN) is used to synthesize and complete imaging features from pathological images to ensure the integrity of the multimodal fusion input. This mechanism improves the prediction stability of the algorithm in scenarios with missing data or noise interference, significantly expanding its application potential in primary medical institutions.

[0054] Example 2

[0055] See also Figure 1-4 , based on the first embodiment, when the imaging data is missing, the imaging data (CT / MRI) cannot be obtained in time due to equipment limitations during rapid diagnosis during surgery.

[0056] Start the Generative Adversarial Network (GAN) to synthesize imaging features from pathological images. Input the intraoperative pathological images into the pre-trained GAN generator and output the synthetic imaging feature matrix F syn ∈R 512×7×7×7 ;

[0057] Use cross-modal attention module to focus on pathological features F s With F syn Dynamic fusion, computing

[0058] F fusion =Conv3D(F s⊕SEBlock(F syn ));

[0059] Predict molecular typing through gated temporal convolutional networks and calculate the uncertainty score U;

[0060] If U>0.15, a manual review will be prompted and the result will be marked as “synthetic data assisted prediction”.

[0061] Example 3

[0062] See also Figure 1-4 , based on Example 1, in the scenario where the time interval of cross-modal data exceeds the limit, the time interval between preoperative imaging examination and intraoperative pathological image exceeds 72 hours (such as 96 hours), resulting in the misalignment of spatiotemporal features.

[0063] Dynamically adjust the expansion coefficient of the causal expansion convolution layer to enhance the time series modeling capability. Calculate the time interval t interval =96 hours, according to the formula Set the expansion coefficient sequence to 6, 12, 24, and 48;

[0064] Adaptive dilated convolutional layers are used in the CausalTCN network to capture long-term temporal dependencies;

[0065] Added time decay factor in fusion module

[0066]

[0067] When outputting the prediction results, if U>0.15, it will automatically mark "long interval data needs to be reviewed", which improves the accuracy.

[0068] Example 4

[0069] See also Figure 1-4 , based on the first embodiment, in the low-resolution data scenario of grassroots hospitals, the pathological images of grassroots medical institutions have low resolution (such as 112×112 pixels) and contain motion artifacts.

[0070] The preprocessing module was optimized and an anti-noise training strategy was introduced. Pathological images were preprocessed using bicubic interpolation upsampling to 224×224 pixels and non-local means denoising was applied.

[0071] Add adversarial training to the channel attention module of ResNet18, and the weight update formula is

[0072]

[0073] Introducing residual connections in the fusion module:

[0074] Random Gaussian noise (σ=0.1) and motion blur are added during training to enhance robustness.

[0075] Example 5

[0076] See also Figure 1-4 ,Based on the first embodiment, in a multi-center heterogeneous data fusion scenario,,when differences in imaging equipment across hospitals (such as,different CT slice thicknesses) lead to feature distribution shift.

[0077] Domain Adaptation is used to align the multi-center feature distribution. A domain classifier is connected after 3D ResNet, and adversarial training is used to minimize the domain difference loss L. DA ;

[0078] Weighted domain-sensitive features during multimodal fusion:

[0079]

[0080] where λ DA is the domain difference coefficient;

[0081] The training joint loss function is expanded to L = L CE +0.1·U+0.05·L DA ;

[0082] Automatically identify data sources and match pre-trained domain parameters during inference.

[0083] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all details in detail, nor do they limit the invention to only the specific implementation methods described. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can better understand and utilize the present invention.

Claims

1. A deep learning prediction algorithm for molecular typing of endometrial cancer based on multimodal data fusion, characterized by: The following steps are involved: Step a: Obtain intraoperative frozen section pathological images and preoperative single imaging examination (MRI) data. The pathological images are segmented into tumor interstitial regions and then the spatial feature matrix is ​​extracted by convolutional neural network. The imaging data is extracted by 3D convolutional network. The spatial feature matrix F of the pathological image is obtained. s ∈R 512×7×7 Generated by the modified ResNet18 network, the ResNet 18 removes the fully connected layer and adjusts the last layer of convolution output channels to 512, the imaging feature matrix F img ∈R 512×7×7×7 Generated by 3D ResNet network; Step b: Construct a cross-modal attention fusion module, and dynamically weight the fusion of pathological image features and imaging features through the spatial attention map. The calculation formula is: F fusion =Conv3D(F s ⊕SEBlock(F img )) Where ⊕ represents channel splicing, and SEBlock is the channel attention module; Step c: Input the fused features into the cross-modal gated temporal convolutional network (CausalTCN). The network contains 4 layers of causal expansion convolution layers with expansion coefficients of 1, 2, 4, and 8, respectively. The convolution kernel size of each layer is 5, the number of output channels is 256, and the calculation formula of the cross-modal gated unit is: G=ReLU(Conv1D(F fusion ))⊙σ(Conv1D(F s )) Where σ is the Sigmoid function, ⊙ represents element-by-element multiplication; Step d: Extract high-level semantic features through cascaded residual blocks and output molecular typing probability distribution P∈R 4 (corresponding to the four categories of TCGA) and uncertainty score U = Var(h final ), where h final ∈R 1024 is the final hidden layer feature; Step e: Model training uses a joint loss function: L=L CE (P,y)+0.1·U。 2. The deep learning prediction algorithm for molecular typing of endometrial cancer based on multimodal data fusion according to claim 1, characterized in that: The pathological image preprocessing in step a includes locating the tumor interstitial area using the Otsu threshold segmentation algorithm, cropping the region of interest with a size of 224×224 pixels, adding a channel attention module to the ResNet 18 network after the first convolution layer, and dynamically adjusting the weight parameters based on the imaging data.

3. The deep learning prediction algorithm for molecular typing of endometrial cancer based on multimodal data fusion according to claim 1, characterized in that: In the cross-modal attention fusion module, the spatial attention weight is generated through interactive learning of pathological image features and imaging features, and the calculation formula is: in Represents element-wise multiplication.

4. The deep learning prediction algorithm for molecular typing of endometrial cancer based on multimodal data fusion according to claim 1, characterized in that: In step c, the spatial feature F s ∈R 512×49 Reshape into R 512×7×7 , the imaging feature F img ∈R 512 ×7×7×7 Dimensionality reduction to R 512×49 Then it is spliced ​​with the pathological features to obtain F fusion ∈R 1024×7×7 .

5. The deep learning prediction algorithm for molecular typing of endometrial cancer based on multimodal data fusion according to claim 1, characterized in that: The calculation of the gate control unit in step c further includes the calculation of the spatial feature F s Perform channel compression, use 1×1 convolution to reduce the number of channels from 512 to 64, fuse the compressed spatial features with the imaging features, and generate the spatial modulation coefficient G∈R 256×49 .

6. The deep learning prediction algorithm for molecular typing of endometrial cancer based on multimodal data fusion according to claim 1, characterized in that: The specific implementation of uncertainty quantification in step d is to calculate the final hidden layer feature h final Perform 100 Monte Carlo samplings to calculate the variance of the prediction results, and trigger the manual review mechanism when U>0.

15.

7. The deep learning prediction algorithm for molecular typing of endometrial cancer based on multimodal data fusion according to claim 1, characterized in that: The imaging data were collected for a single examination within 7 days before the operation, and the time interval between the imaging data and the intraoperative pathological images was no more than 72 hours.

8. The deep learning prediction algorithm for molecular typing of endometrial cancer based on multimodal data fusion according to claim 1, characterized in that: The dilation rate of the causal expansion convolution layer is dynamically adjusted according to the sampling interval, and the adjustment formula is: where t interval = 24 hours is the time interval between the association of imaging data and pathological images, and d is the base of the expansion coefficient.

9. The deep learning prediction algorithm for molecular typing of endometrial cancer based on multimodal data fusion according to claim 1, characterized in that: The model training adopts a progressive alignment strategy, fixing the imaging feature encoding network in the first 10 epochs and training only the pathological image feature extraction module, and jointly optimizing the multimodal fusion network in subsequent epochs.

10. The deep learning prediction algorithm for molecular typing of endometrial cancer based on multimodal data fusion according to claim 1, characterized in that: If imaging data is missing during the intraoperative rapid diagnosis, imaging features are synthesized from pathological images through a generative adversarial network (GAN) to complete the fusion input.

Images