Auxiliary method for prognosis prediction of renal clear cell carcinoma
By constructing a multimodal prediction network and utilizing an improved Transformer module and a sparse Bayesian extreme learning machine, the heterogeneity and redundancy problems in multimodal learning were solved, enabling accurate prognostic assessment of clear cell renal cell carcinoma and early identification of high-risk patients.
Patent Information
- Application Number
- CN202511083558.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-04
- Publication Date
- 2025-11-14
AI Technical Summary
Existing deep learning-based multimodal learning methods suffer from model generalization ability due to heterogeneity between modalities and data redundancy, making it difficult to accurately assess individualized prognostic information for clear cell renal cell carcinoma.
A prognostic prediction network for clear cell renal cell carcinoma was constructed. Multimodal features were extracted using CT feature extractors and pathological feature extractors. An improved Transformer module, multilayer perceptron, and cross-attention module were used to capture individual knowledge, common knowledge, and derived knowledge. The network was then combined with a sparse Bayesian extreme learning machine for adaptive fusion and prediction.
It improves the accuracy of prognostic assessment of clear cell renal cell carcinoma and the generalization ability of the model, enhances the early identification of high-risk patients, and assists in clinical diagnosis.
Smart Images
Figure CN120954751A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and medical imaging technology, specifically to an auxiliary method for predicting the prognosis of clear cell renal cell carcinoma. Background Technology
[0002] Clear cell renal cell carcinoma (ccRCC) is one of the most common types of renal cell carcinoma. Its high heterogeneity results in significant diversity in molecular characteristics and histological morphology, accompanied by high invasiveness and metastatic potential, leading to generally poor prognosis. Although surgical resection is the primary treatment for ccRCC, approximately 20%–30% of patients experience local recurrence after surgery, and treatment after recurrence is more difficult, severely impacting patient survival and quality of life. Traditional prognostic assessment methods, such as the TMN staging system based on clinicopathological features, play an important role in tumor staging and treatment decisions. However, due to the high heterogeneity of tumors, these systems struggle to comprehensively and accurately assess individualized prognostic information. Therefore, developing more accurate prognostic prediction methods capable of identifying high-risk patients early is crucial for improving treatment outcomes and patient survival.
[0003] In recent years, the widespread application of deep learning technology in medical image analysis has provided a new perspective for cancer prognosis prediction. Especially when processing tumor imaging data, deep learning can automatically extract latent features and improve prediction accuracy. With the development of multimodal data analysis technology, deep learning-based multimodal learning methods offer new possibilities for comprehensively utilizing multi-source data such as CT and pathological images, enabling a more comprehensive capture of tumor characteristics. Currently, common multimodal fusion methods simply integrate features from different modalities, but this approach ignores the potential relationships and interactions between different modalities, thus limiting the effectiveness of information sharing and feature fusion. To overcome this problem, many studies have proposed innovative multimodal information fusion methods to improve diagnostic accuracy and model stability. However, heterogeneity between modalities and data redundancy remain challenges in multimodal learning, necessitating further optimization of fusion strategies and improvement of model generalization capabilities. Summary of the Invention
[0004] The present invention addresses the problem that existing deep learning-based multimodal learning methods suffer from poor model generalization ability due to heterogeneity between modalities and data redundancy, and provides an auxiliary method for predicting the prognosis of clear cell renal cell carcinoma.
[0005] To solve the above problems, the present invention is achieved through the following technical solution:
[0006] A prognostic prediction method for clear cell renal cell carcinoma includes the following steps:
[0007] Step 1: Construct a prognostic prediction network for clear cell renal cell carcinoma;
[0008] The aforementioned prognostic prediction network for clear cell renal cell carcinoma consists of a CT feature extractor, a pathological feature extractor, a CT Transformer module, a pathological Transformer module, a CT personalized knowledge encoder, a pathological personalized knowledge encoder, a common knowledge encoder, a derived knowledge encoder, a CT fully connected layer, a pathological fully connected layer, a common fully connected layer, a derived fully connected layer, a normalization layer, a CT multiplier, a pathological multiplier, a common multiplier, a derived multiplier, an adder, and an extreme learning machine.
[0009] The input of the CT feature extractor is used as the input of the CT image to the prognostic prediction network for clear cell renal cell carcinoma. The output of the CT feature extractor is connected to the input of the CT Transformer module. The output of the CT Transformer module is connected to the input of the CT individual knowledge encoder, one input of the common knowledge encoder, and one input of the derived knowledge encoder.
[0010] The input of the pathological feature extractor is used as the input of the pathological image to the prognostic prediction network for clear cell renal cell carcinoma. The output of the pathological feature extractor is connected to the input of the pathological Transformer module. The output of the pathological Transformer module is connected to the input of the pathological individual knowledge encoder, another input of the common knowledge encoder, and another input of the derived knowledge encoder.
[0011] One output of the CT personalized knowledge encoder is connected to the CT input of the normalization layer via the CT fully connected layer, and the CT output of the normalization layer is connected to one input of the CT multiplier; the other output of the CT personalized knowledge encoder is connected to the other input of the CT multiplier.
[0012] One output of the pathological personalized knowledge encoder is connected to the pathological input of the normalization layer via a fully connected pathological layer. The pathological output of the normalization layer is connected to one input of the pathological multiplier. The other output of the pathological personalized knowledge encoder is connected to the other input of the pathological multiplier.
[0013] One output of the common knowledge encoder is connected to the common input of the normalization layer via a common fully connected layer; the common output of the normalization layer is connected to one input of the common multiplier; the other output of the common knowledge encoder is connected to the other input of the common multiplier.
[0014] One output of the derived knowledge encoder is connected to the derived input of the normalization layer via a derived fully connected layer, and the derived output of the normalization layer is connected to one input of the derived multiplier; the other output of the derived knowledge encoder is connected to the other input of the derived multiplier.
[0015] The outputs of the CT multiplier, the pathology multiplier, the common multiplier, and the derived multiplier are all connected to the input of the adder. The output of the adder is connected to the input of the extreme learning machine. The output of the extreme learning machine serves as the prediction output of the renal clear cell carcinoma prognostic prediction network.
[0016] Step 2: Collect sample datasets of CT and pathological images, and use the sample datasets of CT and pathological images to train the prognostic prediction network for clear cell renal cell carcinoma to obtain the prognostic prediction model for clear cell renal cell carcinoma.
[0017] Step 3: Obtain the patient's CT images and pathological images, and input them into the renal clear cell carcinoma prognostic prediction model for prediction to obtain the prognostic prediction results of renal clear cell carcinoma.
[0018] In the above scheme, the CT feature extractor is the ResNet18 feature extractor, and the pathological feature extractor is the UNI feature extractor.
[0019] In the above scheme, both the CT Transformer module and the pathology Transformer module are improved Transformer modules, which include two multi-head attention modules, two multipliers, and a pyramid position code generator. The input of the first multi-head attention module is used as the input of the improved Transformer module. The input and output of the first multi-head attention module are simultaneously connected to the input of the first multiplier. The output of the first multiplier is connected to the input of the pyramid position code generator. The output of the pyramid position code generator is connected to the input of the second multi-head attention module. The input and output of the second multi-head attention module are simultaneously connected to the input of the second multiplier. The output of the second multiplier is used as the output of the improved Transformer module.
[0020] In the above scheme, both the CT personalized knowledge encoder and the pathology personalized knowledge encoder are multilayer perceptrons, while both the common knowledge encoder and the derived knowledge encoder are cross-attention modules.
[0021] In the above scheme, the extreme learning machine is a sparse Bayesian extreme learning machine.
[0022] In the above scheme, the loss function used when training the prognostic prediction network for clear cell renal cell carcinoma is:
[0023] L (i) total =Δl (i) knowledge +βl (i) patient
[0024] l(i) knowledge =W(F R (i) ,R (i) )+W(F R (i) C (i) )+W(F P (i) ,P (i) )+W(F P (i) C (i) )-W(F R (i) ,P (i) )-W(F P (i) ,R (i) )-W(F R (i) E (i) )-W(F P (i) E (i) )
[0025]
[0026] In the formula, L (i) total Let l represent the total loss of the i-th sample group. (i) knowledge Let l represent the knowledge-level loss function for the i-th sample group. (i) patient Let F represent the sample-level loss function for the i-th sample group, α represent the knowledge-level loss weight, and β represent the sample-level loss weight; W(·) represents the Wasserstein distance, and d(·) represents the Euclidean distance; R (i) F represents the intramodal representation of the i-th sample group in CT mode. P (i) R represents the pathological modality representation of the i-th sample group; (i) Let P represent the CT personality knowledge of the i-th sample group. (i) Let C represent the pathological individual knowledge of the i-th sample group. (i) E represents the common knowledge of the i-th group of samples. (i) Represents the derived knowledge of the i-th sample; R + P represents the set of CT-specific knowledge for samples in the CT and pathology image dataset that match the category of the i-th sample group. + C represents the set of pathological individual knowledge for samples in the CT and pathological image sample dataset that match the category of the i-th sample group. +E represents the set of common knowledge among samples in the CT and pathology image sample dataset that belong to the same category as the samples in the i-th group. + R represents the set of derived knowledge for samples in the sample dataset of CT and pathology images that belong to the same category as the samples in the i-th group; - P represents the set of CT-specific knowledge for samples in the CT and pathology image dataset that do not belong to the category of the i-th group of samples. - C represents the set of pathological individual knowledge for samples in the CT and pathological image sample dataset that do not belong to the same category as the samples in the i-th group. - E represents the set of common knowledge among samples in the CT and pathology image sample dataset that do not belong to the same category as the i-th group of samples. - This represents the set of derived knowledge for samples in the CT and pathology image sample dataset that do not belong to the same category as the i-th group of samples; i = 1, 2, ..., n, where n represents the number of samples in the CT and pathology image sample dataset.
[0027] Compared with existing technologies, this invention extracts three types of knowledge components from CT and pathological images through four encoders for multimodal representation extraction and multimodal knowledge acquisition: individual knowledge existing only in a single modality, common knowledge existing across multiple modalities, and derived knowledge obtained through intermodal interactions. An adaptive fusion strategy is then used to dynamically assign optimal weights to different types of knowledge, thereby obtaining optimal fusion features. Finally, an extreme learning machine is used to predict the central features of each feature to obtain prediction results for ccRCC patients. During training, guided losses at the knowledge and sample levels enhance the discriminative and distinguishable nature of different knowledge. This invention effectively improves the prognostic assessment performance of ccRCC, thereby achieving the effect of assisted diagnosis. Attached Figure Description
[0028] Figure 1 This is a schematic diagram of an auxiliary method for predicting the prognosis of clear cell renal cell carcinoma.
[0029] Figure 2 A schematic diagram illustrating the improved structure of the Transformer module. Detailed Implementation
[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific examples and the accompanying drawings.
[0031] See Figure 1 A prognostic aid method for clear cell renal cell carcinoma includes the following steps:
[0032] Step 1: Construct a prognostic prediction network for clear cell renal cell carcinoma.
[0033] Considering that CT images can provide macroscopic imaging information of the kidney region, such as tumor location, morphological characteristics, and the presence of metastasis, they have been proven to be effective in predicting the survival and recurrence risk of clear cell renal cell carcinoma (ccRCC) patients. Pathological images play a crucial role in revealing the tumor microenvironment (TME) and the molecular characteristics of tumor cells; deep learning analysis of pathological images can provide important evidence for tumor classification, staging, and recurrence prediction. Therefore, this invention aims to integrate the features of both CT and pathological images to achieve prognostic prediction of clear cell renal cell carcinoma. To this end, the prognostic prediction network for clear cell renal cell carcinoma designed in this invention consists of a CT feature extractor, a pathological feature extractor, a CT Transformer module, a pathological Transformer module, a CT personalized knowledge encoder, a pathological personalized knowledge encoder, a common knowledge encoder, a derived knowledge encoder, a CT fully connected layer, a pathological fully connected layer, a common fully connected layer, a derived fully connected layer, a normalization layer, a CT multiplier, a pathological multiplier, a common multiplier, a derived multiplier, an adder, and an extreme learning machine.
[0034] The input to the CT feature extractor serves as the input to the CT image for the renal clear cell carcinoma prognostic prediction network. The output of the CT feature extractor is connected to the input of the CT Transformer module, which in turn connects to the input of the CT personalized knowledge encoder, one input of the common knowledge encoder, and one input of the derived knowledge encoder. Similarly, the input to the pathology feature extractor serves as the input to the pathology image for the renal clear cell carcinoma prognostic prediction network. The output of the pathology feature extractor is connected to the input of the pathology Transformer module, which in turn connects to the input of the pathology personalized knowledge encoder, another input of the common knowledge encoder, and another input of the derived knowledge encoder. One output of the CT personalized knowledge encoder is connected to the CT input of the normalization layer via a fully connected CT layer. The CT output of the normalization layer is then connected to one input of the CT multiplier. The other output of the CT personalized knowledge encoder is connected to the other input of the CT multiplier. Likewise, one output of the pathology personalized knowledge encoder is connected to the pathology input of the normalization layer via a fully connected pathology layer. The pathology output of the normalization layer is then connected to one input of the pathology multiplier. The other output of the pathology personalized knowledge encoder is connected to the other input of the pathology multiplier. One output of the common knowledge encoder is connected to the common input of the normalization layer via a common fully connected layer. The common output of the normalization layer is connected to one input of the common multiplier. The other output of the common knowledge encoder is connected to the other input of the common multiplier. One output of the derived knowledge encoder is connected to the derived input of the normalization layer via a derived fully connected layer. The derived output of the normalization layer is connected to one input of the derived multiplier. The other output of the derived knowledge encoder is connected to the other input of the derived multiplier. The outputs of the CT multiplier, the pathology multiplier, the common multiplier, and the derived multiplier are simultaneously connected to the input of the adder. The output of the adder is connected to the input of the extreme learning machine. The output of the extreme learning machine serves as the prediction output of the renal clear cell carcinoma prognostic prediction network.
[0035] The aforementioned prognostic prediction network for clear cell renal cell carcinoma comprises the following four stages:
[0036] (1) Multimodal characterization extraction stage
[0037] The multimodal characterization extraction stage uses a feature extractor and a Transformer module to extract features and intramodal characterizations from CT and pathology.
[0038] CT images are processed by a CT feature extractor to obtain CT features. In this embodiment, the CT feature extractor is a ResNet (Residual Network) 18 feature extractor. ResNet 18 is a deep convolutional neural network, mainly composed of 18 layers, including 16 convolutional layers and 2 fully connected layers. Its main characteristics are extremely strong feature extraction capabilities and low computational complexity.
[0039] Pathological features are obtained from pathological images through a pathological feature extractor. In this embodiment, the pathological feature extractor is a large-scale universal self-supervised pathological visual encoder (UNI) based on ViT-Large. UNI uses ViT-Large (ViT-L) as its backbone network, which consists of 24 transformer blocks, each containing 16 self-attention heads. The hidden layer dimension is 1024, the MLP dimension is 4096, and the number of parameters is approximately 304 million. The patch size is set to 16x16, and its main feature is that it can guide the network to focus more on key pathological regions and suppress redundant information.
[0040] CT features are processed by the CT Transformer module to obtain the CT intramodal representation F. R Pathological features are processed by the Pathology Transformer module to obtain the pathological modality representation F. P The CT Transformer module and the Pathology Transformer module are both improved Transformer modules, which include two multi-head attention modules, two multipliers, and a pyramid positional encoding generator, such as... Figure 2 As shown. The input of the first multi-head attention module is used as the input of the improved Transformer module. The input and output of the first multi-head attention module are simultaneously connected to the input of the first multiplier. The output of the first multiplier is connected to the input of the pyramid position encoding generator. The output of the pyramid position encoding generator is connected to the input of the second multi-head attention module. The input and output of the second multi-head attention module are simultaneously connected to the input of the second multiplier. The output of the second multiplier is used as the output of the improved Transformer module.
[0041] (2) Multimodal knowledge acquisition stage
[0042] The multimodal knowledge acquisition stage employs four encoders to capture different knowledge components present in CT and pathology multimodal data: modality-specific individual knowledge, intermodal shared common knowledge, and derived knowledge generated from modal interactions. Individual knowledge refers to features existing only in a single modality, reflecting the specific information of the CT or pathology modality. Common knowledge consists of features shared between two modalities, revealing similar patterns between CT and pathology images. Derived knowledge is new knowledge obtained through intermodal interactions, capturing potential complementary information between CT and pathology images. Modeling with different encoders effectively mitigates the conflicts and information loss caused by modal heterogeneity differences in multimodal data fusion.
[0043] Significant data heterogeneity exists between CT and pathology images, and high-dimensional image data often introduces a large amount of redundant information into the model, easily leading to fitting problems and limiting the model's diagnostic performance. Therefore, this invention delves into the potential knowledge information between modalities. Four independently constructed encoders capture the unique knowledge within CT and pathology modalities, the common knowledge shared between CT and pathology modalities, and new insights generated during the interaction between CT and pathology modalities—i.e., derived knowledge. This comprehensive and clear division of knowledge components effectively alleviates the conflicts and information loss caused by heterogeneity in multimodal data fusion, thereby improving the model's performance, insights, and decision-making capabilities in cancer prognosis.
[0044] Both the CT Personalized Knowledge Encoder and the Pathology Personalized Knowledge Encoder are multilayer perceptrons (MLPs). A multilayer perceptron is a feedforward artificial neural network composed of multiple neurons (neural nodes) arranged in a hierarchical structure, including an input layer, hidden layers, and an output layer. Neurons between layers are connected by weights, and information propagates sequentially from the input layer to the output layer without feedback connections.
[0045] Enc uses MLP to build a modal personality knowledge encoder R and Enc P This allows for the acquisition of individual knowledge within CT and pathology modalities, with a focus on knowledge that exists only within a single modality. The formula is expressed as follows:
[0046] R = Enc R (F R )
[0047] P = Enc P (F P )
[0048] Where R represents CT personality knowledge, F R Indicates the intramodal characterization in CT, P represents individual pathological knowledge, and F represents the intramodal characterization in CT. P This represents the internal characterization of a pathological modality.
[0049] Both the common knowledge encoder and the derived knowledge encoder are cross-attention modules. A cross-attention module is a special attention mechanism module used to handle the relationship between two different sequences or modalities. Unlike self-attention modules, cross-attention modules allow one sequence (query sequence) to focus on information in another sequence (key-value sequence) through an attention mechanism.
[0050] A multimodal common knowledge encoder Enc was constructed based on the cross-attention mechanism. C and derived knowledge encoder Enc ETo acquire common and derived knowledge generated during multimodal interaction, cross-modal attention weights for CT and pathology are generated to achieve cross-modal interaction. These encoders not only acquire common knowledge between modalities but also capture derived knowledge generated during cross-modal interaction. The formula is expressed as follows:
[0051] C = Enc C (F P ,F R )
[0052] E = Enc E (F P ,F R )
[0053] Where A represents the common attention matrix, C represents common knowledge, E represents derived knowledge, and F represents... R Indicates intramodal characterization in CT, F P This represents the internal characterization of a pathological modality.
[0054] It is worth noting that, although the common knowledge encoder Enc C and derived knowledge encoder Enc E While both are constructed in the same way, their attention weights differ. The common knowledge encoder obtains common knowledge in multimodal data by optimizing weights, while the derived knowledge encoder obtains unique derived knowledge in multimodal data by optimizing weights.
[0055] Based on the above four encoders, F is characterized from CT and pathological modalities. P ,F R The study obtained unique knowledge R specific to the CT modality, unique knowledge P specific to the pathology modality, common knowledge C between CT and pathology, and derived knowledge E generated during the interaction between CT and pathology.
[0056] (3) Multimodal knowledge adaptive fusion stage
[0057] The multimodal knowledge adaptive fusion stage constructs an adaptive fusion strategy based on a gating mechanism through a fully connected layer and a normalization layer. By dynamically adjusting the weights of different knowledge components, it achieves efficient fusion of multimodal knowledge.
[0058] In practical applications, CT and pathology data from different samples exhibit significant differences in information quality, feature distribution, and task relevance, resulting in inconsistent contribution weights of various knowledge components to the task. Fixed weighting or simple splicing fusion methods often fail to adapt to these differences, potentially leading to information redundancy or weakening of key features. To address this issue, this invention designs an adaptive fusion strategy based on a gating mechanism in the knowledge fusion stage. This strategy dynamically learns the attention weights of each knowledge component and adaptively adjusts the importance of different knowledge components according to task requirements, thereby achieving effective multimodal feature fusion. The formula is expressed as follows:
[0059]
[0060] Where, knowledge component k∈{R,P,C,E}, X k It is a characteristic of knowledge component k, W k It is the linear transformation weight matrix of knowledge component k, b k It is the bias term of knowledge component k, ω k It is the knowledge weight of knowledge component k before normalization, α k F is the knowledge weight after normalization of knowledge component k. fused It is the final fusion feature.
[0061] (4) Classification stage
[0062] In the classification stage, an extreme learning machine classifier is used to predict fused features, which improves the model's ability to capture nonlinear features and enhances its robustness to task noise.
[0063] In this embodiment, the Extreme Learning Machine (ELM) employs a Sparse Bayesian Extreme Learning Machine (B-ELM) classifier to classify the fused features, further improving classification performance. As a classifier, the Sparse Bayesian Extreme Learning Machine not only effectively handles multimodal features but also adjusts the classifier's parameters through Bayesian optimization methods, making the model more adaptable to different datasets and sample groups, ultimately providing accurate prognostic predictions.
[0064] To further improve the model's predictive performance for patients with clear cell renal cell carcinoma, this invention introduces B-ELM, based on the fusion feature F fusedThis classifier effectively captures complex nonlinear feature relationships and, within a Bayesian framework, quantifies prediction uncertainty by probabilistically modeling hidden layer parameters, thus enhancing the model's robustness to task noise. Addressing the difference in feature distribution between relapsed and non-relapsed samples, B-ELM leverages its rapid learning and powerful generalization capabilities to further optimize the model's performance in distinguishing different categories of samples. B-ELM constrains the model by introducing the L1 norm into the ELM optimization solution, thereby achieving a sparse solution. The optimization formula is shown below:
[0065]
[0066] Where E(·) represents the expected loss, θ represents the weights between the hidden and output layers, b represents the bias between the hidden and output layers; argmin(·) represents minimization; l represents the true label of the sample; X represents the output of the hidden layer, which is fixed after the connection weights of the input and hidden layers and the bias of the hidden layer are randomly determined; θ m λ represents the weight of the m-th neuron, L is the number of neurons in the hidden layer; λ > 0 represents the coefficient of the constraint term; ||·||2 represents the L2 norm, which is used to calculate the difference between the true label l and the model prediction value Xθ; ||·||1 represents the L1 norm, which is used to prevent overfitting through regularization.
[0067] Step 2: Collect sample datasets of CT and pathological images, and use these datasets to train a prognostic prediction network for clear cell renal cell carcinoma to obtain a prognostic prediction model for clear cell renal cell carcinoma.
[0068] The CT and pathology image sample dataset comprises n sets of samples, each consisting of one CT image and one pathology image from the same patient. During training, the CT and pathology image sample datasets are fed into the ccRCC prognostic prediction network. First, deep feature representations are extracted from the CT and pathology images of each sample set using CT and pathological characteristics, respectively. Then, multimodal knowledge acquisition is performed on the extracted CT and pathology feature representations to achieve in-depth mining and redundancy removal of multimodal information. To improve the model's discriminative and generalization abilities, guidance is provided at both the knowledge and sample levels. Next, an adaptive fusion strategy is employed to fuse multimodal knowledge to obtain the optimal fused features. Finally, an extreme learning machine is used to classify the fused features, thereby achieving accurate prediction of the prognostic status of ccRCC samples and providing effective support for clinical auxiliary diagnosis.
[0069] To improve model training performance, this invention designs a distance-based guided loss function, which utilizes guided losses at both the knowledge level and the sample level to enhance the discriminative feature representation of various types of knowledge.
[0070] At the knowledge level, all knowledge components originate from the source modality representation F. P ,F R Knowledge is acquired and maintains a unique connection with each modality. Personalized knowledge exists only within its corresponding modality and does not overlap with other modalities; common knowledge is the intersection of knowledge from two modalities, reflecting shared characteristics between modalities; derived knowledge is new insights gained about the task during modal interaction and cannot be obtained through any single modality. Based on these differences in knowledge characteristics, this invention uses Wasserstein distance to calculate the probability distribution differences between various types of knowledge and modal feature representations, and designs personalized guidance strategies accordingly. The loss at the knowledge level is... (i) knowledge for:
[0071] l (i) knowledge =
[0072] W(F R (i) ,R (i) )+W(F R (i) C (i) )+W(F P (i) ,P (i) )+W(F P (i) C (i) )
[0073] -W(F R (i) ,P (i) )-W(F P (i) ,R (i) )-W(F R (i) E (i) )-W(F P (i) E (i) )
[0074] Among them, l (i) knowledge Let W(·) represent the knowledge-level loss function for the i-th sample group; F represents the Wasserstein distance. R (i) F represents the intramodal representation of the i-th sample group in CT mode. P (i) R represents the pathological modality representation of the i-th sample group; (i) Let P represent the CT personality knowledge of the i-th sample group. (i) Let C represent the pathological individual knowledge of the i-th sample group. (i)E represents the common knowledge of the i-th group of samples. (i) This represents the derived knowledge of the i-th sample; i = 1, 2, ..., n, where n represents the number of samples in the CT and pathological image sample dataset.
[0075] Loss function l (i) knowledge W(·) is used to guide knowledge components with positive or negative directions, emphasizing the correlation between knowledge and target modal features while weakening irrelevant or interfering influences, thereby effectively improving the representation ability of various types of knowledge. It is worth noting that this invention will use F... R (i) and F P (i) By stripping the gradient from the backpropagation process, it is ensured that the gradient does not participate in backpropagation, thus enabling effective guidance of knowledge.
[0076] At the sample level, the feature representations of recurring samples are generally similar to other recurring samples, but significantly different from non-recurring samples. This assumption aligns with the core idea of contrastive learning, and existing research has shown that networks incorporating contrastive learning strategies can effectively extract discriminative feature representations. Therefore, this invention proposes a queue-based contrastive learning method to enhance the model's discriminative ability and generalization performance. For derived knowledge of recurring samples, by reducing its distance from recurring samples in the derived knowledge queue while increasing its distance from non-recurring samples, its features are made closer to the features of samples of the same category in the derived knowledge queue, while distancing themselves from the features of samples of different categories. This strategy effectively improves the representational ability of derived knowledge and enhances the model's ability to distinguish specific categories. The specific loss design is as follows:
[0077]
[0078] In the formula, d(·) represents the Euclidean distance; R (i) Let P represent the CT personality knowledge of the i-th sample group. (i) Let C represent the pathological individual knowledge of the i-th sample group. (i) E represents the common knowledge of the i-th group of samples. (i) Represents the derived knowledge of the i-th sample; R + P represents the set of CT-specific knowledge for samples in the CT and pathology image dataset that match the category of the i-th sample group. + C represents the set of pathological individual knowledge for samples in the CT and pathological image sample dataset that match the category of the i-th sample group. + E represents the set of common knowledge among samples in the CT and pathology image sample dataset that belong to the same category as the samples in the i-th group. + R represents the set of derived knowledge for samples in the sample dataset of CT and pathology images that belong to the same category as the samples in the i-th group;- P represents the set of CT-specific knowledge for samples in the CT and pathology image dataset that do not belong to the category of the i-th group of samples. - C represents the set of pathological individual knowledge for samples in the CT and pathological image sample dataset that do not belong to the same category as the samples in the i-th group. - E represents the set of common knowledge among samples in the CT and pathology image sample dataset that do not belong to the same category as the i-th group of samples. - This represents the set of derived knowledge for samples in the CT and pathology image sample dataset that do not belong to the same category as the samples in the i-th group.
[0079] The process of guiding other knowledge at the sample level is consistent with that of derived knowledge; therefore, the loss at the sample level is... patient for:
[0080] l (i) patient =l R (i) +l P (i) +l C (i) +l E (i)
[0081] Combining knowledge- and sample-level guided learning, the knowledge-guided learning loss can be expressed as:
[0082] L (i) total =αl (i) knowledge +βl (i) patient
[0083] Among them, L (i) total Let l represent the total loss of the i-th sample group. (i) knowledge Let l represent the knowledge-level loss function for the i-th sample group. (i) patient Let represent the sample-level loss function of the i-th group of samples, α represent the knowledge-level loss weight, and β represent the sample-level loss weight, which are used to adjust the contributions of the knowledge-level and sample-level loss functions, respectively; i = 1, 2, ..., n, where n represents the number of samples in the CT and pathology image sample dataset.
[0084] Designing a distance-based guided loss function is a key step in improving model generalization performance and enhancing the representation ability of knowledge components. By using guided losses at both the knowledge and sample levels, the discriminative feature representations of various knowledge types can be effectively improved. Through dynamic queues and contrastive learning strategies, the representation ability of four types of knowledge components is optimized, making the features of recurrent samples more distinguishable from those of non-recurrent samples.
[0085] Step 3: Obtain the patient's CT images and pathological images, and input them into the renal clear cell carcinoma prognostic prediction model for prediction to obtain the prognostic prediction results of renal clear cell carcinoma.
[0086] It should be noted that although the embodiments described above are illustrative, they are not intended to limit the invention. Therefore, the invention is not limited to the specific embodiments described above. Any other embodiments obtained by those skilled in the art under the guidance of this invention without departing from its principles are considered to be within the protection scope of this invention.
Claims
1. An auxiliary method for predicting the prognosis of clear cell renal cell carcinoma, characterized in that, The steps include the following: Step 1: Construct a prognostic prediction network for clear cell renal cell carcinoma; The aforementioned prognostic prediction network for clear cell renal cell carcinoma consists of a CT feature extractor, a pathological feature extractor, a CT Transformer module, a pathological Transformer module, a CT personalized knowledge encoder, a pathological personalized knowledge encoder, a common knowledge encoder, a derived knowledge encoder, a CT fully connected layer, a pathological fully connected layer, a common fully connected layer, a derived fully connected layer, a normalization layer, a CT multiplier, a pathological multiplier, a common multiplier, a derived multiplier, an adder, and an extreme learning machine. The input of the CT feature extractor is used as the input of the CT image to the prognostic prediction network for clear cell renal cell carcinoma. The output of the CT feature extractor is connected to the input of the CT Transformer module. The output of the CT Transformer module is connected to the input of the CT individual knowledge encoder, one input of the common knowledge encoder, and one input of the derived knowledge encoder. The input of the pathological feature extractor is used as the input of the pathological image to the prognostic prediction network for clear cell renal cell carcinoma. The output of the pathological feature extractor is connected to the input of the pathological Transformer module. The output of the pathological Transformer module is connected to the input of the pathological individual knowledge encoder, another input of the common knowledge encoder, and another input of the derived knowledge encoder. One output of the CT personalized knowledge encoder is connected to the CT input of the normalization layer via the CT fully connected layer, and the CT output of the normalization layer is connected to one input of the CT multiplier; the other output of the CT personalized knowledge encoder is connected to the other input of the CT multiplier. One output of the pathological personalized knowledge encoder is connected to the pathological input of the normalization layer via a fully connected pathological layer. The pathological output of the normalization layer is connected to one input of the pathological multiplier. The other output of the pathological personalized knowledge encoder is connected to the other input of the pathological multiplier. The output of the common knowledge encoder is connected to the common input of the normalization layer via a common fully connected layer, and the common output of the normalization layer is connected to the input of the common multiplier. The other output of the common knowledge encoder is connected to the other input of the common knowledge multiplier; An output of the derived knowledge encoder is connected to the derived input of the normalization layer via a derived fully connected layer, and the derived output of the normalization layer is connected to an input of the derived multiplier. The other output of the derived knowledge encoder is connected to the other input of the derived multiplier; The outputs of the CT multiplier, the pathology multiplier, the common multiplier, and the derived multiplier are all connected to the input of the adder. The output of the adder is connected to the input of the extreme learning machine. The output of the extreme learning machine serves as the prediction output of the renal clear cell carcinoma prognostic prediction network. Step 2: Collect sample datasets of CT and pathological images, and use the sample datasets of CT and pathological images to train the prognostic prediction network for clear cell renal cell carcinoma to obtain the prognostic prediction model for clear cell renal cell carcinoma. Step 3: Obtain the patient's CT images and pathological images, and input the patient's CT images and pathological images into the prognostic prediction model for clear cell renal cell carcinoma to obtain the prognostic prediction results for clear cell renal cell carcinoma.
2. The method for predicting the prognosis of clear cell renal cell carcinoma according to claim 1, characterized in that, The CT feature extractor is the ResNet18 feature extractor, and the pathology feature extractor is the UNI feature extractor.
3. The method for predicting the prognosis of clear cell renal cell carcinoma according to claim 1, characterized in that, Both the CTTransformer and PathologyTransformer modules are improved Transformer modules, each consisting of two multi-head attention modules, two multipliers, and a pyramid position code generator. The input of the first multi-head attention module serves as the input of the improved Transformer module. The input and output of the first multi-head attention module are simultaneously connected to the input of the first multiplier. The output of the first multiplier is connected to the input of the pyramid position code generator. The output of the pyramid position code generator is connected to the input of the second multi-head attention module. The input and output of the second multi-head attention module are simultaneously connected to the input of the second multiplier. The output of the second multiplier serves as the output of the improved Transformer module.
4. The method for predicting the prognosis of clear cell renal cell carcinoma according to claim 1, characterized in that, Both the CT personalized knowledge encoder and the pathology personalized knowledge encoder are multilayer perceptrons, while both the common knowledge encoder and the derived knowledge encoder are cross-attention modules.
5. The method for predicting the prognosis of clear cell renal cell carcinoma according to claim 1, characterized in that, The Extreme Learning Machine is a sparse Bayesian Extreme Learning Machine.
6. The method for predicting the prognosis of clear cell renal cell carcinoma according to claim 1, characterized in that, The loss function used when training the prognostic prediction network for clear cell renal cell carcinoma is: L (i) total =Δl (i) knowledge +βl (i) patient l (i) knowledge ＝ W(F R (i) ,R (i) +W(F R (i) ,C (i) +W(F P (i) ,P (i) +W(F P (i) ,C (i) -W(F R (i) ,P (i) )-W(F P (i) ,R (i) )-W(F R (i) ,E (i) )-W(F P (i) ,E (i) ) In the formula, L (i) total Let l represent the total loss of the i-th sample group. (i) knowledge Let l represent the knowledge-level loss function for the i-th sample group. (i) patient Let F represent the sample-level loss function for the i-th sample group, α represent the knowledge-level loss weight, and β represent the sample-level loss weight; W(·) represents the Wasserstein distance, and d(·) represents the Euclidean distance; R (i) F represents the intramodal representation of the i-th sample group in CT mode. P (i) R represents the pathological modality representation of the i-th sample group; (i) Let P represent the CT personality knowledge of the i-th sample group. (i) Let C represent the pathological individual knowledge of the i-th sample group. (i) E represents the common knowledge of the i-th group of samples. (i) Represents the derived knowledge of the i-th sample; R + P represents the set of CT-specific knowledge for samples in the CT and pathology image dataset that match the category of the i-th sample group. + C represents the set of pathological individual knowledge for samples in the CT and pathological image sample dataset that match the category of the i-th sample group. + E represents the set of common knowledge among samples in the CT and pathology image sample dataset that belong to the same category as the samples in the i-th group. + R represents the set of derived knowledge for samples in the sample dataset of CT and pathology images that belong to the same category as the samples in the i-th group; - P represents the set of CT-specific knowledge for samples in the CT and pathology image dataset that do not belong to the category of the i-th group of samples. - C represents the set of pathological individual knowledge for samples in the CT and pathological image sample dataset that do not belong to the same category as the samples in the i-th group. - E represents the set of common knowledge among samples in the CT and pathology image sample dataset that do not belong to the same category as the i-th group of samples. - This represents the set of derived knowledge for samples in the CT and pathology image sample dataset that do not belong to the same category as the i-th group of samples; i = 1, 2, ..., n, where n represents the number of samples in the CT and pathology image sample dataset.