A bladder cancer preoperative grading prediction method based on consistent constraint modal fusion

Abstract

The application relates to the technical field of medical data processing, natural language processing and artificial intelligence assisted decision making, and discloses a bladder cancer preoperative grading prediction method based on consistency constraint modal fusion, which comprises the following steps: S1: acquiring preoperative clinical information of a bladder cancer patient and completing data preprocessing; S2: inputting the preprocessed data obtained in the step S1 into a multi-modal representation learning module, respectively extracting feature representations of different modes, and mapping to a shared representation space of a unified dimension; and S3: inputting the same dimension feature representations obtained in the step S2 into a cross-modal fusion module, performing cross-modal information interaction, realizing different modal feature fusion, and generating patient fusion representation. The application solves the problems of insufficient multi-source clinical information joint modeling capability and insufficient fusion representation semantic retention capability in the existing multi-modal pathological grading prediction method, and improves the accuracy and stability of the preoperative pathological grading prediction of the bladder cancer.

Description

Technical Field

[0001] This invention relates to the fields of medical data processing, natural language processing, and artificial intelligence-assisted decision-making technology. Specifically, it relates to a method for predicting preoperative grading of bladder cancer based on consistency-constrained modality fusion. Background Technology

[0002] Bladder cancer is one of the most common malignant tumors of the urinary system and has a high global incidence rate. Pathological grading is a crucial basis for the clinical diagnosis and treatment of bladder cancer, playing a significant guiding role in treatment selection, postoperative recurrence risk assessment, and patient prognosis. Currently, bladder cancer pathological grading mainly relies on cystoscopic biopsy or transurethral resection of bladder tumor (TURBT) to obtain tissue specimens for pathological examination. Although pathological examination is currently the clinically recognized diagnostic basis, these examinations are invasive procedures, requiring patients to undergo tissue biopsy or surgical sampling. Furthermore, due to the significant spatial heterogeneity of bladder cancer, the number of tissue samples obtained preoperatively is limited, and local tissue samples may not fully reflect the pathological characteristics of the entire tumor, thus affecting preoperative pathological grading. Therefore, using routine preoperative clinical data to predict bladder cancer pathological grading is of great significance for improving preoperative risk assessment capabilities and assisting clinicians in developing treatment plans.

[0003] In recent years, with the rapid development of artificial intelligence technology, deep learning methods have achieved good application results in the field of medical auxiliary diagnosis. Numerous studies have utilized medical images, pathological images, electronic medical record texts, and structured clinical data to establish bladder cancer pathological grading prediction models, which have improved auxiliary diagnostic performance to some extent. Among these, deep learning methods based on convolutional neural networks, Transformers, and large language models can automatically learn the latent features in different types of clinical data, showing promising application prospects in tasks such as medical image analysis, clinical text understanding, and disease prediction.

[0004] However, most current research builds predictive models based on single-modal data, such as using only medical images, pathological images, laboratory indicators, or electronic medical record text for pathological grading prediction. Different modalities of data reflect different aspects of a patient's disease state, and a single modality cannot comprehensively describe the patient's overall clinical information; therefore, the predictive performance of these models still has certain limitations.

[0005] For bladder cancer patients, preoperative clinical information is typically available from multiple sources. Admission records include comprehensive clinical information such as the patient's chief complaint, present illness, past medical history, physical examination, and preliminary clinical diagnosis; computed tomography urography (CTU) reports describe the location, size, morphology, enhancement characteristics, and invasion of surrounding tissues of the lesion, reflecting its imaging features; and structured clinical variables such as age and laboratory test results objectively reflect the patient's overall physiological state and metabolic level. These multi-source heterogeneous data are highly complementary, and fully integrating information from different modalities is expected to further improve the accuracy of bladder cancer pathological grading prediction.

[0006] Currently, multimodal learning has been gradually applied to the field of intelligent medical diagnosis. Existing multimodal fusion methods mainly employ feature concatenation, element-wise averaging, gating weighting, and attention mechanisms to achieve the fusion of information from different modalities, with the final classification task serving as the model optimization objective. While these methods can achieve information interaction between different modalities, due to significant differences in expression, semantic granularity, statistical distribution, and information density between long clinical texts, image description texts, and structured numerical data, relying solely on the classification objective for model training can easily lead to a bias in the fused representation towards the modality with the larger information content. This can result in semantic interference between different modalities, insufficient utilization of auxiliary modal information, and ultimately affect the model's predictive performance and generalization ability.

[0007] Furthermore, in bladder cancer pathological grading tasks, high-grade and low-grade cases are typically unevenly distributed, resulting in a prevalent class imbalance in the training data. In clinical practice, misclassifying high-grade bladder cancer as low-grade bladder cancer can lead to delayed treatment, with a significantly higher clinical risk than misclassification. However, most current deep learning models still employ cross-entropy-based loss functions for training, making it difficult to simultaneously address class imbalance and the asymmetric clinical risks associated with different misclassifications. This leaves room for improvement in the model's ability to identify high-grade cases.

[0008] In summary, current methods for predicting the histopathological grading of bladder cancer based on artificial intelligence technology mainly suffer from the following problems: (1) Insufficient ability to jointly model multi-source clinical information. Admission records, CTU descriptions, and structured clinical variables reflect different levels of disease information such as clinical manifestations, imaging characteristics, and physiological status, and the three have strong complementarity. Although existing multimodal pathological grading prediction methods can jointly utilize multi-source clinical data, the information interaction and correlation modeling between different modalities are still insufficient, making it difficult to give full play to the complementary advantages of each modality. As a result, the fused representation cannot fully reflect the overall disease status of the patient, which limits the further improvement of the predictive performance of preoperative pathological grading of bladder cancer.

[0009] (2) Insufficient semantic preservation ability of fusion representation. Existing multimodal pathological grading prediction methods mainly use classification tasks as the model optimization target. In the fusion process, there is a lack of effective mechanism to preserve the semantic information of different modalities. This can easily lead to the fusion representation being biased towards the modality with more information. Some key clinical information is gradually weakened or lost during the fusion process, affecting the accurate expression of the patient's overall disease status by the fusion representation, thereby reducing the model's predictive performance and generalization ability.

[0010] Therefore, how to make full use of patients' preoperative multi-source clinical data, improve the joint modeling ability between different clinical modalities, enhance the ability of fusion representation to preserve multimodal semantic information, and achieve accurate representation of the patient's overall disease state are key technical problems that urgently need to be solved in this field. Summary of the Invention

[0011] The technical problem to be solved by the present invention is to provide a method for predicting the preoperative pathological grading of bladder cancer based on consistency constraint modal fusion. This method solves the problems of insufficient joint modeling ability of multi-source clinical information and insufficient semantic preservation ability of fusion representation in existing multimodal pathological grading prediction methods, thereby improving the accuracy and stability of preoperative pathological grading prediction of bladder cancer.

[0012] The present invention achieves its objective by employing the following technical solution: A method for predicting preoperative grading of bladder cancer based on consistency-constrained modality fusion includes the following steps: S1: Obtain preoperative clinical information of bladder cancer patients and complete data preprocessing; S2: Input the preprocessed data obtained in S1 into the multimodal representation learning module, extract the feature representations of different modalities respectively, and map them to a shared representation space of a unified dimension; S3: Input the same-dimensional feature representation obtained in S2 into the cross-modal fusion module to perform cross-modal information interaction, realize the fusion of features from different modalities, and generate a patient fusion representation; S4: Input the fusion representation obtained in S3 into the consistency joint optimization module, construct the modal semantic consistency constraints between the fusion representation and each single modal representation, and perform joint optimization by combining cost-sensitive classification loss to complete the training of the preoperative pathological grading model for bladder cancer.

[0013] Currently, multimodal deep learning models can achieve joint analysis of different types of clinical data. However, most existing methods rely solely on classification objectives to optimize model parameters, making it difficult to maintain semantic consistency between different modalities. Furthermore, they are susceptible to imbalanced class distribution and variations in clinical risk of misclassification. This invention constructs a multimodal deep learning framework for predicting the pathological grade of bladder cancer. It utilizes admission records as the dominant modality, guiding cross-modal feature interaction between computed tomography urinary tract imaging reports and structured clinical variables to achieve effective fusion of multi-source clinical information. A multi-level consistency constraint mechanism is further introduced to ensure semantic consistency between the fused representation and the features of each modality, improving the discriminative power of the fused representation. Simultaneously, an objective optimization function is designed to consider imbalanced class distribution and asymmetric misclassification risk, enhancing the model's ability to identify high-grade bladder cancer, thereby improving the accuracy and stability of preoperative pathological grade prediction for bladder cancer.

[0014] As a further limitation of this technical solution, the specific steps of S1 are as follows: S11: Collect preoperative admission records, computed tomography urinary tract imaging descriptions and structured clinical variables of bladder cancer patients, and remove patient identity information to complete data de-identification processing; S12: The data processed in S11 is filtered according to the examination time to ensure that the information of each modality comes from the same preoperative diagnosis and treatment stage; the text data is cleaned, the missing values ​​of the structured clinical variables are processed, and normalization is performed according to the statistical parameters of the training set to generate a dataset including three clinical modalities. Choose the appropriate normalization method based on the distribution of each variable. For variables that approximately follow a normal distribution, use Z-score standardization, the formula of which is:

[0015] in: Indicates the first The first patient One structured clinical variable; and They represent the first and second training sets, respectively. The mean and standard deviation of each variable; This represents the clinical variable value after Z-score standardization; For variables that do not follow a normal distribution, Min-Max normalization is used, and its calculation formula is as follows:

[0016] in: and They represent the first and second training sets, respectively. The minimum and maximum values ​​of each variable; After preprocessing is completed, the first The multimodal input of a patient can be represented as:

[0017] in: Indicates the first The patient's admission record text, This refers to descriptive text for computed tomography (CT) urinary tract imaging. This represents the normalized structured clinical variables. This indicates that the clinical variable dimension is 10. Indicates the first Multimodal input for each patient.

[0018] As a further limitation of this technical solution, the specific steps of S2 are as follows: S21: Input the admission record text and the computed tomography urography description text generated in S12 into the medical domain large language model to extract the semantic representation of the text; No. The first patient's admission record text The hidden state corresponding to each valid word is: The number of effective tokens is Then the admission record text Represented as:

[0019] No. The first patient's computed tomography urography description text in the first patient's urinary tract imaging description text The hidden state corresponding to each valid word is: The number of effective tokens is The text describing the computed tomography urinary tract imaging is as follows. Represented as:

[0020] S22: Linearly map the structured clinical variables generated in S12 and project them together with the text representation generated in S21 into a shared representation space of the same dimension to obtain the admission record representation, the computed tomography urinary tract imaging representation, and the structured clinical variable representation, respectively. The computation process of mapping the three modalities to a shared representation space of a unified dimension is as follows:

[0021]

[0022]

[0023] in: These represent the hospital admission record representation, computed tomography urography description representation, and structured clinical variable representation, respectively, mapped to a unified dimension. , and These represent the linear projection matrices corresponding to the admission record text, the computed tomography urography description text, and the structured clinical variables, respectively. , and These represent the corresponding bias vectors.

[0024] As a further limitation of this technical solution, the specific steps of S3 are as follows: S31: The admission record obtained in S22 is used as a query. The computed tomography urinary tract imaging representation and the structured clinical variable representation are added to the modality type embedding as keys and values, and input into the multi-head cross-attention module to realize information interaction between different modalities. The corresponding modality embeddings were added to the computed tomography urinary tract imaging descriptive representation and the structured clinical variable representation, respectively. The calculation process is as follows:

[0025]

[0026] in: and These represent the learnable modality type embeddings corresponding to the computed tomography urinary tract imaging descriptive modality and the structured clinical variable modality, respectively; and These represent the descriptive representation of computed tomography urinary tract imaging and the representation of structured clinical variables after incorporating modality type embedding, respectively. Subsequently, using the admission record representation as the query, and combining the computed tomography urinary tract imaging descriptive representation and the structured clinical variable representation as both keys and values, a cross-modal attention input is constructed. The computation process is represented as follows:

[0027] in: This indicates that the input sequence is composed according to the modal order. The query value representing the cross-modal attention input, and These are the key and value of the cross-modal attention input; Subsequently, a multi-head cross-modal attention mechanism is used to calculate the information interaction between different modalities. The calculation process is as follows:

[0028] in: Shared representation space dimension Represents the normalization function; It is the cross-modal attention output value with Q, K, and V as inputs; S32: The feature representation of the cross-attention output in S31 is sequentially subjected to residual connection, layer normalization and feedforward network to obtain the patient fusion representation; The cross-modal feature representation is residually joined with the original admission record representation and then subjected to layer normalization. The calculation process is as follows:

[0029] in: This represents the intermediate feature representation after cross-modal attention fusion; Presentation layer normalization operation; Represent the above intermediate features The input is fed forward into the network, and residual connections and layer normalization are performed again to obtain the final fused representation:

[0030] in: This represents a feedforward neural network; This indicates the patient's final fusion.

[0031] As a further limitation of this technical solution, the specific steps of S4 are as follows: S41: The fusion representation obtained in S32 is combined with the hospital admission record representation, the computed tomography urinary tract imaging representation, and the structured clinical variable representation to construct a directional information noise contrast estimation contrast learning objective, thereby achieving modal semantic consistency constraint; fusion representation As anchor points, respectively, they are related to the admission records obtained in S2. Description of computed tomography (CT) urinary tract imaging and structured clinical variable representation Establish directional contrastive learning relationships for three modalities. Directional consistency loss The calculations are as follows: ; ; ; in: Represents the cosine similarity function; Indicates the temperature coefficient; Indicates the training batch size; We obtain the modal-wise semantic consistency loss by weighted summation of the directional consistency losses corresponding to the three modalities:

[0032] in: , and These represent the consistency loss weights corresponding to admission records, computed tomography urinary tract imaging descriptions, and structured clinical variables, respectively. S42: Input the fusion representation obtained in step 32 into the classifier to predict the pathological grade of bladder cancer. Use cost-sensitive classification loss as the classification optimization objective. Improve the ability to identify high-grade bladder cancer by adjusting the misclassification cost of different categories. A cost-sensitive classification strategy is adopted, which introduces a misclassification cost matrix in addition to the weighted cross-entropy loss. The misclassification cost matrix is ​​expressed as follows:

[0033] Where: rows of the matrix represent the true class, and columns represent the predicted class; This indicates the cost of misdiagnosing low-grade bladder cancer as high-grade bladder cancer. This indicates the cost of misdiagnosing high-grade bladder cancer as low-grade bladder cancer. For a training batch, its cost-sensitive classification loss Represented as:

[0034] in: This represents the weighted cross-entropy loss; Indicates the first The patient was predicted to be the [number]th [patient]. The probability of a class; Indicates the true category; This represents the corresponding misclassification cost; Indicates the weight of the cost item; The modality-wise semantic consistency loss and cost-sensitive classification loss are jointly optimized to achieve the overall model loss function. Represented as:

[0035] in: This represents the weights of the modal semantic consistency loss; S43: By introducing weight coefficients, the classification loss and the modal semantic consistency loss are combined into a joint optimization objective. The cross-modal fusion module and the classifier are jointly trained, so that the fusion representation can fully preserve the semantic information of each modality and improve the overall prediction performance of the model.

[0036] As a further limitation of this technical solution, it also includes S5: After the model training is completed, the input representation is fused into the classifier, and the output is the predicted probability that the patient belongs to low-grade bladder cancer or high-grade bladder cancer.

[0037] As a further limitation of this technical solution, the hierarchical prediction method is implemented based on a hierarchical prediction system, which includes a data acquisition and preprocessing module, a multimodal representation learning module, a cross-modal fusion module, and a consistency joint optimization module.

[0038] Compared with related technologies, the preoperative grading prediction method for bladder cancer based on consistency-constrained modality fusion provided by this invention has the following beneficial effects: (1) This invention fully integrates multi-source heterogeneous clinical data such as patient preoperative admission record text, computed tomography urinary tract imaging description text and structured clinical variables. With the admission record as the dominant modality, it guides computed tomography urinary tract imaging description and structured clinical variables to perform cross-modal feature interaction, realizes full modeling of the correlation and complementary information between different modalities, improves the ability of the fusion representation to express the overall disease status of the patient, and solves the problem of insufficient joint modeling ability of multi-source clinical information in the prior art.

[0039] (2) The present invention constructs a modal semantic alignment mechanism, which performs directional comparative learning with the hospital admission record representation, the computed tomography urinary tract imaging description representation and the structured clinical variable representation, respectively, so that the fusion representation can fully maintain the key semantic information of each modality in the process of multimodal feature interaction, reduce information interference and semantic shift between modalities, improve the discriminative ability and generalization performance of the fusion representation, and solve the problem of insufficient semantic preservation ability of the fusion representation in the prior art.

[0040] (3) This invention addresses two key technical problems in existing methods for predicting the pathological grading of bladder cancer: insufficient ability to jointly model multi-source clinical information and insufficient ability to maintain the semantics of the fusion representation. It proposes a method and system for predicting the preoperative grading of bladder cancer based on consistency-constrained modal fusion, which realizes effective joint modeling of multi-source clinical information and enhances the ability of the fusion representation to maintain multi-modal semantic information, thereby improving the accuracy, stability and clinical application value of the preoperative pathological grading prediction of bladder cancer. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the process of the present invention.

[0042] Figure 2 This is a schematic diagram of the consistency-constrained multimodal representation learning model of the present invention. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0044] A method for predicting preoperative bladder cancer grading based on consistency-constrained modal fusion is proposed. First, preprocessing of the patient's preoperative admission record text, computed tomography (CT) urinary tract imaging description text, and structured clinical variables is performed. Then, a large language model in the medical field is used to extract semantic representations of the two types of clinical texts, and the structured clinical variables are normalized and mapped to a unified shared representation space through independent linear projection. Next, using the admission record representation as a query, the CT urinary tract imaging description features and structured clinical variable features are fused through a cross-modal attention mechanism to obtain the patient's fused features. Finally, a per-modal semantic consistency constraint is constructed using the fused features as anchors, and joint optimization is performed using a cost-sensitive classification loss to output the predicted results of whether the patient has low-grade or high-grade bladder cancer.

[0045] like Figure 1 and 2 As shown, the present invention includes the following steps: S1: Obtain the preoperative admission record text, computed tomography urinary tract imaging description text, and structured clinical variables of bladder cancer patients, and complete the data preprocessing.

[0046] The specific steps of S1 are as follows: S11: Collect preoperative admission records, computed tomography urography (CTU) descriptions and structured clinical variables of bladder cancer patients, and remove patient identity information to complete data de-identification processing; S12: The data processed in S11 is filtered according to the examination time to ensure that the information of each modality comes from the same preoperative diagnosis and treatment stage; the text data is cleaned, the missing values ​​of the structured clinical variables are processed, and normalization is performed according to the statistical parameters of the training set to generate a dataset including three clinical modalities. Choose the appropriate normalization method based on the distribution of each variable. For variables that approximately follow a normal distribution, use Z-score standardization, the formula of which is:

[0047] in: Indicates the first The first patient One structured clinical variable; and They represent the first and second training sets, respectively. The mean and standard deviation of each variable; This represents the clinical variable value after Z-score standardization; For variables that do not follow a normal distribution, Min-Max normalization is used, and its calculation formula is as follows:

[0048] in: and They represent the first and second training sets, respectively. The minimum and maximum values ​​of each variable; All normalized parameters were calculated from the training set and applied unchanged to the validation and test sets to ensure consistency between model training and prediction. After preprocessing, the... The multimodal input of a patient can be represented as:

[0049] in: Indicates the first The patient's admission record text, This refers to descriptive text for computed tomography (CT) urinary tract imaging. This represents the normalized structured clinical variables. This indicates that the clinical variable dimension is 10. Indicates the first Multimodal input for each patient.

[0050] S2: Input the admission record text, computed tomography urinary tract imaging description text, and normalized structured clinical variables obtained in S1 into the modality representation learning module, extract feature representations of different modalities respectively, and map the three modalities to a shared representation space of a unified dimension.

[0051] The specific steps of S2 are as follows: S21: Input the admission record text and the computed tomography urography description text generated in S12 into the medical domain large language model to extract the semantic representation of the text; First, a large language model for the medical field is used to semantically encode the admission record text and the computed tomography urography description text, respectively, to obtain the hidden state corresponding to each effective word. Then, average pooling is performed on the hidden states corresponding to all effective words to obtain the patient-level text representation.

[0052] Let the first The first patient's admission record text The hidden state corresponding to each valid word is: The number of effective tokens is Then the admission record text Represented as:

[0053] Similarly, let the first... The first patient's computed tomography urography description text in the first patient's urinary tract imaging description text The hidden state corresponding to each valid word is: The number of effective tokens is The text describing the computed tomography urinary tract imaging is as follows. Represented as:

[0054] S22: Linearly map the structured clinical variables generated in S12 and project them together with the text representation generated in S21 into a shared representation space of the same dimension to obtain the admission record representation, the computed tomography urinary tract imaging representation, and the structured clinical variable representation, respectively. Because the text representations of admission records, computed tomography urography descriptions, and structured clinical variables differ in feature dimensions and data distribution, direct cross-modal information exchange is not possible. Therefore, this implementation uses three independent linear projection layers to map the three modalities to a shared representation space with a unified dimension. The calculation process is as follows:

[0055]

[0056]

[0057] in: These represent the hospital admission record representation, computed tomography urography description representation, and structured clinical variable representation, respectively, mapped to a unified dimension. , and These represent the linear projection matrices corresponding to the admission record text, the computed tomography urography description text, and the structured clinical variables, respectively. , and These represent the corresponding bias vectors. In this embodiment, Set it to 256.

[0058] The three linear projection layers use independent parameters, enabling the admission record text, computed tomography urography description text, and structured clinical variables to complete feature mapping according to their respective data characteristics. While unifying the feature dimensions, the semantic information of different modalities is preserved, providing a unified data representation for subsequent cross-modal information interaction.

[0059] Through the above-mentioned clinical text encoding and modal projection process, the admission record text, computed tomography urography description text, and structured clinical variables are converted into a modal representation with a unified dimension, providing input for the cross-modal fusion guided by the admission record in S3.

[0060] S3: Input the admission record representation, computed tomography urinary tract imaging description representation, and structured clinical variable representation obtained in S2 into the cross-modal fusion module. Use the admission record representation as a query to guide the computed tomography urinary tract imaging description representation and structured clinical variable representation to perform cross-modal information interaction, realize the fusion of different modal features, and generate a patient fusion representation.

[0061] The specific steps of S3 are as follows: S31: The admission record obtained in S22 is used as a query. The computed tomography urinary tract imaging representation and the structured clinical variable representation are added to the modality type embedding as keys and values, and input into the multi-head cross-attention module to realize information interaction between different modalities. Specifically, to distinguish auxiliary modal information from different sources, this implementation first adds corresponding modal type embeddings to the computed tomography urinary tract imaging descriptive representation and the structured clinical variable representation, respectively. The calculation process is as follows:

[0062]

[0063] in: and These represent the learnable modality type embeddings corresponding to the computed tomography urinary tract imaging descriptive modality and the structured clinical variable modality, respectively; and These represent the descriptive representation of computed tomography urinary tract imaging and the representation of structured clinical variables after incorporating modality type embedding, respectively. Subsequently, using the admission record representation as the query, and combining the computed tomography urinary tract imaging description representation and the structured clinical variable representation as both keys and values, a cross-modal attention input is constructed. The computation process is represented as follows:

[0064] in: This indicates that the input sequence is composed of modalities in order. The query value representing the cross-modal attention input, and These are the key and value of the cross-modal attention input; Subsequently, a multi-head cross-modal attention mechanism is used to calculate the information interaction between different modalities. The calculation process is as follows:

[0065] in: Shared representation space dimension Represents the normalization function; It is the cross-modal attention output value with Q, K, and V as inputs; This implementation uses a multi-head cross-modal attention mechanism to complete information interaction between different modalities and fuses the outputs of each attention head to obtain cross-modal feature representations.

[0066] S32: The feature representation of the cross-attention output in S31 is sequentially subjected to residual connection, layer normalization and feedforward network to obtain the patient fusion representation; The cross-modal feature representation is residually joined with the original admission record representation and then subjected to layer normalization. The calculation process is as follows:

[0067] in: This represents the intermediate feature representation after cross-modal attention fusion; Presentation layer normalization operation; Furthermore, the above intermediate features are represented as follows: The input is fed forward into the network, and residual connections and layer normalization are performed again to obtain the final fused representation:

[0068] in: This represents a feedforward neural network; This indicates the patient's final fusion.

[0069] Through the above steps, with the admission record as the dominant modality, the computed tomography urinary tract imaging description and structured clinical variables are guided to conduct cross-modal information interaction around the patient's overall clinical information. This enables the fusion representation to simultaneously retain multi-source information such as the patient's clinical manifestations, imaging features, and physiological status, thereby improving the ability to express the patient's overall disease status and providing a more accurate fusion feature representation for modal semantic consistency learning and pathological grading prediction in S4.

[0070] S4: Input the fusion representation obtained in S3 into the consistency joint optimization module, construct the modal semantic consistency constraints between the fusion representation and each single modal representation, and perform joint optimization by combining cost-sensitive classification loss to complete the training of the preoperative pathological grading model for bladder cancer.

[0071] The specific steps of S4 are as follows: S41: The fusion representation obtained in S32 is combined with the hospital admission record representation, the computed tomography urinary tract imaging representation, and the structured clinical variable representation to construct a directional information noise contrast estimation contrast learning objective, thereby achieving modal semantic consistency constraint; fusion representation As anchor points, respectively, they are related to the admission records obtained in S2. Description of computed tomography (CT) urinary tract imaging and structured clinical variable representation Establish directional contrastive learning relationships for three modalities. Directional consistency loss The calculations are as follows: ; ; ; in: Represents the cosine similarity function; Indicates the temperature coefficient; Indicates the training batch size; Furthermore, by weighted summing the directional consistency losses corresponding to the three modalities, we obtain the modal-wise semantic consistency loss:

[0072] in: , and These represent the consistency loss weights corresponding to admission records, computed tomography urinary tract imaging descriptions, and structured clinical variables, respectively.

[0073] By separately constraining the semantic relationships between the fusion representation and each individual modal representation, the fusion representation can maintain the key semantic information of each modality while integrating multimodal complementary information.

[0074] S42: Input the fusion representation obtained in S32 into the classifier to predict the pathological grade of bladder cancer. Use cost-sensitive classification loss as the classification optimization objective. Improve the ability to identify high-grade bladder cancer by adjusting the misclassification cost of different categories. To reduce the clinical risk of misclassifying high-grade bladder cancer as low-grade bladder cancer, this implementation adopts a cost-sensitive classification strategy, introducing a misclassification cost matrix in addition to the weighted cross-entropy loss. The misclassification cost matrix is ​​expressed as follows:

[0075] Where: rows of the matrix represent the true class, and columns represent the predicted class; This indicates the cost of misdiagnosing low-grade bladder cancer as high-grade bladder cancer. This represents the cost of misdiagnosing high-grade bladder cancer as low-grade bladder cancer; this implementation method sets... This allows the model to focus more on recognizing high-grade bladder cancer cases during training.

[0076] For a training batch, its cost-sensitive classification loss Represented as:

[0077] in: This represents the weighted cross-entropy loss; Indicates the first The patient was predicted to be the [number]th [patient]. The probability of a class; Indicates the true category; This represents the corresponding misclassification cost; Indicates the weight of the cost item; Finally, the modal semantic consistency loss and cost-sensitive classification loss are jointly optimized to obtain the overall model loss function. Represented as:

[0078] in: This represents the weights of the modal semantic consistency loss; S43: By introducing weight coefficients, the classification loss and the modal semantic consistency loss are combined into a joint optimization objective. The cross-modal fusion module and the classifier are jointly trained, so that the fusion representation can fully preserve the semantic information of each modality and improve the overall prediction performance of the model.

[0079] S5: After the model training is complete, the fused representations will be input into the classifier, and the output will be the predicted probability that the patient has low-grade or high-grade bladder cancer. During model training, the modality projection layer in S2, the modality type embedding in S3, the cross-modality fusion module, and the classifier parameters are updated based on the overall loss function. The parameters of the medical domain large language model used to extract clinical text representations remain fixed and are not involved in subsequent model training.

[0080] After the model is trained, the preoperative admission record text of the patient to be predicted, the description text of the computed tomography urinary tract imaging, and the structured clinical variables are input into the trained model. The model is then processed through clinical text encoding, modal projection, cross-modal fusion, and classifier calculation. The model outputs the predicted probability of the patient belonging to low-grade bladder cancer or high-grade bladder cancer, and the category with the higher predicted probability is used as the final prediction result.

[0081] Through the above joint optimization process, the fusion representation can maintain semantic consistency with each single-modal representation while making full use of multimodal complementary information. It also improves the model's ability to identify high-grade bladder cancer cases by combining asymmetric misclassification costs, thereby further improving the accuracy of preoperative pathological grading prediction of bladder cancer and its clinical application value.

[0082] A preoperative grading prediction system for bladder cancer based on consistency-constrained modality fusion includes: The data acquisition and preprocessing module is responsible for acquiring the preoperative admission records of bladder cancer patients, the text descriptions of computed tomography urinary tract imaging, and structured clinical variables. It performs preprocessing on each modality of data, including de-identification, time consistency screening, text cleaning, missing value handling, and normalization, and generates the model input data.

[0083] Multimodal Representation Learning Module: Responsible for extracting high-level semantic representations of admission record text and computed tomography urography description text using a large language model in the medical field, performing linear mapping on structured clinical variables, and mapping the three modalities to a shared representation space of a unified dimension through independent linear projection, thereby obtaining the admission record representation, computed tomography urography description representation, and structured clinical variable representation.

[0084] The cross-modal fusion module is responsible for using the admission record representation as a query, and the computed tomography urinary tract imaging description representation with modality type embedding and the structured clinical variable representation as keys and values. It realizes cross-modal feature interaction through a multi-head cross-attention mechanism led by the admission record, and generates a patient fusion representation by combining residual connections, layer normalization and feedforward networks.

[0085] The consistency joint optimization module is responsible for using the fusion representation as an anchor point to construct directional InfoNCE contrastive learning objectives with the admission record representation, computed tomography urinary tract imaging description representation, and structured clinical variable representation, forming a modal semantic consistency loss. It also combines this with cost-sensitive classification loss to construct a joint optimization objective and train the model. The trained classifier is then used to classify the fusion representation and output the predicted probability of whether the patient has low-grade or high-grade bladder cancer, thus achieving preoperative histopathological grading prediction for bladder cancer.

[0086] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for predicting preoperative grading of bladder cancer based on consistency-constrained modality fusion, characterized in that, Includes the following steps: S1: Obtain preoperative clinical information of bladder cancer patients and complete data preprocessing; S2: Input the preprocessed data obtained in S1 into the multimodal representation learning module, extract the feature representations of different modalities respectively, and map them to a shared representation space of a unified dimension; S3: Input the same-dimensional feature representation obtained in S2 into the cross-modal fusion module to perform cross-modal information interaction, realize the fusion of features from different modalities, and generate a patient fusion representation; S4: Input the fusion representation obtained in S3 into the consistency joint optimization module, construct the modal semantic consistency constraints between the fusion representation and each single modal representation, and perform joint optimization by combining cost-sensitive classification loss to complete the training of the preoperative pathological grading model for bladder cancer.

2. The method for predicting preoperative grading of bladder cancer based on consistency-constrained modality fusion according to claim 1, characterized in that: The specific steps of S1 are as follows: S11: Collect preoperative admission records, computed tomography urinary tract imaging descriptions and structured clinical variables of bladder cancer patients, and remove patient identity information to complete data de-identification processing; S12: The data processed in S11 is filtered according to the examination time to ensure that the information of each modality comes from the same preoperative diagnosis and treatment stage; the text data is cleaned, the missing values ​​of the structured clinical variables are processed, and normalization is performed according to the statistical parameters of the training set to generate a dataset including three clinical modalities. Choose the appropriate normalization method based on the distribution of each variable. For variables that approximately follow a normal distribution, use Z-score standardization, the formula of which is: ； in: Indicates the first The first patient One structured clinical variable; and They represent the first and second training sets, respectively. The mean and standard deviation of each variable; This represents the clinical variable value after Z-score standardization; For variables that do not follow a normal distribution, Min-Max normalization is used, and its calculation formula is as follows: ； in: and They represent the first and second training sets, respectively. The minimum and maximum values ​​of each variable; After preprocessing is completed, the first The multimodal input of a patient can be represented as: ； in: Indicates the first The patient's admission record text, This refers to descriptive text for computed tomography (CT) urinary tract imaging. This represents the normalized structured clinical variables. This indicates that the clinical variable dimension is 10. Indicates the first Multimodal input for each patient.

3. The method for predicting preoperative grading of bladder cancer based on consistency-constrained modality fusion according to claim 1, characterized in that: The specific steps of S2 are as follows: S21: Input the admission record text and the computed tomography urography description text generated in S12 into the medical domain large language model to extract the semantic representation of the text; No. The first patient's admission record text The hidden state corresponding to each valid word is: The number of effective tokens is Then the admission record text Represented as: ； No. The first patient's computed tomography urography description text in the first patient's urinary tract imaging description text The hidden state corresponding to each valid word is: The number of effective tokens is The text describing the computed tomography urinary tract imaging is as follows. Represented as: ； S22: Linearly map the structured clinical variables generated in S12 and project them together with the text representation generated in S21 into a shared representation space of the same dimension to obtain the admission record representation, the computed tomography urinary tract imaging representation, and the structured clinical variable representation, respectively. The computation process of mapping the three modalities to a shared representation space of a unified dimension is as follows: ； ； ； in: These represent the hospital admission record representation, computed tomography urography description representation, and structured clinical variable representation, respectively, mapped to a unified dimension. , and These represent the linear projection matrices corresponding to the admission record text, the computed tomography urography description text, and the structured clinical variables, respectively. , and These represent the corresponding bias vectors.

4. The method for predicting preoperative grading of bladder cancer based on consistency-constrained modality fusion according to claim 3, characterized in that: The specific steps of S3 are as follows: S31: The admission record obtained in S22 is used as a query. The computed tomography urinary tract imaging representation and the structured clinical variable representation are added to the modality type embedding as keys and values, and input into the multi-head cross-attention module to realize information interaction between different modalities. The corresponding modality embeddings were added to the computed tomography urinary tract imaging descriptive representation and the structured clinical variable representation, respectively. The calculation process is as follows: ； ； in: and These represent the learnable modality type embeddings corresponding to the computed tomography urinary tract imaging descriptive modality and the structured clinical variable modality, respectively; and These represent the descriptive representation of computed tomography urinary tract imaging and the representation of structured clinical variables after incorporating modality type embedding, respectively. Subsequently, using the admission record representation as the query, and combining the computed tomography urinary tract imaging descriptive representation and the structured clinical variable representation as both keys and values, a cross-modal attention input is constructed. The computation process is represented as follows: ； in: This indicates that the input sequence is composed according to the modal order. The query value representing the cross-modal attention input, and These are the key and value of the cross-modal attention input; Subsequently, a multi-head cross-modal attention mechanism is used to calculate the information interaction between different modalities. The calculation process is as follows: ； in: Shared representation space dimension Represents the normalization function; It is the cross-modal attention output value with Q, K, and V as inputs; S32: The feature representation of the cross-attention output in S31 is sequentially subjected to residual connection, layer normalization and feedforward network to obtain the patient fusion representation; The cross-modal feature representation is residually joined with the original admission record representation and then subjected to layer normalization. The calculation process is as follows: ； in: This represents the intermediate feature representation after cross-modal attention fusion; Presentation layer normalization operation; Represent the above intermediate features The input is fed forward into the network, and residual connections and layer normalization are performed again to obtain the final fused representation: ； in: This represents a feedforward neural network; This indicates the patient's final fusion.

5. The method for predicting preoperative grading of bladder cancer based on consistency-constrained modality fusion according to claim 4, characterized in that: The specific steps of S4 are as follows: S41: The fusion representation obtained in S32 is combined with the hospital admission record representation, the computed tomography urinary tract imaging representation, and the structured clinical variable representation to construct a directional information noise contrast estimation contrast learning objective, thereby achieving modal semantic consistency constraint; fusion representation As anchor points, respectively, they are related to the admission records obtained in S2. Description of computed tomography (CT) urinary tract imaging and structured clinical variable representation Establish directional contrastive learning relationships for three modalities. Directional consistency loss The calculations are as follows: ； ； ； in: Represents the cosine similarity function; Indicates the temperature coefficient; Indicates the training batch size; We obtain the modal-wise semantic consistency loss by weighted summation of the directional consistency losses corresponding to the three modalities: ； in: , and These represent the consistency loss weights corresponding to admission records, computed tomography urinary tract imaging descriptions, and structured clinical variables, respectively. S42: Input the fusion representation obtained in step 32 into the classifier to predict the pathological grade of bladder cancer. Use cost-sensitive classification loss as the classification optimization objective. Improve the ability to identify high-grade bladder cancer by adjusting the misclassification cost of different categories. A cost-sensitive classification strategy is adopted, which introduces a misclassification cost matrix in addition to the weighted cross-entropy loss. The misclassification cost matrix is ​​expressed as follows: ； Where: rows of the matrix represent the true class, and columns represent the predicted class; This indicates the cost of misdiagnosing low-grade bladder cancer as high-grade bladder cancer. This indicates the cost of misdiagnosing high-grade bladder cancer as low-grade bladder cancer. For a training batch, the cost-sensitive classification loss Represented as: ； in: This represents the weighted cross-entropy loss; Indicates the first The patient was predicted to be the [number]th [patient]. The probability of a class; Indicates the true category; This represents the corresponding misclassification cost; Indicates the weight of the cost item; The modality-wise semantic consistency loss and cost-sensitive classification loss are jointly optimized to achieve the overall model loss function. Represented as: ； in: This represents the weights of the modal semantic consistency loss; S43: By introducing weight coefficients, the classification loss and the modal semantic consistency loss are combined into a joint optimization objective. The cross-modal fusion module and the classifier are jointly trained, so that the fusion representation can fully preserve the semantic information of each modality and improve the overall prediction performance of the model.

6. The method for predicting preoperative grading of bladder cancer based on consistency-constrained modal fusion according to claim 4, characterized in that: It also includes S5: After the model is trained, it will fuse the input representation into the classifier and output the predicted probability that the patient has low-grade or high-grade bladder cancer.

7. The method for predicting preoperative grading of bladder cancer based on consistency-constrained modality fusion according to claim 1, characterized in that: The hierarchical prediction method is implemented based on a hierarchical prediction system, which includes a data acquisition and preprocessing module, a multimodal representation learning module, a cross-modal fusion module, and a consensus joint optimization module.

Images