Multi-mode pancreatic cancer prognosis classification prediction method and system based on medical knowledge

Abstract

The invention discloses a multi-mode pancreatic cancer prognosis classification prediction method and system based on medical knowledge. The method comprises the following steps: preprocessing acquired image data, clinical data and electronic medical record text data; extracting image data features, clinical data features and electronic medical record text data features from the preprocessed data; performing multi-modal fusion on the extracted image data features, clinical data features and electronic medical record text data features to obtain multi-modal fusion features; and performing pancreatic cancer prognosis classification prediction based on the multi-modal fusion features. Through combination of deep learning and a natural language processing technology, the defect that the prior art only depends on iconography and structured clinical data is overcome, and medical knowledge and clinical reasoning can be comprehensively integrated, so that the prediction precision is improved, especially in the aspects of tumor risk classification and prognosis evaluation.

Description

Technical Field

[0001] This invention belongs to the technical field of medical assistance and artificial intelligence, specifically relating to a multimodal pancreatic cancer prognostic classification and prediction method and system based on medical knowledge. Background Technology

[0002] Pancreatic cancer, a highly malignant digestive system tumor, has always posed a significant challenge to early diagnosis and prognostic assessment in the medical field. Although advancements in imaging techniques and biomarkers have led to increasingly precise diagnosis of pancreatic cancer, the high heterogeneity of the tumor, atypical clinical symptoms, and diverse imaging manifestations still result in a high risk of missed and misdiagnosed cases. Therefore, improving the accuracy of early diagnosis of pancreatic cancer and combining multiple data sources for personalized prognostic prediction has become a key direction in medical research.

[0003] Neoplasm Tumor Risk (NTR) classification for pancreatic cancer, as part of prognostic assessment, combines patient imaging information, clinical data, and tumor markers to categorize the malignancy, metastasis risk, and prognostic outcomes of pancreatic tumors, aiding in the development of personalized treatment plans. Currently, NTR classification for pancreatic cancer primarily relies on imaging examinations, patient clinical information, and biomarkers. However, due to the limited information from a single data source, current NTR prediction methods based on single-modal data have certain limitations, resulting in limited classification accuracy.

[0004] Multimodal learning, as a groundbreaking technology, is beginning to be widely applied in the medical field. By integrating imaging data, clinical data, and medical text data, multimodal learning achieves effective fusion of different data types to provide more comprehensive predictive judgments. In NTR prediction for pancreatic cancer, imaging data (such as CT and MRI) effectively reflects the morphological characteristics and tissue structure of the tumor, clinical data (such as patient age, gender, family history, and tumor marker levels) reflects the patient's overall health status, while medical text data (such as pathology reports and doctor's diagnoses) contains rich medical knowledge and medical record information. These multimodal data can be used for feature extraction and fusion through deep learning and natural language processing (NLP) techniques, thereby improving the accuracy of NTR prediction.

[0005] The application of natural language processing technology based on large-scale AI language models in the medical field is gradually emerging, enabling in-depth analysis of medical texts to extract potential medical knowledge and clinical reasoning. For example, GPT-type large models (Generative PretrAInedTransformer) can acquire knowledge from patient medical records, doctor reports, and medical literature and play a role in prediction tasks.

[0006] Currently, some multimodal cancer prognosis prediction methods exist, but most are limited to combining imaging data and structured clinical data, failing to fully explore the potential information in medical texts. Therefore, how to combine medical information and clinical significance using AI large language models, and integrate them with imaging and clinical data, to further improve the predictive performance of pancreatic cancer NTR classification remains a challenge in the current technological field. Summary of the Invention

[0007] In view of the above, the present invention aims to provide a multimodal pancreatic cancer prognostic classification and prediction method and system based on medical knowledge by utilizing a large-scale AI language model combined with imaging data, clinical data and medical text data through multimodal learning. By combining deep learning and natural language processing technology, it overcomes the shortcomings of existing technologies that rely solely on imaging and structured clinical data, and can comprehensively integrate medical knowledge and clinical reasoning, thereby improving prediction accuracy, especially in tumor risk classification and prognostic assessment.

[0008] To achieve the above-mentioned objectives, the embodiments provide a multimodal pancreatic cancer prognostic classification and prediction method based on medical knowledge, comprising the following steps: Preprocess the collected image data, clinical data, and electronic medical record text data; Extract image data features, clinical data features, and electronic medical record text data features from the preprocessed data; Multimodal fusion was performed on the extracted image data features, clinical data features, and electronic medical record text data features to obtain multimodal fused features; Prognostic classification and prediction of pancreatic cancer based on multimodal fusion features.

[0009] Preferably, the collected image data, clinical data, and electronic medical record text data are preprocessed, including: For the image data, window width and level were adjusted and normalized, ROI segmentation was performed based on the pancreatic tumor mask, and data augmentation was carried out; For clinical data, data cleaning and standardization, as well as feature extraction of numerical data, are performed. For electronic medical record text data, we perform text cleaning and standardization, medical entity recognition, information extraction and structure transformation; Among them, ROI segmentation based on pancreatic tumor masks includes: The smallest bounding cube of the tumor region is extracted based on the pancreatic tumor mask, and the image is expanded outward by 20 pixels as the tumor-peripancreatic junction input. At the same time, the smallest bounding cube is expanded outward by 20 pixels, and the non-tumor mask region is set to 0 as the tumor main body input. In addition, the smallest bounding cube is expanded outward by 20 pixels, and the tumor mask region is set to 0 as the peripancreatic anatomical context input.

[0010] Preferably, a feature extraction network is used to extract features from the image data, including: The feature extraction network decomposes the preprocessed pancreatic cancer image data into three dedicated input branches, each followed by a feature extraction module. A multi-scale interactive attention fusion module is designed to realize a specialized attention mechanism and multi-scale feature fusion, emphasizing the differential and refined feature extraction of the pancreatic cancer tumor body, boundary, and peripancreatic anatomical structure.

[0011] Preferably, there are three dedicated input branches, each followed by a feature extraction module, including: Tumor main body branch: The input data is the tumor main body input. When using the feature extraction model for feature extraction, firstly, 3D depthwise separable convolution (DS3DConv) is used to extract tumor main body features, then the channel attention mechanism is used to enhance the tumor channel features, and finally, the tumor channel features are adaptively pooled and the tumor main body features are output. Tumor-peripancreatic junction branch: The input data is the tumor-peripancreatic junction. When extracting features using a feature extraction model, 3D dilated convolution is used to focus on tumor boundary details; then, a spatial attention mechanism is used to enhance boundary region information; finally, the encoder of the Transformer module models the global relationship between the boundary region and the peripancreatic tissue. Finally, adaptive pooling of global relationships is performed to obtain boundary features. Peripancreatic Anatomical Structure Context Branch: The input data is the peripancreatic anatomical context. When using a feature extraction model for feature extraction, firstly, features are extracted using multi-scale convolution kernels and concatenated to obtain peripancreatic structural features; then, the Transformer is used to encode the anatomical dependencies of the peripancreatic structures; finally, adaptive pooling of the anatomical dependencies is used to obtain anatomical context features.

[0012] Preferably, the multi-scale interactive attention fusion module implements a specialized attention mechanism and multi-scale feature fusion, including: First, the interaction weights of the interaction attention among the boundary features, tumor main features, and anatomical context features corresponding to the tumor-peripancreatic border input, tumor main input, and peripancreatic anatomical context input are calculated, and then feature fusion is performed to obtain the features. ; Then, a multi-scale gated attention mechanism is added to adjust the feature scale: Finally, the current features Dimensionality reduction to obtain image data features : Presentation layer normalization processing, Indicates fusion features, symbol Indicates splicing, Let represent the learnable weight matrix of the gating mechanism. This represents the Sigmoid activation function. and This represents the weights and bias parameters used in the dimensionality reduction process. express Activation function.

[0013] Preferably, clinical data features are extracted, including: Interactive feature construction and statistical feature matrix construction are used to maximize the retention of core features of clinical information, and deep learning feature encoding is used for further optimization to make the feature representation more adaptable to subsequent classification tasks, ultimately forming high-dimensional clinical data features. Constructing the statistical feature matrix includes: extracting the central tendency and distribution characteristics of patients' biomarkers and tumor indicators from normalized clinical data to calculate statistical features; the statistical features of all patients form the statistical feature matrix. The construction of interactive features includes: first, calculating interactive variables, specifically calculating the comprehensive biomarker index feature, inflammation burden index feature, tumor burden index feature, metabolic-inflammatory comprehensive index feature, biomarker ratio, immune-inflammatory ratio, biomarker index transformation, and log-transformed tumor burden index; then, forming interactive features from all calculated interactive variables.

[0014] Preferably, the features extracted from the electronic medical record text data include: We use medical text embedding, entity recognition, relation extraction, and deep learning coding methods to extract features from electronic medical record text data, forming the features of electronic medical record text data.

[0015] Preferably, a multimodal feature fusion network is used to perform multimodal fusion on the extracted image data features, clinical data features, and electronic medical record text data features to obtain multimodal fused features, including: Designing an attention matrix guided by medical knowledge ,in, , and The importance coefficients between modalities are determined a priori by clinical statistical analysis. The correlation between image and clinical data is α, the correlation between image and electronic medical record text data is β, and the correlation between clinical and electronic medical record text data is γ. Based on attention matrix Feature fusion is performed in the cross-modal bidirectional adaptive attention fusion module: First, a linear transformation is performed on each modality to achieve the initial projection of modal features; then, cross-modal bidirectional attention calculation is performed, with each modality as the query and the other modalities as the key and value, to calculate cross-modal attention interaction; next, intra-modal attention gating fusion is performed, that is, a gating coefficient is calculated for each modality, and the original features and cross-modal features are adaptively fused; finally, cross-modal feature aggregation is performed, that is, the three fused modal features are concatenated, and then the Transformer Encoder is used to capture the high-level semantic relationship between the three modalities as a whole to obtain the final multimodal fused features.

[0016] Preferably, the Transformer is used as the prognostic classification model to predict the prognostic status of pancreatic cancer based on multimodal fusion features.

[0017] To achieve the above-mentioned objectives, the embodiments also provide a multimodal pancreatic cancer prognostic classification and prediction system based on medical knowledge, comprising: The preprocessing unit is used to preprocess the acquired image data, clinical data, and electronic medical record text data. The feature extraction unit is used to extract image data features, clinical data features, and electronic medical record text data features from the preprocessed data. The feature fusion unit is used to perform multimodal fusion of extracted image data features, clinical data features, and electronic medical record text data features to obtain multimodal fused features; The classification prediction unit is used for pancreatic cancer prognostic classification prediction based on multimodal fusion features.

[0018] Compared with the prior art, the beneficial effects of the present invention include at least the following: This invention addresses the shortcomings of existing technologies, such as low utilization of multimodal data, weak medical knowledge integration capabilities, and limited predictive performance, by integrating medical text information from medical images, clinical data, and electronic medical records. The invention employs an AI-powered large language model to parse unstructured medical text, extracting key clinical information from electronic medical records, pathology reports, and physician diagnostic records. This information is then deeply integrated with image features and structured clinical indicators to improve the accuracy of NTR classification.

[0019] This invention also aims to improve the feature interaction mechanism of existing multimodal models and address the problem of insufficient information fusion between data sources. Existing methods typically employ simple feature concatenation or attention mechanisms for data fusion, failing to effectively model cross-modal relationships between medical images, structured clinical data, and medical text data, resulting in insufficient information utilization and impacting the model's predictive ability. This invention optimizes cross-modal feature learning methods, enabling AI to fully understand and utilize various types of data, thereby improving the accuracy of predicting pancreatic cancer prognosis.

[0020] This invention employs technologies such as image adaptive attention, medical text embedding (ChatGPT), and cross-modal feature fusion to optimize the entire process from data preprocessing, feature extraction, fusion computation to classification prediction, ensuring the model's efficiency and robustness. Traditional pancreatic cancer prognostic assessment mainly relies on imaging or single clinical indicators, while this invention, by fusing multi-source heterogeneous data, makes the prediction results more medically interpretable, helping clinicians develop personalized treatment plans. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a multimodal pancreatic cancer prognostic classification and prediction method based on medical knowledge, provided in the embodiments. Figure 2 This is a schematic diagram of the structure of PSCMB-Net, a feature extraction network for pancreatic cancer imaging characteristics, provided in the embodiment. Figure 3 This is a schematic diagram of the structure of the multimodal feature fusion network PKMCA-Net provided in the embodiment.

[0023] Figure 4 This is a schematic diagram of the structure of the multimodal pancreatic cancer prognostic classification and prediction system based on medical knowledge provided in the embodiment. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0025] The inventive concept of this invention is as follows: Currently, prognostic prediction of pancreatic cancer mainly relies on imaging examinations, clinical indicators, and biomarker analysis. However, due to the high heterogeneity and complex pathological characteristics of pancreatic cancer, prediction methods based on a single data source have limitations and are difficult to accurately assess a patient's tumor risk and disease progression. Furthermore, while existing multimodal learning methods attempt to combine imaging and clinical data, most are limited to the fusion of structured data, failing to fully utilize unstructured medical text data such as electronic medical records, pathology reports, and imaging descriptions. This limits the model's capabilities in knowledge reasoning and decision support, affecting the accuracy and clinical interpretability of prognostic predictions.

[0026] To this end, embodiments of the present invention provide a multimodal pancreatic cancer prognostic prediction and classification scheme driven by medical images, clinical information, and electronic medical records that integrates AI large language models. Through innovative data fusion and knowledge reasoning strategies, it improves the accuracy and clinical applicability of pancreatic cancer NTR classification, providing stronger technical support for precision medicine.

[0027] like Figure 1 As shown in the embodiment, a multimodal pancreatic cancer prognostic classification and prediction method based on medical knowledge is provided, including the following steps: S1 preprocesses the collected image data, clinical data, and electronic medical record text data.

[0028] In this embodiment, the multimodal data involved includes medical imaging data, clinical data, and electronic medical record text data. Specifically, for imaging data, CT scan images of the patient's pancreas are acquired and stored in DICOM format, including a precise labeled mask file for the pancreatic tumor, also stored in DICOM format. The labeled file and the CT file are linked by file name. For clinical data, structured clinical information of the patient is collected, including age, gender, height, weight, and clinical laboratory indicators (such as CA19-9, CEA, Glb, PLR, AST, AGR, etc.). The clinical data also includes the final classification label of tumor risk NTR. For electronic medical record text data, unstructured medical text of the patient is collected, including doctor's diagnostic reports, pathology reports, postoperative follow-up records, etc., to obtain personalized medical descriptive information for the patient. The above three modalities of data are interconnected through the patient's personal ID.

[0029] The embodiments also include preprocessing of the collected multimodal data, specifically including: For CT imaging data of the pancreas, a multi-step preprocessing technique is employed to improve the quality and consistency of the image data and extract key lesion regions for model training and prediction. This step includes window width and level adjustment and normalization, ROI segmentation based on tumor masks, and data augmentation.

[0030] (a-1) Window width and window level adjustment and normalization: The raw grayscale values ​​of CT image data are represented by Hounsfield Units (HU), which have a wide dynamic range. Directly inputting them into a deep learning model may lead to training instability. Therefore, this invention adjusts and normalizes the window width and window level of the CT image data to enhance the contrast of the target region and improve the model's feature extraction capability. The HU value range for pancreatic tissue and tumor regions in CT images is relatively fixed; therefore, this invention uses a window width of WW=350 and a window level of WL=50.

[0031] (a-2) ROI segmentation based on pancreatic tumor mask CT images typically contain a large amount of irrelevant information, such as surrounding tissue and background areas. To input the deep learning model PSCMB-Net for pancreatic cancer lesion CT images designed in this invention, this invention employs a ROI segmentation method based on the tumor mask. This method accurately extracts the minimum bounding cube of the tumor region and expands it outwards by 20 pixels as the first type of CT input. Simultaneously, the image with the minimum bounding cube expanded outwards by 20 pixels and the non-tumor mask region set to 0 is used as the second type of CT input. Furthermore, the image with the minimum bounding cube expanded outwards by 20 pixels and the tumor mask region set to 0 is used as the third type of CT input. Since the segmented CT images vary in size from patient to patient, to standardize the input size, this invention uses trilinear interpolation to uniformly scale the ROI region to a fixed size (e.g., 96×96×8) to accommodate the input of the deep learning model.

[0032] (a-3) Data Augmentation To improve the model's generalization ability, this invention introduces data augmentation methods in image data preprocessing. By performing different transformations on the image data, multiple variants are generated to enhance the model's robustness. These include rotation, translation, cropping, and affine transformation.

[0033] A standardized data preprocessing workflow was designed for clinical data from pancreatic cancer patients to improve data quality, eliminate outliers, and optimize the representation of input data, ensuring that the model can fully utilize clinical information for prognostic classification and prediction. Specifically, this includes: (b-1) Data cleaning and standardization First, missing values ​​are handled: patient information is directly deleted for key variables that are missing, while ordinary variables are imputed using linear regression or mean. Second, outlier detection is performed using box plot analysis. Outliers exceeding the specified range are confirmed using clinical knowledge, and if deemed abnormal, they are pruned or imputed using medical methods. Finally, data normalization is performed to bring all numerical data to a single dimension.

[0034] (b-2) Feature extraction of numerical data Clinical data contains variables of multiple categories, which are converted from categorical variables to numerical format using one-hot encoding.

[0035] To address the unstructured nature of electronic medical record text data, a systematic preprocessing method is employed, including text cleaning and normalization, medical entity recognition (NER), semantic embedding and feature extraction, information extraction and structure transformation, etc., to improve data quality and optimize information representation, making it suitable for prognostic classification models for pancreatic cancer prognosis prediction.

[0036] (c-1) Text cleaning and normalization Because electronic medical record texts may contain various non-standard expressions, such as spelling errors entered by doctors, redundant punctuation, and irrelevant characters, regular expression matching rules are first used to remove irrelevant characters, making the text clearer. Next, considering that medical texts may contain a large number of stop words that do not contribute to semantics, a medical-specific stop word list is used to remove these meaningless words to reduce noise and improve the effectiveness of feature extraction. Finally, synonym normalization is performed. Due to different writing habits among doctors, the same medical concept may have multiple expressions; for example, "pancreatic cancer" may be described as "malignant pancreatic tumor" or "pancreatic carcinoma." This invention constructs a synonym mapping table based on medical dictionaries and related medical knowledge to unify different expressions into standard terminology.

[0037] (c-2) Medical entity recognition, information extraction and structure transformation Key information in electronic medical records (such as disease names, symptoms, treatment methods, and imaging findings) is usually stored in an unstructured format. This key information needs to be converted into a structured data format for subsequent processing by deep learning models. This invention employs an AI large-scale language model for medical entity recognition and information extraction. Through the design of prompts, the AI ​​large-scale language model accurately extracts key information such as diseases, symptoms, imaging findings, and treatment plans from electronic medical records, and stores information in a structured manner, such as: {"Examination":"Pancreatic Enhanced CT","Lesion":"Low-density mass in the head of the pancreas","Diagnosis":"Pancreatic cancer"}.

[0038] S2 extracts image data features, clinical data features, and electronic medical record text data features from the preprocessed data.

[0039] S2-1, Feature Extraction from Image Data Pancreatic cancer CT imaging exhibits distinct and unique clinical imaging characteristics compared to other cancers, including diverse lesion morphology and blurred boundaries; close relationship between the tumor and surrounding pancreatic anatomical structures; significant intratumoral heterogeneity; and high contrast between pancreatic cancer lesions and normal pancreatic tissue. Given these unique characteristics of pancreatic cancer CT imaging, traditional image analysis networks such as conventional CNNs and 3D EfficientNet struggle to accurately capture and characterize the lesion features and peripancreatic anatomical relationships of pancreatic cancer. Therefore, PSCMB-Net, a feature extraction network specifically designed for the imaging characteristics of pancreatic cancer, is proposed.

[0040] like Figure 2 As shown, the network architecture PSCMB-Net decomposes the preprocessed pancreatic cancer CT image data into three dedicated input branches, each followed by a feature extraction module. A multi-scale interactive attention fusion module is designed to realize a specialized attention mechanism and multi-scale feature fusion, emphasizing the differential and refined feature extraction of the pancreatic cancer tumor body, boundary, and peripancreatic anatomical structures.

[0041] The PSCMB-Net network architecture takes three sets of image tensors as input to the same region from different feature perspectives as input image data. Features are extracted from these image tensors through three input branches, including: Tumor-Branch: The input data is a 20-pixel image of the smallest outer cube of the tumor region, expanded outward from the CT image. It contains only information about the tumor region, with all other locations set to zero to capture the internal texture and heterogeneity of the tumor. In other words, the CT image is only visible within the tumor region, with all other areas set to zero. This is recorded as the tumor-branch input. When using a feature extraction model, firstly, 3D depthwise separable convolution (DS3DConv) is used to extract tumor ontology features. : ,in, We represent a self-regularized non-monotonic activation function; then, a channel attention mechanism (SE-Net) is used to enhance tumor channel features. : Finally, adaptive pooling ( AdaptivePool3D Tumor pathway characteristics And output the main feature vector of the tumor. : .

[0042] Tumor-peripancreatic boundary branch: The input data is an image of the tumor region expanded 20 pixels outward from the smallest outermost cube CT image, fully preserving the boundary information between the tumor and the surrounding pancreatic tissue, emphasizing the boundary features between the tumor and the peripancreatic tissue, and is denoted as the tumor-peripancreatic boundary input. When using a feature extraction model, 3D atrous convolution (Atrous 3DConv) is first employed to focus on the details of the tumor boundary. : Then, a spatial attention mechanism (CBAM) is used to enhance the information in the boundary region. : Next, the global relationship between the boundary region and the peripancreatic tissue is modeled using the encoder of the Transformer module. : Finally, adaptive pooling ( AdaptivePool3D Global Relationships And obtain boundary features : .

[0043] Context-Branch of Peripancreatic Anatomy: The input data is an image of the tumor region expanded 20 pixels outward from the smallest outer cube CT image and zeroed out in the tumor region, preserving information about the peripancreatic tissues (such as the pancreatic duct and blood vessels), capturing the anatomical background structures in which the tumor is located, and recorded as the peripancreatic anatomical context input. When using a feature extraction model for feature extraction, firstly, features are extracted using multi-scale convolution kernels of 1×1×1, 3×3×3, and 5×5×5 (Conv), and then concatenated (Concat) to obtain the peripancreatic structural features. : Then, the Transformer was used to encode the anatomical dependencies of the peripancreatic structures. : Finally, adaptive pooling anatomical dependencies. And obtain anatomical context features : .

[0044] In this embodiment, when designing the Multi-Scale Interactive Attention Fusion (MIAF) module, it was considered that the relationship between pancreatic cancer tumors and their surrounding tissues is unidirectional and well-defined. The pancreatic tumor itself is the core area of ​​focus and the core reference for final classification and prediction, while the boundaries and peripancreatic context represent the information background of the tumor's environment. They are mainly used to assist in understanding the tumor's state, invasion status, and anatomical relationships. In other words, in imaging and clinical practice, the most crucial issue is to observe and analyze the relationship between the tumor and its surrounding tissues from the perspective of tumor ontology. Therefore, based on clinical and imaging understanding, the attention mechanism should be designed with tumor ontology features as the observer (Query) to explore and focus on information about its related surrounding tissues. The surrounding tissues (boundaries + context) provide the Key (the object of attention) and Value (the features used to represent the tumor after attention) to enhance and supplement the tumor's main features. This design not only reflects the lesion-centric medical observation thinking in clinical practice and conforms to the medical logic of pancreatic cancer CT imaging features, but also has clear logic and is targeted within the Transformer mechanism.

[0045] Because of the clear image-clinical relationship between the pancreatic cancer tumor and its surrounding tissues, and the differences in information density and characteristics across different regions, this design becomes an innovative, data-driven, and medically logically sound choice. The Query, generated from the tumor ontology, represents the core task objective, reflecting a tumor-centric approach; the Key and Value, composed of the tumor boundary and peripancreatic anatomy, reflect the logic of surrounding tissues providing supporting information. This effectively achieves unidirectional interaction between tumor information and background information, enhancing the model's interpretability, performance, and medical significance.

[0046] Therefore, in the Multi-Scale Interactive Attention Fusion (MIAF) module, to further highlight pancreatic cancer features, the cross-attention weights among the three CT image features are calculated: Then feature fusion is performed. ： ; Next, a multi-scale gating attention mechanism is added to adjust the feature scale: Finally, the dimensionality of the current feature vector is reduced to obtain the final CT image data features. : in, , , These represent the weights used to calculate K, Q, and V, respectively. Representing feature dimension, Presentation layer normalization processing, Indicates fusion features, symbol Indicates splicing, Let represent the learnable weight matrix of the gating mechanism. This represents the Sigmoid activation function. and This represents the weights and bias parameters used in the dimensionality reduction process. express Activation function.

[0047] In the image data feature extraction section, to address the specificity of the input data, an innovative three-branch input data design was proposed to differentiate and decompose pancreatic cancer lesion regions, accurately capturing specific information about the lesion, its boundaries, and the anatomical background. An innovative multi-scale interactive attention fusion module (MIAF) achieves precise fusion of three features—the lesion itself, its boundary, and the anatomical background—through Cross-Attention and multi-scale gated attention, making the expression of pancreatic cancer image features more accurate and discriminative.

[0048] S2-2, Clinical Data Feature Extraction In the clinical data feature extraction stage, this invention employs feature interaction construction and statistical feature matrix building for structured data such as patient numerical data, biomarkers, pathological stages, histological information, and treatment information. This maximizes the retention of core clinical information features, and further optimizes the feature representation through deep learning feature encoding (MLP encoding) to make the feature representation more adaptable to subsequent classification tasks, ultimately forming high-dimensional clinical data features. .

[0049] (1) Input Clinical Data Tensor: The clinical data provided by the user contains multiple fields, mainly divided into: patient basic information, biomarkers, pathological information, and final classification labels. Let the clinical data input be... Where B is the Batch Size and F is the feature dimension, which is the total dimension of all features.

[0050] (2) Constructing a statistical feature matrix: Calculate statistical features to extract the central tendency and distribution characteristics of patients' biomarkers and tumor markers from the normalized clinical data, thereby enhancing the interpretability of the data. First, assume the patient... The original numerical variables are: Then, the mean, standard deviation, interquartile range (IQR), maximum value, minimum value, and Z-score are calculated as statistical features. The formula for calculating the Z-score is as follows: , Indicates the first The first patient's One clinical variable, This represents the mean of the variable across all patients. This represents the standard deviation of the variable across all patients.

[0051] Then calculate the statistical characteristic matrix. Includes the following information: Each row represents a patient, and each column represents a statistical feature. After calculating all statistical features, a statistical feature matrix is ​​formed. ,in, It represents the total dimension or total number of statistical features.

[0052] (3) Feature Interaction Construction: Since a single variable may not be sufficient to fully describe the patient's survival prognosis, this invention employs a feature interaction construction method, combining multiple variables to generate new features and constructing new features based on existing variables to improve data interpretability. First, variable interaction calculations are performed, specifically including: Biomarker composite index characteristics This characteristic reflects the overall level of the patient's biomarkers: Inflammation burden index characteristics This feature measures the level of inflammation (leukocytes + C-reactive protein - albumin): Tumor burden index characteristics This feature measures a patient's tumor biological burden: Metabolic-inflammatory composite index characteristics This feature comprehensively assesses the patient's metabolic and inflammatory status: Biomarker ratio This ratio can measure the relative influence of different biomarkers, avoiding the influence of the absolute value of CA19-9 on the classification results. immune-inflammatory ratio This ratio reflects the relative relationship between a patient's immune function and inflammation level: Biomarker index transformation This exponential transformation is used to emphasize the impact of high concentrations of CA19-9 on patient prognosis: Log-transformed tumor burden index To reduce the impact of extreme values ​​on the model and improve data stability: , , These represent the patient's carcinoembryonic antigen (CEA), carbohydrate antigen 19-9 (CA19-9), and carbohydrate antigen 125 (CA125) test values, respectively. , , These represent the patient's white blood cell count, C-reactive protein, and albumin levels, respectively. This indicates the patient's body mass index; This represents the characteristics of the calculated tumor burden index.

[0053] Then, the interaction feature matrix is ​​constructed; that is, after all the generated features are constructed, a new interaction feature matrix is ​​formed. : in, The number of interactive features is determined by the number of features constructed.

[0054] (4) Deep Learning Feature Encoding: Deep learning feature encoding is performed using a multilayer perceptron (MLP) to enhance feature representation capabilities and adapt to subsequent multimodal classification models. Specifically, this includes: First, concatenate the statistical feature matrix and the feature interaction matrix to obtain the concatenated matrix. ,in, , .

[0055] Then perform MLP encoding, considering... Therefore, a two-layer fully connected neural network (MLP) is used to extract features, and the final output encoded features are used as the final extracted clinical features. ,

[0056] In the clinical feature extraction section, statistical feature construction (mean, variance, IQR) is used to supplement univariate information and enhance feature expressive power. Ratio, exponential, and interaction feature construction improves data usability and compensates for the limited clinical information. Deep learning feature encoding is employed using MLP to enhance the high-dimensional expressive power of features, making them suitable for subsequent classification tasks.

[0057] 2-3, Feature extraction of electronic medical record text data: After preprocessing the electronic medical record text data, this invention employs an AI large language model for deep semantic feature extraction of unstructured medical record text data. Because electronic medical record text data contains complex medical content such as patient diagnostic information, imaging reports, pathology descriptions, and treatment records, traditional word frequency statistics or TF-IDF methods are insufficient to fully capture semantic information. Therefore, this invention uses Medical Text Embedding, Entity Recognition (NER), Relation Extraction (RE), and deep learning encoding methods for feature extraction, ultimately forming high-dimensional electronic medical record text data features. This ensures consistency in subsequent multimodal fusion.

[0058] (1) Input electronic medical record text data: Suppose that after data preprocessing, the text dataset of electronic medical record text data is represented as: ,in, Indicates batch size. Indicates the first The electronic medical record text of each patient includes unstructured text content such as imaging reports, pathology descriptions, and treatment plans.

[0059] (2) Text feature extraction based on AI large language model: Since electronic medical records contain a large number of professional terms and unstructured medical descriptions, this invention uses AI large language model to extract deep semantic information.

[0060] First, medical text embedding is performed, which involves semantic vector transformation. ChatGPT is used to convert the electronic medical record text into a high-dimensional semantic vector. : ChatGPT-Embedding( ), where ChatGPT-Embedding( ) represents the text encoder of the ChatGPT large language model. It is the embedding vector of the medical record text. For the dimensions of text embedding; ChatGPT computes deep semantic features of medical record text through multi-layer Transformers. : TansformerLayer ( Each Transformer layer contains a multi-head attention mechanism and a feedforward neural network to model long-range dependencies in the text.

[0061] Then, medical entity recognition is performed: ChatGPT is used for medical entity recognition (NER), automatically extracting key information such as disease name, symptoms, imaging findings, pathological diagnosis, treatment methods, medications, and examination methods from electronic medical record text, and converting it into vector representations that can be used for deep learning. Specifically, this includes: For medical entity recognition: ChatGPT is used to recognize medical entities. ChatGPT-NER( ), Medical entities can be identified by writing prompt commands. Entity example: { Disease: Pancreatic cancer Imaging findings: "Low-density mass in the head of the pancreas" Treatment method: Surgery + Chemotherapy "Biomarker": "Elevated CA19-9" } To maintain data consistency, this invention constructs a fixed-dimensional entity matrix to ensure a consistent structure even when some entity categories are empty. Entity embedding for each category Calculated by ChatGPT. Missing entity categories are filled with zero vectors to maintain a fixed dimension of the feature matrix.

[0062] For medical entity vectorization: the ChatGPT large language model is used to calculate the semantic vector of each medical entity. : ,in For the first A medical entity, For the vector representation of the entity, For embedding dimensions, this invention defines 256 dimensions. Example: "Pancreatic cancer" → ChatGPT- → Vector "Surgery + Chemotherapy" → ChatGPT- → Vector Ultimately, the vectors of all entities are combined to form a medical entity feature matrix. : , The feature dimension embedded for the final entity. ,in This represents the maximum number of entities.

[0063] For medical entity category encoding: Since different categories of medical entities may have different levels of importance, this invention employs a category vector encoding method to embed entity category information into a vector space. An embedding weight is assigned to each category according to its entity category (e.g., "disease," "examination," "treatment"). ,in For the number of medical entity categories, Let be the category embedding matrix. For each medical entity, the category vector is: ,in This is a vector representing the entity category. The final medical entity feature vector. ,in , This represents the final entity feature dimension. In this invention, it is 256 dimensions.

[0064] Next, entity relationships are collected and modeled: there are complex relationships between medical entities in electronic medical records, such as: disease-imaging manifestations (e.g., "pancreatic cancer → low-density mass in the head of the pancreas"); disease-diagnostic methods (e.g., "pancreatic cancer → confirmed by tissue biopsy"); disease-treatment plans (e.g., "pancreatic cancer → surgery + chemotherapy"); drugs-disease treatment (e.g., "gemcitabine → pancreatic cancer"); examination-imaging manifestations (e.g., "CT enhanced scan → pancreatic mass"). This invention uses ChatGPT for relationship modeling, automatically extracting the logical relationships between these entities and converting them into structured vector representations for subsequent deep learning model processing.

[0065] For relation extraction, ChatGPT is used to parse electronic medical record text. Extract the logical relationships between entities: ;in These are entity pairs and their relationship categories extracted from text. Example: { Entity pair: ["pancreatic cancer", "surgery + chemotherapy"], Relationship Type: Treatment Plan } Relationship Classification ChatGPT is used to assign a relation label to each entity pair: pathology-imaging findings (e.g., "pancreatic cancer" → "low-density mass in the head of the pancreas"), diagnosis-examination method (e.g., "tissue biopsy" → "confirmed pancreatic cancer"), disease-treatment method (e.g., "pancreatic cancer" → "neoadjuvant chemotherapy"), drug-disease (e.g., "gemcitabine" → "pancreatic cancer"); the relation labels are then vectorized. , One-hot encoding for relation categories, This represents the number of relation categories.

[0066] For relation vectorization: a relation embedding method is used to convert the extracted relation information into high-dimensional vectors, ensuring that it can be used for neural network computation. Specifically, a relation transformation matrix is ​​used. Perform relational mapping on entity embeddings: ,in, These are the head entity and the tail entity, This represents vector concatenation. This is the relational mapping weight matrix.

[0067] By fusing relation category features with entity semantic features, the final medical entity relation feature vector is obtained. ,in The dimension of the final relation vector is 256 in this invention.

[0068] in, To encode the relation categories one-hot, To map to For projection matrices of the same dimension, the symbol + indicates element-wise addition.

[0069] Finally, feature fusion is performed to obtain the electronic medical record text feature vector. This invention extracts medical entity feature vectors during the medical entity recognition (NER) process. In the process of entity relationship modeling (RE), medical entity relationship feature vectors were extracted. To form the final electronic medical record text feature vector. This invention employs self-attention weighted fusion to maximize information retention and optimize the semantic expressive power of features. Specifically, it includes: For the input medical entity feature vector Relationship feature vectors with medical entities To improve feature representation capabilities, this invention employs a self-attention mechanism for learning. and The interaction information between them is used to calculate their respective weight scores. and And adaptively adjust their respective weights and ,Right now , The weighted fusion features are used as features of the case medical record text data. : ,in, Keep the dimensions unchanged.

[0070] The advantage of the self-attention weighted feature fusion method is that the self-attention mechanism can learn the contribution weights of different features, making feature fusion more robust.

[0071] S3 performs multimodal fusion on the extracted image data features, clinical data features, and electronic medical record text data features to obtain multimodal fused features.

[0072] After completing the extraction of image data features Clinical data characteristics Features of electronic medical record text data Then, these features need to be fused to form the final multimodal features. This provides input for subsequent classification models.

[0073] Since imaging data, clinical data, and electronic medical record text data of pancreatic cancer patients represent different dimensions of medical information, and each modality contributes uniquely to patient prognosis and complements the others, this invention, based on prior medical knowledge, designs a feature fusion architecture (PKMCA-Net architecture) suitable for the current multimodal pancreatic cancer data. This is a multimodal feature fusion network architecture for pancreatic cancer based on prior medical knowledge guidance and cross-modal bidirectional adaptive attention. Figure 3 As shown.

[0074] (1) Design an attention matrix guided by medical knowledge ,in, , and The importance coefficients between modalities are determined a priori by clinical statistical analysis. The correlation between image and clinical data is α, the correlation between image and electronic medical record text data is β, and the correlation between clinical and electronic medical record text data is γ.

[0075] Medical knowledge-guided attention matrix The weighting coefficients can be explicitly determined based on clinical consensus regarding the importance of different modalities in the clinical diagnosis and prognostic assessment of pancreatic cancer. In this invention, based on clear prior medical knowledge, the weight of "image-clinical data correlation" is assigned as 0.8, "image-electronic medical record text data correlation" as 0.7, and "data-electronic medical record text data correlation" as 0.9. The high correlation between clinical data and text data (0.9) reflects the close link between structured clinical indicators and unstructured text reports; the secondary correlation between image data and clinical data (0.8) emphasizes the core role of image features in clinical assessment; and the secondary correlation between image data and text data (0.7) clearly indicates an important but slightly weak complementary relationship between image information and text reports. Therefore, the following is explicitly given: Medical knowledge-guided attention matrix .

[0076] (2) Based on attention matrix Feature fusion is performed in the cross-modal bidirectional adaptive attention fusion module: First, a linear transformation is performed on each mode to achieve the initial projection of the modal features: in, , , These are the weight parameters for the corresponding features. , , The Q values ​​for the three models are shown below. and For projection weights, and These are the weight values ​​for all modes.

[0077] Then, cross-modal bidirectional attention is calculated, with each modality as the query and the other modalities as the key and value, to calculate the cross-modal attention interaction. This step reflects the active and passive interactions between modalities. Image modality: ,in yes Image modality corresponding row ( That is, ([1, 0.8, 0.7]); Similarly, the clinical modality and the text modality are: in and They are respectively Corresponding rows for clinical modality and text modality ( That is, [0.8, 1, 0.9] and [0.7, 0.9, 1]. In this way, a two-way interaction of active attention and passive fusion of each modality feature is achieved.

[0078] Next, intra-modal attention-gated fusion is performed, that is, a gating coefficient is calculated for each modality, and the original features and cross-modal features are adaptively fused: Taking image modality as an example: gating coefficient Next, the image modal feature fusion results are as follows: ; The clinical and text modalities are analogous: ; Finally, cross-modal feature aggregation is performed, which involves concatenating the fused features from the three modalities and then using TransformerEncoder to capture the high-level semantic relationships between the three modalities as a whole. The final multimodal fusion features are obtained. .

[0079] This invention features a novel multimodal feature fusion architecture that is significantly different from existing multimodal fusion methods. Specifically designed for the medical scenario of pancreatic cancer prognosis prediction, it has distinct medical logic, data-driven characteristics, and efficient adaptive interactive feature fusion capabilities. It significantly improves the accuracy and interpretability of pancreatic cancer prognostic classification, and has clear clinical application value and strong patent protection potential.

[0080] Medical prior knowledge attention guidance: Determine the initial weight matrix between modalities through expert priors or clinical statistics, and clarify the medical interpretive significance of features in the fusion; Cross-modal bidirectional attention mechanism: Innovatively realizes the coexistence of active attention and passive interaction between modalities, enabling each modality to adaptively select and integrate information from other modalities, reflecting the real medical logic and data characteristics; Adaptive fusion with gating mechanism: further realizes dynamic weight allocation of features and automatically balances the contribution of original modality information and fused information; Transformer Encoder cross-modal aggregation: further captures high-order semantic relationships between modalities, improving the ability to express fused features and generalization performance.

[0081] S4, pancreatic cancer prognostic classification prediction based on multimodal fusion features.

[0082] After completing the multimodal feature fusion, the fused multimodal features are utilized. This invention performs a binary classification of pancreatic cancer prognosis. It employs a Transformer model for classification, ensuring high accuracy, robustness, and strong generalization ability.

[0083] The goal of the classification model is to predict the prognostic risk category (binary classification) of pancreatic cancer patients: , The prediction category is 0 (low risk, 1 high risk). It is a classifier. This invention further employs a Transformer for classification, utilizing a self-attention mechanism to identify key patterns among multimodal features.

[0084] Multimodal feature fusion using linear transformation Feed into Transformer: ; Calculate the self-attention weights based on QKV: ; Calculate the attention-weighted representation: , This is the representation after attention.

[0085] This invention employs a multi-head attention mechanism, using multiple independent attention heads to learn different feature spaces: Where h is the number of attention heads (h=8 in this invention),

[0086] This invention employs a two-layer feedforward neural network after the Transformer computation. Used for feature transformation The data is fed into a fully connected layer for binary classification prediction, and the final binary classification probability is calculated. , To output the weight matrix, For bias, σ ,like Then it is classified as high risk (1), if It is then classified as low risk (0).

[0087] In the classification and prediction stage, feature learning is performed based on Transformer to improve the interactive expressive power of multimodal features. Multi-head attention (MHSA) is used to enhance the global dependencies between features of different modalities. Feedforward neural networks (FFN) are employed to optimize feature transformation and improve the expressive power of the classifier.

[0088] This invention effectively improves the accuracy of prognostic prediction for pancreatic cancer patients by constructing a multimodal data fusion and deep learning classification model. Compared with traditional single-modal methods, this invention achieves deep fusion of imaging, clinical information, and electronic medical record text data, fully exploring the complementary information of each modality's features, thus giving the model stronger predictive capabilities and clinical applicability.

[0089] Experimental results demonstrate that the method of this invention has significant advantages in pancreatic cancer prognostic classification. On a real clinical dataset from a tertiary hospital, the multimodal fusion method of this invention improved the model's AUC (area under the curve) to 0.91, a significant improvement in classification performance compared to using image data alone (AUC = 0.82) or clinical data alone (AUC = 0.79). Furthermore, this invention employs a cross-modal Transformer structure, effectively enhancing the interactive expressive ability between different modal features, enabling the model to maintain high prediction accuracy even with varying data loss. Experimental data show that even with missing image or text data, the model can still maintain a classification accuracy of over 86%, demonstrating greater stability compared to traditional single-modal models (which drop to 75%) and exhibiting strong generalization ability.

[0090] From a social and economic perspective, this invention can improve the early risk assessment capabilities for pancreatic cancer patients, providing more precise stratified management for high-risk patients, thereby enhancing the targeted nature of clinical treatment and reducing unnecessary treatment costs and resource waste. Statistics show that early and accurate assessment of pancreatic cancer can increase the 5-year survival rate by 15%-20%, while reducing treatment costs by approximately 30%-40%. The implementation of this invention helps optimize the allocation of medical resources, reduce unnecessary hospitalizations and surgeries, lower the economic burden on patients, and promote the development of intelligent medical decision-making.

[0091] In summary, this invention overcomes the limitations of single-modal methods, realizes the fusion of multimodal medical data and intelligent prediction of pancreatic cancer prognosis, improves the accuracy and reliability of clinical decision-making, and has significant technical advantages, economic benefits and social value, providing a scientific basis for individualized precision treatment of pancreatic cancer.

[0092] Based on the same inventive concept, such as Figure 4 As shown in the figure, this embodiment of the invention also provides a multimodal pancreatic cancer prognostic classification and prediction system 40 based on medical knowledge, including: a preprocessing unit 41, a feature extraction unit 42, a feature fusion unit 43, and a classification prediction unit 44. The preprocessing unit 41 is used to preprocess the collected image data, clinical data, and electronic medical record text data; the feature extraction unit 42 is used to extract image data features, clinical data features, and electronic medical record text data features from the preprocessed data; the feature fusion unit 43 is used to perform multimodal fusion of the extracted image data features, clinical data features, and electronic medical record text data features to obtain multimodal fused features; and the classification prediction unit 44 is used to perform pancreatic cancer prognostic classification and prediction based on the multimodal fused features.

[0093] It should be noted that the multimodal pancreatic cancer prognostic classification and prediction system based on medical knowledge provided in the above embodiments should be illustrated using the above-described functional module division as an example when performing multimodal pancreatic cancer prognostic classification and prediction. The functions described above can be assigned to different functional modules as needed, that is, the internal structure of the terminal or server can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the multimodal pancreatic cancer prognostic classification and prediction system based on medical knowledge provided in the above embodiments and the embodiment of the multimodal pancreatic cancer prognostic classification and prediction construction method based on medical knowledge belong to the same concept. For details of its specific implementation process, please refer to the embodiment of the multimodal pancreatic cancer prognostic classification and prediction method based on medical knowledge, which will not be repeated here.

[0094] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal pancreatic cancer prognostic classification and prediction method based on medical knowledge, characterized in that, Includes the following steps: Preprocess the collected image data, clinical data, and electronic medical record text data; Extract image data features, clinical data features, and electronic medical record text data features from the preprocessed data; Multimodal fusion was performed on the extracted image data features, clinical data features, and electronic medical record text data features to obtain multimodal fused features; Prognostic classification and prediction of pancreatic cancer based on multimodal fusion features.

2. The multimodal pancreatic cancer prognostic classification and prediction method based on medical knowledge according to claim 1, characterized in that, Preprocessing of the collected image data, clinical data, and electronic medical record text data includes: For the image data, window width and level were adjusted and normalized, ROI segmentation was performed based on the pancreatic tumor mask, and data augmentation was carried out; For clinical data, data cleaning and standardization, as well as feature extraction of numerical data, are performed. For electronic medical record text data, we perform text cleaning and standardization, medical entity recognition, information extraction and structure transformation; Among them, ROI segmentation based on pancreatic tumor masks includes: The smallest bounding cube of the tumor region is extracted based on the pancreatic tumor mask, and the image is expanded outward by 20 pixels as the tumor-peripancreatic junction input. At the same time, the smallest bounding cube is expanded outward by 20 pixels, and the non-tumor mask region is set to 0 as the tumor main body input. In addition, the smallest bounding cube is expanded outward by 20 pixels, and the tumor mask region is set to 0 as the peripancreatic anatomical context input.

3. The multimodal pancreatic cancer prognostic classification and prediction method based on medical knowledge according to claim 2, characterized in that, Feature extraction networks are used to extract features from image data, including: The feature extraction network decomposes the preprocessed pancreatic cancer image data into three dedicated input branches, each followed by a feature extraction module. A multi-scale interactive attention fusion module is designed to realize a specialized attention mechanism and multi-scale feature fusion, emphasizing the differential and refined feature extraction of the pancreatic cancer tumor body, boundary, and peripancreatic anatomical structure.

4. The multimodal pancreatic cancer prognostic classification and prediction method based on medical knowledge according to claim 3, characterized in that, Three dedicated input branches, each followed by a feature extraction module, including: Tumor main body branch: The input data is the tumor main body input. When using the feature extraction model for feature extraction, firstly, 3D depthwise separable convolution (DS3DConv) is used to extract tumor main body features, then the channel attention mechanism is used to enhance the tumor channel features, and finally, the tumor channel features are adaptively pooled and the tumor main body features are output. Tumor-peripancreatic junction branch: The input data is the tumor-peripancreatic junction. When extracting features using a feature extraction model, 3D dilated convolution is used to focus on tumor boundary details; then, a spatial attention mechanism is used to enhance boundary region information; finally, the encoder of the Transformer module models the global relationship between the boundary region and the peripancreatic tissue. Finally, adaptive pooling of global relationships is performed to obtain boundary features. Peripancreatic Anatomical Structure Context Branch: The input data is the peripancreatic anatomical context. When using a feature extraction model for feature extraction, firstly, features are extracted using multi-scale convolution kernels and concatenated to obtain peripancreatic structural features; then, the Transformer is used to encode the anatomical dependencies of the peripancreatic structures; finally, adaptive pooling of the anatomical dependencies is used to obtain anatomical context features.

5. The multimodal pancreatic cancer prognostic classification and prediction method based on medical knowledge according to claim 3, characterized in that, The multi-scale interactive attention fusion module implements specialized attention mechanisms and multi-scale feature fusion, including: First, the interaction weights of the interaction attention among the boundary features, tumor main features, and anatomical context features corresponding to the tumor-peripancreatic border input, tumor main input, and peripancreatic anatomical context input are calculated, and then feature fusion is performed to obtain the features. ; Then, a multi-scale gated attention mechanism is added to adjust the feature scale: Finally, the current features Dimensionality reduction to obtain image data features : Presentation layer normalization processing, Indicates fusion features, symbol Indicates splicing, The learnable weight matrix representing the gating mechanism. This represents the Sigmoid activation function. and This represents the weights and bias parameters used in the dimensionality reduction process. express Activation function.

6. The multimodal pancreatic cancer prognostic classification and prediction method based on medical knowledge according to claim 1, characterized in that, Extracting clinical data features, including: Interactive feature construction and statistical feature matrix construction are used to maximize the retention of core features of clinical information, and deep learning feature encoding is used for further optimization to make the feature representation more adaptable to subsequent classification tasks, ultimately forming high-dimensional clinical data features. Constructing the statistical feature matrix includes: extracting the central tendency and distribution characteristics of patients' biomarkers and tumor indicators from normalized clinical data to calculate statistical features; the statistical features of all patients form the statistical feature matrix. The construction of interactive features includes: first, calculating interactive variables, specifically calculating the comprehensive biomarker index feature, inflammation burden index feature, tumor burden index feature, metabolic-inflammatory comprehensive index feature, biomarker ratio, immune-inflammatory ratio, biomarker index transformation, and log-transformed tumor burden index; then, forming interactive features from all calculated interactive variables.

7. The multimodal pancreatic cancer prognostic classification and prediction method based on medical knowledge according to claim 1, characterized in that, Extract features from electronic medical record text data, including: We use medical text embedding, entity recognition, relation extraction, and deep learning coding methods to extract features from electronic medical record text data, forming the features of electronic medical record text data.

8. The multimodal pancreatic cancer prognostic classification and prediction method based on medical knowledge according to claim 1, characterized in that, A multimodal feature fusion network is used to fuse extracted image data features, clinical data features, and electronic medical record text data features to obtain multimodal fused features, including: Designing an attention matrix guided by medical knowledge ,in, , and The importance coefficients between modalities are determined a priori by clinical statistical analysis. The correlation between image and clinical data is α, the correlation between image and electronic medical record text data is β, and the correlation between clinical and electronic medical record text data is γ. Based on attention matrix Feature fusion is performed in the cross-modal bidirectional adaptive attention fusion module: First, a linear transformation is performed on each modality to achieve the initial projection of modal features; then, cross-modal bidirectional attention calculation is performed, with each modality as the query and the other modalities as the key and value, to calculate cross-modal attention interaction; next, intra-modal attention gating fusion is performed, that is, a gating coefficient is calculated for each modality, and the original features and cross-modal features are adaptively fused; finally, cross-modal feature aggregation is performed, that is, the three fused modal features are concatenated, and then the Transformer Encoder is used to capture the high-level semantic relationship between the three modalities as a whole to obtain the final multimodal fused features.

9. The multimodal pancreatic cancer prognostic classification and prediction method based on medical knowledge according to claim 1, characterized in that, We used Transformer as the prognostic classification model to predict the prognosis of pancreatic cancer based on multimodal fusion features.

10. A multimodal pancreatic cancer prognostic classification and prediction system based on medical knowledge, characterized in that, include: The preprocessing unit is used to preprocess the acquired image data, clinical data, and electronic medical record text data. The feature extraction unit is used to extract image data features, clinical data features, and electronic medical record text data features from the preprocessed data. The feature fusion unit is used to perform multimodal fusion of extracted image data features, clinical data features, and electronic medical record text data features to obtain multimodal fused features; The classification prediction unit is used for pancreatic cancer prognostic classification prediction based on multimodal fusion features.

Images