Gastrointestinal tumor postoperative rehabilitation effect prediction system based on multi-modal data

Abstract

The application discloses a gastrointestinal tumor postoperative rehabilitation effect prediction system based on multi-modal data, relates to the technical field of rehabilitation prediction, and comprises a multi-modal data acquisition and preprocessing module, which is used for collecting preoperative baseline data and postoperative follow-up data at multiple time points, the preoperative baseline data comprises medical image data, pathological image data and surgical record data, and the postoperative follow-up data comprises clinical examination time series data, molecular detection data, life quality scale data and wearable device time series data; the preoperative baseline data and the postoperative follow-up data are preprocessed; compared with an existing static prediction method, the application can track the whole postoperative rehabilitation process through time series position encoding and a time series Transformer architecture, outputs updated predictions at key time points such as preoperation, the first month, the third month, the sixth month and the twelfth month after operation, and realizes dynamic risk assessment.

Description

Technical Field

[0001] This invention relates to the field of rehabilitation prediction technology, specifically to a system for predicting postoperative rehabilitation outcomes of gastrointestinal tumors based on multimodal data. Background Technology

[0002] Gastrointestinal tumors are one of the leading causes of cancer-related deaths worldwide. The recurrence rate within 2 years after surgery can reach 30%-50%, while a considerable proportion of patients are able to achieve good recovery and quality of life.

[0003] Current postoperative rehabilitation assessment for gastrointestinal tumors primarily relies on physicians' clinical experience, using static clinical indicators such as TNM staging and pathological type for risk assessment. This approach has significant limitations: First, while the TNM staging system is widely used in clinical practice, the prognosis of patients within the same stage still varies greatly, making individualized and accurate prediction impossible. Second, traditional assessments, based on static information at the time of surgery, cannot capture the dynamic changes during postoperative rehabilitation, such as the recovery trend of nutritional indicators, the fluctuation pattern of inflammatory indicators, and the gradual changes in quality of life. Third, the existing assessment system lacks granular attention to the quality of rehabilitation, focusing more on the endpoint of "whether recurrence occurs," while paying insufficient attention to rehabilitation indicators that directly affect the patient's long-term prognosis, such as the speed of nutritional recovery, the recovery of gastrointestinal function, and changes in psychological state.

[0004] In recent years, artificial intelligence technology has begun to be applied to the field of prognostic prediction for gastrointestinal tumors. In radiomics, deep learning models based on CT images have been used to predict peritoneal metastasis and assess recurrence risk in gastric cancer, with some models achieving an AUC of 0.85-0.95 in multi-center validation. In pathomics, multi-instance learning frameworks based on whole-slice digital images (WSI) can extract nuclear morphology and tumor microenvironment features for lymph node metastasis prediction and survival analysis. In molecular detection, ctDNA liquid biopsy technology can detect molecular residual lesion signals 3-6 months before radiographic recurrence. However, the above studies mostly focus on the predictive ability of a single modality and lack effective integration of multimodal information.

[0005] In terms of multimodal fusion, existing technologies mainly have the following shortcomings: First, the fusion strategy is mainly static. Existing multimodal fusion methods mostly fuse multiple modal data at a single point in time, without considering the time dimension of data collection. Postoperative rehabilitation of gastrointestinal tumors is a dynamic process that lasts for several months to several years. There is a time-series dependency between preoperative baseline data and follow-up data at 1 month, 3 months, 6 months and 12 months after surgery. Static fusion cannot capture this time-series dynamic feature.

[0006] Second, there is insufficient capacity to handle modality loss. In clinical practice, not all patients have complete modality data. For example, some primary care hospitals cannot provide whole-section pathology images or ctDNA testing data, and some patients lack follow-up data at certain time points due to compliance issues. Existing fusion methods mostly assume complete modality data, and their performance degrades significantly when modality loss occurs.

[0007] Although ZL202610128822.0 discloses a system for predicting the risk of postoperative recurrence of rectal cancer based on multimodal time-series data, which integrates time-series clinical data, time-series MR images and pathological images, it focuses on the single prediction target of recurrence risk and does not involve fine-grained prediction and interpretable reasoning of rehabilitation effects.

[0008] Therefore, existing technologies still lack a systematic solution for predicting postoperative rehabilitation outcomes in gastrointestinal tumors that can integrate multimodal time-series data, achieve dynamic rehabilitation trajectory tracking, provide interpretable reasoning paths, and offer personalized intervention recommendations. This application proposes a multimodal data-based system for predicting postoperative rehabilitation outcomes in gastrointestinal tumors to overcome these deficiencies. Summary of the Invention

[0009] To address the aforementioned technical challenges, this paper presents a system for predicting postoperative rehabilitation outcomes in gastrointestinal tumors based on multimodal data. This solution resolves the issues of lacking integrated multimodal time-series data, failing to achieve dynamic rehabilitation trajectory tracking, providing interpretable reasoning paths, and offering personalized intervention recommendations.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A multimodal data-based system for predicting postoperative rehabilitation outcomes in gastrointestinal tumors includes: a multimodal data acquisition and preprocessing module for acquiring preoperative baseline data and follow-up data at multiple postoperative time points. The preoperative baseline data includes medical imaging data, pathological image data, and surgical record data. The postoperative follow-up data includes clinical laboratory time-series data, molecular detection data, quality of life scale data, and wearable device time-series data. The system also performs preprocessing on the preoperative baseline data and the postoperative follow-up data, including image grayscale normalization, pathological whole-slice image segmentation and block division, and time alignment and missing value imputation of the time-series data. The multimodal feature extraction module is used to extract modal features from the preprocessed modal data, including: extracting image features through a deep learning model, extracting pathological features through a pathological feature extraction network, extracting temporal features of clinical testing and molecular detection through a temporal coding network, and extracting activity pattern features of wearable devices through statistical analysis. The temporal Transformer fusion module is used to inject temporal position codes corresponding to time points into each modality feature, to perform interactive fusion of different modality features through a cross-modal attention mechanism, to build a rehabilitation trajectory pattern library based on contrastive learning, and to output rehabilitation effect prediction results at multiple time points after surgery. An interpretable reasoning module is used to construct a knowledge graph in the field of gastrointestinal tumor rehabilitation. It maps the feature attention weights output by the temporal Transformer fusion module to the entity nodes of the knowledge graph, generates a reasoning path from predicted features to clinical concepts, and quantifies the relative importance of each modality to the prediction result through feature contribution analysis. The clinical decision support module is used to generate personalized rehabilitation intervention suggestions through rule mapping based on the reasoning path of the interpretable reasoning module and the rehabilitation effect prediction results.

[0011] Preferably, in the multimodal feature extraction module: The deep learning model is a pre-trained three-dimensional convolutional neural network used to extract image depth features and to extract radiomics features through radiomics algorithms. The pathological feature extraction network adopts a multi-instance learning framework, using the overall benign and malignant labels of the pathological image as a weakly supervised signal for training. After aggregating the segmented blocks of the whole pathological slice image, it outputs cell nuclear morphology features and tumor microenvironment features. The temporal coding network is a long short-term memory network.

[0012] Preferably, the temporal Transformer fusion module includes a modal encoder group, a temporal position encoding layer, and a fusion output layer; The modal encoder group includes: an image encoder, which uses a three-dimensional convolutional neural network to perform dimensionality reduction and semantic extraction of image depth features and radiomics features; a pathology encoder, which uses a Transformer architecture to globally aggregate pathological features output by a multi-instance learning framework; a laboratory encoder, which uses a long short-term memory network to encode the temporal features of clinical laboratory tests and molecular tests; and a wearable encoder, which uses a fully connected layer to map activity pattern features. The temporal position encoding layer assigns learnable temporal embedding vectors for preoperative, postoperative month 1, month 3, month 6, and month 12, respectively. The temporal embedding vectors are initialized as random vectors and updated through training. The temporal embedding vectors are added to the modal features at the corresponding time points to obtain temporal-aware modal features. The fusion output layer inputs the temporal-aware modal features at each time point into a multi-layer Transformer encoder and outputs the postoperative rehabilitation effect prediction vector at each time point.

[0013] Preferably, the cross-modal attention mechanism adopts a multi-head cross-attention structure; each modality is used as the target modality, the temporal-aware modal features of that modality are used as the query vector, and the temporal-aware modal features of the other modalities are concatenated in the channel dimension as the key vector and value vector, and multi-head cross-attention is calculated to obtain the cross-modal enhancement features of that modality. The dimension of the cross-modal enhancement features is the same as the dimension of the temporal-aware modal features of that modality; after performing the above calculation on all modalities, the cross-modal enhancement features of each modality are concatenated in the channel dimension, and the fused features are output through a linear projection layer; residual connections and layer normalization are introduced in the cross-attention calculation.

[0014] Preferably, the construction of the rehabilitation trajectory pattern library based on contrastive learning includes: The fusion features of the same patient at adjacent time points are used as positive sample pairs, and the fusion features of different patients at the same time point are used as negative sample pairs. By training with the InfoNCE contrastive loss function, the distance between positive sample pairs in the embedding space is reduced, while the distance between negative sample pairs in the embedding space is increased. The trained fusion features are clustered into K rehabilitation trajectory patterns using the K-means clustering algorithm. The value of K is determined by the elbow rule. Each pattern corresponds to an embedding representation of a typical rehabilitation trajectory, thus forming the rehabilitation trajectory pattern library. For new patients, the cosine similarity between their fusion features and each pattern in the rehabilitation trajectory pattern library is calculated, and the pattern with the highest similarity is used as the reference for the patient's rehabilitation trajectory. The deviation of the rehabilitation trajectory is defined as 1 minus the cosine similarity between the current patient's fusion features and the most similar pattern in the rehabilitation trajectory pattern library.

[0015] Preferably, the knowledge graph for gastrointestinal tumor rehabilitation includes an entity layer and a relation layer; The entity layer includes the following entity types: disease characteristic entity, test indicator entity, imaging manifestation entity, pathological characteristic entity, rehabilitation measure entity, and prognostic result entity; The relation layer includes the following relation types: "Indicator" relation, which connects the test indicator entity and the disease feature entity; "Influence" relation, which connects the disease feature entity and the prognostic result entity; "Applicability" relation, which connects the rehabilitation measure entity and the disease feature entity. The mapping of feature attention weights to knowledge graph entity nodes includes: extracting attention weight matrices from the cross-modal attention layer of the temporal Transformer fusion module; averaging the attention weights across the spatial dimension if the modal features are in the form of feature maps, and directly taking the values ​​if the modal features are feature vectors, to obtain the attention weight vectors for each modal feature; pre-constructing a feature-entity mapping dictionary, which establishes a correspondence between radiomics features, pathological features, and test indicator features and entity names in the knowledge graph through annotations by medical experts; and associating features with weight values ​​greater than a preset threshold in the attention weight vectors with the corresponding entity nodes according to the mapping dictionary, forming feature-entity mapping pairs.

[0016] Preferably, the generation of the reasoning path from predicted features to clinical concepts includes: Starting with the feature-entity mapping pair corresponding to the high attention weight, a breadth-first search is performed in the knowledge graph, with a search depth of no more than 3 hops. During the search process, the expansion path is selected according to the preset priority of the relationship type. The preset priority is that "indication" relationship takes precedence over "influence" relationship over "applicability" relationship. The search terminates when the search path reaches the prognosis entity or rehabilitation measure entity; if multiple entities are reached simultaneously, the entity with the shortest path length is selected; if the path lengths are the same, the path with the highest relationship priority is selected. A reasoning path is formed from the characteristic entity to the prognostic outcome entity or the rehabilitation measure entity, and the entity sequence in the reasoning path is converted into a natural language description as an interpretable reasoning output.

[0017] Preferably, the feature contribution analysis uses the Shapley value method; For each mode, the overall contribution value of the mode is obtained by summing the Shapley values ​​of all features of the mode, wherein the Shapley values ​​are calculated using Monte Carlo sampling approximation. Sort the modalities by the absolute value of their overall contribution from largest to smallest to determine the relative importance of each modality to the prediction results; For a target mode, sort the features in the mode from largest to smallest absolute value of Shapley value, and select the top N features as the key contributing features of the mode.

[0018] Preferably, the clinical decision support module includes a nutritional support submodule, a follow-up scheduling submodule, and a lifestyle intervention submodule; The nutritional support submodule generates nutritional supplementation plan adjustment suggestions based on the time-series change trends of albumin, prealbumin, and hemoglobin, combined with the importance ranking of nutritional-related features in Shapley value analysis. The follow-up scheduling submodule dynamically adjusts the time interval for the next follow-up based on the probability of relapse and the deviation of the recovery trajectory. The lifestyle intervention submodule generates suggestions for adjusting exercise intensity and daily routines based on the activity patterns of wearable devices and the deviations from the rehabilitation trajectory reference patterns. The dynamic adjustment rules of the follow-up scheduling submodule include: When the risk of relapse is below the first threshold and the deviation of the recovery trajectory is below the second threshold, the follow-up interval is set to 6 months. When the risk of relapse is lower than the first threshold and the deviation of the recovery trajectory is not lower than the second threshold, the follow-up interval is set to 3 months. When the risk of relapse is not lower than the first threshold and the deviation of the recovery trajectory is lower than the second threshold, the follow-up interval is set to 3 months. When the probability of relapse is not lower than the first threshold and the deviation of the recovery trajectory is not lower than the second threshold, the follow-up interval is set to 1 month.

[0019] Preferably, the rehabilitation effect prediction results include the following prediction targets: Nutritional recovery levels are predicted based on the time-series changes in albumin, prealbumin, and body weight data, and are divided into three levels: complete recovery, partial recovery, and no recovery. The digestive function recovery score is a prediction based on the temporal changes in food intake, defecation frequency, and digestive symptoms, with a continuous value ranging from 0 to 100. The quality of life score is based on the time-series changes of each dimension of the EORTC QLQ-C30 scale, with a value range of 0 to 100. The probability of recurrence risk is predicted based on the fusion of imaging features, pathological features, and molecular detection data, and the value ranges from 0 to 1 as a continuous value.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: Compared to existing static prediction methods, this invention, through temporal location encoding and a temporal Transformer architecture, can track the entire postoperative rehabilitation process, outputting updated predictions at key time points such as preoperative and postoperative months 1, 3, 6, and 12, achieving dynamic risk assessment. By introducing a temporal Transformer architecture and cross-modal attention mechanism into gastrointestinal tumor rehabilitation prediction, it effectively integrates temporal information from multiple heterogeneous data sources such as imaging, pathology, clinical testing, molecular detection, and wearable devices at multiple time points, overcoming the deficiency of static fusion in capturing temporal dynamic features. Through comparative learning, a rehabilitation trajectory pattern library is constructed, which not only predicts risks but also learns typical rehabilitation trajectory patterns, identifies patients deviating from the normal trajectory, and provides a reference for personalized rehabilitation management.

[0021] By using knowledge graph reasoning and Shapley value analysis, the deep learning prediction results are transformed into clinically understandable reasoning paths, clarifying the contribution of each modality to the prediction, and enhancing clinicians' trust in the model's decision-making. In addition to outputting prediction results, this system also generates personalized suggestions such as nutritional support, follow-up scheduling, and lifestyle interventions through rule mapping, realizing a complete closed loop from "prediction" to "interpretation" to "suggestion". The system also focuses on rehabilitation details: in addition to predicting relapse risk, it predicts detailed rehabilitation indicators such as nutritional recovery level, digestive function recovery score, and quality of life score, which are closer to the actual needs of clinical rehabilitation management. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the system architecture of the present invention; Figure 2 This is a schematic diagram of the temporal Transformer fusion module structure of the present invention; Figure 3 This is a schematic diagram of the workflow of the interpretable reasoning module of the present invention; Figure 4 This is a schematic diagram illustrating the comparative learning of rehabilitation trajectories in this invention; Figure 5 This is a schematic diagram of the output interface of the clinical decision support module of the present invention. Detailed Implementation

[0023] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0024] Reference Figure 1 As shown, the gastrointestinal tumor postoperative rehabilitation effect prediction system based on multimodal data includes: a multimodal data acquisition and preprocessing module that collects preoperative baseline data and follow-up data at multiple postoperative time points. Preoperative baseline data includes medical imaging data, pathological image data, and surgical record data; postoperative follow-up data includes clinical laboratory time-series data, molecular diagnostic data, quality of life scale data, and wearable device time-series data. Preprocessing includes image grayscale normalization, pathological whole-slice image segmentation and block division, and time alignment and missing value imputation of time-series data.

[0025] Specifically: In the preoperative baseline data, medical imaging data are preoperative enhanced abdominal CT images (portal venous phase), with a slice thickness of 1-2 mm; pathological image data are postoperative H&E stained whole-section images (WSI), with a resolution of 0.25 μm / pixel; surgical record data include surgical method (laparoscopic / open surgery), operation time, intraoperative blood loss, number of lymph nodes dissected, and anastomosis method.

[0026] Postoperative follow-up data included clinical laboratory time-series data collected at 1, 3, 6, and 12 months postoperatively, including tumor markers (CEA, CA19-9, CA72-4), nutritional indicators (albumin, prealbumin, hemoglobin, body weight), and inflammatory markers (NLR, PLR, CRP); molecular testing data collected at 1, 3, and 6 months postoperatively, including ctDNA dynamic monitoring data (VAF of somatic mutation sites) and gene mutation status (HER2, MSI, KRAS, BRAF, etc.); quality of life scale data, namely the EORTC QLQ-C30 scale score; and wearable device time-series data, namely daily steps, heart rate, and sleep duration collected by smart bracelets, aggregated into weekly averages.

[0027] In preprocessing, image grayscale standardization cropped the grayscale values ​​of CT images to the soft tissue window range (-125 to 225 HU), linearly normalized them to the [0,1] interval, and resampling was performed to unify the voxel spacing. ; Pathological whole slide image segmentation and partitioning will be performed using WSI. At magnification, the image was segmented into 256×256 pixel non-overlapping blocks, and background blocks with a tissue content of less than 50% were removed. The time alignment of the time series data was based on the surgery date as the zero point, and the follow-up data were aligned to the 1st month (30±7 days), 3rd month (90±14 days), 6th month (180±14 days), and 12th month (365±30 days) after surgery. The missing data test index was linearly interpolated, and the missing scale data were marked as missing and not imputed.

[0028] The multimodal feature extraction module extracts modal features from each of the preprocessed modal data, including imaging features, pathological features, temporal features of clinical testing and molecular detection, and activity pattern features of wearable devices (the specific extraction method is described in the implementation of claim 2 below).

[0029] The temporal Transformer fusion module injects temporal position codes corresponding to each time point into each modal feature, performs interactive fusion of different modal features through a cross-modal attention mechanism, constructs a rehabilitation trajectory pattern library based on contrastive learning, and outputs rehabilitation effect prediction results at multiple time points after surgery.

[0030] The interpretable reasoning module constructs a knowledge graph for the field of gastrointestinal tumor rehabilitation, maps the feature attention weights output by the temporal Transformer fusion module to entity nodes in the knowledge graph, generates a reasoning path from predicted features to clinical concepts, and quantifies the relative importance of each modality to the prediction results through feature contribution analysis.

[0031] The clinical decision support module generates personalized rehabilitation intervention recommendations through rule mapping, based on the reasoning path of the interpretable reasoning module and the rehabilitation effect prediction results.

[0032] In this embodiment, the deep learning model is a pre-trained 3D convolutional neural network used to extract image depth features: a ResNet50 pre-trained on ImageNet is used as the backbone network, the last fully connected layer is removed, and the network is input at the maximum transverse section of the tumor in the CT image and two layers above and below it (a total of 5 layers). The feature maps of the last convolutional layer of each layer are extracted, and global average pooling is performed to obtain the desired feature map. 3D image depth feature vector .

[0033] Radiomics features were extracted using radiomics algorithms: 258 radiomics features (18 first-order statistical features, 24 shape features, 72 GLCM features, 48 ​​GLRLM features, 48 ​​GLSZM features, and 48 NGTDM features) were extracted from the 3D ROI of the tumor using the Pyradiomics library. After ICC screening (>0.75) and LASSO regression selection, 15-25 features were retained to form the radiomics feature vector. .

[0034] The pathological feature extraction network employs a multi-instance learning framework: valid image patches after WSI segmentation are used as instances, and the overall benign / malignant label of the pathological image is used as a weakly supervised signal to train the MIL model. An attention-based aggregation-based MIL method (AB-MIL) is used, where a 512-dimensional feature vector is extracted from each image patch using a pre-trained ResNet18, and the attention weights for each image patch are calculated through an attention network. in, Here is the weight matrix of the hidden layer of the attention network. =256 is the hidden layer dimension. =512 represents the feature dimension of the image patch; This is the attention score vector, used to map the hidden layer output to a scalar attention score; The bias vector of the hidden layer; For the first Feature vectors of image patches This represents the total number of image patches. Weighted aggregation yields the WSI-level feature vector: Simultaneously, a ResNet18 classifier was trained at the image patch level to identify eight TME components (tumor cells, lymphocytes, plasma cells, eosinophils, stroma, necrosis, mucus, and normal glands), and the proportion and spatial distribution characteristics of each component were statistically analyzed as a supplement to the characteristics of the tumor microenvironment.

[0035] Temporal coding networks are Long Short-Term Memory (LSTM) networks: They process data at each time point... The test index values ​​are concatenated into an input vector. Input the data into a two-layer LSTM (64-dimensional hidden layers) and extract the hidden state at the last time step. As a time series feature vector It has 128 dimensions.

[0036] Activity pattern characteristics of wearable devices are extracted through statistical analysis: weekly average steps, weekly average heart rate, weekly average sleep duration, step count coefficient of variation, heart rate coefficient of variation, week-on-week change rate of steps, and week-on-week change rate of heart rate are calculated on a weekly basis to form an activity pattern feature vector. .

[0037] like Figure 2 As shown, the temporal Transformer fusion module includes a modal encoder group, a temporal position encoding layer, and a fusion output layer.

[0038] The modal encoder group consists of four independent encoders: Image encoder: Employs a 3-layer 3D convolutional neural network to analyze image depth features. and radiomics features Dimensionality reduction and semantic extraction are performed to output 256-dimensional image coding features. ; Pathological encoder: A 4-layer Transformer encoder (8 attention heads, 1024 feedforward network dimensions) is used to process the pathological features output by MIL. Perform global aggregation to output 256-dimensional pathological coding features. ; Encoder verification: Timing characteristics of LSTM output The features are mapped to 256-dimensional test codes through a fully connected layer. ; Wearable encoder: Encodes activity patterns through fully connected layers Mapped to 256-dimensional wearable encoded features .

[0039] The temporal position coding layer defines 5 time points: (Preoperative) (1 month post-surgery) (3 months post-surgery) (6 months post-surgery) (12 months post-surgery). Learnable temporal embedding vectors were assigned to each time point. ( =0,1,2,3,4), initialized as a standard normally distributed random vector. During training, updates are performed via backpropagation. The temporal embedding vector is added to the modality-encoded features at the corresponding time points: in, This yields time-aware modal features. It should be noted that preoperative data only exists in... Time points, postoperative follow-up data exist to Time point; for a mode that is missing at a certain time point, use the all-zero vector of that mode to replace it.

[0040] The fusion output layer inputs the temporal-aware modal features of each time point into a 4-layer Transformer encoder (8 attention heads, 1024 feedforward network dimensions) to model the dependencies between time points. Finally, prediction heads are connected to each time point to output the postoperative rehabilitation effect prediction vector at each time point.

[0041] In this embodiment, cross-modal attention employs a multi-head cross-attention structure, sequentially using each modality as the target modality for enhancement. Based on its time-sensing modal features as query vector The temporal-aware modal features of the remaining modalities are concatenated along the channel dimension to form the key vector. Sum value vector : in This indicates that the target mode will be excluded. The temporal-aware modal features of the other three modalities are concatenated along the channel dimension, resulting in a 768-dimensional concatenation. , , Modal The query projection matrix, key projection matrix, and value projection matrix. For the projection dimension, the multi-head cross-attention is calculated as follows: in, For the number of attention heads, The output projection matrix for multi-head attention; , and These are query, key, and value submatrices, respectively. For the query representation of the target modality in the h-th subspace, The bond representation of the non-target mode in the h-th subspace The content information to be transmitted in the h-th subspace of the non-target mode is introduced by residual connections and layer normalization: That is, mode Cross-modal enhancement features, whose dimensions are similar to those of temporally aware modal features. Same (256 dimensions).

[0042] After performing the above calculations on all modalities, the cross-modal enhancement features of each modality are concatenated along the channel dimension: ,in This indicates splicing along the channel dimension; via a linear projection layer. Output fusion features .

[0043] like Figure 4 As shown, the construction of the rehabilitation trajectory pattern library is based on comparative learning.

[0044] Training phase: The fusion features of the same patient at adjacent time points are used as positive sample pairs. Using the fusion characteristics of different patients at the same time point as negative samples The InfoNCE contrast loss function is used: in For cosine similarity, For temperature parameters, Batch size; Clustering Phase: After training convergence, the fusion features of all trained patients are extracted, and the K-means clustering algorithm is used to cluster them into K rehabilitation trajectory patterns. The value of K is determined by the elbow rule: the sum of squared errors (SSE) within the cluster is calculated when K ranges from 2 to 10, and the inflection point where the rate of decrease in SSE significantly slows down is selected as the value of K. In this embodiment, K=4, corresponding to four typical rehabilitation trajectories: "rapid recovery," "steady recovery," "slow recovery," and "recovery stagnation." The cluster center of each pattern... As an embedded representation of this pattern, it constitutes the rehabilitation trajectory pattern library.

[0045] Inference phase: For new patients, calculate their fusion features. Cosine similarity with the cluster centers of each pattern in the rehabilitation trajectory pattern library: The most similar patterns This serves as a reference for the patient's recovery trajectory.

[0046] The deviation from the rehabilitation trajectory is defined as: Deviation The value ranges from [0,1], and the larger the value, the further the patient deviates from the most similar typical trajectory.

[0047] like Figure 3 As shown, the knowledge graph in the field of gastrointestinal tumor rehabilitation includes an entity layer and a relation layer; The entity layer includes six types of entities: disease characteristic entities (such as "poorly differentiated adenocarcinoma", "serosa invasion", "lymph node metastasis", "vascular invasion"); laboratory indicator entities (such as "low albumin", "elevated CEA", "elevated NLR", "ctDNA positive"); imaging manifestation entities (such as "increased tumor wall thickness", "extraserosa invasion", "lymph node enlargement"); pathological characteristic entities (such as "dense tumor cells", "insufficient lymphocyte infiltration", "abundant stroma"); rehabilitation measures entities (such as "enteral nutrition support", "immunotherapy", "psychological intervention", "exercise rehabilitation"); and prognostic outcome entities (such as "good nutritional recovery", "high risk of recurrence", "delayed recovery of gastrointestinal function").

[0048] The relationship layer contains three types of relationships: "indicator" relationship, which connects the entity of the test indicator with the entity of the disease characteristic, such as "low albumin" → indicator → "poor nutritional status"; "influence" relationship, which connects the entity of the disease characteristic with the entity of the prognostic result, such as "poorly differentiated adenocarcinoma" → influence → "high risk of recurrence"; and "applicability" relationship, which connects the entity of the rehabilitation measure with the entity of the disease characteristic, such as "enteral nutrition support" → applicable → "poor nutritional status".

[0049] The process of mapping feature attention weights to knowledge graph entity nodes is as follows: First, the attention weight matrix is ​​extracted from the cross-modal attention layer of the temporal Transformer fusion module. For each attention head in the multi-head cross-attention, the attention score matrix is ​​extracted. If the modal features are in the form of feature maps (such as image depth features), the attention weights are averaged over the spatial dimensions (height and width); if the modal features are feature vectors (such as temporal features or activity pattern features), the attention weight values ​​are directly taken. The attention weight vector for each modality feature is obtained by averaging the weights of each attention head. .

[0050] A feature-entity mapping dictionary is pre-constructed. The first category is interpretable features, including radiomics features (such as tumor volume and grayscale heterogeneity), pathological statistical features (such as tumor cell percentage and lymphocyte infiltration density), and test indicator features (such as albumin level and CEA level). These features have clear clinical semantics and can be directly associated with knowledge graph entities through the mapping dictionary.

[0051] The second category is abstract deep features, including high-dimensional embedding vectors output by the image deep feature extraction submodule (such as 512-dimensional feature maps extracted based on ResNet50) and WSI embedding vectors output by the pathology multi-instance learning submodule. These features are automatically learned by neural networks, and a single dimension does not have a clinically interpretable meaning and cannot be directly associated with knowledge graph entities through a mapping dictionary.

[0052] For abstract deep features, this system adopts an indirect mapping method based on attention weights. Interpretable features are used as an intermediary to establish associations between abstract deep features and knowledge graph entities. The specific steps are as follows: Step 1: Cross-modal attention weight extraction In the cross-modal attention layer of the temporal Transformer fusion module, interpretable feature vectors of the same modality are simultaneously input. and abstract deep feature vectors Taking image modalities as an example, features can be explained. To represent the number of radiomics features, abstract deep features The number of channels in the ResNet50 output feature map. During the computation of multi-head cross attention, an attentional interaction occurs between the abstract deep feature dimension and the interpretable features.

[0053] Extracting the attention weight matrix of abstract deep features to interpretable features in the cross-modal attention layer , among which, element Indicates the first The abstract deep feature dimension for the first The degree of attention given to interpretable features.

[0054] Step 2: Filtering Interpretable Features Attention weight matrix For each row (i.e., each abstract deep feature dimension), the top K interpretable features with the largest attention weights are selected as its associated interpretable features. The value of K ranges from 1 to K and from 3; in this embodiment, K = 2. That is, for each abstract deep feature dimension... The set of features that can be interpreted by association is as follows: in, This indicates selecting the value with the largest weight. The interpretable feature index corresponding to each element.

[0055] Step 3: Indirect Entity Mapping Based on the set of related interpretable features obtained in step 2 Query the feature-entity mapping dictionary to obtain the knowledge graph entity corresponding to each associated interpretable feature, and map the abstract deep feature dimension to the set of entities corresponding to all its associated interpretable features to form indirect mapping pairs: in, For the first An abstract deep feature, To look up via a mapping dictionary The set of knowledge graph entities corresponding to the interpretable features in the text. For the first in the knowledge graph Entity node identifier.

[0056] Step 4: Merge mapping results Direct mapping to the set (Direct mappings from interpretable features) and indirect mapping pairs Merging them yields a complete set of feature-entity mapping pairs: Subsequent inference path generation and Shapley value analysis are both based on the merged complete set of mapping pairs. conduct.

[0057] For example, taking image modalities as an example, assuming ResNet50 outputs 512-dimensional abstract deep features, Pyradiomics extracts 20 interpretable radiomics features. In the cross-modal attention layer, the attention weight matrix is ​​extracted. .

[0058] For the 128th dimension abstract depth feature The top two interpretable features with the highest attention weight are "tumor volume". And "grayscale non-uniformity" Then the associated interpretable feature set = {Tumor volume, grayscale heterogeneity}.

[0059] The query mapping dictionary shows that "tumor volume" corresponds to the knowledge graph entity "increased tumor volume" (entity identifier E03), and "grayscale non-uniformity" corresponds to the knowledge graph entity "high tumor heterogeneity" (entity identifier E07). Therefore, an indirect mapping pair is formed. That is, the 128th dimension of abstract deep features is indirectly mapped to two knowledge graph entities: "tumor volume increase" and "tumor heterogeneity".

[0060] Similarly, for the abstract embedding vector output by the MIL framework in the pathological modality, through the attention weight between it and pathological statistical features (such as the proportion of tumor cells and the density of lymphocyte infiltration), it is indirectly mapped to pathological feature entities such as "dense tumor cells" and "insufficient lymphocyte infiltration" in the knowledge graph.

[0061] Based on the mapping dictionary, the attention weight vector The weight value is greater than the preset threshold. (This embodiment is set as follows) Features with a value of 0.05 are associated with corresponding entity nodes, forming a set of feature-entity mapping pairs. .

[0062] With feature-entity mapping pairs of sets The entity node in the graph is the starting node, and a breadth-first search (BFS) is performed in the knowledge graph with a search depth of no more than 3 hops.

[0063] During the search process, the expansion path is selected according to the preset priority of the relationship type, with priority from high to low as follows: "Indicator" relationship > "Influence" relationship > "Applicable" relationship. When there are outgoing edges of multiple relationship types for the same node, the relationship type with the higher priority is expanded first.

[0064] The search path terminates when it reaches the prognosis entity or rehabilitation measure entity. If multiple entities are reached simultaneously, the entity with the shortest path length is selected as the endpoint. If the path lengths are the same, the entity reached by the path with the highest relation priority is selected as the endpoint.

[0065] The entity and relation sequences in the reasoning path are converted into natural language descriptions. For example, the path "low albumin" → indication → "poor nutritional status" → impact → "delayed recovery" → applicable → "enteral nutrition support" is converted into: "The patient's low albumin indicates poor nutritional status, which may affect the recovery process. It is recommended to strengthen enteral nutrition support."

[0066] In this embodiment, the Shapley value method is used to quantify the relative importance of each mode to the prediction result.

[0067] For each mode All features within this mode are treated as a whole and included in the Shapley value calculation. Let there be a total of... = 4 modes, modes The Shapley value is: in, To not include modes modal subset, For using subsets The model's predicted values ​​in the intermediate mode. Because... =4, requires traversal =16 subsets, directly and precisely calculated.

[0068] Based on the overall contribution value of each modality Sort the modalities from largest to smallest to determine their relative importance to the prediction results.

[0069] For the target mode The Shapley values ​​of each feature within this modality were further calculated. Due to the high dimensionality of features within a single modality (e.g., 15-25 dimensions for radiomics features), Monte Carlo sampling was used for approximate calculation, with 1000 sampling iterations. The features were then sorted by absolute value of the Shapley values ​​from largest to smallest, and the top [values] were selected. One feature (in this embodiment) =5) as a key contributing feature of this mode.

[0070] like Figure 5 As shown, the clinical decision support module includes a nutritional support submodule, a follow-up scheduling submodule, and a lifestyle intervention submodule.

[0071] The nutritional support submodule generates recommendations for adjusting nutritional supplementation plans based on the temporal trends of albumin, prealbumin, and hemoglobin, combined with the importance ranking of nutritional-related characteristics in Shapley value analysis. Specific rules: If albumin is below 35 g / L for two consecutive time points, enteral nutrition supplementation is recommended; if prealbumin is below 180 mg / L, protein intake is recommended; if hemoglobin is below 110 g / L (women) or 120 g / L (men), iron and vitamin B12 supplementation is recommended.

[0072] The follow-up scheduling submodule is based on the probability of recurrence. Deviation from recovery trajectory The time interval for the next follow-up visit is dynamically adjusted. A first threshold is set. =0.3, second threshold =0.4, the adjustment rule is: when and The follow-up interval was set at 6 months; when and The follow-up interval was set at 3 months; when and The follow-up interval was set at 3 months; when and The follow-up interval was set at 1 month.

[0073] The lifestyle intervention submodule generates suggestions for adjusting exercise intensity and daily routines based on the activity patterns of wearable devices and the deviations from the rehabilitation trajectory reference model. Specific rules: If the average weekly steps are less than 60% of the average for the corresponding stage of the rehabilitation trajectory reference model, it is recommended to gradually increase walking volume under the guidance of a doctor, with a weekly increase not exceeding 10%; if sleep duration is less than 6 hours or the coefficient of variation is greater than 0.3, it is recommended to adjust the daily routine and consider sleep intervention.

[0074] This system outputs predictions of rehabilitation outcomes in the following four dimensions at various postoperative time points: Nutritional recovery level: Based on the time-series change trend prediction of albumin, prealbumin and weight data, it is divided into three levels: complete recovery (albumin ≥40g / L, prealbumin ≥250mg / L, weight recovered to more than 90% of the preoperative weight), partial recovery (the above indicators improved but did not reach the standard of complete recovery), and no recovery (the above indicators did not improve or worsened). The Softmax classifier is used to output the three-class probability.

[0075] Digestive function recovery score: Based on the temporal changes in food intake, defecation frequency and digestive symptoms, the score is a continuous value in the range of [0,100]. 0 points indicates complete loss of function and 100 points indicates complete recovery of function. The score is output using a regression prediction head.

[0076] Quality of life score: Based on the time-series changes of each dimension of the EORTC QLQ-C30 scale, the scores are predicted as continuous values ​​in the range of [0,100]. The regression prediction head is used to predict the scores of each functional dimension.

[0077] Recurrence risk probability: Based on the fusion feature prediction of imaging features, pathological features and molecular detection data, the value is a continuous value in the range of [0,1], and the probability value is output by the Sigmoid output layer.

[0078] The training of this system employs a multi-task learning strategy, and the total loss function is: in, The cross-entropy loss is the result of the nutrient recovery level. The mean squared error loss is calculated for both the digestive function score and the quality of life score. Cox partial likelihood loss for predicting recurrence risk The learning loss is compared to the rehabilitation trajectory. The weighting coefficients are set to... =1.0 =0.5、 =1.0、 =0.1.

[0079] The training data came from a multicenter follow-up cohort of gastrointestinal tumor patients. The inclusion criteria were: (1) pathologically confirmed gastric or colorectal cancer; (2) radical surgery; (3) follow-up time ≥ 12 months; and (4) data from at least one preoperative imaging and one postoperative time point. The training set, validation set, and test set were divided in a 7:1:2 ratio to ensure that the data from different centers were distributed consistently across the sets.

[0080] Training uses the AdamW optimizer with an initial learning rate of Weight decay The batch size is 32, the number of training rounds is 100, and cosine annealing learning rate scheduling is used.

[0081] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A system for predicting postoperative rehabilitation outcomes of gastrointestinal tumors based on multimodal data, characterized in that, include: A multimodal data acquisition and preprocessing module is used to acquire preoperative baseline data and follow-up data at multiple postoperative time points. The preoperative baseline data includes medical imaging data, pathological image data, and surgical record data. The postoperative follow-up data includes clinical laboratory time-series data, molecular detection data, quality of life scale data, and wearable device time-series data. The module performs preprocessing on the preoperative baseline data and the postoperative follow-up data. The preprocessing includes image grayscale standardization, pathological whole-slice image segmentation and block division, time alignment of time-series data, and imputation of missing values. The multimodal feature extraction module is used to extract modal features from the preprocessed modal data, including: extracting image features through a deep learning model, extracting pathological features through a pathological feature extraction network, extracting temporal features of clinical testing and molecular detection through a temporal coding network, and extracting activity pattern features of wearable devices through statistical analysis. The temporal Transformer fusion module is used to inject temporal position codes corresponding to time points into each modal feature, to perform interactive fusion of different modal features through a cross-modal attention mechanism, to build a rehabilitation trajectory pattern library based on contrastive learning, and to output rehabilitation effect prediction results at multiple time points after surgery. An interpretable reasoning module is used to construct a knowledge graph in the field of gastrointestinal tumor rehabilitation. It maps the feature attention weights output by the temporal Transformer fusion module to the entity nodes of the knowledge graph, generates a reasoning path from predicted features to clinical concepts, and quantifies the relative importance of each modality to the prediction result through feature contribution analysis. The clinical decision support module is used to generate personalized rehabilitation intervention suggestions through rule mapping based on the reasoning path of the interpretable reasoning module and the rehabilitation effect prediction results.

2. The gastrointestinal tumor postoperative rehabilitation effect prediction system based on multimodal data according to claim 1, characterized in that: In the multimodal feature extraction module: The deep learning model is a pre-trained three-dimensional convolutional neural network used to extract image depth features and to extract radiomics features through radiomics algorithms. The pathological feature extraction network adopts a multi-instance learning framework, using the overall benign and malignant labels of the pathological image as a weak supervision signal for training. After aggregating the segmented blocks of the whole pathological slice image, it outputs cell nuclear morphology features and tumor microenvironment features. The temporal coding network is a long short-term memory network.

3. The gastrointestinal tumor postoperative rehabilitation effect prediction system based on multimodal data according to claim 2, characterized in that: The temporal Transformer fusion module includes a modal encoder group, a temporal position encoding layer, and a fusion output layer; The modal encoder group includes: an image encoder, which uses a three-dimensional convolutional neural network to perform dimensionality reduction and semantic extraction of image depth features and radiomics features; a pathology encoder, which uses a Transformer architecture to globally aggregate pathological features output by a multi-instance learning framework; a laboratory encoder, which uses a long short-term memory network to encode the temporal features of clinical laboratory tests and molecular tests; and a wearable encoder, which uses a fully connected layer to map activity pattern features. The temporal position encoding layer assigns learnable temporal embedding vectors for preoperative, postoperative month 1, month 3, month 6, and month 12, respectively. The temporal embedding vectors are initialized as random vectors and updated through training. The temporal embedding vectors are added to the modal features at the corresponding time points to obtain temporal-aware modal features. The fusion output layer inputs the temporal-aware modal features at each time point into a multi-layer Transformer encoder and outputs the postoperative rehabilitation effect prediction vector at each time point.

4. The gastrointestinal tumor postoperative rehabilitation effect prediction system based on multimodal data according to claim 3, characterized in that: The cross-modal attention mechanism employs a multi-head cross-attention structure. Each modality is used as the target modality, its temporal-aware modal features are used as the query vector, and the temporal-aware modal features of the remaining modalities are concatenated along the channel dimension to serve as the key and value vectors. Multi-head cross-attention is then calculated to obtain the cross-modal enhancement features for that modality. The dimension of these cross-modal enhancement features is the same as the dimension of the temporal-aware modal features of that modality. After performing the above calculation on all modalities, the cross-modal enhancement features of each modality are concatenated along the channel dimension, and a fused feature is output through a linear projection layer. Residual connections and layer normalization are introduced in the cross-attention calculation.

5. The gastrointestinal tumor postoperative rehabilitation effect prediction system based on multimodal data according to claim 4, characterized in that: The rehabilitation trajectory pattern library constructed based on contrastive learning includes: The fusion features of the same patient at adjacent time points are used as positive sample pairs, and the fusion features of different patients at the same time point are used as negative sample pairs. By training with the InfoNCE contrastive loss function, the distance between positive sample pairs in the embedding space is reduced, while the distance between negative sample pairs in the embedding space is increased. The trained fusion features are clustered into K rehabilitation trajectory patterns using the K-means clustering algorithm. The value of K is determined by the elbow rule. Each pattern corresponds to an embedding representation of a typical rehabilitation trajectory, thus forming the rehabilitation trajectory pattern library. For new patients, the cosine similarity between their fusion features and each pattern in the rehabilitation trajectory pattern library is calculated, and the pattern with the highest similarity is used as the reference for the patient's rehabilitation trajectory. The deviation of the rehabilitation trajectory is defined as 1 minus the cosine similarity between the current patient's fusion features and the most similar pattern in the rehabilitation trajectory pattern library.

6. The gastrointestinal tumor postoperative rehabilitation effect prediction system based on multimodal data according to claim 1, characterized in that: The knowledge graph for the field of gastrointestinal tumor rehabilitation includes an entity layer and a relation layer; The entity layer includes the following entity types: disease characteristic entity, test indicator entity, imaging manifestation entity, pathological characteristic entity, rehabilitation measure entity, and prognostic result entity; The relation layer includes the following relation types: "Indicator" relation, which connects the test indicator entity and the disease feature entity; "Influence" relation, which connects the disease feature entity and the prognostic result entity; "Applicability" relation, which connects the rehabilitation measure entity and the disease feature entity. The mapping of feature attention weights to knowledge graph entity nodes includes: extracting attention weight matrices from the cross-modal attention layer of the temporal Transformer fusion module; averaging the attention weights across the spatial dimension if the modal features are in the form of feature maps, and directly taking the values ​​if the modal features are feature vectors, to obtain the attention weight vectors for each modal feature; pre-constructing a feature-entity mapping dictionary, which establishes a correspondence between radiomics features, pathological features, and test indicator features and entity names in the knowledge graph through annotations by medical experts; and associating features with weight values ​​greater than a preset threshold in the attention weight vectors with the corresponding entity nodes according to the mapping dictionary, forming feature-entity mapping pairs.

7. The gastrointestinal tumor postoperative rehabilitation effect prediction system based on multimodal data according to claim 6, characterized in that: The reasoning path from predicted features to clinical concepts includes: Starting with the feature-entity mapping pair corresponding to the high attention weight, a breadth-first search is performed in the knowledge graph, with a search depth of no more than 3 hops. During the search process, the expansion path is selected according to the preset priority of the relationship type. The preset priority is that "indication" relationship takes precedence over "influence" relationship over "applicability" relationship. The search terminates when the search path reaches the prognosis entity or rehabilitation measure entity; if multiple entities are reached simultaneously, the entity with the shortest path length is selected; if the path lengths are the same, the path with the highest relationship priority is selected. A reasoning path is formed from the characteristic entity to the prognostic outcome entity or the rehabilitation measure entity, and the entity sequence in the reasoning path is converted into a natural language description as an interpretable reasoning output.

8. The gastrointestinal tumor postoperative rehabilitation effect prediction system based on multimodal data according to claim 1, characterized in that: The feature contribution analysis uses the Shapley value method; For each mode, the overall contribution value of the mode is obtained by summing the Shapley values ​​of all features of the mode, wherein the Shapley values ​​are calculated using Monte Carlo sampling approximation. Sort the modalities by the absolute value of their overall contribution from largest to smallest to determine the relative importance of each modality to the prediction results; For a target mode, sort the features in the mode from largest to smallest absolute value of Shapley value, and select the top N features as the key contributing features of the mode.

9. The gastrointestinal tumor postoperative rehabilitation effect prediction system based on multimodal data according to claim 5, characterized in that: The clinical decision support module includes a nutritional support submodule, a follow-up scheduling submodule, and a lifestyle intervention submodule. The nutritional support submodule generates recommendations for adjusting nutritional supplementation plans based on the temporal variation trends of albumin, prealbumin, and hemoglobin, combined with the importance ranking of nutritional-related features in Shapley value analysis. The follow-up scheduling submodule dynamically adjusts the time interval for the next follow-up based on the probability of relapse and the deviation of the recovery trajectory. The lifestyle intervention submodule generates suggestions for adjusting exercise intensity and daily routines based on the activity patterns of wearable devices and the deviations from the rehabilitation trajectory reference patterns. The dynamic adjustment rules of the follow-up scheduling submodule include: When the risk of relapse is below the first threshold and the deviation of the recovery trajectory is below the second threshold, the follow-up interval is set to 6 months. When the risk of relapse is lower than the first threshold and the deviation of the recovery trajectory is not lower than the second threshold, the follow-up interval is set to 3 months. When the risk of relapse is not lower than the first threshold and the deviation of the recovery trajectory is lower than the second threshold, the follow-up interval is set to 3 months. When the probability of relapse is not lower than the first threshold and the deviation of the recovery trajectory is not lower than the second threshold, the follow-up interval is set to 1 month.

10. The gastrointestinal tumor postoperative rehabilitation effect prediction system based on multimodal data according to claim 9, characterized in that: The predicted rehabilitation outcomes include the following prediction objectives: Nutritional recovery levels are predicted based on the time-series changes in albumin, prealbumin, and body weight data, and are divided into three levels: complete recovery, partial recovery, and no recovery. The digestive function recovery score is a prediction based on the temporal changes in food intake, defecation frequency, and digestive symptoms, with a continuous value ranging from 0 to 100. The quality of life score is based on the time-series changes of each dimension of the EORTC QLQ-C30 scale, with a value range of 0 to 100. The probability of recurrence risk is predicted based on the fusion of imaging features, pathological features, and molecular detection data, and the value ranges from 0 to 1 as a continuous value.

Images