Disease progression prediction method and system based on time-space multi-modal graph convolutional network

Abstract

The application discloses a disease progression prediction method and system based on a time-space multi-modal graph convolution network. The method comprises the following steps: acquiring multi-modal medical follow-up data of a target patient at multiple time points; constructing a time multi-modal medical knowledge graph based on the multi-modal medical follow-up data; performing spatial graph convolution on a multi-modal medical knowledge subgraph at each time point to aggregate spatial medical features of medical entity nodes along medical relationship edges; performing time graph convolution on time edges between different time points to learn time evolution features of the medical entity nodes between the different time points; fusing the spatial medical features and the time evolution features to obtain a patient dynamic representation; and predicting a future disease progression of the target patient based on the patient dynamic representation. Thus, the structured relationship information of the medical knowledge graph and the time evolution information of the patient follow-up sequence can be simultaneously utilized to realize accurate prediction of a disease progression risk.

Description

Technical Field

[0001] This application relates to the field of medical artificial intelligence technology, and more specifically, to a method and system for predicting disease progression based on temporal-spatial multimodal graph convolutional networks. Background Technology

[0002] Disease progression prediction is an important direction for precision medicine and clinical decision support. Cancer, chronic diseases, and neurodegenerative diseases often exhibit distinct temporal evolution characteristics, and static data from a single visit are usually insufficient to accurately assess future progression risk. With the continuous accumulation of electronic medical records, medical imaging, and laboratory indicators, how to comprehensively utilize multimodal follow-up data from multiple time points to accurately predict disease progression risk has become a research hotspot in the field of medical artificial intelligence.

[0003] Existing methods for predicting disease progression mainly fall into two categories. One category comprises time-series models (such as RNNs, LSTMs, and Transformers), which can utilize longitudinal patient follow-up data but struggle to explicitly leverage the medical relationship structures between diseases, symptoms, laboratory indicators, and imaging findings. The other category includes graph neural networks and knowledge graph methods, which can utilize medical relationship structures, but typically treat the graph as static, making it difficult to model state transitions and disease evolution processes between different follow-up time points. Some studies have attempted to combine the two, but these often involve simple fusion at the feature or decision levels, failing to synchronously propagate spatial medical relationships and temporal evolutionary relationships within a unified graph structure. Furthermore, existing methods lack effective mechanisms to handle real-world clinical situations such as irregular follow-up intervals, missing modal data, and the need for continuous integration of new follow-up data. Summary of the Invention

[0004] The main objective of this application is to provide a disease progression prediction method based on temporal-spatial multimodal graph convolutional networks, in order to solve the technical problem that existing technologies cannot simultaneously take into account the relationship between medical outcomes and temporal evolution patterns. This method can simultaneously utilize the structured relationships of medical knowledge graphs and the temporal evolution information of patient follow-up sequences to achieve accurate prediction of disease progression risk.

[0005] To achieve the above objectives, the first aspect of this application proposes a disease progression prediction method based on a temporal-spatial multimodal graph convolutional network, comprising: acquiring multimodal medical follow-up data of a target patient at multiple time points; constructing a temporal multimodal medical knowledge graph corresponding to the target patient based on the multimodal medical follow-up data, wherein each time point corresponds to a multimodal medical knowledge subgraph, each multimodal medical knowledge subgraph contains multiple medical entity nodes and medical relation edges connecting the medical entity nodes, and different time points are connected by temporal edges connecting the medical entity nodes; and performing spatial graph convolution on the multimodal medical knowledge subgraph at each time point. Spatial medical features of medical entity nodes are aggregated along medical relationship edges; temporal graph convolution is performed on temporal edges between different time points to learn the temporal evolution features of medical entity nodes between different time points; wherein, copies of the same medical entity node at different time points are connected through temporal edges, the temporal edges including at least one of the following: edge connecting the same lesion, edge connecting the same detection index, edge connecting disease state transition, and edge connecting the order of visits; spatial medical features and temporal evolution features are fused to obtain a dynamic representation of the patient to characterize the current and historical comprehensive state of the target patient; and the future disease progression of the target patient is predicted based on the dynamic representation of the patient.

[0006] In the above scheme, spatial graph convolution is performed on the multimodal medical knowledge subgraph at each time point to aggregate the spatial medical features of medical entity nodes along medical relation edges. This includes: obtaining the type identifier of each medical relation edge within the current time point; assigning different relation transformation matrices to different medical relation types according to the type identifier; and for each medical entity node within the current time point, aggregating the features of neighboring nodes connected to the medical entity node through various types of medical relation edges. The aggregation results of different relation types are added after being mapped by the corresponding relation transformation matrices to obtain the spatial medical features of the medical entity node.

[0007] In the above scheme, temporal graph convolution is performed on the time edges between different time points to learn the temporal evolution characteristics of medical entity nodes between different time points. This includes: obtaining the actual calendar time interval between two adjacent time points; generating a time interval code based on the actual calendar time interval, which is used to represent the duration information between adjacent follow-ups; inputting the time interval code and the type identifier of the current time edge into a gating function to calculate the propagation gating value on the time edge; and multiplying the evolution characteristics of the medical entity node at the previous time point by the propagation gating value along the time edge and then passing it to the corresponding medical entity node at the time point to obtain the temporal evolution characteristics of the medical entity node between different time points.

[0008] In the above scheme, spatial medical features and temporal evolution features are integrated to obtain a dynamic representation of the target patient in the current and historical comprehensive state. This includes: concatenating or weighting the spatial medical features of the target time point with the temporal evolution features corresponding to the time point to obtain a fused representation of the time point; and mapping the fused representation to a fixed-dimensional state vector through a fully connected layer as the dynamic representation of the target patient.

[0009] In the above scheme, the method further includes: obtaining a modality missing mask at each time point, where each bit of the modality missing mask corresponds to a modality type, used to identify whether data of the corresponding modality was collected at each time point; when performing spatial graph convolution, the modality missing mask is used to participate in the feature aggregation process of neighboring nodes to adjust the aggregation weight according to the missing status of each modality type at each time point; when performing temporal graph convolution, the modality missing mask is used to participate in the feature propagation process along the temporal edge to control the propagation gate value of historical features along the temporal edge according to the modality missing status at the current time point, wherein the propagation gate value is used to adjust the intensity of the evolution features of the medical entity node at the previous moment being transmitted to the corresponding node at the current moment along the temporal edge.

[0010] In the above scheme, predicting the future disease progression of a target patient based on the patient's dynamic representation includes: inputting the patient's dynamic representation into a prediction decoder, and the prediction decoder outputting a disease progression probability sequence within Q preset time windows, where Q is greater than or equal to 1; wherein, for each time window, the disease progression probability sequence includes at least one of the following: stable probability, progression probability, remission probability, and relapse probability.

[0011] The above scheme further includes: generating predictive interpretations, including: recording the first attention weight of each medical relationship edge during spatial graph convolution and recording the second attention weight of each temporal edge during temporal graph convolution; backtracking based on the first and second attention weights in response to the prediction results of disease progression; outputting the two medical entity nodes connected by the top M weighted medical relationship edges as key spatial evidence; and outputting the two time points connected by the top N weighted temporal edges as key time periods.

[0012] In the above scheme, the method further includes: when new follow-up data is added for the target patient, a new time point is added to the end of the temporal multimodal medical knowledge graph corresponding to the target patient; a multimodal medical knowledge subgraph corresponding to the new time point is constructed based on the new follow-up data; a temporal edge is established between the multimodal medical knowledge subgraph corresponding to the new time point and the multimodal medical knowledge subgraph corresponding to the previous adjacent time point; an affected update set is determined, the affected update set including the nodes and edges of the multimodal medical knowledge subgraph corresponding to the new time point, the temporal edge connected to the new time point, the medical entity nodes in the previous adjacent time point that are directly connected to the new time point through the temporal edge, and the k-hop neighborhood nodes of the medical entity nodes in the new time point on the medical relation edge; the historical cache representation outside the affected update set is reused, and spatial graph convolution and temporal graph convolution are re-executed only on the nodes within the affected update set to obtain the updated dynamic representation of the patient.

[0013] To achieve the above objectives, a second aspect of this application proposes a disease progression prediction system based on a temporal-spatial multimodal graph convolutional network, comprising: an acquisition module for acquiring multimodal medical follow-up data of a target patient at multiple time points; a construction module for constructing a temporal multimodal medical knowledge graph corresponding to the target patient based on the multimodal medical follow-up data, wherein each time point corresponds to a multimodal medical knowledge subgraph, each multimodal medical knowledge subgraph includes multiple medical entity nodes and medical relationship edges connecting the medical entity nodes, and different time points are connected by temporal edges connecting the medical entity nodes; and a spatial graph convolution module for performing spatial graph convolution on the multimodal medical knowledge subgraph at each time point. The system aggregates spatial medical features of medical entity nodes along medical relationship edges; a temporal graph convolution module performs temporal graph convolution on temporal edges between different time points to learn the temporal evolution features of medical entity nodes between different time points; wherein, different time points are connected by temporal edges to copies of the same medical entity node at different time points, and the temporal edges include at least one of the following: edge connecting the same lesion, edge connecting the same detection index, edge connecting disease state transition, and edge connecting the order of visits; a fusion module is used to fuse spatial medical features and temporal evolution features to obtain a dynamic representation of the patient's current and historical comprehensive state; and a prediction module is used to predict the future disease progression of the target patient based on the dynamic representation of the patient.

[0014] To achieve the above objectives, a third aspect of this application provides a computing device including a processor and a memory storing a computer program, wherein when the processor runs the computer program, it implements any of the above-described methods for predicting disease progression based on temporal-spatial multimodal graph convolutional networks.

[0015] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described methods for predicting disease progression based on temporal-spatial multimodal graph convolutional networks.

[0016] The technical solutions provided by the embodiments of this application may include the following beneficial effects: In this application, multimodal medical follow-up data of a target patient at multiple time points are obtained; a temporal multimodal medical knowledge graph is constructed based on the multimodal medical follow-up data; spatial graph convolution is performed on the multimodal medical knowledge subgraph at each time point to aggregate the spatial medical features of medical entity nodes along medical relationship edges; temporal graph convolution is performed on the temporal edges between different time points to learn the temporal evolution features of medical entity nodes between different time points; wherein, copies of the same medical entity node at different time points are connected through temporal edges, and the temporal edges include at least one of the following: edge connecting the same lesion, edge connecting the same detection index, edge connecting disease state transition, and edge connecting the order of visits; spatial medical features and temporal evolution features are fused to obtain a dynamic representation of the patient; and the future disease progression of the target patient is predicted based on the dynamic representation of the patient. Thus, by constructing a knowledge subgraph containing medical relationship edges at each time point and establishing temporal edges across time points, spatial graph convolution and temporal graph convolution can propagate features along medical relationship edges and temporal edges respectively within a unified graph structure. This simultaneously captures medical associations within the same time point and disease evolution patterns between different time points, enabling the simultaneous use of medical relationship structure and temporal evolution information for disease progression prediction. Furthermore, by mapping each follow-up visit to a multimodal medical knowledge subgraph, each subgraph contains medical entity nodes and their relationship edges, and different time points connect different time point copies of the same medical entity node through temporal edges, the multimodal data of patients at multiple time points, including images, tests, and text, are uniformly organized into a graph data form that combines spatial medical semantics and temporal sequence structure. This provides a structured data foundation for subsequent dual-path graph convolution, enabling the organization of patient follow-up data at multiple time points into a structured temporal graph representation. Moreover, by fusing the spatial medical features output by spatial graph convolution and the temporal evolution features output by temporal graph convolution, a dynamic representation of the patient's current and historical comprehensive state is obtained, and future disease progression can be predicted based on this representation. This dynamic representation simultaneously encodes the aggregated results of medical evidence for the patient at the same point in time and the trend of state changes between different points in time, which can more comprehensively reflect the patient's disease progression and thus improve prediction accuracy.

[0017] In summary, the embodiments of this application construct a temporal multimodal medical knowledge graph containing medical relationship edges at each time point and time edges across time points. Spatial graph convolution and temporal graph convolution are performed along the medical relationship edges and time edges respectively, so that spatial medical features and temporal evolution features are synchronously learned and fused in a unified graph structure. This results in a dynamic representation of the patient's current and historical comprehensive state to predict future disease progression. This achieves joint modeling of medical structural relationships and temporal evolution information in patient follow-up data at multiple time points, improving the accuracy of disease progression prediction. Attached Figure Description

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings: Figure 1 A flowchart illustrating the disease progression prediction method based on temporal-spatial multimodal graph convolutional networks provided in this application embodiment; Figure 2 A schematic diagram of the technical architecture for disease progression prediction based on temporal-spatial multimodal graph convolutional networks provided in this application embodiment; Figure 3 A schematic diagram illustrating the application scenarios of time points and time edges in the temporal multimodal medical knowledge graph provided in this application embodiment; Figure 4 A schematic diagram illustrating a spatiotemporal fusion scenario in disease progression prediction based on a temporal-spatial multimodal graph convolutional network, as provided in an embodiment of this application. Figure 5 A schematic diagram illustrating the application of incremental updates of new follow-up data provided in a specific embodiment of this application; Figure 6 A flowchart illustrating the disease progression prediction method based on temporal-spatial multimodal graph convolutional networks provided in a specific embodiment of this application; Figure 7 A schematic diagram illustrating an application scenario of the disease progression prediction method based on temporal-spatial multimodal graph convolutional networks provided in a specific embodiment of this application; Figure 8 A schematic diagram of the composition structure of a disease progression prediction system based on a temporal-spatial multimodal graph convolutional network provided in an embodiment of this application; Figure 9 A schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] In this application, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this application and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation.

[0022] Furthermore, in addition to indicating location or positional relationship, some of the aforementioned terms may also have other meanings. For example, the term "above" may also be used in some cases to indicate a certain dependency or connection relationship. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances.

[0023] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linked," and "socketing" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0024] It should be noted that the embodiments of this application can be applied to clinical decision support fields such as tumor progression prediction, chronic disease exacerbation risk prediction, relapse risk assessment, and treatment response prediction. For example, by acquiring multimodal data such as CT / MRI images, image reports, electronic medical record texts, structured laboratory indicators, vital signs, and treatment plans at multiple follow-up time points from a hospital's radiology department, oncology department, chronic disease management department, or health checkup center, the method of this application is used to construct a multimodal medical knowledge subgraph for each time point, including image region nodes, text fragment nodes, laboratory indicator nodes, treatment plan nodes, and disease status nodes. These subgraphs are connected by medical relationship edges such as description, mention, and support status. Furthermore, they are connected by time edges such as continuous edges for the same lesion, continuous edges for the same indicator, and disease status transition edges. Simultaneously, a patient-level temporal multimodal medical knowledge graph is formed. Spatial medical features of each medical entity node within the same time point are aggregated along the medical relationship edges through spatial graph convolution. Temporal evolution features of lesion changes, indicator fluctuations, and state transitions between different time points are learned along the temporal edges through temporal graph convolution. After fusion, a dynamic representation of the patient's current and historical comprehensive state is obtained. Finally, the probability of progression, recurrence risk, deterioration level, or treatment response probability within one or more future time windows are output. This aims to solve the problem that existing technologies cannot simultaneously utilize medical relationship structure and temporal evolution information for disease progression prediction.

[0025] Understandably, the clinical application of disease progression prediction currently faces two major challenges: the disconnect between time-series models and graph structure modeling, and the difficulty in unifying the modeling of multi-time-point and multi-modal data. First, the disconnect between time-series models and graph structures is evident in the fact that while sequence models such as RNNs and LSTMs can utilize longitudinal patient follow-up data, they struggle to explicitly express the medical relationship structure between disease, symptoms, laboratory indicators, imaging findings, and treatment plans. While graph neural networks or knowledge graph methods can utilize medical relationship structures, they typically treat the graph as static, making it difficult to model state transitions and disease evolution processes between different follow-up time points. Even when some methods attempt to combine the two, they often involve simple splicing at the feature layer, failing to allow spatial medical relationships and temporal evolutionary relationships to propagate synchronously within the same graph structure. Second, the difficulty in unifying the modeling of multi-time-point and multi-modal data stems from the lack of effective means to organize different modalities of patient follow-up data, such as imaging, laboratory tests, text data, and treatment data, into a unified graph structure. This makes it difficult for the model to learn simultaneously from the aggregation of medical evidence at each time point and the state changes between time points, resulting in an incomplete representation of disease progression and affecting prediction accuracy.

[0026] Based on this, the embodiments of this application aim to provide a disease progression prediction method that can integrate multimodal follow-up data of patients at multiple time points into the medical relationship structure and time evolution pattern for unified modeling, so as to assist doctors in oncology, chronic disease management and radiology departments to accurately assess the risk of future disease progression of patients, identify key changing trends, and improve the predictive ability and clinical application value of medical AI systems in longitudinal follow-up scenarios.

[0027] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Figure 1 This is a flowchart illustrating the medical interpretation and processing method provided in the embodiments of this application, such as... Figure 1 As shown, applied to a computing device, the method includes: Step 101: Obtain multimodal medical follow-up data of the target patient at multiple time points.

[0029] The computing device can be a fixed terminal or a mobile terminal. Fixed terminals can be diagnostic workstations in hospital oncology or radiology departments, servers in chronic disease management departments, local servers in health checkup centers, local inference servers in hospitals, or computing nodes in cloud-based medical AI platforms; mobile terminals can be laptops, tablets, or handheld terminals used by doctors, or smartphones or tablets used by authorized family members of patients. For example, it could be a desktop computer deployed in a hospital's oncology department, used to receive multimodal data such as CT / MRI images, image reports, structured laboratory indicators, treatment plans, and disease status from multiple follow-up time points of the patient, and run the temporal-spatial multimodal graph convolutional network model described in this application to output the probability of future disease progression and key time point evidence paths for the patient; it could also be a workstation deployed in a chronic disease management department, used to obtain the patient's long-term follow-up laboratory indicators, medication records, and symptom text, predict the risk of future deterioration, and mark the changing trends of key indicators; or it could be a cloud server, which receives multimodal data from patients uploaded from primary healthcare institutions or health check-up centers via an API interface, performs temporal graph construction and dual-path graph convolutional inference, and returns the disease progression prediction results and key evidence backtracking information to the doctor's end, supporting remote consultation and hierarchical medical treatment scenarios. The computing device obtains data through a file reading interface, database query interface, or application programming interface (API) and loads it into memory for subsequent steps.

[0030] In this context, the target patient refers to the specific subject being evaluated who requires prediction of disease progression, i.e., the subject targeted by the temporal multimodal medical knowledge graph constructed and the prediction method implemented in this application. The target patient can be a patient already diagnosed with a certain type of disease (such as a cancer patient, a chronic disease patient, or a neurodegenerative disease patient), or a screening subject with specific risk factors who requires dynamic monitoring and follow-up evaluation.

[0031] It should be added that, during the training phase, the model needs to be trained using a large number of sample patients (i.e., historical patient groups) with multi-time-point and multi-modal follow-up data and their corresponding known progression outcome labels; during the inference phase, for the specific patient individual for whom prediction is needed (i.e. target patient), the trained model is used to generate disease progression prediction results for that patient individual based on the patient's existing multi-time-point multi-modal follow-up data.

[0032] Here, the target patient's identity information is anonymized upon entering the system, retaining only medical data features related to disease progression prediction and excluding privacy information directly associated with personal identity. Follow-up data for each time point of the target patient is represented in the temporal multimodal medical knowledge graph as corresponding time point nodes and associated medical entity nodes. The prediction result is output based on the patient's current dynamic representation.

[0033] Multimodal medical follow-up data refers to medical-related data generated by target patients during the follow-up process from multiple data modalities, including imaging data (such as medical images and their derivative data such as CT, MRI, PET-CT, X-ray, ultrasound, etc.), text data (such as text descriptions in imaging reports, electronic medical records, discharge summaries, progress notes, surgical records, and chief complaint texts in laboratory reports), structured laboratory indicator data (such as laboratory test results such as complete blood count, tumor markers, biochemical indicators, coagulation function, and blood gas analysis), vital sign data (such as blood pressure, heart rate, body temperature, respiratory rate, and blood oxygen saturation), treatment plan data (such as surgery, radiotherapy, chemotherapy, targeted therapy, immunotherapy, drug therapy, and their start and end times, dosages, and courses of treatment), and disease status label data (such as disease stage, pathological grade, efficacy evaluation, progression, recurrence, and remission). Data from different modalities reflects a patient's disease status from different dimensions: imaging data provides information on anatomical structure and lesion morphology; laboratory indicators provide information on the internal environment and functional status of the body; textual data provides clinical descriptions, diagnostic opinions, and treatment process information; treatment plans reflect interventions and their changes; and disease status labels mark the outcome or prognosis of the disease at corresponding time points. These multiple modalities of data collectively constitute a complete information foundation for predicting disease progression.

[0034] Here, a time point refers to a time marker indicating when a medical action occurs for the target patient, including but not limited to the date of consultation, examination, hospitalization, discharge, follow-up, or a preset time window. Each time point corresponds to a complete medical event or data collection event for the patient, and different time points for the same patient have a clear chronological order. For example, a cancer patient may undergo multiple CT scans during treatment, and the date of each scan is a time point; a patient with a chronic disease may have a blood test once a month, and the date of each test is a time point; for scenarios with irregular follow-ups, data can be divided according to preset time windows (e.g., every 3 months), and data within the same time window are grouped into the same time point. The time interval between adjacent time points can be equal (e.g., once a month for follow-up) or unequal (e.g., follow-up at the 1st month, 3rd month, and 6th month after surgery), and this application embodiment does not limit this.

[0035] It is understandable that each "time point" corresponds to a set of "multimodal medical follow-up data," and the "multimodal medical follow-up data" at each "time point" may contain one or more of the aforementioned modalities. The modality types and data volumes contained at different time points may be the same or different, depending on the actual examination items and data collection at each time point in clinical practice. Step 101 of this application obtains all available multimodal data of the target patient at all time points, which will be used to subsequently construct a temporal multimodal medical knowledge graph.

[0036] For example, please refer to Figure 2 ,like Figure 2 As shown, the patient's multi-time point follow-up data input includes time points t1, t2...t3, etc. Each time point follow-up data includes, but is not limited to: images, reports, laboratory indicators, medication, symptoms, and disease status.

[0037] Step 102: Construct a time-series multimodal medical knowledge graph corresponding to the target patient based on multimodal medical follow-up data.

[0038] Each time point corresponds to a multimodal medical knowledge subgraph, and each multimodal medical knowledge subgraph contains multiple medical entity nodes and medical relationship edges connecting the medical entity nodes. Different time points are connected through time edges connecting the medical entity nodes.

[0039] For example, please refer to Figure 2 ,like Figure 2 As shown, in the construction of the temporal multimodal medical knowledge graph, the time slice subgraph, namely the multimodal medical knowledge subgraph mentioned above, consists of multiple medical entity nodes, namely different categories of medical follow-up data, and medical relationship edges, namely medical relationship edges connecting various medical entity nodes, such as connecting images and reports.

[0040] Each time point corresponds to a multimodal medical knowledge subgraph (i.e., a time slice subgraph). Each multimodal medical knowledge subgraph contains multiple medical entity nodes and medical relationship edges connecting the medical entity nodes. Different time points are connected through the time edges connecting the medical entity nodes.

[0041] For example, please refer to Figure 2 ,like Figure 2 As shown, the temporal multimodal medical knowledge graph consists of multiple time slice subgraphs (such as t1, t2, and t3 subgraphs). Each time slice subgraph contains multiple medical entity nodes and medical relationship edges connecting the medical entity nodes. Different time slice subgraphs are connected by time edges.

[0042] Specifically, the medical entity nodes refer to graph nodes used to represent objective entities with independent semantics and their attribute information in the medical field. Each medical entity node corresponds to a specific medical concept or data unit, and different medical entity nodes within the same time slice subgraph can represent different modalities or types of medical data. For example... Figure 2 As shown, medical entity nodes include, but are not limited to, image nodes (such as CT images, MRI images and lesion areas within them), report nodes (such as image report text, medical record text), test indicator nodes (such as blood routine tests, tumor markers), medication nodes (such as chemotherapy drugs, targeted drugs), symptom nodes (such as cough, chest pain, fatigue), and disease status nodes (such as stable, progressive, remission, relapse). Different categories of medical entity nodes are identified by different shapes or colors to distinguish their respective modalities. Each medical entity node has a corresponding feature representation; for example, an image node corresponds to the feature vector of the image region, a report node corresponds to the semantic embedding vector of the text fragment, and a test indicator node corresponds to the encoding vector of the test value.

[0043] Here, the medical relation edge (i.e., spatial edge) refers to the edge used to connect two medical entity nodes within the same time slice subgraph, representing the medical semantic relationship between different medical entities at the same point in time. For example... Figure 2As shown, medical relationship edges include, but are not limited to, the following types: Depicts, connecting image nodes and image report nodes, indicating that the signs shown in the image are described in the report; Mentions, connecting report nodes and symptom nodes or disease state nodes, indicating that the corresponding symptom or disease is mentioned in the text; Located in, connecting lesion nodes and anatomical location nodes, indicating the anatomical location of the lesion; Has Indicator, connecting disease state nodes and test indicator nodes, indicating that the test indicator is used to support or reflect the disease state; Treated by, connecting disease state nodes and medication nodes, indicating that the drug or treatment plan is used to treat the disease state; Supports State, connecting image nodes, report nodes, or test indicator nodes and disease state nodes, indicating that the evidence supports the corresponding disease state determination; Contradicts State, connecting evidence nodes and disease state nodes, indicating that the evidence contradicts the corresponding disease state determination. Different types of medical relationship edges are identified by different line styles or colors. Figure 2 Solid lines represent spatial relationship edges. Medical entity nodes within the same time slice subgraph are interconnected through these medical relationship edges, forming a multimodal medical knowledge subgraph for that time point. This subgraph is used in subsequent spatial graph convolutions to aggregate spatial medical features along these medical relationship edges.

[0044] Similarly, as Figure 3 As shown, edges represented by solid lines within the same time point are spatial edges. For example, an image region ROI node is connected to a report text node via a "descriptive" relationship edge, indicating that what is seen in the image is described in the report; a test indicator CEA node is connected to a disease state node via a "has indicator" relationship edge, indicating that the test indicator is used to support or reflect the determination of the disease state; a disease state node is connected to a treatment node via a "treated_by" relationship edge, indicating that the disease state is treated with the corresponding treatment plan. Spatial edges connect different medical entity nodes within the same time slice subgraph (such as an image region ROI node and a report text node), rather than copies of the same entity at different times. The role of spatial edges is to construct structured relationships between multimodal medical evidence within the same time point, enabling subsequent spatial graph convolution to perform message passing along the spatial edges between different medical entity nodes, aggregating spatial medical features within the same time point. The types of spatial edges include, but are not limited to, those mentioned above. Figure 3 The relationships shown include "descriptive" relationships, "has indicator" relationships, "treated_by" relationships, as well as descriptive relationships, mention relationships, location relationships, supporting state relationships, and refuting state relationships.

[0045] Here, the time edge refers to the edge used to connect corresponding medical entity nodes in different time slice subgraphs, representing the temporal sequence and evolution path of the same medical entity at different time points. For example... Figure 2 As shown, time edges connect image nodes in subgraph t1 to corresponding image nodes in subgraph t2, representing the continuous observation relationship of the lesion corresponding to the image at different time points. Time edges can also connect test indicator nodes in subgraph t1 to corresponding test indicator nodes in subgraph t2, representing the numerical change relationship of the same test indicator at different time points. Time edges can also connect disease state nodes at different time points, representing the transition relationship of disease state over time. Specifically, time edges include, but are not limited to, the following types: consecutive edges for the same lesion (same_lesion_next), connecting image nodes representing the same lesion at different time points, representing continuous tracking and observation of the lesion; consecutive edges for the same test indicator (same_indicator_next), connecting nodes representing the same test indicator at different time points, representing dynamic monitoring of the indicator; disease state transition edges (state_transition), connecting disease state nodes at adjacent time points, representing the transition of disease state from the previous state to the current state over time; and visitation sequence edges (visit_next), connecting visitation event nodes at adjacent time points, representing the temporal order between two follow-up visits. The different time points are connected by the aforementioned time edges to form a patient-level temporal multimodal medical knowledge graph, which is used for subsequent temporal graph convolution to learn the temporal evolution characteristics of medical entity nodes between different time points along the time edges.

[0046] Similarly, as Figure 3 As shown, the edges represented by dashed lines / thick arrows between different points in time are called time edges. Unlike spatial edges, which connect different entity nodes within the same point in time, time edges are specifically used to connect the same entity nodes between different points in time. For example... Figure 3 As shown: the ROI nodes of the imaging region are connected sequentially by dashed arrows between t1, t2, and t3, indicating continuous tracking and observation of the same lesion region; the CEA nodes of the test index are connected sequentially by dashed arrows between t1, t2, and t3, indicating dynamic monitoring of the same tumor marker; the disease state nodes are connected sequentially by dashed arrows between t1, t2, and t3, indicating the state transition path of the disease state over time.

[0047] It should be noted that the time edges connect copies of the same medical entity at different time points (e.g., the CEA node at time t1 and the CEA node at time t2), rather than different entities within the same time point. The purpose of the time edges is to associate the same medical entity of the same patient at different time points, forming a temporal evolution chain. This allows subsequent temporal graph convolutions to pass messages between corresponding nodes at different time points along the time edges, learning the temporal evolution characteristics of medical entity nodes across different time points.

[0048] Furthermore, the difference and synergy between spatial and temporal edges can be as follows: spatial edges connect different entity nodes within the same point in time (e.g., the ROI of an image at time t1 and the report text at time t1), achieving structured association of cross-modal medical evidence within the same point in time; temporal edges connect the same entity node at different points in time (e.g., the ROI of an image at time t1 and the ROI of an image at time t2), achieving the evolution tracking of the same entity along the time axis. Both... Figure 2 Different line types are used to clearly distinguish spatial edges: solid lines represent spatial edges, and dashed lines / thick arrows represent temporal edges. Spatial edges provide the graph with structured information about "which medical evidences are related at the same point in time," while temporal edges provide the graph with temporal information about "how the same entity changes at different points in time." Together, they constitute the "space-time" two-dimensional graph structure of the temporal multimodal medical knowledge graph, providing a structural foundation for the subsequent synchronous propagation of features along the two types of edges within the same framework through spatial graph convolution and temporal graph convolution.

[0049] Furthermore, such as Figure 3 As shown, the time edge also carries corresponding time edge attributes, including but not limited to: interval length Δt, which is the actual calendar time interval between two adjacent time points (e.g., ...). Figure 2 The Δt labels between t1 and t2, and between t2 and t3, are used to modulate the propagation intensity at different time intervals in temporal graph convolution; the treatment stage, i.e., the treatment stage the patient receives within the time period spanned by the temporal edge; and the follow-up type, i.e., the follow-up nature corresponding to the temporal edge. Temporal edge attributes are stored together with the temporal edge and are retrieved in subsequent temporal graph convolutions.

[0050] Therefore, by defining, distinguishing and co-connecting spatial and temporal edges as described above, the multimodal follow-up data of the target patient at multiple time points are organized into a unified temporal multimodal medical knowledge graph that combines spatial medical semantics and temporal evolution structure. This provides a structured graph data foundation for subsequent spatial graph convolution to aggregate spatial medical features along spatial edges and temporal graph convolution to learn temporal evolution features along temporal edges.

[0051] Specifically, the system represents patient p's follow-up data as follows: G_p = {G_p^1, G_p^2, …, G_p^T, E_time} Wherein, G_pt represents the multimodal medical knowledge subgraph of the patient at time point t, and E_time represents the set of time edges across time points. Each time slice subgraph G_pt includes: Image nodes and image region nodes.

[0052] Image report text node and medical record text node.

[0053] Verify indicator nodes and vital sign nodes.

[0054] Symptoms, signs, treatment plan, and medication timing.

[0055] Disease status, risk level, or progression status nodes.

[0056] Medical entity nodes and candidate conclusion nodes.

[0057] Spatial edges within the same time point include depicts, mentions, located_in, has_indicator, treated_by, supports_state, and contrasts_state. Temporal edges between different time points include same_lesion_next, same_indicator_next, state_transition, treatment_followed_by, and visit_next.

[0058] Step 103: Perform spatial graph convolution on the multimodal medical knowledge subgraph at each time point to aggregate the spatial medical features of medical entity nodes along the medical relation edges.

[0059] Spatial graph convolution refers to a graph convolution operation performed based on the graph structure of the multimodal medical knowledge subgraph, with the medical entity nodes as the processing objects and the medical relationship edges as the message propagation paths. For example... Figure 2 As shown, within each time slice subgraph (t1, t2, or t3), the spatial graph convolution treats all medical entity nodes within that time point as the processing object, with... Figure 2 or Figure 3The medical relationship edges (i.e., spatial edges), represented by solid lines, serve as message propagation paths. Features of each medical entity node are propagated along these solid lines to its neighboring nodes. A learnable relational transformation matrix maps the features of neighboring nodes, which are then aggregated onto the target node, thereby achieving information fusion between cross-modal medical evidence at the same time point. Spatial graph convolution is performed only within a single time slice subgraph and does not involve cross-time point information transfer. Its output is the updated feature representation of each medical entity node, i.e., the spatial medical features.

[0060] In this context, spatial medical features refer to the feature representations carried by each medical entity node in the multimodal medical knowledge subgraph after convolution of the spatial graph. These feature representations are not the original input data (such as image pixel values, test values, or text word vectors), but rather updated features obtained after the original features are propagated and aggregated along the medical relationship edges during spatial graph convolution. For example... Figure 2 As shown, after spatial graph convolution, the ROI nodes at time t1 not only carry their own image features but also incorporate the semantic features of the report text nodes propagated through "descriptive" relation edges; the CEA test indicator nodes not only carry their own test value features but also incorporate the state features of the disease state nodes propagated through "has indicator" relation edges. Therefore, the spatial medical features are a comprehensive representation of multimodal medical evidence within the same time point after structured aggregation, including the node's own information and the information of its neighboring nodes connected by its medical relation edges, for subsequent fusion with temporal evolution features.

[0061] It is understandable that aggregation along medical relation edges refers to the specific path and method of message passing during the execution of the spatial graph convolution. For example... Figure 2 or Figure 3 As shown, message passing in spatial graph convolution strictly follows the path... Figure 2 or Figure 3 The medical relationship edges, represented by solid lines, are used for convolution, without crossing dashed lines / thick arrows (i.e., time edges) between different time points. Specifically, for each target medical entity node, spatial graph convolution collects the features of all neighboring nodes through the medical relationship edges connected to it. The neighboring features corresponding to different relationship types are mapped by the corresponding relationship transformation matrix and then added together to obtain the aggregation result. Figure 2 In the process, the features of the report text node are aggregated along the "descriptive" solid line edge of the ROI node in the image region; the features of the disease status node are aggregated along the "has indicator" solid line edge of the CEA node; and the features of the treatment node are aggregated along the "treated_by" solid line edge of the disease status node. The "medical relationship edge" in "aggregation along medical relationship edge" refers to... Figure 2 or Figure 3The solid lines representing spatial edges in the graph are strictly distinguished from the dashed lines / thick arrows (temporal edges) connecting different points in time. Spatial graph convolution propagates information only along the solid lines, not along the dashed lines / thick arrows. This is the essential difference between spatial graph convolution and subsequent temporal graph convolution in terms of propagation path.

[0062] Thus, through step 103, the original multimodal features of each medical entity node in the multimodal medical knowledge subgraph at each time point are spatially propagated and aggregated along the medical relationship edges to obtain the spatial medical features of each node, providing a spatial dimension feature representation for fusion with temporal evolution features in subsequent steps.

[0063] In some implementations, spatial graph convolution is performed on the multimodal medical knowledge subgraph at each time point to aggregate the spatial medical features of the medical entity nodes along the medical relation edges, including: Obtain the type identifier of each medical relationship edge within the current time point; Based on the type identifier, different relationship transformation matrices are assigned to different medical relationship types; For each medical entity node at the current time point, the features of neighboring nodes connected to the medical entity node through various medical relationship edges are aggregated. The aggregation results of different relationship types are added together after being mapped by the corresponding relationship transformation matrix to obtain the spatial medical features of the medical entity node.

[0064] Among them, the type identifier is used to distinguish different medical semantic relationship categories, such as descriptive relationship, mention relationship, indicator relationship, treatment relationship, support status relationship, etc.

[0065] It is understandable that the neighbor nodes connected by different medical relationship types have different semantic meanings. For example, the "descriptive" relationship conveys semantic description information of imaging signs, the "has indicator" relationship conveys quantitative test information reflecting disease status, and the "treated_by" relationship conveys intervention information. These information are distributed differently and have different semantic dimensions in the original feature space. Therefore, it is necessary to configure a learnable transformation matrix independently for each relationship type in order to map the neighbor features of different relationship types to a unified target feature space.

[0066] Specifically, within each time slice G_pt, the system performs spatial graph convolution. For the representation h_v^{t,l} of node v at layer l, spatial propagation can be represented as: h_v^{t,l+1} = sigma( sum_{r in R} sum_{u in N_r(v)} alpha_{u,r,v}^tW_r h_u^{t,l} ) Where R represents the set of relation types, N_r(v) represents the set of neighbor nodes connected to v through relation r, W_r represents the transformation matrix corresponding to the relation type, and alpha represents the relation attention weight. Through spatial propagation, the model can learn the medical relationships between image features, abnormalities, symptoms, and treatment plans at the same point in time.

[0067] The above formula means that for the target node v, all relation types r are traversed. For each relation type r, all neighbor nodes u connected to v through that relation are traversed. The current layer features of the neighbor node u are mapped by the transformation matrix Wr corresponding to that relation type and multiplied by the attention weight α. The mapping results of all neighbor nodes of all relation types are added together and then activated by the activation function σ to obtain the features of the target node v in the next layer.

[0068] It should be noted that the attention weight α is normalized across all node pairs at the same time point to ensure that the contributions of different neighboring nodes are comparable. The time point superscript t indicates that the above spatial graph convolution is only performed within a single time slice subgraph and does not involve information transfer across time points. The attention weights and transformation matrix are shared between different time points.

[0069] In the above implementation, through the spatial graph convolution, the model can perform message passing and feature aggregation along medical relationship edges within each time point. This ensures that the spatial medical features of each medical entity node not only include its own original features but also incorporate feature information from neighboring nodes propagated through various medical relationship edges. Therefore, the model can learn the structured medical relationships between imaging features, abnormal test results, symptoms, treatment plans, and disease states within the same time point, providing rich spatial-dimensional feature representations for subsequent fusion with temporal evolution features.

[0070] Step 104: Perform temporal graph convolution on the temporal edges between different time points to learn the temporal evolution characteristics of medical entity nodes between different time points. Specifically, temporal edges connect copies of the same medical entity node at different time points, and these temporal edges include at least one of the following: edges connecting the same lesion, edges connecting the same detection index, edges connecting disease state transitions, and edges connecting the order of medical visits.

[0071] Temporal graph convolution refers to a graph convolution operation performed using time edges across time points in the temporal multimodal medical knowledge graph as message propagation paths and replicas of the same medical entity node at different time points as processing objects. For example... Figure 2 or Figure 3 As shown, the time-plot convolution along Figure 2 or Figure 3The temporal graph convolution, represented by dashed lines / thick arrows (such as dashed arrows connecting ROI nodes in the image region at times t1, t2, and t3, or dashed arrows connecting nodes of the same CEA index), propagates the evolutionary features of the medical entity node from the previous time step to the corresponding node at the current time step, thereby capturing the change pattern of the medical entity on the time axis. Unlike the spatial graph convolution in step 103, which propagates features between different entities at the same time point along solid lines, the temporal graph convolution propagates features between copies of the same entity at different time points along dashed lines / thick arrows. Temporal graph convolution can be implemented using a temporal graph network, gated recurrent units, or a temporal attention mechanism. Its core lies in using temporal edges to establish dependencies of the same entity on the time axis, modeling the dynamic evolution of the entity over time.

[0072] The temporal evolution feature refers to the feature representation carried by medical entity nodes at each time point after convolution of the temporal graph, which integrates information from historical time points. This feature representation reflects the changing trend and evolution pattern of the corresponding medical entity (such as lesions, test indicators, and disease states) over time. Specifically, after convolution of the temporal graph, the ROI node of the image region at time t2 not only carries its own spatial medical features at time t2, but also integrates the evolutionary information of the same lesion at time t1 propagated along the temporal edge; the disease state node at time t3 not only carries its own spatial medical features at time t3, but also integrates the disease state evolutionary information from times t1 and t2 propagated along the disease state transfer edge. Therefore, the temporal evolution feature is a comprehensive representation of the state information of the same medical entity at historical time points accumulated to the current time point, used for subsequent fusion with spatial medical features to characterize the patient's current and historical comprehensive state.

[0073] In some implementations, temporal graph convolution is performed on the temporal edges between different time points to learn the temporal evolution characteristics of medical entity nodes between different time points, including: Get the actual calendar time interval between two adjacent time points; A time interval code is generated based on the actual calendar time interval. The time interval code is used to represent the duration information between adjacent follow-ups. The time interval code and the type representation of the current time edge are input together into the gating function to calculate the propagation gate control on the time edge; Along the time edge, the evolutionary characteristics of the medical entity node at the previous time point are multiplied by the propagation gating value and then passed to the corresponding entity node at the current time point to obtain the temporal evolution characteristics of the medical entity node between different time points.

[0074] The actual calendar time interval can be, for example, Figure 3As shown, Δt between t1 and t2, and Δt between t2 and t3, represent the actual number of days or months between two adjacent follow-up visits. The follow-up intervals may differ between different patients, and the intervals between different follow-up stages for the same patient may also differ (e.g., follow-up at the 1st month, 3rd month, and 6th month post-surgery). Therefore, it is necessary to obtain the specific time difference for each pair of adjacent time points.

[0075] The time interval encoding can take various forms, including but not limited to: normalized time difference (normalized by dividing Δt by the maximum follow-up interval), logarithmic transformation time difference log(1+Δt), exponential decay coefficient exp(-β·Δt) based on the relation type, or periodic time encoding (used to express the periodic pattern of treatment cycle or follow-up cycle).

[0076] The time edge type identifier is used to distinguish different types of time edges (such as lesion continuity edges, same indicator continuity edges, disease state transition edges, and visit sequence edges). Different types of time edges should have different propagation intensities within the same time interval. For example, for the same_lesion_next edge, a longer time interval can enhance the progressive meaning of lesion size changes; that is, the longer the interval, the more significant the actual changes in the lesion. Conversely, for the visit_next edge, a longer time interval reduces the direct propagation intensity of ordinary visit sequence edges; that is, the longer the interval, the greater the random fluctuation interference between two visits. Furthermore, the gating value can be calculated as follows: After linear transformation of the time interval encoding and the type identifier of the time edge, the sigmoid function is activated to output a gating value between 0 and 1; wherein, the time interval encoding includes at least one of log(1+Δt) transform value, normalized Δt value or exponential decay value in the form of exp(-β·Δt), where Δt is the actual calendar time interval.

[0077] Furthermore, after spatial propagation, the system performs temporal propagation along the temporal edge. For node sequences of the same lesion, the same indicator, or the same disease state at different time points, the system uses temporal graph convolution, gated recurrent units, or temporal attention to compute evolutionary representations: g_v^{t+1} = TGCN(h_v^{t}, h_v^{t-1}, Delta t, edge_time) Where Delta_t represents the interval between adjacent time points, and edge_time represents the time edge type. The system can learn the decay coefficient based on the time interval, avoiding treating follow-ups separated by one day and one year as equal distances.

[0078] In the above implementation, through the temporal graph convolution, the model performs message passing and feature aggregation on copies of the same medical entity at different time points along the temporal edge. This ensures that the temporal evolution features of medical entity nodes at each time point not only include the spatial medical features of the current time point but also accumulate evolutionary information from historical time points. Therefore, the model can learn the changing trends of the same lesion across multiple follow-up time points (e.g., lesion continuously increasing, stabilizing, or shrinking), the temporal fluctuation patterns of the same test indicator (e.g., CEA continuously increasing, fluctuating, or decreasing), and the temporal transition path of disease states (e.g., from stable to progressive, from progressive to remission), providing a temporal dimension feature representation for subsequent fusion with spatial medical features.

[0079] In some implementations, performing temporal graph convolution on temporal edges between different time points includes: generating time interval codes based on the actual time intervals between adjacent time points, and calculating the temporal edge message propagation strength based on the time interval codes and temporal edge types to regulate the propagation of evolutionary characteristics of lesions, test indicators, treatment plans, or disease states between different follow-up time points. Thus, through time interval coding and temporal edge propagation gating, the model can distinguish lesion changes, indicator fluctuations, and state transitions at different follow-up intervals, avoiding treating all adjacent follow-ups as equidistant events.

[0080] In some implementations, the method further includes: Obtain the modality missing mask at each time point. Each bit of the modality missing mask corresponds to a modality type and is used to identify whether data of the corresponding modality was collected at the time point. When performing spatial graph convolution, the modality missing mask is used to participate in the feature aggregation process of neighboring nodes to adjust the aggregation weights according to the missing status of each modality type at each time point; When performing temporal graph convolution, a modality missing mask is used to participate in the feature propagation process along the temporal edge to control the propagation gate value of historical features along the temporal edge according to the modality missing status at the current time point. This propagation gate value is used to adjust the intensity of the evolution features of the medical entity node at the previous time point being transmitted to the corresponding node at the current time point along the temporal edge.

[0081] The missing mask, denoted as m_t, is a vector or matrix used to identify whether data for each modality at each time point has been collected, where t represents the time point index. Each bit of the missing mask corresponds to a modality type, indicating whether data for that modality was collected at that time point.

[0082] The missing mask is generated as follows: When constructing the temporal multimodal medical knowledge graph in step 102, for each time slice subgraph, the system checks whether each modality's data exists. Specifically, the missing data for the image modality is determined by whether a usable image file was collected at that time point; the missing data for the test indicators is determined by whether there is a corresponding test report value at that time point; the missing data for the text modality is determined by whether there is a usable image report text or electronic medical record at that time point; and the missing data for the treatment plan is determined by whether there is a medication record or treatment record at that time point. If a modality's data is not collected or is unavailable after collection at that time point, the missing mask bit corresponding to that modality is marked as missing; if it is collected and available, it is marked as present.

[0083] It should be noted that the missing mask does not indicate that the medical fact is negative (i.e., it does not mean "the modality is normal" or "the test indicator is within the normal range"), but only that no evidence of the corresponding modality was observed at that time point. For example, if no CT scan was performed at a certain time point, the image modality missing mask bit will be marked as missing. This does not mean that the patient's lesion is absent or normal, but only that there are no CT images available for analysis at that time point. The missing mask is unrelated to the feature value itself and is metadata independent of the feature value, used to guide the model to distinguish between "missing evidence" and "negative evidence".

[0084] Understandably, in spatial graph convolution, a missing data mask is used in the feature aggregation process of neighboring nodes. When a certain modality of data is missing at a certain time point, the medical entity node corresponding to that modality may not exist in that time slice subgraph, or its node features may be missing embedding vectors. The missing data mask reduces the information contribution of the node corresponding to the missing modality to other nodes within the same time slice by affecting the aggregation weight of the node or its associated edges, thus avoiding misjudging "uncollected evidence" as "evidence that does not support the current conclusion."

[0085] Understandably, in temporal graph convolution, the missing mask participates in the feature propagation process along temporal edges. When data for a certain modality is missing at a target time point, the corresponding medical entity node for that modality has no new observations to update its features. In this case, the missing mask reduces the propagation gating value of the temporal edge corresponding to that modality, allowing the evolutionary features of historical time points to be transmitted to the current time point with greater intensity, thereby maintaining the continuity of historical information; at the same time, it avoids excessive amplification of missing embedding noise caused by the lack of observations at the current time point. The mechanism by which the missing mask participates in both spatial graph convolution and temporal graph convolution ensures that the model can still run stably under common clinical scenarios with incomplete follow-up data.

[0086] The propagation gating value refers to the degree of attenuation or enhancement of information transmitted along the time edge from a medical entity node in the previous time step to the corresponding node in the current time step. The propagation gating value controls the contribution of historical evolution features to the features of the current node in the temporal graph convolution, determining the balance between "how much historical information is retained and how much current observation is adopted." The propagation intensity is comprehensively regulated by multiple factors, including the type identifier of the current time edge, the actual calendar time interval between adjacent time points, and the missing mask state of the modality corresponding to the target time point.

[0087] Understandably, the propagation gating value ranges from 0 to 1 (inclusive). A propagation strength close to 1 indicates that historical features are almost completely transmitted to the current time point, and the current observation contributes very little to the update. This is suitable for the following situations: data for this modality is missing at the current time point, and there is a lack of new observations to update the current state, so historical evolution information should be relied upon; or the disease state has not changed significantly within a short time interval. A propagation strength close to 0 indicates that historical features are almost completely attenuated, and the spatial medical features at the current time point dominate the state update. This is suitable for the following situations: modality data is available at the current time point, and new observations can accurately reflect the current state; or the disease state may have changed significantly over a longer time interval, reducing the reference value of historical information.

[0088] Specifically, different modalities may be collected at different times. The system establishes time slices based on medical visits, examination dates, or preset time windows. If a modality is missing in a certain time slice, the system generates a modality missing mask m_t, which can be represented using nearest neighbor time points, population statistical references, or missing embeddings. The model receives both modal features and the missing mask simultaneously during spatiotemporal propagation to avoid misclassifying missing features as normal ones.

[0089] In the above embodiments, by jointly regulating the propagation intensity through the time interval encoding, time edge type, and modality missing mask, the temporal graph convolution can adaptively adjust the contribution of historical information to the current state update according to the actual clinical scenario, avoid treating all adjacent follow-ups as equidistant events, and improve the model's adaptability to irregular clinical follow-up data.

[0090] In some implementations, the modality missing mask participates in both spatial graph convolution and temporal graph convolution. Specifically, in spatial graph convolution, the aggregation weights of different modality evidence nodes or relation edges are adjusted based on the modality missing mask; in temporal graph convolution, the modality missing mask is used to determine whether to propagate historical representations or missing embeddings along the corresponding temporal edges. Thus, using the missing mask simultaneously for spatial neighbor aggregation and temporal edge propagation reduces the risk of misclassifying missing evidence as negative evidence and improves the model's robustness in cases of incomplete image, text, or test data.

[0091] Step 105: Integrate spatial medical features and temporal evolution features to obtain a dynamic representation of the current and historical comprehensive state of the target patient.

[0092] The patient dynamic representation refers to a vector representation that comprehensively characterizes the overall disease progression of a target patient from the first follow-up time point to the current time point by fusing the spatial medical features and temporal evolution features of the target patient at each time point. This vector representation is the final output feature of the temporal-spatial multimodal graph convolutional network and is used as input to the prediction decoder to generate future disease progression prediction results. Specifically, the construction process of the patient dynamic representation is as follows: For each time point of the target patient, after spatial graph convolution in step 103, spatial medical features of each medical entity node that incorporate multimodal medical evidence within the same time point are obtained; after temporal graph convolution in step 104, temporal evolution features of each medical entity node that accumulate historical time point information are obtained. Step 105 fuses the above two types of features.

[0093] The patient dynamics are represented as a fixed-dimensional feature vector.

[0094] For example, the patient dynamic representation has at least one of the following characteristics: It is cumulative. The dynamic representation of the patient includes not only multimodal medical evidence information at the current point in time (i.e., spatial medical features), but also evolutionary information of the same medical entity at historical points in time (i.e., temporal evolution features). Figure 2 For example, the dynamic representation of the patient at time t3 includes not only the spatial medical characteristics of the ROI, CEA test indicators, and disease status at time t3, but also the cumulative lesion change trend (whether the lesions continue to grow), indicator fluctuation patterns (whether CEA continues to rise), and state transition path (the evolution of the disease state from stable to progressive). Therefore, this representation is a numerical representation of the "current and historical comprehensive state".

[0095] It is patient-specific. The patient dynamic representation is constructed for a single target patient. Different patients have different temporal multimodal medical knowledge graphs, and the patient dynamic representations obtained after spatial graph convolution and temporal graph convolution are also different. This representation encodes the patient-specific lesion change patterns, indicator fluctuation patterns, and treatment response characteristics, rather than population statistical patterns.

[0096] It has a fixed dimension. Regardless of the number of time points in the target patient or the number of medical entity nodes at each time point, the dynamic representation of the patient, after mapping by the fully connected layer, is a vector of fixed dimension. This fixed dimension is determined by the input dimension of the predictive decoder, enabling the model to handle patient data with different follow-up lengths and different numbers of modalities, and exhibiting good generalization ability.

[0097] It possesses temporal alignment. The patient dynamic representation is based on the current latest time point, with all historical information accumulating and propagating along the timeline to the current time point. Therefore, this representation is aligned with the current follow-up time on the timeline and can be directly used to predict future disease progression starting from the current time point.

[0098] Through the above fusion method, step 105 encodes the spatial dimension multimodal medical evidence aggregation results and the temporal dimension disease evolution accumulation results into a vector, namely the patient dynamic representation, which provides a comprehensive and compact feature input for subsequent disease progression prediction.

[0099] In some implementations, spatial medicine features and temporal evolution features are integrated to obtain a dynamic representation of the patient's current and historical overall condition, including: The spatial medical features of the target time point are concatenated or weighted and summed with the temporal evolution features corresponding to the time point to obtain the fused representation of the time point; The fused representation is mapped to a fixed-dimensional state vector through a fully connected layer, which serves as the dynamic representation of the target patient.

[0100] Specifically, the above-mentioned fusion of spatial medical features and temporal evolution features yields a dynamic representation of the patient, characterizing the current and historical combined state of the target patient, specifically including: First, the spatial medical features of the target time point are concatenated or weighted and summed with the corresponding temporal evolution features to obtain a fused representation of that time point. The spatial medical features, output by the spatial graph convolution module, represent the spatial dimension features of each medical entity node within that time point after aggregating multimodal medical evidence along medical relationship edges. The temporal evolution features, output by the temporal graph convolution module, represent the temporal dimension features of each medical entity node after accumulating historical time point information along temporal edges. Concatenating or weighting and summing these two features yields a fused representation of that time point, which includes both spatial and temporal dimension information.

[0101] Then, the fused representation is mapped to a fixed-dimensional state vector through a fully connected layer, serving as the dynamic representation of the target patient. Since different target patients have different numbers of time points and medical entity nodes, and the modal types contained at different time points may also differ, directly using the fused representation as the prediction input would lead to inconsistent dimensions. Therefore, mapping the fused representation to a fixed-dimensional state vector through a fully connected layer unifies the representation dimension across different patients and different follow-up lengths, facilitating input processing by the subsequent prediction module.

[0102] Specifically, please refer to Figure 4 , Figure 4This is a schematic diagram illustrating a spatiotemporal fusion scenario in disease progression prediction based on a temporal-spatial multimodal graph convolutional network, as provided in an embodiment of this application. Figure 4 As shown, the temporal-spatial multimodal graph convolutional network model includes a multimodal input encoder, a spatial graph convolutional layer, a temporal graph convolutional layer, a spatiotemporal fusion layer, and a multi-step prediction head. Figure 4 As shown, the multimodal input encoder receives multimodal data at various time points and extracts initial feature representations for each modality through an image encoder, text encoder, MLP (Multi-Level Processing) indicator, and treatment protocol encoder, respectively. These initial feature representations, combined with a modality missing mask, are then input into a spatial graph convolutional layer. In the spatial graph convolutional layer, the model performs spatial propagation along medical relation edges within each time slice to obtain spatial representations. Then, in the temporal graph convolutional layer, the model performs temporal propagation along temporal edges such as visit_next, same_lesion_next, same_indicator_next, and state_transition to obtain temporal representations. Figure 4 As shown, the spatial graph convolutional layer and the temporal graph convolutional layer can be stacked alternately as needed (L layers) to achieve deeper spatial-temporal feature interaction. After the alternating propagation of the L layers is completed, the spatiotemporal fusion layer fuses the final spatial and temporal representations and combines them with time interval encoding to obtain the patient dynamic representation z_p^t. Finally, the multi-step prediction head receives the patient dynamic representation z_p^t and outputs the disease state prediction result for the next h steps.

[0103] Furthermore, such as Figure 4 As shown, the multi-step prediction head is used to output the prediction results of disease progression within one or more future time windows. Its input is the patient dynamic representation z_p^t, and its output is y_p^{t+h} = Decoder(z_p^t, h). Wherein, Decoder represents the prediction decoder, and h represents the number of future steps or time windows. The prediction results include, but are not limited to, the following types: (1) the probability of disease progression within the next h months (such as the next 3 months, 6 months, 12 months); (2) the risk curve within multiple future time windows, with a risk score given for each time window; (3) the sequence of disease states, such as stable, remission, progression, relapse, etc.; (4) the probability of treatment response, used to assess the likelihood of the patient responding to the current treatment plan; and (5) key evidence and key time points, used to assist doctors in understanding the basis of the prediction.

[0104] For example Figure 4As shown, the multi-step prediction head can simultaneously output prediction results for multiple future time windows (such as risk at t+1, t+2, and t+3), enabling the model to provide not only a risk score at a single time point but also the dynamic trend of disease progression over time. This helps doctors understand the possible evolution trajectory of the disease over time; for example, the clinical treatment strategies corresponding to a sharp increase in risk in the short term differ from those corresponding to a slow increase. The training objective of the multi-step prediction head includes a multi-step state sequence loss, used to constrain the predicted state sequence within multiple future time windows to approximate the true sequence.

[0105] Thus, through the above implementation method, spatial medical features and temporal evolution features are uniformly encoded into a fixed-dimensional dynamic representation of the patient in the spatiotemporal fusion layer, and then the multi-step prediction head outputs the disease progression prediction results for multiple future time windows, providing clinicians with comprehensive information on the assessment of disease progression trends.

[0106] Step 106: Predict the future disease progression of the target patient based on the patient's dynamic representation.

[0107] Disease progression refers to the direction and extent of changes in the disease status of a target patient over a future period, starting from the current point in time. Unlike "disease diagnosis," which determines whether a patient currently has a disease or what pathological state they are in—a cross-sectional assessment—disease progression is predictive information looking towards the future, answering the question, "How will the patient's disease evolve in the future?" Specifically, disease progression includes, but is not limited to, disease progression probability or disease status classification results. In practical applications, disease progression includes, but is not limited to, the following: Firstly, the probability of disease progression within a specified future time window. This probability refers to the likelihood of disease progression in the target patient within the specified future time window, expressed as a probability value between 0 and 1. For example, the probability of tumor progression within the next 3 months, the probability of chronic disease worsening within the next 6 months, and the probability of recurrence and metastasis within the next 12 months. This probability value provides clinicians with an intuitive quantitative basis for risk, facilitating risk stratification management.

[0108] Secondly, risk curves across multiple future time windows. These risk curves divide the future timeline into multiple consecutive time windows (e.g., every 3 months), outputting the corresponding progression probability or risk score for each window, forming a trend curve that changes over time. For example, outputting a risk sequence within the next 12 months, with each 3-month window as a separate window (t+3 month risk, t+6 month risk, t+9 month risk, t+12 month risk). The risk curves reflect the dynamic trend of disease progression risk over time—for example, a gradual increase in risk, a rise followed by a decrease, or a sustained stable risk—providing physicians with more precise references for developing follow-up plans and determining intervention timing.

[0109] Third, the future disease state sequence. This disease state sequence refers to the disease state category and its progression path that the target patient may be in at multiple future time points. Disease states include, but are not limited to, stable (no significant change in disease), remission (lesion shrinkage or indicator improvement), progression (lesion enlargement or indicator deterioration), and relapse (recurrence of the disease). For example, the model can predict the state every 3 months within the next 6 months as "stable to stable to progression," meaning the patient progresses after two months of stability. The state sequence helps doctors understand the phased characteristics of disease evolution.

[0110] Fourth, treatment response probability. The treatment response probability refers to the likelihood that a target patient will achieve the expected effect with the current or planned treatment regimen. For example, it predicts whether a patient will respond to the current chemotherapy regimen, whether drug resistance will develop, whether it is suitable to continue the current treatment, or whether a change in regimen is necessary. This output can assist physicians in making individualized choices among multiple available treatment options.

[0111] Fifth, key evidence and key time points. Key evidence refers to medical entities that contribute significantly to the prediction process and their corresponding modal evidence (such as signs of lesion volume increase in imaging at a certain time point, or a trend of continuously rising CEA at a certain time point). Key time points refer to follow-up time points that contribute significantly to the prediction process. For example, the model can output "During t2 to t3, lesion volume increased and CEA continued to rise; both jointly drove the high-risk judgment of this progression prediction," helping doctors understand the evidentiary basis of the prediction results.

[0112] Sixth, the severity level or stage change. The severity level refers to classifying the degree of disease progression into several levels (such as mild, moderate, and severe), and the stage change refers to predicting the possible changes in the disease stage at future points in time (such as progressing from stage I to stage II, or from local progression to distant metastasis). This output can help doctors assess the severity of disease progression.

[0113] The above-mentioned disease progression information can be output individually or in combination. For example, the model can simultaneously output "the probability of progression in the next 3 months is 0.76" and "the key time point is the lesion enlargement + CEA elevation during t2 to t3", providing clinicians with complete predictive information from quantitative risk to evidence attribution.

[0114] It should be noted that the above predictions are probabilistic assessments calculated based on the target patients' historical follow-up data and medical knowledge graph structure. They aim to provide physicians with clinical decision-making references, assisting them in identifying high-risk patients and developing individualized follow-up plans and treatment protocols. The final diagnosis, staging adjustment, or treatment plan change should be determined by the physician based on a comprehensive assessment of the patient's clinical presentation, laboratory tests, and imaging evidence.

[0115] In some implementations, predicting the future disease progression of the target patient based on the patient dynamic representation includes: The patient's dynamic representation is input into a prediction decoder, which outputs a disease progression probability sequence within Q preset time windows, where Q is greater than or equal to 1; wherein, for each time window, the disease progression probability sequence includes at least one of a stable probability, a progression probability, a remission probability, and a relapse probability.

[0116] Specifically, the above-mentioned prediction of the future disease progression of the target patient based on the patient dynamic representation includes: The patient's dynamic representation is input into a predictive decoder, which outputs a sequence of disease progression probabilities over Q preset time windows, where Q is an integer greater than or equal to 1. The preset time windows can be of a fixed length, such as every 3 months or every 6 months, or they can be set according to the natural history of the disease or clinical follow-up practices. For example, cancer patients typically undergo follow-up assessments at 3, 6, and 12 months after treatment, so the preset time windows can be set to 3 months, 6 months, and 12 months.

[0117] For example, when Q=3, the predictive decoder outputs a sequence of disease progression probabilities for three preset time windows (e.g., the next 3 months, the next 6 months, the next 9 months, or the next 3-6 months, 6-9 months, and 9-12 months). For each time window, the predictive decoder outputs a set of disease state probability distributions, which include at least one of the following: stable probability, progression probability, remission probability, and relapse probability. The sum of all probability values ​​is 1. For example, the predicted outcomes for a cancer patient at time t3 might be as follows: within a 3-month time window, the probability of stable disease is approximately 0.15, the probability of progression is approximately 0.76, the probability of remission is approximately 0.06, and the probability of recurrence is approximately 0.03, indicating a high risk of progression within the next 3 months; within a 6-month time window, the probability of stable disease is approximately 0.10, the probability of progression is approximately 0.72, the probability of remission is approximately 0.08, and the probability of recurrence is approximately 0.10, indicating a persistently high risk of progression; within a 9-month time window, the probability of stable disease is approximately 0.12, the probability of progression is approximately 0.58, the probability of remission is approximately 0.15, and the probability of recurrence is approximately 0.15, indicating a decrease in the risk of progression but an increase in the risk of recurrence. Through these probability sequences across multiple time windows, physicians can understand the dynamic trend of disease progression risk over time, providing a quantitative reference for developing individualized follow-up plans and determining the timing of interventions.

[0118] It should be noted that the above probability values ​​are predictive assessment results calculated by the model based on the historical follow-up data of the target patients. They are intended to provide doctors with clinical decision-making references and assist doctors in identifying high-risk patients. The final diagnosis, staging adjustment or treatment plan change should be determined by the doctor based on a comprehensive judgment of the patient's clinical manifestations, laboratory tests and imaging evidence.

[0119] In other implementations, such as Figure 4 As shown and in conjunction with the descriptions of steps 103 to 105 above, the prediction decoder can also work in conjunction with the multi-step prediction head described in step 105. Specifically, the multi-step prediction head receives the patient dynamic representation z_p^t output by the spatiotemporal fusion layer, and outputs the disease state prediction result y_p^{t+h}=Decoder(z_p^t, h) for the next h steps through the decoder, where h represents the number of future steps or the number of time windows. In addition to the probability sequence described above, this prediction result may also include risk curves within multiple future time windows, disease state sequences (e.g., "stable to progressive to relapse"), treatment response probabilities, and key evidence and key time points, such as... Figure 4 The output of the multi-step prediction head is shown below.

[0120] In the above implementation, the predictive decoder directly outputs the disease progression probability sequence within multiple preset time windows in the future. Each time window contains fine-grained state probability distributions such as stability probability, progression probability, remission probability, and relapse probability. On the one hand, the risk curve of multiple time windows depicts the dynamic evolution trend of the disease on a continuous time axis, enabling doctors to distinguish between different clinical patterns such as "short-term rapid deterioration" and "long-term slow progression". On the other hand, the prediction granularity is refined by using multiple state probability distributions instead of a single binary classification result, enabling the model to reflect the possibility of transition between disease states. This provides a quantitative decision-making reference for clinicians to formulate individualized follow-up plans, select intervention timing, and evaluate the risk-benefit ratio under different treatment strategies.

[0121] In some implementations, the method further includes: generating a predictive explanation, including: During the spatial graph convolution process, the first attention weight of each medical relation edge is recorded, and during the temporal graph convolution process, the second attention weight of each temporal edge is recorded. In response to the predicted disease progression, backtracking is performed based on the first attention weight and the second attention weight: The two medical entity nodes connected by the top M weighted medical relationship edges are output as key spatial evidence. The two time points connected by the N time edges with the highest weights are output as the key time periods.

[0122] Specifically, during the spatial graph convolution process in step 103, the model records the first attention weight of each medical relationship edge. The first attention weight refers to the contribution score of the neighboring nodes connected to each medical relationship edge to the feature aggregation of the target node within the same time slice subgraph. In a specific implementation, the attention weight α_{u,r,v}^t from step 103 can be used. Figure 2 As shown, within the t3 time slice subplot, the model records the attention weights between "descriptive" relationship edges connecting image region (ROI) nodes and report text nodes, and the attention weights between "has indicator" relationship edges connecting CEA nodes and disease state nodes. The first attention weight of each medical relationship edge represents the importance of the features of the neighboring nodes connected by that relationship edge during aggregation; that is, the higher the weight, the greater the contribution of the information propagated by that medical relationship edge to the features of the target node.

[0123] During the temporal graph convolution in step 104, the model records the second attention weights for each temporal edge. These second attention weights refer to the propagation gating values ​​or attention scores when propagating messages along each temporal edge, used to measure the contribution of the evolutionary characteristics of the medical entity node at the previous time step to the corresponding node at the current time step. For example... Figure 2As shown, the model records the second attention weights of the same_lesion_next time edge from t1 to t2, the same_indicator_next time edge from t2 to t3, and the state_transition time edge from t1 to t2. The second attention weight of each time edge represents the importance of the historical features propagated by that time edge in updating the features of the current node. In other words, the higher the weight, the greater the contribution of the evolutionary information of that time period to the current prediction.

[0124] Furthermore, in response to the disease progression prediction result output in step 106, the model backtracks based on the first attention weight and the second attention weight. Specifically, the model sorts all the first attention weights recorded during the spatial graph convolution process, extracts the two medical entity nodes connected by the top M medical relationship edges with the highest weights, and outputs them as key spatial evidence, where M is a preset positive integer (e.g., M=3). Figure 3 As shown, if the first attention weight of a certain "descriptive" medical relationship edge ranks first within the t3 time slice sub-image, then the ROI node and report text node connected by that relationship edge are output as key spatial evidence, indicating that the image sign information and text description information carried by these two nodes contribute the most to the prediction result. If the first attention weight of a certain "has indicator" medical relationship edge ranks second, then the CEA node and disease state node are output as key spatial evidence, indicating that this test indicator contributes significantly to the judgment of disease state.

[0125] Then, the model simultaneously sorts all the second attention weights recorded during the temporal graph convolution process, extracts the two time points connected by the top N time edges with the highest weights, and outputs them as the key time periods, where N is a preset positive integer (e.g., N=2). Figure 3 As shown, if the second attention weight of the same_lesion_next time edge from t2 to t3 ranks first among all time edges, then the time points t2 and t3 connected by this time edge are output as the key time period, indicating that the lesion change characteristics during the period from t2 to t3 contribute the most to the prediction result; if the second attention weight of the state_transition time edge from t1 to t2 ranks second, then the time points t1 and t2 connected by this time edge are output as the key time period, indicating that the disease state transition during the period from t1 to t2 also contributes to the prediction.

[0126] For example, the model can output a complete prediction explanation text based on the backtracking results: "The prediction is based on the following factors: the lesion volume increased during t2 to t3 (the ROI of the imaging region is consistent with the description in the report text, with a spatial side contribution of 0.25), the CEA continued to rise (the index side contribution of 0.21), and the disease status did not improve after treatment (the status transition side contribution of 0.12). These three factors together drive the high probability judgment of this progression risk." In the above implementation, by recording the attention weights of each medical relationship edge and each time edge during spatial graph convolution and temporal graph convolution respectively, and by backtracking according to the weight ranking after the prediction results are generated, the medical entity nodes connected by the medical relationship edges with the highest weight ranking are extracted as key spatial evidence, and the time points connected by the time edges with the highest weight ranking are extracted as key time periods. This realizes the automatic conversion from the internal attention distribution of the model to clinically readable interpretation, enabling doctors to quickly locate the specific evidence type, medical entity and time stage on which the model relies when making predictions. This solves the problem that deep learning models are difficult to provide traceable reasoning paths, and enhances the interpretability and clinical credibility of the prediction results.

[0127] In some embodiments, the method further includes: When new follow-up data is added for the target patient, a new time point is added to the end of the temporal multimodal medical knowledge graph corresponding to the target patient, and a multimodal medical knowledge subgraph corresponding to the new time point is constructed based on the new follow-up data. Establish a time edge between the multimodal medical knowledge subgraph corresponding to the newly added time point and the multimodal medical knowledge subgraph corresponding to the previous adjacent time point; Determine the affected update set, which includes the nodes and edges of the multimodal medical knowledge subgraph corresponding to the new time point, the time edge connected to the new time point, the medical entity nodes in the previous adjacent time point that are directly connected to the new time point through the time edge, and the k-hop neighborhood nodes of the medical entity nodes in the new time point on the medical relation edge. The historical cache representation outside the affected update set is reused, and spatial graph convolution and temporal graph convolution are re-performed only on the nodes within the affected update set to obtain the updated patient dynamic representation.

[0128] For example, when new follow-up data is added for the target patient, the method performs a local incremental update step.

[0129] Specifically, please refer to Figure 5 , Figure 5 This application schematic diagram illustrates the incremental update of new follow-up data provided in a specific embodiment of this application, such as... Figure 5 As shown, the target patient already has a temporal multimodal medical knowledge graph with T historical time points (t1 to t2).T Each time point contains medical entity nodes such as images, laboratory indicators, medical record text, and disease status, and adjacent time points are connected by time edges. When a patient completes a new follow-up examination, the system obtains the newly added time point t. T+1 Multimodal medical follow-up data (including new images, new test indicators, new disease state nodes, etc.).

[0130] At the graph structure level, the system adds a new time slice G_p^{T+1} at the end of the target patient's temporal multimodal medical knowledge graph, constructing a new multimodal medical knowledge subgraph containing the new medical entity nodes and their corresponding medical relation edges. Simultaneously, the system establishes temporal edges (such as same_lesion_next, same_indicator_next, state_transition, etc.) between the new time slice G_p^{T+1} and the previous adjacent time slice G_p^T for the same medical entity node, integrating the new time slice into the temporal evolution chain of the patient's temporal graph.

[0131] At the level of indicating updates, such as Figure 5 As shown, the system reuses the cached representation of the target patient's previous T historical time points (such as the spatial medical features and temporal evolution features of each medical entity node). Specifically, the cached representation includes the spatial medical features obtained by spatial graph convolution in step 103, the temporal evolution features obtained by temporal graph convolution in step 104, and the patient dynamic representation obtained in step 105. When a new time slice is added, the system only re-executes spatial graph convolution, temporal graph convolution, and spatiotemporal fusion on the new time slice and its affected k-hop neighborhood nodes. For nodes outside the affected update set, the system directly reuses the historical cached representation without recalculation. Figure 4 As shown, the dashed box marks the scope of "only updating newly added slices and k-hop neighborhoods", indicating that the computational cost is much less than a full recalculation.

[0132] Specifically, the affected update set A_update includes the following nodes and edges: (1) all nodes and edges within the newly added time slice G_p^{T+1}; (2) all newly established time edges between the newly added time slice and the previous adjacent time point G_p^T; (3) medical entity nodes in the previous adjacent time point G_p^T that are directly connected to the newly added time slice through time edges; (4) k-hop neighbor nodes of the medical entity nodes in the newly added time slice in the medical relation space (k is a preset positive integer, such as k=1 or k=2), that is, the set of neighbor nodes connected to the newly added node through no more than k medical relation edges in the spatial medical relation. For all nodes within the above A_update, the system re-executes the spatial graph convolution of step 103, the temporal graph convolution of step 104, and the spatiotemporal fusion of step 105 based on the historical cache representation to obtain the updated feature representation of each node and the dynamic representation of the patient. For nodes in the historical time points before the newly added time slice that do not belong to A_update, their cache representation remains unchanged, and the cache results of the previous T time points are directly reused.

[0133] For example, taking a chronic disease patient with 40 historical time points as an example, if each time point contains an average of about 20 medical entity nodes, then the spatial representation update of all historical nodes requires the computation of about 800 nodes. When a new follow-up time point is added, the affected update set A_update only includes about 20 nodes in the new time slice, about 3-5 nodes directly connected by time edges in the previous adjacent time point, and the k-hop neighborhood of these nodes in the medical relation space (about 30-40 more nodes when k=2), totaling about 55-65 nodes that need to be recalculated, which is less than one-tenth of the computation of all historical nodes, and all historical cache representations can be reused.

[0134] After completing the aforementioned partial update, the system re-executes step 106 based on the updated patient dynamic representation, outputting new future disease progression prediction results through the prediction decoder, such as... Figure 5 As shown, the "Output New Future Progress Prediction" module receives the updated representation and outputs multi-step prediction results such as t+1 risk, t+2 risk, and t+3 risk.

[0135] like Figure 5As shown, upon the arrival of new follow-up data, the system does not need to recalculate the entire patient atlas; it only needs to append new time slices to the end of the time-series atlas and update the affected local node set. This significantly reduces the computational latency after new follow-up data, enabling the model to support the rapid response requirements in continuous clinical monitoring scenarios. Furthermore, since the cached representation is an intermediate result calculated based on model parameters and historical data, reusing the cached representation is mathematically equivalent to a full recalculation. That is, for nodes unaffected by the new nodes, the neighbor set of their spatial graph convolution and the historical dependencies of their temporal graph convolution remain unchanged. Therefore, directly reusing the cached representation does not introduce any errors, and the updated prediction results are consistent with the results of a full recalculation.

[0136] In the above embodiments, the newly added follow-up data local update mechanism can incorporate new follow-up data into the model with reduced computational cost, update patient status in real time, and output the latest progression prediction, providing technical support for continuous clinical monitoring and dynamic assessment.

[0137] In some implementations, when new patient follow-up data is added, a new time slice is added to the end of the patient-level temporal multimodal medical knowledge graph. An affected update set is determined, including the new time slice, adjacent time edges, consecutive nodes of the same lesion or the same indicator, and their k-hop neighborhoods. Historical cache representations outside of this affected update set are reused, and representation updates are performed only on the affected update set. Thus, through time slice appending, affected set definition, and historical cache reuse, new follow-up data can be locally updated and quickly used for online progression prediction, reducing the cost of full graph recalculation and full retraining.

[0138] In summary, this application's embodiments, by constructing a knowledge subgraph containing medical relationship edges for each time point and establishing temporal edges across time points, enable spatial graph convolution and temporal graph convolution to synchronously propagate features along medical relationship edges and temporal edges respectively within a unified graph structure. This achieves joint modeling of medical structural relationships and temporal evolution information in multi-time point follow-up data of patients, overcoming the dual shortcomings of time series models' difficulty in utilizing medical relationship structures and static graph neural networks' difficulty in modeling temporal evolution. By fusing spatial medical features and temporal evolution features into a dynamic patient representation characterizing the patient's current and historical comprehensive state, this representation simultaneously encodes the aggregation results of multimodal medical evidence within the same time point and the trend of state changes between different time points, thereby more comprehensively reflecting the disease progression and improving prediction accuracy. Through mechanisms such as adding new time slices, defining the affected update set, and reusing historical caches, when new follow-up data arrives, only the affected node set needs to be locally updated without requiring a full update. The recalculation decouples computational complexity from the number of historical time points, significantly reducing computational latency and deployment costs in continuous monitoring scenarios (verification experiments show an acceleration ratio of 10x to 69x with update results consistent bit-by-bit with full recalculation). By recording attention weights in spatial and temporal graph convolutions and backtracking after prediction, key spatial evidence and key time periods can be output, providing doctors with traceable explanatory evidence and overcoming the limitation of deep learning model prediction results being difficult to verify. Through time interval encoding and propagation gating mechanisms, the propagation intensity of historical information is adaptively adjusted according to the actual time interval between adjacent follow-ups. By participating in the aggregation and propagation process of spatial and temporal graph convolutions through modality missing masks, the model can distinguish between "evidence missing" and "evidence negative," maintaining stable prediction performance even in irregular follow-up and modality missing clinical scenarios. This effectively improves the prediction accuracy, deployment efficiency, and clinical credibility of medical AI systems in longitudinal follow-up scenarios.

[0139] Based on the same inventive concept as the foregoing embodiments, the foregoing embodiments will be described in detail below through a specific example.

[0140] The purpose of this application is to provide a method and system for predicting disease progression based on a temporal-spatial multimodal graph convolutional network. This scheme upgrades a static multimodal medical knowledge graph into a patient-level temporal graph, attaching images, tests, text, medications, and status from each patient follow-up visit as time-series nodes or time-slice subgraphs to the graph. A temporal-spatial multimodal graph convolutional module is designed to propagate spatial features along medical relationships within the same time slice and propagate status change features along follow-up time edges between different time slices.

[0141] Through the above design, the embodiments of this application aim to achieve the following objectives: The patient follow-up sequence, medical relationship structure, and multimodal evidence are unified into a temporal multimodal medical knowledge graph.

[0142] Simultaneously, we study spatial medical dependence within the same time point and disease evolution dependence between different time points.

[0143] Supports irregular follow-up intervals and missing modalities.

[0144] Output multi-step disease progression risk, progression time window, and key evidence pathways.

[0145] It supports local map updates and subsequent incremental training for newly added follow-up data.

[0146] Specifically, please combine Figure 2 This application proposes a temporal-spatial multimodal graph convolutional framework for disease progression prediction. The system first receives multimodal follow-up data from multiple time points of the patient, including medical images, image reports, structured laboratory indicators, vital signs, medical record text, treatment plans, and disease status labels. The system constructs a temporal slice subgraph for each time point, containing image evidence nodes, text evidence nodes, laboratory indicator nodes, symptom nodes, treatment nodes, disease status nodes, and medical entity nodes for that time point, connected by medical relation edges. Between different time points, the system connects consecutive nodes of the same patient, the same lesion, the same indicator, or the same disease status through temporal edges.

[0147] The temporal-spatial multimodal graph convolutional network receives the aforementioned temporal graph as input. The model first obtains feature representations of image, text, examination, and treatment information through a modal encoder; then, it performs spatial graph convolution or relational graph attention propagation within each time slice to aggregate medical structural information at the same time point; next, it performs temporal graph convolution, gated recurrent propagation, or temporal attention propagation along the temporal edges to learn lesion changes, indicator changes, and state transitions. Finally, the model outputs the probability of disease progression, severity level, recurrence risk, or treatment response prediction within one or more future time windows.

[0148] Specifically, the system of the present invention may include the following modules.

[0149] The multi-time point data receiving module is used to receive images, reports, test indicators, medical records, treatment plans, and disease status of patients at multiple follow-up time points.

[0150] The time alignment module is used to align different modal data to time slices based on examination time, consultation time, and follow-up time.

[0151] The temporal multimodal knowledge graph construction module is used to construct a medical knowledge subgraph for each time point and establish time edges across time points.

[0152] The multimodal coding module is used to encode image regions, text fragments, test indicators, treatment plans, and status labels respectively.

[0153] The spatial graph convolution module is used to aggregate features along medical relation edges within the same time slice.

[0154] The temporal graph convolution module is used to propagate the evolutionary features of lesions, indicators, and disease states along the follow-up time.

[0155] The spatiotemporal fusion module is used to fuse spatial propagation results and temporal propagation results to form a dynamic representation of the patient.

[0156] The progression prediction module is used to output the probability of future disease progression, severity level, risk of relapse, treatment response, or multi-step state sequence.

[0157] The Explanation and Evidence Backtracking module is used to output time points, modal evidence, medical entities, and reasoning paths that contribute significantly to the prediction results.

[0158] The follow-up incremental update module is used to update the patient time series diagram and local node representation after new follow-up data arrives.

[0159] Specifically, please refer to Figure 6 , Figure 6 A flowchart illustrating the disease progression prediction method based on temporal-spatial multimodal graph convolutional networks provided in a specific embodiment of this application is shown below. Figure 6 As shown, the method includes: S100: Receives patient follow-up data at multiple time points. The system receives medical images, image reports, medical record texts, consultation audio, test results, vital signs, treatment plans and disease status of patients at multiple time points.

[0160] S200: Execution of time alignment and modal coding The system performs time alignment of multimodal data according to medical visit events or time windows, and encodes images, text, laboratory indicators, and treatment plans separately. For missing modalities, the system generates a missing mask.

[0161] S300: Constructing a Temporal Multimodal Medical Knowledge Graph The system constructs a multimodal medical knowledge subgraph for each time point and establishes time edges between different time points for the same lesion, the same indicator, the same treatment plan, and the same disease state.

[0162] S400: Perform spatial graph convolution The model propagates features along the medical relationship edges within each time slice to obtain a spatial medical representation of the same time point.

[0163] S500: Execution-Time Graph Convolution The model propagates characteristics along the time horizon, learning changes in lesions, indicators, treatment responses, and state transitions.

[0164] S600: Perform spatiotemporal fusion The system integrates spatial representation, temporal representation, time interval encoding, and modality missing mask to obtain a dynamic representation of the patient.

[0165] S700: Output Disease Progression Prediction The progression prediction module outputs the probability of progression, severity, risk of relapse, treatment response, or state sequence for one or more future time windows.

[0166] S800: Generate explanation and audit logs Based on attention weights, path contributions, and time contributions, the system outputs key time points, key evidence, key indicator changes, and key inference paths, and records model versions, graph versions, and data time windows.

[0167] It should be added that the model training data includes patient historical follow-up sequences, multimodal evidence at specific time points, medical knowledge graph relationships, and future progression labels. Training objectives may include: Progression prediction loss is used to constrain future progression, relapse, deterioration, or treatment response predictions to be close to the true label.

[0168] Multi-step state sequence loss is used to predict state sequences within multiple future time windows.

[0169] Graph relation consistency loss is used to constrain spatial propagation results to conform to the medical relation schema.

[0170] Temporal consistency loss is used to constrain changes in disease state to conform to temporal order and follow-up intervals.

[0171] Modal alignment loss is used to align images, text, and test indicator evidence at the same time point.

[0172] Missing modality robustness loss is used to train the model to still output risk predictions when images, tests, or text are missing.

[0173] In terms of training strategy, the multimodal encoder and spatial graph convolutional module can be trained first, and then the temporal graph convolutional module can be added for end-to-end training. If the amount of data is limited, the image encoder and text encoder can be frozen, and only the graph convolutional layer, temporal propagation layer and prediction head can be trained.

[0174] Taking a tumor patient with three follow-up time points as an example, at t1, the imaging showed a lesion diameter of 10mm and normal CEA; at t2, the lesion diameter was 12mm and CEA was slightly elevated; at t3, the lesion diameter was 16mm, CEA continued to rise, and the report text indicated "increased compared to before". The system constructs three time slice sub-graphs of t1, t2, and t3, and connects the same lesion node with same_lesion_next, connects the CEA index node with same_indicator_next, and connects the disease status node with state_transition.

[0175] Spatial graph convolution aggregates the relationships between "lesion-anatomical location-imaging features-candidate status" and "CEA-abnormal indicators-candidate status" at each time point; temporal graph convolution learns the changing trends along the temporal edges of lesion and CEA. The final model outputs a disease progression risk of 0.76 in the next 3 months, and provides a key explanation: "The lesion enlargement was relatively high between t2 and t3, CEA continued to rise, and the consistency between the report text and imaging changes supports the increased risk of progression." For example, for tumor progression prediction, the system receives CT images, imaging reports, tumor markers, treatment plans, and disease status from multiple follow-up time points. A knowledge subgraph is constructed for each time point, containing the lesion region, imaging features, anatomical location, tumor markers, and treatment nodes. The same lesion and the same indicator are connected between different time points. The model learns the relationship between lesions, imaging features, abnormal indicators, and treatment through spatial graph convolution, and learns the trends in lesion size and indicator changes through temporal graph convolution, outputting the progression risk for the next 3 or 6 months.

[0176] For example, for predicting the deterioration of chronic diseases, the system receives long-term laboratory indicators, medication records, symptom records, and outpatient follow-up texts from patients with chronic diseases. Indicator nodes at each time point are connected to disease status nodes, and the same indicator at different time points is connected via time edges. The model learns the long-term trends of indicators, changes in medication use, and changes in symptoms to predict the risk of future deterioration.

[0177] For example, for incremental updates of new follow-up data, when a patient completes a new imaging and laboratory examination, the system adds a time slice at the end of the patient's time series graph. The model reuses the cached representation of historical time points, calculating only the newly added slice and its k-hop neighborhood. If the new follow-up data is confirmed by the physician and carries a true progression label, the sample enters the incremental training sample pool for subsequent periodic fine-tuning.

[0178] Therefore, compared with the prior art, the embodiments of this application have at least the following beneficial effects: Simultaneously, it utilizes medical spatial relationships and temporal evolution relationships. The system not only considers the medical entity associations within a single visit, but also the state changes between multiple follow-up time points.

[0179] It supports multimodal longitudinal data fusion. The system can uniformly process images, text, test indicators, treatment plans, and disease states, and explicitly model modality gaps.

[0180] Improve the traceability of disease progression prediction. The system can output key time points, key lesion changes, key test indicators, and key reasoning paths, making it easier for doctors to review.

[0181] Supports low-cost access to new follow-up data. The system can append new time slices to the end of the patient's time series graph and locally update the node representation, avoiding full recalculation each time.

[0182] This approach is applicable to various disease progression tasks. It can be used in scenarios such as tumor progression, chronic disease exacerbation, relapse risk, treatment response, and rehabilitation risk stratification.

[0183] It should be noted that spatial graph convolution can employ GCN, GAT, R-GCN, graph transformer, or knowledge graph embedding models. Temporal graph convolution can employ TCN, GRU, LSTM, temporal attention, temporal graph network, or continuous temporal neural network. Multimodal encoders can employ medical imaging CNN, vision transformer, medical text transformer, structured index MLP, or treatment protocol encoder. Time slices can be constructed based on visit events, fixed time windows, examination dates, or treatment cycles. Progression prediction outputs can be binary progression risk, multi-class status, risk curves, survival time, or treatment response probability. Incremental updates can be performed at the patient level, disease level, lesion level, or local subgraph level.

[0184] In some implementations, based on the disease progression prediction method of the above-mentioned temporal-spatial multimodal graph convolutional network, this application also provides a complete closed-loop implementation method that includes model training, online inference, local hot updates, and periodic incremental training.

[0185] Specifically, please refer to Figure 7 ,like Figure 7 As shown, the closed-loop system includes a historical patient follow-up sample library, a model training module, an online inference module, a local hot update module, and an incremental training sample pool.

[0186] I. Model Training Phase like Figure 7As shown, the system first obtains a large amount of follow-up data from a historical patient follow-up sample database, including time-series images, laboratory indicators, medical records, and treatment records at multiple time points, as well as corresponding real progression labels. The historical patients refer to the group of patients who have completed follow-up or have obtained a clear outcome (such as progression / non-progression, relapse / non-relapse, survival / death, etc.), and their data are used for model training and parameter optimization.

[0187] like Figure 7 As shown, the model training module receives the historical patient's temporal multimodal data, organizes it into a temporal multimodal medical knowledge graph through the training set construction module, and inputs it into the temporal-spatial multimodal GCN model. The temporal-spatial multimodal GCN model includes a multimodal input encoder, a spatial graph convolutional layer, a temporal graph convolutional layer, a spatiotemporal fusion layer, and a multi-step progression prediction head. During training, the model calculates the loss between the predicted results and the actual progression labels, and optimizes the model parameters through backpropagation. The loss function may include at least one of the following: (1) Progression prediction loss, used to constrain the difference between the predicted disease progression and the real label; (2) Multi-step state sequence loss, used to constrain the predicted state sequence within multiple future time windows to be close to the real sequence; (3) Graph relation consistency loss, used to constrain the propagation result of spatial graph convolution to conform to the preset medical relation pattern; (4) Temporal consistency loss, used to constrain the temporal evolution of disease state to conform to the temporal order and follow-up interval; (5) Modality alignment loss, used to constrain the medical entity node features of different modalities to align with each other within the same time point; (6) Missing modality robustness loss, used to perform random mask simulation on images, text or test indicators during training to constrain the model to maintain stable prediction performance in the case of missing modalities.

[0188] like Figure 7 As shown, after training, the system uses a validation set to evaluate model performance. Evaluation metrics include AUC (area under the curve), C-index (consistency index), risk stratification effect, calibration, and multi-time-window prediction performance. Based on the evaluation results, model selection and parameter tuning are performed, and finally, the trained model is deployed for online inference.

[0189] II. Online Reasoning Stage like Figure 7 As shown, during the online inference phase, the system receives temporal multimodal data (including temporal images, test indicators, medical record text, and treatment records) from the target patient and inputs it into the pre-trained temporal-spatial multimodal GCN model. The data is processed through a multimodal input encoder, spatial graph convolutional layer, temporal graph convolutional layer, and spatiotemporal fusion layer. Finally, the multi-step progression prediction head outputs real-time risk prediction results and disease progression information for clinicians to view and use as a reference for decision-making.

[0190] like Figure 7 As shown, during the online inference process, while generating prediction results, the model can also output key evidence paths through the attention backtracking mechanism to explain the time points, modal types, and medical relationship edges on which the prediction depends.

[0191] III. New Follow-up Data Access and Partial Hot Updates like Figure 7 As shown, when new follow-up data (new images, new laboratory indicators, new medical record text) is added for the target patient, the system does not trigger a full retraining; instead, it first enters the local hot update module. This local hot update module reuses the cached historical time point representation, recalculating only the newly added time slice and its k-hop neighborhood nodes (see the aforementioned embodiments for the specific local update mechanism). Figure 4 (As described), once updated, it can be used for instant prediction without waiting for the model parameters to be retrained. For example... Figure 5 As shown, after a local hot update, the system can quickly output the updated prediction results, meeting the low latency requirements of continuous clinical monitoring scenarios.

[0192] IV. Periodic Incremental Training like Figure 7 As shown, once new follow-up data is confirmed by a doctor and labeled with actual progression, the sample enters the incremental training sample pool. The system waits to accumulate a certain number of labeled new samples before performing periodic incremental training—using the new data in the incremental training sample pool to fine-tune and update the model parameters. For example... Figure 7 As shown, the samples in the incremental training sample pool are "included in the sample pool after obtaining the real labels, and used for subsequent model parameter updates", forming a complete closed loop from "new data to local hot update to instant prediction to waiting for real labels to inclusion in the incremental sample pool to periodic incremental training to updating model parameters".

[0193] In terms of training strategy, the multimodal encoder and spatial graph convolutional module can be trained first, followed by the addition of a temporal graph convolutional module for end-to-end training. If the amount of data is limited, the image encoder and text encoder can be frozen, and only the graph convolutional layer, temporal propagation layer, and prediction head can be trained to reduce the risk of overfitting. During the incremental training phase, a lower learning rate can be used to fine-tune the model to avoid catastrophic forgetting problems.

[0194] Thus, through the above... Figure 7 The training, inference, and incremental update closed loop shown in this application not only enables real-time prediction of follow-up data of new patients after model deployment, but also allows for periodic updates of model parameters in a low-cost manner after a certain scale of new data has been accumulated. This enables the model performance to continuously improve with the accumulation of clinical data, while avoiding the high computational cost of full retraining after each new data addition. This achieves sustainable evolution of the model after deployment and continuous clinical monitoring.

[0195] Figure 8 A schematic diagram of the composition structure of the disease progression prediction system based on temporal-spatial multimodal graph convolutional networks provided in this application embodiment is shown below. Figure 8 As shown, the system includes: Module 81 is used to acquire multimodal medical follow-up data of the target patient at multiple time points; Module 82 is used to construct a time-series multimodal medical knowledge graph corresponding to the target patient based on multimodal medical follow-up data. Each time point corresponds to a multimodal medical knowledge subgraph, and each multimodal medical knowledge subgraph includes multiple medical entity nodes and medical relationship edges connecting the medical entity nodes. Different time points are connected through time edges connecting the medical entity nodes. Spatial graph convolution module 83 is used to perform spatial graph convolution on the multimodal medical knowledge subgraph at each time point to aggregate the spatial medical features of medical entity nodes along the medical relation edges; The temporal graph convolution module 84 is used to perform temporal graph convolution on the temporal edges between different time points to learn the temporal evolution characteristics of medical entity nodes between different time points; The fusion module 85 is used to fuse spatial medical features and temporal evolution features to obtain a dynamic representation of the current and historical comprehensive state of the target patient. Prediction module 86 is used to predict the future disease progression of a target patient based on the patient's dynamic representation.

[0196] Based on the same inventive concept as the foregoing embodiments, this application provides a computing device, such as... Figure 9 As shown, the device includes: a processor 310 and a memory 311 storing a computer program; wherein, Figure 9 The processor 310 shown in the diagram does not indicate that there is only one processor 310, but only indicates the positional relationship of the processor 310 relative to other devices. In practical applications, there can be one or more processors 310; similarly, Figure 9 The memory 311 illustrated in the diagram has the same meaning, that is, it is only used to indicate the positional relationship of memory 311 relative to other devices. In practical applications, the number of memories 311 can be one or more. When the processor 310 runs the computer program, the medical interpretation processing method applied to the above-mentioned device is implemented. The device may also include: at least one network interface 312. The various components in the device are coupled together through a bus system 313. It is understood that the bus system 313 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 313 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in Figure 9 The general designated all buses as Bus System 313.

[0197] The memory 311 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 311 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0198] The memory 311 in this embodiment is used to store various types of data to support the operation of the device. Examples of this data include: any computer programs used to operate on the device, such as operating systems and applications; contact data; phonebook data; messages; pictures; videos, etc. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications, such as media players, browsers, etc., used to implement various application services. Here, the program implementing the method of this embodiment can be included in the application.

[0199] Based on the same inventive concept as the foregoing embodiments, this embodiment also provides a computer-readable storage medium storing a computer program. The computer-readable storage medium can be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc. When the computer program stored in the computer-readable storage medium is executed by a processor, it implements the above-described disease progression prediction method based on a temporal-spatial multimodal graph convolutional network. For the specific steps implemented when the computer program is executed by the processor, please refer to [link to relevant documentation]. Figure 1 or Figure 6 The description of the illustrated embodiments will not be repeated here.

[0200] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0201] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0202] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for predicting disease progression based on temporal-spatial multimodal graph convolutional networks, characterized in that, Includes the following steps: Acquire multimodal medical follow-up data of target patients at multiple time points; Based on the multimodal medical follow-up data, a temporal multimodal medical knowledge graph corresponding to the target patient is constructed. Each time point corresponds to a multimodal medical knowledge subgraph, and each multimodal medical knowledge subgraph contains multiple medical entity nodes and medical relationship edges connecting the medical entity nodes. Different time points are connected through the time edges connecting the medical entity nodes. Spatial graph convolution is performed on the multimodal medical knowledge subgraph at each time point to aggregate the spatial medical features of the medical entity nodes along the medical relation edges; Temporal graph convolution is performed on the temporal edges between different time points to learn the temporal evolution characteristics of the medical entity node between different time points; wherein, the replicas of the same medical entity node at different time points are connected by temporal edges, and the temporal edges include at least one of the following: edge connecting the same lesion, edge connecting the same detection index, edge connecting the disease state transition, and edge connecting the order of medical visits; By integrating the spatial medicine features and the temporal evolution features, a dynamic representation of the patient, characterizing the current and historical combined state of the target patient, is obtained. Predict the future disease progression of the target patient based on the patient dynamic representation; Obtain the modality missing mask at each time point. Each bit of the modality missing mask corresponds to a modality type and is used to identify whether data of the corresponding modality was collected at the time point. When performing the spatial graph convolution, the modality missing mask is used to participate in the feature aggregation process of neighboring nodes to adjust the aggregation weights according to the missing status of each modality type at each time point. When performing the temporal graph convolution, the modality missing mask is used to participate in the feature propagation process along the temporal edge to control the propagation gate value of historical features along the temporal edge according to the modality missing situation at the current time point. The propagation gate value is used to adjust the intensity of the evolution features of the medical entity node at the previous time point being transmitted to the corresponding node at the current time point along the temporal edge.

2. The method according to claim 1, characterized in that, The step of performing spatial graph convolution on the multimodal medical knowledge subgraph at each time point to aggregate the spatial medical features of the medical entity nodes along the medical relation edges includes: Obtain the type identifier of each medical relationship edge within the current time point; Based on the type identifier, different relationship transformation matrices are assigned to different medical relationship types; For each medical entity node at the current time point, the features of neighboring nodes connected to the medical entity node through various medical relationship edges are aggregated. The aggregation results of different relationship types are added together after being mapped by the corresponding relationship transformation matrix to obtain the spatial medical features of the medical entity node.

3. The method according to claim 1, characterized in that, Performing temporal graph convolution on the temporal edges between different time points to learn the temporal evolution features of the medical entity nodes between different time points includes: Get the actual calendar time interval between two adjacent time points; A time interval code is generated based on the actual calendar time interval, and the time interval code is used to characterize the duration information between adjacent follow-ups; The time interval code and the type identifier of the current time edge are input into the gating function to calculate the propagation gating value on the time edge; Along the time edge, the evolutionary characteristics of the medical entity node at the previous time point are multiplied by the propagation gating value and then passed to the corresponding medical entity node at the time point to obtain the temporal evolution characteristics of the medical entity node between different time points.

4. The method according to claim 1, characterized in that, The fusion of the spatial medical features and the temporal evolution features yields a dynamic representation of the patient, characterizing the current and historical combined state of the target patient, including: The spatial medical features of the target time point are concatenated or weighted and summed with the temporal evolution features corresponding to the time point to obtain the fused representation of the time point; The fused representation is mapped to a fixed-dimensional state vector through a fully connected layer, which serves as the dynamic representation of the target patient.

5. The method according to claim 1, characterized in that, The prediction of future disease progression of the target patient based on the patient dynamic representation includes: The patient's dynamic representation is input into a prediction decoder, which outputs a disease progression probability sequence within Q preset time windows, where Q is greater than or equal to 1; wherein, for each time window, the disease progression probability sequence includes at least one of a stable probability, a progression probability, a remission probability, and a relapse probability.

6. The method according to claim 1, characterized in that, The method further includes: generating predictive explanations, including: During the spatial graph convolution process, the first attention weight of each medical relation edge is recorded, and during the temporal graph convolution process, the second attention weight of each temporal edge is recorded. In response to the predicted disease progression, backtracking is performed based on the first attention weight and the second attention weight: The two medical entity nodes connected by the top M weighted medical relationship edges are output as key spatial evidence. The two time points connected by the N time edges with the highest weights are output as the key time periods.

7. The method according to claim 1, characterized in that, The method further includes: When new follow-up data is added for the target patient, a new time point is added to the end of the temporal multimodal medical knowledge graph corresponding to the target patient, and a multimodal medical knowledge subgraph corresponding to the new time point is constructed based on the new follow-up data. Establish a time edge between the multimodal medical knowledge subgraph corresponding to the newly added time point and the multimodal medical knowledge subgraph corresponding to the previous adjacent time point; Determine the affected update set, which includes the nodes and edges of the multimodal medical knowledge subgraph corresponding to the new time point, the time edge connected to the new time point, the medical entity nodes in the previous adjacent time point that are directly connected to the new time point through the time edge, and the k-hop neighborhood nodes of the medical entity nodes in the new time point on the medical relation edge. The historical cache representation outside the affected update set is reused, and spatial graph convolution and temporal graph convolution are re-performed only on the nodes within the affected update set to obtain the updated patient dynamic representation.

8. A disease progression prediction system based on temporal-spatial multimodal graph convolutional networks, characterized in that, include: The acquisition module is used to acquire multimodal medical follow-up data of the target patient at multiple time points; The construction module is used to construct a time-series multimodal medical knowledge graph corresponding to the target patient based on the multimodal medical follow-up data. Each time point corresponds to a multimodal medical knowledge subgraph, and each multimodal medical knowledge subgraph includes multiple medical entity nodes and medical relationship edges connecting the medical entity nodes. Different time points are connected through the time edges connecting the medical entity nodes. A spatial graph convolution module is used to perform spatial graph convolution on the multimodal medical knowledge subgraph at each time point to aggregate the spatial medical features of the medical entity nodes along the medical relation edges; The temporal graph convolution module is used to perform temporal graph convolution on the temporal edges between different time points to learn the temporal evolution characteristics of the medical entity node between different time points; wherein, the copies of the same medical entity node at different time points are connected by temporal edges, and the temporal edges include at least one of the following: edge connecting the same lesion, edge connecting the same detection index, edge connecting the disease state transition, and edge connecting the order of medical visits. The fusion module is used to fuse the spatial medical features and the temporal evolution features to obtain a dynamic representation of the patient's current and historical overall state. The prediction module is used to predict the future disease progression of the target patient based on the patient's dynamic representation. The system is also configured to obtain modality missing masks at each time point, wherein each bit of the modality missing mask corresponds to a modality type and is used to identify whether data of the corresponding modality has been collected at the time point. When performing the spatial graph convolution, the modality missing mask is used to participate in the feature aggregation process of neighboring nodes to adjust the aggregation weights according to the missing status of each modality type at each time point. When performing the temporal graph convolution, the modality missing mask is used to participate in the feature propagation process along the temporal edge to control the propagation gate value of historical features along the temporal edge according to the modality missing situation at the current time point. The propagation gate value is used to adjust the intensity of the evolution features of the medical entity node at the previous time point being transmitted to the corresponding node at the current time point along the temporal edge.

9. A computing device, characterized in that, The device includes a processor and a memory storing a computer program, wherein when the processor runs the computer program, it implements the disease progression prediction method based on a temporal-spatial multimodal graph convolutional network as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the disease progression prediction method based on a temporal-spatial multimodal graph convolutional network as described in any one of claims 1 to 7.

Images