Cerebral stroke risk and prognosis-based prediction system and method

Abstract

The invention discloses a cerebral apoplexy risk and prognosis prediction system and method, and relates to the field of intelligent medical treatment, and the system comprises a data processing and knowledge construction layer which is used for extracting, cleaning and constructing a space-time multi-modal knowledge graph and structured clinical features from multi-source heterogeneous medical data; the feature engineering and fusion layer is used for deeply fusing dynamic semantic information in the space-time multi-modal knowledge graph and the structured clinical features through a graph embedding and attention mechanism to generate a fusion feature vector for a cerebral apoplexy prediction task; and the prediction model and output layer is used for performing cerebral apoplexy risk and prognosis prediction based on the fusion feature vector to obtain a prediction result, and generating a decision result for assisting a doctor in understanding the model through an interpretable mechanism. The method provided by the invention can improve the accuracy of stroke recurrence, bleeding transformation or function prognosis prediction, and provides a new way for accurate stroke management.

Description

Technical Field

[0001] This specification relates to the field of smart healthcare, and more specifically, this application relates to a system and method for predicting stroke risk and prognosis. Background Technology

[0002] Stroke is one of the leading causes of death and disability worldwide. Its etiology is complex, diagnostic procedures are diverse, and treatment options are constantly evolving. Management relies heavily on the integration of information from multiple sources of medical data, including literature, guidelines, electronic health records (EHRs), and imaging reports. To effectively organize and utilize this information, knowledge graphs have become a key tool. They assist clinical decision-making and research analysis by structurally representing stroke-related entities (such as subtypes, symptoms, risk factors, drugs, treatments, and genes) and their relationships (such as "treatment," "cause," and "diagnostic basis").

[0003] However, the construction and maintenance of current stroke knowledge graphs still face several challenges: traditional manual construction is inefficient and difficult to scale; existing NLP (Natural Language Processing) / machine learning methods rely on a large amount of labeled data, making it difficult to accurately extract complex relationships in medical language; the graph content is updated slowly and lacks timeliness, and it is still difficult to extract knowledge uniformly from multi-source heterogeneous data; in addition, there are bottlenecks in the automatic verification of extraction results.

[0004] To address the aforementioned issues, there is an urgent need for a more efficient, scalable system and method for predicting stroke risk and prognosis with dynamic update capabilities. Summary of the Invention

[0005] The summary section introduces a series of simplified concepts, which will be further explained in detail in the detailed description section. This summary section is not intended to limit the key and essential technical features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0006] In a first aspect, the present invention proposes a stroke risk and prognosis prediction system, comprising: The data processing and knowledge construction layer is used to extract, clean, and construct spatiotemporal multimodal knowledge graphs and structured clinical features from multi-source heterogeneous medical data. The feature engineering and fusion layer is used to deeply fuse the dynamic semantic information in the spatiotemporal multimodal knowledge graph and the structured clinical features mentioned above through graph embedding and attention mechanisms, generating a fusion feature vector for stroke prediction tasks. The prediction model and output layer are used to predict stroke risk and prognosis based on the above-mentioned fused feature vectors, obtain prediction results, and generate decision results that help doctors understand the model through an interpretable mechanism.

[0007] In one feasible implementation, the aforementioned data processing and knowledge construction layer includes a data access and preprocessing module, a spatiotemporal multimodal knowledge graph construction module, and a clinical feature extraction module. The aforementioned data access and preprocessing module is responsible for extracting and cleaning data from multi-source heterogeneous medical data to obtain preprocessed data. The aforementioned spatiotemporal multimodal knowledge graph construction module constructs the aforementioned spatiotemporal multimodal knowledge graph based on the aforementioned preprocessed data. The aforementioned spatiotemporal multimodal knowledge graph includes entities, relationships, temporal information, and multimodal features. The aforementioned clinical feature extraction module extracts key clinical indicators for stroke risk and prognosis modeling from the structured clinical data in the preprocessed data, and completes the corresponding feature engineering to obtain the aforementioned structured clinical features.

[0008] In one feasible implementation, the aforementioned feature engineering and fusion layer includes a spatiotemporal multimodal knowledge graph embedding module and an attention mechanism-driven multi-feature fusion module. The aforementioned spatiotemporal multimodal knowledge graph embedding module is used to perform time-aware and multimodal feature fusion graph embedding encoding operations on the aforementioned spatiotemporal multimodal knowledge graph to generate graph embedding features; The attention mechanism-driven multi-feature fusion module is used to learn the interaction relationship between the graph embedding features and the structured clinical features through a cross-attention mechanism, so as to generate the fusion feature vector for the stroke prediction task.

[0009] In one feasible implementation, the aforementioned prediction model and output layer include a downstream prediction model module and a result display and presentation module. The downstream prediction model module is used to identify the above-mentioned fused feature vectors based on the intelligent recognition model in order to obtain prediction results; The results display and presentation module is used to generate decision results that help doctors understand the model based on the above prediction results through an interpretability mechanism.

[0010] Secondly, the present invention also proposes a method for predicting stroke risk and prognosis, used in the stroke risk and prognosis prediction system described in the first aspect, comprising: Based on the above data processing and knowledge construction layer, spatiotemporal multimodal knowledge graphs and structured clinical features are extracted, cleaned and constructed from multi-source heterogeneous medical data. Based on the aforementioned feature engineering and fusion layer, the dynamic semantic information in the aforementioned spatiotemporal multimodal knowledge graph and the aforementioned structured clinical features are deeply fused through graph embedding and attention mechanisms to generate a fusion feature vector for stroke prediction tasks. Based on the aforementioned prediction model and output layer, stroke risk and prognosis are predicted using the aforementioned fused feature vectors. The prediction results are then generated through an interpretability mechanism to help doctors understand the model's decision-making results.

[0011] In one feasible implementation, when the aforementioned feature engineering and fusion layer includes a spatiotemporal multimodal knowledge graph construction module, the specific processing steps of the aforementioned spatiotemporal multimodal knowledge graph construction module include: Natural language processing technology is used to extract entity, relation and attribute information from the text data of the preprocessed data. The entities include diseases, symptoms, drugs, genes and examinations, and the relations include causal relations, treatment relations and concurrency relations. Associating the state changes of the aforementioned relationships or entities with the corresponding event time information generates a graph structure with time-series attributes, wherein the aforementioned event time information includes specific timestamps, time intervals, and fuzzy time period labels; Quantitative image features are extracted from the image data of the preprocessed data, and physiological waveform parameters are extracted from the physiological signals of the preprocessed data. The aforementioned quantified image features and physiological waveform parameters are then linked to relevant entity or relation nodes in the aforementioned graph through specific edges or attributes to form multimodal enhanced knowledge nodes. The extracted entities are uniformly mapped to unique identifiers in the medical standard ontology library to form connected entities; The extracted entities, relationships, attribute information, temporally ordered attribute graph structure, multimodal enhanced knowledge nodes, and connected entities are stored in a graph database to construct the spatiotemporal multimodal knowledge graph.

[0012] In one feasible implementation, when the aforementioned feature engineering and fusion layer includes a spatiotemporal multimodal knowledge graph embedding module, the specific processing steps of the aforementioned spatiotemporal multimodal knowledge graph embedding module include: Load the above spatiotemporal multimodal knowledge graph; An initial embedding operation is performed on the above spatiotemporal multimodal knowledge graph to obtain initial graph embedding features; The initial graph embedding features are subjected to time-aware and graph embedding iterative operations to generate the graph embedding features.

[0013] In one feasible implementation, the above-described time-aware and graph embedding iterative operations on the initial graph embedding features to generate the graph embedding features include: Sample positive triplet pairs and corresponding negative triplet pairs from the initial graph embedding features described above; For each pair of positive and negative sample triples, the spatiotemporal multimodal embedding representations of the head and tail entities are calculated, along with the temporal-encoded dynamic embedding representation of the correspondence. These embeddings are generated by fusing the entity's fundamental vector with its associated multimodal features through modal projection and modal attention mechanisms. Based on the above spatiotemporal multimodal embedding representation and time-coded dynamic embedding representation, the positive sample scoring function value and negative sample scoring function value of each triple are calculated; Based on the above positive sample scoring function values ​​and the above negative sample scoring function values, a joint loss function is constructed; The above-mentioned joint loss function and optimizer are used for training to update parameters until convergence, thereby generating the above-mentioned graph embedding features.

[0014] In one feasible implementation, where the feature engineering and fusion layer includes an attention mechanism-driven multi-feature fusion module, the specific processing procedure of the attention mechanism-driven multi-feature fusion module includes: Calculate the first attention-weighted representation of the clinical features by the above graph embedding features; Calculate the second attention-weighted representation of the above clinical features on the above graph embedding features; The above-mentioned graph embedding features, clinical features, first attention-weighted representation, and second attention-weighted representation are pooled to obtain a fixed-dimensional representation vector. The pooling results are then concatenated and input into the fusion structure to generate the fusion feature vector used for stroke prediction task.

[0015] In one feasible implementation, the prediction of stroke risk and prognosis in the above-mentioned prediction model and output layer is obtained by any one or more of the following: extreme gradient boosting algorithm, lightweight gradient boosting machine, feedforward neural network and long short-term memory network.

[0016] In summary, this invention proposes a stroke risk and prognosis prediction system based on Spatio-Temporal Multimodal Knowledge Graph Embedding (STM-KGE) and Attention-driven Multi-feature Fusion (AMFF). By constructing an STM-KG containing temporal and multimodal information, an innovative STM-KGE algorithm is used to learn embedding vectors that capture temporal dynamics and multimodal characteristics. Furthermore, an AMFF module incorporating cross-attention intelligently fuses graph embedding features with clinical features to generate richer fused features. Inputting these fused features into a downstream prediction model significantly improves the accuracy of predicting stroke recurrence, hemorrhagic transformation, or functional prognosis, providing a new approach for precise stroke management.

[0017] Other advantages, objectives and features of this application will be apparent in part from the description which follows, and in part from what those skilled in the art will understand through study and practice of this application. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit this specification. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A structural schematic diagram of a stroke risk and prognosis prediction system provided for an embodiment of this application; Figure 2 A schematic diagram illustrating the principle of a stroke risk and prognosis prediction system provided in this application embodiment; Figure 3 A flowchart illustrating a method for predicting stroke risk and prognosis is provided in this application embodiment. Figure 4 A schematic diagram of an example fragment of a spatiotemporal multimodal knowledge graph provided in this application embodiment; Figure 5 A schematic diagram illustrating the principle of a spatiotemporal multimodal knowledge graph embedding algorithm provided in this application embodiment; Figure 6 This is a schematic diagram of the core structure of an attention mechanism-driven multi-feature fusion algorithm provided in an embodiment of this application. Detailed Implementation

[0019] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0020] Please see Figure 1 and Figure 2 , Figure 1 A structural schematic diagram of a stroke risk and prognosis prediction system provided for an embodiment of this application; Figure 2 A schematic diagram illustrating the principle of a stroke risk and prognosis prediction system provided in this application embodiment; the schematic diagram of a stroke risk and prognosis prediction system provided in this application embodiment may specifically include: The data processing and knowledge construction layer 10 is used to extract, clean, and construct spatiotemporal multimodal knowledge graphs and structured clinical features from multi-source heterogeneous medical data. Feature engineering and fusion layer 11 is used to deeply fuse the dynamic semantic information in the above spatiotemporal multimodal knowledge graph and the above structured clinical features through graph embedding and attention mechanisms to generate a fusion feature vector for stroke prediction tasks. The prediction model and output layer 12 are used to predict stroke risk and prognosis based on the above-mentioned fused feature vectors, obtain prediction results, and generate decision results that help doctors understand the model through an interpretability mechanism.

[0021] For example, in this embodiment, the overall system architecture includes a data processing and knowledge construction layer 10, a feature engineering and fusion layer 11, and a prediction model and output layer 12. The three layers work together in sequence to form an intelligent analysis process for predicting stroke risk and prognosis.

[0022] First, in the data processing and knowledge construction layer 10, the system obtains stroke-related information from multi-source heterogeneous medical data. Data sources include Electronic Health Record (EHR), Public / Publisher MEDLINE (PubMed), Picture Archiving and Communication System (PACS), and various physiological signal monitoring devices. This layer, through the data access and preprocessing module, cleans, standardizes (e.g., DICOM image decoding, signal filtering), anonymizes, and preprocesses the raw data, ensuring reliable data quality and good consistency in subsequent processes. Based on this, the spatio-temporal multimodal knowledge graph construction module integrates extracted entities (e.g., symptoms, drugs, examination items), relationships (e.g., "treatment," "concurrency," "cause"), time information (e.g., examination time, medication time), and multimodal features (e.g., radiomics parameters, physiological signal features) to construct a structured spatio-temporal multimodal knowledge graph (STM-KG). Meanwhile, the clinical feature extraction module extracts key variables needed for stroke prediction from structured clinical data, including demographic information, history of underlying diseases, laboratory test results, vital signs and rating scale data, to form a structured clinical feature vector.

[0023] Secondly, in the feature engineering and fusion layer 11, the system introduces a spatio-temporal multimodal knowledge graph embedding (STM-KGE) module. Through graph neural networks and time-aware mechanisms, it performs low-dimensional embedding modeling of entities and relations in STM-KG, generating graph embedding features that simultaneously reflect graph structure, temporal dynamics, and multimodal information. Next, it utilizes an attention-driven multi-feature fusion (AMFF) module to model the dependency relationship between graph embedding features and structured clinical features using a cross-attention mechanism, realizing mutual weighting and semantic interaction between cross-domain features, and finally generating a fused feature vector for downstream tasks.

[0024] Finally, in the prediction model and output layer 12, the system inputs the fused feature vector to the downstream prediction model module. This module can perform predictions based on one or more model combinations, including: Extreme Gradient Boosting (XGBoost), Light Gradient Boosting Machine (LightGBM), Feedforward Neural Network (FNN), and Long Short-Term Memory (LSTM). This module outputs predicted probabilities or risk levels based on the prediction target (such as stroke recurrence, hemorrhage transformation, or mRS score). To enhance clinical usability, the results display and presentation module combines interpretability techniques (such as Local Interpretable Model-agnostic Explanations (LIME) and Shapley Additive explanations (SHAP)) or embedded layer attention weights to highlight key prediction criteria (such as a specific risk factor node) and graphically display the prediction results and risk interpretations to help physicians understand the system's decision-making process.

[0025] In summary, this embodiment achieves a complete closed-loop process from multi-source data access, knowledge graph modeling, deep feature fusion to prediction and interpretation output through the linkage of three modules. It has powerful spatiotemporal perception, multimodal integration and clinical application capabilities, and is suitable for intelligent prediction of stroke risk and personalized prognostic assessment scenarios.

[0026] In summary, this invention proposes a stroke risk and prognosis prediction system based on Spatio-Temporal Multimodal Knowledge Graph Embedding (STM-KGE) and Attention-driven Multi-feature Fusion (AMFF). By constructing an STM-KG containing temporal and multimodal information, an innovative STM-KGE algorithm is used to learn embedding vectors that capture temporal dynamics and multimodal characteristics. Furthermore, an AMFF module incorporating cross-attention intelligently fuses graph embedding features with clinical features to generate richer fused features. Inputting these fused features into a downstream prediction model significantly improves the accuracy of predicting stroke recurrence, hemorrhagic transformation, or functional prognosis, providing a new approach for precise stroke management.

[0027] In one feasible implementation, the aforementioned data processing and knowledge construction layer 10 includes a data access and preprocessing module 101, a spatiotemporal multimodal knowledge graph construction module 102, and a clinical feature extraction module 103. The aforementioned data access and preprocessing module 101 is responsible for extracting and cleaning data from multi-source heterogeneous medical data to obtain preprocessed data. The aforementioned spatiotemporal multimodal knowledge graph construction module 102 constructs the aforementioned spatiotemporal multimodal knowledge graph based on the aforementioned preprocessed data. The aforementioned spatiotemporal multimodal knowledge graph includes entities, relationships, temporal information, and multimodal features. The aforementioned clinical feature extraction module 103 extracts key clinical indicators for stroke risk and prognosis modeling from the structured clinical data in the preprocessed data and completes the corresponding feature engineering to obtain the aforementioned structured clinical features.

[0028] For example, the data access and preprocessing module 101 is responsible for accessing stroke-related information from multi-source heterogeneous medical data. The data sources include Electronic Health Record (EHR) systems, medical literature databases (such as PubMed), Picture Archiving and Communication Systems (PACS), and physiological signal monitoring devices. This module performs a series of preprocessing operations, such as data cleaning, format standardization (e.g., DICOM image format decoding, waveform signal filtering), data anonymization (to protect patient privacy), and preliminary text normalization (e.g., sentence segmentation, noise removal), transforming the raw unstructured or semi-structured medical information into well-structured and semantically complete preprocessed data, providing a unified input for subsequent knowledge graph construction and clinical feature extraction.

[0029] Based on this, the spatio-temporal multimodal knowledge graph construction module 102 uses the preprocessed data as input to construct a spatio-temporal multimodal knowledge graph (STM-KG) for the task of predicting stroke risk and prognosis. The atlas not only includes traditional medical entities (such as disease subtypes, symptoms, signs, drugs, genes, and examination items) and relationship types (such as "cause," "treatment," and "association"), but more importantly, it introduces two enhancement mechanisms: First, the fusion of temporal information, which adds standardized temporal expressions (such as specific timestamps, time intervals, or fuzzy time periods) to event-type relationships (such as "medication start," "symptom onset," and "examination") or entity state changes (such as NIHSS score records); second, the linking of multimodal features, which links quantitative parameters extracted from imaging data (such as CT and MRI) (such as lesion volume, spatial location, and radiomics features) and key indicators extracted from physiological signals (such as ECG and EEG) (such as heart rate variability and specific waveforms) to corresponding entity or event nodes in the atlas in a structured manner. This can take the form of additional attributes or feature nodes, and the connections are made through semantic relationships such as "possessing imaging features" or "manifesting as a signal pattern." The final generated STM-KG is stored in a graph database that supports attribute expansion and temporal queries, exhibiting good scalability and query efficiency.

[0030] Meanwhile, the clinical feature extraction module 103 extracts key indicators for stroke risk assessment and prognostic modeling from the preprocessed structured clinical data. The extracted data includes demographic information (such as age and gender), history of underlying diseases (such as hypertension, diabetes, atrial fibrillation, etc.), admission assessment scales (such as NIHSS score, Glasgow Coma Scale), vital signs data, and laboratory test results (such as blood glucose level, blood lipid level, coagulation function, etc.). Furthermore, this module performs necessary feature engineering operations, including numerical normalization, missing value imputation, and categorical variable encoding, ensuring that all clinical features have a uniform scale and high-quality expression, thereby providing accurate numerical input for downstream models.

[0031] In summary, this implementation method, through the close collaboration of modules 101-103, achieves efficient transformation from raw heterogeneous medical data to a well-structured spatiotemporal multimodal knowledge graph and structured clinical feature vectors, providing strong data support and knowledge representation foundation for the intelligent stroke prediction system.

[0032] In one feasible implementation, the feature engineering and fusion layer 11 includes a spatiotemporal multimodal knowledge graph embedding module 111 and an attention mechanism-driven multi-feature fusion module 112. The aforementioned spatiotemporal multimodal knowledge graph embedding module 111 is used to perform time-aware and multimodal feature fusion graph embedding encoding operations on the aforementioned spatiotemporal multimodal knowledge graph to generate graph embedding features; The attention mechanism-driven multi-feature fusion module 112 is used to learn the interaction relationship between the graph embedding features and the structured clinical features through the cross-attention mechanism, so as to generate the fusion feature vector for the stroke prediction task.

[0033] For example, in one feasible implementation, the feature engineering and fusion layer 11 includes a spatiotemporal multimodal knowledge graph embedding module 111 and an attention mechanism-driven multi-feature fusion module 112. The two work together to deeply fuse structured clinical features with complex graph structures and multimodal semantic information in the spatiotemporal multimodal knowledge graph to construct more expressive input features to support downstream stroke risk and prognosis prediction tasks.

[0034] Specifically, the spatiotemporal multimodal knowledge graph embedding module 111 takes the constructed spatiotemporal multimodal knowledge graph (STM-KG) as input and performs graph embedding encoding operations to transform entities and relations in the graph into low-dimensional embedding vectors. The embedding algorithm used in this module has two core designs: On the one hand, time-aware embedding mechanisms have been introduced to handle dynamic information about the evolution of entities or relationships over time. This mechanism embeds the time of event occurrence, duration, or time interval into a vector representation by introducing time encoding methods (such as recurrent neural networks (RNN), Transformer encoders, or time representations using Fourier transforms), thereby capturing temporal characteristics such as "symptoms precede medication" or "delayed updates to test results." On the other hand, multimodal feature fusion mechanisms are used to jointly model heterogeneous modal features associated with atlas entities (such as lesion volume and texture features in image data, and heart rate variability and waveform patterns in physiological signals) with the basic graph structure. This process can achieve information-weighted fusion through cross-modal attention mechanisms, or optimize atlas representations through joint loss function design (while simultaneously constraining semantic structure preservation and multimodal collaborative consistency).

[0035] Ultimately, the graph embedding vectors output by this module not only contain the static semantic information of the nodes and the neighborhood features of the graph structure, but also carry their temporal dynamic features and multimodal attribute content, providing semantically rich input for subsequent fusion learning.

[0036] Next, the attention-driven multi-feature fusion module 112 receives graph embedding features (such as embedding vectors of patient nodes, diagnosis nodes, risk factor nodes, etc.) generated by the aforementioned embedding module, as well as structured clinical feature vectors from the clinical feature extraction module. This module uses a cross-attention mechanism to model the interaction dependencies between the two types of features.

[0037] Specifically, this module calculates the attention distribution of graph embedding features to different clinical features, thereby identifying which structured indicators are more discriminative in the current graph semantic context. Simultaneously, it allows clinical feature vectors to act inversely on graph embedding features, extracting more instructive representations by focusing on key node relationships within the graph. This cross-attention mechanism effectively enhances the semantic coupling between cross-domain information, compensating for the shortcomings of traditional concatenation or simple weighting methods in modeling complex feature interactions.

[0038] Finally, the graph embedding and clinical feature vector, after cross-attention weighting, will be jointly input into the fusion structure (such as a multilayer perception mechanism, fusion transformation layer, etc.) to generate a fusion feature vector that is rich in information, has strong semantic relevance, and has high discriminative ability, which will serve as the final input feature of the stroke prediction model.

[0039] In summary, this implementation method, through the combination of modules 111 and 112, constructs a deep semantic expression mechanism that retains both the semantics of the graph structure and temporal dynamics, while integrating multimodal features and structured indicators, providing a solid feature foundation and model interpretability support for intelligent prediction of stroke.

[0040] In one feasible implementation, the aforementioned prediction model and output layer 12 includes a downstream prediction model module 121 and a result display and presentation module 122. The downstream prediction model module 121 is used to identify the above-mentioned fused feature vectors according to the intelligent recognition model in order to obtain prediction results; The results display and presentation module 122 is used to generate decision results that help doctors understand the model based on the above prediction results through an interpretability mechanism.

[0041] For example, the prediction model and output layer 12 includes a downstream prediction model module 121 and a result display and presentation module 122, which are used to realize intelligent prediction and interpretable result output of stroke risk and prognosis based on fused feature vectors, thereby improving the usability and credibility of the model in clinical practice.

[0042] Specifically, the downstream prediction model module 121 receives the final fused feature vector from the attention mechanism-driven multi-feature fusion module (AMFF) and selects the appropriate intelligent model structure based on different stroke prediction targets. Prediction tasks may include: stroke recurrence risk (binary classification), hemorrhage transformation risk (binary classification), and functional prognostic score (mRS score, multi-class or regression) at a future time point. To meet these diverse task requirements, this module supports various machine learning and deep learning algorithms, including Extreme Gradient Boosting (XGBoost), Light Gradient Boosting Machine (LightGBM), Feedforward Neural Network (FNN), Long Short-Term Memory (LSTM), and Transformer architecture models. The module performs a complete model training, validation, and testing process, ultimately outputting results including predicted probabilities, risk stratification labels, and prognostic scores to support subsequent clinical risk assessment and decision support.

[0043] To further enhance the model's transparency and clinical interpretability, the results display and presentation module 122 interprets and visualizes the aforementioned prediction results. On one hand, this module can directly utilize attention weight information from the AMFF module to determine which graph entities, relationship nodes, or clinical features play a key role in the prediction process. On the other hand, it can also integrate mainstream interpretability AI technologies, such as Local Interpretable Model-agnostic Explanations (LIME) and SHapley Additive explanations (SHAP), to perform local feature contribution analysis on the model output. In this way, physicians can view the core risk factors or graph paths that the model "focuses" on, such as an important symptom, laboratory indicator, or key relationship nodes in the graph, thereby understanding the basis behind the model's decisions.

[0044] Ultimately, the prediction results and explanations will be presented visually in the user interface, such as highlighted prediction paths, color-coded risk levels, and feature weight ranking charts, to assist clinicians in making scientific decisions and personalized interventions. Overall, this implementation, through the collaboration of modules 121 and 122, achieves a closed-loop design from in-depth prediction to interpretable display, improving the practicality, reliability, and clinical friendliness of the stroke risk and prognosis prediction system.

[0045] The second aspect, such as Figure 3 As shown, the present invention also proposes a method for predicting stroke risk and prognosis, used in the stroke risk and prognosis prediction system described in the first aspect, comprising: S210. Based on the above data processing and knowledge construction layer, extract, clean, and construct a spatiotemporal multimodal knowledge graph and structured clinical features from multi-source heterogeneous medical data. S220. Based on the above feature engineering and fusion layer, the dynamic semantic information in the above spatiotemporal multimodal knowledge graph and the above structured clinical features are deeply fused through graph embedding and attention mechanisms to generate a fusion feature vector for stroke prediction task. S230. Based on the above prediction model and the output layer, the risk and prognosis of stroke are predicted using the above fused feature vector. The prediction results are obtained, and the decision results are generated to help doctors understand the model through an interpretability mechanism.

[0046] For example, this solution provides a method for predicting stroke risk and prognosis based on spatiotemporal multimodal knowledge graph and intelligent fusion mechanism, which mainly includes the following three steps, corresponding to steps S210, S220 and S230 respectively, forming a closed-loop process from multi-source data processing to interpretable prediction output.

[0047] In step S210, the system actively connects to or receives data from multiple pre-configured data sources, including but not limited to: Electronic Health Record (EHR) systems, medical literature databases (such as PubMed), Picture Archiving and Communication Systems (PACS), physiological signal monitoring databases (such as ECG databases), and Laboratory Information Systems (LIS). For different data sources and formats, corresponding data extraction operations are performed. For example, parsing HL7 messages, querying SQL databases, processing DICOM files, reading CSV / JSON files, and extracting literature abstracts. The extracted raw data is cleaned to remove noise, handle outliers, and standardize units and codes (such as drug names and diagnostic codes). Basic NLP processing, such as sentence segmentation, stemming, or lemmatization, is performed on text data. Necessary preprocessing (such as denoising and registration) is performed on image data. Signal data is filtered. Data involving patient privacy (such as names and ID numbers) is anonymized or pseudonymous to ensure compliance with data security and privacy regulations. In step S210, the system performs unified processing and structural modeling of the accessed multi-source heterogeneous medical data through a data processing and knowledge construction layer. Specifically, this step first extracts stroke-related data from various data sources, such as electronic medical record systems (EHRs), medical literature databases (e.g., PubMed), medical imaging systems (PACS), and physiological signal monitoring equipment. These data are then cleaned, formatted (e.g., image DICOM decoding, signal filtering), anonymized, and preprocessed to obtain high-quality, standardized preprocessed data. Based on this, a spatiotemporal multimodal knowledge graph (STM-KG) for stroke prediction is constructed. This graph includes not only medical entities (e.g., diseases, symptoms, drugs, examinations) and their semantic relationships, but also event timestamps (e.g., medication start time, examination time) and multimodal features extracted from image and signal data (e.g., lesion volume, heart rate variability), integrating these features into the graph as node attributes or independent feature nodes. Meanwhile, key features, including demographic information, basic medical history, admission indicators, and laboratory tests, were extracted from structured clinical data. Clinical feature engineering was then performed through normalization and missing value processing to obtain structured clinical feature vectors that can be used for subsequent modeling.

[0048] In step S220, the system utilizes two key modules in the feature engineering and fusion layer: a spatiotemporal multimodal knowledge graph embedding module and an attention-driven multi-feature fusion module, to deeply fuse the aforementioned graph and clinical features. First, entities and relationships in the STM-KG are modeled using a graph embedding encoding algorithm. A time-aware mechanism (such as a time encoder) and a multimodal fusion strategy (such as a modal attention mechanism) are employed to compress the graph structure, temporal dynamics, and multimodal attributes into a unified low-dimensional graph embedding vector. Then, a cross-attention mechanism is used to model the interaction between graph embedding features and structured clinical features, extracting key feature interactions. Finally, a highly expressive fusion feature vector is generated through a fusion structure (such as a multilayer perceptron) for use in intelligent stroke prediction tasks.

[0049] In step S230, the system invokes the prediction model and output layer to perform downstream intelligent prediction and interpretable output on the fused feature vector. Specifically, various models, including Extreme Gradient Boosting (XGBoost), Lightweight Gradient Boosting Machine (LightGBM), Feedforward Neural Network (FNN), and Long Short-Term Memory (LSTM), can be selected for training and inference based on different prediction objectives (such as stroke recurrence risk, hemorrhage transformation risk, and future functional prognosis score). The prediction results can be in the form of probability distribution, risk level, or score value. Furthermore, the system combines attention mechanism weights or integrates interpretable AI methods (such as LIME and SHAP) to explain the model's decision-making process. For example, it identifies graph nodes or key clinical indicators that play a decisive role in the prediction and presents these results to doctors through a visual interface, helping them better understand the model's decision-making basis and improving clinical trust and efficiency.

[0050] In summary, this invention proposes a stroke risk and prognosis prediction method based on Spatio-Temporal Multimodal Knowledge Graph Embedding (STM-KGE) and Attention-driven Multi-feature Fusion (AMFF). By constructing an STM-KG containing temporal and multimodal information, an innovative STM-KGE algorithm is used to learn embedding vectors that capture temporal dynamics and multimodal characteristics. Furthermore, an AMFF module incorporating cross-attention intelligently fuses graph embedding features with clinical features to generate richer fused features. Inputting these fused features into a downstream prediction model significantly improves the accuracy of predicting stroke recurrence, hemorrhagic transformation, or functional prognosis, providing a new approach for precise stroke management.

[0051] In one feasible implementation, when the aforementioned feature engineering and fusion layer includes a spatiotemporal multimodal knowledge graph construction module, the specific processing steps of the aforementioned spatiotemporal multimodal knowledge graph construction module include: Natural language processing technology is used to extract entity, relation and attribute information from the text data of the preprocessed data. The entities include diseases, symptoms, drugs, genes and examinations, and the relations include causal relations, treatment relations and concurrency relations. Associating the state changes of the aforementioned relationships or entities with the corresponding event time information generates a graph structure with time-series attributes, wherein the aforementioned event time information includes specific timestamps, time intervals, and fuzzy time period labels; Quantitative image features are extracted from the image data of the preprocessed data, and physiological waveform parameters are extracted from the physiological signals of the preprocessed data. The aforementioned quantified image features and physiological waveform parameters are then linked to relevant entity or relation nodes in the aforementioned graph through specific edges or attributes to form multimodal enhanced knowledge nodes. The extracted entities are uniformly mapped to unique identifiers in the medical standard ontology library to form connected entities; The extracted entities, relationships, attribute information, temporally ordered attribute graph structure, multimodal enhanced knowledge nodes, and connected entities are stored in a graph database to construct the spatiotemporal multimodal knowledge graph.

[0052] For example, when the feature engineering and fusion layer includes a spatiotemporal multimodal knowledge graph construction module, the specific processing of this module is geared towards the stroke prediction task, fusing multimodal information such as natural language, images, and physiological signals to construct a high-quality knowledge graph with temporal and semantic attributes.

[0053] First, this module uses Natural Language Processing (NLP) technology to extract structured information from preprocessed text data (such as electronic medical records, medical literature summaries, etc.), identifying the medical entities, semantic relationships, and related attribute information involved. The entity types include, but are not limited to, diseases, symptoms, drugs, genes, and examination items, while the relationship types cover various semantic associations such as causal relationships, treatment relationships, and concurrent relationships, forming a preliminary entity-relationship-attribute triple structure.

[0054] Secondly, after extracting entities and relationships, the system enters the time information association stage, further identifying the event times described in the text (such as "medication started three days ago," "symptoms appeared on the first day of hospitalization"), and parsing them into structured time information, specifically represented as standard timestamps, time intervals, or fuzzy time period labels (such as "preoperative," "during hospitalization"). This time information is attached to the corresponding relationship or entity state changes, giving the graph temporal attributes, thereby supporting time-aware modeling.

[0055] Next, the system performs feature extraction and atlas linking operations on non-textual modal data sources. For image data (such as CT and MRI), the system can call pre-trained image models or run radiomics workflows to extract quantitative image features such as lesion volume, shape, texture, and density. For physiological signal data (such as ECG and EEG), parameters such as heart rate variability, frequency domain energy distribution, and specific waveform features are calculated. These multimodal features are then structurally linked to the corresponding patient entities, examination events, or treatment behavior nodes in the atlas. The linking form can be node attribute embedding or newly created feature nodes, connected to the main atlas nodes through relationship types such as "has image features" or "represents a signal pattern," forming a multimodal enhanced knowledge structure.

[0056] Subsequently, to improve the semantic consistency and standardization of the graph in cross-system interaction, the system performs entity parsing and standard linking operations, aligning and matching the extracted medical entities (including their synonyms, aliases, abbreviations, etc.) with the unique identifiers in standard medical ontology libraries (such as SNOMED CT, ICD, RxNorm, LOINC), thereby achieving entity uniqueness and ontology standardization, and solving the problems of entity redundancy and inaccurate matching caused by semantic ambiguity.

[0057] Finally, the system stores all the constructed content, including parsed entities, relationships with associated time information, multimodal enhanced nodes, and standardized entities with unified identifiers, in a graph database in the form of triples or extended graph structures. This database supports attribute-level and time-based query operations and has an incremental update mechanism, allowing for the real-time addition of nodes, supplementation of edges, or updating of existing node attributes and time labels based on new data streams. Ultimately, the module outputs a structured spatiotemporal multimodal stroke knowledge graph (STM-KG) with temporal representation capabilities and multimodal information linking capabilities, providing a rich, reliable, and queryable semantic foundation for downstream embedding learning and predictive modeling.

[0058] Figure 4 This is a schematic diagram of an example segment of a spatiotemporal multimodal knowledge graph provided in an embodiment of this application. In this graph, patient A (P1) is used as the central node, and its related medical entities and event information are systematically organized. First, the patient is diagnosed with ischemic stroke (Cond1), and this diagnosis is connected to the patient entity through the "have" relationship. The patient also participates in a diagnosis event (R_Diag), which occurs on July 26, 2024 at 10:00 (T1). The time information is attached to the diagnosis event as an independent node through the "occurrence time" relationship, thereby realizing the time representation capability of the graph.

[0059] Following diagnosis, the patient underwent a medication use. The patient is linked to the specific drug entity alteplase (Drug1) via a "use" relationship, with a dose of 0.9 mg / kg. This action is modeled as a medication use event (R_Use) in the graph and linked back to the patient node via a "experience medication use" relationship. This medication use event has explicit time information—July 26, 2024, 10:05 (T2)—indicating that it occurred shortly after diagnosis. This time point is also represented by an explicit node and linked via a "started at" relationship.

[0060] Furthermore, the atlas also records examination behaviors and results related to the patient. The patient "underwent" a head CT scan (Test1), which is further connected to an image feature node named lesion ROI_123 (ImgFeat1), representing a quantitative radiomics indicator extracted from the image data. This type of structure enhances the atlas's multimodal representation capabilities through "image feature" relationships.

[0061] In terms of physiological signals, the atlas also includes a feature node from ECG (electrocardiogram) analysis, namely the atrial fibrillation pattern (SigFeat1). This feature is connected to the patient entity through a "representation" relationship, indicating that the patient exhibits atrial fibrillation signal characteristics during physiological monitoring, which is a typical way of expressing physiological modality information.

[0062] Overall, this knowledge graph not only encompasses the basic structural elements of a medical knowledge graph (entities, relationships), but also organically combines temporal information, event modeling, and modal features (images, signals). Each medical action (such as medication or diagnosis) is modeled as an event node and linked to modal features through explicit time tags, achieving a realistic reconstruction and temporal modeling of the diagnosis and treatment process. This structure provides significant data support for subsequent graph embedding learning, attention feature fusion, and stroke risk and prognosis prediction modeling.

[0063] This solution enables unified modeling of structured and unstructured information in medical data, enhances the ability of knowledge graphs to express temporal and modal dimensions, and significantly improves the practicality and intelligent performance of the graph in stroke risk identification and prognosis modeling scenarios.

[0064] In one feasible implementation, when the aforementioned feature engineering and fusion layer includes a spatiotemporal multimodal knowledge graph embedding module, the specific processing steps of the aforementioned spatiotemporal multimodal knowledge graph embedding module include: Load the above spatiotemporal multimodal knowledge graph; An initial embedding operation is performed on the above spatiotemporal multimodal knowledge graph to obtain initial graph embedding features; The initial graph embedding features are subjected to time-aware and graph embedding iterative operations to generate the graph embedding features.

[0065] In one feasible implementation, Figure 5 This application provides a schematic diagram of a spatiotemporal multimodal knowledge graph embedding algorithm. The algorithm involves performing time-aware and iterative graph embedding operations on the initial graph embedding features to generate the new graph embedding features, including: Sample positive triplet pairs and corresponding negative triplet pairs from the initial graph embedding features described above; For each pair of positive and negative sample triples, the spatiotemporal multimodal embedding representations of the head and tail entities are calculated, along with the temporal-encoded dynamic embedding representation of the correspondence. These embeddings are generated by fusing the entity's fundamental vector with its associated multimodal features through modal projection and modal attention mechanisms. Based on the above spatiotemporal multimodal embedding representation and time-coded dynamic embedding representation, the positive sample scoring function value and negative sample scoring function value of each triple are calculated; Based on the above positive sample scoring function values ​​and the above negative sample scoring function values, a joint loss function is constructed; The above-mentioned joint loss function and optimizer are used for training to update parameters until convergence, thereby generating the above-mentioned graph embedding features.

[0066] For example, the spatiotemporal multimodal knowledge graph embedding STM-KGE module system first loads the constructed spatiotemporal multimodal knowledge graph (STM-KG) from the graph database, which contains information such as entities (e.g., diseases, drugs, patients), relationships (e.g., "treatment" and "induction"), event timestamps (e.g., medication time and examination time), and multimodal features (e.g., CT images and EEG signals).

[0067] Initialize embedding vectors for all entities and relation nodes in the knowledge graph. Initialization can use randomly distributed or pre-trained vectors (such as those based on TransE, GloVe, BioBERT, etc.) to generate initial feature vectors. This lays the foundation for subsequent training.

[0068] The STM-KGE core graph embedding training process includes: 1. Sample collection: Triple samples were sampled from the STM-KG: Positive samples: , indicating time Upper Entity and There is a relationship Negative samples: ( ), obtained by replacing entities.

[0069] 2. Multimodal entity embedding computation: Head entity (or tail entity) Multimodal feature fusion and embedding includes the following steps: Entity The original modal features (such as images and signals) are represented as multiple modal feature vectors. ; Mapped to a unified embedding space via modal projection operations:

[0070] An attention mechanism is applied to all modal projection vectors to obtain a weighted fusion representation:

[0071] in, Indicates the first The importance weights of each modality (which can be obtained through a learnable attention layer); Final fused entity embedding vector:

[0072] in, The modal fusion control coefficient adjusts the degree of influence of modal information on structural embedding.

[0073] 3. Time-aware relation embedding computation: relation The embedding requires the introduction of time encoding to construct a time-aware relationship embedding: through the time encoding function TimeEncode. Mapping timestamps to time vectors yields a dynamic relationship representation:

[0074] in, It is the fusion weight that controls the time offset.

[0075] 4. Scoring function calculation: Calculate the score (i.e., the degree of structural consistency) for each triple:

[0076] The L1 or L2 norm is used to measure the distance between vectors, and a lower score indicates a more reasonable embedding structure.

[0077] 5. Joint Loss Function Design: The training objective is to distinguish between positive and negative samples. The following joint loss function is constructed:

[0078] The specific meanings are as follows: For structural loss terms:

[0079] Used to ensure that the score of positive samples is lower than that of negative samples, margin is the interval hyperparameter; To mitigate modal alignment loss, improve modal projection consistency; To mitigate the time consistency loss, we maintain the effective constraint of time information on the model. The weighting hyperparameters are used to control various losses.

[0080] 6. Parameter optimization and update: The gradients of all parameters are calculated using the backpropagation algorithm, including: entity / relation embedding vectors; modal projection matrix (ModalProject); and attention weights. Fusion factor Time offset Use optimizers (such as Adam, SGD) for training and updates until the loss converges.

[0081] The gradients of all parameters are calculated using the backpropagation algorithm, including: entity / relation embedding vectors; modal projection matrix (ModalProject); attention weights α, fusion factor β, and time offset γ; and then the parameters are updated using an optimizer (such as Adam or SGD) until the loss converges.

[0082] Once the model training is complete, the system outputs the final embedding vectors of all entities and relationships in the STM-KG, forming a spatiotemporally aware, multimodal fusion graph embedding feature library. These embeddings will serve as inputs to the subsequent attention fusion module (AMFF) and downstream prediction models, supporting stroke risk assessment and prognosis prediction tasks.

[0083] In summary, this STM-KGE module fully leverages the temporal evolution and modal semantic information of entities and relationships in the graph through a triple fusion mechanism of structural modeling, temporal modeling, and modal perception. It achieves low-dimensional embedding and compression of high-dimensional medical knowledge and is the core feature extraction subsystem of the entire intelligent stroke prediction system.

[0084] In one feasible implementation, Figure 6 This application provides a schematic diagram of the core structure of an attention mechanism-driven multi-feature fusion algorithm. In the case where the feature engineering and fusion layer includes an attention mechanism-driven multi-feature fusion module, the specific processing steps of the attention mechanism-driven multi-feature fusion module include: Calculate the first attention-weighted representation of the clinical features by the above graph embedding features; Calculate the second attention-weighted representation of the above clinical features on the above graph embedding features; The above-mentioned graph embedding features, clinical features, first attention-weighted representation, and second attention-weighted representation are pooled to obtain a fixed-dimensional representation vector. The pooling results are then concatenated and input into the fusion structure to generate the fusion feature vector used for stroke prediction task.

[0085] For example, for each patient to be predicted, the embedding library output by STM-KGE obtains a set of relevant graph embedding vectors according to preset rules (such as selecting patient nodes, primary diagnosis nodes, key risk factor nodes, etc.). , fusion weight wait.

[0086] Obtain the set of clinical feature vectors corresponding to this patient. .

[0087] Calculate the first attention-weighted representation of the above graph embedding features on the clinical features. . Specifically: .

[0088] Calculate the second attention-weighted representation of the above clinical features on the above graph embedding features. . Specifically:

[0089] .

[0090] Pooling is performed on the original features and weighted features to obtain a fixed-size representation, for example... The pooled feature vectors are then concatenated.

[0091] The concatenated vectors are then input into a multilayer perceptron (MLP) or other fusion layers for final fusion.

[0092]

[0093] The final fusion feature vector for each patient .

[0094] In one feasible implementation, the prediction of stroke risk and prognosis in the above-mentioned prediction model and output layer is obtained by any one or more of the following: extreme gradient boosting algorithm, lightweight gradient boosting machine, feedforward neural network and long short-term memory network.

[0095] For example, during model selection and loading, the system selects an appropriate model structure based on the specific prediction target. For instance, for binary classification tasks such as stroke recurrence or hemorrhage conversion, XGBoost (Extreme Gradient Boosting) or LightGBM (Lightweight Gradient Boosting Machine) can be used; for multi-class or regression tasks such as mRS scoring, Feedforward Neural Network (FNN) or Long Short-Term Memory (LSTM) can be used. If in the inference phase, pre-trained model weights are loaded; if in the training phase, model initialization is performed.

[0096] Next comes the model training phase (executed only during training). The system inputs the fused feature vectors from the training set along with their corresponding ground truth labels (e.g., relapse status, mRS score) into the model, defines an appropriate loss function (e.g., cross-entropy loss for classification, mean squared error for regression), and selects an optimizer (e.g., Adam, SGD) for parameter learning and model tuning. The system periodically evaluates performance metrics, such as accuracy, AUC, or RMSE, on the validation set and automatically or manually adjusts hyperparameters such as learning rate and regularization strength accordingly to ensure optimal model performance in generalization ability and robustness.

[0097] During model prediction (inference phase), the system takes the fused feature vector of each patient as input and passes it to the trained model to obtain its output. The output includes the probability of stroke recurrence, functional prognostic scores (such as mRS) at specific future time points, or rank labels based on risk distribution. Finally, this module returns a quantitative stroke risk or prognostic prediction result for each patient.

[0098] To improve the transparency and reliability of the predictive model in a clinical setting, this module performs interpretive analysis and visualizes the results of the model output. The processing procedure is as follows: In interpretability analysis, the system first extracts attention weight information from the AMFF module to analyze the input sources that play a crucial role in the final prediction during the fusion process. Specifically, the system can identify which graph embedding features (such as disease nodes and image feature nodes) and which clinical features (such as NIHSS scores and blood pressure levels) have high weights in the attention distribution, thereby determining their importance to the model input. Furthermore, the system integrates mainstream interpretability AI technologies (such as LIME: Local Model-Independent Explanation Mechanism, and SHAP: Shapley Value-Based Feature Contribution Evaluation Method) to analyze the model's sensitivity and dependence on different feature dimensions during decision-making, helping users understand the causal basis of the predictions.

[0099] In the results visualization and presentation, the system displays the prediction results to users in an intuitive and easy-to-understand format. For example, risk levels can be displayed using color-coded labels, prediction scores can be displayed using scale graphs, and confidence intervals can be displayed using risk curves or bar charts. Simultaneously, explanatory information (such as a ranking list of important features, a heatmap of attention to nodes in the graph, and SHAP bar charts) will also be displayed, forming a traceable and interpretable prediction report to assist doctors in the diagnosis and intervention process.

[0100] In summary, this invention, by deeply integrating the complex biomedical relationships, temporal dynamics, and multimodal data features implicit in knowledge graphs, and combining them with traditional clinical indicators, significantly improves the accuracy and robustness of predicting key stroke events (such as recurrence risk and hemorrhagic transformation risk) and long-term functional prognosis (such as the degree of improvement in mRS score), surpassing traditional models that rely on only partial information. This invention overcomes the limitations of traditional methods in information utilization, enabling a more comprehensive and in-depth mining and utilization of the patient's full-dimensional information, including structured clinical data, unstructured knowledge graph relationships, temporal evolution patterns of key events, and unique biomarker information contained in multimodal data such as images and signals. For complex cases with multiple comorbidities and convoluted treatment processes, this invention integrates richer contextual knowledge and dynamic information, enabling the model to more accurately understand the complexity of the case, thereby making more reliable risk assessments and prognostic judgments. The innovative attention fusion mechanism (AMFF) not only optimizes the feature fusion effect, but the attention weights it generates also provide interpretability clues for model prediction. By analyzing which knowledge graph entities / relationships (via graph embedding features) or clinical features generate higher "attention" to the prediction results, clinicians can better understand the key basis for model decision-making, enhance their trust in the prediction results, and promote human-machine collaboration.

[0101] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A system for predicting stroke risk and prognosis based on, characterized by, Comprise: a data processing and knowledge construction layer for extracting, cleaning and constructing a spatio-temporal multi-modal knowledge graph and structured clinical features from multi-source heterogeneous medical data; a feature engineering and fusion layer for deep fusion of dynamic semantic information in the spatio-temporal multi-modal knowledge graph and the structured clinical features through graph embedding and attention mechanism to generate a fusion feature vector for stroke prediction task; a prediction model and output layer for stroke risk and prognosis prediction based on the fusion feature vector, obtaining a prediction result, and generating a decision result of the model through an explainability mechanism to assist doctors in understanding the model.

2. The stroke risk and prognosis prediction system based on the above claim 1, characterized in that, The data processing and knowledge construction layer comprises a data access and preprocessing module, a spatio-temporal multi-modal knowledge graph construction module, and a clinical feature extraction module, The data access and preprocessing module is responsible for extraction and cleaning operations from multi-source heterogeneous medical data to obtain preprocessed data; The spatio-temporal multi-modal knowledge graph construction module constructs the spatio-temporal multi-modal knowledge graph based on the preprocessed data, and the spatio-temporal multi-modal knowledge graph comprises entities, relationships, time information and multi-modal features; The clinical feature extraction module extracts key clinical indicators for stroke risk and prognosis modeling from structured clinical data in the preprocessed data and completes corresponding feature engineering to obtain the structured clinical features. 3.The stroke risk and prognosis prediction system based on the same according to claim 1, wherein, The feature engineering and fusion layer comprises a spatio-temporal multi-modal knowledge graph embedding module and an attention mechanism driven multi-feature fusion module, The spatio-temporal multi-modal knowledge graph embedding module is used for time-aware and multi-modal feature fusion graph embedding coding operation of the spatio-temporal multi-modal knowledge graph to generate graph embedding features; The attention mechanism driven multi-feature fusion module is used for learning the interaction between the graph embedding features and the structured clinical features through cross-attention mechanism to generate the fusion feature vector for stroke prediction task. 4.The system based on stroke risk and prognosis prediction according to claim 1, characterized in that, The prediction model and output layer comprises a downstream prediction model module and a result display and presentation module, The downstream prediction model module is used for identifying the fusion feature vector according to an intelligent recognition model to obtain a prediction result; The result display and presentation module is used for generating a decision result of the model through an explainability mechanism to assist doctors in understanding the model according to the prediction result.

5. A method for predicting stroke risk and prognosis based on the system for predicting stroke risk and prognosis according to any one of claims 1 to 4, characterized in that, Comprise: extracting, cleaning and constructing a spatio-temporal multi-modal knowledge graph and structured clinical features from multi-source heterogeneous medical data according to the data processing and knowledge construction layer; deep fusion of dynamic semantic information in the spatio-temporal multi-modal knowledge graph and the structured clinical features through graph embedding and attention mechanism according to the feature engineering and fusion layer to generate a fusion feature vector for stroke prediction task; stroke risk and prognosis prediction based on the fusion feature vector according to the prediction model and output layer, obtaining a prediction result, and generating a decision result of the model through an explainability mechanism to assist doctors in understanding the model. 6.The method of predicting stroke risk and prognosis based on the method according to claim 5, characterized in that, In the case that the feature engineering and fusion layer comprises a spatio-temporal multi-modal knowledge graph construction module, the specific processing process of the spatio-temporal multi-modal knowledge graph construction module comprises: extracting entities, relations and attribute information from the text data of the preprocessed data using natural language processing technology, the entities including diseases, symptoms, drugs, genes and examinations, the relations including causal relations, treatment relations and concurrent relations; associating the state changes of the relations or the entities with corresponding event time information to generate a time attribute graph structure, wherein the event time information includes specific timestamps, time intervals and fuzzy time period labels; extracting quantitative image features from the image data of the preprocessed data, and extracting physiological waveform parameters from the physiological signals of the preprocessed data; and link the quantitative image features and the physiological waveform parameters to the related entity or relation node in the graph through a specific edge or attribute to form a multi-modal enhanced knowledge node; mapping the extracted entities to unique identifiers in a medical standard ontology library to form connected entities; storing the extracted entities, relations, attribute information, time attribute graph structure, multi-modal enhanced knowledge node and connected entities in a graph database to construct the spatio-temporal multi-modal knowledge graph. 7.The method of predicting stroke risk and prognosis based on the method according to claim 5, characterized in that, In the case where the feature engineering and fusion layer includes a spatio-temporal multi-modal knowledge graph embedding module, the specific processing process of the spatio-temporal multi-modal knowledge graph embedding module includes: loading the spatio-temporal multi-modal knowledge graph; performing an initial embedding operation on the spatio-temporal multi-modal knowledge graph to obtain initial graph embedding features; performing time-aware and graph embedding iteration operations on the initial graph embedding features to generate the graph embedding features. 8.The method of predicting stroke risk and prognosis based on the method according to claim 7, characterized in that, The time-aware and graph embedding iteration operations on the initial graph embedding features to generate the graph embedding features include: sampling positive sample triples and corresponding negative sample triples from the initial graph embedding features; for each pair of positive sample triples and negative sample triples, calculating the spatio-temporal multi-modal embedding representation of the head entity and the tail entity, and calculating the time-encoding dynamic embedding representation of the corresponding relation, wherein the embedding is generated by fusing the entity base vector and its associated multi-modal features through modal projection and modal attention mechanism, calculating the positive sample scoring function value and the negative sample scoring function value of each triple based on the spatio-temporal multi-modal embedding representation and the time-encoding dynamic embedding representation; constructing a joint loss function according to the positive sample scoring function value and the negative sample scoring function value; training based on the joint loss function and the optimizer to update the parameters until convergence to generate the graph embedding features. 9.The method of predicting stroke risk and prognosis based on the method according to claim 5, characterized in that, In the case where the feature engineering and fusion layer includes a multi-feature fusion module driven by an attention mechanism, the specific processing process of the multi-feature fusion module driven by an attention mechanism includes: calculating a first attention weighted representation of the graph embedding features on the clinical features; calculating a second attention weighted representation of the clinical features on the graph embedding features; The graph embedding feature, the clinical feature, the first attention weighted representation and the second attention weighted representation are subjected to pooling processing to obtain a fixed-dimensional representation vector, and the pooling result is spliced and input into a fusion structure to generate a fusion feature vector for the stroke prediction task. 10.The method of predicting stroke risk and prognosis based on the method according to claim 5, characterized in that, The prediction model and the output layer are used for stroke risk and prognosis prediction by any one or more of extreme gradient boosting algorithm, light gradient boosting machine, feedforward neural network and long short-term memory network.

Images