Vascular embolism clinical risk prognosis evaluation system based on multi-source data

Abstract

The invention relates to the technical field of clinical auxiliary decision making, in particular to a vascular embolism clinical risk prognosis evaluation system based on multi-source data. The system comprises a patient multi-source data acquisition module, a clinical data feature processing module, a multi-source data fusion module and a clinical explanatory enhancement module. The method comprises the following steps: acquiring basic information, clinical examination results, vital signs and medical image data of a patient, and respectively performing structured and image feature extraction to form multi-modal feature representation; a key area is mined in combination with a self-attention mechanism, and the feature expression ability is improved; and through comparison of historical similar cases and contribution analysis of local features, interpretability enhancement of model output is realized, and an embolism risk prediction result with a clinical reference value is generated. The method effectively improves the prediction accuracy, stability and medical readability of the model, and is suitable for auxiliary diagnosis and treatment scenes of diseases such as vascular embolism.

Description

Technical Field

[0001] The present invention relates to the technical field of clinical decision support, and in particular to a clinical risk prognosis assessment system for vascular embolism based on multi-source data. Background Art

[0002] In clinical practice, vascular embolism, a common and high-risk vascular disease, is characterized by rapid onset, rapid progression, and high rates of disability and mortality, particularly in deep vein thrombosis, pulmonary embolism, and cerebral thrombosis. Early identification of high-risk individuals and scientific prognostic assessment are key to improving treatment efficiency and reducing mortality.

[0003] With the development of medical information technology, patients generate a large amount of heterogeneous data during their medical consultations, including basic demographic information, laboratory test results, vital sign monitoring data, and medical imaging data. This multi-source data contains a rich set of potential risk characteristics. However, due to the complex data types, strong temporal characteristics, high noise, and the fact that traditional risk assessment methods rely primarily on static single-modal indicators, how to further intelligent medical-assisted diagnosis based on multi-source data has become a challenge. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a clinical risk prognosis assessment system for vascular embolism based on multi-source data to solve at least one of the above technical problems.

[0005] The present application provides a clinical risk prognosis assessment system for vascular embolism based on multi-source data, the system comprising:

[0006] Patient multi-source data acquisition module, used to obtain patient basic data, clinical examination data, vital signs data and patient imaging data;

[0007] The patient clinical data feature processing module is used to extract image features based on the patient's image data to obtain image feature data; perform structural characterization based on the patient's basic data, clinical examination data, and vital signs data to obtain the patient's clinical structural feature data;

[0008] The patient clinical multi-source data fusion module is used to perform multimodal fusion based on the imaging feature data and the patient's clinical structural feature data to obtain the patient's multimodal data; self-attention processing is performed on the patient's multimodal data to obtain local area weighted data;

[0009] The patient clinical interpretability enhancement module is used to obtain historical similar data based on the user's patient basic data, clinical examination data and vital signs data to obtain historical similar data; and to enhance the interpretability based on historical similar data and local area weighted data to obtain clinical risk prediction data.

[0010] This invention achieves intelligent, high-precision prediction of clinical risk for vascular embolism by fusing multi-source heterogeneous medical data and combining image feature extraction, structural feature modeling, self-attention mechanisms, and interpretability enhancement techniques. The system fully exploits key patient risk factors in structured examination and imaging data, identifies local high-value / areas of interest through self-attention mechanisms, and performs label attribution and feature contribution analysis based on historical similar cases. This makes predictions more interpretable and clinically valuable, improving the accuracy, timeliness, and rationality of physicians' embolism risk assessments.

[0011] Preferably, the acquisition of patient basic data, clinical examination data, vital signs data and patient imaging data is used to:

[0012] Obtain basic patient data from the electronic medical record system, including age, gender, height, weight, and history of vascular embolism; obtain clinical examination data from the clinical laboratory electronic system; obtain vital sign data from vital sign monitoring equipment or nursing record system; and obtain patient imaging data through the medical imaging system.

[0013] This system integrates electronic medical record systems, clinical laboratory systems, vital sign monitoring equipment, and medical imaging systems to achieve automated and standardized collection of multi-source patient data. It efficiently and comprehensively acquires multidimensional health data covering both static attributes and dynamic physiological states, providing an accurate, authentic, and timely data foundation for feature extraction and risk prediction, thereby improving the overall system data quality and the reliability of intelligent analysis.

[0014] Preferably, the image feature extraction is performed based on the patient image data to obtain image feature data; and the structural characterization is performed based on the patient basic data, clinical examination data, and vital signs data to obtain the patient clinical structural feature data, which is used for:

[0015] Perform image registration based on patient image data to obtain image registration data; perform key area detection based on the image registration data to obtain key area detection data; segment the image registration data based on the key area detection data to obtain image segmentation data; perform depth feature extraction based on the image segmentation data to obtain image depth feature data; perform image feature aggregation on the image depth feature data to obtain image feature aggregation data;

[0016] Fields are screened based on the patient's basic data, clinical examination data, and vital signs data to obtain field screening data; numerical processing is performed on the field screening data to obtain field numerical data; missing marks are performed on the field numerical data to obtain missing marked data; feature vectors are constructed based on the missing marked data to obtain the patient's clinical structure feature data.

[0017] The present invention achieves deep extraction and standardized expression of multi-source features by separately and finely processing imaging data and structured clinical data. The imaging pathway sequentially completes registration, key region detection, segmentation, deep feature extraction, and aggregation, effectively improving the accuracy of identifying lesion areas and spatial expression capabilities. The structural pathway improves the expression integrity of clinical indicators and model readability through field screening, digitization, missing markers, and vector construction, thereby providing a high-quality, highly controllable input foundation for multimodal fusion and risk modeling, significantly enhancing the system's feature expression and prediction accuracy.

[0018] Preferably, multimodal fusion is performed on the image feature data and the patient's clinical structural feature data to obtain the patient's multimodal data; self-attention processing is performed on the patient's multimodal data to obtain local area weighted data, which is used for:

[0019] Performing feature dimension alignment based on the image feature data and the patient's clinical structural feature data to obtain feature dimension alignment data; performing dual-channel gating processing based on the feature dimension alignment data to obtain first patient multimodal data; performing contextual multi-head self-attention calculation based on the first patient multimodal data to obtain first multimodal weighted data; performing position mask weighting on the first multimodal weighted data to obtain first local area weighted data;

[0020] Performing multi-layer perceptron processing on the feature dimension aligned data to obtain second patient multimodal data; performing masked multi-head self-attention calculation on the second patient multimodal data to obtain second multimodal weighted data; performing maximum pooling on the second multimodal weighted data to obtain second local area weighted data;

[0021] Residual connection is performed according to the first local area weighted data and the second local area weighted data to obtain local area weighted data.

[0022] The present invention realizes the collaborative perception and deep representation of image features and structural features by constructing a dual-path multimodal fusion and self-attention calculation mechanism. Through feature dimension alignment and dual-channel gating processing, the feature interaction and importance control between modalities are effectively enhanced; the contextual multi-head attention mechanism is combined with position mask weighting to improve the spatial attention ability of the image area; the multi-layer perceptron and mask attention path are combined with the maximum pooling operation to realize the nonlinear expression of structural features and the extraction of high-weight regions; through the dual-path residual connection fusion, local region weighted data with both spatial locality and semantic weight is formed, which significantly improves the expression ability of multi-source feature fusion and the accuracy of downstream risk prediction. Although some methods in the prior art involve multimodal data input, they mostly adopt simple splicing or early fusion strategies, which fail to fully consider the semantic differences and complementary relationships between different modal features, resulting in information redundancy or the submerging of key features. The present invention realizes the compensatory fusion of multimodal and multi-mechanism information by performing residual connection on the dual-path attention results, and implements adjustments based on data characteristics in the process, effectively improving the robustness of the model to local area variations and enhancing the consistency and reliability of risk assessment results.

[0023] Preferably, the dual-channel gating processing is performed on the data aligned according to the feature dimensions to obtain the multimodal data of the first patient, which is used for:

[0024] Mutual information perception is performed based on the feature dimension alignment data to obtain mutual information perception data; gated channel generation is performed based on the mutual information perception data and the feature dimension alignment data to obtain gated channel data, wherein the gated channel data includes structure gated channel data and image gated channel data; dual-channel reconstruction and output are performed on the feature dimension alignment data based on the gated channel data to obtain feature reconstructed data; difference residual connection is performed based on the feature reconstructed data to obtain multimodal data of the first patient.

[0025] The present invention achieves dynamic modeling of the correlation and difference between image features and structural features by introducing a mutual information perception mechanism and a dual-channel gating structure. Mutual information perception data can accurately reflect the degree of semantic coupling between different modalities, thereby guiding the generation of more discriminative structural gating channels and image gating channels; through the reconstruction output of dual-channel alignment features, the expressive power of key modalities can be effectively enhanced; on this basis, a differential residual connection mechanism is further introduced to retain the residual information between heterogeneous features, improving the diversity and integrity of the fusion features, thereby constructing semantically rich, structurally clear, and informative first-patient multimodal data, providing more discriminative input support for downstream attention modeling and risk prediction.

[0026] Preferably, the contextual multi-head self-attention calculation is performed based on the first patient multimodal data to obtain first multimodal weighted data, which is used for:

[0027] Context expansion is performed on the first patient's multimodal data to obtain multimodal context expansion data; multi-head self-attention weight calculation is performed on the multimodal context expansion data to obtain multi-head self-attention weight data; multi-head splicing output is performed on the multimodal context expansion data based on the multi-head self-attention weight data to obtain multi-head splicing data; main position features are output based on the multi-head splicing data to obtain first multimodal weighted data.

[0028] The present invention enhances the ability of multimodal fusion features to perceive and model contextual information by introducing context expansion and multi-head self-attention mechanisms. Its beneficial effects are: by performing context expansion on the first patient's multimodal data, a local pseudo-sequence can be constructed to simulate time evolution or similar patient feature environments, thereby improving the model's understanding of cross-sample or local feature associations; multi-head self-attention weight calculation can achieve parallel attention to different semantic dimensions and explore deep dependencies between features; multi-head splicing output integrates multi-view attention information to further enrich the representation level; extracting the main position feature as the first multimodal weighted data gives the system stronger local context expression and feature selection capabilities, effectively improving the accuracy and robustness of the subsequent discriminant model.

[0029] Preferably, the first multimodal weighted data is weighted using a position mask to obtain first local area weighted data, which is used for:

[0030] Credibility calculation is performed based on the first multimodal weighted data to obtain credibility data; a position mask is constructed based on the credibility data to obtain position mask data; weighted calculation is performed on the first multimodal weighted data based on the position mask data to obtain multimodal position mask data; local activation calculation is performed based on the multimodal position mask data to obtain first local area weighted data.

[0031] The present invention introduces credibility perception and position mask mechanism to perform refined regional weighted processing on the first multimodal weighted data. Its beneficial effects are: by performing credibility calculation on multimodal weighted data, the reliability of each feature dimension or region can be dynamically evaluated, providing a data-driven basis for mask construction; the position mask data generated based on credibility can realize spatial modeling of information importance; weighted calculation of multimodal data under the guidance of mask effectively suppresses invalid feature interference and highlights key areas; high response areas are extracted through local activation calculation to obtain the first local area weighted data, which significantly improves the local recognition and regional perception ability of feature expression, and provides a more accurate information basis for risk prediction and interpretability analysis.

[0032] Preferably, the data aligned according to the feature dimensions is processed by a multi-layer perceptron to obtain the second patient multimodal data, which is used for:

[0033] Layer normalization is performed on the feature dimension aligned data to obtain feature layer normalized data; noise perturbation enhancement is performed on the feature layer normalized data to obtain feature enhanced data; nonlinear multi-layer perceptron stacking is performed on the feature enhanced data to obtain asymmetric network structure data; multi-scale residual cross-connection is performed on the asymmetric network structure data to obtain feature cross-layer connection data; dimension compression is performed on the feature cross-layer connection data to obtain multimodal data of the second patient.

[0034] The present invention enhances the expressiveness and robustness of features by constructing a multi-layer nonlinear perceptron structure and integrating normalization, noise perturbation, and multi-scale residual mechanisms. Layer normalization improves the stability of feature distribution and suppresses gradient explosion or vanishing problems; the introduction of noise perturbation enhancement operations can effectively improve the model's generalization ability for abnormal inputs or small changes; the asymmetric multi-layer perceptron stacking structure combined with multiple activation functions can fully model the nonlinear relationship between features of different dimensions; the multi-scale residual cross-connection mechanism realizes information fusion and compensation between deep and shallow features, avoiding information loss; and through dimensional compression, compact and semantically rich second patient multimodal data is obtained, providing more expressive and adaptable high-quality feature input for downstream attention modeling and risk prediction.

[0035] Preferably, the masked multi-head self-attention calculation is performed on the second patient multimodal data to obtain second multimodal weighted data, which is used for:

[0036] A pseudo-sequence is constructed on the multimodal data of the second patient to obtain pseudo-sequence data; low-weight indicator shielding is performed on the pseudo-sequence data to obtain low-weight indicator shielding data; channel opening is performed on the low-weight indicator shielding data according to preset indicator opening channel data to obtain channel opening data; position control is performed according to the channel opening data to obtain mask matrix data; multi-head self-attention calculation is performed on the multimodal data of the second patient according to the mask matrix data to obtain second multimodal weighted data.

[0037] The present invention introduces mask control and pseudo-sequence construction mechanisms to accurately perform self-attention modeling on the second patient's multimodal data. By constructing a pseudo-sequence of multimodal feature vectors, a substructure unit with semantic discrimination is constructed to facilitate local interaction modeling. A low-weight indicator shielding mechanism is used to filter noise information and reduce interference from invalid dimensions. Channel-level control is performed in conjunction with preset open channel data to ensure the integrity of the expression of key medical indicators during the attention propagation process. Furthermore, a mask matrix is ​​generated through position control to make the attention mechanism spatially selective and semantically targeted. Masked multi-head self-attention calculations are performed on the original features to effectively enhance the interaction modeling capabilities between high-risk features, thereby obtaining more discriminative second multimodal weighted data and providing more refined representation support for embolic risk assessment.

[0038] Preferably, the historical similarity data is acquired based on the user's patient basic data, clinical examination data, and vital signs data to obtain historical similarity data; and interpretability enhancement is performed based on the historical similarity data and local area weighted data to obtain clinical risk prediction data for:

[0039] Historical similar data is acquired based on the user's patient basic data, clinical examination data, and vital signs data to obtain historical similar data; local feature contribution is calculated based on historical similar data and local area weighted data to obtain local feature contribution data; visual attention visualization and structural attention visualization are performed based on the local feature sharing degree to obtain visual attention visualization data and structural attention visualization data respectively; similar case label attribution analysis is performed on historical similar data based on the visual attention visualization data and structural attention visualization data to obtain clinical risk prediction data.

[0040] The present invention enhances the interpretability of clinical risk prediction by introducing local feature contribution analysis and multimodal visualization based on historical similar samples. By obtaining similar historical cases from user basic data, clinical examination data, and vital signs, the model's reference accuracy for the current patient status is improved; feature contribution is calculated based on local area weighted data, which can identify the key factors that have the greatest impact on risk prediction; combined with the visualization output of visual attention and structural attention, it can not only intuitively display high-risk image areas and key indicators, but also assist doctors in understanding the focus of the model; further, through visualization-guided similar case label attribution analysis, the model's capabilities in predictive result traceability, explanation credibility, and decision support are improved, thereby enhancing the system's clinical applicability and doctors' willingness to adopt it.

[0041] The beneficial effects of the present invention are: through modular design, it realizes the whole process of intelligent clinical risk assessment of vascular embolism from multi-source data collection, feature extraction, fusion modeling to interpretability enhancement, and has high technical integration and clinical practicality. By collecting the patient's basic information, laboratory tests, vital signs and medical imaging data, the system realizes the comprehensive integration of structured and unstructured data, and can build a more accurate multimodal patient status representation; the self-attention mechanism is used to perform local weighted processing on the fused data, which strengthens the model's ability to identify key areas and important indicators; the introduction of historical similar case comparison and local feature contribution analysis, combined with visual and structural attention visualization methods, significantly improves the transparency and interpretability of the prediction results; the output clinical risk prediction results are more valuable for reference, which helps doctors to achieve earlier and more accurate risk intervention, improve clinical decision-making efficiency and patient prognosis management level. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0043] Figure 1 A module flow chart of a vascular embolism clinical risk prognosis assessment system based on multi-source data according to an embodiment is shown;

[0044] Figure 2 A flowchart of a patient multi-source data acquisition module according to an embodiment is shown;

[0045] Figure 3 A flow chart of a patient clinical data feature processing module according to an embodiment is shown;

[0046] Figure 4 A flowchart of a patient clinical multi-source data fusion module according to an embodiment is shown;

[0047] Figure 5 A flow chart of a patient clinical interpretability enhancement module according to an embodiment is shown. DETAILED DESCRIPTION

[0048] The following is a clear and complete description of the technical method of the present invention in conjunction with the accompanying drawings. It is obvious that the embodiments described are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts are within the scope of protection of the present invention.

[0049] Furthermore, the accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor and / or microcontroller approaches.

[0050] It should be understood that although the terms "first," "second," and the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the exemplary embodiments. The term "and / or" as used herein includes any and all combinations of one or more of the listed associated items.

[0051] See also Figures 1 to 5 The present application provides a clinical risk prognosis assessment system for vascular embolism based on multi-source data, the system comprising:

[0052] The patient multi-source data acquisition module 10 is used to acquire patient basic data, clinical examination data, vital sign data and patient image data.

[0053] In an embodiment, the patient basic data acquisition includes age, gender, height, weight, basic medical history, medication history, etc., which are automatically acquired through the interface of the electronic health record system (EHR) and standardized in FHIR format. The clinical examination data acquisition includes blood routine examination, biochemical examination (such as D-dimer, blood lipid, blood glucose, etc.), coagulation function indicators (such as PT, APTT, INR), etc. Real-time acquisition is achieved by connecting the hospital LIS system. The vital sign data acquisition includes heart rate, blood pressure, body temperature, respiratory rate, blood oxygen saturation, etc. Timely acquisition is achieved by connecting the monitor device or wearable device (such as a smart wristband), and time stamp synchronization is performed at a minute level. The image data acquisition, such as CT, MRI, DSA images, etc., uses the DICOM protocol to call from the PACS system, and is matched by time in combination with the hospital image database index.

[0054] The patient clinical data feature processing module 20 is used to extract image features from the patient image data to obtain image feature data, and to structure features from the patient basic data, clinical examination data and vital sign data to obtain patient clinical structure feature data.

[0055] In an embodiment, the CT / MRI image is segmented using the U-Net architecture for blood vessel structure; the spatial distribution features such as embolism position, shape, density, etc. are extracted based on the ResNet or 3D-CNN network; the image is windowed and enhanced (such as window width and window level adjustment) to ensure the stability of the enhanced features; and the output is an image feature tensor with consistent dimensions.

[0056] The clinical indicators are processed according to the standard normalization rules (such as Z-score standardization); the discrete features such as gender and basic diseases are encoded using one-hot encoding; the time series data (such as heart rate fluctuation) uses a sliding window to extract mean, variance, trend and other statistics; and all the processed data form a unified dimension structured feature vector.

[0057] The patient clinical multi-source data fusion module 30 is used to perform multi-modal fusion according to the image feature data and the patient clinical structure feature data to obtain patient multi-modal data, and to perform self-attention processing according to the patient multi-modal data to obtain local region weighted data.

[0058] In one embodiment, the image feature tensor and structured feature vector are aligned using a cross-attention mechanism; a multi-layer Transformer architecture is used to achieve inter-modal interaction; residual connections and layer normalization are performed on the features to prevent gradient vanishing; and a fused feature vector of uniform length is output. Multi-head self-attention is used to identify important local regions; position encoding is introduced to distinguish features from different sources; and weighted data for local regions is generated to represent the vascular locations and related clinical indicators that the model focuses on.

[0059] The patient clinical interpretability enhancement module 40 is used to obtain historical similar data based on the user's patient basic data, clinical examination data and vital signs data to obtain historical similar data; and to enhance interpretability based on historical similar data and local area weighted data to obtain clinical risk prediction data.

[0060] In one embodiment, a k-nearest neighbor (KNN) or Faiss-based vector search engine is used to match historical patient data with the smallest distance in the structural feature space. Similarity is calculated based on weighted Euclidean distance, with a weighting coefficient of 0.6 for imaging features and 0.4 for structural features. The top-N (e.g., five) similar cases are retrieved, including their diagnostic results and prognoses. The model's focus areas are visualized using attention weights. SHAP values ​​are used to assess the contribution of each structural feature to the prediction results. The clinical pathways of similar historical cases are integrated with the current patient's clinical pathways, and risk prediction results (e.g., mortality rate, complication probability) and corresponding clinical interpretation reports are output.

[0061] In an embodiment, according to the basic information, clinical examination data and vital sign data of the current patient, the system constructs a structured feature vector (for example, length 128), and combines the generated local region weighted vector (for example, length 512) to form a unified query vector through vector splicing. The vector is input into the KNN or FAISS vector retrieval engine to perform similarity comparison with the feature vectors in the historical patient case database. The top-N (for example, N = 5) most similar historical patient data are selected, including their image data, structure data, diagnosis label (such as whether it is a pulmonary embolism, cerebral infarction, etc.) and prognosis outcome (such as death, recovery, complications), to form a set of historical similar data. To enhance the explainability of the risk prediction result of the current patient, the system uses the SHAP (SHapley Additive exPlanations) method to evaluate the contribution of the structure feature to the model output. The system uses models such as XGBoost, LightGBM, MLP, etc. for risk prediction (output mortality rate, complication probability, prediction of specific clinical index changes, etc.). Use the SHAP library (such as the shap module in Python) to interpret the trained model, only keep the first 128 dimensions (i.e. the structure feature part), and arrange them in descending order according to the SHAP value size; display format example (Top-5):

[0062] The structure feature screening result table 1 is as follows:

[0063]

[0064]

[0065] Use the multi-head attention weight matrix generated during training or the Grad-CAM (Gradient-weighted Class Activation Mapping) technique to perform reverse propagation calculation on the input image; generate a heat map on the original CT / MRI image, and the red area represents the area that the model focuses on (such as the main trunk of the pulmonary artery), which is used to assist the doctor in judging the reason for the model's risk conclusion. Map the SHAP value to the structure field and draw a bar chart or matrix heat map; explanatory text can be superimposed to assist the doctor in quickly identifying the main risk factors.

[0066] Preferably, the patient basic data, clinical examination data, vital sign data and patient image data are used for:

[0067] 11. Obtain patient basic data from an electronic medical record system, wherein the patient basic data includes age, gender, height, weight, and vascular embolism history;

[0068] In an embodiment, data is pulled from the electronic medical record system (EMR) using HL7 or FHIR standard protocol; field definition and mapping rules: age is the Date_of_Birth field minus current system time, and the integer part is kept to get the current age; gender is the field value converted to 0 / 1 binary encoding if it is 'M' / 'F' or'male' / 'female'; height and weight are the Height_cm (height in cm) and Weight_kg (weight in kg) fields, respectively, with one decimal place kept; embolic disease history: extract the keywords from the medical record "past history" part, such as "cerebral infarction", "pulmonary embolism", "DVT", etc., and generate binary disease history labels using rule-based keyword matching + word embedding similarity judgment.

[0069] 12. Obtain clinical examination data from a clinical laboratory electronic system;

[0070] In an embodiment, LIS (laboratory information system) is connected; index selection rules: select the latest data within the last 7 days; if there are multiple samples for the same test, the value order is: abnormal value > latest value; obtain clinical monitoring data through typical test items, such as coagulation function: D-Dimer, Fibrinogen, PT, APTT, INR; inflammatory response: CRP, WBC, Neutrophils; metabolic indicators: LDL, HDL, blood sugar, etc. The collected data is unified in units (such as g / L→g / dL), and the values are normalized (Z-score or min-max scaling); if the value is outside the reference interval, it is labeled for model learning.

[0071] 13. Obtain vital sign data from a vital sign monitoring device or a nursing record system;

[0072] In an embodiment, data is collected from a vital sign monitoring device (such as a multi-parameter monitor), such as heart rate, blood pressure, SpO2, respiratory rate, body temperature; from a nursing information system (NIS), such as manually recorded body temperature, consciousness state, skin temperature, etc.; with a sampling period of minutes, the mean, maximum, and fluctuation range within the past 24 hours are saved; missing data is linearly interpolated or filled in units of hours. The fluctuation trend (such as blood pressure slope change) is extracted by a sliding window; static values and dynamic changes (such as ΔSpO2) are combined to input the model; non-numeric features (such as consciousness state) are mapped to labels (awake / drowsy / coma→0 / 1 / 2).

[0073] 14. Obtain patient image data through a medical imaging system.

[0074] In one embodiment, the system is connected to a PACS system, adopts the DICOM protocol, and pulls images based on the patient ID and examination time range. It can be configured to support CT, MRI, DSA and other modes, with CT being prioritized by default. The resolution is standardized (e.g., unified to 512×512); the image window width and window position are adjusted to the vascular mode (e.g., W=100, L=40 for cerebrovascular CT); multiple slice sequences are synthesized into a 3D volume tensor, preserving spatial hierarchical information; image acquisition time, scanning site, and scanning protocol are checked; and the images are compared with EMR examination records to ensure that the images are consistent with the patient data.

[0075] Preferably, the image feature extraction is performed based on the patient image data to obtain image feature data; and the structural characterization is performed based on the patient basic data, clinical examination data, and vital signs data to obtain the patient clinical structural feature data, which is used for:

[0076] 21. Perform image registration based on the patient's image data to obtain image registration data;

[0077] In one embodiment, image data from multiple time points, different modalities (e.g., CT and MRI), or multiple slices are aligned to facilitate analysis. Image similarity-based registration methods (e.g., Mutual Information (NCC)) or deep learning methods (e.g., VoxelMorph) are used. For homomodal sequences (e.g., CT multi-slice), rigid registration is used; for heteromodal sequences (e.g., CT + MRI), non-rigid registration is used. The registered image data is output, maintaining spatial coordinate consistency.

[0078] If images A and B are well registered, the grayscale values ​​at their corresponding locations have a small joint distribution entropy and a large mutual information. Let image A be the fixed image and image B be the image to be registered. Define the grayscale distribution probability P(a) of image A, the grayscale distribution probability P(b) of image B, the joint probability distribution P(a,b), and MI(A,B) as the mutual information between the two. Normalize the grayscale of the fixed image A and the moving image B (e.g., 0 to 1 range), and downsample the image to accelerate (e.g., 256×256). Initialize the rigid or non-rigid transformation matrix (e.g., rotation, translation, scaling parameters). Use the current transformation parameters to affine / nonlinearly transform image B into B. ′ For each paired pixel A[i], B[i], the two-dimensional histogram H(A,B) is statistically converted into a joint probability: P(a,b)=H(a,b) / N. Calculate the edge probability: p(a)=∑ b p(a,b),p(b)=∑ aBased on the above distribution, the mutual information between images A and B is calculated. The system uses optimization algorithms (such as gradient descent, Powell's method, and L-BFGS) to iteratively update the transformation parameters, with maximizing MI(A,B) as the objective function. The optimization terminates when the change in mutual information, ΔMI, between two consecutive iterations is less than a preset convergence threshold, ε. The output includes the optimal spatial transformation parameters and the registered image.

[0079] 22. Perform key area detection based on image registration data to obtain key area detection data;

[0080] In one embodiment, a deep object detection network such as YOLOv5 and FasterR-CNN is used to locate key areas where vascular embolism occurs (such as cerebral vascular trunks, pulmonary artery trunks, lower limb veins, etc.) using image registration data; the deep object detection network is trained by constructing a training set of manually annotated vascular abnormality area box maps; intermediate network training is performed using YOLOv5 and FasterR-CNN algorithms; the output format is bounding box coordinates (x, y, w, h) or 3D area mask; those with confidence levels below 0.5 are discarded, and the remaining areas are passed to the next step.

[0081] 23. Segment the image registration data according to the key area detection data to obtain image segmentation data;

[0082] In one embodiment, the vascular structure and embolic lesions in the key area are finely segmented. An image segmentation network such as U-Net or nnU-Net is used; a mixed loss function of Dice Loss and Cross Entropy is adopted; the segmentation objects include arterial / venous pathways, plaques, and embolic areas; the output is a binary mask (or one-hot encoding) with the same size as the original. Figure 1 To.

[0083] 24. Perform depth feature extraction based on the image segmentation data to obtain image depth feature data;

[0084] In one embodiment, a ResNet, EfficientNet, or 3D-CNN network is used to extract features from the segmented area; the extracted content includes perfusion distribution (CT value distribution, density change); embolism size (number of voxels, edge complexity); surrounding tissue changes (neighborhood contrast, texture gradient); and the output is a fixed-length feature vector (e.g., 1×512).

[0085] 25. Perform image feature aggregation on the image depth feature data to obtain image feature aggregation data;

[0086] In one embodiment, deep features from multiple slices, multiple channels, or multiple regions are fused. Feature pooling is a hybrid of average pooling (AvgPool) and maximum pooling (MaxPool). Time series slices are aggregated using LSTM. The output is image feature aggregation data of a uniform dimension (e.g., 1×1024) for multimodal fusion.

[0087] 26. Perform field screening based on patient basic data, clinical examination data, and vital signs data to obtain field screening data;

[0088] In one embodiment, structural data fields highly correlated with the prognosis of vascular embolism were selected. Clinical expert rules and logistic regression with L1 regularization penalty were used for initial screening. Required fields included age, gender, BMI, D-Dimer, INR, heart rate, SpO2, and medical history. Fields with a missing rate >40% and an IV value (information value) <0.02 were deleted. The output was a list of several standard fields for downstream modeling.

[0089] 27. Perform numerical processing on the data filtered by the field to obtain the numerical data of the field;

[0090] In one embodiment, categorical fields are converted to numeric values ​​acceptable to the model. Categorical fields such as gender and consciousness state are converted to one-hot encoding (each value is generated in a separate column, with the current position being 1 and the rest being 0); ordinal fields such as graded scores (e.g., NYHA cardiac function class) are converted to ordinal encoding (mapping the ordinal grades to numeric labels); and continuous variables such as blood pressure and body temperature are converted to Z-score normalization (subtracting the mean from each field value and dividing by the standard deviation). The output is a structured, purely numeric feature table (DataFrame structure).

[0091] 28. Mark the missing data of the field numerical data to obtain the missing marked data;

[0092] In one embodiment, missing values ​​are explicitly modeled rather than simply deleted or filled. A new column with a "missing" label (0 / 1) is added to each field. Continuous values ​​are filled with the mean, while a "missing flag" column is retained. The model automatically learns the impact of missing values ​​on risk. For example, missing items in numeric fields are filled with the mean calculated from the entire sample (or training set). The output is extended feature data containing the original value and the missing flag.

[0093] In one embodiment, missing data are interpolated using the patient's historical data, or marked as missing.

[0094] 29. Construct feature vectors based on missing labeled data to obtain the patient's clinical structural feature data.

[0095] In one embodiment, all structured field data is integrated to construct a unified input vector. All fields are concatenated into a one-dimensional vector (e.g., 1×128 dimensions) in a preset order. A field mapping table is used to ensure a one-to-one correspondence between fields and positions. If the model supports feature embedding, the field ID index can also be output. The output vector is standardized and directly input into a deep learning network or fusion module.

[0096] Preferably, multimodal fusion is performed on the image feature data and the patient's clinical structural feature data to obtain the patient's multimodal data; self-attention processing is performed on the patient's multimodal data to obtain local area weighted data, which is used for:

[0097] 31. Perform feature dimension alignment based on the image feature data and the patient's clinical structure feature data to obtain feature dimension alignment data;

[0098] In one embodiment, the dimensions of image features (e.g., 1×1024) and structural features (e.g., 1×128) are inconsistent and cannot be directly concatenated or fused. Instead, a unified mapping is performed. A fully connected layer is used to reduce the image features to the target dimension, such as 1×256. A dimensionality-raising network (e.g., using BatchNorm, Linear, and ReLU) is used to map the structural features to 1×256. The lengths of the two vectors are kept consistent to facilitate channel concatenation. The output is feature dimension-aligned data (1×256×2, concatenated).

[0099] 32. Perform dual-channel gating processing on the data according to the feature dimension alignment to obtain multimodal data of the first patient;

[0100] In one embodiment, after alignment of multimodal features, the influence of each modality in different tasks or stages is controlled. A dual-channel gating network is constructed: the input is image features and structural features; each channel is input into the Sigmoid gating layer: G img =sigmoid(W img *F img +b img ), G img is the importance weight (gate value) of each dimension feature of the image channel, sigmoid is the Sigmoid activation function, which is used to compress the gate value to the [0,1] interval, W img is the linear transformation weight matrix of the image channel, dimension 256×256, F img is the feature vector of the image modality, dimension 1×256, b img G is the bias vector of the image channel, dimension 1×256. struct =sigmoid(W struct *F struct +b struct ), G structis the structural channel gating weight, dimension 1×256, sigmoid is the Sigmoid activation function, which is used to compress the gate value to the [0,1] interval, W struct is the linear weight matrix of the structure channel, dimension 256×256, F struct is the structural modal eigenvector, dimension 1×256, b struct is the structural channel bias vector, dimension 1×256; the output feature is: F fused =G img ⊙F img +G struct ⊙F struct , F fused The fused feature vector and the feature dimension alignment data are packaged to represent the multimodal data of the first patient. ⊙ represents element-by-element multiplication. The multimodal data of the first patient (the fused feature vector with modality adaptation) is output. If a modality is not important in the current task, its gate value approaches 0, otherwise it is 1. A certain input is: After calculating the gate weight, we get: G img =[0.9,0.2,...,0.4], G struct =[0.1,0.8,...,0.6], the fusion result of one dimension (the first dimension) is: F fused =0.75.

[0101] 33. Perform contextual multi-head self-attention calculation based on the first patient's multimodal data to obtain first multimodal weighted data;

[0102] In one embodiment, the input is reshaped / sliced ​​to form a pseudo-sequence representation. Each segment can be regarded as a "context unit". Based on each segment, a standard Transformer multi-head self-attention structure is used: Query (Q), Key (K), Value (V) matrix is ​​constructed; self-attention is performed: Atten (Q, K, V) = Softmax (QK T / √d k )*V, Atten(Q,K,V), Softmax is normalized by row to form the attention weight matrix, QK T To calculate the similarity of each position to other positions, Q is the Query matrix, which represents the query vector, K is the Key matrix, which represents the content vector, and d k is the scaling factor of each attention dimension, V is the Value matrix, which represents the actual carrier of information transmission; multiple heads are parallel, the feature dimension of each head is 64, and the total output dimension is consistent with the input; residual connection and LayerNorm are added; the first multimodal weighted data is output.

[0103] 34. Perform position mask weighting on the first multimodal weighted data to obtain first local area weighted data;

[0104] In one embodiment, the importance of different positions / regions is controlled to guide the model to focus on local structures. A position mask matrix is ​​constructed, based on the importance of image regions or structure fields; the mask is multiplied into the attention weight matrix: A masked =Atten(Q,K,V)⊙M pos , A masked is the first local area weighted data, Atten(Q,K,V) is the first multimodal weighted data, M pos It is a position mask matrix; the mask source is, for example, from the medical key area box for images; for structural data, it comes from expert knowledge (such as D-Dimer has a higher weight); the output is the first local area weighted data.

[0105] 35. Perform multi-layer perceptron processing on the data according to the feature dimension alignment to obtain the multimodal data of the second patient;

[0106] In one embodiment, the potential mode of modal interaction is mined through deep nonlinear mapping. A standard three-layer MLP is constructed: input → Linear (256 → 128) → ReLU → Dropout; Linear (128 → 64) → ReLU; Linear (64 → 256) → output; BatchNorm is added to all layers to accelerate convergence; and the second patient multimodal data is output (another fusion path). The first layer is transformed into wide compression, input data, dimension compression, activation calculation, regularization, and the first layer output is obtained; the second layer is transformed into feature compression, input data, weight calculation, activation calculation, and the second layer output is obtained; the third layer is transformed into dimensionality increase back to the original dimension, input data, weight parameter calculation, and direct output.

[0107] 36. Perform masked multi-head self-attention calculation on the second patient's multimodal data to obtain second multimodal weighted data;

[0108] In one embodiment, the results of the second patient's multimodal data after processing by the multi-layer perceptron module include, in some cases, some fields from the original missing data. The system avoids focusing attention on these missing fields, and therefore introduces a missing-aware mask matrix. In the aforementioned module, for the structural feature processing stage, a corresponding missing flag field (0 / 1) is generated for each field, and the missing vector is expanded into a two-dimensional mask matrix; a missing-aware mask is constructed, and the missing field is set to 0 and the rest are 1; Masked Multi-Head Attention is used, which is consistent with traditional Attention, but the QK weight matrix is ​​multiplied by the mask; invalid fields can be skipped to avoid attention misleading; and the second multimodal weighted data is output.

[0109] 37. Perform maximum pooling on the second multimodal weighted data to obtain second local area weighted data;

[0110] In one embodiment, global maximum pooling is performed on the second multimodal weighted data, that is, on each feature dimension, the maximum activation value of the dimension is selected from all positions / segments to form a compressed representation; the second local area weighted data is output as the maximum activation response of each dimension, which is used to represent the strongest feature.

[0111] 38. Perform residual connection based on the first local area weighted data and the second local area weighted data to obtain local area weighted data.

[0112] In one embodiment, two parallel attention paths model multimodal features from different perspectives. output =First Local +Second Local , (element-wise addition), F output For local area weighted data, First Local is the weighted data of the first local area, Second Local is the weighted data of the second local area, + is element-by-element addition by dimension; or, it is processed through the weighted residual structure: F output =α·First Local +(1-α)·Second Local , where α is a learnable parameter obtained by fitting historical data; the output is local area weighted data for interpretability enhancement and clinical risk prediction.

[0113] Preferably, the dual-channel gating processing is performed on the data aligned according to the feature dimensions to obtain the multimodal data of the first patient, which is used for:

[0114] Align the data according to the feature dimension to perform mutual information perception and obtain mutual information perception data;

[0115] In one embodiment, the strength of the mutual information between image features and structural features is evaluated as a reference for generating gating weights. Using the MINE method, the image feature vector and the structural feature vector (both 1×256) are input; an estimation network is trained (the network includes two fully connected layers; the first layer dimension is mapped from 512 to 256, and the second layer dimension is mapped from 256 to 1; the ReLU activation function is used after each layer; the input is the vector of the concatenation of image features and structural features (dimension is 1×512); the network is trained using the Adam optimizer with a learning rate set to 1e-4; the mini-batch method is used for estimation with a batch size of 32; and the following dual mutual information estimation is maximized: in is the estimated mutual information value, E is the mathematical expectation operation, which is approximated by taking the average within the batch, Tθ is the score of the input pairs using the neural network to reflect their correlation, and F img is the original image feature vector, with a dimension of (1×256), F struct is the original structural feature vector with a dimension of (1×256), log is the logarithmic operation, which is used to stabilize the scale of the estimated term, e is the base of the natural exponential, Tθ(F img ′,F struct ) is to score the input pairs using a neural network to reflect their relevance, F img ′ is the image feature after the pairing is destroyed and used for comparative learning. After the training is completed, a perception value of the mutual information strength is obtained. This value is used as the weight prior or gate activation factor for channel selection. The mutual information perception data is output with a dimension of (1×1) or expanded to a broadcast vector (1×256).

[0116] Generate gated channels based on mutual information perception data and feature dimension alignment data to obtain gated channel data, where the gated channel data includes structure gated channel data and image gated channel data;

[0117] In one embodiment, based on the mutual information perception results, weight modulation channels are generated for the image channel and the structure channel respectively to achieve modal weighted harmonization. img and F struct After dimension alignment, they are spliced ​​into F fused is the fused feature vector, is a one-dimensional 512-dimensional vector (single-channel splicing feature). Perform channel separation and convert F fused Split into two channels, such as image channel Structural Channel Gating weight generation (per channel): for F img Use a fully connected network to generate the gating vector G img :G img =sigmoid(W1·F img +b1+α*MI score ), G img is the image gated channel weight vector, sigmoid is the Sigmoid activation function, W1 is the image channel weight matrix, F img is the image gated channel data, b1 is the image channel bias vector, α is the mutual information control coefficient (image), MI score is the mutual information perception value. struct Generate G using a similar structure struct :G struct=sigmoid(W2·F struct +b2+β*MI score ), G struct is the structural gated channel weight vector, sigmoid is the Sigmoid activation function, W2 is the structural channel weight matrix, F struct is the structure gated channel data, b2 is the structure channel bias term, β is the mutual information control coefficient (structure), MI score is the mutual information perception value, where α and β are adjustable hyperparameters that regulate the weight influence of mutual information; the output gated channel data includes structural gated channel data: Image gated channel data:

[0118] Perform dual-channel reconstruction and output on the feature dimension alignment data according to the gated channel data to obtain feature reconstructed data;

[0119] In one embodiment, the original features of the modalities are re-weighted and combined using gated channels to emphasize high-information channels and suppress redundant modal content. Channel product reconstruction is performed, and image reconstruction is performed: R img =F img ⊙G img , R img is the reconstructed image feature vector, F img is the image feature vector, G img For image gating weights, structure reconstruction: R struct =F struct ⊙G struct , R struct is the reconstructed structural feature vector, F struct is the structural eigenvector, ⊙ is the element-by-element multiplication, G struct is the structural gating weight, channel fusion: splicing the two to get R fused =[R img ,R struct ], the dimension is still 1×512; or use linear fusion, R fused =W f concat(R img ,R struct )+b f , R fused is the fusion feature vector, W f is a learnable fusion weight matrix, concat is a concatenation function that concatenates two 1×256 vectors into 1×512, and R img is the reconstructed image feature vector, R struct is the reconstructed structural feature vector, b f is the fusion bias term. The output is the feature reconstruction data with the new fusion state after gate adjustment.

[0120] Perform differential residual connection based on the feature-reconstructed data to obtain the multimodal data of the first patient.

[0121] In one embodiment, the difference information between the initial input and the gated modulation output is introduced to strengthen the inter-modal residual signal and improve the model's ability to perceive modal differences. Calculate the difference vector: ΔF = R fused -F fused , F fused is the initial splicing input, R fused Reconstruct output for gate control; residual connection method: residual connection after adding activation function: F output =LayerNorm(ReLU(ΔF+R fused )), F output is the multimodal data of the first patient, LayerNorm is the layer normalization function, ReLU is the activation function, ΔF is the difference vector, which represents the residual signal before and after gating, F fused The original splicing features are used as the principal components of the residual connection to retain the original modal information; the first patient multimodal data (including modal features, gated fusion information and difference correction capabilities) are output.

[0122] Preferably, the contextual multi-head self-attention calculation is performed based on the first patient multimodal data to obtain first multimodal weighted data, which is used for:

[0123] Performing context expansion based on the multimodal data of the first patient to obtain multimodal context expansion data;

[0124] In one embodiment, the multimodal data of the first patient is expanded into multiple context windows for parallel attention mechanism modeling, that is, multi-perspective understanding is introduced. For example, 1×512; to build the context, generate K context perspectives: Use linear transformation to generate context sub-vectors: Context k =Linear k (F multi )for k∈[1,...,K],CDontext k is the kth context sub-vector, the sub-representation after context expansion, Linear k is the full connection calculation corresponding to the k-th context sub-vector, F multi is the first patient's multimodal fusion feature vector, k is the context subvector order item, K is the number of context perspectives, and each Linear k Independently learn the projection matrix W k ; Merged into For example, 8×64 (8 attention heads, 64 dimensions per head); output multimodal context extension data.

[0125] Multi-head self-attention weights are calculated based on the multimodal context extension data to obtain multi-head self-attention weight data;

[0126] In one embodiment, the query, key, and value in each context perspective (each attention head) are calculated, and the weight matrix of each head is obtained. For each context vector, the following is performed: k-th query vector Q k =W Q k·Context k , the kth Key vector K k =W K k·Context k , the kth Value vector V k =W V k·Context k All W Q k (kth Query projection matrix), W K k (kth Key projection matrix), W V k (kth Value projection matrix), can be shared or independent; calculate attention weight: Attn k =Softmax(Q k ·K k T / √d k ), Attn k is the attention weight coefficient of the kth attention head, a scalar (scalar multiplied by V k ), Softmax is a normalization function that maps the score to the probability distribution interval, Q k is the kth query vector, K k is the kth key vector, T is the transpose symbol, d k The dimension of the Key vector, used as a scaling factor to prevent the inner product result from being too large.

[0127] Perform multi-head splicing output on the multimodal context extension data according to the multi-head self-attention weight data to obtain multi-head splicing data;

[0128] In one embodiment, the output values ​​(Weighted V) of multiple attention heads are concatenated into a complete semantic vector to represent multi-view information. k =Attn k ·V k ,Output k is the weighted output result of the kth attention head, Attn k is the attention weight (scalar or vector) of the kth attention head, V k is the value vector of the kth attention head. Concatenation: As 1x512, F concat is the fusion semantic vector after multi-head splicing, which integrates all attention outputs, Concat is the splicing operation, which splices the outputs of each head along the feature dimension, Output1 is the output of the first attention head, Output K is the output of the Kth attention head, is the real number field where the vector is located, K is the number of attention heads (i.e., the number of context perspectives), d k is the feature dimension of each head (e.g., 64). Select residual connection and normalization: is the output result, which has a smoother semantic expression capability, is the main position feature representation input, LayerNorm is the layer normalization operation, which normalizes the fusion result in dimension to improve training stability and generalization ability, F concat is the multi-head splicing output, the fusion semantic result of the current stage, F multi is the first patient multi-modal data of the original input, which is used as a reference for the residual path.

[0129] According to the multi-head splicing data, the main position feature output is obtained, and the first multi-modal weighted data is obtained.

[0130] In an embodiment, from the spliced multi-perspective information, the dominant feature position is focused and the first multi-modal weighted output is generated. The main position selection weight vector is constructed, where is the dimension data of the weight vector, which is used to locate the key position P = sigmoid (Linear (F concat )), P is the main position selection weight vector (the significance of each feature dimension), sigmoid is the activation function, Linear is the linear transformation layer, i.e., by multiplying the preset weight with F concat and adding a preset bias term to obtain F concat is the multi-head splicing feature vector; or the main position fragment is selected using the Top-K weight; and the original splicing result is weighted: F primary = F concat ⊙ P, F primary is the weighted main position feature, F concat is the multi-head splicing feature vector, ⊙ is the bitwise multiplication, P is the main position selection weight vector (the significance of each feature dimension). The dimension is compressed through the linear layer: F output1 is the first multi-modal weighted data after linear mapping, Linear is the linear transformation layer, F primary is the weighted main position feature, is the feature dimension.

[0131] In one embodiment, in order to extract dominant information from the fused multi-head attention features, the system performs weighted processing on the multi-head splicing data based on the main position feature selection mechanism and outputs the first multimodal weighted data. Specifically, the splicing feature vector output by the multi-head attention module is used as input, and a set of trainable linear mapping parameters are used to generate the importance weights on the feature dimensions. In order to ensure that the weight value is stable and has normalization characteristics, the system inputs the linear mapping result into the Sigmoid function to obtain a main position weight vector between 0 and 1, which is used to represent the importance of each splicing dimension under the current sample. The system performs element-by-element weighting operations on the original multi-head splicing feature vector and the main position weight vector to highlight the representative feature positions and suppress the interference of non-dominant areas. The weighted result is a set of feature representations strengthened by the main position. In order to unify the dimensions of the multimodal feature vector, the system further inputs the weighted result into the linear compression module, compresses its dimensions to the preset output feature length through linear transformation, and obtains the first multimodal weighted data.

[0132] Preferably, the first multimodal weighted data is weighted using a position mask to obtain first local area weighted data, which is used for:

[0133] Performing credibility calculation based on the first multimodal weighted data to obtain credibility data;

[0134] In one embodiment, a credibility score is assigned to each dimension (or region) feature to identify which features are more reliable. Input: First multimodal weighted data For example, 1×256; build a shallow MLP module as a credibility predictor: C is, Sgimoid is the activation function, Linear is the linear transformation layer, the second linear transformation layer projects the hidden space back to the space with the same dimension as the original feature to obtain the credibility score of each dimension, ReLU is the activation function, Linear is the linear transformation layer, the first linear transformation layer extracts the high-order combination relationship of the input features and captures the local feature pattern. The first layer extracts the feature pattern; the second layer compresses it into the credibility dimension; Sigmoid output range [0,1], indicating the confidence score; the output is the credibility data Each dimension corresponds to a confidence score.

[0135] Constructing a position mask according to the credibility data to obtain position mask data;

[0136] In one embodiment, a mask matrix is ​​constructed to mark the trusted location area and guide local weighting. The credibility data C is converted into a mask: threshold truncation (such as 0.5): M = where (C ≥ 0.5, 1, 0), a credibility threshold (such as 0.5) is set. When the credibility score of a feature dimension is higher than the threshold, the dimension is considered to be a "trusted location" and is represented by 1; otherwise, the dimension is considered untrustworthy and is represented by 0. Method 2: Top-K mask selection, take the top K dimensions with the highest confidence (such as K = 32) and set them to 1, and the rest are set to 0; the mask dimensions are consistent with the original data Output position mask data, identifying the trusted area.

[0137] Performing weighted calculation on the first multimodal weighted data according to the position mask data to obtain multimodal position mask data;

[0138] In one embodiment, only the reliable positions are significantly weighted to suppress the interference of low-confidence features. masked =F multi ⊙M+(1-M)⊙γ,F masked is the output feature (after masking), F multi is the original multimodal weighted feature, ⊙ is, M is the position mask matrix (hard mask), γ is the default value filling item, and is the default filling value of the low confidence area (can be 0 or the average value).

[0139] In one embodiment, instead of using a hard 0 / 1 mask, the credibility score C∈[0,1]^d calculated in the previous stage is directly used as a weighting factor, F masked =F multi ⊙M, F masked is the weighted feature result, F multi is the original multimodal feature, M is the position mask matrix (hard mask). The confidence value itself is used as a continuous weighting factor; the multimodal position mask data is output

[0140] A local activation calculation is performed according to the multimodal position mask data to obtain first local region weighted data.

[0141] In one embodiment, a reinforcement operation is performed on the high confidence region so that it generates a larger activation response in the final model. A local amplification process is performed, A = ReLU (F masked *α), A is the local enhancement response result, ReLU is a non-negative activation function, filtering low activation areas, F masked is the weighted multimodal position mask data, α is the activation amplification factor (such as 1.5 or 2.0); after activation, a sparse response map is formed, retaining only reliable strong features. The first local area weighted data F1 = A + F multi .

[0142] In an embodiment, attention-weighted remapping is performed A is the activated feature map, Softmax is a normalization function, and outputs the relative attention weight of each position, W att is the position attention weight learning parameter, is the transpose matrix of the feature dimension, used to generate the position score, F masked is the feature data after multi-modal position mask weighting. The first local area weighted data F1=A+F multi .

[0143] Preferably, the multi-layer perception processing is performed on the feature dimension alignment data to obtain second patient multi-modal data, which is used for:

[0144] Layer normalization is performed on the feature dimension alignment data to obtain feature layer normalization data;

[0145] In an embodiment, the distribution difference of different channels / fields is eliminated to accelerate model convergence and improve feature stability. The input feature dimension alignment data For example, 1x256; a standard LayerNorm operation is used: μ=mean(F aligh ), σ 2 =var(F align ), F norm =(F align -μ) / sqrt(σ 2 +ε), μ is the mean value of all dimensions of the feature vector, mean is the mean operation, F align is the aligned feature input vector, σ is the standard deviation, var is the variance operation, F norm is the normalized feature vector, sqrt is the square root function, and ε is a small constant to prevent division by zero (such as 1e-6).

[0146] Noise perturbation enhancement is performed on the feature layer normalization data to obtain feature enhancement data;

[0147] In an embodiment, the robustness of the model to input perturbations is enhanced, the generalization ability is improved, and overfitting is prevented. A controllable noise term is added to the normalized feature vector, such as Gaussian perturbation: Noise N(0, σ 2 ), F aug =F norm +λ*Noise, Noise is a noise perturbation vector, N(0, σ 2 ) is a zero-mean Gaussian distribution, σ is the Gaussian standard deviation, F aug is the feature enhancement data, F norm is the feature layer normalization data, and λ is the noise intensity coefficient. Wherein σ 2= 0.01, λ is the perturbation intensity coefficient, set to 0.1. After adding, ReLU processing is performed to remove the offset caused by negative perturbations;

[0148] In one embodiment, several dimensions are randomly set to zero: F aug [i]=0,with probability p,F aug [i] is i, with probability p. After adding, ReLU processing is performed to remove the offset caused by negative disturbance;

[0149] According to the feature enhancement data, nonlinear multi-layer perceptron stacking is performed to obtain asymmetric network structure data;

[0150] In one embodiment, the enhanced features are mapped to a high-order nonlinear space to extract deep semantic expressions. An asymmetric MLP network (non-equidimensional structure) is designed: L1: Linear (d→4d) → GELU → Dropout, L2: Linear (4d→2d) → ReLU, L3: Linear (2d→d) → ReLU, L1 is a width expansion layer (feature diversity), Linear is a linear transformation, d is, GELU is a Gaussian error linear unit, Dropout is a random deactivation, L2 is an intermediate compression layer (redundancy removal), ReLU is an activation function to remove negative values, and L3 is a regression dimension (restore the original structure). LayerNorm and Residual: F are used between each layer. out =LayerNorm(F in +L(F in )), F out For asymmetric network structure data, LayerNorm is a layer normalization operation to eliminate the difference between dimensions and stabilize training. in is the current input feature vector, L(F in ) is the output after passing through the current MLP sublayer (e.g., L1, L2, L3). Asymmetry reflects the wide expansion (feature diversity) of the first layer; rapid compression (redundancy removal) of subsequent layers; and the output of asymmetric network structure data.

[0151] Perform multi-scale residual cross-connection based on asymmetric network structure data to obtain feature cross-layer connection data;

[0152] In one embodiment, the feature abstract representations at different scales are integrated, and the local, mid-scale and full-scale feature expressions are extracted through three branch paths, and they are fused through residual cross-connection, so as to construct a cross-layer connection feature with multi-layer perception capabilities, thereby improving the expressiveness and robustness of the model. Cross-scale connections are established between different perception layers so that the model retains low-level abstract capabilities and obtains high-level abstract expressions. Construct a three-branch cross-channel connection structure (corresponding to the three routes of the nonlinear multi-layer perceptron stacking in the previous steps), F1 = MLP1 (F_aug) (local low-dimensional processing), F2 = MLP2 (F_aug) (mid-scale), F3 = F_asym (full scale). F cross =F1+F2+F3, or, introduce attention weights: α1, α2, α3 = Softmax(W·Concat(F1,F2,F3)), that is, α1 is obtained by paying attention to F1, and the rest of the weights are similar. W is a linear transformation matrix that maps the concatenated features to three fusion weight values, F cross =α1·F1+α2·F2+α3·F3, and obtain the feature cross-layer connection data.

[0153] Dimensionality compression is performed on the feature cross-layer connection data to obtain the second patient multimodal data.

[0154] In one embodiment, the multi-scale features after cross-layer fusion are compressed into a vector of standard uniform length for processing by the downstream attention module. Linear projection layer and Dropout are used to compress the dimensions: The target dimension is d ′ =128, and obtain the second patient multimodal data.

[0155] Preferably, the masked multi-head self-attention calculation is performed on the second patient multimodal data to obtain second multimodal weighted data, which is used for:

[0156] Performing pseudo-sequence construction on the second patient's multimodal data to obtain pseudo-sequence data;

[0157] In one embodiment, the original multimodal features Construct a pseudo sequence representation that can be processed by multi-head attention and introduce a local context structure. Input the second patient's multimodal data (e.g. 1×256); construct a sliding window or cut into sub-blocks: construct it with a window size of w=16 as follows: Get a shape of L×d s Pseudo sequence data P seq (e.g. 16×16). Each sub-block is treated as a pseudo time step, allowing attention to model inter-feature context along the sequence direction.

[0158] Perform low-weight index shielding on pseudo-sequence data to obtain low-weight index shielded data;

[0159] In one embodiment, the sub-segments with small attention contributions are suppressed so that the attention calculation is more focused on the key feature areas, thereby improving the efficiency and performance of the model. Identify and temporarily block the feature segments with low importance to improve the concentration of the attention mechanism. Calculate the feature importance score for each vector of the pseudo sequence, and the mean attention score is as follows: Score[i] = mean(P seq [i]), Score[i] is the representative score of the i-th segment, mean is the mean function, the average value of all dimensions, P seq [i] is the feature vector of the i-th segment. Alternatively, the weight network can be trained: Score[i]=Linear(P seq [i]), Score[i] is the representative score of the i-th segment, Linear is a learnable scoring function, P seq [i] is the feature vector of the i-th segment. Set the masking threshold or masking ratio (such as masking the lowest 30% of the scores): Mask Low [i]=1if Score[i]≥T else0,Mask Low [i] is the keep / mask flag, which is input to the subsequent mask matrix. Score[i] is the importance score of the i-th segment, and T is the masking threshold (which can be set or trained). Alternatively, a Top-K retention strategy is used; the masked positions will be set to 0 in the mask matrix.

[0160] According to the preset indicator opening channel data, the low-weight indicator shielding data is opened to obtain the channel opening data;

[0161] In one embodiment, according to a preset strategy, mask channels are forcibly opened for certain important domain indicators, even if they have low weights. The opening strategy comes from a rule base or domain expert knowledge, such as the position Pos corresponding to features such as "D-Dimer" and "SpO2". open =[3,7,11], set the mask at these positions to 1: For1in Pos open :M low [i]=1,Pos open To specify the fragment position index that must be retained, M low [i] is the mask vector after the channel is opened (i.e., channel opening data). It is the reverse complement of the low-weight indicator masking to prevent key features from being mistakenly masked.

[0162] Perform position control based on channel opening data to obtain mask matrix data;

[0163] In one embodiment, a complete L×L mask matrix is ​​constructed for attention control, indicating which positions are allowed to interact. Mask generation rule: Mask[i][j]=1if M open [i]==1and M open [j]==1else0, Mask[i][j] is whether the attention connection between the i-th and j-th segments is feasible, M open [i] is the open flag of the ith pseudo time step, which comes from the channel open vector of the previous step, M open [j] is the open flag of the jth pseudo time step, which comes from the channel open vector of the previous step. The generated mask matrix Used to mask the weight matrix in the Attention mechanism (non-1 values ​​are forced to -∞); prevent low-quality features from affecting each other.

[0164] Multi-head self-attention calculation is performed on the second patient's multimodal data according to the mask matrix data to obtain second multimodal weighted data.

[0165] In one embodiment, attention is focused based on the mask matrix, and context modeling and interaction are performed on the retained feature areas. Perform standard long-term attention: Q = Linear Q (P seq ), K=Linear K (P seq ), V=Liinear V (P seq ), Q is the output query matrix, Linear Q To calculate the query weight matrix, P seq is the input pseudo-sequence feature matrix, K is the key query matrix, Linear K is the key weight matrix calculation, V is the output value matrix, Linear V Calculate the value weight matrix. Add the mask when calculating the attention score: A=Softmax(Attn), A is the attention score after activation calculation, Attn[i][j] is the attention score of the i-th segment to the j-th segment, M mask [i][j] is the mask matrix value, which controls whether the attention pair is effective. s is the feature dimension scaling factor to prevent the inner product value from being too large. Output the attention weighted result: O is the context-enhanced representation of the fragment, A is the attention score after activation calculation, and V is the output value matrix. is the feature dimension data, where L is the upper limit of j, representing the pseudo sequence length, and the attention weight is concentrated on M maskThe region with =1 suppresses the influence of low-quality features. Multi-head parallel, spliced into the second multi-modal weighted data: F out The context output after multi-head splicing is Concat, and the splicing function is is the feature dimension data.

[0166] Preferably, the historical similar data is obtained according to the user patient basic data, clinical examination data and vital sign data; and the clinical risk prediction data is obtained by performing explainability enhancement according to the historical similar data and the local region weighted data, and is used for:

[0167] 41. Obtain historical similar data according to user patient basic data, clinical examination data and vital sign data;

[0168] In an embodiment, input user patient basic data such as age, gender, medical history; clinical examination data such as blood / coagulation / metabolic indicators; vital sign data such as blood pressure, SpO2, heart rate, etc.; and process the above data into a structured feature vector F user is a fixed dimension vector corresponding to the user patient basic data, is the feature dimension representation, the method is the same as described above; Z-score or Min-Max Scaling, ensure the same scale with the historical case vector; use vector database (such as FAISS) to calculate the similarity, similarity calculation, such as using Euclidean distance, cosine similarity, weighted Euclidean (some fields are weighted); return Top-N similar samples, such as N=5, containing the label, examination record, image information, etc. of each sample, to obtain the historical similar data.

[0169] 42. Calculate the local feature contribution degree according to the historical similar data and the local region weighted data to obtain the local feature contribution degree data;

[0170] In an embodiment, the contribution degree of each modality feature to the prediction result is calculated by combining the similar cases and the local region weighted data. Input historical similar data H cases (including label), local region weighted data F weighted(such as 1×256), perform SHAP value analysis (using the SHAP method to evaluate the actual impact of each feature in the current prediction. SHAP is an explanation method based on game theory, applicable to various prediction models such as tree models and neural networks), run KernelSHAP or DeepSHAP on the trained prediction model, KernelSHAP: applicable to any black box model, based on sampling; DeepSHAP: applicable to deep networks, integrating gradients and SHAP ideas, more efficient; output the SHAP value of each dimension feature S is the output feature contribution (SHAP value), Output feature dimension, positive or negative indicates positive / negative influence; or, based on input → prediction gradient derivative: is the local gradient (derivative vector), Risk is the model output risk value, F weighted It is the input feature vector and can be used in model-free explanation scenarios (black box method); the local feature contribution data represents the positive / negative influence of each feature dimension in the current prediction result and can be regarded as a local explanation vector.

[0171] 43. Perform visual attention visualization and structural attention visualization based on the local feature sharing degree to obtain visual attention visualization data and structural attention visualization data respectively;

[0172] In one embodiment, the calculated local feature contribution is modally decomposed into image dimension visualization (visual attention) and structural field visualization (structural attention) to assist physicians in their judgment. The input contribution data is the dimension index related to the imaging modality (e.g., the first 64 dimensions); the corresponding image from the historical case (e.g., CT image); Grad-CAM or Attention Map projection onto the original image; a color overlay (e.g., red areas indicate high risk); and multiple historical images are superimposed to display the distribution of the focal area. The output visual attention visualization data can be an image or an image + heat map overlay.

[0173] Enter the contribution score of the corresponding structure field (such as age, D-Dimer, INR, etc.); bar chart / radar chart / thermal matrix representation; strong contribution fields are placed at the top, and weak fields are grayed out; field explanation notes can be attached, such as: D-Dimer↑→Needs attention↑↑↑.

[0174] 44. Based on the visual attention visualization data and structural attention visualization data, similar case label attribution analysis is performed on historical similar data to obtain clinical risk prediction data.

[0175] In one embodiment, the current prediction result is compared with similar cases, and explanatory labels and risk conclusions are output. Compare the differences between each key feature in similar historical cases and the current patient; use contribution data to highlight the source of risk: "The current patient's D-Dimer level is highly similar to that of the death case #3, and both have large heart rate fluctuations, suggesting that the degree of coagulation will be further aggravated." Label generation is rule-generated or filled with natural language templates; medical knowledge graph comparison can be introduced to supplement the reasoning logic. Visual risk conclusions are proposed, such as "a high-concern area appears in the lower lobe of the lung, which is consistent with the pulmonary embolism area of ​​case #4", and "the imaging risk area is highly matched with the focus area of ​​previous cerebral hemorrhage patients." Integrate the above attribution results to generate structured or natural language description prediction outputs: risk conclusions (such as high, medium, and low risks) (risk conclusions are obtained by training based on risk labels in past cases); sample attribution (such as which cases are similar, which fields are concerned); medical prompts (such as pathological mechanisms that can be cited from knowledge graphs);

[0176] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, the scope of the present invention is limited by the attached application documents rather than the above description, and it is intended that all changes that fall within the meaning and scope of equivalent elements of the application documents are included in the present invention.

[0177] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is to be construed in the widest possible manner consistent with the principles and novel features disclosed herein.

Claims

1. A clinical risk prognosis assessment system for vascular embolism based on multi-source data, characterized in that: The system includes: Patient multi-source data acquisition module, used to obtain patient basic data, clinical examination data, vital signs data and patient imaging data; The patient clinical data feature processing module is used to extract image features based on the patient's image data to obtain image feature data; perform structural characterization based on the patient's basic data, clinical examination data, and vital signs data to obtain the patient's clinical structural feature data; The patient clinical multi-source data fusion module is used to perform multimodal fusion based on the imaging feature data and the patient's clinical structural feature data to obtain the patient's multimodal data; self-attention processing is performed on the patient's multimodal data to obtain local area weighted data; The patient clinical interpretability enhancement module is used to obtain historical similar data based on the user's patient basic data, clinical examination data and vital signs data to obtain historical similar data; and to enhance the interpretability based on historical similar data and local area weighted data to obtain clinical risk prediction data.

2. The system according to claim 1, wherein: The acquisition of patient basic data, clinical examination data, vital signs data and patient imaging data is used to: Obtain basic patient data from the electronic medical record system, including age, gender, height, weight, and history of vascular embolism; obtain clinical examination data from the clinical laboratory electronic system; obtain vital sign data from vital sign monitoring equipment or nursing record system; and obtain patient imaging data through the medical imaging system.

3. The system according to claim 1, wherein: The image feature extraction is performed based on the patient image data to obtain image feature data; the structural characterization is performed based on the patient's basic data, clinical examination data, and vital signs data to obtain the patient's clinical structural feature data, which is used for: Perform image registration based on patient image data to obtain image registration data; perform key area detection based on the image registration data to obtain key area detection data; segment the image registration data based on the key area detection data to obtain image segmentation data; perform depth feature extraction based on the image segmentation data to obtain image depth feature data; perform image feature aggregation on the image depth feature data to obtain image feature aggregation data; Perform field screening based on patient basic data, clinical examination data, and vital signs data to obtain field screening data; perform numerical processing on the field screening data to obtain field numerical data; The missing mark is performed on the field numerical data to obtain the missing mark data; the feature vector is constructed based on the missing mark data to obtain the patient's clinical structure feature data.

4. The system according to claim 1, wherein: The method performs multimodal fusion based on the image feature data and the patient's clinical structural feature data to obtain the patient's multimodal data; performs self-attention processing based on the patient's multimodal data to obtain local area weighted data, which is used for: Perform feature dimension alignment based on the image feature data and the patient's clinical structure feature data to obtain feature dimension alignment data; Performing dual-channel gating processing on the feature dimension aligned data to obtain first patient multimodal data; performing contextual multi-head self-attention calculation on the first patient multimodal data to obtain first multimodal weighted data; performing position mask weighting on the first multimodal weighted data to obtain first local area weighted data; Performing multi-layer perceptron processing on the feature dimension aligned data to obtain second patient multimodal data; performing masked multi-head self-attention calculation on the second patient multimodal data to obtain second multimodal weighted data; performing maximum pooling on the second multimodal weighted data to obtain second local area weighted data; Residual connection is performed according to the first local area weighted data and the second local area weighted data to obtain local area weighted data.

5. The system according to claim 4, characterized in that The dual-channel gating process is performed on the data aligned according to the feature dimensions to obtain multimodal data of the first patient, which is used for: Perform mutual information perception based on the feature dimension alignment data to obtain mutual information perception data; perform gated channel generation based on the mutual information perception data and the feature dimension alignment data to obtain gated channel data, wherein the gated channel data includes structure gated channel data and image gated channel data; perform dual-channel reconstruction and output on the feature dimension alignment data based on the gated channel data to obtain feature reconstructed data; Perform differential residual connection based on the feature-reconstructed data to obtain the multimodal data of the first patient.

6. The system according to claim 4, characterized in that The context multi-head self-attention calculation is performed based on the first patient multimodal data to obtain first multimodal weighted data, which is used to: Context expansion is performed on the first patient's multimodal data to obtain multimodal context expansion data; multi-head self-attention weight calculation is performed on the multimodal context expansion data to obtain multi-head self-attention weight data; multi-head splicing output is performed on the multimodal context expansion data based on the multi-head self-attention weight data to obtain multi-head splicing data; main position features are output based on the multi-head splicing data to obtain first multimodal weighted data.

7. The system according to claim 4, characterized in that The first multimodal weighted data is weighted using a position mask to obtain first local area weighted data, which is used for: Credibility calculation is performed based on the first multimodal weighted data to obtain credibility data; a position mask is constructed based on the credibility data to obtain position mask data; weighted calculation is performed on the first multimodal weighted data based on the position mask data to obtain multimodal position mask data; local activation calculation is performed based on the multimodal position mask data to obtain first local area weighted data.

8. The system according to claim 4, wherein: The multi-layer perceptron processing is performed on the data aligned according to the feature dimensions to obtain the second patient multimodal data, which is used for: Perform layer normalization on the data according to the feature dimension alignment to obtain feature layer normalized data; Perform noise perturbation enhancement on the normalized data of the feature layer to obtain feature enhanced data; According to the feature enhancement data, nonlinear multi-layer perceptron stacking is performed to obtain asymmetric network structure data; according to the asymmetric network structure data, multi-scale residual cross connection is performed to obtain feature cross-layer connection data; Dimensionality compression is performed on the feature cross-layer connection data to obtain the second patient multimodal data.

9. The system according to claim 4, wherein: The masked multi-head self-attention calculation is performed on the second patient multimodal data to obtain second multimodal weighted data, which is used for: Performing pseudo-sequence construction on the multimodal data of the second patient to obtain pseudo-sequence data; performing low-weight indicator shielding on the pseudo-sequence data to obtain low-weight indicator shielding data; performing channel opening on the low-weight indicator shielding data according to preset indicator opening channel data to obtain channel opening data; Perform position control based on channel opening data to obtain mask matrix data; Multi-head self-attention calculation is performed on the second patient's multimodal data according to the mask matrix data to obtain second multimodal weighted data.

10. The system according to claim 1, wherein: The method acquires historical similar data based on the user's patient basic data, clinical examination data, and vital signs data to obtain historical similar data; and enhances interpretability based on the historical similar data and local area weighted data to obtain clinical risk prediction data, which is used for: Historical similarity data is acquired based on the user's patient basic data, clinical examination data, and vital signs data to obtain historical similarity data; local feature contribution is calculated based on the historical similarity data and local area weighted data to obtain local feature contribution data; visual attention visualization and structural attention visualization are performed based on the local feature sharing degree to obtain visual attention visualization data and structural attention visualization data respectively; Based on the visual attention visualization data and structural attention visualization data, similar case label attribution analysis is performed on historical similar data to obtain clinical risk prediction data.

Images