Ventricular tachycardia and ventricular fibrillation prediction method based on multi-modal clinical data and related equipment

Abstract

The invention relates to a ventricular tachycardia and ventricular fibrillation prediction method based on multi-modal clinical data and related equipment. The method comprises the following steps: acquiring structured clinical data and unstructured clinical data of a target patient; preprocessing the structured clinical data and the unstructured clinical data; converting the pre-processed structured clinical data and the pre-processed unstructured clinical data into embedded vectors to obtain structured clinical features and unstructured clinical features; fusing the structured clinical features and the unstructured clinical features based on a bidirectional attention mechanism and a dynamic gating fusion mechanism to obtain multi-modal fusion features; time sequence features are extracted from the multi-modal fusion features through time sequence modeling; and performing ventricular tachycardia and ventricular fibrillation risk prediction according to the time sequence characteristics to obtain a ventricular tachycardia and ventricular Structured and unstructured clinical data are effectively integrated, the multi-modal time sequence prediction model is established, and the prediction accuracy of the ventricular tachycardia and ventricular fibrillation is remarkably improved.

Description

Technical Field

[0001] The present application relates to the technical field of ventricular tachycardia and ventricular fibrillation prediction, and in particular to a ventricular tachycardia and ventricular fibrillation prediction method based on multimodal clinical data and related equipment. Background Art

[0002] Ventricular tachycardia (VT) and ventricular fibrillation (VF) are extremely serious clinical manifestations of cardiovascular disease, posing a direct threat to patients' lives. Ventricular tachycardia (VT) is a tachycardia consisting of three or more wide, distorted QRS complexes originating below the His bundle bifurcation, while ventricular fibrillation (VF) is a rapid, weak contraction of the ventricular muscles, severely impairing blood flow. Both conditions can lead to rapid loss of consciousness or even death. Accurately predicting VT and VF is a pressing issue. Summary of the Invention

[0003] In response to the above-mentioned deficiencies or shortcomings, the present application provides a ventricular tachycardia and ventricular fibrillation prediction method and related equipment based on multimodal clinical data. The embodiments of the present application effectively integrate structured and unstructured clinical data, establish a multimodal time series prediction model, and significantly improve the prediction accuracy of ventricular tachycardia and ventricular fibrillation.

[0004] According to a first aspect, the present application provides a method for predicting ventricular tachycardia and ventricular fibrillation based on multimodal clinical data. In some embodiments, the method includes:

[0005] Obtain structured and unstructured clinical data of target patients;

[0006] Preprocess structured and unstructured clinical data;

[0007] The preprocessed structured clinical data and unstructured clinical data are converted into embedding vectors to obtain structured clinical features and unstructured clinical features;

[0008] Based on the bidirectional attention mechanism and dynamic gating fusion mechanism, structured clinical features and unstructured clinical features are fused to obtain multimodal fusion features;

[0009] Extracting temporal features from the multimodal fusion features through temporal modeling;

[0010] The risk of ventricular tachycardia and ventricular fibrillation is predicted based on the time series characteristics, and the ventricular tachycardia and ventricular fibrillation prediction results are obtained.

[0011] In some embodiments, the structured clinical data includes multiple clinical indicators; the preprocessing includes feature alignment processing; and the feature alignment processing includes:

[0012] Obtain the importance score of each predetermined clinical indicator in the prediction of VT and VF;

[0013] A weight coefficient is set for each clinical indicator according to the importance score of each clinical indicator, and the size of the weight coefficient of the clinical indicator is positively correlated with the size of the importance score of the clinical indicator.

[0014] In some embodiments, structured clinical features and unstructured clinical features are fused based on a bidirectional attention mechanism and a dynamic gating fusion mechanism to obtain multimodal fusion features, including:

[0015] calculating the attention weight of the structured clinical feature to the unstructured clinical feature as the first attention weight, and generating the first feature according to the first attention weight and the unstructured clinical feature;

[0016] Calculating the inverse attention weight of the unstructured clinical feature to the structured clinical feature as the second attention weight, and generating the second feature according to the second attention weight and the structured clinical feature;

[0017] The first feature and the second feature are dynamically mixed through a dynamic gated fusion mechanism to obtain a multimodal fusion feature.

[0018] In some embodiments, extracting a temporal feature from the multimodal fusion feature through temporal modeling includes:

[0019] Obtaining a pre-set attenuation factor value rule for the acute attack characteristics of ventricular tachycardia and ventricular fibrillation, and performing dynamic time attenuation processing on the multimodal fusion feature using the attenuation factor value rule to obtain the attenuated fusion feature;

[0020] Perform time series modeling on the attenuated fusion features from the forward and backward directions respectively to obtain bidirectional splicing features;

[0021] The convolution kernel parameters are generated dynamically according to time through a multi-layer perceptron, and the convolution kernel parameters are used to extract temporal features from the bidirectional splicing features.

[0022] In some embodiments, the attenuation factor value selection rules include:

[0023] When the difference between the current prediction time and the current data point time is less than or equal to 24 hours, the attenuation factor is set to the first threshold;

[0024] When the difference between the current prediction time and the current data point time is greater than 24 hours and less than or equal to 72 hours, the attenuation factor is set to the second threshold;

[0025] When the difference between the current prediction time and the current data point time is greater than 72 hours, the attenuation factor is set to the third threshold; among the first threshold, the second threshold and the third threshold, the first threshold is the largest and the third threshold is the smallest.

[0026] In some embodiments, the time series features include multiple single-point time series features corresponding to different time steps; performing ventricular tachycardia and ventricular fibrillation risk prediction based on the time series features to obtain a ventricular tachycardia and ventricular fibrillation prediction result includes:

[0027] Calculate the time attention weight of each single-point time series feature, aggregate the single-point time series features included in the time series features according to the time attention weight of each single-point time series feature and the preset time attenuation gain term to obtain the deep time series feature;

[0028] Combine the deep temporal features, the initial fusion features in the multimodal fusion features, and the latest fusion features to obtain residual features;

[0029] Generate ventricular tachycardia and ventricular fibrillation risk assessment values ​​based on residual characteristics.

[0030] In some embodiments, generating a ventricular tachycardia / ventricular fibrillation risk assessment value based on the residual features includes:

[0031] Calculate static risk assessment value based on residual characteristics;

[0032] Calculate dynamic risk assessment values ​​based on temporal gradient characteristics;

[0033] A ventricular tachycardia and ventricular fibrillation risk assessment value is generated according to the static risk assessment value and the dynamic risk assessment value.

[0034] According to a second aspect, the present application provides a device for predicting ventricular tachycardia and ventricular fibrillation based on multimodal clinical data. In some embodiments, the device includes:

[0035] Clinical data acquisition module, used to obtain structured and unstructured clinical data of target patients;

[0036] A preprocessing module, used to preprocess structured clinical data and unstructured clinical data;

[0037] An embedding module, which is used to convert the preprocessed structured clinical data and unstructured clinical data into embedding vectors to obtain structured clinical features and unstructured clinical features;

[0038] The feature fusion module is used to fuse structured clinical features and unstructured clinical features based on the bidirectional attention mechanism and dynamic gated fusion mechanism to obtain multimodal fusion features;

[0039] A time series modeling module, configured to extract time series features from the multimodal fusion features through time series modeling;

[0040] The ventricular tachycardia and ventricular fibrillation prediction module is used to predict the risk of ventricular tachycardia and ventricular fibrillation based on time series characteristics and obtain ventricular tachycardia and ventricular fibrillation prediction results.

[0041] According to a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for predicting ventricular tachycardia and ventricular fibrillation based on multimodal clinical data provided in any of the above embodiments is implemented.

[0042] According to a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for predicting ventricular tachycardia and ventricular fibrillation based on multimodal clinical data provided in any of the above embodiments is implemented.

[0043] In the above-mentioned embodiment of the present application, the structured clinical data and the unstructured clinical data are first preprocessed; then the processed data is converted into feature vectors to obtain structured clinical features and unstructured clinical features. On this basis, the bidirectional attention mechanism and the dynamic gated fusion mechanism are innovatively adopted to realize multimodal feature fusion and generate fusion features. Then, the timing features in the fusion features are extracted through timing modeling, and finally a deep learning model is constructed based on these timing features to realize accurate prediction of ventricular tachycardia and ventricular fibrillation. This method effectively integrates structured and unstructured clinical data, establishes a multimodal timing prediction model, and significantly improves the prediction accuracy of ventricular tachycardia and ventricular fibrillation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a method for predicting ventricular tachycardia and ventricular fibrillation based on multimodal clinical data according to one or more embodiments of the present application is provided;

[0045] Figure 2 A data flow diagram provided by this application according to one or more embodiments;

[0046] Figure 3 This is a structural block diagram of a device for predicting ventricular tachycardia and ventricular fibrillation based on multimodal clinical data according to one or more embodiments of the present application;

[0047] Figure 4 This is a diagram of the internal structure of a computer device provided in accordance with one or more embodiments of the present application. DETAILED DESCRIPTION

[0048] To make the purpose, technical solutions, and advantages of this application more clear, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0049] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0050] In the description of this application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances. In addition, in the description of this application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0051] In response to the shortcomings of related technologies, this application provides a ventricular tachycardia and ventricular fibrillation prediction method based on multimodal clinical data. This method can effectively integrate structured and unstructured clinical data, and achieve accurate prediction of ventricular tachycardia and ventricular fibrillation by establishing a multimodal time series prediction model.

[0052] In some embodiments, the method includes Figure 1 The steps shown are described in detail below using the method applied to a server as an example.

[0053] S110: Acquire structured clinical data and unstructured clinical data of the target patient.

[0054] The target patient refers to the patient whose ventricular tachycardia or ventricular fibrillation is to be predicted, and specifically may be a patient with cardiovascular disease, such as a myocardial infarction patient.

[0055] The server can obtain multimodal clinical data generated during the patient's medical treatment process from the data center of the medical institution, including structured clinical data and unstructured clinical data.

[0056] Structured clinical data can be clinical indicator information of patients within a specific time period (such as hospitalization, from admission to discharge), such as basic information, clinical scores, medical history information, blood test indicators and echocardiography indicators, etc. Basic information can include the patient's gender, age, number of days of hospitalization, etc. Clinical scores can include GRACE (Global Registry of Acute Coronary Events) score and TIMI (Thrombolysis In Myocardial Infarction) score. Medical history information can include history of hypertension, atrial fibrillation, diabetes, hyperlipidemia, cerebrovascular disease, gastric ulcer, old myocardial infarction, smoking history, whether to quit smoking, previous stents, and previous bypass (Coronary Artery Bypass Grafting, CABG). Blood test indicators may include total white blood cell count, neutrophil count, lymphocyte count, monocyte count, glycated hemoglobin, platelets, blood glucose, alanine aminotransferase (ALT), aspartate aminotransferase (AST), creatinine clearance rate (CCR), total cholesterol (TC), low-density lipoprotein cholesterol (LDL-C), high-density lipoprotein cholesterol (HDL-C), triglyceride (TG), uric acid (UA), glycated hemoglobin (HbA1c), high-sensitivity C-reactive protein (hs-CRP), interleukin-6 (IL-6), NT-pro BNP (N-terminal Pro-B-type natriuretic peptide (B-type natriuretic peptide) and TNI (Troponin I) peak levels. Echocardiographic indices may include left ventricular ejection fraction (LVEF) and left ventricular end-diastolic diameter (LVEDD).

[0057] Unstructured clinical data can be the patient's medical records within a specific time period (such as during hospitalization). For example, it can be the patient's hospitalization medical records and other medical records. These medical records record the patient's entire hospitalization process in detail, including but not limited to the patient's admission record, medical history record, vascular intervention examination and treatment record sheet, coronary angiography report sheet and discharge record and other text content. The admission record usually records the patient's initial admission in detail, including the patient's chief complaint, current medical history, past medical history and comprehensive physical examination results. This information can be used as a preliminary diagnostic basis for the prediction of ventricular tachycardia and ventricular fibrillation. The medical history record records in detail every physical examination and treatment progress of the patient during hospitalization. These records follow a strict chronological order and are usually updated every two or three days. They include information such as changes in the patient's condition, physician rounds, treatment medication regimens and their effects. The temporal nature of the medical history record can provide valuable time series data for subsequent ventricular tachycardia and ventricular fibrillation predictions. The Vascular Intervention Examination and Treatment Record accurately records the vascular intervention procedures a patient undergoes, including information such as the surgical method, operation details, and surgical results. This information provides direct evidence for evaluating the patient's vascular condition and treatment effectiveness. The Coronary Angiography Report details the patient's coronary angiography results, including information such as the coronary artery's anatomical structure, degree of stenosis, and blood flow conditions. These results provide a scientific basis for doctors to formulate treatment plans. The Discharge Record is the final summary of the patient's hospitalization, including summary information about the patient's hospitalization experience and health status at discharge.

[0058] S120: Preprocessing the structured clinical data and the unstructured clinical data.

[0059] This step uses preprocessing to clean, transform, and align the patient's multimodal clinical data (i.e., structured and unstructured clinical data) to ensure that the processed multimodal clinical data meets the requirements of subsequent steps. Cleaning primarily eliminates noise and ensures data quality, and can include processing missing values, outliers, and duplicate values. Transformation can involve converting data formats to unify different data formats.

[0060] Alignment includes time alignment, feature alignment, and content alignment. Aligning multimodal clinical data based on time and feature dimensions ensures that structured and unstructured clinical data can be effectively integrated within a unified spatiotemporal framework.

[0061] Time alignment can include the following operations:

[0062] (1) Establish a unified time axis: With the patient's admission time as the time origin (t = 0), a unified time axis coordinate system is established. All structured clinical data (such as laboratory test results, vital signs, etc.) and unstructured clinical data (such as medical records, surgical records, etc.) are timestamped to ensure that all events have a clear time point on the time axis;

[0063] (2) Timestamp parsing and alignment: For unstructured text data (such as medical records), a timestamp parsing model based on natural language processing (NLP) (such as BERT-BiLSTM) is used to extract time information from the text. For example, from "The patient experienced chest pain on the third day after surgery", the time point "3 days after surgery" is extracted and mapped to a unified timeline. For structured clinical data, the recorded timestamps can be used directly for alignment.

[0064] (3) Sliding window alignment mechanism: Set a sliding time window (e.g., 24 hours) to match structured clinical data and unstructured clinical data within the same time window. For data with missing time points, interpolation methods (e.g., cubic spline interpolation) can be used to fill in the missing time points during cleaning to ensure the continuity of the time series.

[0065] Feature alignment can include the following operations:

[0066] (1) Feature alignment for structured clinical data: Standardize the structured clinical data to eliminate the dimensional differences of different indicators. For example, use the Z-score standardization method to process structured clinical data; use time series interpolation methods (such as linear interpolation or spline interpolation) to fill in missing clinical indicators in structured clinical data.

[0067] In some embodiments, when feature alignment is performed on structured clinical data, the importance score of each clinical indicator in the predetermined structured clinical data in the prediction of ventricular tachycardia and ventricular fibrillation can be obtained, and then a weight coefficient is set for each clinical indicator based on the importance score of each clinical indicator. The size of the weight coefficient of each clinical indicator is positively correlated with the size of the importance score of the clinical indicator, so that key indicators (such as TNI peak) account for a higher proportion in the model, thereby improving the accuracy of ventricular tachycardia and ventricular fibrillation prediction.

[0068] For example, a dynamic weight can be applied to each clinical indicator by the following formula to achieve weight enhancement:

[0069] ω i =α·ICD score (x i )+β;

[0070] ICD score (x i ) represents the clinical index xi Importance score in the prediction of ventricular tachycardia and ventricular fibrillation. i represents the weight coefficient for the i-th clinical indicator. The higher the weight coefficient for a clinical indicator, the higher its status in the model. α and β are hyperparameters that can be set in advance based on reliable clinical knowledge, for example, α = 0.8 and β = 0.2. Dynamic weighting will be implemented at multiple stages. In the feature dimension alignment stage, assigning higher weight coefficients to key indicators (i.e., clinical indicators with high importance scores) can strengthen these indicators, enabling them to actively influence model learning. Simultaneously, assigning lower weight coefficients to less relevant clinical indicators (i.e., clinical indicators with low importance scores) can weaken their influence and reduce noise interference. In the subsequent cross-modal fusion stage, the attention weight of clinical indicators with high weight coefficients on unstructured feature vectors can be increased, making these clinical indicators more closely associated with unstructured clinical data such as text content. In time series modeling, the weight coefficients of clinical indicators will work synergistically with the piecewise decay factor, allowing the model to pay more attention to recent changes in key indicators, thereby improving prediction accuracy.

[0071] (2) Feature alignment for unstructured clinical data: Use a pre-trained language model (such as the BERT model) to embed the text in the unstructured clinical data (hereinafter referred to as unstructured text) to convert the unstructured text into a feature vector of fixed dimension; perform entity recognition and feature extraction on key medical events in the text (such as "chest pain attack", "high blood pressure", etc.) to ensure that the text features are semantically aligned with the structured clinical data.

[0072] Content alignment can include the following operations:

[0073] There may be potential connections between structured and unstructured clinical data. For example, patient symptoms, treatment responses, or complications mentioned in medical records may be associated with changes in clinical indicators in structured data. Content alignment can be performed by performing the following operations to fully explore the potential connections between structured and unstructured clinical data:

[0074] (1) Use a named entity recognition model (e.g., a NER model) to extract key medical entities from medical records, such as symptoms (chest pain, dyspnea), treatments (drug names, surgery types), and complications (arrhythmia, heart failure). Match the extracted entities with the corresponding fields in the structured clinical data. For example, associate "chest pain" in the medical record with "TNI peak" in the structured clinical data.

[0075] (2) Identify the associations between entities in the medical records through the relationship extraction model. For example, identify the therapeutic relationship between "chest pain" and "nitroglycerin treatment", or the causal relationship between "elevated blood pressure" and "history of hypertension". Perform correlation analysis on the extracted relationships and the changes in clinical indicators in the structured clinical data. For example, associate "nitroglycerin treatment" with "blood pressure changes" in the structured data.

[0076] (3) Use a semantic similarity calculation model (such as BERT-based sentence embedding) to semantically match the text descriptions in the medical record with the fields in the structured data. For example, semantically align "the patient experienced dyspnea" in the medical record with "blood oxygen saturation" in the structured clinical data. Then, dynamically adjust the weight of the semantic match through the attention mechanism to ensure the accuracy of the content alignment.

[0077] S130: Convert the preprocessed structured clinical data and unstructured clinical data into embedding vectors to obtain structured clinical features and unstructured clinical features.

[0078] The preprocessed structured clinical data includes structured feature vectors of multiple time steps; the preprocessed unstructured clinical data includes unstructured feature vectors of multiple time steps.

[0079] This step uses deep learning methods to convert preprocessed structured and unstructured clinical data into low-dimensional embedding representations to facilitate subsequent analysis and model training.

[0080] In some embodiments, the pre-processed structured clinical data can be converted into an embedding vector using the following formula:

[0081]

[0082] Among them, W s ∈R d×m ,b s ∈R d is a trainable parameter, is the structured feature vector at time step t, d is the unified embedding dimension, and LayerNorm represents layer normalization.

[0083] This embodiment uses a simple linear transformation plus Layer Normalization to transform the structured feature vector of time step t into Convert to embedding vector Among them, the linear transformation is achieved through a weight matrix W s ∈R d×m and the bias term b sImplementation, where d is the target embedding dimension and m is the dimension of the input feature vector. Layer normalization is used to standardize the output to speed up the training process and improve model performance.

[0084] In some embodiments, the pre-processed unstructured clinical data can be converted into an embedding vector using the following formula:

[0085]

[0086] Among them, W t ∈R d×768 is the dimension projection matrix.

[0087] For unstructured clinical data, the BERT model is first used to convert the unstructured feature vectors of each time step into the initial embedding vector. Specifically, the BERT model is used to transform the unstructured feature vectors of the input time step t into the initial embedding vectors. Encode and take the output corresponding to the [CLS] tag as the sentence level representation to obtain the initial embedding vector Since the default output vector dimension of the BERT model is 768, in order to match the embedding vector of structured clinical data Align to a unified embedding dimension d, and then further process the initial embedding vector output by the BERT model through a linear transformation (i.e. a fully connected layer) and a ReLU activation function Get the final embedding vector The linear transformation used here includes a weight matrix W t ∈R d×768 and the bias term b t , used to project the output of the BERT model onto the specified dimension d.

[0088] In order to distinguish the embedding vectors of different modalities, the embedding vectors It is called structured embedding vector, which embeds the vector It is called unstructured embedding vector.

[0089] S140: Based on the bidirectional attention mechanism and dynamic gating fusion mechanism, structured clinical features and unstructured clinical features are fused to obtain multimodal fusion features.

[0090] This step aims to fuse and interact the embedding vectors of each modality to achieve information complementarity. By bidirectionally calculating the attention weights of structured clinical features on unstructured clinical features and the anti-attention weights of unstructured clinical features on structured clinical features, dynamic gating fusion technology is then used to dynamically adjust the contribution ratios of different modalities through a learnable gating variable (γ) to achieve feature fusion and avoid information redundancy caused by simple splicing.

[0091] In some embodiments, the structured clinical features and the unstructured clinical features are fused based on a bidirectional attention mechanism and a dynamic gated fusion mechanism to obtain a multimodal fusion feature, including the following steps:

[0092] (1) Calculate the attention weight of the structured clinical feature to the unstructured clinical feature as the first attention weight, and generate the first feature according to the first attention weight and the unstructured clinical feature.

[0093] The attention weight of the structured clinical features to the unstructured clinical features refers to the attention weight of the structured embedding vector of each time step in the structured clinical features to the unstructured embedding vector of each time step in the unstructured clinical features.

[0094] The attention weight can be represented by the similarity score between the two embedding vectors. For example, the similarity score can be calculated by the following formula.

[0095]

[0096] Among them, α ij Represents the structured embedding vector at the jth time step and the unstructured embedding vector at the jth time step The similarity score between them is obtained by embedding the structured vector and unstructured embedding vectors The dot product of (where d is the embedding dimension), and then the softmax function is applied to normalize the obtained. T represents the total number of time steps; represents the unstructured embedding vector at the kth time step.

[0097] After obtaining the attention weight of the structured embedding vector of each time step to the unstructured embedding vector of each time step, the unstructured embedding vector of each time step is weighted summed based on this attention weight to obtain the first feature F s→t The weighted summation operation is implemented by the following formula:

[0098]

[0099] (2) Calculate the inverse attention weight of the unstructured clinical feature to the structured clinical feature as the second attention weight, and generate the second feature according to the second attention weight and the structured clinical feature.

[0100] The attention weight of unstructured clinical features to structured clinical features refers to the reverse attention weight of the unstructured embedding vector of each time step in the unstructured clinical features to the structured embedding vector of each time step in the structured clinical features.

[0101] The reverse attention weight can be represented by the degree of correlation between the two embedding vectors. For example, the degree of correlation can be calculated by the following formula.

[0102]

[0103] β ji Represents the unstructured embedding vector at the jth time step and the structured embedding vector at the i-th time step The degree of correlation between them. Among them, a multi-layer perceptron (MLP) is used to process the unstructured embedding vector of the j-th time step and the structured embedding vector at the kth time step The concatenated vectors can be used to obtain more complex interaction representations.

[0104] After obtaining the reverse attention weight of the unstructured embedding vector of each time step to the structured embedding vector of each time step, the structured embedding vector of each time step is weighted summed based on this reverse attention weight to obtain the second feature F t→s The weighted summation operation is implemented by the following formula:

[0105]

[0106] (3) The first feature and the second feature are dynamically mixed through a dynamic gated fusion mechanism to obtain a multimodal fusion feature.

[0107] This step dynamically mixes the first and second features through a dynamic gated fusion mechanism. The relevant operations can be achieved through the following formula:

[0108] γ=σ(W g ·[F s→t ; F t→s ]+b g );

[0109] F c =γ⊙F s→t +(1-γ)⊙F t→s ;

[0110] The dynamic gating fusion mechanism specifically uses a learnable gating variable γ to determine how to combine the information F extracted from the two modalities. s→t and F t→s . And γ is transformed by a linear transformation (weight matrix W g and the bias term b g ) acts on the concatenated vector [F s→t ; F t→s], and then pass through the sigmoid function σ to control the contribution ratio of the two source information. The multimodal fusion feature F c is through γ and F s→t The element-wise product of (1-γ) and F t→s The sum of the element-wise products of .

[0111] S150: Extracting temporal features from the multimodal fusion features through temporal modeling.

[0112] This step uses dynamic time series modeling to capture the pattern of disease evolution over time from multimodal fusion features, especially the acute precursors of ventricular tachycardia and ventricular fibrillation. The captured information is the time series feature.

[0113] In some embodiments, extracting temporal features from the multimodal fusion features through temporal modeling includes the following steps:

[0114] (1) Obtain a pre-set attenuation factor value rule for the acute attack characteristics of ventricular tachycardia and ventricular fibrillation, use the attenuation factor value rule to perform dynamic time attenuation processing on the multimodal fusion feature, and obtain the attenuated fusion feature.

[0115] In order to enable the model to adjust the degree of influence according to the distance between the event and the current moment, thereby more accurately capturing the development trend of ventricular tachycardia and ventricular fibrillation, this embodiment introduces a dynamic time attenuation factor into the model, that is, an attenuation factor that changes with time. In addition, based on the acute attack characteristics of ventricular tachycardia and ventricular fibrillation, a segmented attenuation factor is designed to enable the model to focus on the changes in key indicators 24 hours before the attack, which helps to enhance the model's predictive ability for acute events.

[0116] The value rules set based on the piecewise attenuation factor include:

[0117] a. When the difference between the current prediction time and the current data point time is less than or equal to 24 hours, the attenuation factor is set to the first threshold;

[0118] b. When the difference between the current prediction time and the current data point time is greater than 24 hours and less than or equal to 72 hours, the attenuation factor is set to the second threshold;

[0119] c. When the difference between the current prediction time and the current data point time is greater than 72 hours, the attenuation factor is set to the third threshold; among the first threshold, the second threshold and the third threshold, the first threshold is the largest and the third threshold is the smallest.

[0120] For example, the piecewise attenuation factor may be as follows:

[0121]

[0122] λ(t e ,t c ) indicates that the corresponding timestamp is t e The attenuation factor corresponding to the multimodal fusion feature. e Refers to the current data point time, indicating the timestamp of the current data point (such as the current multimodal fusion feature). c Refers to the current prediction time, which indicates the time when the model makes predictions or evaluations.

[0123] For any multimodal fusion feature, it can be attenuated using the following formula:

[0124]

[0125] represents the multimodal fusion features at time step t, Represents the multimodal fusion feature at time step t after attenuation. If the difference between the timestamp of the multimodal fusion feature and the current prediction time is greater than 72 hours, the attenuation factor applied to the multimodal fusion feature is 1.0. If the difference between the timestamp of the multimodal fusion feature and the current prediction time is greater than 24 hours and less than or equal to 72 hours, the attenuation factor applied to the multimodal fusion feature is 1.5. If the difference between the timestamp of the multimodal fusion feature and the current prediction time is less than or equal to 24 hours, the attenuation factor applied to the multimodal fusion feature is 2.0. By assigning a higher weight (e.g., λ = 2.0) to the data 24 hours before the onset, the model can focus on the changes in key indicators 24 hours before the onset.

[0126] (2) The attenuated fusion features are temporally modeled from the forward and backward directions respectively to obtain bidirectional splicing features.

[0127]

[0128] Refer to the above formula. In this step, a bidirectional LSTM (Long Short-Term Memory) can be used to model the sequence from both the forward and backward directions, thereby capturing the dependencies between the time series (such as the temporal changes in the course of the disease). and backward state The information from the beginning of the sequence to the current time t and the information from the end of the sequence to the current time t are captured respectively. The bidirectional splicing feature H is formed by splicing the information from these two directions t , which can more comprehensively represent the state of each time step.

[0129] (3) The convolution kernel parameters are generated dynamically according to time through a multi-layer perceptron, and the temporal features are extracted from the bidirectional splicing features using the convolution kernel parameters.

[0130] The time series features include features of multiple time steps, and the features of each time step in the time series features will be referred to as single-point time series features below.

[0131] In some embodiments, the extraction of temporal features can be achieved through adaptive temporal convolution.

[0132]

[0133] Among them, C t Represents the single-point time series feature from time step t.

[0134] K is the convolution kernel window size (can be set according to actual needs, for example, set K = 3), w k (t) refers to the convolution kernel parameters dynamically generated by the multilayer perceptron (MLP) according to the time step t, w k (t) = MLP(t). This processing method allows the convolution operation to take into account temporal changes, enhancing the model's ability to distinguish features at different input moments.

[0135] S160: Predicting the risk of ventricular tachycardia and ventricular fibrillation based on the time series characteristics to obtain a ventricular tachycardia and ventricular fibrillation prediction result.

[0136] In some embodiments, performing ventricular tachycardia and ventricular fibrillation risk prediction based on time series characteristics to obtain a ventricular tachycardia and ventricular fibrillation prediction result includes the following steps:

[0137] (1) Calculate the time attention weight of each single-point time series feature, aggregate the single-point time series features included in the time series feature according to the time attention weight of each single-point time series feature and the preset time attenuation gain term, and obtain the deep time series feature.

[0138] The temporal attention weight of each single-point temporal feature can be determined through the temporal attention mechanism.

[0139]

[0140] is the weight matrix used to project single-point temporal features into the hidden space, d h is the hidden layer dimension, d c is the output dimension of the temporal convolution. represents the bias term, represents the attention score vector; a t Represents the temporal attention weight of the single-point temporal feature at time step t.

[0141] In the above formula, Indicates that the linear transformation (W a C t +b a ) The single point time series feature C of time step t t Mapping to latent space Then, nonlinear activation is introduced through the tanh function to extract the deep temporal pattern of time step t (such as the precursor features of disease worsening). The importance score of the single-point temporal feature of time step t is calculated through the vector v. The vector v can learn which hidden features are most critical for the prediction of ventricular tachycardia and ventricular fibrillation.

[0142] Last passed Convert the importance score of the single-point temporal feature at time step t into a probability distribution, i.e., the temporal attention weight a t The higher the weight of a single-point time series feature, the greater its impact on the final prediction result.

[0143] After obtaining the temporal attention weights of the single-point temporal features at each time step, the following formula can be used to aggregate the single-point temporal features included in the temporal features. This formula can simultaneously consider the importance and timeliness of the single-point temporal features during aggregation, making the model more closely aligned with the temporal dynamics of the real world, thereby achieving more robust and efficient fusion, and ultimately significantly improving the model's ability to model dynamic and complex data:

[0144]

[0145] h g Represents the deep temporal features obtained by aggregation.

[0146] λ(t,T) represents the time decay gain term, which can be calculated by the following formula:

[0147] λ(t,T)=1+sigmoid(w λ ·(Tt));

[0148] t represents the time corresponding to the relevant single-point time series feature, and T represents the time of the latest clinical data in the target patient's multimodal clinical data (e.g., if the multimodal clinical data is the clinical data from the patient's admission to discharge, T can be the discharge time). By introducing the time-attenuation gain term, a higher gain coefficient can be assigned to recent features (the gain coefficient value range can be, for example, 1.2-2.0), and a lower gain coefficient can be assigned to long-term features (the gain coefficient value range can be, for example, 1.0-1.5), thereby highlighting the important value of recent features for ventricular tachycardia and ventricular fibrillation prediction, while retaining the reference value of long-term features for ventricular tachycardia and ventricular fibrillation prediction.

[0149] (2) The deep temporal features, the initial fusion features in the multimodal fusion features and the latest fusion features are combined to obtain the residual features.

[0150] In order to prevent information attenuation in the deep network, this step combines the deep temporal features with the initial fusion features and the latest fusion features in the multimodal fusion features to preserve the original cross-modal interaction features.

[0151] The above combination operation can be achieved through the following formula:

[0152]

[0153] h f Represents the residual feature. LayerNorm represents layer normalization. This step uses layer normalization to stabilize gradient propagation. is the residual projection matrix, which is used to integrate the multimodal fusion features of time step 1 (such as admission time) and multimodal fusion features of time step T (such as discharge time or latest time) The concatenated vector is projected into the latent space, d h is the hidden layer dimension, and d is the unified embedding dimension, which is used to ensure that the dimensions of the residual connections match.

[0154] (3) Generate ventricular tachycardia and ventricular fibrillation risk assessment values ​​based on residual characteristics.

[0155] In some embodiments, generating a ventricular tachycardia / ventricular fibrillation risk assessment value based on the residual features comprises the following steps:

[0156] a. Calculate the static risk assessment value based on the residual characteristics;

[0157] In this step, the static risk assessment value can be calculated using the following formula:

[0158] s c =σ(W c h f +b c );

[0159] s c represents the static risk assessment value, h f is the residual feature, W c is the weight matrix, b c is the bias term.

[0160] b. Calculate dynamic risk assessment values ​​based on temporal gradient characteristics;

[0161] This step uses temporal gradient features to capture the disease development trend (represented by dynamic risk assessment values), which can be used to correct the static risk assessment value calculated in the previous step.

[0162] The dynamic risk assessment value can be calculated using the following formula:

[0163]

[0164] Δs represents the dynamic risk assessment value, W Δ is the weight matrix, b Δ is the bias term. ReLU represents the activation function. H T represents the bidirectional splicing feature at time step T, Represents the temporal gradient feature. It can be calculated by the following formula

[0165]

[0166] c. Generate a ventricular tachycardia or ventricular fibrillation risk assessment value based on the static risk assessment value and the dynamic risk assessment value.

[0167] The VT-VF risk estimate can be generated using the following formula:

[0168]

[0169] represents the ventricular tachycardia / ventricular fibrillation risk assessment value, and Clip represents the Clip operation, which ensures that the model's final output conforms to the probability range. η refers to Gaussian noise and is only enabled during model training. By adding Gaussian noise during training, the model can simulate the uncertainty of real clinical data, enhancing its robustness.

[0170] In some embodiments, a ventricular tachycardia and ventricular fibrillation prediction model based on deep learning and multimodal clinical data fusion can be constructed. The ventricular tachycardia and ventricular fibrillation prediction model is used to implement the above steps S130-S160. Its input data is the structured clinical features and unstructured clinical features output by step S120, and the output data is the ventricular tachycardia and ventricular fibrillation prediction results.

[0171] The ventricular tachycardia and ventricular fibrillation prediction model includes multiple multimodal embedding modules, cross-modal fusion interaction modules, time series modeling modules and output modules that are connected in sequence.

[0172] A multimodal embedding module is used to convert the preprocessed unstructured clinical data into low-dimensional embedding vectors (i.e., unstructured clinical features) through BERT encoding and dimensionality projection, and to convert the preprocessed structured clinical data into low-dimensional embedding vectors (i.e., structured clinical features) through linear transformation and layer normalization.

[0173] The cross-modal fusion interaction module is used to dynamically mix the unstructured clinical features and structured clinical features output by the multimodal embedding module through a bidirectional attention mechanism and a dynamic gated fusion mechanism to obtain multimodal fusion features;

[0174] The timing modeling module is used to extract the timing features in the multimodal fusion features output by the cross-modal fusion interaction module through dynamic time decay, bidirectional timing modeling and adaptive timing convolution.

[0175] The output module consists of a sequentially connected dynamic temporal attention pooling layer, a multimodal residual connection layer, and a risk dynamic prediction layer. The dynamic temporal attention pooling layer aggregates the individual point temporal features output by the temporal modeling module into deep temporal features based on a dynamic temporal attention mechanism and a time decay gain term.

[0176] The multimodal residual connection layer is also connected to the cross-modal fusion interaction module, which is used to combine the deep temporal features output by the dynamic time attention pooling layer and the initial fusion features and the latest fusion features output by the cross-modal fusion interaction module to obtain residual features.

[0177] The risk dynamic prediction layer has two paths: the main path and the auxiliary path. The main path is used to calculate the static risk assessment value based on the residual characteristics, and the auxiliary path is used to calculate the dynamic risk assessment value based on the time gradient characteristics. The risk dynamic prediction layer can combine the information generated by the two paths (i.e., the static risk assessment value and the dynamic risk assessment value) and output a prediction result that is both immediate and trend-based (i.e., the ventricular tachycardia and ventricular fibrillation risk assessment value).

[0178] The VT / VF prediction model can comprehensively improve the prediction accuracy of VT / VF, achieve early warning, and thus provide important clinical evidence for precise intervention and personalized treatment. The methods or steps implemented by each module of the VT / VF prediction model can be found in the description of the above embodiments and will not be repeated here.

[0179] It should be noted that, in the embodiments of the present application, data related to user information or user data (for example, structured and unstructured clinical data of patients during hospitalization, etc.) need to be obtained and processed after authorization and consent from the user. When the embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. In addition, with respect to the various steps included in the method for predicting ventricular tachycardia and ventricular fibrillation based on multimodal clinical data provided in any of the above embodiments, unless otherwise clearly stated in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of these steps can include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0180] Based on the same inventive concept, the present application also provides a device for predicting ventricular tachycardia and ventricular fibrillation based on multimodal clinical data. In some embodiments, Figure 3 As shown in FIG, the ventricular tachycardia and ventricular fibrillation prediction device based on multimodal clinical data includes the following modules:

[0181] A clinical data acquisition module 110 is used to acquire structured clinical data and unstructured clinical data of a target patient;

[0182] A preprocessing module 120 is used to preprocess structured clinical data and unstructured clinical data;

[0183] An embedding module 130, configured to convert the pre-processed structured clinical data and unstructured clinical data into embedding vectors to obtain structured clinical features and unstructured clinical features;

[0184] A feature fusion module 140 is configured to fuse structured clinical features and unstructured clinical features based on a bidirectional attention mechanism and a dynamic gated fusion mechanism to obtain multimodal fusion features;

[0185] A time series modeling module 150 is configured to extract time series features from the multimodal fusion features through time series modeling;

[0186] The ventricular tachycardia and ventricular fibrillation prediction module 160 is used to predict the risk of ventricular tachycardia and ventricular fibrillation according to the time series characteristics and obtain a ventricular tachycardia and ventricular fibrillation prediction result.

[0187] In some embodiments, the structured clinical data includes multiple clinical indicators; the preprocessing includes feature alignment processing; the feature alignment processing includes: obtaining the importance score of each predetermined clinical indicator in the prediction of ventricular tachycardia and ventricular fibrillation, setting a weight coefficient for each clinical indicator according to the importance score of each clinical indicator, and the size of the weight coefficient of each clinical indicator is positively correlated with the size of the importance score of the clinical indicator.

[0188] In some embodiments, the feature fusion module 140 includes:

[0189] a first feature processing submodule, configured to calculate an attention weight of the structured clinical feature to the unstructured clinical feature as a first attention weight, and generate a first feature according to the first attention weight and the unstructured clinical feature;

[0190] a second feature processing submodule, configured to calculate an inverse attention weight of the unstructured clinical feature to the structured clinical feature as a second attention weight, and generate a second feature according to the second attention weight and the structured clinical feature;

[0191] The dynamic gated fusion submodule is used to dynamically mix the first feature and the second feature through a dynamic gated fusion mechanism to obtain a multimodal fusion feature.

[0192] In some embodiments, the timing modeling module 150 includes:

[0193] The rule acquisition submodule is used to obtain the attenuation factor value rule pre-set for the acute attack characteristics of ventricular tachycardia and ventricular fibrillation, and use the attenuation factor value rule to perform dynamic time attenuation processing on the multimodal fusion feature to obtain the attenuated fusion feature;

[0194] The bidirectional temporal modeling submodule is used to perform temporal modeling on the attenuated fusion features from the forward and backward directions respectively to obtain bidirectional splicing features;

[0195] The adaptive temporal convolution submodule is used to dynamically generate convolution kernel parameters according to time through a multi-layer perceptron, and use the convolution kernel parameters to extract temporal features from the bidirectional splicing features.

[0196] In some embodiments, the attenuation factor value selection rules include: when the difference between the current prediction time and the current data point time is less than or equal to 24 hours, the attenuation factor is taken as the first threshold value; when the difference between the current prediction time and the current data point time is greater than 24 hours and less than or equal to 72 hours, the attenuation factor is taken as the second threshold value; and when the difference between the current prediction time and the current data point time is greater than 72 hours, the attenuation factor is taken as the third threshold value; among the first threshold value, the second threshold value and the third threshold value, the first threshold value is the largest and the third threshold value is the smallest.

[0197] In some embodiments, the time series features include multiple single-point time series features corresponding to different time steps; the ventricular tachycardia and ventricular fibrillation prediction module 160 includes:

[0198] The temporal feature aggregation submodule is used to calculate the temporal attention weight of each single-point temporal feature, aggregate the single-point temporal features included in the temporal features according to the temporal attention weight of each single-point temporal feature and the preset time attenuation gain term, and obtain the deep temporal features;

[0199] The feature combination submodule is used to combine the deep temporal features, the initial fusion features and the latest fusion features in the multimodal fusion features to obtain the residual features;

[0200] The risk prediction submodule is used to generate a ventricular tachycardia or ventricular fibrillation risk assessment value based on the residual features.

[0201] In some embodiments, the risk prediction submodule includes:

[0202] A static risk assessment unit, used for calculating a static risk assessment value based on residual characteristics;

[0203] A dynamic risk assessment unit, configured to calculate a dynamic risk assessment value based on temporal gradient characteristics;

[0204] The ventricular tachycardia and ventricular fibrillation risk output unit is used to generate a ventricular tachycardia and ventricular fibrillation risk assessment value according to the static risk assessment value and the dynamic risk assessment value.

[0205] Regarding the specific limitations of the ventricular tachycardia and ventricular fibrillation prediction device based on multimodal clinical data, please refer to the limitations of the ventricular tachycardia and ventricular fibrillation prediction method based on multimodal clinical data above, and will not be repeated here. The various modules in the above-mentioned ventricular tachycardia and ventricular fibrillation prediction device based on multimodal clinical data can be implemented in whole or in part through software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above-mentioned modules.

[0206] The present application also provides a computer device. In some embodiments, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for predicting ventricular tachycardia and ventricular fibrillation based on multimodal clinical data provided in any of the above embodiments can be implemented.

[0207] Furthermore, in some embodiments, the internal structure diagram of the computer device may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as structured clinical data and unstructured clinical data of the target patient. The specific stored data can also be found in the definitions in the above method embodiments. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for predicting ventricular tachycardia and ventricular fibrillation based on multimodal clinical data is implemented.

[0208] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0209] The present application also provides a computer-readable storage medium. In some embodiments, a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the method for predicting ventricular tachycardia and ventricular fibrillation based on multimodal clinical data provided in any of the above embodiments is implemented.

[0210] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0211] Those skilled in the art will appreciate that all or part of the processes in the above method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink), DRAM (SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0212] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0213] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for predicting ventricular tachycardia and ventricular fibrillation based on multimodal clinical data, characterized in that: The method comprises: Obtain structured and unstructured clinical data of target patients; Preprocessing the structured clinical data and the unstructured clinical data; The preprocessed structured clinical data and unstructured clinical data are converted into embedding vectors to obtain structured clinical features and unstructured clinical features; The structured clinical features and the unstructured clinical features are fused based on a bidirectional attention mechanism and a dynamic gating fusion mechanism to obtain a multimodal fusion feature; Extracting temporal features from the multimodal fusion features through temporal modeling; The risk of ventricular tachycardia and ventricular fibrillation is predicted according to the time series characteristics to obtain a ventricular tachycardia and ventricular fibrillation prediction result.

2. The method according to claim 1, wherein The structured clinical data includes multiple clinical indicators; the preprocessing includes feature alignment processing; The feature alignment process includes: Obtaining a predetermined importance score of each clinical indicator in predicting ventricular tachycardia and ventricular fibrillation; A weight coefficient is set for each clinical indicator according to the importance score of each clinical indicator; the size of the weight coefficient of the clinical indicator is positively correlated with the size of the importance score of the clinical indicator.

3. The method according to claim 1, wherein The structured clinical features and the unstructured clinical features are fused based on a bidirectional attention mechanism and a dynamic gating fusion mechanism to obtain multimodal fusion features, including: calculating an attention weight of the structured clinical feature to the unstructured clinical feature as a first attention weight, and generating a first feature according to the first attention weight and the unstructured clinical feature; calculating an inverse attention weight of the unstructured clinical feature to the structured clinical feature as a second attention weight, and generating a second feature according to the second attention weight and the structured clinical feature; The first feature and the second feature are dynamically mixed through a dynamic gated fusion mechanism to obtain a multimodal fusion feature.

4. The method according to claim 1, wherein Extracting temporal features from the multimodal fusion features through temporal modeling includes: Obtaining a pre-set attenuation factor value rule for the acute attack characteristics of ventricular tachycardia and ventricular fibrillation, and performing dynamic time attenuation processing on the multimodal fusion feature using the attenuation factor value rule to obtain an attenuated fusion feature; Perform time series modeling on the attenuated fusion features from the forward and backward directions respectively to obtain bidirectional splicing features; A multi-layer perceptron is used to dynamically generate convolution kernel parameters according to time, and the convolution kernel parameters are used to extract temporal features from the bidirectional splicing features.

5. The method according to claim 4, wherein The attenuation factor value selection rules include: When the difference between the current prediction time and the current data point time is less than or equal to 24 hours, the attenuation factor is set to the first threshold; When the difference between the current prediction time and the current data point time is greater than 24 hours and less than or equal to 72 hours, the attenuation factor is set to the second threshold; When the difference between the current prediction time and the current data point time is greater than 72 hours, the attenuation factor is taken as the third threshold; among the first threshold, the second threshold and the third threshold, the first threshold is the largest and the third threshold is the smallest.

6. The method according to claim 1, wherein The time series features include multiple single-point time series features corresponding to different time steps; The risk of ventricular tachycardia and ventricular fibrillation is predicted based on the time series characteristics, and the ventricular tachycardia and ventricular fibrillation prediction results are obtained, including: Calculating the time attention weight of each of the single-point time series features, and aggregating the single-point time series features included in the time series features according to the time attention weight of each of the single-point time series features and a preset time attenuation gain term to obtain a deep time series feature; Combining the deep temporal features, the initial fusion features and the latest fusion features in the multimodal fusion features to obtain residual features; A ventricular tachycardia and ventricular fibrillation risk assessment value is generated according to the residual features.

7. The method according to claim 6, wherein Generating a ventricular tachycardia and ventricular fibrillation risk assessment value according to the residual feature, including: Calculating a static risk assessment value based on the residual characteristics; Calculate dynamic risk assessment values ​​based on temporal gradient characteristics; A ventricular tachycardia and ventricular fibrillation risk assessment value is generated according to the static risk assessment value and the dynamic risk assessment value.

8. A device for predicting ventricular tachycardia and ventricular fibrillation based on multimodal clinical data, characterized in that: The device comprises: Clinical data acquisition module, used to obtain structured and unstructured clinical data of target patients; A preprocessing module, configured to preprocess the structured clinical data and the unstructured clinical data; An embedding module, which is used to convert the preprocessed structured clinical data and unstructured clinical data into embedding vectors to obtain structured clinical features and unstructured clinical features; A feature fusion module, configured to fuse the structured clinical features and the unstructured clinical features based on a bidirectional attention mechanism and a dynamic gated fusion mechanism to obtain a multimodal fusion feature; A time series modeling module, configured to extract time series features from the multimodal fusion features through time series modeling; The ventricular tachycardia and ventricular fibrillation prediction module is used to predict the risk of ventricular tachycardia and ventricular fibrillation according to the time series characteristics and obtain a ventricular tachycardia and ventricular fibrillation prediction result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images