A cardiovascular disease risk prediction method based on multi-modal data fusion

Abstract

This invention relates to the field of cardiovascular disease risk prediction technology, and particularly to a cardiovascular disease risk prediction method based on multimodal data fusion. The method includes: acquiring four types of multimodal data—genetic data, imaging data, electronic medical record data, and physiological time-series data—based on hospital systems and wearable devices; processing the four types of multimodal data separately using a multimodal feature extraction strategy to obtain medical record semantic feature vectors, time-series feature vectors, genetic feature vectors, and imaging feature vectors; fusing these vectors using a multi-level feature fusion strategy to obtain a multimodal fused feature vector; and inputting the multimodal fused feature vector into a pre-trained cardiovascular disease risk prediction model to output the cardiovascular disease risk prediction result. This invention enables the synergistic utilization of risk information from each modality while preserving its own clinical semantics, improving the effectiveness of integrating multi-source heterogeneous information in long-term cardiovascular disease risk prediction.

Description

Technical Field

[0001] This invention relates to the field of cardiovascular disease risk prediction technology, and in particular to a method for cardiovascular disease risk prediction based on multimodal data fusion. Background Technology

[0002] Cardiovascular disease is one of the leading causes of death among chronic diseases worldwide. Its pathogenesis involves complex interactions across multiple dimensions, including genetic factors, organic lesions, disease progression history, and immediate physiological state. With the advancement of precision medicine, researchers are attempting to integrate heterogeneous clinical information such as genetic data, medical imaging data, electronic medical records (EMR) data, and time-series physiological monitoring data into a single predictive framework to achieve a comprehensive assessment of patients' long-term cardiovascular event risks. However, these four types of data exhibit significant hierarchical differences in clinical temporal semantics: the genetic risk information encoded by genomic data remains stable throughout an individual's lifespan, representing a lifelong, unchanging genetic foundation; coronary CT images reflect organic pathological structures at specific cross-sectional moments; clinical events recorded in EMRs, such as diagnostic changes, medication adjustments, and surgical procedures, describe the longitudinal evolution of an individual's risk factors; and physiological time-series signals such as heart rate and blood pressure continuously collected by wearable devices reflect the patient's current dynamic physiological state. The inherent hierarchical relationship in semantic stability and temporal scale among these four types of data makes the effective fusion of multimodal data a core challenge in the field of long-term cardiovascular disease risk prediction.

[0003] In feature extraction from physiological time-series data, existing methods typically employ a fixed-time-window uniform aggregation of continuous time-series signals. This involves extracting statistical measures such as the mean or standard deviation of physiological measurements within a pre-defined equal-length window, using these as time-series features input into subsequent models. However, the physiological time-series signals of cardiovascular disease patients often exhibit significant non-stationary responses before and after clinical events (such as new diagnoses, medication dosage adjustments, and surgical interventions). Before an event, there may be a gradual accumulation of abnormal trends, while after the event, there may be delayed physiological responses or disease rebound. These high-information event response segments are treated equally with the low-information signals of the stable background period under a fixed-window uniform aggregation strategy, resulting in the most diagnostically valuable event trigger signals being diluted by a large amount of steady-state noise. Furthermore, when summarizing information from multiple clinical events in a patient's history, existing methods often assign the same aggregation weight to clinical events of different types (such as surgical interventions and daily medication adjustments) and at different time intervals (such as recent and long-term events), ignoring the differentiated contribution of the clinical severity and relevance of events to risk prediction, further diluting and weakening key event signals.

[0004] Chinese invention patent application CN202110486200.2 discloses a disease risk prediction method based on multimodal fusion. This patent uses structured data (such as laboratory test results and vital sign records) and unstructured text data (such as diagnostic records and medical orders) from electronic medical records as input. It employs a fully convolutional network (FCN) to extract features from the structured data and a pre-trained BERT model to extract semantic features from the text. The two types of features are then concatenated along a specified dimension and subjected to segmented pooling to obtain fused features. Finally, the disease risk prediction result is output through a fully connected layer and a Softmax classifier. To address the problem of imbalanced sample classes, the patent introduces a synthetic minority class oversampling technique (SMOTE) to generate synthetic minority class samples in the feature space to reduce the imbalance rate. However, the predictive ability of this patent is limited to the phased information presented in hospital visit records, and the constructed risk assessment perspective is relatively singular. It is difficult to comprehensively reflect the patient's comprehensive risk across multiple dimensions such as genetic background, organic lesions, and dynamic physiological state. Therefore, its prediction accuracy and clinical applicability for long-term prognosis of cardiovascular diseases remain significantly limited. Summary of the Invention

[0005] In view of this, the present invention provides a cardiovascular disease risk prediction method based on multimodal data fusion to solve the problems of single dimension of cardiovascular disease risk prediction and assessment and insufficient ability to comprehensively utilize multi-source heterogeneous clinical information in the prior art, thereby improving the accuracy of long-term risk prediction.

[0006] The technical solution of this invention is implemented as follows: This invention provides a method for predicting cardiovascular disease risk based on multimodal data fusion, comprising: S1. Acquire four types of multimodal data—genetic data, imaging data, electronic medical record data, and physiological time-series data—based on hospital systems and wearable devices; S2. A multimodal feature extraction strategy is used to process four types of multimodal data. For electronic medical record data, a pre-trained language model is used for semantic encoding and clinical event records are extracted to obtain medical record semantic feature vectors. For physiological time series data, a non-uniform feature extraction method based on clinical event anchors is used and processed through a dual heterogeneity-perceived weighted aggregation strategy to obtain time series feature vectors. For gene data and image data, encoding networks are used to process them to obtain genetic feature vectors and image feature vectors, respectively. S3. The genetic feature vector, image feature vector, medical record feature vector and time series feature vector are fused using a multi-level feature fusion strategy. The different modal features are conditionally modulated step by step along a preset semantic hierarchy, and a gated residual path is set between each modal feature. The output of the step-by-step conditional modulation and the output of the gated residual path are spliced ​​and normalized to obtain the multi-modal fused feature vector. S4. Input the multimodal fusion feature vector into the pre-trained cardiovascular disease risk prediction model and output the cardiovascular disease risk prediction result.

[0007] Based on the above technical solutions, preferably, in step S1, the gene data is allelic data of single nucleotide polymorphism sites from a gene database, the image data is coronary CT angiography image data from an image archiving and communication system, the electronic medical record data is electronic medical record text data from a hospital information system, including diagnostic records, medication adjustment records, surgical operation records and symptom descriptions, and the physiological time series data is physiological monitoring time series data from a wearable device, including at least continuous heart rate signals and continuous blood pressure signals.

[0008] Based on the above technical solutions, preferably, in step S2, the process of semantically encoding the electronic medical record data using a pre-trained language model and extracting clinical event records includes: de-identifying and segmenting the electronic medical record data; inputting each medical record text into a pre-trained BERT model to obtain the corresponding... The labeled hidden layer vectors; for all medical records. The vector is subjected to mean pooling and then mapped to a dimension through a linear projection layer. The semantic feature vector of the medical record is obtained. .

[0009] Based on the above technical solutions, preferably, in step S2, the extraction of the time-series feature vector includes: A1. Extract timestamped clinical event records from electronic medical record data. Each record includes an event timestamp and event type label. The extracted data... A collection of clinical events constitutes a set of clinical event records. ,in For the first The timestamp of each event The event type is labeled, corresponding to new diagnosis, drug activation or dosage adjustment, surgery or interventional procedure, respectively; A2. Assume that physiological time-series data includes At time step, at time step place The physiological measurement vector is For each clinical event Using event timestamps Define the forward window as the anchor point. and back window ,in ; A3. For each event Forward window calculation Dimensional physiological trend vector The detrending counterfactual effect is calculated based on the backward window. ,right Event construction noise normalization event response pattern to ; A4. A dual heterogeneity-perceived weighted aggregation strategy is used to weight and aggregate the feature vectors of each clinical event. The aggregated trend vector, offset vector, response pattern vector, and background physiological state vector are then concatenated and mapped to the dimension via a linear projection layer. The time-series feature vector is obtained. .

[0010] Based on the above technical solutions, preferably, the dual heterogeneity-aware weighted aggregation strategy in step A4 includes: defining severity weights based on event type. and exponential decay weights based on time relevance ,in The interval between the event and the predicted time point. Let be the time decay scale parameter; combine the event type severity weight and the time decay weight in product form and normalize to obtain the _th _ The joint aggregate weight of the events: ; Based on joint aggregation weight Trend vectors for each clinical event Offset vector and response mode vector Weighted aggregation is performed to obtain aggregated trend vectors and aggregated offset vectors, and the subset of intervention events is then analyzed. The weights are renormalized to obtain the aggregated response pattern vector.

[0011] Based on the above technical solutions, preferably, in step S2, processing the gene data using a coding network includes: performing one-hot encoding on each SNP site in the gene data, compressing the obtained genetic coding vector through a two-layer fully connected network, and then mapping it to a dimension through a linear projection layer. The genetic feature vector is obtained. The image data is processed using an encoding network, which includes: normalizing and denoising the image data before inputting it into a ResNet-50 convolutional neural network to obtain the feature vector of the last residual block output after global average pooling, and then mapping it to the dimension through a linear projection layer. The image feature vector is obtained. .

[0012] Based on the above technical solutions, preferably, step S3 specifically includes: S31. Following the stepwise conditional modulation main path along the clinical semantic hierarchy, the semantic stability of genetic feature vectors, imaging feature vectors, medical record semantic feature vectors, and temporal feature vectors decreases. Affine modulation is then applied to higher-level semantic features, conditioned on lower-level semantic features, to obtain the main path output. ; S32. Construct gated residual connections between genetic feature vectors and medical record semantic feature vectors, genetic feature vectors and temporal feature vectors, and image feature vectors and temporal feature vectors, respectively, to obtain residual path outputs. , , ; S33, Output the main path Output of three residual paths , , The multimodal fusion feature vector is obtained by concatenating along the feature dimension and then performing layer normalization.

[0013] Based on the above technical solutions, preferably, the step-by-step conditional modulation in step S31 includes: Setting the modulation level , respectively corresponding to genetic feature vectors As a first-level condition for image feature vectors Modulation, image feature vector As a second-level condition, the semantic feature vector of medical records Modulation, medical record semantic feature vector As a third-level condition pair, the time series feature vector Modulation; In the In the affine modulation process of the level condition features onto the target features, a modulation intensity coefficient is introduced: ; in For the first The learnable basic strength parameters of the level, For learnable hierarchical distance decay rate, For the sigmoid function, For the first The cumulative semantic hierarchy span of the modulation level; Calculate the first The original FiLM residual of the stage modulation And calculate the trust domain scaling factor: ; in For the first The relative norm budget parameter of the level, To prevent small positive numbers from being divided by zero, For the first The modulated target characteristics at the level; The modulation intensity coefficient With trust domain scaling factor Acting on the original FiLM residual And superimposed on the target feature to obtain the first Level modulation output: ; The modulation outputs at each stage are chained together: First-stage modulation output The conditional feature input for the second-level modulation, and the output for the second-level modulation. As the conditional feature input for the third-level modulation, the final third-level modulation output... As the main path output .

[0014] Based on the above technical solution, preferably, the gated residual connection in step S32 is implemented using a gating mechanism, wherein the gating coefficient of each gated residual connection is calculated by concatenating the two modal feature vectors at both ends of the gated residual connection and then passing them through a gating network; wherein, the gating coefficient of the gated residual connection between the genetic feature vector and the temporal feature vector is: ; The residual path output is: ; in For the learnable weight matrix of the gated network, For the corresponding bias vector, The characteristic transformation matrix, It is a completely one vector; and based on the same method respectively and Calculate the gating coefficients and obtain the residual path output. and .

[0015] Based on the above technical solutions, preferably, the pre-trained cardiovascular disease risk prediction model is an encoder model based on a Transformer structure, including L stacked Transformer encoder blocks. Each encoder block consists of a multi-head self-attention sublayer and a feedforward network sublayer, and each sublayer adopts the Pre-LayerNorm form. The model training uses a binary cross-entropy loss function with positive class weights, where the positive class sample weights... , and These represent the number of negative and positive samples in the training set, respectively; end-to-end joint training is performed using the AdamW optimizer, with a learning rate scheduling strategy employing linear warm-up followed by cosine decay. The present invention has the following advantages over the prior art: This invention explicitly models the clinical semantic differences among four modalities of data: genes, images, electronic medical records, and physiological time series. It constructs a complete processing chain from data acquisition and feature extraction to fusion prediction, solving the semantic confusion problem caused by the equal weighting of the four types of data in existing multimodal fusion methods. This allows the risk information of each modality to be used synergistically while retaining its own clinical semantics, thus improving the effectiveness of integrating multi-source heterogeneous information in long-term risk prediction of cardiovascular diseases.

[0016] This invention uses clinical event timestamps as anchors to extract non-uniformly structured features from physiological time-series data. It then performs weighted aggregation by normalizing the product of event type severity weights and time-recent decay weights. Compared to a fixed-window uniform aggregation strategy, this approach independently preserves physiological change signals triggered by clinical events (including pre-event trends, post-event detrending counterfactual effects, and intervention response patterns). At the same time, it gives higher-severity recent events that contribute significantly to prediction results a more reasonable weight in the aggregation results. This helps improve the ability of time-series feature vectors to represent cardiovascular risk-related physiological changes, thereby improving the model's accuracy in identifying MACE risk.

[0017] This invention employs a hierarchical conditional modulation mechanism along the clinical semantic hierarchy, supplemented by dual constraints of modulation intensity and trust domain. Outside the main modulation path, gated residual pathways are established for three non-adjacent modal pairs: genetic-medical record, genetic-temporal, and image-temporal. Hierarchical modulation allows each modal feature to perceive its semantic context during fusion. The dual constraints adaptively tighten with the semantic hierarchy span, reducing the risk of feature numerical distortion between distant semantic hierarchy combinations in chained modulation. The gated residual pathways preserve direct information channels for known cross-layer biological associations outside the main path. The gating mechanism adaptively determines the fusion intensity based on the correlation between two input features, avoiding forced mixing between unrelated modal pairs. The synergistic effect of these mechanisms ensures that the final multimodal fusion feature vector simultaneously contains hierarchical semantic calibration information and cross-layer direct association information, contributing to improved cardiovascular disease risk prediction models' comprehensive discrimination ability against multi-dimensional clinical risk factors. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the cardiovascular disease risk prediction method of the present invention; Figure 2 This is a flowchart of the multi-level feature fusion strategy of the present invention; Figure 3 This is a structural diagram of the cardiovascular disease risk prediction model of the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] like Figure 1 As shown, this invention provides a cardiovascular disease risk prediction method based on multimodal data fusion, comprising: S1. Acquire four types of multimodal data—genetic data, imaging data, electronic medical record data, and physiological time-series data—based on hospital systems and wearable devices; S2. A multimodal feature extraction strategy is used to process four types of multimodal data. For electronic medical record data, a pre-trained language model is used for semantic encoding and clinical event records are extracted to obtain medical record semantic feature vectors. For physiological time series data, a non-uniform feature extraction method based on clinical event anchors is used and processed through a dual heterogeneity-perceived weighted aggregation strategy to obtain time series feature vectors. For gene data and image data, encoding networks are used to process them to obtain genetic feature vectors and image feature vectors, respectively. S3. The genetic feature vector, image feature vector, medical record feature vector and time series feature vector are fused using a multi-level feature fusion strategy. The different modal features are conditionally modulated step by step along a preset semantic hierarchy, and a gated residual path is set between each modal feature. The output of the step-by-step conditional modulation and the output of the gated residual path are spliced ​​and normalized to obtain the multi-modal fused feature vector. S4. Input the multimodal fusion feature vector into the pre-trained cardiovascular disease risk prediction model and output the cardiovascular disease risk prediction result.

[0022] In one embodiment of the present invention, step S1 includes: obtaining the patient's single nucleotide polymorphism (SNP) allele genotype data from a gene database, denoted as... Coronary CT angiography (CTA) image data is acquired from a Picture Archiving and Communication System (PACS), denoted as... ; Obtain electronic medical record text data from the Hospital Information System (HIS), denoted as It includes structured and unstructured content such as diagnostic records, medication adjustment records, surgical procedure records, and symptom descriptions; physiological monitoring time-series data are acquired from wearable devices and denoted as... This includes at least continuous heart rate and continuous blood pressure signals. These four types of data form a progressive hierarchical relationship in clinical temporal semantics: Describing the underlying nature of lifelong genetic risks, Describe the organic pathological state of the current cross-section. Describe the longitudinal evolution trajectory of historical risk factors. Describe real-time dynamic physiological manifestations.

[0023] In one embodiment of the present invention, in step S2 Processing of genetic data includes: Each SNP locus is one-hot encoded. The resulting high-dimensional sparse genetic coding vector is compressed through a two-layer fully connected network (containing ReLU activation function) and then mapped to the desired dimension through a linear projection layer. , to obtain the genetic feature vector .

[0024] Furthermore, the processing of image data includes: After normalization and denoising preprocessing, the data is fed into a ResNet-50 convolutional neural network. The feature vector output from the last residual block, after global average pooling, is then mapped to the desired dimension via a linear projection layer. To obtain the image feature vector .

[0025] Furthermore, the processing of electronic medical record data includes: After de-identification and sentence segmentation, each medical record text is input into a pre-trained BERT model to obtain the corresponding... The labeled hidden vector serves as the semantic representation of that particular medical visit, and is applied to all medical records. The vector is subjected to mean pooling and then mapped to a dimension through a linear projection layer. The semantic feature vector of the medical record is obtained. .

[0026] Furthermore, temporal feature vectors The extraction is divided into two stages: the first stage extracts structured features of a single event response for each clinical event; the second stage performs dual heterogeneity-aware weighted aggregation on the feature vectors of all events, and the specific implementation includes: A1, from Extract clinical event records with clear timestamps. Each record includes an event timestamp and an event type label. Corresponding to three categories: new diagnosis, drug initiation or dosage adjustment, and surgery or interventional procedure, the extracted data will be... A set of clinical events constitutes a clinical event record: in For the first The timestamp of each event ( ), time unit and The sampling time axis is consistent.

[0027] A2. Assume physiological time series data Include At time step, at time step place The physiological measurement vector is ( For each clinical event. Using event timestamps Define the forward window as the anchor point before the event occurs. A set consisting of time steps: ; The back window is after the event occurs. A set consisting of time steps: ; in , This is the length of the forward view window. The asymmetry between the two, which refers to the length of the backward observation window, is based on considerations of cardiovascular clinical mechanisms: physiological states begin to change gradually before clinical events occur, while the physiological response triggered by clinical interventions (such as drug adjustments) is delayed, so the backward observation window should be longer. and These are learnable scalar hyperparameters that are optimized in a data-driven manner during the training phase.

[0028] A3. For each event Forward window calculation Dimensional physiological trend vector The detrending counterfactual effect is calculated based on the backward window. ,right Event construction noise normalization event response pattern to .

[0029] Specifically, for each event The forward window is used to calculate the linear regression slope for each dimension. Dimensional physiological trend vector: ; in The average of the forward window time steps. The vector represents the mean of the physiological signals in the forward window, and the division is an element-wise division. The components and Each dimension corresponds one-to-one, and its first... Each component )express No. The linear regression slope of a measurement index before the event. A positive value indicates that the corresponding physiological index shows a continuous upward trend before the event, while a negative value indicates a continuous downward trend.

[0030] Calculate the detrending counterfactual effect (offset vector). First, the regression intercept vector is calculated based on the linear trend before the event. ,in This represents element-wise multiplication; let the mean of the backward window time steps be $. The mean vector of the physiological signal in the backward window is Construct the counterfactual expected mean (i.e., the expected mean of a backward window if there is no event intervention and only the pre-event trend continues): ,in, Therefore, the detrending counterfactual effect is calculated: ; The Each component A positive value indicates that the actual mean of the indicator after the event is higher than the expected continuation of the trend, while a negative value indicates that it is lower than expected. The calculation of the detrended counterfactual effect eliminates the influence of the natural trend that existed before the event, and can more accurately reflect the true physiological effect of the clinical event.

[0031] Only for event types with explicit intervention semantics ( (i.e., drug adjustment and surgical intervention) to construct a noise-normalized event response pattern vector First, calculate the residual variance vector of the trend fit before the event (calculated dimension by dimension): ; Then, construct the noise-normalized event response pattern vector: ; in To prevent small positive numbers from being divided by zero. The When one component is positive, it indicates that the clinical intervention has successfully reversed the first [condition / condition]. The pre-event abnormal trend of the dimensional index (with treatment response); a negative value indicates failure to reverse (insufficient treatment response or disease progression). For newly diagnosed events ( Its essence is a record of disease state rather than an intervention, and it only uses trend vectors. and offset vector Not calculated To maintain the clinical semantic consistency of each feature quantity.

[0032] A4. A dual heterogeneity-perceived weighted aggregation strategy is used to weight and aggregate the feature vectors of each clinical event. The aggregated trend vector, offset vector, response pattern vector, and background physiological state vector are then concatenated and mapped to the dimension via a linear projection layer. The temporal feature vector is obtained. .

[0033] Specifically, we first define prior severity weights based on event type, as the three types of events differ in their cardiovascular prognostic significance: surgery or interventional procedures correspond to a patient's disease reaching a severity requiring direct intervention; new diagnoses correspond to the confirmation of organic lesions; and medication adjustments correspond to the routine management of risk factors. Based on this, we define the severity weights for each event type: ; in, Weights are assigned to the severity of surgical or interventional events. To add severity weights to the types of diagnostic events, The severity weights for drug activation or dosage adjustment events are determined by type; all three are learnable positive scalar parameters, ordered a priori. For initialization, optimization is performed during the training phase using a data-driven approach, allowing data to correct prior rankings. Next, exponentially decaying weights based on temporal relevance are defined, assuming... To predict the time point, the event The interval from the predicted time point is Define the time decay factor: ; in The time decay scale parameter is a learnable parameter that controls the rate at which the influence of historical events decays over time, with units consistent with the timeline of the event timestamps. The event type severity weight and the time decay weight are combined in a product form and then normalized to obtain the [missing parameter]. The joint aggregate weight of the events: ; pass This also reflects the severity of clinical intervention in the event. ) and time proximity ( Events that are "severe in type and recent in time" have the highest weight, and normalization ensures that... ,make The numerical scale is not affected by the total number of events. The impact.

[0034] Based on joint aggregation weight The feature vectors of each clinical event are grouped and weighted for aggregation. For all... The trend vector and offset vector of each event are aggregated by joint weights: ; Only for a subset of events with intervention semantics Aggregate the response pattern vectors according to their normalized weights: ; in This is a subset of the subscripts for interventional events (medication adjustments and surgical interventions). A subset of intervention events Event index within, For the first The joint aggregate weight of the events, To find the dummy indicator, traverse the subset of intervention events. All event indices within, For the first The joint aggregation weight of events, its definition is the same as... same, The sum of the joint aggregate weights of all events within the subset of intervention-type events is used as the normalization denominator. for In the subset of intervention events The renormalized weights within satisfy , For the first Noise-normalized event response pattern vectors for each type of intervention event To be according to The weighted aggregated response pattern vector. Additionally, a set of non-event background time steps is defined. Calculate the background physiological state vector This reflects the patient's baseline physiological level during non-event-stabilized periods. The aggregated trend vector... Aggregate offset vector Aggregate response mode vector and background physiological state vector After stitching, the image is mapped to the dimension via a linear projection layer. The temporal feature vector is obtained as follows: ; in For a learnable projection matrix, This indicates vector concatenation. It also carries four types of structured time-series features: pre-event trend information, post-event response information, intervention-type event response pattern information, and background baseline information.

[0035] After processing electronic medical record data, genetic data, image data, and time-series feature data as described above, four feature vectors are output. Dimensions are unified as These correspond to four clinical semantic levels, respectively.

[0036] The non-uniform feature extraction method based on clinical event anchors described above constructs an asymmetric observation window using clinical event timestamps as anchors. This allows for the structured decomposition of physiological changes before and after the event, and distinguishes between the feature semantics of diagnostic and interventional events. This enables high-information event response fragments to be independently represented, avoiding the problem of key event signals being diluted by the steady-state background in the fixed-window uniform aggregation strategy. The dual heterogeneity perception weighted aggregation, through the joint normalization of event type severity weights and time-recent decay weights, ensures that clinical events of different types and time distances receive weights that match their clinical contributions during aggregation, thus mitigating the weakening of key historical event signals by equal-weighted aggregation to some extent.

[0037] In one embodiment of the present invention, such as Figure 2 As shown, step S3 specifically includes: S31. Following the stepwise conditional modulation main path along the clinical semantic hierarchy, the semantic stability of genetic feature vectors, imaging feature vectors, medical record semantic feature vectors, and temporal feature vectors decreases. Affine modulation is then applied to higher-level semantic features, conditioned on lower-level semantic features, to obtain the main path output. .

[0038] Specifically, setting the modulation level These correspond to: Level 1, based on genetic feature vectors. For conditional features ( Image feature vectors For target features ( The second stage modulates the output of the first stage. For conditional features ( ), medical record semantic feature vector For target features ( The third stage, with output modulated by the second stage. For conditional features ( Temporal feature vectors For target features ( Semantic stability decreases from genetic to temporal, with each level applying affine modulation of more dynamic semantic features to a more stable semantic background, so that the fusion result at each level carries semantic background information from all levels below it.

[0039] In the In level modulation, affine scaling parameter vectors are generated through two lightweight linear networks. Translation parameter vector ,in For the first Level conditional eigenvectors, To generate a learnable weight matrix for a linear network with scaled parameter vectors, For the corresponding learnable bias vector, To generate a learnable weight matrix for a linear network with translation parameter vectors, For the corresponding learnable bias vector, . and Dynamically generated from conditional features, used for element-wise affine transformation of target features. Calculate the... The original FiLM residual of the modulation stage: ; in, For the first The modulated target feature vector at the level. For element-wise multiplication, The difference between the standard FiLM modulation output and the target feature represents the original modification of the target feature by affine modulation. However, directly using standard FiLM modulation without intensity constraints presents two problems: First, in the three-level chain structure, the semantic hierarchy span between the conditional signal and the modulated feature increases progressively. In the third-level modulation, the conditional signal contains lifelong unchanging genetic components, while the modulated target is an instantaneous dynamic physiological feature. The semantic hierarchy span between the two is the largest, and direct modulation easily introduces semantic confusion. Second, even with a small modulation intensity, if the modulation parameter value is too large, it can still lead to uncontrolled target feature values. To address this, a semantic hierarchy distance-aware modulation mechanism with trust domain projection constraints is introduced.

[0040] In the In the affine modulation process of the level condition features onto the target features, a modulation intensity coefficient is introduced: ; in For the first The learnable basic strength parameters of the level, For learnable hierarchical distance decay rate, For the sigmoid function, For the first The cumulative semantic hierarchy span of the modulation levels (0 for the first level, 1 for the second, and 2 for the third). When At that time, the third-level modulation strength satisfies That is, the modulation intensity of the instantaneous time-series features is adaptively constrained to be minimized.

[0041] Calculate the trust domain scaling factor: ; in For the first The relative norm budget parameter (learnable scalar) of the level controls the upper limit of the allowable modulation amplitude relative to the target feature. The proportion of norms To prevent small positive numbers from being divided by zero, For the first The modulated target features at the level of [level]. The modulation intensity coefficient. When When smaller, Trust domain constraints are not activated, and modulation is performed at its original strength; when Exceeding the budget limit hour, The original FiLM residual is scaled proportionally to ensure that the final modulation amplitude meets the requirements. .

[0042] Modulation intensity coefficient With trust domain scaling factor Acting on the original FiLM residual This is superimposed on the target feature to obtain the first... Level modulation output: ; The modulation outputs at each stage are chained together: First-stage modulation output As the conditional feature input for the second-level modulation, the second-level modulation output... As the conditional feature input for the third-level modulation, the final third-level modulation output... As the main path output It carries complete fusion information across four clinical semantic levels.

[0043] The three-level modulation corresponds to the following: Level 1 is as follows: For conditional pair Modulation revealed that, for the same degree of coronary artery stenosis, the corresponding probability of cardiovascular events differed under different genetic risk backgrounds. The modulation results were... Level 2 and above For conditional pair Modulation, the same historical medical records should have different weights given a combined background of known genetic risk level and degree of organic lesion; the modulation result is... Level 3 and above For conditional pair Modulation, where heart rate variability fluctuations of equal amplitude represent different immediate risk intensities across three layers of background: known genetic risk, organic lesions, and medical history. The modulation result is... The parameter matrix of all linear networks and bias vector ( All of these are learnable parameters, which are updated synchronously with the parameters of other modules during end-to-end training.

[0044] By modulation intensity coefficient Adaptive constraints are applied to modulation with large semantic differences across layers, based on the monotonically decreasing characteristic of semantic hierarchy span; trust domain scaling factor. By further limiting the absolute size of the modulation residual at the order of magnitude level, the two form a dual constraint, which achieves cross-layer semantic calibration while preserving the independent expressive ability of each modality feature, thereby improving the fusion robustness of chain modulation among modal combinations with large semantic heterogeneity.

[0045] S32. Construct gated residual connections between genetic feature vectors and medical record semantic feature vectors, genetic feature vectors and temporal feature vectors, and image feature vectors and temporal feature vectors, respectively, to obtain residual path outputs. , , .

[0046] Specifically, gated residual connections are implemented using a gating mechanism. The gating coefficient of each gated residual connection is calculated by concatenating the two modal feature vectors at both ends of the connection and then passing the result through a gating network. Taking the genetic-temporal direct pathway as an example, the gating coefficient of the gated residual connection between the genetic feature vector and the temporal feature vector is as follows: ; The residual path output is: ; in For the learnable weight matrix of the gated network, For the corresponding bias vector, The characteristic transformation matrix, It is a vector of all ones. This represents the concatenation of two feature vectors. It is the sigmoid activation function. This is element-wise multiplication. This corresponds to the direct genetic-temporal pathway output, reflecting the biological pathway by which gene variations directly affect cardiac electrophysiological temporal signals.

[0047] The genetic-pathological direct pathway (corresponding to the direct impact of gene polymorphism on disease history) and the imaging-temporal direct pathway (corresponding to the direct constraint relationship between coronary artery morphology and structure on immediate physiological fluctuations, such as the direct limitation of the degree of coronary artery stenosis on the amplitude of exercise heart rate response) adopt the exact same gated residual connection form and are based on the same method. and Calculate the gating coefficients and obtain the residual path output. and The specific formula is as follows: ; ; ; ; in For the learnable weight matrix of the corresponding gated network, For the corresponding bias vector The above parameters are all learnable parameters and are updated synchronously with the parameters of other modules during end-to-end training.

[0048] Gating coefficient element-wise value in The gating is adaptively determined by the concatenation of the two input features: when the semantic correlation between the two features is strong, the gating tends to fuse cross-layer information. When the correlation is close to 1, the output is mainly dominated by the transformed source features; when the correlation is weak, the gating tends to preserve the original target features. (Approaching 0), to avoid the forced mixing of cross-layer information without biological basis.

[0049] The aforementioned gated residual pathway mechanism compensates for the dilution effect of chain modulation on the direct biological association between non-adjacent semantic layers by establishing independent direct connection pathways for each group of non-adjacent semantic layers. The gate coefficient is adaptively determined by the two-path features, so that the information fusion strength of each pathway matches the cross-layer correlation of the current input features, avoiding the noise interference introduced by fixed weight fusion when the cross-layer semantic correlation is weak.

[0050] S33, Output the main path Output of three residual paths , , After concatenation along the feature dimension and layer normalization, the final multimodal fused feature vector is obtained: ; in This represents vector concatenation. The dimension is It also includes hierarchical semantic calibration information of the main path and cross-layer direct biological association information of the three residual pathways. LayerNorm normalizes the mean and variance of the concatenated vectors to eliminate numerical scale differences between the outputs of different pathways and ensure the stability of the input to the subsequent prediction model.

[0051] In one embodiment of the present invention, such as Figure 3 As shown, step S4 includes: fusing the multimodal feature vectors Input the data into a pre-trained cardiovascular disease risk prediction model and output the cardiovascular disease risk prediction results.

[0052] Specifically, the pre-trained cardiovascular disease risk prediction model is an encoder model based on a Transformer structure, including... The Transformer encoder blocks are stacked in layers. Each layer consists of a multi-head self-attention (MHSA) sublayer and a feedforward network (FFN) sublayer. Each sublayer adopts the Pre-LayerNorm form. First, it is mapped back to the dimension through a linear projection layer. ,get: ; in The learnable weight matrix of the linear projection layer. For the corresponding learnable bias vector, This is the initial hidden layer representation after projection. After being expanded into a single token sequence, it is sent in. A stacked Transformer encoder block, the first layer( The calculation process for ) is as follows: ; ; in For the first The input hidden layer representation of the layer, For layer normalization operations, MHSA adopts Each attention head has a key, query, and value dimension. ,in The feature dimension for each attention head; the FFN consists of two linear transformations and an intermediate GELU activation function, with a hidden layer dimension of... ; This is the intermediate representation after the MHSA sublayer. For the first Output of the layer encoder block. Final encoder output. The output of the Lth layer encoder block is used as a comprehensive representation vector of the patient's multimodal information and is passed into the classification head.

[0053] The encoder's final output The data is fed into a classification head, which consists of a linear transformation and a sigmoid activation function. ; in The learnable weight vector for the linear layer of the classification head. For the corresponding learnable bias scalar, It is the sigmoid activation function. This represents the probability of a patient experiencing a major adverse cardiovascular event (MACE, including myocardial infarction, stroke, and cardiac death) within the next 10 years, as predicted by the model. Exceeding the preset threshold (e.g.) When a patient is identified as high-risk, they are considered to be in a high-risk condition.

[0054] The model training method includes using whether the patient develops MACE within 10 years after the prediction time point as a binary label. (1 indicates MACE occurred, 0 indicates it did not occur) Supervised training is performed. The proportion of positive samples with MACE occurring within the next 10 years in the training dataset is typically significantly lower than that of negative samples. A binary cross-entropy loss function with positive class weights is used. ; in The number of samples in the training batch. For the first The label of each sample For the model to the first The predicted probability of a sample. The weights for positive class samples are determined by the ratio of positive to negative samples in the training set. ,in and The numbers represent the number of negative and positive samples in the training set, respectively. By increasing the loss weight of positive samples, the model can maintain sufficient detection sensitivity for MACE-positive patients even under class imbalance.

[0055] The AdamW optimizer was used (weight decay factor set to 1). End-to-end joint training is performed, with parameters of all modules (feature extraction networks in each sub-process of step S2, FiLM modulation network in step S31, gating network in step S32, and Transformer encoder and classification head in step S4) updated synchronously. The learning rate employs a linear warm-up plus cosine decay scheduling strategy: in the pre-training phase... Within each training step, the learning rate increases linearly from 0 to the peak learning rate. ; thereafter, following the cosine curve from Decay to minimum learning rate This stabilizes the multi-module parameter updates in the early stages of training and converges to local optima with fine-grained precision in later stages. Dropout is applied after each sublayer of the Transformer encoder and the fully connected layers of the feature extraction network. To prevent overfitting, the parameters of the pre-trained BERT model in step S2 are frozen during the initial training phase and then unfrozen for fine-tuning after other modules converge, avoiding large-scale pre-training parameters from damaging the early gradient signals of other modules. The AUROC on the validation set is used as the primary evaluation metric, employing an early stopping strategy: training is terminated when the validation set AUROC no longer improves after several consecutive training rounds, and the optimal model checkpoint with the optimal validation set AUROC is saved as the final model. Model performance is simultaneously evaluated using AUPRC, sensitivity, specificity, and calibration curves. AUPRC has higher discriminative power for detecting positive samples in class imbalance scenarios, while the calibration curve evaluates the predicted probability. Consistency with actual incidence rates ensures that the predicted results have reliable clinical quantitative significance.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for predicting cardiovascular disease risk based on multimodal data fusion, characterized in that, include: S1. Acquire four types of multimodal data—genetic data, imaging data, electronic medical record data, and physiological time-series data—based on hospital systems and wearable devices; S2. A multimodal feature extraction strategy is used to process the four types of multimodal data respectively. Among them, the electronic medical record data is semantically encoded using a pre-trained language model and clinical event records are extracted to obtain the medical record semantic feature vector. Physiological time-series data were processed using a non-uniform feature extraction method based on clinical event anchors and a dual heterogeneity-sensing weighted aggregation strategy to obtain time-series feature vectors; genetic data and image data were processed using encoding networks to obtain genetic feature vectors and image feature vectors, respectively. S3. The genetic feature vector, image feature vector, medical record feature vector and time series feature vector are fused using a multi-level feature fusion strategy. The different modal features are conditionally modulated step by step along a preset semantic hierarchy, and a gated residual path is set between each modal feature. The output of the step-by-step conditional modulation and the output of the gated residual path are spliced ​​and normalized to obtain the multi-modal fused feature vector. S4. Input the multimodal fusion feature vector into the pre-trained cardiovascular disease risk prediction model and output the cardiovascular disease risk prediction result.

2. The cardiovascular disease risk prediction method based on multimodal data fusion as described in claim 1, characterized in that, In step S1, the gene data is allelic data of single nucleotide polymorphism sites from a gene database, the image data is coronary CT angiography image data from an image archiving and communication system, the electronic medical record data is electronic medical record text data from a hospital information system, including diagnostic records, medication adjustment records, surgical operation records and symptom descriptions, and the physiological time series data is physiological monitoring time series data from a wearable device, including at least continuous heart rate signals and continuous blood pressure signals.

3. The cardiovascular disease risk prediction method based on multimodal data fusion as described in claim 1, characterized in that, Step S2, the process of semantically encoding the electronic medical record data using a pre-trained language model and extracting clinical event records, includes: de-identifying and segmenting the electronic medical record data; inputting the text of each medical record into a pre-trained BERT model to obtain the corresponding... The labeled hidden layer vectors; for all medical records The vector is subjected to mean pooling and then mapped to a dimension through a linear projection layer. The semantic feature vector of the medical record is obtained. .

4. The cardiovascular disease risk prediction method based on multimodal data fusion as described in claim 3, characterized in that, In step S2, the extraction of time-series feature vectors includes: A1. Extract timestamped clinical event records from electronic medical record data. Each record includes an event timestamp and event type label. The extracted data... A collection of clinical events constitutes a set of clinical event records. ,in For the first The timestamp of each event For the first The type labeling of each clinical event ,in This indicates a newly diagnosed case. This indicates that the medication has been initiated or the dosage has been adjusted. Indicates a surgical or interventional procedure; A2. Assume that physiological time-series data includes At time step, at time step place The physiological measurement vector is For each clinical event Using event timestamps Define the forward window as the anchor point. and back window ,in , The length of the forward view window. This is the length of the backward viewing window; A3. For each event Forward window calculation Dimensional physiological trend vector The detrending counterfactual effect, i.e., the offset vector, is calculated based on the backward window. ,right Construct noise-normalized event response pattern vectors ; A4. A dual heterogeneity-perceived weighted aggregation strategy is used to weight and aggregate the feature vectors of each clinical event. The aggregated trend vector, offset vector, response pattern vector, and background physiological state vector are then concatenated and mapped to the dimension via a linear projection layer. The time-series feature vector is obtained. .

5. The cardiovascular disease risk prediction method based on multimodal data fusion as described in claim 4, characterized in that, The dual heterogeneity-aware weighted aggregation strategy in step A4 includes: defining severity weights based on event type. and exponential decay weights based on time relevance ,in The interval between the k-th event and the predicted time point. For time decay scale parameters, To predict the time point, the severity weight of the event type and the time decay weight are combined in product form and then normalized to obtain the first... The joint aggregate weight of the events: in j To obtain the summation dummy index, based on joint aggregation weights Trend vectors for each clinical event Offset vector and response mode vector Weighted aggregation is performed to obtain aggregated trend vectors and aggregated offset vectors, and the subset of intervention events is then analyzed. The weights are renormalized to obtain the aggregated response pattern vector.

6. The cardiovascular disease risk prediction method based on multimodal data fusion as described in claim 1, characterized in that, In step S2, processing the gene data using a coding network includes: performing one-hot encoding on each SNP site in the gene data; compressing the obtained genetic coding vector through a two-layer fully connected network; and then mapping it to a dimension through a linear projection layer. The genetic feature vector is obtained. The image data processing using an encoding network includes: normalizing and denoising the image data before inputting it into a ResNet-50 convolutional neural network to obtain the feature vector of the last residual block output after global average pooling, which is then mapped to the dimension via a linear projection layer. The image feature vector is obtained. .

7. The cardiovascular disease risk prediction method based on multimodal data fusion as described in claim 1, characterized in that, Step S3 specifically includes: S31. Following the stepwise conditional modulation main path along the clinical semantic hierarchy, the semantic stability of genetic feature vectors, imaging feature vectors, medical record semantic feature vectors, and temporal feature vectors decreases. Affine modulation is then applied to higher-level semantic features, conditioned on lower-level semantic features, to obtain the main path output. ; S32. Construct gated residual connections between genetic feature vectors and medical record semantic feature vectors, genetic feature vectors and temporal feature vectors, and image feature vectors and temporal feature vectors, respectively, to obtain residual path outputs. , , ; S33, Output the main path Output of three residual paths , , The multimodal fusion feature vector is obtained by concatenating along the feature dimension and then performing layer normalization.

8. The cardiovascular disease risk prediction method based on multimodal data fusion as described in claim 7, characterized in that, The step-by-step conditional modulation in step S31 includes: Setting the modulation level , respectively corresponding to genetic feature vectors As a first-level condition for image feature vectors Modulation, image feature vector As a second-level condition, the semantic feature vector of medical records Modulation, semantic feature vector of medical records As a third-level condition pair, the time series feature vector Modulation; In the In the affine modulation process of the level condition features onto the target features, a modulation intensity coefficient is introduced: ； in For the first The learnable basic strength parameters of the level, For learnable hierarchical distance decay rate, For the sigmoid function, For the first The cumulative semantic hierarchy span of the modulation level; Calculate the first The original FiLM residual of the stage modulation And calculate the trust domain scaling factor: ； in For the first The relative norm budget parameter of the level, To prevent small positive numbers from being divided by zero, For the first The modulated target characteristics at the level; The modulation intensity coefficient With trust domain scaling factor Acting on the original FiLM residual And superimposed on the target feature to obtain the first Level modulation output: ； The modulation outputs at each stage are chained together: First-stage modulation output The conditional feature input for the second-level modulation, and the output for the second-level modulation. As the conditional feature input for the third-level modulation, the final third-level modulation output... As the main path output .

9. The cardiovascular disease risk prediction method based on multimodal data fusion as described in claim 7, characterized in that, In step S32, the gated residual connections are implemented using a gating mechanism. The gating coefficient of each gated residual connection is calculated by concatenating the two modal feature vectors at both ends of the connection and then passing the result through a gating network. Specifically, the gating coefficient of the gated residual connection between the genetic feature vector and the temporal feature vector is: ； The residual path output is: ； in, This is the gating coefficient vector. For the learnable weight matrix of the gated network, This represents the learnable bias vector corresponding to the gated network. Source features The characteristic transformation matrix to be transformed. It is a vector of all ones. This is the output of the genetic-temporal direct pathway. For element-wise multiplication; and based on the same method respectively and Calculate the gating coefficients and obtain the residual pathway output and the genetic-medical record direct pathway output. With image-time direct path output .

10. The cardiovascular disease risk prediction method based on multimodal data fusion as described in claim 1, characterized in that, The pre-trained cardiovascular disease risk prediction model is a Transformer-based encoder model, comprising L stacked Transformer encoder blocks. Each encoder block consists of a multi-head self-attention sublayer and a feedforward network sublayer, with each sublayer employing a Pre-LayerNorm structure. Model training uses a binary cross-entropy loss function with positive class weights, where the positive class sample weights... , and These represent the number of negative and positive samples in the training set, respectively; the AdamW optimizer is used for end-to-end joint training, and the learning rate adopts a scheduling strategy of linear warm-up plus cosine decay.

Images