Breast cancer cardiotoxicity risk prediction system based on multi-modal data fusion

Abstract

The present application relates to a breast cancer cardiotoxicity risk prediction system based on multi-modal data fusion. The system collects multi-source heterogeneous medical data such as electronic health records, dynamic electrocardio monitoring, echocardiogram video stream, chemotherapy medication time series data and pathological section images in real time through a distributed architecture, and optimizes storage efficiency by adopting a hierarchical compression strategy. The system maps the risk index to a three-dimensional heart heat map, displays the key feature contribution degree, and establishes a dynamic risk knowledge graph to reveal the toxicity occurrence path. The system deploys an incremental learning mechanism to realize continuous optimization of the model, ensures data security through differential privacy and federated learning, sets up a real-time early warning feedback loop, triggers clinical intervention when the risk exceeds the threshold, and dynamically adjusts the model weight according to the feedback of the physician, thereby significantly improving the prediction accuracy and clinical applicability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical big data, and particularly relates to a breast cancer cardiotoxicity risk prediction system based on multi-modal data fusion. BACKGROUND

[0002] Early prediction of breast cancer chemotherapy-related cardiotoxicity mainly relies on single modal clinical indicators (such as left ventricular ejection fraction) or static pathological features, which is difficult to fully capture the dynamic correlation between drug metabolism kinetics, cardiac electrophysiological activity and tissue microstructure damage. Traditional methods lack effective fusion mechanism for multi-source heterogeneous medical data, resulting in that the spatio-temporal interaction relationship between key biological signals is not fully utilized, which limits the sensitivity and specificity of the prediction model.

[0003] The existing system has significant limitations in data processing: the spatio-temporal feature heterogeneity of electrocardiogram and ultrasound data hinders cross-modal alignment, the deep microstructure information of pathological images is not fully extracted, and the individual differences of chemotherapy drug metabolism are not quantified and integrated. At the same time, the traditional architecture cannot support real-time joint analysis of streaming multi-modal data, and the distributed storage and computing efficiency is low, which is difficult to meet the timeliness requirements of clinical dynamic risk monitoring.

[0004] Current clinical decision-making faces double challenges: on the one hand, the visualization of cardiotoxicity risk is insufficient, and there is a lack of three-dimensional anatomical positioning and feature contribution explanation; on the other hand, the prediction model lacks incremental learning and feedback optimization mechanism, and cannot evolve continuously with the accumulation of diagnosis and treatment data. In addition, there are privacy protection defects in cross-institutional data collaboration, which restricts the feasibility of multi-center joint modeling, and ultimately affects the accuracy and clinical applicability of risk early warning. SUMMARY

[0005] To achieve the above purpose, the present application provides the following technical scheme:

[0006] According to the first aspect of the present application, the present application claims a breast cancer cardiotoxicity risk prediction system based on multi-modal data fusion, comprising:

[0007] A distributed multi-modal data acquisition module: real-time acquisition of patient's electronic health records (EHR), dynamic electrocardiogram monitoring data, echocardiogram video stream, chemotherapy drug dosage time series data and pathological section images through medical Internet of Things devices, and storage in a distributed columnar database in the form of streaming data;

[0008] A multi-modal feature joint extraction engine: inter-frame optical flow feature extraction is performed on the echocardiogram video stream to generate a dynamic vector of cardiac contraction function, 3D convolutional neural network is used to extract tissue microstructure topological features from the pathological section images, pharmacokinetic decay curve is constructed based on the chemotherapy drug dosage time series data, and area under the curve feature is calculated;

[0009] spatiotemporal feature fusion module: use graph neural network GNN to align the heterogeneous features of the spatiotemporal topological features of the dynamic electrocardiogram monitoring data, the dynamic vector of cardiac contraction function, and the topological features of tissue microstructure, and generate a fusion feature tensor;

[0010] dynamic risk prediction model: the input layer receives the fusion feature tensor and the pharmacokinetic characteristics, processes the time series dependence relationship through the gated recurrent unit GRU layer, dynamically weights the multi-modal features using the multi-head self-attention mechanism, and the output layer generates the cardiotoxicity risk index and the risk trend curve in the next 30 days;

[0011] risk visualization interface: map the risk index to a heat map of a three-dimensional heart model, and record the prediction results on a blockchain notarization platform.

[0012] Further, the distributed multi-modal data acquisition module adopts a hierarchical data compression strategy:

[0013] The echocardiogram video stream is compressed using H.265 encoding and key frame difference compression, and only the motion difference data between key frames is retained;

[0014] The dynamic electrocardiogram monitoring data is compressed based on wavelet transform, retaining 0.5-40Hz frequency band features, filtering out high-frequency noise, and retaining electrocardiogram waveform core frequency band features;

[0015] In the distributed storage layer, a time series partition index of columnar storage is established, which is double-dimensionally fragmented according to patient ID + chemotherapy cycle, and is stored in segments according to the patient ID combined with the chemotherapy cycle, accelerating data retrieval in a specific treatment phase.

[0016] Further, the multi-modal feature joint extraction engine includes an imageomics feature enhancement unit:

[0017] The imageomics feature enhancement unit extracts the contrast and entropy features of the gray level co-occurrence matrix GLCM after superpixel segmentation of the pathological section image, first segments the superpixel region, and extracts the texture contrast and complexity indicators;

[0018] The pre-trained ResNet-50 network extracts deep semantic features of cells, and the traditional imageomics features and deep learning features are fused. After traditional texture features and deep features are fused, principal component analysis PCA dimensionality reduction is performed to eliminate redundant information.

[0019] Further, the spatiotemporal feature fusion module uses a heterogeneous feature alignment algorithm:

[0020] The graph isomorphism network GIN is used to model the electrocardiogram lead topology as an unweighted graph, model the electrocardiogram lead position as a graph node, and project the ultrasound dynamic vector to the corresponding electrocardiogram node.

[0021] mapping the dynamic vector of the echocardiogram to a graph node space by a feature projection matrix, aggregating cross-modal node features using a graph attention mechanism GAT, calculating the correlation strength between cross-modal nodes using a graph attention mechanism, and aggregating features.

[0022] Further, the dynamic risk prediction model comprises an incremental learning mechanism:

[0023] triggering model updating when the amount of new patient data reaches a threshold, and using an elastic weight consolidation EWC algorithm to prevent catastrophic forgetting;

[0024] using elastic weight consolidation technology to retain old knowledge and prevent new training from covering historical patterns;

[0025] parallelizing parameter fine-tuning on a Spark cluster, with an update cycle of less than 24 hours.

[0026] Further, the system sets up a real-time early warning feedback loop:

[0027] When the risk index exceeds the clinical threshold, an early warning is automatically pushed to the attending physician's terminal. The feedback loop receives the physician's labeled false positive or false negative data, and dynamically adjusts the weight distribution ratio of the GRU layer and the attention layer, reduces the weight of noise features, and enhances the influence of key features.

[0028] Further, the pharmacokinetic feature calculation introduces individualized correction factors:

[0029] Based on the CYP3A4 enzyme active site variation in patient genomic data, adjust the curve decay coefficient, combine the kidney function index eGFR to correct the drug clearance rate parameter, and output the individualized drug time area under the curve AUC feature;

[0030] Adjust the drug metabolism rate parameter according to the patient's genetic test results, and combine the kidney function index to correct the calculation of drug clearance rate.

[0031] Further, the risk visualization interface realizes multi-dimensional correlation analysis:

[0032] Superimposed display of myocardial strain rate abnormal areas on a three-dimensional heart model, correlation display of the structural similarity of the current chemotherapy regimen and historical high-risk regimens, generation of an interactive decision tree explanation graph, and display of key feature contribution ranking.

[0033] Further, the system deploys a differential privacy protection mechanism:

[0034] Add Laplace noise before feature fusion, use a federated learning architecture, train local models at each hospital, and only upload model gradients from the hospital local node. Through homomorphic encryption, the risk index calculation is processed in ciphertext.

[0035] Further, the system further comprises:

[0036] A dynamic risk knowledge graph is established, nodes of the dynamic risk knowledge graph include four types of entities of patients, drugs, toxicity phenotypes and biomarkers, and four types of entity networks corresponding to the four types of entities are constructed: patients→drugs→toxicity phenotypes→biomarkers.

[0037] The edge relationship defines the causal path probability of the chemotherapy scheme-heart damage, defines the causal relationship chain of the chemotherapy scheme and the heart damage, and predicts the unobserved toxicity path based on the graph embedding technology TransE.

[0038] The present application relates to a breast cancer cardiotoxicity risk prediction system based on multi-modal data fusion. The system collects real-time electronic health records, dynamic electrocardiogram monitoring, echocardiogram video streams, chemotherapy medication time series data and pathological section images and other multi-source heterogeneous medical data through a distributed architecture, and optimizes storage efficiency using a hierarchical compression strategy. The system maps the risk index to a three-dimensional heart heat map, displays the key feature contribution, and establishes a dynamic risk knowledge graph to reveal the toxicity path. The system deploys an incremental learning mechanism to continuously optimize the model, ensures data security through differential privacy and federated learning, sets up a real-time warning feedback loop, triggers clinical intervention when the risk exceeds the threshold, and dynamically adjusts the model weight according to the feedback of the physician, significantly improving the prediction accuracy and clinical applicability. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The structure module diagram of a breast cancer cardiotoxicity risk prediction system based on multi-modal data fusion claimed by the embodiments of the present application;

[0040] Figure 2 The workflow diagram of a breast cancer cardiotoxicity risk prediction system based on multi-modal data fusion claimed by the embodiments of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0042] The terms "first", "second", "third", etc. in the present application are only for descriptive purpose and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second", "third" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise explicitly and specifically limited. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present application are only used to explain the relative position relationship, movement condition, etc. between components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0043] Reference herein to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily mutually exclusive of other embodiments. It is explicitly and implicitly understood that the embodiments described herein can be combined with other embodiments.

[0044] According to the first embodiment of the present application, the present application claims a breast cancer cardiotoxicity risk prediction system based on multi-modal data fusion, referring to Figure 1 and Figure 2 , comprising:

[0045] Distributed multi-modal data acquisition module: real-time acquisition of patient's electronic health record (EHR), dynamic electrocardio monitoring data, echocardiogram video stream, chemotherapy drug dosage time series data and pathological section image through medical Internet of Things device, and storage in distributed columnar database in the form of streaming data;

[0046] Multi-modal feature joint extraction engine: inter-frame optical flow feature extraction is performed on the echocardiogram video stream to generate a dynamic vector of cardiac contraction function, 3D convolutional neural network is used to extract tissue microstructure topological features from the pathological section image, pharmacokinetic decay curve is constructed based on the chemotherapy drug dosage time series data, and area under curve feature is calculated;

[0047] The spatio-temporal feature fusion module: uses a graph neural network (GNN) to align the heterogeneous features of the spatio-temporal topological features of the dynamic electrocardiogram monitoring data, the dynamic vector of cardiac contraction function, and the topological features of tissue microstructure, and generates a fusion feature tensor;

[0048] The dynamic risk prediction model: the input layer receives the fusion feature tensor and the pharmacokinetic characteristics, processes the time sequence dependence relationship through a gated recurrent unit (GRU) layer, dynamically weights the multi-modal features using a multi-head self-attention mechanism, and generates a cardiotoxicity risk index and a 30-day risk trend curve at the output layer;

[0049] The risk visualization interface: maps the risk index to a heat map of a three-dimensional heart model and records the prediction results on a blockchain notarization platform.

[0050] Further, the distributed multi-modal data acquisition module adopts a hierarchical data compression strategy:

[0051] The echocardiogram video stream is compressed using H.265 encoding and key frame difference compression, and only the motion difference data between key frames is retained;

[0052] The dynamic electrocardiogram monitoring data is compressed using a wavelet transform-based lossy compression method, which retains the 0.5-40 Hz frequency band features, filters out high-frequency noise, and retains the core frequency band features of the electrocardiogram waveform;

[0053] A time series partition index of columnar storage is established in the distributed storage layer, which is double-dimensionally sliced according to patient ID + chemotherapy cycle, and is stored in segments according to the patient ID combined with the chemotherapy cycle to accelerate data retrieval in a specific treatment phase.

[0054] In this embodiment, the data acquisition implementation details include medical Internet of Things device configuration, using a Philips MX450 ultrasound device to collect 1080p resolution video stream at a rate of 30 frames per second, and transmitting through a dedicated medical-grade WiFi 6 network; a GE MAC 5500 electrocardiogram monitor records 12-lead data at a sampling frequency of 500 times per second, and generates a compressed data packet every 5 minutes; a chemotherapy infusion pump is equipped with an RFID sensor that records the drug name, dose, and infusion rate in real time, with millisecond-level timestamp accuracy;

[0055] For ultrasound video processing, the key frame difference technique is used, with a key frame selected every 0.5 seconds, and only the motion vector data between adjacent frames is stored, reducing the data volume of a single examination from 3.2 GB to 864 MB;

[0056] The electrocardiogram data compression retains the 0.5-40 Hz clinically relevant frequency band through the wavelet transform filter, eliminates high-frequency noise above 50 Hz, and ensures the integrity of the QRS complex form; in the storage optimization, a double-dimensional index is established in the Apache Cassandra database, first hashed according to the patient ID, and then partitioned according to the chemotherapy cycle range, achieving millisecond query of billion-level records.

[0057] The compressed data quality evaluation is performed on the clinical verification data, and the blind test by 3 cardiologists shows that the consistency Kappa value of the electrocardiogram waveform diagnosis after compression is 0.92; on an 8-node cluster, the average response time for searching the data of the 3rd cycle of a specific patient is 0.87 seconds, and the standard deviation is 0.12 seconds.

[0058] Further, the multi-modal feature joint extraction engine includes an imageomics feature enhancement unit:

[0059] The imageomics feature enhancement unit extracts the contrast and entropy features of the gray level co-occurrence matrix GLCM after superpixel segmentation of the pathological section image, first segments the superpixel region, and extracts the texture contrast and complexity indicators;

[0060] The pre-trained ResNet-50 network extracts deep semantic features of cells, and the traditional imageomics features and deep learning features are fused. After the fusion of traditional texture features and deep features, principal component analysis PCA dimension reduction is performed to eliminate redundant information.

[0061] In this embodiment, the pathological feature extraction is performed for superpixel processing. The pathological section scanning is a 40-fold resolution 20000x20000 pixel full field image. The adaptive grid segmentation technology is used to generate 5000+ superpixel regions, ensuring that each region contains 10-15 complete cells. 22 kinds of texture features are extracted from each superpixel, including the contrast and entropy of the gray level co-occurrence matrix;

[0062] In the deep feature fusion, a pre-trained deep neural network is used to extract the nuclear morphology features of cells. The traditional texture features and 2048-dimensional deep features are spliced, and the dimension is reduced to 256-dimensional through principal component analysis, retaining 95% of the original information amount;

[0063] In the ultrasound video processing in the dynamic analysis of cardiac function, the motion trajectories of 200 endocardial feature points are tracked; in the pharmacokinetic modeling, the area under the curve is calculated based on the patient's weight and liver and kidney function; the feature extraction system has an AUC of 0.89 in the TCGA-BRCA data set to distinguish malignant tissues as a verification result.

[0064] Further, the spatio-temporal feature fusion module uses a heterogeneous feature alignment algorithm:

[0065] The electrocardiogram lead topology is modeled as an unweighted graph using graph isomorphism network (GIN), and the electrocardiogram lead position is modeled as a graph node. The ultrasound dynamic vector is projected to the corresponding electrocardiogram node.

[0066] The dynamic vector of the echocardiogram is mapped to the graph node space through a feature projection matrix, the cross-modal node features are aggregated using a graph attention mechanism (GAT), the correlation strength between cross-modal nodes is calculated using the graph attention mechanism, and the features are aggregated.

[0067] In this embodiment, 18 anatomical nodes are defined during the topology construction of the graph neural network: 12 standard electrocardiogram lead positions and 6 echocardiogram key regions. A node space relationship matrix is established, and the connection weight of nodes with a distance of less than 5 cm is set to 0.9.

[0068] During the cross-modal alignment process, the ultrasound dynamic vector is projected to the corresponding anatomical node, and a three-layer graph attention network is used to learn the feature correlation weight and output a 512-dimensional fusion tensor.

[0069] After performance verification, the fusion feature has a detection sensitivity of 91.2% for early cardiotoxicity, and the F1 score is increased from 0.71 to 0.89, achieving feature discrimination improvement.

[0070] Further, the dynamic risk prediction model comprises an incremental learning mechanism:

[0071] When the amount of new patient data reaches a threshold, the model is updated, and the elastic weight consolidation (EWC) algorithm is used to prevent catastrophic forgetting.

[0072] The elastic weight consolidation technology is used to retain old knowledge and prevent new training from covering historical patterns.

[0073] Parameter fine-tuning is performed in parallel on a Spark cluster, and the update cycle is less than 24 hours.

[0074] Further, the system sets a real-time warning feedback loop:

[0075] When the risk index exceeds the clinical threshold, an early warning is automatically pushed to the attending physician terminal. The feedback loop receives the doctor's labeled false positive or false negative data, and dynamically adjusts the weight distribution ratio of the GRU layer and the attention layer, reduces the weight of noise features, and enhances the influence of key features.

[0076] In this embodiment, the threshold is set when the warning system is in operation. When the risk index is greater than 0.82, an orange warning is triggered, and when the risk index is greater than 0.90, a red warning is triggered. For the feedback mechanism, after the doctor marks a false positive, the system automatically reduces the weight of noise features. The effect is verified by reducing the false positive rate from 23.1% to 6.3% within 3 months.

[0077] For the construction of a heart model in three-dimensional visualization, a personalized three-dimensional model is generated based on CT scans of a patient; a heat map mapping marks the myocardial strain rate <-18% area as red; a decision tree explanation shows the TOP3 risk factors and their contribution weights; and the clinical verification doctor decision compliance rate reaches 93.7%.

[0078] Further, the pharmacokinetic characteristic calculation introduces an individualized correction factor:

[0079] Based on the CYP3A4 enzyme active site variation in the patient's genomic data, the curve decay coefficient is adjusted, the drug clearance rate parameter is corrected in combination with the renal function index eGFR, and the individualized area under the drug time curve AUC feature is output;

[0080] According to the patient's genetic test results, the drug metabolism rate parameter is adjusted, and the drug clearance rate is corrected in combination with the renal function index.

[0081] In this embodiment, the prediction model architecture includes an input layer that receives 512-dimensional fusion features + individualized pharmacokinetic parameters, a time series processing layer that uses a 128-unit gated recurrent network to process continuous 30-day data, and an attention mechanism that dynamically allocates feature weights, such as a chemotherapy dose feature weight of 0.35. The output layer generates a 0-1 risk index and a daily risk value for the next 30 days.

[0082] For individualized pharmacokinetic correction, the CYP3A4 slow metabolism genotype reduces drug clearance rate by 37%, and the drug half-life is prolonged by 1.8 times when eGFR <60ml / min, and the prediction error of the area under the corrected curve is reduced to 6.7%;

[0083] In the incremental learning mechanism, 500 new patient data are accumulated and updated to trigger, lock the key parameter change range ±15% to perform elastic weight solidification, and a 20-node Spark cluster completes model optimization and distributed update in 18.7 minutes.

[0084] Further, the risk visualization interface realizes multi-dimensional correlation analysis:

[0085] The myocardial strain rate abnormal area is displayed on the three-dimensional heart model, the structural similarity between the current chemotherapy regimen and the historical high-risk regimen is displayed, an interactive decision tree explanation diagram is generated, and the key feature contribution ranking is displayed.

[0086] Further, the system deploys a differential privacy protection mechanism:

[0087] Laplace noise is added before feature fusion, a federated learning architecture is used, local training models are trained in each hospital, and only model gradients are uploaded by hospital local nodes. Through homomorphic encryption, the ciphertext processing of the risk index calculation is realized.

[0088] Further, the system further comprises:

[0089] A dynamic risk knowledge graph is established, nodes of the dynamic risk knowledge graph include four types of entities of patients, drugs, toxicity phenotypes and biomarkers, and four types of entity networks are constructed: patients→drugs→toxicity phenotypes→biomarkers.

[0090] The edge relationship defines the causal path probability of the chemotherapy scheme-heart injury, defines the causal relationship chain of the chemotherapy scheme and the heart injury, and predicts the unobserved toxicity path based on the graph embedding technology TransE.

[0091] In the embodiment, noise injection is performed in the privacy protection implementation, random disturbance conforming to medical privacy standards is added before feature fusion, a federated learning architecture is adopted, only encrypted model parameters are shared by 5 hospitals, and the prediction difference is less than 0.03 in the privacy protection mode.

[0092] For knowledge graph construction, four types of nodes of entities of patients, chemotherapy drugs, toxicity phenotypes and biomarkers are defined, a causal chain of "paclitaxel→ventricular premature beat→troponin increase" is established through relationship modeling, and the accuracy rate of predicting unobserved toxicity path is 92.4% in path prediction.

[0093] The whole system verification data of the application are shown in Table 1.

[0094] Table 1: Prediction performance table

[0095]

[0096] In the clinical effect, the incidence of cardiac events in the early warning group of the prospective cohort (n=216) is 6.1%, and the incidence of cardiac events in the control group is 19.2%, and in the doctor satisfaction survey, 94.3% of doctors think that the system effectively assists clinical decision-making.

[0097] In the system performance of the embodiment, the end-to-end delay is 8.7 seconds, from data input to risk output, the continuous 3-month operation availability is 99.2%, the resource consumption is less than 4GB of single processing peak memory and less than 75% of CPU occupancy, all data come from FDA approved clinical trials, conform to the HIPAA privacy standard, and pass the IRB ethical review.

[0098] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the division of the units is only a logical function division, and there can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, or the among different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0099] In addition, each function unit in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist alone physically, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit. The above is only an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

[0100] The specific embodiments of the application are described in detail above, but they are only examples. The present application is not limited to the specific embodiments described above. Any equivalent modification or substitution to the application by those skilled in the art is also within the scope of the present application. Therefore, any equivalent transformation and modification, improvement, etc. made without departing from the spirit and principle range of the present application should be included in the scope of the present application.

Claims

1. A breast cancer cardiotoxicity risk prediction system based on multimodal data fusion, characterized in that, include: Distributed multimodal data acquisition module: Real-time acquisition of patients' electronic health records (EHR), dynamic electrocardiogram monitoring data, echocardiogram video streams, chemotherapy drug dosage time-series data, and pathological slide images through medical IoT devices, and storage of these data in a distributed columnar database in the form of streaming data; Multimodal feature joint extraction engine: extracts inter-frame optical flow features from the echocardiogram video stream to generate a dynamic vector of cardiac contractile function, uses a 3D convolutional neural network to extract tissue microstructure topological features from the pathological slice images, constructs a pharmacokinetic decay curve based on the chemotherapy drug dosage time series data, and calculates the area under the curve feature. Spatiotemporal feature fusion module: Using graph neural network (GNN), the spatiotemporal topological features, dynamic vector of cardiac contraction function, and tissue microstructure topological features of the dynamic electrocardiogram monitoring data are heterogeneously aligned to generate a fused feature tensor; Dynamic risk prediction model: The input layer receives fused feature tensors and pharmacokinetic features, processes temporal dependencies through a gated recurrent unit (GRU) layer, dynamically weights multimodal features using a multi-head self-attention mechanism, and generates a cardiotoxicity risk index and a risk trend curve for the next 30 days in the output layer. Risk visualization interface: Maps the risk index to a heat map of a 3D heart model and records the prediction results on a blockchain-based evidence storage platform; The multimodal feature joint extraction engine includes a radiomics feature enhancement unit: The radiomics feature enhancement unit performs superpixel segmentation on the pathological slide image and extracts the contrast and entropy features of the gray-level co-occurrence matrix (GLCM). It first segments the superpixel region and extracts the texture contrast and complexity index. Deep semantic features of cells are extracted by pre-trained ResNet-50 network. Traditional image omics features and deep learning features are fused together with traditional texture features and deep features. Principal component analysis (PCA) is then used for dimensionality reduction and fusion to eliminate redundant information. The spatiotemporal feature fusion module employs a heterogeneous feature alignment algorithm: The topology of ECG leads is modeled as an unweighted graph using graph isomorphic networks (GIN), the locations of ECG leads are modeled as graph nodes, and the ultrasound dynamic vectors are projected onto the corresponding ECG nodes. The dynamic vectors of the echocardiogram are mapped to the graph node space through the feature projection matrix. The graph attention mechanism (GAT) is used to aggregate cross-modal node features, and the correlation strength between cross-modal nodes is calculated using the graph attention mechanism to aggregate features.

2. The breast cancer cardiotoxicity risk prediction system based on multimodal data fusion according to claim 1, characterized in that, The distributed multimodal data acquisition module adopts a layered data compression strategy: The echocardiogram video stream is encoded with H.265 and compressed differentially with keyframes, retaining only the motion difference data between keyframes; The dynamic electrocardiogram monitoring data is compressed using lossy compression based on wavelet transform, retaining the 0.5-40Hz frequency band characteristics. After filtering out high-frequency noise, the data is compressed to retain the core frequency band characteristics of the electrocardiogram waveform. A columnar storage time-series partition index is established in the distributed storage layer, and the data is divided into two dimensions according to patient ID and chemotherapy cycle, and stored in segments according to the patient ID and chemotherapy cycle.

3. The breast cancer cardiotoxicity risk prediction system based on multimodal data fusion according to claim 1, characterized in that, The dynamic risk prediction model includes an incremental learning mechanism: When the amount of new patient data reaches a threshold, the model is updated, and the EWC algorithm with elastic weights is used to prevent catastrophic amnesia. The elastic weighting technique is used to retain old knowledge and prevent new training from overwriting historical patterns. Parameter fine-tuning is performed in parallel on a Spark cluster, with an update cycle of less than 24 hours.

4. The breast cancer cardiotoxicity risk prediction system based on multimodal data fusion according to claim 1, characterized in that, The system is equipped with a real-time early warning feedback loop: When the risk index exceeds the clinical threshold, an alert is automatically pushed to the attending physician's terminal. The feedback loop receives false or missed data labeled by the physician and dynamically adjusts the weight distribution ratio of the GRU layer and the attention layer based on the feedback data, reduces the weight of noise features and increases the influence of key features.

5. A breast cancer cardiotoxicity risk prediction system based on multimodal data fusion according to claim 1, characterized in that, The pharmacokinetic characteristic calculation incorporates an individualized correction factor: Based on the variation of CYP3A4 enzyme activity site in patient genomic data, the curve attenuation coefficient is adjusted, and the drug clearance parameter is corrected by combining the renal function index eGFR, and the area under the curve (AUC) characteristic of individualized drug administration is output. The drug metabolism rate parameters are adjusted based on the patient's gene testing results, and the drug clearance rate is calculated by correcting the renal function indicators.

6. A breast cancer cardiotoxicity risk prediction system based on multimodal data fusion according to claim 1, characterized in that, The risk visualization interface enables multi-dimensional correlation analysis: The system overlays abnormal myocardial strain rate regions onto a 3D heart model, correlates and displays the structural similarity between the current chemotherapy regimen and historical high-risk regimens, generates an interactive decision tree interpretation diagram, and displays the ranking of key feature contributions.

7. A breast cancer cardiotoxicity risk prediction system based on multimodal data fusion according to claim 1, characterized in that, The system deploys a differential privacy protection mechanism: Laplacian noise is added before feature fusion. A federated learning architecture is adopted, and each hospital trains the model locally. The local hospital nodes only upload the model gradients. Homomorphic encryption is used to achieve ciphertext processing for risk index calculation.

8. A breast cancer cardiotoxicity risk prediction system based on multimodal data fusion according to claim 1, characterized in that, Also includes: A dynamic risk knowledge graph is established, wherein the nodes of the dynamic risk knowledge graph include four types of entities: patients, drugs, toxic phenotypes, and biomarkers. A corresponding network of four types of entities is constructed: patients → drugs → toxic phenotypes → biomarkers. Edge relationships define the causal path probability of chemotherapy regimen-cardiac injury, define the causal relationship chain between chemotherapy regimen and cardiac injury, and predict unobserved toxicity pathways based on the graph embedding technique TransE.

Images