A real-time condition deterioration prediction system based on multi-modal data

Abstract

The application discloses a kind of real-time illness deterioration prediction systems based on multi-modal data, belong to the auxiliary medical technical field based on deep learning.This application is based on bidirectional encoder representation model, Transformer model and text generation model, can access monitor and electronic medical record system and so on to obtain multi-modal data, including real-time high-frequency physiological signal and clinical text record, image report etc., predict the illness deterioration probability of ICU patient in future multiple different times, and generate the basis of interpretable report to prompt doctor model evaluation.This application can handle complex multi-modal medical data, real-time detect the illness change of ICU patient, achieve early detection, early intervention, early treatment, has great potential in optimizing medical resource allocation, improving the work efficiency of medical staff, reducing the mortality of ICU patient.In addition, the generalization ability of the present application is strong, and a special system can be developed for a specific department or a specific disease to improve the accuracy of prediction and the credibility of the results.

Description

Technical Field

[0001] This invention relates to the field of deep learning-based assistive medical technology, and in particular to a real-time disease deterioration prediction system based on multimodal data. Background Technology

[0002] Intensive care units (ICUs) require continuous and close monitoring of patients and rapid intervention when their condition deteriorates. However, the condition of ICU patients changes rapidly, and any undetected changes can increase the risk of death. For every hour of delay in treatment, the mortality rate is expected to increase by 3%. Therefore, an effective ICU early warning system can detect changes in the condition of critically ill patients early, allowing for early intervention and treatment, thereby improving the success rate of treatment.

[0003] However, current early warning score (EWS) systems applied to some patient monitors, such as the modified early warning score (MEWS) and the national early warning score (NEWS), have significant limitations: high false alarm rates, inability to capture dynamic trends over time, limited predictive capabilities, and significant individual variability. Furthermore, methods based on machine learning models (such as Long Short-Term Memory networks and Logistic models) suffer from low prediction accuracy when faced with massive amounts of medical data, making it difficult to predict the probability of disease deterioration in real time within the next two hours.

[0004] With the development of artificial intelligence, especially deep learning and Transformer models, the ability to process large-scale multimodal datasets has been significantly improved. Sajad Darabi et al. trained a Transformer model and a Bidirectional Encoder Representations from Transformers (BERT) model based on text and medical codes from the MIMIC database to predict mortality, length of hospital stay, and readmission probability for ICU patients. Benjamin Shickel... [2]Researchers designed a Longformer model that integrates static and temporal data from electronic health records, achieving an AUC of 0.927 on the 7-day mortality prediction task. In summary, using a Transformer model to fuse massive amounts of multimodal data has proven feasible, and it performs exceptionally well on the task of predicting 7-day mortality in ICU patients based on EHR data. However, few studies currently utilize textual data from electronic health records and monitor data to predict the probability of ICU patients' condition deteriorating in the next two hours in real time.

[0005] Therefore, it is of great significance to design a system that can combine textual data of ICU patients (such as medical records, imaging reports, etc.) and real-time physiological data from monitors (such as ECG, blood pressure, body temperature, blood oxygen, etc.) to predict the probability of patient deterioration in different time windows such as 1, 2, and 6 hours in the future and generate interpretable reports. This system can help medical staff to detect, intervene, and treat patients early, reduce the mortality rate of ICU patients, and improve the work efficiency of medical staff. Summary of the Invention

[0006] The purpose of this invention is to provide a real-time disease deterioration prediction system based on multimodal data (hereinafter referred to as ICUwatch). This system can process complex multimodal medical data, detect changes in the condition of ICU patients in real time, optimize the allocation of medical resources, improve work efficiency, and provide assistance in reducing the mortality rate of ICU patients.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A real-time disease deterioration prediction system based on multimodal data includes: a data processing module, a clinical text analysis module, a multimodal physiological data prediction module, an interpretable report generation module, and a system front-end interface;

[0009] The data processing module is used to connect the monitor and the electronic health record system, to obtain the clinical text record information of ICU patients through the electronic health record system, and to obtain the current multimodal physiological data of ICU patients by receiving the data collected from the monitor.

[0010] The data processing module performs text preprocessing on clinical text record information and sends the resulting text data to the clinical text analysis module; it also performs time alignment and normalization on multimodal physiological data, segments the time window according to the set time window length, and then sends the segmented physiological data of different modalities to the multimodal physiological data prediction module.

[0011] The clinical text analysis module employs the ClinicalBERT model based on the BERT architecture to perform deep semantic analysis on the input text data, thereby generating channel attention weights for each modality of the multimodal physiological prediction module. In other words, the clinical text analysis module of this invention performs deep semantic analysis on unformed clinical text data, and can extract key clinical semantic features and contextual information from complex clinical narratives. This information is used to understand the overall relevance of the patient's condition.

[0012] The multimodal physiological data prediction module employs a multimodal Transformer deep learning architecture. It assigns channel attention weights generated by the clinical text analysis module to the corresponding physiological data input channels. A memory-based multimodal attention fusion module fuses physiological data from different modalities. After generating the target sequence via a Transformer encoder and decoder, the prediction head predicts the probability of ICU patient deterioration within different time windows in the future, generating a corresponding self-attention matrix and sending it to the interpretability report generation module. Specifically, the memory-based multimodal attention fusion module uses the physiological data features of a specified initial time window as the key vector for the attention mechanism and stores it as a memory key. It uses physiological data from each subsequent time window as the query vector input to calculate the attention score between the long-term query vector and the stored memory key. Cross-attention fusion between different modalities of physiological data yields the fusion result of physiological data from different modalities, which is then fed into the Transformer encoder. Therefore, the multimodal physiological data prediction module of this invention is used to process and analyze time-series physiological data, capturing the dynamic patterns of physiological signals changing over time and their complex interrelationships.

[0013] The interpretability report generation module uses a large language model to generate an interpretability report on the predicted probability based on the channel attention weights output by the clinical text analysis module and the self-attention matrix output by the multimodal physiological data prediction module. The report is then visualized and output through the system's front-end interface.

[0014] Furthermore, clinical textual records include, but are not limited to: medical records, nursing records, and imaging reports.

[0015] Furthermore, multimodal physiological data include, but are not limited to: heart rate, blood pressure, respiratory rate, and electrocardiogram waveform.

[0016] Furthermore, the multimodal physiological data prediction module first converts the physiological data of each input modality into a sum of numerical embedding and positional embedding through the transformation embedding module to retain its numerical information and positional information in the sequence. Then, it is multiplied by the corresponding channel attention weights and sent to the memory-based multimodal attention fusion module.

[0017] Furthermore, the memory-based multimodal attention fusion module specifically involves: using a cross-attention mechanism to perform cross-attention fusion on physiological data from any two modalities to obtain the current modal fusion result; then performing the current modal fusion result with any physiological data that has not undergone modal fusion in the next round of cross-attention fusion, until all physiological data from all modalities have been processed; and finally, sending the modal fusion result obtained in the last round into the Transformer encoder.

[0018] Furthermore, the cross-attention mechanism for fusing physiological data from any two modalities specifically includes:

[0019] Define A and B as the modal representations currently undergoing cross-attention fusion;

[0020] The memory bonds of modes A and B are mapped to obtain the key vectors of modes A and B;

[0021] The physiological data of the currently input modes A and B are mapped to obtain the corresponding query vector and value vector of mode A, and the query vector and value vector of mode B, respectively.

[0022] The first vector of mode A is obtained by performing matrix multiplication between the key vector and the query vector of mode A and then scaling it using Softmax.

[0023] The first vector of mode B is obtained by performing matrix multiplication between the key vector and the query vector of mode B and then scaling it using Softmax.

[0024] The first fusion vector of mode A is obtained by performing matrix multiplication between the key vector of mode A and the query vector of mode B, followed by scaling with Softmax.

[0025] The query vector of modality A and the key vector of modality B are multiplied by a matrix and then scaled by Softmax to obtain the first fusion vector of modality B.

[0026] Perform matrix multiplication between the first vector of mode A and the value vector of mode A, and perform matrix multiplication between the first fusion vector of mode A and the value vector of mode B. Then add the two matrix multiplications together to obtain the initial fusion result 1.

[0027] Perform matrix multiplication between the first vector of mode B and the value vector of mode B, and perform matrix multiplication between the first fusion vector of mode B and the value vector of mode A. Then add the two matrix multiplications together to obtain the initial fusion result 2.

[0028] Add the initial fusion result 1 and the initial fusion result 2 to obtain the mode fusion result of mode A and mode B.

[0029] Furthermore, the model parameter optimization for the clinical text analysis module, multimodal physiological data prediction module, and interpretability report generation module is as follows:

[0030] The model parameters of the multimodal physiological data prediction module are trained based on the collected training dataset and cross-entropy loss. Then, the model parameters of the clinical text analysis module are fine-tuned based on the trained multimodal physiological data prediction module. Finally, the model parameters of the interpretability report generation module are fine-tuned. The loss function used during fine-tuning is the cross-entropy loss based on the prediction output of the multimodal physiological data prediction module.

[0031] Furthermore, when the multimodal physiological data prediction module predicts the probability of ICU patients' condition deteriorating within different time windows in the future, the future time includes the next 1 hour, 2 hours, and 6 hours.

[0032] Furthermore, the initial time window corresponding to the memory key is set to the first time window after the patient enters the ICU and reaches a stable state. That is, after the patient enters the ICU, the physiological data of each adjacent time window output by the monitoring equipment are detected. When the number of convergences of the physiological data changes between adjacent time windows reaches the set value, it is determined to be a stable state.

[0033] Furthermore, the visual presentation content output by the system's front-end interface includes: an explanatory report of the predicted probability, patients whose risk of deterioration reaches the warning value and their risk of deterioration, monitoring data collected by the monitor, and clinical text record information.

[0034] The technical solution provided by this invention brings at least the following beneficial effects:

[0035] (1) Improved accuracy and timeliness of prediction: Through innovative text-guided multimodal data fusion methods, memory-based multimodal data fusion mechanisms, and advanced deep learning models (such as ClinicalBERT and multimodal Transformer), this system significantly outperforms traditional EWS scoring systems and baseline machine learning models in predicting the deterioration of ICU patients in the short term (such as in the next 1 hour, 2 hours, 6 hours).

[0036] (2) Enhanced clinical applicability: By generating interpretable "key focus reports," this system not only provides the prediction results but, more importantly, explains the rationale behind the model's predictions, highlighting key clinical indicators and textual information that require attention. This provides clinicians with clear and transparent decision support, helping to translate the powerful predictive capabilities of AI models into practical clinical actions and overcoming the acceptance challenges of traditional "black box" AI models in clinical applications.

[0037] (3) Reduced information overload and potential alarm fatigue: Due to more accurate and real-time predictions and the provision of interpretable reports, it helps to reduce unnecessary false alarms, thereby reducing the stress caused by information overload, reducing the risk of fatigue, and improving the overall alarm efficiency and patient safety in the ICU.

[0038] (4) Better adapt to individual differences among patients: By using a memory-based fusion module, data characteristics of ICU patients in their initial "normal state" are preserved, and an individualized center is established for ICU patients. This can better capture early subtle changes in the patient's condition and improve the accuracy of prediction.

[0039] (5) User-friendly interface: A simple and easy-to-use interface was designed to connect the ICUwatch system to the monitor and electronic health record system. The interface directly displays real-time physiological data, the predicted probability of disease deterioration in different time windows, and corresponding explanatory reports. This concise and intuitive design reduces the learning cost for users and improves work efficiency. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A schematic diagram of the structure of a real-time disease deterioration prediction system based on multimodal data provided in this application embodiment;

[0042] Figure 2 A flowchart illustrating the operation of a real-time disease deterioration prediction system based on multimodal data, provided in this application embodiment;

[0043] Figure 3 This application provides a structural diagram of a clinical text analysis module in a real-time disease deterioration prediction system based on multimodal data, as shown in the embodiments of this application.

[0044] Figure 4 A structural diagram of a multimodal physiological data prediction module in a real-time disease deterioration prediction system based on multimodal data, provided in an embodiment of this application;

[0045] Figure 5 This application provides a structural diagram of the Transformer model encoder in the multimodal physiological data prediction module of a real-time disease deterioration prediction system based on multimodal data, as provided in an embodiment of the present application.

[0046] Figure 6 This application provides a structural diagram of the Transformer model decoder in the multimodal physiological data prediction module of a real-time disease deterioration prediction system based on multimodal data, as shown in the embodiments of this application.

[0047] Figure 7 A memory-based multimodal fusion mechanism structure diagram is provided in a real-time disease deterioration prediction system based on multimodal data, as shown in the embodiments of this application.

[0048] Figure 8 This application provides a real-time physiological data display interface for a real-time disease deterioration prediction system based on multimodal data, as an embodiment of the present application.

[0049] Figure 9 This application provides an embodiment of a text data display interface for a real-time disease deterioration prediction system based on multimodal data, including medical records and image reports.

[0050] Figure 10 This application provides an embodiment of a real-time disease deterioration prediction system based on multimodal data, which provides a display interface for predicting the probability of disease deterioration in different time windows.

[0051] Figure 11 This application provides an interpretive focus report display interface for a real-time disease deterioration prediction system based on multimodal data, as an embodiment of the present application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described in detail and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present invention.

[0053] This invention provides a real-time disease deterioration prediction system based on multimodal data (ICUwatch). The model is based on a bidirectional encoder representations from transformers (BERT), a Transformer model, and a text generation model. It can access multimodal data from monitors and electronic medical record systems, including real-time high-frequency physiological signals, clinical text records, and image reports. It predicts the probability of disease deterioration in ICU patients within the next 1, 2, and 6 hours and generates interpretable reports to inform doctors' model evaluation. Mimicking the process of doctors reading medical records and imaging reports before examining specific indicators, the ICUwatch system can handle complex multimodal medical data, monitor changes in ICU patients' conditions in real time, and enable early detection, early intervention, and early treatment. It has significant potential in optimizing medical resource allocation, improving the efficiency of medical staff, and reducing ICU patient mortality. Furthermore, the ICUwatch system has strong generalization capabilities; dedicated ICUwatch systems can be developed for specific departments and diseases to improve prediction accuracy and reliability.

[0054] One embodiment of this application uses the MIMIC-IV database for model training and validation. The MIMIC-IV database collects clinical data from over 190,000 patients and 450,000 hospitalization records treated at Beth Israel Deaconess Medical Center from 2008 to 2019. The data primarily consists of three categories: text data extracted from the electronic health record system, waveform data collected from ICU bedside monitoring devices, vital signs, fluid management and event records, and death follow-up records. Death records can be used as tags, and the text and physiological data from the database can be used to train various modules in the ICUwatch system.

[0055] The ICUwatch system provided in this invention draws on the diagnostic approach of doctors. It first analyzes textual data such as the patient's medical records and imaging reports to arrive at a general diagnosis or focus for the patient's condition. Then, it makes a diagnosis by focusing on specific indicators or examination results. The overall structure of the ICUwatch system is as follows: Figure 1As shown, the system includes a data processing module, a clinical text analysis module, a multimodal physiological data prediction module, an interpretable report generation module, and a system front-end interface. First, in the data preprocessing stage, the data processing module cleans and standardizes the text data (including medical records and image reports) of the MIMIC dataset, removing irrelevant information and performing word embedding transformation to facilitate input to the ClinicalBERT model. Simultaneously, the vital signs data are arranged in time series, missing values ​​are processed, normalized, and segmented according to time windows to prepare the training format for the multimodal Transformer model. Then, the clinical text analysis module performs deep semantic analysis on the input text data to generate channel attention weights for each modality of the multimodal physiological prediction module. The multimodal physiological data prediction module assigns the channel attention weights generated by the clinical text analysis module to the corresponding physiological data input channels. The memory-based multimodal attention fusion module fuses the physiological data from different modalities. After generating the target sequence through the Transformer encoder and Transformer decoder, the prediction head predicts the probability of ICU patients' condition deterioration within different time windows in the future, generates the corresponding self-attention matrix, and sends it to the interpretability report generation module. Finally, the interpretability report generation module generates an interpretive report on the predicted probability based on the channel attention weights output by the clinical text analysis module and the self-attention matrix output by the multimodal physiological data prediction module, and displays it on the system's front-end interface.

[0056] See Figure 2 The operation flow of the system proposed in this embodiment of the invention includes:

[0057] S1. The model first performs basic data processing (such as text data cleaning);

[0058] S2. Input the text data into the ClinicalBERT-based clinical text analysis module to deeply understand the semantic information of the text and generate the channel attention weights for the multimodal physiological data prediction module.

[0059] S3. Assign channel attention weights to the corresponding physiological data input channels. After fusing physiological data from different modalities, the multimodal Transformer module predicts the probability of ICU patients’ condition deteriorating within different time windows in the future.

[0060] S4. Input the output of the clinical text analysis module and the self-attention score of the multimodal physiological data prediction module into the interpretability report generation module to generate an interpretable report on the predicted probability, such as complications that need to be focused on, whether an infection has occurred, etc.

[0061] In this embodiment of the invention, the model training of the network models involved (clinical text analysis module, multimodal physiological data prediction module, and interpretability report generation module) includes:

[0062] The ClinicalBERT model, optimized for the clinical domain, is used for deep learning training on textual data to capture key clinical features. Simultaneously, a multimodal Transformer model is trained on vital sign data; this model is designed to understand the temporal dynamics of physiological parameters and their interrelationships. Then, the attention mechanism generated by the ClinicalBERT model is integrated into the multimodal prediction model to achieve cross-modal guidance. By adjusting the attention allocation in the Transformer model, the focus on specific physiological indicators is strengthened based on text analysis results, thereby improving the accuracy of predicting disease deterioration and the model's interpretability. This step also includes fine-tuning the fusion model to ensure efficient collaboration between the two parts. Finally, validation and system integration are performed. Cross-validation techniques are used to evaluate the system's performance on independent test sets, focusing on key metrics such as AUROC and AUCPR. Based on the validation results, model parameters are fine-tuned to ensure that the ICUwatch system can not only accurately predict the probability of disease deterioration within the next n hours but also stably generate targeted medical advice reports. When the system is finally deployed, it seamlessly integrates real-time monitor data with updated medical records, and displays prediction results and automatically generated medical concern reports through a user-friendly interface, effectively assisting medical staff in making rapid responses and demonstrating a complete chain from data to clinical application.

[0063] In one embodiment, the training process is as follows:

[0064] First, the multimodal physiological data prediction module is trained using data from the cleaned and labeled MIMIC dataset. Then, the clinical text analysis module and the interpretable report generation module are fine-tuned based on the trained module.

[0065] The binary cross-entropy loss function was used for both training and model fine-tuning to predict the risk of disease changes implied in the text.

[0066]

[0067] Where N is the number of samples in the batch, l n The loss corresponding to the nth sample:

[0068] l n =-[y n *log x n +(1-y n )*log(1-x n )] 2)

[0069] Where, x n Let y represent the probability that the nth sample is a positive example. n This represents the label of the nth sample.

[0070] During training, the calculated loss is backpropagated, and the network parameters are updated using the Adam optimization algorithm. The learning rate is set to 0.0003. After each complete training epoch, the samples in the training set are randomly shuffled before entering the next epoch, thus updating and iterating the network parameters. In one example, the GPU used during training is 1×NVIDIA TeslaV100 (32GB), the CPU is an i7-14900K, the PyTorch training framework is used, the batch size is 24 (global batch size 96), 50 epochs are used, the initial learning rate is set to 5e-5, ranging from [1e-5, 1e-4], and a cosine annealing strategy is employed.

[0071] In fine-tuning the ClinicalBERT model (clinical text analysis module), this embodiment performed task-specific fine-tuning based on 28,970 ICU admission records and clinical documents, using disease deterioration labels (based on clinical events 24 hours later) as supervision signals. The fine-tuning process employed a binary cross-entropy loss function to predict the risk of disease changes implied in the text.

[0072] When training the ClinicalBERT model, the weights output by the model are assigned to the corresponding physiological signal channels. For vital signs (BP, Temp) encoding, a sliding window (1h, 2h, 6h) and a TemporalTransformer (4 layers, 8 heads) with positional encoding are used. For ECG waveform data, waveform features are extracted using a 1D-CNN and then input into the Transformer layer. The hidden layer dimension is set to 512, the feedforward network dimension is set to 2048, and the dropout rate is set to 0.1.

[0073] Finally, the text generation model was fine-tuned using the output of ClinicalBERT, the self-attention matrix of the multimodal Transformer model, and expert evaluation and modified reports.

[0074] After training, unlabeled text data and physiological data (i.e., clinically generated data in real time) can be input into ICUwatch and the following steps can be performed to obtain the probability of ICU patients' condition deteriorating within the next 1 hour, 2 hours, 6 hours and other time windows, as well as corresponding interpretive reports.

[0075] Specifically, in the data processing stage of step S1, the embodiments of this application are applied to the ICU for neurological diseases, defining deterioration of condition as death. Screening criteria include: patients with neurological diseases; ICU stay of at least 24 hours to ensure sufficient data for analysis; exclusion of patients who develop severe illness within a very short period after ICU admission (e.g., the first 2 hours), as such events may be difficult to predict early with data; and exclusion of patients with severely missing data (e.g., unrecorded electrocardiogram waveforms or excessively long intervals between recordings of key vital signs). For other embodiments, there may be different definitions of deterioration of condition and screening criteria.

[0076] First, the text data is constructed, including converting the text to lowercase, avoiding unnecessary special characters and HTML tags, and standardizing abbreviations and medical terms. Second, the text data is segmented into words or sub-words using a word processor such as the one built into ClinicalBERT.

[0077] Then, physiological data is constructed. Since different physiological signals may have different sampling frequencies and recording interruptions may occur, time synchronization is required for non-uniformly sampled physiological signals (such as heart rate and blood oxygen saturation). For interpolation within a window, methods such as linear interpolation, forward padding, or backward padding can be used. The ECG signal clock also needs to be synchronized with other physiological signals, and continuous physiological indicators are z-score normalized. Continuous time-series data is divided into fixed-length observation windows (e.g., data from the past 1, 2, or 6 hours), which will serve as the input sequence for the model. For ECG waveform data, appropriate filters (such as bandpass filters) are first used to eliminate interference and high-frequency noise, followed by QRS complex detection to segment the continuous ECG signal into individual beat ECG segments centered on the R wave.

[0078] During step S2, the clinical text contains a wealth of valuable information about the patient's medical history, current condition, treatment response, and potential risks. To effectively utilize this unstructured data, this system employs the ClinicalBERT model for deep semantic analysis. ClinicalBERT is a deep neural network model based on the BERT architecture, with the Transformer encoder at its core, which uses a self-attention mechanism to learn the contextual representation of the text. Figure 3 As shown, ClinicalBERT, based on the general BERT, has undergone domain-adaptive pre-training using a large amount of medical literature and clinical text (such as medical knowledge from BooksCorpus, Wikipedia, and clinical notes from the MIMIC database). This enables it to better understand the specific patterns of medical terminology and clinical narratives. Its main functions include:

[0079] (1) Deep semantic feature extraction: Input the patient's medical history, nursing records, and report text data. ClinicalBERT can capture deep semantic information and key clinical concepts in the text, overcoming the complexity and limitations of traditional or keyword rule-based text semantic processing methods.

[0080] (2) Attention-Guided Signal Generation: The ClinicalBERT model is not only used for independent text analysis, but its learned text representations are also used to adjust the attentional reallocation of different physiological signal channels in subsequent multimodal Transformer models when processing physiological data. Specifically, ClinicalBERT can analyze the potential risk directions of patients based on text content (e.g., Figure 3 The example shown is ClinicalBERT learning from text data such as medical records of heart patients, suggesting that the electrocardiogram and blood pressure channels need to be given special attention. It outputs a set of probability values ​​or weights corresponding to the input channels of the multimodal Transformer model, thereby realizing intelligent guidance of physiological data analysis through text information.

[0081] In some embodiments, the monitor may generate other physiological signals, such as blood oxygen and blood carbon dioxide concentration, and the number of channels needs to be adjusted according to the types of data the monitor is connected to the ICUwatch system.

[0082] During step S3, the dynamic changes in physiological time-series data (such as heart rate, blood pressure, respiratory rate, electrocardiogram waveforms, etc.) are of great significance for judging the trend of the patient's condition. The ICUwatch system uses a multimodal Transformer model to process massive and complex clinical data. Figure 4 The overall structure of the multimodal physiological data prediction module is demonstrated. The model first retrieves multimodal physiological signals from the MIMIC database as multiple inputs. Each modality's input is first converted into a sum of numerical and positional embeddings by a free-form embedding module to preserve its numerical information and positional information in the sequence. Then, it is multiplied by the corresponding channel attention weights and fed into a memory-based multimodal attention fusion module. Figure 5 This is the main structure of the Transformer model encoder, which consists of N (N=6) identical layers stacked together. Each layer contains two sub-modules: a multi-head self-attention mechanism and a feedforward neural network. Multi-head self-attention allows the model to simultaneously focus on different positions in the input sequence, assigning weights by calculating the similarity between the query (Q), key (K), and value (V). The feedforward network consists of two linear transformations and a ReLU activation function for feature transformation. Each sub-module is equipped with residual connections and layer normalization, effectively preventing gradient problems common in deep networks. Figure 6The main structure of the Transformer model decoder consists of N identical layers, but each layer contains three sub-modules: masked multi-head self-attention (to prevent the seer from seeing future information), encoder-decoder attention (combining the decoder's query with the encoder's key), and a feedforward neural network. The decoder achieves autoregressive generation through a masking mechanism, ensuring that each position can only access previous outputs. The core innovations of the physiological data prediction module lie in the following two major mechanisms:

[0083] (1) Text-guided multimodal attention: such as Figure 3 As shown, after analyzing textual data, the ClinicalBERT model generates a set of guiding weights. These weights are assigned to the corresponding physiological signal modalities in the multimodal Transformer model. This establishes a knowledge mapping from the textual semantic space to the physiological feature space, realizing cross-modal knowledge transfer and focus guidance. This mechanism draws inspiration from the diagnostic process of clinicians: doctors first make a judgment on the patient's condition based on textual information such as medical history and imaging reports, and then focus on specific data to confirm the diagnosis. Through textual guidance, the model can more intelligently allocate its cognitive resources to different physiological signals. For example, if the text suggests that the patient has a respiratory problem, the model may pay more attention to changes in physiological indicators such as respiratory rate and blood oxygen saturation, thereby improving the accuracy and efficiency of prediction. Compared with traditional simple multimodal fusion or methods using general attention, this is an information fusion strategy that is more in line with clinical cognitive patterns.

[0084] (2) Memory-based Multimodal Attention Fusion Module: Patients in the ICU exhibit significant individual differences, with their physiological indicators often differing from those of healthy individuals. Furthermore, there are significant differences in physiological data characteristics among different ICU patients. Therefore, there is no unified standard for "normal" ICU data. To address this issue and effectively capture the dynamic changes in patients' conditions, this invention innovatively designs a memory-based multimodal attention fusion module, the specific structure of which is as follows: Figure 7 As shown. The core idea of ​​this module is to establish a dynamic and individualized physiological monitoring system for patients.

[0085] In the memory-based multimodal attention fusion module, a cross-attention mechanism is first used to perform cross-attention fusion on physiological data from any two modalities to obtain the current modal fusion result. Then, the current modal fusion result is combined with any physiological data that has not yet undergone modal fusion for the next round of cross-attention fusion. This process is repeated until all physiological data from all modalities have been processed. The modal fusion result obtained in the last round is then fed into the Transformer encoder.

[0086] In one embodiment, see Figure 7 The cross-attention mechanism for fusing physiological data from any two modalities specifically includes:

[0087] Let A and B represent the modal representations currently undergoing cross-attention fusion; for example, modality A corresponds to ECG data, and modality B corresponds to BP data. The memory keys of modalities A and B are mapped (multiplied by the weight matrices corresponding to the key vectors) to obtain the key vectors of modalities A and B. The currently input physiological data of modalities A and B are mapped (multiplied by the weight matrices corresponding to the query vector and value vector, respectively) to obtain the corresponding query vector and value vector of modality A, and the query vector and value vector of modality B, respectively. The key vector of modality A is multiplied by the query vector and then scaled using Softmax to obtain the first vector of modality A. The key vector of modality B is multiplied by the query vector and then scaled using Softmax to obtain the first vector of modality B. The key vector of modality A and the query vector of modality B... The query vector is multiplied by matrix and then scaled by Softmax to obtain the first fusion vector of modality A. The query vector of modality A and the key vector of modality B are multiplied by matrix and then scaled by Softmax to obtain the first fusion vector of modality B. The first vector of modality A and the value vector of modality A are multiplied by matrix, and the first fusion vector of modality A and the value vector of modality B are multiplied by matrix. The two matrix multiplications are then added to obtain the initial fusion result 1. The first vector of modality B and the value vector of modality B are multiplied by matrix, and the first fusion vector of modality B and the value vector of modality A are multiplied by matrix. The two matrix multiplications are then added to obtain the initial fusion result 2. The initial fusion result 1 and the initial fusion result 2 are added to obtain the modality fusion result of modality A and modality B.

[0088] In practice, when a patient is first connected to monitoring equipment (e.g., in the initial time window after a patient has stabilized in the ICU), their corresponding physiological data features are considered to be in a "normal" state and stored as a "key vector" for that modality. In each subsequent time window, when new physiological data (as a "query" vector) is input, the model calculates an attention score between the current query and the initially stored key vector. This allows the model to assess the degree and trend of change in the current physiological state relative to the patient's own ontological state, rather than comparing it to a fixed, universal "normal range." This memory-based comparison method can more effectively identify subtle changes that are clinically significant for a specific patient, thereby improving the early predictive tendency and accuracy while reducing false alarms caused by individual physiological differences.

[0089] Step S4: Although the learning model performs well on prediction tasks, its "black box" nature often makes the decision-making process difficult to understand, which greatly limits its acceptance and application in clinical practice. To address this key issue, this system training incorporates an interpretable report generation module. Its core is a large language model (LLM) in the biomedical field, such as ContactDoctor / Bio-Medical-MultiModal-Llama-3-8B-V1 in this embodiment. This model has 8 billion parameters and has been pre-trained using high-quality biomedical text and image data, enabling it to understand and generate biomedical-related text. To adapt the model to the task of this embodiment, it is fine-tuned using the following methods:

[0090] (1) Focus information from upstream models: ClinicalBERT and polymorphic Transformer models generate self-attention matrices during prediction, reflecting the degree of attention the model pays to the input data (vocabulary in the text, different time points or channels of physiological signals) when making decisions. These focus weights can be extracted as important clues for LLM to understand predictions.

[0091] (2) Expert-written analysis reports: To guide the LLM in generating reports that conform to clinical logic and expression, the research team invited clinicians to write analysis reports on real cases of deterioration (positive examples). These reports typically include an analysis of the causes of deterioration, key indicators and situations requiring attention. By using the above attention information and expert reports as training data to fine-tune the pre-trained LLM, the model's internal state is linked to clinically relevant interpretations. Finally, when the ICUwatch system predicts that a patient's condition is about to deteriorate, the module not only outputs a risk probability but also automatically generates a "key attention report." The report, combined with a medical knowledge base, provides relevant hints or suggestions, such as: "Patient reports 'worsening chest pain,' ECG shows 'ST segment elevation with upward convexity,' current heart rate 115 bpm. Knowledge base suggests: highly suspected acute myocardial infarction." Based on this method of constructing the LLM with a focus matrix and clinical knowledge, the "black box" of the prediction model is effectively opened, significantly enhancing the interpretability and clinical applicability of the prediction results, enabling clinicians to better understand and trust AI-assisted diagnosis and take timely and effective actions accordingly.

[0092] In this embodiment, the designed system front-end interface includes a patient drop-down menu, a real-time monitor data display module, a data display module for medical records, image reports, etc., a module for predicting the probability of disease deterioration in different time windows, and a key attention report module. Figure 8This is the physiological data display interface in this embodiment. In other embodiments, the monitor data input module can increase or decrease the display window according to the types of data collected by different models of monitors. Figure 9 This embodiment provides a text data display interface for patients, including medical records, image reports, diagnoses, etc. The content of the interface can be adjusted according to different embodiments. Figure 10 This embodiment uses prediction windows of 1 hour, 2 hours, and 6 hours to display the probability of patient condition deterioration at different timeframes. Other embodiments can adjust the prediction window as needed. This embodiment provides possible risk factors for patients whose condition is worsening within this module. Figure 11 This embodiment focuses on the report display interface. The model's attention matrix indicates the reasons for the patient's deterioration. In this embodiment, the patient may have severe heart rate abnormalities, respiratory dysfunction, or infection, requiring examination and timely treatment intervention by medical personnel.

[0093] In one embodiment, the ICUwatch of this invention can be integrated into a monitor to directly display the probability of a patient's condition worsening in different windows. It can also be connected to a doctor's terminal to transmit data for processing and display. Simultaneously, an ICU patient management interface can be designed to monitor all ICU patients at the same time, reporting when the probability of a patient's condition worsening exceeds a threshold, prompting medical staff to conduct timely examinations and treatments.

[0094] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0095] Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include at least one of those features.

[0096] Any process or method description described in this specification can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0097] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0098] Note that, unless otherwise explicitly stated, all features disclosed in this specification can be replaced by alternative features for achieving the same, equivalent, or similar purpose. Therefore, unless explicitly stated otherwise, each disclosed feature is merely one example of a set of equivalent or similar features. Where used, "further," "preferably," "even further," and "more preferably" are simply starting points for describing another embodiment based on the foregoing embodiments, the combination of which with the foregoing embodiments constitutes the complete configuration of another embodiment. Any combination of several "further," "preferably," "even further," or "more preferably" settings following the same embodiment constitutes yet another embodiment.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application, and are not intended to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the application.

[0100] The above descriptions are merely some embodiments of this application. For those skilled in the art, various modifications and improvements can be made without departing from the inventive concept of this application, and all such modifications and improvements fall within the scope of protection of this application.

Claims

1. A real-time disease deterioration prediction system based on multimodal data, characterized in that, include: The system includes a data processing module, a clinical text analysis module, a multimodal physiological data prediction module, an interpretable report generation module, and a system front-end interface. The data processing module connects the monitor and the electronic health record system, acquires the clinical text record information of ICU patients through the electronic health record system, and acquires the current multimodal physiological data of ICU patients through the data collected by the monitor. The data processing module performs text preprocessing on clinical text record information and sends the resulting text data to the clinical text analysis module; it also performs time alignment and normalization on multimodal physiological data, segments the time window according to the set time window length, and then sends the segmented physiological data of different modalities to the multimodal physiological data prediction module. The clinical text analysis module uses the ClinicalBERT model based on the BERT architecture to perform deep semantic analysis on the input text data, in order to generate the channel attention weights of each modality channel in the multimodal physiological data prediction module. The multimodal physiological data prediction module adopts a multimodal Transformer deep learning architecture. It assigns the channel attention weights generated by the clinical text analysis module to the corresponding physiological data input channels. The memory-based multimodal attention fusion module fuses the physiological data of different modalities. After generating the target sequence through the Transformer encoder and Transformer decoder, the prediction head predicts the probability of ICU patients' condition deterioration within different time windows in the future, generates the corresponding self-attention matrix, and sends it to the interpretability report generation module. The memory-based multimodal attention fusion module uses the physiological data features of a specified initial time window as the key vector of the attention mechanism and stores them as memory keys. It uses the physiological data in each subsequent time window as the input query vector to calculate the attention score between the query vector and the stored memory keys. The module also performs cross-attention fusion between different modal physiological data to obtain the fusion results of different modal physiological data and sends them to the Transformer encoder. The initial time window corresponding to the memory key is set as the first time window after the patient enters the ICU and reaches a stable state. That is, after the patient enters the ICU, the physiological data of each adjacent time window output by the monitoring equipment is detected. When the number of convergences of the physiological data changes between adjacent time windows reaches a set value, the patient is determined to be in a stable state. The interpretability report generation module uses a large language model to generate an interpretability report on the predicted probability based on the channel attention weights output by the clinical text analysis module and the self-attention matrix output by the multimodal physiological data prediction module. The report is then visualized and output through the system's front-end interface.

2. The system as described in claim 1, characterized in that, Clinical written records include: medical records, nursing records, and imaging reports.

3. The system as described in claim 1, characterized in that, Multimodal physiological data include: heart rate, blood pressure, respiratory rate, and electrocardiogram waveform.

4. The system as described in claim 1, characterized in that, The multimodal physiological data prediction module first converts the physiological data of each input modality into a sum of numerical embedding and positional embedding through the transformation embedding module to preserve its numerical information and positional information in the sequence. Then, it is multiplied by the corresponding channel attention weights and sent to the memory-based multimodal attention fusion module.

5. The system as described in claim 1, characterized in that, The memory-based multimodal attention fusion module specifically employs a cross-attention mechanism to perform cross-attention fusion on physiological data from any two modalities, thereby obtaining the modal fusion result of the current two modalities. The current modality fusion result is then combined with any physiological data that has not undergone modality fusion for the next round of cross-attention fusion, until all modal physiological data has been processed; the modality fusion result obtained in the last round is then fed into the Transformer encoder.

6. The system as described in claim 5, characterized in that, The cross-attention mechanism for fusing physiological data from any two modalities specifically includes: Define A and B as the modal representations currently undergoing cross-attention fusion; The memory bonds of modes A and B are mapped to obtain the key vectors of modes A and B; The physiological data of the currently input modes A and B are mapped to obtain the corresponding query vector and value vector of mode A, and the query vector and value vector of mode B, respectively. The first vector of mode A is obtained by performing matrix multiplication between the key vector and the query vector of mode A and then scaling it using Softmax. The first vector of mode B is obtained by performing matrix multiplication between the key vector and the query vector of mode B and then scaling it using Softmax. The first fusion vector of mode A is obtained by performing matrix multiplication between the key vector of mode A and the query vector of mode B, followed by scaling with Softmax. The query vector of modality A and the key vector of modality B are multiplied by a matrix and then scaled by Softmax to obtain the first fusion vector of modality B. Perform matrix multiplication between the first vector of mode A and the value vector of mode A, and perform matrix multiplication between the first fusion vector of mode A and the value vector of mode B. Then add the two matrix multiplications together to obtain the initial fusion result 1. Perform matrix multiplication between the first vector of mode B and the value vector of mode B, and perform matrix multiplication between the first fusion vector of mode B and the value vector of mode A. Then add the two matrix multiplications together to obtain the initial fusion result 2. Add the initial fusion result 1 and the initial fusion result 2 to obtain the mode fusion result of mode A and mode B.

7. The system as described in claim 1, characterized in that, The specific optimization of model parameters for the clinical text analysis module, multimodal physiological data prediction module, and interpretable report generation module is as follows: The model parameters of the multimodal physiological data prediction module are trained based on the collected training dataset and cross-entropy loss. Then, the model parameters of the clinical text analysis module are fine-tuned based on the trained multimodal physiological data prediction module. Finally, the model parameters of the interpretability report generation module are fine-tuned. The loss function used during fine-tuning is the cross-entropy loss based on the prediction output of the multimodal physiological data prediction module.

8. The system as described in claim 1, characterized in that, When the multimodal physiological data prediction module predicts the probability of ICU patients' condition deteriorating within different time windows in the future, the future time includes the next 1 hour, 2 hours, and 6 hours.

9. The system as described in claim 1, characterized in that, The system's front-end interface outputs the following visual content: an explanatory report of the predicted probability, patients whose risk of deterioration reaches the warning value and their risk of deterioration, monitoring data collected in real time by the monitor, and clinical text record information.

Images