A method and system for predicting death risk oriented to incomplete multi-modal electronic health records, a terminal and a storage medium

Abstract

The application belongs to the technical field of data processing, and discloses a death risk prediction method, system, terminal and storage medium for incomplete multi-modal electronic health records, which comprises the following steps: judging the missing type of multi-modal EHR; aligning the observation modal sequence and adding a modal signal prompt to obtain an observation modal sequence representation; generating an initial generative missing modal sequence representation through cross attention and adding a modal signal prompt to obtain a generative missing modal sequence representation; performing cross-modal fusion and memory coding on the observation modal sequence representation and the generative missing modal sequence representation through cross-modal attention to obtain a cross-modal fusion sequence; performing missing mode perception time series aggregation on the cross-modal fusion sequence through multi-head attention, and obtaining the death risk prediction probability of the patient through a residual multi-layer perception machine. The application significantly improves the accuracy of the in-hospital death risk prediction for incomplete multi-modal electronic health records.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, system, terminal, and computer-readable storage medium for predicting mortality risk based on incomplete multimodal electronic health records. Background Technology

[0002] The clinical status of patients in the Intensive Care Unit (ICU) is typically reflected by a combination of information in Electronic Health Records (EHRs), including multivariate irregularly sampled time series (MISTS) of vital signs and laboratory tests, as well as clinical text recorded by physicians and nurses. In real-world clinical settings, missing information can be categorized into two distinct types: intramodal missing information, where a modality exists but is sampled irregularly and sparsely within the observation window; and modal-level missing information, where an entire modality is completely unavailable for a particular hospitalization, such as the complete absence of clinical text or numerical time series. For the binary classification task of predicting in-hospital mortality risk within the first 48 hours after admission to the ICU (48-Hour In-Hospital Mortality, 48-IHM), in the case of modality-level missing data, the encoder branch on the missing side has no observations to process. The model that relies on two-stream cross-modal fusion will degenerate into a single-stream network, and cross-modal attention cannot function properly, resulting in a decrease in model prediction performance.

[0003] The existing technologies still have the following shortcomings: (1) Existing irregular EHR models lack a unified processing mechanism for the three types of missing modes at the sample level: missing text, missing values, and complete data. (2) Some missing completion methods only generate the missing mode representation directly based on the observed mode, without explicitly informing the subsequent fusion module of the source of the representation. (3) Existing fusion networks lack learnable conditional expressions for the three types of missing modes: missing text, missing values, and complete data, making it difficult to adjust the fusion strategy for different missing scenarios. (4) Existing methods mostly use unified mean pooling or shared attention for temporal aggregation of fused sequences, without coordinating with the missing modes. These shortcomings significantly reduce the accuracy of predicting the risk of inpatient mortality within the first 48 hours of ICU patient admission, resulting in insufficient "early warning" capabilities of hospitals, which may "underreport" high-risk patients or "over-alarm" leading to a waste of medical resources.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a method, system, terminal, and computer-readable storage medium for predicting mortality risk based on incomplete multimodal electronic health records, aiming to solve the problem of low accuracy in existing methods for predicting mortality risk based on incomplete multimodal electronic health records.

[0006] To achieve the above-mentioned objective, this invention provides a method for predicting mortality risk based on incomplete multimodal electronic health records. The method includes: Obtain the patient's multimodal electronic health record and the corresponding mask of the multimodal electronic health record, and determine the missing type of the multimodal electronic health record based on the mask; Based on the missing type, the observation modality sequence in the multimodal electronic health record is encoded and time-aligned to obtain an initial observation modality sequence characterization; The initial generative missing mode sequence representation is obtained by extracting sequence information related to the missing mode from the initial observed mode sequence representation through cross attention. Modal signal cues are added to the initial observed modal sequence representation and the initial generative missing modal sequence representation, respectively, to obtain the observed modal sequence representation and the generative missing modal sequence representation; The observed modal sequence representation and the generative missing modal sequence representation are bidirectionally cross-modal fused by cross-modal attention to obtain a cross-modal interactive representation. The cross-modal interactive representation with missing type hints is then memorized and fused to obtain a cross-modal fused sequence. Based on the temporal cue corresponding to the missing type, the cross-modal fusion sequence is temporally aggregated using multi-head attention to obtain a temporal aggregation vector; By using a residual multilayer perceptron and a classification layer, the temporal aggregation vector is nonlinearly transformed and mapped to obtain the predicted probability of the patient's mortality risk.

[0007] Optionally, the step of obtaining the patient's multimodal electronic health record and the mask corresponding to the multimodal electronic health record, and determining the missing type of the multimodal electronic health record based on the mask, specifically includes: The system acquires the patient's multimodal electronic health record within a set time period before admission to the department and the corresponding mask for the multimodal electronic health record. The multimodal electronic health record includes a numerical sequence or clinical text, and the mask includes a numerical observation mask corresponding to the numerical sequence and a text time mask corresponding to the clinical text. Based on the numerical observation mask and the text time mask, determine the missing type of the multimodal electronic health record; If the numerical observation mask indicates the existence of valid numerical observations, and the text time mask indicates the absence of valid clinical text records, then the missing type of the multimodal electronic health record is determined to be missing text modality. If the numerical observation mask indicates that there are no valid numerical observations, and the text time mask indicates that there are valid clinical text records, then the missing type of the multimodal electronic health record is determined to be missing numerical modality. If the numerical observation mask indicates the existence of valid numerical observations, and the text time mask indicates the existence of valid clinical text records, then the missing type of the multimodal electronic health record is determined to be complete data.

[0008] Optionally, the initial observation modality sequence representation includes an initial numerical modality sequence representation or an initial text modality sequence representation, wherein the numerical sequence includes numerical observations and observation timestamps, and the clinical text includes clinical text records and writing timestamps; The step of encoding and time-aligning the observation modality sequences in the multimodal electronic health record according to the missing type to obtain an initial observation modality sequence characterization specifically includes: When the missing type is a missing text modality, the observation modality sequence in the multimodal electronic health record is a numerical sequence, which includes irregular numerical sequences and regular numerical sequences; For the irregular numerical sequence, the numerical observations in the irregular numerical sequence are concatenated with the numerical observation mask corresponding to the irregular numerical sequence to obtain the concatenated numerical observations and numerical observation mask. The concatenated numerical observations and numerical observation mask are then aligned to a shared time grid through temporal embedding and a multi-temporal attention network to obtain the irregular branch representation. For the rule-based numerical sequence, a one-dimensional convolution is performed on the rule-based numerical sequence to obtain the rule-based branch representation; Learnable weights are assigned to the irregular branch representation and the regular branch representation respectively through a gating network. The irregular branch representation and the regular branch representation are then weighted and fused according to the learnable weights to obtain the initial numerical mode sequence representation. When the missing type is a missing numerical modality, the observation modality sequence in the multimodal electronic health record is clinical text; A medical text encoder is used to extract the classification label vector of the clinical text record in the clinical text; Based on the writing timestamp in the clinical text and the text time mask corresponding to the clinical text, the classification label vector is aligned to the shared time grid through multi-temporal attention to obtain the initial text modality sequence representation; When the missing type is complete data, the observation modality sequence in the multimodal electronic health record is a numerical sequence and clinical text; The numerical sequence and the clinical text are encoded and time-aligned respectively to obtain the initial numerical modality sequence representation and the initial text modality sequence representation.

[0009] Optionally, the initial generative missing modality sequence representation includes an initial generative text modality sequence representation or an initial generative numerical modality sequence representation; The step of extracting sequence information related to the missing modes from the initial observed mode sequence representation through cross-attention to obtain the initial generative missing mode sequence representation specifically includes: When the missing type is a missing text modality, the initial observed modality sequence representation is an initial numerical modality sequence representation. The text target query library is used as the query, and the output of the initial numerical modality sequence representation after linear projection is used as the key and value. Sequence information related to the target text modality is extracted from the initial numerical modality sequence representation through cross attention to obtain the initial generative text modality sequence representation. When the missing type is a missing numerical mode, the initial observed mode sequence representation is an initial text mode sequence representation. The numerical target query library is used as the query, and the output of the initial text mode sequence representation after linear projection is used as the key and value. Sequence information related to the target numerical mode is extracted from the initial text mode sequence representation through cross attention to obtain the initial generative numerical mode sequence representation. When the missing type is complete data, the initial observation modality sequence representation includes the initial numerical modality sequence representation and the initial text modality sequence representation.

[0010] Optionally, the modal signal cue includes text observation cue, text generation cue, numerical observation cue, and numerical generation cue; the observation modal sequence representation includes numerical modal sequence representation or text modal sequence representation; and the generative missing modal sequence representation includes generative text modal sequence representation or generative numerical modal sequence representation. The step of adding modal signal cues to the initial observed modal sequence representation and the initial generative missing modal sequence representation to obtain the observed modal sequence representation and the generative missing modal sequence representation, specifically includes: When the missing type is a missing text modality, an additive injection method is used to add the numerical observation cue to the initial numerical modality sequence representation to obtain the numerical modality sequence representation, and to add the text generation cue to the initial generative text modality sequence representation to obtain the generative text modality sequence representation. When the missing type is a missing numerical mode, an additive injection method is used to add the text observation cue to the initial text mode sequence representation to obtain the text mode sequence representation, and to add the numerical generation cue to the initial generative numerical mode sequence representation to obtain the generative numerical mode sequence representation. When the missing type is complete data, an additive injection method is used to add the numerical observation cue to the initial numerical modality sequence representation to obtain the numerical modality sequence representation, and to add the text observation cue to the initial text modality sequence representation to obtain the text modality sequence representation.

[0011] Optionally, the cross-modal interaction representation includes a first cross-modal interaction representation, a second cross-modal interaction representation, a third cross-modal interaction representation, a fourth cross-modal interaction representation, a fifth cross-modal interaction representation, and a sixth cross-modal interaction representation; the missing type prompt includes a missing text prompt, a missing numerical value prompt, and a complete data prompt; and the cross-modal fusion sequence includes a first cross-modal fusion sequence, a second cross-modal fusion sequence, and a third cross-modal fusion sequence. The process involves bidirectional cross-modal fusion of the observed modal sequence representation and the generative missing modal sequence representation through cross-modal attention to obtain a cross-modal interactive representation. Furthermore, the cross-modal interactive representation, after adding missing type hints, undergoes memory encoding and fusion to obtain a cross-modal fused sequence. Specifically, this includes: When the missing type is missing text modality, the generative text modality sequence representation is used as the query, and the numerical modality sequence representation is used as the key and value. The first cross-modal interaction representation of the text after being constrained by numerical conditions is calculated through cross-modal attention. The second cross-modal interaction representation of the numerical value after being constrained by text conditions is calculated using the numerical modality sequence representation as the query and the generative text modality sequence representation as the key and value. The missing text prompts corresponding to the missing text modalities are concatenated to the end of the first cross-modal interaction representation and the second cross-modal interaction representation, respectively; The first cross-modal interaction representation after splicing the missing text prompt is input into the text memory encoder. The text memory encoder performs memory encoding on the first cross-modal interaction representation after splicing the missing text prompt through self-attention and feedforward network to obtain the first text memory representation. The second cross-modal interaction representation after splicing the missing text prompt is input into the numerical memory encoder. The numerical memory encoder performs memory encoding on the second cross-modal interaction representation after splicing the missing text prompt through self-attention and feedforward network to obtain the first numerical memory representation. The first text memory representation and the first numerical memory representation are concatenated along the feature dimension to obtain the first cross-modal fusion sequence; When the missing type is a missing numerical modality, the text modality sequence representation is used as the query, the generative numerical modality sequence representation is used as the key and value, and the third cross-modal interaction representation after the text is constrained by numerical conditions is calculated through cross-modal attention. The fourth cross-modal interaction representation after the numerical value is constrained by text conditions is calculated using the generative numerical modality sequence representation as the query and the text modality sequence representation as the key and value. The missing value prompts corresponding to the missing value modalities are concatenated to the end of the third cross-modal interaction representation and the fourth cross-modal interaction representation, respectively; The third cross-modal interaction representation after splicing the missing numerical prompt is input into the text memory encoder. The text memory encoder performs memory encoding on the third cross-modal interaction representation after splicing the missing numerical prompt through self-attention and feedforward network to obtain the second text memory representation. The fourth cross-modal interaction representation after splicing missing numerical prompts is input into the numerical memory encoder. The numerical memory encoder performs memory encoding on the fourth cross-modal interaction representation after splicing missing numerical prompts through self-attention and feedforward network to obtain the second numerical memory representation. The second text memory representation and the second numerical memory representation are concatenated along the feature dimension to obtain the second cross-modal fusion sequence; When the missing type is complete data, the text modal sequence representation is used as the query and the numerical modal sequence representation is used as the key and value. The fifth cross-modal interaction representation after the text is constrained by numerical conditions is calculated through cross-modal attention. The sixth cross-modal interaction representation after the numerical value is constrained by text conditions is calculated using the numerical modal sequence representation as the query and the text modal sequence representation as the key and value. The complete data prompts corresponding to the complete data are concatenated to the end of the fifth cross-modal interaction representation and the sixth cross-modal interaction representation, respectively; The fifth cross-modal interaction representation after piecing together complete data prompts is input into a text memory encoder. The text memory encoder uses self-attention and a feedforward network to memorize and encode the fifth cross-modal interaction representation after piecing together complete data prompts, resulting in a third text memory representation. The sixth cross-modal interaction representation after splicing complete data prompts is input into the numerical memory encoder. The numerical memory encoder performs memory encoding on the sixth cross-modal interaction representation after splicing complete data prompts through self-attention and feedforward network to obtain the third numerical memory representation. The third text memory representation and the third numerical memory representation are concatenated along the feature dimension to obtain the third cross-modal fusion sequence.

[0012] Optionally, the timing prompts include timing prompts for missing text modalities, timing prompts for missing numerical modalities, and timing prompts for complete data; the timing aggregation vectors include a first timing aggregation vector, a second timing aggregation vector, and a third timing aggregation vector. The step of performing temporal aggregation on the cross-modal fusion sequence using multi-head attention based on the temporal cue corresponding to the missing type to obtain a temporal aggregation vector specifically includes: When the missing type is a missing text modality, the missing text modality time sequence hint corresponding to the missing text modality is used as the query, and the representation of the first cross-modal fusion sequence on the shared time grid is used as the key and value. The first attention weight is obtained by multi-head attention calculation. Based on the first attention weight, the representation of the first cross-modal fusion sequence on the shared time grid is weighted and aggregated to obtain the first temporal aggregation vector; When the missing type is a missing numerical mode, the missing numerical mode time sequence hint corresponding to the missing numerical mode is used as the query, and the representation of the second cross-modal fusion sequence on the shared time grid is used as the key and value. The second attention weight is obtained by multi-head attention calculation. Based on the second attention weight, the representation of the second cross-modal fusion sequence on the shared time grid is weighted and aggregated to obtain the second temporal aggregation vector; When the missing type is complete data, the complete data time series prompt corresponding to the complete data is used as the query, and the representation of the third cross-modal fusion sequence on the shared time grid is used as the key and value. The third attention weight is obtained by multi-head attention calculation. Based on the third attention weight, the representation of the third cross-modal fusion sequence on the shared time grid is weighted and aggregated to obtain the third temporal aggregation vector.

[0013] To achieve the above-mentioned objectives, the present invention also provides a mortality risk prediction system for incomplete multimodal electronic health records, wherein the mortality risk prediction system for incomplete multimodal electronic health records includes: The missing type determination module is used to obtain the patient's multimodal electronic health record and the mask corresponding to the multimodal electronic health record, and determine the missing type of the multimodal electronic health record based on the mask; The encoding and alignment module is used to encode and time-align the observation modality sequence in the multimodal electronic health record according to the missing type, so as to obtain an initial observation modality sequence characterization; A completion module is used to extract sequence information related to missing modes from the initial observed mode sequence representation through cross attention, so as to obtain an initial generative missing mode sequence representation. The modal signal cueing module is used to add modal signal cueing to the initial observed modal sequence representation and the initial generative missing modal sequence representation respectively, so as to obtain the observed modal sequence representation and the generative missing modal sequence representation; The cross-modal fusion and memory encoding module is used to perform bidirectional cross-modal cross-fusion of the observed modal sequence representation and the generative missing modal sequence representation through cross-modal attention to obtain cross-modal interactive representation, and to perform memory encoding and fusion of the cross-modal interactive representation after adding missing type hints to obtain cross-modal fusion sequence; The temporal aggregation module is used to perform temporal aggregation on the cross-modal fusion sequence through multi-head attention based on the temporal cue corresponding to the missing type, so as to obtain a temporal aggregation vector. The mortality risk prediction module is used to perform nonlinear transformation and mapping on the time-series aggregated vector through a residual multilayer perceptron and a classification layer to obtain the predicted mortality risk probability of the patient.

[0014] To achieve the above-mentioned objectives, the present invention also provides a terminal, the terminal comprising: a memory, a processor, and a mortality risk prediction program for incomplete multimodal electronic health records stored in the memory and executable on the processor, wherein the mortality risk prediction program for incomplete multimodal electronic health records, when executed by the processor, implements the steps of the mortality risk prediction method for incomplete multimodal electronic health records as described above.

[0015] To achieve the above-mentioned objectives, the present invention also provides a computer-readable storage medium storing a mortality risk prediction program for incomplete multimodal electronic health records. When executed by a processor, the mortality risk prediction program for incomplete multimodal electronic health records implements the steps of the mortality risk prediction method for incomplete multimodal electronic health records as described above.

[0016] In this invention, a patient's multimodal electronic health record and a corresponding mask are obtained. Based on the mask, the missing type of the multimodal electronic health record is determined. According to the missing type, the observed modality sequence in the multimodal electronic health record is encoded and time-aligned to obtain an initial observed modality sequence representation. Sequence information related to the missing modality is extracted from the initial observed modality sequence representation through cross-attention to obtain an initial generative missing modality sequence representation. Modality signal cues are added to both the initial observed modality sequence representation and the initial generative missing modality sequence representation to obtain the observed modality sequence representation. The invention employs a two-stream fusion method: generating modal sequence representation and generative missing modal sequence representation. Cross-modal attention is used to perform bidirectional cross-modal cross-fusion of the observed modal sequence representation and the generative missing modal sequence representation to obtain a cross-modal interactive representation. This cross-modal interactive representation, after adding missing type cues, undergoes memory encoding and fusion to obtain a cross-modal fused sequence. Based on the temporal cues corresponding to the missing type, multi-head attention is used to perform temporal aggregation on the cross-modal fused sequence to obtain a temporal aggregation vector. A residual multilayer perceptron and a classification layer are used to perform nonlinear transformation and mapping on the temporal aggregation vector to obtain the predicted probability of patient mortality risk. This invention maintains the dual-stream fusion structure through generative cues, explicitly labels feature sources through modal signal cues, adjusts the cross-modal fusion strategy for different missing scenarios through missing type cues, and adapts the temporal evidence focused on by the temporal aggregation to the missing pattern through missing pattern awareness. This significantly improves the accuracy of inpatient mortality risk prediction for incomplete multimodal electronic health records, enhances the hospital's early warning capabilities, avoids underreporting of high-risk patients, and prevents excessive alarms leading to wasted medical resources. Attached Figure Description

[0017] Figure 1 This is a flowchart of a preferred embodiment of the present invention for predicting mortality risk based on incomplete multimodal electronic health records; Figure 2 This is another flowchart of a preferred embodiment of the present invention for predicting mortality risk based on incomplete multimodal electronic health records; Figure 3 This is a structural diagram of a preferred embodiment of the mortality risk prediction system for incomplete multimodal electronic health records of the present invention; Figure 4 This is another structural diagram of a preferred embodiment of the mortality risk prediction system for incomplete multimodal electronic health records of the present invention; Figure 5 This is a structural diagram of the generative prompt module of the present invention; Figure 6 This is a structural diagram of the time reminder enhancement module of the present invention; Figure 7This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] The clinical status of patients in the intensive care unit (ICU) is typically reflected by a combination of information from electronic health records, including multivariate irregular sampling time series of vital signs and laboratory tests, as well as clinical texts recorded by physicians and nurses. Both types of information are generated at irregular time points: numerical variables are monitored or tested at different frequencies, while clinical texts are written at irregular times. For a binary classification task of predicting in-hospital mortality risk within the first 48-hour observation window after ICU admission (48-IHM), structured numerical time series and clinical texts can be used for joint modeling to achieve in-hospital mortality risk prediction.

[0020] In real-world clinical scenarios, missing information can be categorized into two distinct types: intramodal missing information, where a modality exists but is sampled irregularly and sparsely within the observation window; and modality-level missing information, where an entire modality is completely unavailable for a particular hospitalization, such as the complete absence of clinical text or numerical time series data. In the case of modality-level missing information, the encoder branch on the missing side has no observations to process, causing models relying on two-stream cross-modal fusion to degenerate into single-stream networks. Cross-modal attention cannot function properly, leading to a decline in model predictive performance.

[0021] To address the modality-level missing problem in the 48-IHM binary classification task, existing technologies include: The first category comprises methods based on interpolation or temporal discretization. These methods map irregularly sampled time series onto a regular grid with fixed time intervals, generating regularized features through forward padding, mean interpolation, kernel interpolation, or learnable interpolation networks, and then using convolutional neural networks, recurrent neural networks, or Transformers for prediction. While these methods can handle irregular sampling within a modality to some extent, they struggle to maintain the original multimodal fusion structure when the entire text modality or the entire numerical temporal modality is missing.

[0022] The second category comprises methods based on time-aware recurrent neural networks (RNNs). These methods introduce time decay mechanisms into RNNs, GRUs (Gated Recurrent Units), and LSTMs (Long Short-Term Memory) to model irregular time intervals between adjacent observations. Typical examples include GRU-D (GRU with Decay) and T-LSTM (Time-aware LSTM). These methods are suitable for handling missing data and irregular intervals in time series, but when clinical texts and structured time series coexist, additional cross-modal fusion mechanisms are usually required; furthermore, the fusion structure may degrade when a particular modality is entirely missing.

[0023] The third category consists of graph neural network-based methods. These methods treat different clinical variables as nodes and model the dependencies between variables through a graph structure, such as RAINDROP. These methods can enhance the expression of variable relationships within multivariate time series, but they are primarily geared towards numerical time series modalities and struggle to directly handle clinical text modalities and cross-modal missing data completion.

[0024] The fourth category comprises clinical text modeling methods based on pre-trained language models. These methods utilize BERT (Bidirectional Encoder Representations from Transformers), BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), ClinicalBERT (Clinical Bidirectional Encoder Representations from Transformers), and Clinical-Longformer to extract clinical text representations. Patient-level text representations are then obtained through average pooling, hierarchical Transformers, or time-aware text models. While these methods can handle the semantics of clinical text, their reliance on and fusion of structured temporal modalities are often insufficient, and most methods ignore the irregularities in the writing time of clinical records.

[0025] The fifth category is multimodal electronic health record fusion methods. These methods encode structured temporal features and clinical text features separately, then perform joint prediction through concatenation, attention mechanisms, cross-modal Transformers, or gated fusion. These methods perform well when both modalities are present, but it is generally assumed that multiple modalities are available. When a complete text modality or a complete numerical temporal modality is missing, the two-stream structure degenerates into a single-stream structure, cross-modal attention fails, and predictive performance declines.

[0026] The sixth category is multimodal cue learning methods for missing modalities. These methods address missing modalities in the latent space through learnable cues, but existing work mainly focuses on general multimodal tasks such as regular sampling or modality time alignment, and has not been designed for irregular sampling, clinical text timestamps, and ICU 48-hour inpatient death binary classification tasks, making it difficult to directly transfer to irregular multimodal EHR.

[0027] The shortcomings of existing technologies include: (1) Existing irregular EHR models are usually applicable to situations where numerical time series and clinical text are available at the same time, or only model a single modality. They lack a unified processing mechanism for the three types of missing states at the sample level: missing text, missing numerical values, and complete data.

[0028] (2) The partial missing completion method directly generates the missing mode representation based on the observed mode, without explicitly informing the subsequent fusion module which representations come from real observations and which representations come from the generated completion, which can easily cause confusion between real information and generated information.

[0029] (3) Existing fusion networks often treat different missing patterns as the same input distribution, lacking learnable conditional representations for the three types of missing states: missing text, missing values, and complete data, making it difficult to adjust the fusion strategy for different missing scenarios.

[0030] (4) The impact of missing data and observations at different time points in irregular EHRs on the risk assessment of 48-IHM varies, and the temporal evidence that should be considered differs under different missing data patterns. Existing methods mostly use uniform mean pooling or shared attention for temporal aggregation of fused sequences, without coordinating with missing data patterns.

[0031] (5) If only complete modal samples are used during the training phase, the model is not robust enough in actual missing modal reasoning scenarios; if only simple random discarding is used, it is difficult to coordinate with missing types, generation completion and temporal aggregation mechanisms.

[0032] (6) The 48-IHM inpatient mortality binary classification task has significant class imbalance, and conventional cross-entropy training is insufficient for learning a small number of positive samples.

[0033] Therefore, there is a need for a risk prediction method that can maintain a dual-stream fusion structure, explicitly label the source of generated features, conditionally fuse according to the missing type, and perform time-series aggregation with missing pattern awareness even when there are modality-level missing data in multimodal EHRs, and is robust to class imbalance in 48-IHM binary classification.

[0034] To address the aforementioned technical problems, this invention provides a method for predicting mortality risk based on incomplete multimodal electronic health records. The method involves acquiring the patient's multimodal electronic health records and a corresponding mask, and determining the missing type of the multimodal electronic health records based on the mask. According to the missing type, the observed modality sequences in the multimodal electronic health records are encoded and time-aligned to obtain an initial observed modality sequence representation. Sequence information related to the missing modalities is extracted from the initial observed modality sequence representation through cross-attention to obtain an initial generative missing modality sequence representation. The initial observed modality sequence representation and the initial generative missing modality sequence representation are then used as the basis for this method. Modal sequence representations are augmented with modal signal cues to obtain observed modal sequence representations and generative missing modal sequence representations. These representations are then bidirectionally cross-modal fused using cross-modal attention to obtain cross-modal interactive representations. The cross-modal interactive representations, after adding missing type cues, are then memorized and fused to obtain a cross-modal fused sequence. Based on the temporal cues corresponding to the missing type, the cross-modal fused sequence is temporally aggregated using multi-head attention to obtain a temporal aggregated vector. Finally, a residual multilayer perceptron and a classification layer are used to perform nonlinear transformations and mappings on the temporal aggregated vector to obtain the predicted mortality risk probability for the patient. This invention maintains the dual-stream fusion structure through generative prompts, explicitly labels feature sources through modal signal prompts, adjusts cross-modal fusion and memory encoding strategies for different missing scenarios through missing type prompts, and adapts the temporal evidence focused on by missing pattern-aware temporal aggregation to the missing pattern. This significantly improves the accuracy of inpatient mortality risk prediction for incomplete multimodal electronic health records, which is beneficial to improving the early warning capabilities of hospitals, avoiding underreporting of high-risk patients, and avoiding excessive alarms that lead to waste of medical resources.

[0035] The application content will be further explained below with reference to the accompanying drawings and the description of the embodiments.

[0036] A preferred embodiment of the present invention for a mortality risk prediction method based on incomplete multimodal electronic health records, such as... Figure 1 and Figure 2 As shown, it specifically includes: S1. Obtain the patient's multimodal electronic health record and the mask corresponding to the multimodal electronic health record, and determine the missing type of the multimodal electronic health record based on the mask.

[0037] In one implementation of this embodiment, the step of acquiring the patient's multimodal electronic health record and the mask corresponding to the multimodal electronic health record, and determining the missing type of the multimodal electronic health record based on the mask, specifically includes: The system acquires the patient's multimodal electronic health record within a set time period before admission to the department and the corresponding mask for the multimodal electronic health record. The multimodal electronic health record includes a numerical sequence or clinical text, and the mask includes a numerical observation mask corresponding to the numerical sequence and a text time mask corresponding to the clinical text. Based on the numerical observation mask and the text time mask, determine the missing type of the multimodal electronic health record; If the numerical observation mask indicates the existence of valid numerical observations, and the text time mask indicates the absence of valid clinical text records, then the missing type of the multimodal electronic health record is determined to be missing text modality. If the numerical observation mask indicates that there are no valid numerical observations, and the text time mask indicates that there are valid clinical text records, then the missing type of the multimodal electronic health record is determined to be missing numerical modality. If the numerical observation mask indicates the existence of valid numerical observations, and the text time mask indicates the existence of valid clinical text records, then the missing type of the multimodal electronic health record is determined to be complete data.

[0038] Specifically, multimodal electronic health record data (i.e., samples) and their corresponding masks are obtained from ICU patients within the first 48 hours after admission. The first modality is a structured irregular multivariate time series (i.e., numerical series), and the second modality is timestamped clinical text. Based on the text time mask corresponding to the clinical text and the numerical observation mask corresponding to the numerical time series, the modality availability status of each sample within the batch is determined. If the text time mask indicates the existence of at least one valid clinical text record, and the numerical observation mask indicates the absence of a valid numerical observation, the sample is determined to be a missing numerical modality. If the numerical observation mask indicates the existence of a valid numerical observation, and the text time mask indicates the absence of a valid clinical text record, the sample is determined to be a missing text modality. If both the text modality and the numerical modality have valid observations, the sample is determined to be complete data. The result of this missing modality determination is represented by a sample-level missing modality index and serves as the condition input for subsequent generative prompt selection, modality signal prompt selection, missing type prompt selection, and time series prompt query selection.

[0039] S2. Based on the missing type, the observation modality sequence in the multimodal electronic health record is encoded and time-aligned to obtain an initial observation modality sequence characterization.

[0040] In one implementation of this embodiment, the initial observation modality sequence representation includes an initial numerical modality sequence representation or an initial text modality sequence representation, wherein the numerical sequence includes numerical observations and observation timestamps, and the clinical text includes clinical text records and writing timestamps; The step of encoding and time-aligning the observation modality sequences in the multimodal electronic health record according to the missing type to obtain an initial observation modality sequence characterization specifically includes: When the missing type is a missing text modality, the observation modality sequence in the multimodal electronic health record is a numerical sequence, which includes irregular numerical sequences and regular numerical sequences; For the irregular numerical sequence, the numerical observations in the irregular numerical sequence are concatenated with the numerical observation mask corresponding to the irregular numerical sequence to obtain the concatenated numerical observations and numerical observation mask. The concatenated numerical observations and numerical observation mask are then aligned to a shared time grid through temporal embedding and a multi-temporal attention network to obtain the irregular branch representation. For the rule-based numerical sequence, a one-dimensional convolution is performed on the rule-based numerical sequence to obtain the rule-based branch representation; Learnable weights are assigned to the irregular branch representation and the regular branch representation respectively through a gating network. The irregular branch representation and the regular branch representation are then weighted and fused according to the learnable weights to obtain the initial numerical mode sequence representation. When the missing type is a missing numerical modality, the observation modality sequence in the multimodal electronic health record is clinical text; A medical text encoder is used to extract the classification label vector of the clinical text record in the clinical text; Based on the writing timestamp in the clinical text and the text time mask corresponding to the clinical text, the classification label vector is aligned to the shared time grid through multi-temporal attention to obtain the initial text modality sequence representation; When the missing type is complete data, the observation modality sequence in the multimodal electronic health record is a numerical sequence and clinical text; The numerical sequence and the clinical text are encoded and time-aligned respectively to obtain the initial numerical modality sequence representation and the initial text modality sequence representation.

[0041] Specifically, based on the sample-level missing patterns obtained in step S1, only modalities with valid observations in the current sample (i.e., observation modal sequences) are subjected to raw data encoding and time alignment. For numerical modalities with valid observations, the raw numerical observations, numerical observation masks, and observation timestamps are received. The observations and masks are concatenated and input into a time embedding and multi-temporal attention network to obtain irregular branch representations on a 48-hour shared time grid. In the case of regularized numerical sequences, one-dimensional convolution is performed on the most recently observed forward-filled or equivalently regularized numerical sequences to obtain regular branch representations. Then, a gating network is used to assign learnable weights to the irregular branch representations and regular branch representations, and they are fused to obtain the numerical modal sequence representation (i.e., the initial numerical modal sequence representation). For text modalities with valid observations, the classification label vector of each clinical text record is first extracted using a medical text encoder. Then, based on the writing timestamp and text time mask of the clinical text record, the classification label vector is aligned to the same 48-hour shared time grid through multi-temporal attention to obtain the text modal sequence representation (i.e., the initial text modal sequence representation). For missing modes, encoding based on the original observations is not performed; instead, a latent sequence representation is generated in a subsequent step using a generative prompting module. Thus, both numerical and textual modes are mapped to sequence representations with a length corresponding to a 48-hour shared time grid, providing a unified input for subsequent completion and cross-modal fusion.

[0042] S3. Extract sequence information related to missing modes from the initial observed mode sequence representation through cross attention to obtain the initial generative missing mode sequence representation.

[0043] In one implementation of this embodiment, the initial generative missing modality sequence representation includes an initial generative text modality sequence representation or an initial generative numerical modality sequence representation. The step of extracting sequence information related to the missing modes from the initial observed mode sequence representation through cross-attention to obtain the initial generative missing mode sequence representation specifically includes: When the missing type is a missing text modality, the initial observed modality sequence representation is an initial numerical modality sequence representation. The text target query library is used as the query, and the output of the initial numerical modality sequence representation after linear projection is used as the key and value. Sequence information related to the target text modality is extracted from the initial numerical modality sequence representation through cross attention to obtain the initial generative text modality sequence representation. When the missing type is a missing numerical mode, the initial observed mode sequence representation is an initial text mode sequence representation. The numerical target query library is used as the query, and the output of the initial text mode sequence representation after linear projection is used as the key and value. Sequence information related to the target numerical mode is extracted from the initial text mode sequence representation through cross attention to obtain the initial generative numerical mode sequence representation. When the missing type is complete data, the initial observation modality sequence representation includes the initial numerical modality sequence representation and the initial text modality sequence representation.

[0044] Specifically, for samples identified as having missing modalities, a sequence representation corresponding to the missing modality is generated in the latent representation space, rather than directly imputing in the original data space. For samples with missing text modalities, a text target query library is used as the query. The numerical modality sequence representation (i.e., the initial numerical modality sequence representation) is linearly projected and used as the key and value. Sequence information related to the target text modality is extracted from the numerical modality through multi-head cross-attention to generate a text modality sequence representation (i.e., the initial generative text modality sequence representation). For samples with missing numerical modalities, a numerical target query library is used as the query. The text modality sequence representation (i.e., the initial text modality sequence representation) is linearly projected and used as the key and value. Sequence information related to the target numerical modality is extracted from the text modality through multi-head cross-attention to generate a numerical modality sequence representation (i.e., the initial generative numerical modality sequence representation). The target query library is initialized by the global mean vector of the corresponding target modality in the training set and expanded into a learnable query sequence according to a 48-hour time grid, and is updated together with the prediction target during training. The generated missing modality sequence representations are processed by layer normalization and then used to replace the empty input of the missing side branch. For complete data samples, the original encoded representations of both modalities are retained, and no replacement is performed. Through this step, when either textual or numerical modality is completely missing, the model can still construct two sequence representations compatible with the downstream two-stream cross-modal fusion structure. In other words, the missing modality sequence representation generation is used to recover or maintain the two-stream fusion structure.

[0045] S4. Add modal signal cues to the initial observed modal sequence representation and the initial generative missing modal sequence representation respectively to obtain the observed modal sequence representation and the generative missing modal sequence representation.

[0046] In one implementation of this embodiment, the modal signal cue includes text observation cue, text generation cue, numerical observation cue, and numerical generation cue; the observation modal sequence representation includes numerical modal sequence representation or text modal sequence representation; and the generative missing modal sequence representation includes generative text modal sequence representation or generative numerical modal sequence representation. The step of adding modal signal cues to the initial observed modal sequence representation and the initial generative missing modal sequence representation to obtain the observed modal sequence representation and the generative missing modal sequence representation, specifically includes: When the missing type is a missing text modality, an additive injection method is used to add the numerical observation cue to the initial numerical modality sequence representation to obtain the numerical modality sequence representation, and to add the text generation cue to the initial generative text modality sequence representation to obtain the generative text modality sequence representation. When the missing type is a missing numerical mode, an additive injection method is used to add the text observation cue to the initial text mode sequence representation to obtain the text mode sequence representation, and to add the numerical generation cue to the initial generative numerical mode sequence representation to obtain the generative numerical mode sequence representation. When the missing type is complete data, an additive injection method is used to add the numerical observation cue to the initial numerical modality sequence representation to obtain the numerical modality sequence representation, and to add the text observation cue to the initial text modality sequence representation to obtain the text modality sequence representation.

[0047] Specifically, after generating the missing modal sequence representation, learnable modal signal cues are added to both the text branch and the numerical branch to explicitly distinguish between real observation representations and generated complete representations. Modal signal cues include at least four categories: text observation cues, text generation cues, numerical observation cues, and numerical generation cues. For missing text samples, text generation cues are added to the generated text modal sequence representation, and numerical observation cues are added to the real numerical modal sequence representation; for missing numerical samples, text observation cues are added to the real text modal sequence representation, and numerical generation cues are added to the generated numerical modal sequence representation; for complete data samples, text observation cues and numerical observation cues are added to the real text modal sequence representation and the real numerical modal sequence representation, respectively. These cues are added additively to each time grid position of the corresponding modal sequence representation, enabling the subsequent cross-modal fusion module to identify the source of each representation and reducing the risk of real observation information and generated complete information being mixed up.

[0048] S5. Perform bidirectional cross-modal cross-fusion of the observed modal sequence representation and the generative missing modal sequence representation through cross-modal attention to obtain cross-modal interactive representation, and perform memory encoding and fusion on the cross-modal interactive representation after adding missing type prompts to obtain cross-modal fused sequence.

[0049] In one implementation of this embodiment, the cross-modal interaction representation includes a first cross-modal interaction representation, a second cross-modal interaction representation, a third cross-modal interaction representation, a fourth cross-modal interaction representation, a fifth cross-modal interaction representation, and a sixth cross-modal interaction representation; the missing type prompt includes a missing text prompt, a missing numerical value prompt, and a complete data prompt; and the cross-modal fusion sequence includes a first cross-modal fusion sequence, a second cross-modal fusion sequence, and a third cross-modal fusion sequence. The process involves bidirectional cross-modal fusion of the observed modal sequence representation and the generative missing modal sequence representation through cross-modal attention to obtain a cross-modal interactive representation. Furthermore, the cross-modal interactive representation, after adding missing type hints, undergoes memory encoding and fusion to obtain a cross-modal fused sequence. Specifically, this includes: When the missing type is missing text modality, the generative text modality sequence representation is used as the query, and the numerical modality sequence representation is used as the key and value. The first cross-modal interaction representation of the text after being constrained by numerical conditions is calculated through cross-modal attention. The second cross-modal interaction representation of the numerical value after being constrained by text conditions is calculated using the numerical modality sequence representation as the query and the generative text modality sequence representation as the key and value. The missing text prompts corresponding to the missing text modalities are concatenated to the end of the first cross-modal interaction representation and the second cross-modal interaction representation, respectively; The first cross-modal interaction representation after splicing the missing text prompt is input into the text memory encoder. The text memory encoder performs memory encoding on the first cross-modal interaction representation after splicing the missing text prompt through self-attention and feedforward network to obtain the first text memory representation. The second cross-modal interaction representation after splicing the missing text prompt is input into the numerical memory encoder. The numerical memory encoder performs memory encoding on the second cross-modal interaction representation after splicing the missing text prompt through self-attention and feedforward network to obtain the first numerical memory representation. The first text memory representation and the first numerical memory representation are concatenated along the feature dimension to obtain the first cross-modal fusion sequence; When the missing type is a missing numerical modality, the text modality sequence representation is used as the query, the generative numerical modality sequence representation is used as the key and value, and the third cross-modal interaction representation after the text is constrained by numerical conditions is calculated through cross-modal attention. The fourth cross-modal interaction representation after the numerical value is constrained by text conditions is calculated using the generative numerical modality sequence representation as the query and the text modality sequence representation as the key and value. The missing value prompts corresponding to the missing value modalities are concatenated to the end of the third cross-modal interaction representation and the fourth cross-modal interaction representation, respectively; The third cross-modal interaction representation after splicing the missing numerical prompt is input into the text memory encoder. The text memory encoder performs memory encoding on the third cross-modal interaction representation after splicing the missing numerical prompt through self-attention and feedforward network to obtain the second text memory representation. The fourth cross-modal interaction representation after splicing missing numerical prompts is input into the numerical memory encoder. The numerical memory encoder performs memory encoding on the fourth cross-modal interaction representation after splicing missing numerical prompts through self-attention and feedforward network to obtain the second numerical memory representation. The second text memory representation and the second numerical memory representation are concatenated along the feature dimension to obtain the second cross-modal fusion sequence; When the missing type is complete data, the text modal sequence representation is used as the query and the numerical modal sequence representation is used as the key and value. The fifth cross-modal interaction representation after the text is constrained by numerical conditions is calculated through cross-modal attention. The sixth cross-modal interaction representation after the numerical value is constrained by text conditions is calculated using the numerical modal sequence representation as the query and the text modal sequence representation as the key and value. The complete data prompts corresponding to the complete data are concatenated to the end of the fifth cross-modal interaction representation and the sixth cross-modal interaction representation, respectively; The fifth cross-modal interaction representation after piecing together complete data prompts is input into a text memory encoder. The text memory encoder uses self-attention and a feedforward network to memorize and encode the fifth cross-modal interaction representation after piecing together complete data prompts, resulting in a third text memory representation. The sixth cross-modal interaction representation after splicing complete data prompts is input into the numerical memory encoder. The numerical memory encoder performs memory encoding on the sixth cross-modal interaction representation after splicing complete data prompts through self-attention and feedforward network to obtain the third numerical memory representation. The third text memory representation and the third numerical memory representation are concatenated along the feature dimension to obtain the third cross-modal fusion sequence.

[0050] Specifically, after incorporating modal signal cues, bidirectional cross-modal cross-fusion is performed using both text and numerical branches as queries. Using text modal sequence representations as queries and numerical modal sequence representations as keys and values, cross-modal interaction representations of text constrained by numerical conditions (i.e., text cross-modal interaction sequences) are calculated; simultaneously, using numerical modal sequence representations as queries and text modal sequence representations as keys and values, cross-modal interaction representations of numerical conditions constrained by text conditions (i.e., numerical cross-modal interaction sequences) are calculated. Subsequently, based on the obtained sample-level missing patterns, corresponding learnable cue token sequences are selected from the missing type cue library. The missing type cue library includes at least three categories: missing numerical cue, missing text cue, and complete data cue, and the missing type cue selected for the same sample is shared between the text and numerical branches. The selected missing type cue sequences are concatenated to the end of both the text and numerical cross-modal interaction sequences, and then input into the text memory encoder and numerical memory encoder, respectively. The memory encoder performs deep encoding on the concatenated cross-modal interaction sequence using self-attention and a feedforward network, enabling both modal representations to retain their respective semantics while acquiring contextual memory under the current missing state (i.e., missing type / missing pattern). After encoding, the text memory representation and the numerical memory representation are concatenated along the feature dimension to form a cross-modal fusion sequence. Through this step, the three states—missing numerical values, missing text, and complete data—are no longer considered as the same input distribution, but rather participate in cross-modal fusion and deep memory encoding in the form of learnable cues.

[0051] S6. Based on the temporal cue corresponding to the missing type, perform temporal aggregation on the cross-modal fusion sequence through multi-head attention to obtain a temporal aggregation vector.

[0052] In one implementation of this embodiment, the timing prompts include missing text modal timing prompts, missing numerical modal timing prompts, and complete data timing prompts, and the timing aggregation vectors include a first timing aggregation vector, a second timing aggregation vector, and a third timing aggregation vector; The step of performing temporal aggregation on the cross-modal fusion sequence using multi-head attention based on the temporal cue corresponding to the missing type to obtain a temporal aggregation vector specifically includes: When the missing type is a missing text modality, the missing text modality time sequence hint corresponding to the missing text modality is used as the query, and the representation of the first cross-modal fusion sequence on the shared time grid is used as the key and value. The first attention weight is obtained by multi-head attention calculation. Based on the first attention weight, the representation of the first cross-modal fusion sequence on the shared time grid is weighted and aggregated to obtain the first temporal aggregation vector; When the missing type is a missing numerical mode, the missing numerical mode time sequence hint corresponding to the missing numerical mode is used as the query, and the representation of the second cross-modal fusion sequence on the shared time grid is used as the key and value. The second attention weight is obtained by multi-head attention calculation. Based on the second attention weight, the representation of the second cross-modal fusion sequence on the shared time grid is weighted and aggregated to obtain the second temporal aggregation vector; When the missing type is complete data, the complete data time series prompt corresponding to the complete data is used as the query, and the representation of the third cross-modal fusion sequence on the shared time grid is used as the key and value. The third attention weight is obtained by multi-head attention calculation. Based on the third attention weight, the representation of the third cross-modal fusion sequence on the shared time grid is weighted and aggregated to obtain the third temporal aggregation vector.

[0053] Specifically, after obtaining the cross-modal fusion sequence, the fusion representation corresponding to the 48-hour time grid position is taken as the key and value for temporal aggregation. For missing type cue tokens appended to the end of the sequence, they are not included in the temporal aggregation as time grid positions. A set of learnable temporal cue vectors is pre-set, the number of which corresponds to the number of missing pattern categories. The corresponding temporal cue vector is selected as the query according to the obtained sample-level missing patterns. Multi-head attention is performed on the selected temporal cue query (i.e., temporal cue / temporal cue vector) and the fusion representation on the 48-hour time grid to obtain the attention weight for the current missing pattern. Based on this attention weight, the representation of the fusion sequence on the time grid is weighted and aggregated to generate a temporal aggregation vector related to the missing pattern. This temporal aggregation vector is the output of the temporal prompt module and is fed into the prediction output module as a patient-level prediction feature. Therefore, in three different scenarios—missing text, missing numerical values, and complete data—the model can select different time-series cue queries and conditionally aggregate different temporal evidence within a 48-hour observation window, instead of using the same mean pooling or shared attention aggregation for all missing states.

[0054] S7. Using a residual multilayer perceptron and a classification layer, the temporal aggregation vector is nonlinearly transformed and mapped to obtain the predicted probability of the patient's mortality risk.

[0055] Specifically, the patient-level prediction features obtained from S6 are input into the residual multilayer perceptron (MLP) and classification layer to obtain binary classification output logits (unnormalized raw output); then, the probability values ​​corresponding to the inpatient mortality category are obtained through softmax (normalized exponential function) (normalized output), which serve as the prediction results of inpatient mortality risk in the first 48 hours after ICU admission.

[0056] Furthermore, during the training phase, the missing modality robust risk prediction model constructed from the above steps is used as the training object, and the real labels of hospitalized deaths are used as the supervision signal. To address the class imbalance in the 48-IHM binary classification task, cross-entropy loss with class weights can be used to enable the model to learn the shared prediction structure of complete samples, missing text samples, and missing numerical samples, while improving the ability to identify a small number of death samples.

[0057] This invention proposes a mortality risk prediction method for incomplete multimodal electronic health records, specifically for a binary classification task predicting in-hospital mortality risk in ICU patients within the first 48 hours of admission. The method acquires the patient's currently observable numerical irregular time series and / or clinical text sequences, and determines sample-level missing patterns based on numerical observation masks and text time masks. Subsequently, modalities with valid observations are encoded and aligned to a 48-hour shared time grid, introducing generative cues, modal signal cues, missing type cues, and temporal cues into the latent representation space. Specifically, when sample-level modal missingness exists, a learnable query library initialized with the global mean of the target modality training set is used to generate missing modality sequence representations from the observed modality through multi-head cross-attention. Furthermore, modal signal cues are used to label the observed and generated representations, and conditional modeling is performed on three types of missing states: missing values, missing text, and complete data, based on missing type cues. After bidirectional cross-modal interaction and deep memory encoding, learnable temporal cues selected according to the missing pattern are used as queries to perform temporal aggregation of the fused sequence representations at 48-hour time grid positions, resulting in a temporal aggregation vector related to the missing pattern. This temporal aggregation vector serves as the output of the temporal prompt and is fed into the prediction output module as a patient-level prediction feature. Finally, the inpatient mortality risk probability is output via a residual multilayer perceptron. During the training phase, a robust risk prediction model for missing modalities, formed by missing pattern judgment, observed modality encoding, missing modality representation generation, cue injection, cross-modal fusion, temporal aggregation, and prediction output, is used as the training object, and supervised training is performed using cross-entropy loss with class weights to address class imbalance.

[0058] This invention has at least the following technical effects: (1) Generative prompts are used to generate missing modal representations from the observed modalities, so that the model can maintain a two-branch fusion structure when text or numerical modalities are missing.

[0059] (2) By using modal signals to explicitly mark the sources of real observations and generated completions, the model can interpret the observed features and generated features in different ways.

[0060] (3) By using missing type prompts, the three states of missing values, missing text, and complete data are injected into memory encoding as learnable global conditions, so that the model can adjust cross-modal interaction strategies for different missing scenarios.

[0061] (4) By using time-series cue aggregation with missing pattern conditionalization, the time evidence that the time-series aggregation focuses on adapts to the missing pattern and forms patient-level predictive features for the current missing state.

[0062] (5) By simulating modal missing scenarios during the training phase, the model can share parameters between complete data and missing modal data and improve robustness.

[0063] (6) The class imbalance in the 48-IHM binary classification task can be alleviated by using weighted Focal Loss or weighted cross-entropy.

[0064] The present invention also provides the following embodiments: Example 1 (Complete Data Scenario): The input consists of numerical time series data and clinical text from the first 48 hours after a patient enters the ICU. Both modalities exist. No generation and completion are performed. The original coded representation is preserved. Observation signal prompts and complete data type prompts are added. The output is the probability of in-hospital mortality risk.

[0065] Example 2 (Missing Text Scenario): Input only irregular time series values, use a value-to-text generator to generate text branch representations, add text generation hints and missing text type hints, use time series aggregation to query the time series hints corresponding to the missing text, and output the probability of hospitalization mortality risk.

[0066] Example 3 (Missing Value Scenario): Input only a clinical text sequence, use a text-to-value generator to generate numerical branch representations, add numerical generation hints and missing value type hints, use time series aggregation to query the time series hints corresponding to the missing values, and output the probability of inpatient mortality risk.

[0067] Example 4 (Implementation of Temporal Hint Aggregation Features): Select temporal hints as queries according to the missing patterns, perform multi-head attention on the representation of the fused sequence at the time grid position to obtain the temporal aggregation vector related to the missing patterns; use this temporal aggregation vector as the patient-level prediction feature, input it into the residual multilayer perceptron and the classification layer to obtain the in-hospital mortality risk probability.

[0068] Furthermore, based on the aforementioned method for predicting mortality risk using incomplete multimodal electronic health records, this invention also provides a system for predicting mortality risk using incomplete multimodal electronic health records. A preferred embodiment of the system for predicting mortality risk using incomplete multimodal electronic health records is as follows: Figure 3 As shown, it specifically includes: The missing type determination module 01 is used to obtain the patient's multimodal electronic health record and the mask corresponding to the multimodal electronic health record, and determine the missing type of the multimodal electronic health record based on the mask; The encoding and alignment module 02 is used to encode and time-align the observation modality sequence in the multimodal electronic health record according to the missing type, so as to obtain the initial observation modality sequence characterization; The generation completion module 03 is used to extract sequence information related to the missing modes from the initial observed mode sequence representation through cross attention, so as to obtain the initial generative missing mode sequence representation. Modal signal prompting module 04 is used to add modal signal prompts to the initial observed modal sequence representation and the initial generative missing modal sequence representation respectively, so as to obtain the observed modal sequence representation and the generative missing modal sequence representation; The cross-modal fusion and memory encoding module 05 is used to perform bidirectional cross-modal cross-fusion of the observed modal sequence representation and the generative missing modal sequence representation through cross-modal attention to obtain cross-modal interactive representation, and to perform memory encoding and fusion of the cross-modal interactive representation after adding missing type prompts to obtain cross-modal fusion sequence; The temporal aggregation module 06 is used to perform temporal aggregation on the cross-modal fusion sequence through multi-head attention based on the temporal cues corresponding to the missing type, so as to obtain a temporal aggregation vector; The mortality risk prediction module 07 is used to perform nonlinear transformation and mapping on the time-series aggregated vector through a residual multilayer perceptron and a classification layer to obtain the patient's mortality risk prediction probability.

[0069] This invention also provides another preferred embodiment of a mortality risk prediction system for incomplete multimodal electronic health records, such as... Figure 4 As shown, from top to bottom, they are: multimodal EHR input (data acquisition module) 100, missing mode judgment module 130, numerical time series encoding module 110 and text encoding module 120, generative prompt module 140, modal signal prompt module 150, missing type prompt module 160, cross-modal fusion and memory encoding module 170, time prompt enhancement module 180, and prediction output module 190.

[0070] The internal structure of the generative prompt module 140 is as follows: Figure 5As shown: the observed modality sequence representation (i.e., the observed modality representation) is linearly projected and used as the key K and value V, the target modality query library is used as the query Q, and after multi-head cross-attention and layer normalization, the missing modality sequence representation is output.

[0071] The internal structure of the time reminder enhancement module 180 is as follows: Figure 6 As shown: the corresponding time-series cue is selected from the time-series cue library according to the sample missing pattern as query Q, and the representation of the cross-modal fusion sequence at the time grid position is used as key K and value V. The missing pattern-related time-series aggregation vector is obtained through multi-head attention. This time-series aggregation vector is sent to the prediction output module as a patient-level prediction feature.

[0072] Numerical temporal coding module 110: receives raw numerical observations, numerical observation masks and observation timestamps, obtains irregular branch representations through temporal embedding and multi-temporal attention networks, and can obtain regular branch representations by one-dimensional convolution of regularized sequences. Then, the two branches are mixed by a gating network according to learnable weights to output a numerical mode sequence representation.

[0073] Text encoding module 120: Receives clinical text token, text timestamp and writing timestamp, encodes it line by line by line by medical text encoder, and then aligns it to a 48-hour shared time grid based on writing time through multi-time attention.

[0074] Missing Mode Judgment Module 130: Automatically determines the sample-level missing type based on text time mask and numerical observation mask, providing conditions for the selection of subsequent generative hints, modal signal hints, missing type hints and timing hints.

[0075] Generative hint module 140: includes a number-to-text generator and a text-to-number generator, uses a learnable query library initialized with the mean of the target modality in the training set, generates missing modality representations from the observed modality through multi-head cross-attention, and restores the missing side branches to sequence representations compatible with downstream.

[0076] Modal signal cueing module 150: Uses four learnable cues to label observed text, generated text, observed values, and generated values ​​respectively, so that the fusion network can distinguish the source of features.

[0077] Missing type prompt module 160: Sets different prompt token sequences for missing values, missing text, and complete data, concatenates them into the cross-modal interaction sequence before deep memory encoding, and shares them between the two modal branches.

[0078] Cross-modal fusion and memory encoding module 170: First, shallow cross-modal interaction is performed through bidirectional cross-attention, then the missing perception representation of each modality is refined through deep memory encoder, and finally the cross-modal fusion sequence is formed by splicing in the feature dimension.

[0079] Time-based cue enhancement module 180: Maintains a set of learnable temporal cues selected by missing patterns as queries, performs missing pattern-aware temporal aggregation on the representation of the fused sequence at the time grid position, and sends the resulting aggregated vector as a patient-level prediction feature to the prediction output module.

[0080] Predictive output module 190: Outputs 48-IHM binary classification of inpatient mortality risk probability via residual multilayer perceptron and classification layer.

[0081] Furthermore, based on the aforementioned method and system for predicting mortality risk for incomplete multimodal electronic health records, this invention also provides a terminal, wherein a preferred embodiment of the terminal is as follows: Figure 7 As shown, it specifically includes a processor 10, a memory 20, and a display 30. Figure 7 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0082] In some embodiments, memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, or flash card. Furthermore, memory 20 may include both internal and external storage units. Memory 20 is used to store application software and various types of data installed on the terminal, such as the terminal's program code. Memory 20 may also be used to temporarily store data that has been output or will be output. In one embodiment, memory 20 stores a mortality risk prediction program 40 for incomplete multimodal electronic health records. This mortality risk prediction program 40 for incomplete multimodal electronic health records can be executed by processor 10 to implement the steps of the mortality risk prediction method for incomplete multimodal electronic health records in this application.

[0083] In some embodiments, processor 10 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 20 or process data, such as executing a mortality risk prediction program 40 for incomplete multimodal electronic health records.

[0084] In some embodiments, display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. Display 30 is used to display information on the terminal and to display a visual user interface.

[0085] In one embodiment, when the processor 10 executes the mortality risk prediction program 40 for incomplete multimodal electronic health records stored in the memory 20, the steps of the mortality risk prediction method for incomplete multimodal electronic health records as described above are implemented.

[0086] The present invention also provides a computer-readable storage medium storing a mortality risk prediction program for incomplete multimodal electronic health records, wherein the mortality risk prediction program for incomplete multimodal electronic health records, when executed by a processor, implements the steps of the mortality risk prediction method for incomplete multimodal electronic health records as described above.

[0087] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0088] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0089] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for predicting mortality risk based on incomplete multimodal electronic health records, characterized in that, The method for predicting mortality risk based on incomplete multimodal electronic health records includes: Obtain the patient's multimodal electronic health record and the corresponding mask of the multimodal electronic health record, and determine the missing type of the multimodal electronic health record based on the mask; Based on the missing type, the observation modality sequence in the multimodal electronic health record is encoded and time-aligned to obtain an initial observation modality sequence characterization; The initial generative missing mode sequence representation is obtained by extracting sequence information related to the missing mode from the initial observed mode sequence representation through cross attention. Modal signal cues are added to the initial observed modal sequence representation and the initial generative missing modal sequence representation, respectively, to obtain the observed modal sequence representation and the generative missing modal sequence representation; The observed modal sequence representation and the generative missing modal sequence representation are bidirectionally cross-modal fused by cross-modal attention to obtain a cross-modal interactive representation. The cross-modal interactive representation with missing type hints is then memorized and fused to obtain a cross-modal fused sequence. Based on the temporal cue corresponding to the missing type, the cross-modal fusion sequence is temporally aggregated using multi-head attention to obtain a temporal aggregation vector; By using a residual multilayer perceptron and a classification layer, the temporal aggregation vector is nonlinearly transformed and mapped to obtain the predicted probability of the patient's mortality risk. The process of acquiring the patient's multimodal electronic health record and the corresponding mask, and determining the missing type of the multimodal electronic health record based on the mask, specifically includes: The system acquires the patient's multimodal electronic health record within a set time period before admission to the department and the corresponding mask for the multimodal electronic health record. The multimodal electronic health record includes a numerical sequence and / or clinical text, and the mask includes a numerical observation mask corresponding to the numerical sequence and a text time mask corresponding to the clinical text. Based on the numerical observation mask and the text time mask, determine the missing type of the multimodal electronic health record; If the numerical observation mask indicates the existence of valid numerical observations, and the text time mask indicates the absence of valid clinical text records, then the missing type of the multimodal electronic health record is determined to be missing text modality. If the numerical observation mask indicates that there are no valid numerical observations, and the text time mask indicates that there are valid clinical text records, then the missing type of the multimodal electronic health record is determined to be missing numerical modality. If the numerical observation mask indicates the existence of valid numerical observations and the text time mask indicates the existence of valid clinical text records, then the missing type of the multimodal electronic health record is determined to be complete data. The initial generative missing modality sequence representation includes an initial generative text modality sequence representation or an initial generative numerical modality sequence representation; The step of extracting sequence information related to the missing modes from the initial observed mode sequence representation through cross-attention to obtain the initial generative missing mode sequence representation specifically includes: When the missing type is a missing text modality, the initial observed modality sequence representation is an initial numerical modality sequence representation. The text target query library is used as the query, and the output of the initial numerical modality sequence representation after linear projection is used as the key and value. Sequence information related to the target text modality is extracted from the initial numerical modality sequence representation through cross attention to obtain the initial generative text modality sequence representation. When the missing type is a missing numerical mode, the initial observed mode sequence representation is an initial text mode sequence representation. The numerical target query library is used as the query, and the output of the initial text mode sequence representation after linear projection is used as the key and value. Sequence information related to the target numerical mode is extracted from the initial text mode sequence representation through cross attention to obtain the initial generative numerical mode sequence representation. When the missing type is complete data, the initial observation modality sequence representation includes the initial numerical modality sequence representation and the initial text modality sequence representation.

2. The method for predicting mortality risk based on incomplete multimodal electronic health records according to claim 1, characterized in that, The initial observation modality sequence representation includes the initial numerical modality sequence representation and / or the initial text modality sequence representation. The numerical sequence includes numerical observations and observation timestamps, and the clinical text includes clinical text records and writing timestamps. The step of encoding and time-aligning the observation modality sequences in the multimodal electronic health record according to the missing type to obtain an initial observation modality sequence characterization specifically includes: When the missing type is a missing text modality, the observation modality sequence in the multimodal electronic health record is a numerical sequence, which includes irregular numerical sequences and regular numerical sequences; For the irregular numerical sequence, the numerical observations in the irregular numerical sequence are concatenated with the numerical observation mask corresponding to the irregular numerical sequence to obtain the concatenated numerical observations and numerical observation mask. The concatenated numerical observations and numerical observation mask are then aligned to a shared time grid through temporal embedding and a multi-temporal attention network to obtain the irregular branch representation. For the rule-based numerical sequence, a one-dimensional convolution is performed on the rule-based numerical sequence to obtain the rule-based branch representation; Learnable weights are assigned to the irregular branch representation and the regular branch representation respectively through a gating network. The irregular branch representation and the regular branch representation are then weighted and fused according to the learnable weights to obtain the initial numerical mode sequence representation. When the missing type is a missing numerical modality, the observation modality sequence in the multimodal electronic health record is clinical text; A medical text encoder is used to extract the classification label vector of the clinical text record in the clinical text; Based on the writing timestamp in the clinical text and the text time mask corresponding to the clinical text, the classification label vector is aligned to the shared time grid through multi-temporal attention to obtain the initial text modality sequence representation; When the missing type is complete data, the observation modality sequence in the multimodal electronic health record is a numerical sequence and clinical text; The numerical sequence and the clinical text are encoded and time-aligned respectively to obtain the initial numerical modality sequence representation and the initial text modality sequence representation.

3. The method for predicting mortality risk based on incomplete multimodal electronic health records according to claim 2, characterized in that, The modal signal cues include text observation cues, text generation cues, numerical observation cues, and numerical generation cues; the observation modal sequence representations include numerical modal sequence representations and / or text modal sequence representations; and the generative missing modal sequence representations include generative text modal sequence representations or generative numerical modal sequence representations. The step of adding modal signal cues to the initial observed modal sequence representation and the initial generative missing modal sequence representation to obtain the observed modal sequence representation and the generative missing modal sequence representation, specifically includes: When the missing type is a missing text modality, an additive injection method is used to add the numerical observation cue to the initial numerical modality sequence representation to obtain the numerical modality sequence representation, and to add the text generation cue to the initial generative text modality sequence representation to obtain the generative text modality sequence representation. When the missing type is a missing numerical mode, an additive injection method is used to add the text observation cue to the initial text mode sequence representation to obtain the text mode sequence representation, and to add the numerical generation cue to the initial generative numerical mode sequence representation to obtain the generative numerical mode sequence representation. When the missing type is complete data, an additive injection method is used to add the numerical observation cue to the initial numerical modality sequence representation to obtain the numerical modality sequence representation, and to add the text observation cue to the initial text modality sequence representation to obtain the text modality sequence representation.

4. The method for predicting mortality risk based on incomplete multimodal electronic health records according to claim 3, characterized in that, The cross-modal interaction representation includes a first cross-modal interaction representation, a second cross-modal interaction representation, a third cross-modal interaction representation, a fourth cross-modal interaction representation, a fifth cross-modal interaction representation, and a sixth cross-modal interaction representation; the missing type prompt includes missing text prompt, missing numerical value prompt, and complete data prompt; the cross-modal fusion sequence includes a first cross-modal fusion sequence, a second cross-modal fusion sequence, and a third cross-modal fusion sequence. The process involves bidirectional cross-modal fusion of the observed modal sequence representation and the generative missing modal sequence representation through cross-modal attention to obtain a cross-modal interactive representation. Furthermore, the cross-modal interactive representation, after adding missing type hints, undergoes memory encoding and fusion to obtain a cross-modal fused sequence. Specifically, this includes: When the missing type is missing text modality, the generative text modality sequence representation is used as the query, and the numerical modality sequence representation is used as the key and value. The first cross-modal interaction representation of the text after being constrained by numerical conditions is calculated through cross-modal attention. The second cross-modal interaction representation of the numerical value after being constrained by text conditions is calculated using the numerical modality sequence representation as the query and the generative text modality sequence representation as the key and value. The missing text prompts corresponding to the missing text modalities are concatenated to the end of the first cross-modal interaction representation and the second cross-modal interaction representation, respectively; The first cross-modal interaction representation after splicing the missing text prompt is input into the text memory encoder. The text memory encoder performs memory encoding on the first cross-modal interaction representation after splicing the missing text prompt through self-attention and feedforward network to obtain the first text memory representation. The second cross-modal interaction representation after splicing the missing text prompt is input into the numerical memory encoder. The numerical memory encoder performs memory encoding on the second cross-modal interaction representation after splicing the missing text prompt through self-attention and feedforward network to obtain the first numerical memory representation. The first text memory representation and the first numerical memory representation are concatenated along the feature dimension to obtain the first cross-modal fusion sequence; When the missing type is a missing numerical modality, the text modality sequence representation is used as the query, the generative numerical modality sequence representation is used as the key and value, and the third cross-modal interaction representation after the text is constrained by numerical conditions is calculated through cross-modal attention. The fourth cross-modal interaction representation after the numerical value is constrained by text conditions is calculated using the generative numerical modality sequence representation as the query and the text modality sequence representation as the key and value. The missing value prompts corresponding to the missing value modalities are concatenated to the end of the third cross-modal interaction representation and the fourth cross-modal interaction representation, respectively; The third cross-modal interaction representation after splicing the missing numerical prompt is input into the text memory encoder. The text memory encoder performs memory encoding on the third cross-modal interaction representation after splicing the missing numerical prompt through self-attention and feedforward network to obtain the second text memory representation. The fourth cross-modal interaction representation after splicing missing numerical prompts is input into the numerical memory encoder. The numerical memory encoder performs memory encoding on the fourth cross-modal interaction representation after splicing missing numerical prompts through self-attention and feedforward network to obtain the second numerical memory representation. The second text memory representation and the second numerical memory representation are concatenated along the feature dimension to obtain the second cross-modal fusion sequence; When the missing type is complete data, the text modal sequence representation is used as the query and the numerical modal sequence representation is used as the key and value. The fifth cross-modal interaction representation after the text is constrained by numerical conditions is calculated through cross-modal attention. The sixth cross-modal interaction representation after the numerical value is constrained by text conditions is calculated using the numerical modal sequence representation as the query and the text modal sequence representation as the key and value. The complete data prompts corresponding to the complete data are concatenated to the end of the fifth cross-modal interaction representation and the sixth cross-modal interaction representation, respectively; The fifth cross-modal interaction representation after piecing together complete data prompts is input into a text memory encoder. The text memory encoder uses self-attention and a feedforward network to memorize and encode the fifth cross-modal interaction representation after piecing together complete data prompts, resulting in a third text memory representation. The sixth cross-modal interaction representation after splicing complete data prompts is input into the numerical memory encoder. The numerical memory encoder performs memory encoding on the sixth cross-modal interaction representation after splicing complete data prompts through self-attention and feedforward network to obtain the third numerical memory representation. The third text memory representation and the third numerical memory representation are concatenated along the feature dimension to obtain the third cross-modal fusion sequence.

5. The method for predicting mortality risk based on incomplete multimodal electronic health records according to claim 4, characterized in that, The timing prompts include timing prompts for missing text modalities, timing prompts for missing numerical modalities, and timing prompts for complete data. The timing aggregation vectors include a first timing aggregation vector, a second timing aggregation vector, and a third timing aggregation vector. The step of performing temporal aggregation on the cross-modal fusion sequence using multi-head attention based on the temporal cue corresponding to the missing type to obtain a temporal aggregation vector specifically includes: When the missing type is a missing text modality, the missing text modality time sequence hint corresponding to the missing text modality is used as the query, and the representation of the first cross-modal fusion sequence on the shared time grid is used as the key and value. The first attention weight is obtained by multi-head attention calculation. Based on the first attention weight, the representation of the first cross-modal fusion sequence on the shared time grid is weighted and aggregated to obtain the first temporal aggregation vector; When the missing type is a missing numerical mode, the missing numerical mode time sequence hint corresponding to the missing numerical mode is used as the query, and the representation of the second cross-modal fusion sequence on the shared time grid is used as the key and value. The second attention weight is obtained by multi-head attention calculation. Based on the second attention weight, the representation of the second cross-modal fusion sequence on the shared time grid is weighted and aggregated to obtain the second temporal aggregation vector; When the missing type is complete data, the complete data time series prompt corresponding to the complete data is used as the query, and the representation of the third cross-modal fusion sequence on the shared time grid is used as the key and value. The third attention weight is obtained by multi-head attention calculation. Based on the third attention weight, the representation of the third cross-modal fusion sequence on the shared time grid is weighted and aggregated to obtain the third temporal aggregation vector.

6. A mortality risk prediction system for incomplete multimodal electronic health records, wherein the mortality risk prediction system for incomplete multimodal electronic health records is used to implement the mortality risk prediction method for incomplete multimodal electronic health records as described in any one of claims 1-5, characterized in that, The mortality risk prediction system for incomplete multimodal electronic health records includes: The missing type determination module is used to obtain the patient's multimodal electronic health record and the mask corresponding to the multimodal electronic health record, and determine the missing type of the multimodal electronic health record based on the mask; The encoding and alignment module is used to encode and time-align the observation modality sequence in the multimodal electronic health record according to the missing type, so as to obtain an initial observation modality sequence characterization; A completion module is used to extract sequence information related to missing modes from the initial observed mode sequence representation through cross attention, so as to obtain an initial generative missing mode sequence representation. The modal signal cueing module is used to add modal signal cueing to the initial observed modal sequence representation and the initial generative missing modal sequence representation respectively, so as to obtain the observed modal sequence representation and the generative missing modal sequence representation; The cross-modal fusion and memory encoding module is used to perform bidirectional cross-modal cross-fusion of the observed modal sequence representation and the generative missing modal sequence representation through cross-modal attention to obtain cross-modal interactive representation, and to perform memory encoding and fusion of the cross-modal interactive representation after adding missing type hints to obtain cross-modal fusion sequence; The temporal aggregation module is used to perform temporal aggregation on the cross-modal fusion sequence through multi-head attention based on the temporal cue corresponding to the missing type, so as to obtain a temporal aggregation vector. The mortality risk prediction module is used to perform nonlinear transformation and mapping on the time-series aggregated vector through a residual multilayer perceptron and a classification layer to obtain the predicted mortality risk probability of the patient.

7. A terminal, characterized in that, The terminal includes: a memory, a processor, and a mortality risk prediction program for incomplete multimodal electronic health records stored in the memory and executable on the processor. When the mortality risk prediction program for incomplete multimodal electronic health records is executed by the processor, it implements the steps of the mortality risk prediction method for incomplete multimodal electronic health records as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a mortality risk prediction program for incomplete multimodal electronic health records, which, when executed by a processor, implements the steps of the mortality risk prediction method for incomplete multimodal electronic health records as described in any one of claims 1-5.

Images