A delirium intelligent identification system based on multi-modal features and explainability analysis
The intelligent identification system, which utilizes multimodal features and interpretability analysis, automatically identifies delirium states, solving the inconsistencies in delirium assessment and data acquisition challenges in existing technologies, and achieving accurate differentiation and real-time monitoring of delirium subtypes.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI CHILDRENS MEDICAL CENT AFFILIATED TO SHANGHAI JIAOTONG UNIV SCHOOL OF MEDICINE
- Filing Date
- 2026-03-11
- Publication Date
- 2026-06-05
AI Technical Summary
Current technologies rely on manual observation to assess delirium, which lacks repeatability and consistency, making it difficult to achieve continuous and real-time monitoring. Furthermore, AI-based methods face high data acquisition barriers in real-world environments, limiting their universality.
An intelligent recognition system based on multimodal features and interpretability analysis is adopted. Multimodal data is acquired through a video acquisition module, spatiotemporal behavioral features are automatically extracted using a deep learning model, multiple binary classification recognition models are constructed, and a voting mechanism is used for decision fusion to achieve automated recognition of delirium.
It achieves accurate differentiation of delirium subtypes, significantly improves the identification rate of suppressed delirium, provides continuous and non-intrusive real-time monitoring capabilities, is highly adaptable, and alleviates the problem of data scarcity.
Smart Images
Figure CN122156768A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent diagnostic technology, and more specifically, to an intelligent delirium recognition system based on multimodal features and interpretability analysis. Background Technology
[0002] In applications such as continuous monitoring and intensive care, monitored subjects may exhibit an acute, fluctuating disturbance of consciousness, accompanied by cognitive and perceptual impairments, clinically defined as delirium. The development of delirium is often dispersed across long-term video sequences, making it difficult to consistently capture its patterns in a single or short observation. More seriously, delirium can lead to prolonged hospital stays and increased all-cause mortality. Early identification of delirium throughout hospitalization is crucial; timely detection facilitates early intervention, ensures high-quality clinical care, and helps improve treatment outcomes.
[0003] Current technologies for assessing delirium primarily rely on manual observation or periodic evaluation, which suffers from the following shortcomings: First, the assessment process depends on human experience, making it difficult to generate repeatable and quantifiable objective parameters; second, manual assessments are usually based on limited time segments, making it difficult to reflect the temporal characteristics of behavioral changes; and third, the consistency among different observers is low, making it difficult to meet the engineering requirements of continuous, real-time monitoring. While artificial intelligence-based solutions exist, their models typically rely on single, specific data sources (such as structured electronic medical records or specific EEG signals), which present challenges in real-world clinical settings due to high data acquisition barriers and limited universality. Summary of the Invention
[0004] This invention provides a delirium intelligent identification system based on multimodal features and interpretability analysis to solve the problems of existing artificial intelligence identification systems lacking accurate differentiation of clinical subtypes in delirium assessment and having a low identification rate for suppressive delirium.
[0005] To achieve the above objectives, the present invention provides the following technical solution:
[0006] A delirium intelligent identification system based on multimodal features and interpretability analysis includes a video acquisition module, a data storage module, a data preprocessing and inference module, and a clinical interaction module. The video acquisition module is used to acquire continuous video data of the monitored subject in a clinical environment. The data includes multimodal features such as the monitored subject's facial and upper body behavior to meet the needs of subsequent time-series analysis.
[0007] Through the collaborative work of the above modules, an automated processing flow is achieved from video acquisition to delirium recognition result output, thereby reducing the subjectivity and instability caused by manual observation.
[0008] Preferably, the data storage module is used to store the collected video data and its corresponding analysis results, and supports management by case or time dimension.
[0009] Preferably, the data preprocessing and inference module is used to perform frame segmentation, cropping, time segment construction and feature standardization on the original video data, and automatically extract the spatiotemporal behavioral features of the video based on a deep learning model to achieve intelligent analysis of delirium.
[0010] Preferably, the clinical interaction module is used to provide medical staff with a visual display of recognition results, including delirium occurrence status and type prediction results, and supports access on clinical terminals.
[0011] Preferably, the intelligent analysis step of delirium state specifically includes:
[0012] The process includes video data acquisition, video temporal modeling, feature extraction and modeling, and delirium detection. The video data acquisition includes continuous video data of the behavior of the monitored object. During the acquisition process, the video data should be free from obstruction and severe interference.
[0013] Preferably, the intelligent analysis step of delirium state specifically includes:
[0014] Video temporal modeling: The video data is divided into multiple video segments according to a preset time window, and each video segment is processed to a uniform length to construct standardized video input samples.
[0015] Preferably, the intelligent analysis step of delirium state specifically includes:
[0016] Feature extraction and modeling: Video segments are modeled using a spatiotemporal convolutional neural network to automatically learn changes in the behavior, motion patterns, and attention features of the monitored objects, thereby extracting high-level semantic features related to the occurrence of delirium.
[0017] Preferably, the intelligent analysis step of delirium state specifically includes:
[0018] Delirium occurrence determination: Based on the aforementioned features, a binary classification recognition model is constructed to determine the input video and output the recognition result indicating whether the monitored object is at risk of delirium.
[0019] Preferably, the method for determining the occurrence of delirium includes:
[0020] Multimodal feature construction: Based on the spatiotemporal features of the video, auxiliary behavior-related features are introduced, including emotion change features and eye movement behavior features, to construct a multimodal feature representation;
[0021] Binary sub-model construction: To address the differences between different delirium types, multiple binary classification identification models are constructed to distinguish between different delirium subtypes and the normal state.
[0022] Preferably, the method for determining the occurrence of delirium further includes:
[0023] Decision fusion mechanism: The voting device is used to fuse the output results of multiple binary classification models, and a comprehensive judgment is made based on the prediction results of each model to finally output the delirium type of the monitored object;
[0024] Classification results output: The output includes fine-grained classification results including agitated delirium, suppressed delirium, and non-delirious states.
[0025] The principle and beneficial effects of this technical solution:
[0026] 1. Achieved accurate differentiation of clinical subtypes: Through a multi-model integrated voting mechanism, it effectively distinguished between agitated and inhibitory subtypes of delirium in children for the first time in automated identification, providing key decision support for targeted clinical intervention and filling the gap in existing technology.
[0027] 2. Significantly improves the recognition rate of suppressed delirium: By explicitly modeling eye movement temporal features and micro-expression gating mechanisms, the system can capture the subtle core features of suppressed subjects, such as inattention, eye movement lag, and emotional apathy, greatly reducing the risk of misjudgment and missed diagnosis of this easily missed subtype.
[0028] 3. More comprehensive feature representation and stronger discriminative power: By integrating multimodal information from video (macro-behavior), eye movement (cognitive attention) and micro-expression (emotional fluctuation), a high-dimensional and complementary deep spatiotemporal feature is constructed, which overcomes the defect of single modality being insensitive to subtle anomalies. The overall discriminative power and robustness of the model are significantly enhanced.
[0029] 4. Good clinical applicability: This invention adopts a non-contact video analysis method, which is easy to deploy in wards such as PICU, enabling continuous and non-disruptive real-time monitoring. The integrated voting framework has a certain tolerance for errors in a single model and can handle clinical video clips of different lengths, making it highly practical.
[0030] 5. Provides an effective example for few-shot learning: Through the design of a multi-task binary classifier, pre-trained transfer learning, and intermodal information complementarity, it effectively utilizes limited labeled clinical data, alleviating the common contradiction between data scarcity and model overfitting in the field of medical AI. Attached Figure Description
[0031] Figure 1 This is a block diagram of the delirium recognition system in this invention;
[0032] Figure 2This is a flowchart of delirium identification in this invention;
[0033] Figure 3 This is a schematic diagram of the emotion-video fusion network in this invention;
[0034] Figure 4 This is a schematic diagram of the eye-tracking-video fusion network in this invention;
[0035] Figure 5 This is a schematic diagram of the voting device's workflow in this invention; Detailed Implementation
[0036] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments:
[0037] like Figure 1 As shown, a delirium intelligent identification system based on multimodal features and interpretability analysis includes: a video acquisition module, a data storage module, a data preprocessing and inference module, and a clinical interaction module. The video acquisition module is used to acquire continuous video data of the monitored object in a clinical environment. The video data includes facial and upper body behavioral information of the monitored object to meet the needs of subsequent time-series analysis.
[0038] Through the collaborative work of the above modules, an automated processing flow is achieved from video acquisition to delirium recognition result output, thereby reducing the subjectivity and instability caused by manual observation.
[0039] like Figure 1 As shown, the data storage module is used to store the collected video data and its corresponding analysis results, and supports management by case or time dimension.
[0040] like Figure 1 As shown, the data preprocessing and inference module is used to perform frame segmentation, cropping, time segment construction and feature standardization on the original video data, and automatically extracts the spatiotemporal behavioral features of the video based on a deep learning model to achieve intelligent analysis of delirium.
[0041] like Figure 1 As shown, the clinical interaction module is used to provide medical staff with a visual display of recognition results, including the delirium occurrence status and type prediction results, and also supports access on clinical terminals.
[0042] like Figure 2 As shown, the intelligent analysis steps for delirium state specifically include:
[0043] The process includes video data acquisition, video temporal modeling, feature extraction and modeling, and delirium detection. The video data acquisition includes continuous video data of the behavior of the monitored object. During the acquisition process, the video data should be free from obstruction and severe interference.
[0044] like Figure 2 As shown, the intelligent analysis steps for delirium state specifically include:
[0045] Video temporal modeling: The video data is divided into multiple video segments according to a preset time window, and each video segment is processed to a uniform length to construct standardized video input samples.
[0046] like Figure 2 As shown, the intelligent analysis steps for delirium state specifically include:
[0047] Feature extraction and modeling: Video segments are modeled using a spatiotemporal convolutional neural network to automatically learn changes in the behavior, motion patterns, and attention features of the monitored objects, thereby extracting high-level semantic features related to the occurrence of delirium.
[0048] like Figure 2 As shown, the intelligent analysis steps for delirium state specifically include:
[0049] Delirium occurrence determination: Based on the aforementioned features, a binary classification recognition model is constructed to determine the input video and output the recognition result indicating whether the monitored object is at risk of delirium.
[0050] The evaluation metrics on the standard test set for judging the occurrence of delirium are as follows:
[0051]
[0052] like Figures 3 to 5 As shown, the method for determining the occurrence of delirium includes:
[0053] Multimodal feature construction: Based on the spatiotemporal features of the video, auxiliary behavior-related features are introduced, including emotion change features and eye movement behavior features, to construct a multimodal feature representation;
[0054] Binary sub-model construction: To address the differences between different delirium types, multiple binary classification identification models are constructed to distinguish between different delirium subtypes and the normal state.
[0055] like Figures 3 to 5 As shown, the method for determining the occurrence of delirium further includes:
[0056] Decision fusion mechanism: The voting device is used to fuse the output results of multiple binary classification models, and a comprehensive judgment is made based on the prediction results of each model to finally output the delirium type of the monitored object;
[0057] Classification results output: The output includes fine-grained classification results including agitated delirium, suppressed delirium, and non-delirious states.
[0058] When assessing the three states of delirium—agitation, inhibition, and non-delirium—the evaluation metrics on the standard test set are as follows:
[0059]
[0060] Example:
[0061] (1) Alternatives for multimodal input and feature extraction layers
[0062] 1. Alternatives to micro-expression or emotion feature modeling methods
[0063] In this embodiment, DeepFace is used to extract the proportion of emotion categories as micro-expression features;
[0064] Use other facial expression recognition or emotion computing models, such as FER+, AffectNet, EmotionNet, or expression recognition models based on the Transformer architecture;
[0065] Micro-expression modeling is performed directly based on facial muscle action unit sequences, and emotional states are characterized by the activation intensity, duration, and change patterns of the AU.
[0066] Use temporal emotion embedding vectors to replace global emotion proportion features, or further introduce higher-order dynamic indicators such as emotion change rate, emotion fluctuation amplitude, and emotion stability.
[0067] All of the above schemes can serve as equivalent expressions of emotions or micro-expression information, and can be used to modulate or enhance the ability to discriminate video spatiotemporal features.
[0068] 2. Alternatives to eye-tracking feature acquisition methods
[0069] High-precision eye movement data can be directly collected using an infrared eye tracker, wearable eye tracking device, or bedside fixed eye camera.
[0070] Use other facial landmark detection or gaze estimation algorithms, such as OpenFace, Gaze360, RT-GENE, etc., to extract eye movement features such as gaze direction, blinking behavior, and pupil movement trajectory;
[0071] The original frame-by-frame eye movement time-series data can be further processed into statistical features, such as blink frequency, fixation stability, saccade velocity distribution, and fixation point dispersion, and then used as model input.
[0072] The above-mentioned alternatives differ in their implementation paths and data sources, but they can all characterize children's attentional state, arousal level and its changing features, and are equivalent substitutes for the technical ideas of this invention.
[0073] 3. Replacement of video backbone network structure
[0074] Three-dimensional convolutional neural network structures, such as C3D, I3D, SlowFast, etc.
[0075] Hybrid architectures based on temporal modeling, such as convolutional neural networks with long short-term memory networks, or convolutional neural networks with temporal convolutional networks;
[0076] Pure Transformer or video Transformer architectures based on attention mechanisms, such as TimeSformer and ViViT.
[0077] Any method that can effectively model the spatiotemporal sequence of a video and output discriminative features for delirium identification can be considered an equivalent alternative implementation of the present invention.
[0078] (2) Alternatives to multimodal fusion mechanisms
[0079] 1. Substitution in the integration phase
[0080] In this embodiment, multimodal information is primarily fused at the feature layer. Other alternative approaches include:
[0081] Fusion is carried out at the decision-making level, that is, after each modality outputs its prediction results, a comprehensive decision is made through weighted rules or logical rules;
[0082] Fusion is performed in the middle layer of the network, that is, eye-tracking or emotion features are introduced into the middle temporal layer of the video backbone network to interact with video features;
[0083] Use cross-modal attention mechanisms or bidirectional cross-attention mechanisms to replace simple feature splicing or gating operations.
[0084] 2. Replacement of gating and modulation mechanisms
[0085] The emotion gating module in this embodiment can be replaced in the following ways:
[0086] Channel attention or temporal attention modules are used to weight video features in the channel or time dimension;
[0087] A dynamic weight generation module based on self-attention or conditional attention is used to dynamically adjust the importance of video features according to the auxiliary modality;
[0088] Introduce learnable modal weight coefficients, or an adaptive weighting strategy based on prediction uncertainty estimation.
[0089] The above-mentioned alternative methods are all used to dynamically modulate video feature streams according to auxiliary modalities, and are equivalent technical solutions of the present invention.
[0090] (3) Alternatives to classification strategies and decision-making logic
[0091] 1. In this embodiment, a rule-based voting matrix is used to complete the three-class classification decision. The following alternatives can be used:
[0092] The weighted voting strategy dynamically allocates voting weights based on the performance or prediction uncertainty of each sub-model on the validation set.
[0093] The hierarchical classification strategy first determines whether delirium exists, and then further distinguishes between agitated and inhibited delirium samples.
[0094] Bayesian fusion methods based on probability distributions, or multi-model decision fusion methods based on evidence theory (such as Dempster-Shafer theory).
[0095] 2. Replacement of model organizational form
[0096] The combination of multiple binary classifiers can also be replaced by the following structure:
[0097] A multi-task learning model with a shared backbone network and multiple task output heads is adopted.
[0098] An end-to-end three-class classification model is adopted, combined with a class imbalance loss function or a cost-sensitive learning strategy;
[0099] A confidence or uncertainty assessment module is introduced at the model output. When the uncertainty of the prediction result is high, a prompt message indicating that manual review is required is output.
[0100] 3. Alternatives to training strategies and data processing workflows
[0101] 3.1 Data Augmentation and Alternative Sample Construction Methods
[0102] The sliding window video slicing strategy in this embodiment can be replaced by the following:
[0103] An adaptive temporal sampling strategy based on keyframe selection or the intensity of motion changes;
[0104] Parallel modeling using multi-scale time windows is employed to simultaneously capture short-term and long-term behavioral features;
[0105] Pre-training can be performed using weakly supervised or self-supervised learning methods to reduce reliance on precise manual annotation.
[0106] 3.2 Replacement of Transfer Learning and Training Process
[0107] In addition to using publicly available video datasets for pre-training, the following methods can also be used:
[0108] Self-supervised temporal pre-training on unlabeled or weakly labeled clinical video data;
[0109] Federated learning or multi-center collaborative training methods are used to improve the model's generalization ability in different hospitals and different populations;
[0110] The model is continuously learned or fine-tuned online based on different hospital environments or population characteristics.
[0111] The above descriptions are merely embodiments of the present invention, and common knowledge such as specific technical solutions and / or characteristics are not described in detail here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the technical solutions of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.
Claims
1. A delirium intelligent recognition system based on multimodal features and interpretability analysis, characterized in that, include: The system includes a video acquisition module, a data storage module, a data preprocessing and inference module, and a clinical interaction module. The video acquisition module is used to acquire continuous video data of the monitored subject in a clinical environment. The data includes multimodal features such as the monitored subject's facial and upper body behavior to meet the needs of subsequent time-series analysis. Through the collaborative work of the above modules, an automated processing flow is achieved from video acquisition to delirium recognition result output, thereby reducing the subjectivity and instability caused by manual observation.
2. The delirium intelligent recognition system based on multimodal features and interpretability analysis according to claim 1, characterized in that: The data storage module is used to store the collected video data and its corresponding analysis results, and supports management by case or time dimension.
3. The delirium intelligent recognition system based on multimodal features and interpretability analysis according to claim 2, characterized in that: The data preprocessing and inference module is used to perform frame segmentation, cropping, time segment construction, and feature standardization on the original video data, obtain eye-tracking features, and automatically extract the spatiotemporal behavioral features of the video based on a deep learning model to achieve intelligent analysis of delirium.
4. The delirium intelligent recognition system based on multimodal features and interpretability analysis according to claim 3, characterized in that: The clinical interaction module is used to provide medical staff with a visual display of recognition results, including the delirium occurrence status and type prediction results.
5. The delirium intelligent recognition system based on multimodal features and interpretability analysis according to claim 3, characterized in that, The intelligent analysis steps for delirium state specifically include: The process includes video data acquisition, video temporal modeling, feature extraction and modeling, and delirium detection. The video data acquisition includes continuous video data of the behavior of the monitored object. During the acquisition process, the video data should be free from obstruction and severe interference.
6. The delirium intelligent recognition system based on multimodal features and interpretability analysis according to claim 5, characterized in that, The intelligent analysis steps for delirium state specifically include: Video temporal modeling: The video data is divided into multiple video segments according to a preset time window, and each video segment is processed to a uniform length to construct standardized video input samples.
7. The delirium intelligent recognition system based on multimodal features and interpretability analysis according to claim 5, characterized in that, The intelligent analysis steps for delirium state specifically include: Feature extraction and modeling: Video segments are modeled using a spatiotemporal convolutional neural network to automatically learn changes in the behavior, motion patterns, and attention features of the monitored objects, thereby extracting high-level semantic features related to the occurrence of delirium.
8. The delirium intelligent recognition system based on multimodal features and interpretability analysis according to claim 5, characterized in that, The intelligent analysis steps for delirium state specifically include: Delirium occurrence determination: Based on the aforementioned features, a binary classification recognition model is constructed to determine the input video and output the recognition result indicating whether the monitored object is at risk of delirium.
9. A delirium intelligent recognition system based on multimodal features and interpretability analysis according to claim 8, characterized in that, The method for determining the occurrence of delirium includes: Multimodal feature construction: Based on the spatiotemporal features of the video, auxiliary behavior-related features are introduced, including emotion change features and eye movement behavior features, to construct a multimodal feature representation; Binary sub-model construction: To address the differences between different delirium types, multiple binary classification identification models are constructed to distinguish between different delirium subtypes and the normal state.
10. A delirium intelligent recognition system based on multimodal features and interpretability analysis according to claim 9, characterized in that, The method for determining the occurrence of delirium also includes: Decision fusion mechanism: The voting device is used to fuse the output results of multiple binary classification models, and a comprehensive judgment is made based on the prediction results of each model to finally output the delirium type of the monitored object; Classification results output: The output includes fine-grained classification results including agitated delirium, suppressed delirium, and non-delirious states.