Early warning dataset construction method and system for embodied intelligent nursing scene and computer readable storage medium

Abstract

The application discloses a method and system for constructing an early warning data set for embodied intelligent nursing scene and a computer readable storage medium. The method comprises the following steps: acquiring multi-modal perception data of a target object, calculating the cumulative deviation degree based on the personalized multi-modal baseline, intercepting the associated multi-modal data segment when the abnormal triggering condition is met, performing causal analysis, generating an initial causal subgraph of each mode and performing conflict detection, triggering manual review when a conflict is detected, forming a multi-modal causal graph with causal annotation and confidence annotation according to the modification information of the initial causal subgraph and the confirmation information of the causal edge confidence, performing intervention simulation with the causal graph as a constraint, generating counterfactual samples and pairing them with the original data segment, obtaining causal comparison training samples and collecting them to form an early warning data set, improving the causal fidelity of the early warning data set, enabling the model trained by using the data set to learn causal logic rather than statistical correlation, and reducing the false positive rate.

Description

Technical Field

[0001] This application relates to the fields of intelligent nursing and artificial intelligence technology, and in particular to a method, system and computer-readable storage medium for constructing early warning datasets for embodied intelligent nursing scenarios. Background Technology

[0002] In nursing homes and home care settings, embodied intelligent robots need to anticipate acute risk events such as falls and sudden respiratory and cardiac arrest in advance to buy caregivers a window of opportunity to intervene. This predictive capability relies on large-scale, high-quality training datasets.

[0003] Currently, some technologies have attempted to collect nursing scenario data through multimodal sensors and use neural network models to calculate the correlation weights between various modal data. However, such methods are essentially based on statistical correlation and cannot distinguish between true causal relationships and spurious correlations. Summary of the Invention

[0004] This application proposes a method for constructing an early warning dataset for embodied intelligent nursing scenarios, aiming to solve the technical problems of unreliable causal labeling and unresolved cross-modal causal conflicts in the prior art.

[0005] This application provides a method for constructing an early warning dataset for embodied intelligent nursing scenarios, the method including: Acquire multimodal perception data of the target object over a continuous time period; Based on the personalized multimodal baseline of the target object, calculate the cumulative deviation of each modal sensing data from the corresponding baseline; If the cumulative deviation meets the anomaly triggering condition, then extract the multimodal data segment associated with the anomaly event from the multimodal sensing data; Causal analysis is performed on each modality data segment to generate an initial causal subgraph for each modality. Conflict detection is performed on the initial causal subgraphs of different modalities, and manual review is triggered when a conflict is detected. The system obtains manually input modification information of the initial causal subgraph and confirmation information of causal edge confidence, forming a multimodal causal graph with causal labels and confidence labels. Using a multimodal causal graph as a constraint, intervention simulations are performed on multimodal data segments to generate corresponding counterfactual samples. Multimodal data segments are then paired with counterfactual samples to obtain causal comparison training samples. Multiple sets of causal comparison training samples are collected to form an early warning dataset.

[0006] Accordingly, this application also provides a system for constructing an early warning dataset for embodied intelligent nursing scenarios, including: The synchronous acquisition module is used to acquire multimodal sensing data of the target object over a continuous time period; An adaptive truncation module is used to calculate the cumulative deviation of each modal sensing data from the corresponding baseline based on the personalized multimodal baseline of the target object; and to truncate multimodal data segments associated with abnormal events from the multimodal sensing data when the cumulative deviation meets the abnormal triggering conditions. The human-machine collaborative annotation module is used to perform causal analysis on each modality data segment and generate an initial causal subgraph for each modality; to perform conflict detection on the initial causal subgraphs of different modalities and trigger manual review when a conflict is detected; and to obtain the modification information of the initial causal subgraphs and the confirmation information of the confidence of the causal edges by manual input, forming a multimodal causal graph with causal annotation and confidence annotation. The counterfactual sample generation module is used to simulate intervention on multimodal data segments using a multimodal causal graph as a constraint, generate corresponding counterfactual samples, and pair multimodal data segments with counterfactual samples to obtain causal comparison training samples. The sample output module is used to aggregate multiple sets of causal comparison training samples to form an early warning dataset.

[0007] Accordingly, this application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute the above-described method for constructing an early warning dataset for embodied intelligent care scenarios.

[0008] This application acquires multimodal sensing data of a target object over a continuous time period; calculates the cumulative deviation of each modal sensing data point from the corresponding baseline based on the target object's personalized multimodal baseline; if the cumulative deviation meets the anomaly triggering condition, it extracts multimodal data segments associated with the anomaly from the multimodal sensing data; performs causal analysis on each modal data segment to generate an initial causal subgraph for each modality; performs conflict detection on the initial causal subgraphs of different modalities and triggers manual review when a conflict is detected; acquires manually input modification information for the initial causal subgraph and confirmation information for the confidence of causal edges to form a multimodal causal graph with causal and confidence labels; uses the multimodal causal graph as a constraint to perform intervention simulation on the multimodal data segments, generating corresponding counterfactual samples; pairs the multimodal data segments with the counterfactual samples to obtain causal comparison training samples; and aggregates multiple sets of causal comparison training samples to form an early warning dataset. This improves the causal fidelity of the early warning dataset, enabling the early warning model trained using this dataset to learn causal logic rather than statistical correlation, thus reducing the false alarm rate. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] in: Figure 1 This is a schematic diagram of an early warning dataset construction system for embodied intelligent nursing scenarios, provided as an embodiment of this application.

[0011] Figure 2 This is a flowchart illustrating a method for constructing an early warning dataset for embodied intelligent nursing scenarios, as provided in an embodiment of this application.

[0012] Figure 3 This is a flowchart illustrating a personalized baseline cumulative deviation and anomaly determination process provided in an embodiment of this application.

[0013] Figure 4 This is a schematic diagram of a modal-independent causal discovery and conflict detection process provided in an embodiment of this application.

[0014] Figure 5 This is a schematic diagram of a counterfactual sample generation process provided in an embodiment of this application.

[0015] Figure 6 This is a schematic diagram of a causal closed-loop verification iteration process provided in an embodiment of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0017] This application provides a method, system, and computer-readable storage medium for constructing an early warning dataset for embodied intelligent nursing scenarios. Specifically, the method for constructing an early warning dataset for embodied intelligent nursing scenarios can be executed by an electronic device, which can be a terminal or a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0018] To address the aforementioned issues, this application provides a method, system, and computer-readable storage medium for constructing an early warning dataset for embodied intelligent nursing scenarios. It solves the technical problems of unreliable causal labeling, unresolved cross-modal causal conflicts, and lack of counterfactual control samples in existing technologies. This improves the causal fidelity of the early warning dataset, enabling early warning models trained on this dataset to learn causal logic rather than statistical correlation, thus reducing false alarm rates. Simultaneously, personalized baselines enable precise monitoring and adaptive cropping tailored to individual needs, while a closed-loop verification and iteration mechanism continuously optimizes the dataset quality, forming an end-to-end high-causal-fidelity early warning dataset construction scheme that is easy to deploy at scale.

[0019] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0020] This application provides an early warning dataset construction system for embodied intelligent nursing scenarios. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of an early warning dataset construction system for embodied intelligent nursing scenarios, provided as an embodiment of this application.

[0021] like Figure 1 As shown, the system includes a synchronous acquisition module, an adaptive truncation module, a human-machine collaborative annotation module, a counterfactual sample generation module, and a sample output module. The functions of each module are as follows: The synchronous acquisition module is used to acquire multimodal sensing data of the target object within a continuous time period. The synchronous acquisition module performs frame-level alignment of the data frames acquired by each sensor using a unified time reference to obtain time-synchronized multimodal sensing data, and then sends the multimodal sensing data to the adaptive interception module.

[0022] The adaptive truncation module is used to calculate the cumulative deviation of each modal sensing data from the corresponding baseline based on the personalized multimodal baseline of the target object; and to truncate multimodal data segments associated with abnormal events from the multimodal sensing data when the cumulative deviation meets the abnormal triggering conditions, and send the truncated multimodal data segments to the human-machine collaborative annotation module.

[0023] The human-machine collaborative annotation module is used to perform causal analysis on each modality data segment, generate initial causal subgraphs for each modality, perform conflict detection on the initial causal subgraphs of different modalities, and trigger manual review when a conflict is detected; it is also used to obtain the modification information of the initial causal subgraphs and the confirmation information of the confidence of causal edges input by humans, form a multimodal causal graph with causal annotation and confidence annotation, and send the generated multimodal causal graph to the counterfactual sample generation module.

[0024] The counterfactual sample generation module is used to simulate intervention on multimodal data segments using a multimodal causal graph as a constraint, generate corresponding counterfactual samples, pair multimodal data segments with counterfactual samples to obtain causal comparison training samples, and send the obtained causal comparison training samples to the sample output module.

[0025] The sample output module is used to aggregate multiple sets of causal comparison training samples to form an early warning dataset, and store the early warning dataset in a database or make it available for external access.

[0026] In some embodiments, the system further includes a causal closed-loop verification module, which is used to train a lightweight early warning model using the early warning dataset as training data, perform causal chain prediction on unlabeled multimodal sensing data using the lightweight early warning model, filter out high uncertainty samples based on the comparison between the prediction results and existing labels and submit them for review, and obtain the correction information after review to update the early warning dataset.

[0027] For further details, please refer to Figure 2 , Figure 2 This is a flowchart illustrating a method for constructing an early warning dataset for embodied intelligent nursing scenarios, provided in an embodiment of this application. The specific process of this method for constructing an early warning dataset for embodied intelligent nursing scenarios can be as follows: 101. Obtain multimodal perception data of the target object within a continuous time period.

[0028] In this application, the target audience refers to the individuals served by the monitoring system in the nursing care setting, namely, individuals who require health status monitoring and risk warnings and are served by the embodied intelligent robot. This includes, but is not limited to, people in nursing homes, people in home care settings, and other individuals who require health monitoring and risk warnings.

[0029] In some embodiments, the step "acquiring multimodal sensing data of the target object over a continuous time period" may include the following operations: Multimodal perception data is collected by a heterogeneous sensor array deployed in a nursing setting. The heterogeneous sensor array includes at least a visual sensor, a radar sensor, an infrared temperature sensor, and an acoustic sensor. A timestamp is added to the data frames collected by each sensor using a unified time reference, and the data frames of each sensor are aligned at the frame level according to the timestamps to obtain time-synchronized multimodal sensing data.

[0030] In this embodiment, a wheeled, embodied intelligent robot is deployed in the space where the target object (e.g., an elderly person) is located. The robot is approximately 1.2 meters tall. The robot body integrates a heterogeneous sensor array, specifically including a visual semantic acquisition unit, a spatial perception and micro-motion detection unit, a non-contact physiological perception module, and an environmental acoustic event monitoring unit.

[0031] The visual semantic acquisition unit uses an RGB-D camera mounted on the robot's head-mounted pitch platform to acquire color images (1920×1080 resolution) and depth images (1280×720 resolution) at 30fps. The camera's field of view is 87°×58°, covering the target object's daily activity area. The acquired color images are used to extract facial motion units and skeletal joints, while the depth images are used for 3D environmental reconstruction and target object distance estimation.

[0032] The spatial perception and micro-motion detection unit employs a 60GHz millimeter-wave radar module, mounted on the robot's chest, and transmits frequency-modulated continuous wave signals. The radar, through a one-transmitter, three-receiver antenna array, simultaneously acquires target distance, velocity, and micro-displacement waveforms on the chest cavity surface. With a radar sampling rate of 1kHz, it can resolve micro-motions at the 0.1mm level. The acquired radar signals are used to extract respiratory waveforms and heart rate variability.

[0033] The non-contact physiological sensing module integrates a thermopile infrared array sensor, which is installed next to the robot's head camera. It collects the temperature distribution of the target object's face and exposed skin area at 10fps, and estimates the body surface temperature and heart rate non-contactly after calibration.

[0034] The environmental acoustic event monitoring unit employs a microphone array, supporting 360° sound source localization and speech enhancement, and acquires environmental audio at a 16kHz sampling rate. The audio stream is processed by a local speech activity detection module to extract non-verbal acoustic events such as groans and sighs, as well as voice commands actively issued by the target object.

[0035] The aforementioned visual semantic acquisition unit, spatial perception and micro-motion detection unit, non-contact physiological perception module, and environmental acoustic event monitoring unit are all connected to the robot's internal edge computing unit.

[0036] Thus, the obtained multimodal perception data includes: visual behavior data (color and depth images acquired by RGB-D cameras, used to extract facial action units, skeletal joints, gait parameters, etc.), spatial posture and micro-motion data (target distance, velocity, and chest cavity surface micro-displacement waveforms acquired by millimeter-wave radar, used to extract respiratory waveforms and heart rate variability), non-contact physiological sign data (temperature distribution of the face and exposed skin areas acquired by infrared array sensors, used to estimate body surface temperature and heart rate), and environmental acoustic data (environmental audio acquired by microphone arrays, used to extract non-verbal acoustic events such as groans and sighs, as well as voice commands).

[0037] To ensure precise frame-level alignment of multimodal data, all sensors are synchronized to the same master clock source via the IEEE 1588 Precision Time Protocol. This master clock source is provided by the PTP Grandmaster clock from the GPS timing module of the robot chassis. Specifically, the edge computing unit runs a PTP slave clock service, and each sensor connects to the edge computing unit via Gigabit Ethernet, achieving nanosecond-level clock synchronization at the hardware level. Each frame of data is appended with a 64-bit absolute timestamp based on PTP synchronization by the sensor driver upon output.

[0038] All raw data is stored locally on the edge computing unit in ROS 2 bag format. Meanwhile, structured features extracted from the data from each sensor (including facial motion unit intensity sequences, respiratory rate, time-domain indices of heart rate variability, acoustic event labels, etc.) are written to the time-series database in real time for subsequent anomaly monitoring.

[0039] Using the above method, multimodal perception data of the target object with time synchronization over a continuous time period was obtained.

[0040] 102. Based on the personalized multimodal baseline of the target object, calculate the cumulative deviation of each modal sensing data from the corresponding baseline.

[0041] In this application, the personalized multimodal baseline refers to a normal state reference benchmark established individually for each target subject. Because different target subjects have individual differences in age, physical condition, and behavioral habits, a uniform standard threshold cannot accurately reflect the true state of each target subject; therefore, it is necessary to establish a unique baseline for each target subject.

[0042] In some embodiments, the method may further include the step of constructing a personalized multimodal baseline for the target object; Constructing a personalized multimodal baseline for the target object can specifically include the following operations: Obtain multimodal data samples of the target object in a stationary state; The modal parameters in the multimodal data sample were statistically analyzed to obtain the mean and standard deviation of each modal parameter; Based on the mean and standard deviation of each modal parameter, the normal fluctuation range of each modal parameter is constructed to obtain a personalized multimodal baseline.

[0043] In this embodiment, in the initial stage after the target object (e.g., an elderly person) moves into the target environment (e.g., a nursing home), multimodal data of the target object in its natural living state are collected to establish a unique personalized baseline model. The data collection period can be set to a preset duration (e.g., one week) to cover the target object's daily activity patterns.

[0044] During data collection, nursing staff observed the target subjects' conditions and manually labeled "stable state" periods. A stable state refers to a period in which the target subject's physical condition is relatively stable without any risk events, such as sitting quietly watching television or taking a slow indoor walk. Based on the stable state periods labeled by the nursing staff, data samples under the stable state were extracted from the collected multimodal data.

[0045] For stationary state data, the corresponding modal parameters are extracted from each modal data, specifically including: Gait parameters: gait speed, stride variation coefficient, double support ratio, and standard deviation of lateral displacement of body center of gravity are extracted from the skeletal point trajectory of the depth camera.

[0046] Physiological parameters: respiratory rate, heart rate variability SDNN (standard deviation of RR interval throughout sinus beats) and RMSSD (root mean square of the difference between adjacent RR intervals) were extracted from radar micro-motion waveforms; average facial temperature was obtained from thermal imaging.

[0047] Acoustic parameters: mean ambient sound pressure level, mean and variance of speech fundamental frequency.

[0048] For each modal parameter, assuming it follows a normal distribution, calculate the mean. and standard deviation Construct a personalized baseline normal fluctuation range. -2 , +2 For example, if the average steady pace of the target object is 0.8 m / s and the standard deviation is 0.1 m / s, then a pace below 0.6 m / s is considered a slight deviation.

[0049] In this way, a unique normal fluctuation range is established for each modal parameter. The normal fluctuation ranges of all modal parameters together constitute the personalized multimodal baseline of the target object. This baseline reflects the normal data distribution of the target object under a healthy and stable state, and is used as the benchmark for determining whether the data deviates during subsequent monitoring.

[0050] In some embodiments, the step "calculating the cumulative deviation of each modality sensing data from the corresponding baseline based on the personalized multimodal baseline of the target object" may include the following operations: Each modal parameter is extracted from each modal sensing data, and the instantaneous deviation of each modal parameter is calculated. The instantaneous deviation is used to characterize the degree of deviation of the value of the modal parameter from the corresponding baseline. The instantaneous deviations of each modal parameter are weighted and summed, and the weighted summation results are accumulated over a preset time period to obtain the cumulative deviation.

[0051] During the real-time monitoring phase, modal parameters are extracted from each modal sensing data at preset time intervals (e.g., per second), and the instantaneous deviation of each modal parameter is calculated. Each modal parameter includes gait parameters, physiological parameters, and acoustic parameters.

[0052] Specifically, for each modal parameter, its real-time value is compared with the corresponding baseline value. The baseline value is the mean of the modal parameter under the steady state of the target object and is recorded in the personalized multimodal baseline. The instantaneous deviation is used to characterize the degree and direction of the modal parameter's deviation from its baseline at a certain moment.

[0053] In some embodiments, instantaneous deviation The calculation formula is: in, For the modal parameters at time... The real-time value, This represents the baseline mean of the modal parameters. The standard deviation of this modal parameter is... , All values ​​are derived from the target object's personalized multimodal baseline. A positive instantaneous deviation indicates that the parameter is above the baseline mean; a negative instantaneous deviation indicates that the parameter is below the baseline mean. For example, in a fall warning scenario, a negative instantaneous deviation of walking speed (i.e., walking speed below the baseline value) indicates that the target object's walking speed has slowed down, which may indicate an anomaly.

[0054] After obtaining the instantaneous deviation of each modal parameter, the instantaneous deviations are weighted and summed, and the weighted sum is accumulated over a preset time period to obtain the cumulative deviation. This process integrates the instantaneous abnormal signals of multiple modal parameters into a comprehensive index, reflecting the overall deviation trend of the target object in the time dimension.

[0055] Specifically, for each moment Calculate the instantaneous deviation of each modal parameter at that moment. A preset weight is applied to the instantaneous deviation of each modal parameter. The values ​​are then weighted and summed to obtain the overall deviation value at that moment. The weight values ​​corresponding to each modal parameter can be preset according to the historical discrimination ability of that modality for a specific anomaly.

[0056] In some embodiments, weights can be assigned based on the sensitivity and specificity of different modalities in the early warning task; for example, assigning higher weights to modalities with higher discriminative power for specific anomalies. Then, the overall deviation value is calculated from the initial time... up to the current moment Accumulate (integrate) to obtain the cumulative deviation. .

[0057] In some embodiments, cumulative deviation The calculation formula is: in, This means that the accumulation is only counted when the absolute value of the instantaneous deviation exceeds a preset threshold (e.g., 1, i.e., one standard deviation); when it is less than or equal to the threshold, it does not contribute. The weights of the i-th modal parameters; the starting time of integration. This is the time since the last accumulated deviation returned to zero. The significance of this formula is that slight fluctuations are not included in the accumulation; accumulation only occurs when the deviation exceeds a certain level (i.e., exceeds one standard deviation), thus avoiding false triggering due to normal physiological fluctuations.

[0058] For example, when this method is applied to a fall warning scenario, the weights of each modal parameter can be set as follows: gait parameter weight 0.4, radar micro-motion parameter weight 0.3, physiological parameter weight 0.2, and acoustic parameter weight 0.1. Through the above calculations, a comprehensive cumulative deviation is obtained, which serves as the basis for subsequent anomaly determination.

[0059] For example, please see Figure 3 , Figure 3 This is a flowchart illustrating a personalized baseline cumulative deviation and anomaly determination method provided in an embodiment of this application. Specifically, it may include the following steps: Collect multimodal data of the target object under stable conditions, calculate the mean and standard deviation of each modal parameter, and establish a personalized baseline fluctuation range as a benchmark for subsequent anomaly judgment.

[0060] During the real-time monitoring phase, each modal parameter is continuously detected, and the deviation of each parameter from its baseline value is calculated. The deviations of each modal parameter are weighted and summed, and the weighted summation results are accumulated within a preset time period to obtain the cumulative deviation.

[0061] Subsequently, the degree of cumulative deviation is determined. When the degree of cumulative deviation increases rapidly in a short period of time, it is determined to be a sudden abnormal trigger type, triggering a sudden event (such as a fall), and the truncation window width is determined accordingly (e.g., 90 seconds); when the degree of cumulative deviation accumulates slowly over a longer period of time, it is determined to be a gradual abnormal trigger type, triggering a gradual event (such as delirium), and the truncation window width is determined accordingly (e.g., 600 seconds).

[0062] Finally, using the trigger time as the end point of the extraction, multimodal data segments are extracted from the multimodal sensing data in reverse according to the determined extraction window width, and the multimodal data segments are output for subsequent causal analysis and annotation.

[0063] 103. If the cumulative deviation meets the abnormal triggering conditions, extract the multimodal data segment associated with the abnormal event from the multimodal sensing data.

[0064] During real-time monitoring, the system needs to determine whether any abnormal events requiring attention have occurred based on the cumulative deviation, and trigger subsequent data capture and labeling processes accordingly. Therefore, pre-defined abnormal trigger conditions serve as the basis for determining whether the cumulative deviation has reached the trigger standard.

[0065] In this embodiment, the anomaly triggering condition is used to determine whether a risk event requiring attention has occurred in the target object. When the cumulative deviation meets the anomaly triggering condition, it indicates that the target object may have an abnormal trend or a risk event has occurred. At this time, it is necessary to extract multimodal data segments before and after this moment for subsequent causal analysis and annotation.

[0066] In some embodiments, the step "if the cumulative deviation meets the anomaly triggering condition, then extract the multimodal data segment associated with the anomaly event from the multimodal sensing data" may include the following operations: When the cumulative deviation rises to a first threshold within a preset first time window, it is determined to be a first abnormal trigger type, and the corresponding first interception window width is determined according to the first abnormal trigger type. When the cumulative deviation accumulates to a second threshold within a preset second duration window, it is determined to be a second abnormal trigger type, and the corresponding second interception window width is determined according to the second abnormal trigger type. The second duration window is longer than the first duration window, and the first interception window width is smaller than the second interception window width. Using the trigger time as the endpoint, multimodal data segments are extracted from the multimodal sensing data in reverse, according to the determined extraction window width.

[0067] Real-time monitoring of cumulative deviation It then compares the result with preset exception triggering conditions. In this embodiment, two types of exception triggering are preset.

[0068] The first abnormal trigger type corresponds to sudden abnormalities and is applicable to sudden risk events such as falls and acute cardiac events. When the cumulative deviation... A rapid increase in deviation within a short period, i.e., an increase in the cumulative deviation from a low level to a first threshold within a preset first time window, is identified as a first abnormal trigger type. In some embodiments, the determination criteria may be: The value increased from below 1.0 to above 5.0 within 10 seconds. It should be understood that the specific values ​​of the first duration window and the first threshold can be adjusted according to the individual characteristics of the target subject and clinical experience, and will not be illustrated here.

[0069] The second type of abnormal trigger corresponds to progressive abnormalities, applicable to progressive risk events such as delirium and worsening respiratory infections. When the cumulative deviation... When the deviation accumulates slowly and continuously over a relatively long period, reaching a second threshold within a preset second time window, it is determined to be a second abnormal trigger type. The second time window is longer than the first time window. In some embodiments, the determination condition may be: The rate should rise slowly and continuously, exceeding the threshold of 12.0, and maintain this upward trend for at least 300 seconds. Optionally, a sliding window linear fitting can be used to determine if the slope of the upward trend is greater than 0.02 / min. It should be understood that the specific values ​​of the second duration window and the second threshold can be adjusted based on the individual characteristics of the target subject and clinical experience; examples will not be provided here.

[0070] When any abnormal trigger type is met, an event trigger signal is generated, and the current time is recorded as the trigger time T.

[0071] Based on the determined exception trigger type, the corresponding truncated window width is adaptively determined, specifically: For the first type of anomaly triggering (sudden), the precursor time is relatively short, so a corresponding first intercept window width is determined. This width is relatively narrow to accurately cover the critical precursor period before the sudden risk event occurs. For the second type of anomaly triggering (gradual), the precursor time is relatively long, so a corresponding second intercept window width is determined. This width is relatively wide to cover the longer behavioral evolution process before the gradual anomaly occurs. The first intercept window width is smaller than the second intercept window width.

[0072] In some embodiments, the first capture window width is 90 seconds, and the second capture window width is 600 seconds. It should be understood that the specific value of the window width can be adjusted according to the individual characteristics and clinical needs of the target object, and will not be illustrated here.

[0073] Using the trigger time T as the endpoint, multimodal data segments are extracted from the multimodal sensing data in a forward direction according to the determined extraction window width. Specifically, for the first abnormal trigger type, a multimodal data segment with a length equal to the first extraction window width is extracted forward from the trigger time T; for the second abnormal trigger type, a multimodal data segment with a length equal to the second extraction window width is extracted forward from the trigger time T.

[0074] The extracted multimodal data segments contain all timestamped original multimodal data within the interval from T-Δt to T (Δt is the determined extraction window width), covering complete precursor information before the occurrence of risk events, which is used for subsequent causal analysis and sample labeling.

[0075] In some embodiments, the original multimodal data within the range is read from the cache of the edge computing unit, the captured multimodal data fragments are packaged into a preset format, and uploaded to the annotation server for use in subsequent steps.

[0076] 104. Perform causal analysis on each modality data segment to generate an initial causal subgraph for each modality. Perform conflict detection on the initial causal subgraphs of different modalities and trigger manual review when a conflict is detected.

[0077] Multimodal data segments record the behavior, physiology, and environment of a target object in the period preceding a risk event. This information includes both precursory signals with a genuine causal relationship to the risk event and irrelevant signals that occur simultaneously only due to temporal coincidence. To identify truly predictive causal relationships from the data, causal analysis of each modal data segment is necessary.

[0078] In this embodiment, causal analysis refers to the process of inferring the causal direction between features within each modal data segment. Specifically, for a given modal data segment, any two features within that modality are selected as the analysis objects, one feature as the cause and the other as the effect. By calculating the causal strength between the two features, it is determined whether the causal direction is valid. The result of causal analysis is a directed causal relationship, that is, it clearly points in one direction (from cause to effect).

[0079] In some embodiments, the step "perform causal analysis on each modality data segment, generate an initial causal subgraph for each modality, perform conflict detection on the initial causal subgraphs of different modalities, and trigger manual review when a conflict is detected" may include the following operations: Extract each modal feature from each modal data segment; Causal analysis is performed on each modal feature to calculate the causal direction and causal strength between any two features within each modality, and an initial causal subgraph for each modality is generated. The initial causal subgraph contains each feature node within the modality and the directed edges between feature nodes. The initial causal subgraphs of different modalities are merged into a global candidate causal graph, and the direction consistency of directed edges from different modalities between the same pair of feature nodes in the global candidate causal graph is compared. When different modalities give opposite causal directions for the same pair of feature nodes, it is determined to be a bidirectional edge conflict; When directed edges of different modalities form a circular dependency structure, it is determined to be a circular dependency conflict; The multimodal data segments corresponding to the initial causal subgraphs with bidirectional edge conflicts or circular dependency conflicts are marked as conflict samples, and the directed edges that have conflicts are highlighted in a visual manner before manual review is triggered.

[0080] For the extracted multimodal data segments, feature sequences are first extracted for each modality. The purpose of feature extraction is to convert the raw sensor data into feature representations that can be used for causal analysis. Each modality employs an appropriate feature extraction method based on its data characteristics.

[0081] For the visual modality, facial action unit intensities were extracted from color images, including pain-related AU4 (contraction of the corrugator supercilii muscle, causing the eyebrows to lower and converge), AU6 (contraction of the orbicularis oculi muscle (orbital portion), causing the cheeks to lift and the eyes to narrow), AU7 (contraction of the orbicularis oculi muscle (eyelid portion), causing the eyelids to tighten and narrow), AU9 (contraction of the levator labii superioris and alarplasty muscle, pulling the root of the nose upward and forming wrinkles at the bridge of the nose (wrinkled nose)), and AU10 (contraction of the levator labii superioris muscle, lifting the upper lip upward, usually accompanied by deepening of the nasolabial fold), etc., to characterize the target subject's pain level and emotional state; coordinates of 18 joint points throughout the body were extracted from depth images, and gait variation coefficients and body curl index (trunk to thigh angle) were calculated based on the joint point coordinates to characterize the target subject's motor ability and posture changes.

[0082] For spatial micro-motion modes, respiratory waveforms and heart rate interval sequences are obtained from radar I / Q signals (two orthogonal signals, in-phase (I) and quadrature (Q), output from the radar receiver) through phase demodulation. Respiratory rate, respiratory depth coefficient of variation, and time-domain and frequency-domain indices of heart rate variability (such as the LF / HF ratio) are calculated to characterize the respiratory function and autonomic nervous state of the target object.

[0083] For physiological modalities, the rate of change of facial temperature gradient is extracted from thermal imaging sequences, and the SDNN is calculated from the beat-by-beat heart rate jointly estimated by radar and thermal imaging to characterize the thermoregulation ability and heart rate variability of the target object.

[0084] For acoustic modalities, a pre-trained audio event detection model is used to detect audio events such as groans and sighs, and the frequency of these events is statistically analyzed. Paralinguistic features such as fundamental frequency, loudness, and speech rate are extracted to characterize the vocal state and emotional changes of the target subject.

[0085] It should be understood that the above feature extraction methods are only examples. The specific feature types of each modality can be adjusted and expanded according to the actual application scenario. They will not be illustrated here.

[0086] Causal analysis is performed on each modal feature to calculate the causal direction and causal strength between any two features within each modality. Causal direction represents the directed relationship from one feature to another (e.g., feature A → feature B, indicating that a change in A directly leads to a change in B), and causal strength represents the confidence level of the directed relationship.

[0087] In some embodiments, causal analysis is implemented using the transfer entropy method. Transfer entropy is an asymmetric measure based on information theory, capable of detecting the direction of directed information transfer between two time series. Specifically, for any two features X and Y, the transfer entropy TEX→Y from X to Y and the transfer entropy TEY→X from Y to X are calculated, and the direction of the directed edges is determined through statistical tests. If the transfer entropy from X to Y is significantly greater than the transfer entropy from Y to X, a causal relationship from X to Y is determined; otherwise, a causal relationship from Y to X is determined. In this way, directed edges are formed between feature nodes within each mode, and all directed edges of each mode together constitute the initial causal subgraph of that mode.

[0088] The initial causal subgraph for each modality contains the feature nodes within that modality and the directed edges between the feature nodes. For example, the visual subgraph may contain a causal chain of "increased AU4 intensity → increased gait variability → increased lateral sway of body center of gravity"; the radar subgraph may contain a causal chain of "increased respiratory rate → decreased heart rate variability → increased trunk micro-movement amplitude"; and the acoustic subgraph may contain a causal chain of "increased groaning frequency → increased speech fundamental frequency".

[0089] It should be understood that transitive entropy is only an exemplary implementation of causal analysis. Other causal discovery algorithms, such as Granger causality test and structural equation modeling, can also be used, which will not be illustrated here.

[0090] Initial causal subgraphs from different modalities are merged into a single global candidate causal graph. During the merging process, different modalities may assign different directed edge directions to the same pair of feature nodes, leading to conflicts. Conflict detection is used to identify these cross-modal causal direction contradictions.

[0091] Collision detection specifically includes the following two types: Bidirectional edge conflict: Detects whether directed edges pointing to the same feature node have opposite directions. When different modalities give opposite causal directions for the same pair of feature nodes (e.g., the visual subgraph infers "increased gait variation coefficient → increased body center of gravity sway", while the radar subgraph infers "increased body center of gravity sway → increased gait variation coefficient"), it is determined to be a bidirectional edge conflict, that is, the two modalities give opposite conclusions about the causal relationship of the same pair of feature nodes.

[0092] Circular dependency conflict: Detects whether directed edges from different modalities form a circular dependency structure. When directed edges from different modalities form a circular structure A→B→C→A, it is determined to be a circular dependency conflict. A circular dependency structure means that the causal directions supported by each modality form a closed loop globally, making it impossible to determine the true causal starting point, requiring manual intervention for arbitration.

[0093] The multimodal data segments corresponding to the initial causal subgraphs with bidirectional edge conflicts or circular dependency conflicts are marked as conflict samples, and the directed edges that have conflicts are highlighted in a visual manner before manual review is triggered.

[0094] Manual review: On the annotation interface, conflicting directed edges are highlighted visually, for example, by displaying conflicting edges with red dashed lines, and a pop-up window informs the reviewers of the specific details of the conflict. The reviewers observe the synchronously played multi-channel data (including camera footage, radar point cloud dynamics, vital sign waveforms, and acoustic spectra) and arbitrate the conflicting directed edges.

[0095] The annotation interface is a visual interactive platform provided by the human-computer collaborative annotation module for reviewers to perform causal annotation operations. It is used to display multimodal data fragments and their corresponding initial causal subgraphs, and to receive editing instructions and annotation information from reviewers.

[0096] Reviewers can click on any conflicting edge to view its corresponding causal strength and significance level, and then decide whether to retain, delete, or reverse the direction based on clinical knowledge. For edges without conflicts, reviewers can also add causal connections that the algorithm did not detect based on experience. Through manual review, cross-modal causal direction conflicts are eliminated, ensuring the consistency of multimodal causal labeling.

[0097] 105. Obtain the modification information of the initial causal subgraph and the confirmation information of the confidence of causal edges from the manual input, and form a multimodal causal graph with causal labels and confidence labels.

[0098] In some embodiments, the step of "obtaining manually input modification information of the initial causal subgraph and confirmation information of causal edge confidence to form a multimodal causal graph with causal labels and confidence labels" may include the following operations: Display the data segments of each modality and their corresponding initial causal subgraphs; In response to manual operation, obtain editing instructions for the target directed edges in the initial causal subgraph. Editing instructions include: confirm retention, delete, reverse direction, or add new directed edges in the initial causal subgraph. Obtain the confidence level information of each causal edge input by the user. The confidence level information includes three levels: high confidence, medium confidence, and low confidence. In response to manual operation, type labeling information of each node in the initial causal subgraph is obtained. Node types include behavioral precursor nodes, emotional drift nodes, and vital sign deviation nodes. The initial causal subgraph is updated based on the editing instructions, confidence level information, and type labeling information, generating a multimodal causal graph with node type labels, causal edge direction labels, and causal edge confidence labels.

[0099] The annotation interface displays multimodal data segments and their corresponding initial causal subgraphs in a synchronized playback manner. The interface employs a four-channel synchronized playback layout, simultaneously displaying camera feeds, dynamic radar point cloud images, vital sign waveforms, and acoustic spectra, allowing reviewers to make comprehensive judgments based on multi-dimensional information. The central area of ​​the annotation interface displays the initial causal subgraph in the form of a directed graph, with nodes representing modal features and directed edges representing causal directions.

[0100] Reviewers edit the directed edges in the initial causal subgraph through the annotation interface. The system responds to the reviewer's manual operation by obtaining editing instructions for the target directed edges. Editing instructions include confirming retention (confirming a directed edge is valid), deleting (determining a directed edge is invalid), reversing direction (correcting the causal direction), or adding new directed edges to the initial causal subgraph (adding causal connections not detected by the algorithm based on clinical knowledge). Through these editing operations, potential errors or omissions in the initial causal subgraph are eliminated.

[0101] Reviewers input confidence level information for each causal edge. Confidence levels include high, medium, and low confidence, representing the degree of certainty that the causal edge is valid. In some embodiments, the system automatically calculates initial confidence values ​​based on the effect size of the causal analysis and displays them on the interface; reviewers can manually adjust these initial values.

[0102] The reviewers also labeled the types of each feature node in the initial causal subgraph. Node types included behavioral precursor nodes (representing changes in the target object's behavioral patterns), emotion drift nodes (representing changes in the target object's emotional state), and vital sign deviation nodes (representing abnormalities in the target object's physiological indicators). Through node type labeling, clinical semantic information was assigned to each feature node in the causal graph.

[0103] Based on the editing instructions, confidence level information, and type labeling information input by the reviewers, the initial causal subgraph is updated, and a multimodal causal graph with node type labels, causal edge direction labels, and causal edge confidence labels is generated for subsequent counterfactual sample generation.

[0104] For example, please see Figure 4 , Figure 4 This is a schematic diagram of a modality-independent causality discovery and conflict detection process provided for an embodiment of this application. Specifically, it may include the following steps: Feature extraction was performed on the multimodal data segments. Visual modality extracted AU (facial action unit) intensity and skeletal joint coordinates; spatial micromotion modality extracted respiratory waveforms and HRV (heart rate variability) indices; physiological modality extracted body surface temperature and heart rate; acoustic modality extracted acoustic features such as groaning event frequency and fundamental frequency.

[0105] For each modality feature, a transfer entropy causality test is performed to calculate the causal direction and causal strength between features within each modality, generating an initial causal subgraph for each modality, including a visual causal subgraph, a radar causal subgraph, a physiological causal subgraph, and an acoustic causal subgraph.

[0106] Subsequently, the initial causal subgraphs of each modality are merged into a global candidate causal graph, and conflict detection is performed on the initial causal subgraphs of different modalities. When the causal directions of the directed edges given by different modalities for the same pair of feature nodes are opposite, it is determined to be a directional conflict; when the directed edges of different modalities form a cyclic dependency structure, it is determined to be a cyclic conflict.

[0107] If a conflict exists, the corresponding multimodal data segment is marked as a modal conflict sample, and the conflict edge is displayed in a highlighted manner (e.g., a red dashed line) and an expert review is triggered; if there is no conflict, the initial candidate DAG (directed acyclic graph) is directly generated.

[0108] After experts confirm or modify the initial candidate DAG, they output the final multimodal causal graph and the confidence level of each causal edge.

[0109] 106. Using multimodal causal graphs as constraints, intervention simulations are performed on multimodal data segments to generate corresponding counterfactual samples. Multimodal data segments are paired with counterfactual samples to obtain causal comparison training samples.

[0110] Counterfactual samples, in particular, are hypothetical data samples generated by intervening in key factor nodes (i.e., modifying the state of those nodes) in the original multimodal data segment while keeping other nodes unchanged. Their core idea is to answer the counterfactual question: "If a certain precursor factor had not occurred, would the event still have happened?"

[0111] In some embodiments, the step "using a multimodal causal graph as a constraint to perform intervention simulation on multimodal data segments, generating corresponding counterfactual samples, and pairing multimodal data segments with counterfactual samples to obtain causal comparison training samples" may include the following operations: Obtain the target time window data corresponding to the target causal edge in the multimodal data segment; Encode the target time window data into the latent space to obtain latent variables; Based on the multimodal causal graph, determine the dimension of the latent variable corresponding to the causal node of the target causal edge in the latent space; The latent variables are modified in a targeted manner along the dimension of the latent variables to generate the modified latent variables; The modified latent variables are decoded into counterfactual multimodal data fragments. In the counterfactual multimodal data fragments, the states of the nodes are changed, while the states of other nodes remain unchanged. Pair multimodal data segments with counterfactual multimodal data segments to obtain causal comparison training samples.

[0112] Before generating counterfactual samples, a generative model needs to be trained to encode multimodal data fragments into a latent space and decode them back into the data space. This generative model uses previously collected multimodal data (including stationary and outlier data) of the target object as training samples. Through unsupervised learning, it maps multimodal feature vectors to a low-dimensional latent space, ensuring a high degree of consistency between the decoded data and the original data. After training, the encoder of this generative model can compress the input multimodal data fragments into low-dimensional latent variables, and the decoder can restore the latent variables to multimodal data fragments.

[0113] In some embodiments, the generative model can be implemented using a Transformer-based variational autoencoder (VAE), with the latent space dimension set to 64 dimensions, and the reconstruction error controlled within a preset range. It should be understood that the variational autoencoder is only an example; it can also be implemented using generative adversarial networks (GANs) or other generative models, which will not be illustrated here.

[0114] For each target causal edge in a multimodal data segment, the target time window data corresponding to that edge is first obtained from the multimodal data segment. The target time window refers to the time interval during which the causal node involved in the causal edge undergoes significant changes (e.g., the time period during which the gait variation coefficient increases). This target time window data is then input into the encoder of the generative model to obtain its latent variables in the latent space.

[0115] Based on the multimodal causal graph, the latent variable dimension corresponding to the cause node of the target causal edge in the latent space is determined. Specifically, since there is a certain correspondence between the various dimensions of the latent space and the features of the data space, the latent variable dimension corresponding to the cause node is located by analyzing the impact of changes in the dimensions of the latent variables on the decoding output. Then, the latent variable is modified in a targeted manner along this latent variable dimension. For example, moving a certain distance in the opposite direction along this dimension is equivalent to changing the state of the cause node from "abnormal" to "normal", thereby generating the modified latent variable.

[0116] In some embodiments, the dimension of the latent variable corresponding to the cause node is located by applying a small perturbation along that dimension and observing the change in the decoder output, and then moving backward a certain distance along that dimension. The specific moving distance can be adaptively adjusted according to the degree of deviation of the cause node's state.

[0117] The modified latent variables are input into the decoder of the generative model, which decodes and generates new multimodal data fragments, namely counterfactual multimodal data fragments. Since the intervention only modifies the latent variable dimensions corresponding to the cause node, in the decoded counterfactual multimodal data fragments, the state of the cause node is changed (e.g., gait returns to normal), while the states of other nodes (conditionally independent of the characteristics of the cause node) remain unchanged (e.g., the decreasing trend of heart rate variability remains unchanged), which is guaranteed by the structural constraints of the causal graph.

[0118] Using the above method, counterfactual multimodal data segments corresponding to the original multimodal data segments were obtained. The counterfactual segments present a hypothetical scenario: "If the cause node had not experienced an anomaly and the other modal data remained unchanged, would the event still have occurred?"

[0119] The original multimodal data fragments are paired with the generated counterfactual multimodal data fragments to form a set of causal comparison training samples. The original multimodal data fragments correspond to the real-world scenario where the event occurred (labeled "Event Occurred"), while the counterfactual multimodal data fragments correspond to the hypothetical scenario where the event did not occur after intervention at a node (labeled "Event Did Not Occur"). The above intervention simulation and pairing operation is performed on each causal edge in the multimodal causal graph to form multiple sets of causal comparison training samples. The paired metadata records the corresponding causal edge identifier and confidence level for reference during subsequent training.

[0120] For example, please see Figure 5 , Figure 5 This is a schematic diagram of a counterfactual sample generation process provided in an embodiment of this application. Specifically, it may include the following steps: Obtain the original multimodal data fragments and select a target causal edge in the causal graph as the intervention object, such as "gait disorder → fall".

[0121] The target time window data corresponding to the causal edge is input into the VAE encoder to obtain latent variables. Based on the multimodal causal graph, the latent variable dimension direction corresponding to the cause node (e.g., "gait disorder") of the target causal edge is located in the latent space. A reverse movement intervention is performed along the latent variable dimension direction of the cause node (i.e., moving towards the direction in which the cause node's state tends to normalize), generating modified latent variables.

[0122] The modified latent variables are input into the VAE decoder to generate counterfactual multimodal data fragments. In these counterfactual fragments, because the states of nodes are changed (e.g., gait returns to normal), while the states of other nodes remain unchanged (e.g., the original physiological indicators do not change trend), a counterfactual fragment corresponding to the original fragment is obtained.

[0123] Finally, the original multimodal data fragments (labeled "fall down") are used as original samples and paired with the generated counterfactual multimodal data fragments (labeled "not fall down") to form a set of causal comparison training samples.

[0124] 107. Collect multiple sets of causal comparison training samples to form an early warning dataset.

[0125] In this embodiment, the original multimodal data segments are packaged with a corresponding set of counterfactual samples into a sample group. Each sample group contains the original data segment, the counterfactual data segment, and metadata. Multiple sample groups are aggregated to form an early warning dataset. Each sample group in this dataset consists of a pair of original samples and corresponding counterfactual samples, ensuring causal fidelity of the dataset.

[0126] The early warning dataset is stored in a sample library for training or updating early warning models in nursing scenarios. After the above steps, a high-quality nursing early warning dataset with causal fidelity is generated.

[0127] In some embodiments, the method may further include a causal loop closure verification iteration step, as follows: Use the early warning dataset as training data to train a lightweight early warning model; A lightweight early warning model is used to predict causal chains in unlabeled multimodal sensing data, and the predicted causal chains are output. The predicted causal chain is compared with the multimodal causal graph of the corresponding sample. When the prediction confidence of the target causal edge in the predicted causal chain is greater than or equal to the confidence threshold and there is no target causal edge in the multimodal causal graph, or when the prediction direction of the target causal edge in the predicted causal chain is opposite to the labeling direction of the corresponding causal edge in the multimodal causal graph, the multimodal data segment corresponding to the target causal edge is judged as a high uncertainty sample. High-uncertainty samples are submitted for review, correction information is obtained after review, and the early warning dataset is updated based on the correction information.

[0128] In this embodiment, the causal closed-loop verification iteration step is used to continuously optimize the dataset quality. This step uses the early warning dataset as training data to train a lightweight early warning model. In some embodiments, the lightweight early warning model can be a lightweight temporal network model, which has fewer parameters than the final deployed nursing early warning model, has a fast inference speed, and is suitable for batch processing of unlabeled data. The lightweight early warning model is used to predict causal chains in unlabeled multimodal sensing data and outputs the predicted causal chains.

[0129] The predicted causal chains are compared with the multimodal causal graphs of the corresponding samples to filter out samples with high uncertainty. Specific screening criteria include the following: The prediction confidence of the target causal edge in the predicted causal chain is greater than or equal to the confidence threshold, but the target causal edge does not exist in the multimodal causal graph (i.e. the model predicts a causal edge that has not been labeled by the expert). The prediction direction of the target causal edge in the causal chain is opposite to the labeling direction of the corresponding causal edge in the multimodal causal graph (i.e., the causal direction predicted by the model is opposite to the expert labeling).

[0130] If any condition is met, the multimodal data segment corresponding to the causal edge of the target is determined to be a high-uncertainty sample.

[0131] High-uncertainty samples are submitted for review. Reviewers compare the predicted causal chains with the expert-annotated causal chains to confirm the accuracy of the model's predictions and correct any erroneous predictions. The corrected information is then obtained and used to update the early warning dataset. The updated dataset can be used to retrain the lightweight early warning model, forming a new round of iterations.

[0132] Through the aforementioned causal closed-loop verification iteration, a continuous optimization closed loop of "labeling—training—verification—correction" is formed. After each iteration, the quality of the dataset gradually improves, and inconsistencies in expert labeling are discovered and corrected, enabling the early warning dataset to be continuously optimized during the iteration process.

[0133] For example, please see Figure 6 , Figure 6 This is a schematic diagram of an iterative process for causal loop closure verification provided in an embodiment of this application. Specifically, it may include the following steps: A lightweight early warning model was trained using the initial causal comparison training sample set as training data. This lightweight early warning model has fewer parameters than the final deployed nursing early warning model, has a fast inference speed, and is suitable for batch processing of unlabeled data.

[0134] A trained lightweight early warning model is used to predict unlabeled multimodal sensing data and output a predicted causal chain.

[0135] The predicted causal chain is compared with the expert-annotated causal chain in the corresponding sample. When the predicted confidence of the target causal edge in the predicted causal chain differs greatly from the expert annotation result (e.g., the predicted confidence is greater than or equal to the confidence threshold but the expert did not annotate the edge), or the predicted causal direction is opposite to the expert annotation direction, the sample is screened as a questionable sample (i.e., a high uncertainty sample).

[0136] The questionable samples were submitted to experts for review and correction. The experts compared the predicted causal chains with the expert-annotated causal chains to confirm the accuracy of the model's predictions and corrected any erroneous predictions. For samples consistent with the expert annotations, their sample quality was confirmed.

[0137] Finally, the dataset is updated based on the corrections made after expert review, completing one round of closed-loop verification iteration. The updated dataset can be used to retrain the lightweight early warning model, forming a continuous optimization closed loop of "labeling—training—verification—correction".

[0138] This application offers the following advantages: By employing a multimodal independent causal discovery and cross-modal conflict detection mechanism, it effectively avoids causal misleading caused by single-modal data, ensuring the consistency of multimodal causal labeling. Simultaneously, through counterfactual sample generation, the early warning model learns the causal inevitability of "if it's not A, then it's not B," rather than merely learning statistical correlation, significantly reducing the number of false alarms caused by spurious correlations. Anomaly judgment is based on personalized multimodal baselines of the target object, avoiding missed or false alarms caused by uniform thresholds. Different judgment conditions and truncation window widths are configured for sudden and gradual anomalies, resulting in more complete and higher signal-to-noise ratio truncated precursor data segments. It provides an end-to-end integrated solution from synchronous acquisition, adaptive truncation, causal labeling to counterfactual generation and closed-loop verification, without relying on additional offline processing equipment or third-party labeling platforms, making it easy to deploy at scale in various scenarios such as nursing homes and home care.

[0139] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a digital signal processor to execute the steps in any of the methods for constructing an early warning dataset for embodied intelligent nursing scenarios provided in embodiments of this application.

[0140] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0141] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0142] Since the computer program stored in the computer-readable storage medium can execute the steps in any of the methods for constructing an early warning dataset for embodied intelligent nursing scenarios provided in the embodiments of this application, the beneficial effects that any of the methods for constructing an early warning dataset for embodied intelligent nursing scenarios provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0143] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A method for constructing an early warning dataset for embodied intelligent nursing scenarios, characterized in that, The method includes: Acquire multimodal perception data of the target object over a continuous time period; Based on the personalized multimodal baseline of the target object, calculate the cumulative deviation of each modal sensing data from the corresponding baseline; If the cumulative deviation meets the anomaly triggering condition, then extract the multimodal data segment associated with the anomaly event from the multimodal sensing data; Causal analysis is performed on each modality data segment to generate an initial causal subgraph for each modality. Conflict detection is performed on the initial causal subgraphs of different modalities, and manual review is triggered when a conflict is detected. The system obtains manually input modification information of the initial causal subgraph and confirmation information of causal edge confidence, forming a multimodal causal graph with causal labels and confidence labels. Using the multimodal causal graph as a constraint, intervention simulation is performed on the multimodal data fragments to generate corresponding counterfactual samples. The multimodal data fragments are then paired with the counterfactual samples to obtain causal comparison training samples. Multiple sets of causal comparison training samples are collected to form an early warning dataset.

2. The method according to claim 1, characterized in that, The method further includes: Construct a personalized multimodal baseline for the target object; The construction of the personalized multimodal baseline for the target object includes: Obtain multimodal data samples of the target object in a stable state; The modal parameters in the multimodal data samples are statistically analyzed to obtain the mean and standard deviation of each modal parameter; Based on the mean and standard deviation of each modal parameter, the normal fluctuation range of each modal parameter is constructed to obtain the personalized multimodal baseline.

3. The method according to claim 1, characterized in that, The personalized multimodal baseline based on the target object calculates the cumulative deviation of each modal sensing data from the corresponding baseline, including: Each modal parameter is extracted from each modal sensing data, and the instantaneous deviation of each modal parameter is calculated. The instantaneous deviation is used to characterize the degree of deviation of the value of the modal parameter from the corresponding baseline. The instantaneous deviations of each modal parameter are weighted and summed, and the weighted summation results are accumulated over a preset time period to obtain the accumulated deviation. If the cumulative deviation meets the anomaly triggering condition, then a multimodal data segment associated with the anomaly event is extracted from the multimodal sensing data, including: When the cumulative deviation rises to a first threshold within a preset first time window, it is determined to be a first abnormal trigger type, and the corresponding first interception window width is determined according to the first abnormal trigger type. When the cumulative deviation accumulates to a second threshold within a preset second duration window, it is determined to be a second abnormal trigger type, and the corresponding second interception window width is determined according to the second abnormal trigger type. The second duration window is longer than the first duration window, and the first interception window width is smaller than the second interception window width. Using the trigger time as the endpoint, the multimodal data segments are extracted from the multimodal sensing data in reverse order according to the determined extraction window width.

4. The method according to claim 1, characterized in that, The process of performing causal analysis on each modality data segment to generate an initial causal subgraph for each modality, performing conflict detection on the initial causal subgraphs of different modalities, and triggering manual review when a conflict is detected includes: Extract each modal feature from each modal data segment; Causal analysis is performed on each modal feature to calculate the causal direction and causal strength between any two features within each modality, and an initial causal subgraph for each modality is generated. The initial causal subgraph contains each feature node within the modality and the directed edges between feature nodes. The initial causal subgraphs of different modalities are merged into a global candidate causal graph, and the directional consistency of directed edges from different modalities between the same pair of feature nodes in the global candidate causal graph is compared. When different modalities give opposite causal directions for the same pair of feature nodes, it is determined to be a bidirectional edge conflict; When directed edges of different modalities form a circular dependency structure, it is determined to be a circular dependency conflict; The multimodal data segments corresponding to the initial causal subgraphs with bidirectional edge conflicts or circular dependency conflicts are marked as conflict samples, and the directed edges that have conflicts are highlighted in a visual manner before manual review is triggered.

5. The method according to claim 1, characterized in that, The process of obtaining manually input modification information of the initial causal subgraph and confirmation information of causal edge confidence to form a multimodal causal graph with causal labels and confidence labels includes: Display the data segments of each modality and their corresponding initial causal subgraphs; In response to manual operation, an editing instruction is obtained for the target directed edge in the initial causal subgraph, the editing instruction including: confirm retention, delete, reverse direction, or add a new directed edge in the initial causal subgraph; Obtain the confidence level information of each causal edge input by the user, wherein the confidence level information includes three levels: high confidence, medium confidence, and low confidence; In response to manual operation, type labeling information of each node in the initial causal subgraph is obtained. The node types include behavioral precursor nodes, emotional drift nodes, and vital sign deviation nodes. The initial causal subgraph is updated according to the editing instructions, the confidence level information, and the type labeling information to generate a multimodal causal graph with node type labels, causal edge direction labels, and causal edge confidence labels.

6. The method according to claim 1, characterized in that, The step involves using the multimodal causal graph as a constraint to perform intervention simulations on the multimodal data segments, generating corresponding counterfactual samples, and pairing the multimodal data segments with the counterfactual samples to obtain causal comparison training samples, including: Obtain the target time window data corresponding to the target causal edge in the multimodal data segment; The target time window data is encoded into the latent space to obtain latent variables; Based on the multimodal causal graph, determine the latent variable dimension corresponding to the causal node of the target causal edge in the latent space; The latent variables are modified in a targeted manner along the dimensions of the latent variables to generate modified latent variables; The modified latent variables are decoded into counterfactual multimodal data fragments, in which the state of the cause node is changed while the state of other nodes remains unchanged; The multimodal data segments are paired with the counterfactual multimodal data segments to obtain causal comparison training samples.

7. The method according to claim 1, characterized in that, The acquisition of multimodal perception data of the target object over a continuous time period includes: Multimodal perception data is collected by a heterogeneous sensor array deployed in a nursing setting. The heterogeneous sensor array includes at least a visual sensor, a radar sensor, an infrared temperature sensor, and an acoustic sensor. A timestamp is added to the data frames collected by each sensor using a unified time reference, and the data frames of each sensor are aligned at the frame level according to the timestamps to obtain the time-synchronized multimodal sensing data.

8. The method according to claim 1, characterized in that, The method further includes: The aforementioned early warning dataset was used as training data to train a lightweight early warning model. The lightweight early warning model is used to predict causal chains in unlabeled multimodal sensing data and output the predicted causal chains. The predicted causal chain is compared with the multimodal causal graph of the corresponding sample. When the prediction confidence of the target causal edge in the predicted causal chain is greater than or equal to the confidence threshold and the target causal edge does not exist in the multimodal causal graph, or the prediction direction of the target causal edge in the predicted causal chain is opposite to the labeling direction of the corresponding causal edge in the multimodal causal graph, the multimodal data segment corresponding to the target causal edge is determined as a high uncertainty sample. The high-uncertainty samples are submitted for review, and the correction information after review is obtained. The early warning dataset is then updated based on the correction information.

9. A system for constructing an early warning dataset for embodied intelligent nursing scenarios, characterized in that, include: The synchronous acquisition module is used to acquire multimodal sensing data of the target object over a continuous time period; An adaptive cropping module is used to calculate the cumulative deviation of each modal sensing data from the corresponding baseline based on the personalized multimodal baseline of the target object. And, when the cumulative deviation meets the abnormal triggering condition, extract a multimodal data segment associated with the abnormal event from the multimodal sensing data; The human-computer collaborative annotation module is used to perform causal analysis on each modality data segment and generate an initial causal subgraph for each modality. Conflict detection is performed on the initial causal subgraphs of different modalities, and manual review is triggered when a conflict is detected; In addition, it is used to obtain manually input modification information of the initial causal subgraph and confirmation information of causal edge confidence, forming a multimodal causal graph with causal labeling and confidence labeling; The counterfactual sample generation module is used to use the multimodal causal graph as a constraint to perform intervention simulation on the multimodal data segment, generate corresponding counterfactual samples, and pair the multimodal data segment with the counterfactual samples to obtain causal comparison training samples. The sample output module is used to aggregate multiple sets of causal comparison training samples to form an early warning dataset.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the method for constructing an early warning dataset for embodied intelligent care scenarios as described in any one of claims 1 to 8.

Images