Multi-modal data fusion-based accurate monitoring method for tumble of old people

Abstract

The invention discloses a multi-modal data fusion-based accurate monitoring method for tumble of old people. The method comprises the following steps: deploying a fixed-point camera to collect continuous frame image streams, and collecting acceleration and angular velocity signals based on a wearable IMU (Inertial Measurement Unit) device; collecting parameter data of heart rate (HR) and electrodermal response (EDA); performing unified timestamp synchronization on all the modal data, and constructing Xinput (t); then respectively carrying out feature analysis on the acquired image data, inertial data and physiological data, and carrying out weighted fusion based on multi-modal features; according to the method, a multi-modal input sequence is constructed by fusing camera images, inertial data and physiological parameter data, and precise recognition of the current activity state of the old people is realized by combining a Bi-LSTM network with front and back time sequence behavior characteristics; and when a fall candidate or a confirmation state is detected, a consistency score value Scorefall (t) is further calculated based on each modal confidence coefficient, so that the accuracy of fall identification can be improved, and the reliability of fall identification can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of monitoring methods, and specifically to a method for accurately monitoring falls of the elderly based on multimodal data fusion. Background Art

[0002] With the accelerated development of an aging society, the personal safety of elderly people living alone has attracted attention. In particular, with the increasing aging of the population, falls have become one of the main risk factors threatening their life safety and quality of life. Falls can lead to serious consequences such as fractures and brain injuries, and can also cause psychological fear and reduced mobility, which in turn affects the elderly's ability to live independently. Falls have become one of the main causes of abnormal death and disability among the elderly. Studies have found that more than 80% of accidental injuries to the elderly are related to falls. Therefore, the development of monitoring systems that can accurately identify falls in the elderly in real time has become an important research direction in the fields of smart elderly care and safety care.

[0003] Existing fall detection technologies use a single modality for behavior recognition, relying solely on camera image analysis or accelerometer judgment. This can lead to inaccurate and timely identification of falls when the image is obstructed, the ambient light changes, or the wearable device is worn improperly. This leads to high false alarm and missed alarm rates, making it difficult to meet the dual requirements of accuracy and stability in actual homes or nursing homes. Furthermore, the technology lacks a continuous analysis and confidence assessment mechanism for fall status.

[0004] In view of the above technical defects, a solution is now proposed. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the present invention provides a method for accurately monitoring falls of the elderly based on multimodal data fusion.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for accurately monitoring falls of the elderly based on multimodal data fusion, a method for accurately monitoring falls of the elderly based on multimodal data fusion, characterized in that it includes the following steps:

[0007] S1. Deploy fixed-point cameras to collect continuous frame image streams. Use wearable IMU devices to collect acceleration and angular velocity signals; collect heart rate (HR) and galvanic skin response (EDA) parameter data.

[0008] S2. Synchronize all modal data with unified timestamps to construct X_input(t). Then, perform feature analysis on the collected image data, inertial data, and physiological data, perform weighted fusion based on multimodal features, and output the fused feature vector F_fusion(t).

[0009] S3. Identify the current state of the elderly based on the fusion feature vector, and use the Bi-LSTM network to classify F_fusion(t). The identification results include safety, fall post-selection, and fall;

[0010] S4. Based on the confidence output results of each modal feature, a consistency scoring function Score_fall(t) is constructed. When Score_fall(t) is greater than or equal to the set threshold θ_fall, a fall confirmation signal Fall_confirm(t) is output. When Score_fall(t) is lower than the threshold, a delayed secondary recognition is performed. If the user's vital signs recover after recognition or the IMU detects standing up, the confirmation signal is canceled.

[0011] S5. After the fall is confirmed, the alarm mechanism is triggered and the data is pushed to the family terminal and the nursing APP.

[0012] The step 1 includes installing a wide-angle camera in the elderly's activity area, the camera collecting continuous image frame stream data, and connecting the wearable IMU device to the main control unit via Bluetooth; the IMU device collects three-axis acceleration values ​​Accel(t) = [Ax, Ay, Az] and three-axis angular velocity values ​​Gyro(t) = [Gx, Gy, Gz] in real time; and downsampling to 1Hz through sliding average, and using a physiological monitoring instrument to collect heart rate HR and skin electrodermal response EDA physiological data.

[0013] The second step is to use the visual modality Video_frame as the main clock every second, and align the other modal data by linear interpolation; construct the standard input vector X_input(t):

[0014] X_input(t)=[Video_frame(t),IMU_vector(t),Bio_vector(t)]

[0015] Among them, Video_frame(t) is the image frame stream data; IMU_vector(t) is the combination of the three-axis acceleration value and the three-axis angular velocity value collected by the IMU device; Bio_vector(t) is the physiological data collected by the physiological monitoring instrument.

[0016] The synchronized X_input(t) data enters feature extraction;

[0017] For the video_frame(t) image frame, a 3D-CNN structure is used to extract spatiotemporal motion features. A posture time series vector P(t:t+k) is constructed to encode the body tilt angle, falling speed, and balance axis offset features. The video feature vector F_video(t) is output.

[0018] Construct a sliding window IMU_seq(t-k+1→t) of length k for the IMU_vector(t)=[Ax,Ay,Az,Gx,Gy,Gz] data; identify motion impact peak, posture flip, and direction switching features through the GRU structure; and output the inertial feature vector: F_IMU(t).

[0019] Extract the HR change rate (ΔHR / HRV), EDA slope, and decrease amplitude from the data Bio_vector(t) = [HR, HRV, EDA, ΔEDA]; use a lightweight fully connected network to extract key signal combination features; and output the physiological feature vector F_bio(t);

[0020] Finally, the above three types of feature vectors are spliced ​​together to output the fused feature vector F_fusion(t)=TransformerFusion(F_video(t),F_IMU(t),F_bio(t)).

[0021] The step three specifically uses a bidirectional long short-term memory network Bi-LSTM to capture the impact, acceleration, displacement before the fall and the static features after the fall; the model inputs the fusion feature vector F_fusion(t), and the model outputs the category label set Y(t)∈{1: safe, 2: fall candidate, 3: confirmed fall};

[0022] The model judgment strategy is to meet two of the following multimodal trigger conditions, marking the case where the model output is Y(t) = 2 as a "fall candidate state". The trigger conditions are: the video posture angle change Δθ> 60°, the instantaneous trunk flip, the physiological HR mutation ± 20bpm, or the EDA slope is greater than the set threshold, indicating a stress response after a fall;

[0023] If two of the following multimodal trigger conditions are met, the model outputs a formal fall (Y(t) = 3). The trigger conditions are: the posture remains horizontal for more than 3 seconds; the IMU shows almost no change, indicating stillness; and the EDA remains high.

[0024] If two of the following multimodal trigger conditions are met, the model output is safe Y(t) = 1. The trigger conditions are: the IMU detects standing / sitting movement; the HR returns to the normal range, the EDA drops, and the posture returns to the standing state.

[0025] Calculate the fall judgment credibility score for each type of modal data:

[0026] Quantify the intensity of posture changes in the image; if a large torso drop or posture twist is detected, a higher video modality confidence Conf_video(t) is assigned;

[0027] Based on the acceleration peak, angular velocity variation range, and stable stationary time; if the impact peak is detected to be greater than 3g or the roll angular velocity is greater than 200° / s, and is accompanied by a stationary state, the IMU modal confidence Conf_IMU(t) increases;

[0028] Determine whether the heart rate drops / rises suddenly, whether the EDA slope changes suddenly, or whether it falls below the safety line, and assign the physiological modal confidence Conf_bio(t).

[0029] The above modal confidence factors are weighted and combined to form the overall judgment score Score_fall(t). The calculation formula is as follows:

[0030] Score_fall(t)=w1×Conf_video(t)+w2×Conf_IMU(t)+w3×Conf_bio(t), where w1, w2, and w3 are modal weight parameters. The output range of Score_fall(t) is between 0 and 1, indicating the overall credibility. When the value exceeds the set threshold θ_fall, the official fall confirmation signal Fall_confirm(t)=1 is output.

[0031] If the fall is confirmed (Fall_confirm(t) = 1), but any of the following conditions are observed later, it is considered a false alarm: the IMU detects that the body turns from horizontal to vertical; the heart rate and EDA physiological data return to the normal range; the video posture is re-identified as normal activity state walking and standing upright; the fall mark is revoked and Fall_confirm(t) = 0 is output.

[0032] At the current moment t, all of the following conditions are met: Fall_confirm(t) = 1 has been output; the state has lasted for ≥ 3 seconds; the modal confidence consistency score Score_fall(t) ≥ θ_fall; the alarm mechanism is activated and the alarm information is pushed to the bound family mobile phone APP and the nursing system terminal.

[0033] This invention provides a method for accurately monitoring elderly falls based on multimodal data fusion. Compared with existing technologies, it has the following advantages:

[0034] The present invention constructs a multimodal input sequence by fusing camera images, inertial data, and physiological parameter data. It then uses a Bi-LSTM network to combine the preceding and following temporal behavior features to accurately identify the current activity state of the elderly. When a fall candidate or confirmation state is detected, a consistency score value (Score_fall(t)) is further calculated based on the confidence level of each modality, which can improve the accuracy and reliability of fall recognition.

[0035] After identifying a fall, the present invention triggers an alarm mechanism when the modal confidence score Score_fall(t) is greater than a preset threshold θ_fall, pushing the alarm to family members and nursing terminals. It also has the functions of false alarm cancellation and stand-up recovery judgment, which can avoid false alarms and improve the response efficiency and timeliness of rescue after the elderly fall.

[0036] The present invention calculates the credibility scores Conf_video(t), Conf_IMU(t), and Conf_bio(t) for the three types of modal features, calculates the consistency score Score_fall(t) according to the set weights, and sets the fall confirmation threshold θ_fall. This allows the consistency analysis mechanism between modalities to effectively avoid interference from extreme outliers, thereby enhancing the system's comprehensive judgment ability for falls in complex states.

[0037] The present invention automatically triggers the alarm mechanism after meeting the modal score and state persistence conditions, and pushes a structured data packet containing a timestamp, video snapshot, IMU trajectory map and physiological signal map to family members and nursing terminals, and sets up misjudgment cancellation logic for later state reversal identification. The alarm and closed-loop response mechanism can effectively improve event processing efficiency, thereby ensuring the safety of the elderly and improving the response decision-making quality of guardians. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the principle framework of the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] See also Figure 1 This application provides a method for accurately monitoring falls of the elderly based on multimodal data fusion, comprising the following steps:

[0041] S1. Deploy fixed-point cameras to collect continuous frame image streams. Use wearable IMU devices to collect acceleration and angular velocity signals; collect heart rate (HR) and galvanic skin response (EDA) parameter data.

[0042] S2. Synchronize all modal data with unified timestamps to construct X_input(t). Then, perform feature analysis on the collected image data, inertial data, and physiological data, perform weighted fusion based on multimodal features, and output the fused feature vector F_fusion(t).

[0043] S3, based on the fusion feature vector, the current state of the elderly is identified, and the Bi-LSTM network is used to classify F_fusion(t). The identification results include safety, post-fall selection, and fall;

[0044] S4. Based on the confidence output results of each modal feature, a consistency scoring function Score_fall(t) is constructed. When Score_fall(t) is greater than or equal to the set threshold θ_fall, a fall confirmation signal Fall_confirm(t) is output. When Score_fall(t) is lower than the threshold, a delayed secondary recognition is performed. If the user's vital signs recover after recognition or the IMU detects standing up, the confirmation signal is canceled.

[0045] S5. After the fall is confirmed, the alarm mechanism is triggered and the data is pushed to the family terminal and the nursing APP.

[0046] Step 1 includes installing a wide-angle camera in the elderly's activity area to collect continuous image frame stream data. The camera is connected to the wearable IMU device via Bluetooth to the main control unit. The IMU device collects three-axis acceleration values ​​Accel(t) = [Ax, Ay, Az] and three-axis angular velocity values ​​Gyro(t) = [Gx, Gy, Gz] in real time; and uses sliding average downsampling to 1Hz, and uses physiological monitoring instruments to collect heart rate HR and skin electrodermal response (EDA) physiological data.

[0047] Install fixed wide-angle cameras in the elderly's activity areas (bedrooms, living rooms, and bathrooms) to collect continuous image frame stream data within the coverage area. The frame rate can be set to 30fps. This image stream contains the elderly's real-time behavioral characteristics, environmental information, and background dynamics, providing input for human posture estimation and action discrimination in the visual modality.

[0048] The elderly person needs to wear an IMU device that supports Bluetooth communication (a smart device worn on the wrist or waist). This module collects three-axis acceleration data Accel(t) = [Ax(t), Ay(t), Az(t)] and three-axis angular velocity data Gyro(t) = [Gx(t), Gy(t), Gz(t)] in real time. The device transmits data via the Bluetooth BLE low-power protocol.

[0049] Step 2 is to use the visual modality Video_frame as the main clock every second, and align the other modal data by linear interpolation; construct the standard input vector X_input(t):

[0050] X_input(t)=[Video_frame(t),IMU_vector(t),Bio_vector(t)]

[0051] Among them, Video_frame(t) is the image frame stream data; IMU_vector(t) is the combination of the three-axis acceleration value and the three-axis angular velocity value collected by the IMU device; Bio_vector(t) is the physiological data collected by the physiological monitoring instrument.

[0052] The synchronized X_input(t) data enters feature extraction;

[0053] For the video_frame(t) image frame, a 3D-CNN structure is used to extract spatiotemporal motion features. A posture time series vector P(t:t+k) is constructed to encode the body tilt angle, falling speed, and balance axis offset features. The video feature vector F_video(t) is output.

[0054] For the video_frame(t) image sequence data, a 3D convolutional neural network (3D-CNN) structure is used to extract cross-frame spatiotemporal features. The specific process is as follows: a sliding time window (k=16 frames) is constructed and the video segment Video_seq(t-k+1→t) is input; the 3D-CNN model captures the action posture, speed change, limb tilt angle and other information in the image; combined with the human body key point estimation, the video feature vector F_video(t) is output, which includes: body tilt angle θ, fall acceleration estimate a_fall, and balance axis offset amplitude d_balance.

[0055] Construct a sliding window IMU_seq(t-k+1→t) of length k for the IMU_vector(t)=[Ax,Ay,Az,Gx,Gy,Gz] data; identify motion impact peak, posture flip, and direction switching features through the GRU structure; and output the inertial feature vector: F_IMU(t).

[0056] Extract the HR change rate (ΔHR / HRV), EDA slope, and decrease amplitude from the data Bio_vector(t) = [HR, HRV, EDA, ΔEDA]; use a lightweight fully connected network to extract key signal combination features; and output the physiological feature vector F_bio(t);

[0057] Multilayer perceptron (MLP) has good performance in extracting linear and nonlinear combination features. In this system, the input Bio_vector(t) = [HR, HRV, EDA, ΔEDA] is used to identify stress reactions after a fall, such as sudden changes in heart rate, increased skin conductance, and decreased HRV, and output the physiological stress state encoding feature F_bio(t).

[0058] Finally, the above three types of feature vectors are spliced ​​together to output the fused feature vector F_fusion(t)=TransformerFusion(F_video(t),F_IMU(t),F_bio(t)).

[0059] Step 3 specifically uses a bidirectional long short-term memory network (Bi-LSTM) to capture the impact, acceleration, displacement before the fall, and the static features after the fall. The model inputs the fusion feature vector F_fusion(t), and the model outputs the category label set Y(t)∈{1: safe, 2: fall candidate, 3: confirmed fall}.

[0060] The model judgment strategy is to meet two of the following multimodal trigger conditions, marking the case where the model output is Y(t) = 2 as a "fall candidate state". The trigger conditions are: the video posture angle change Δθ> 60°, the instantaneous trunk roll physiological HR sudden change ± 20bpm, or the EDA slope greater than the set threshold stress response after the fall;

[0061] If two of the following multimodal trigger conditions are met, the model output is a fall Y(t) = 3. The trigger conditions are: the posture remains horizontal for more than 3 seconds; the IMU has almost no change, indicating stillness; the EDA remains high;

[0062] If two of the following multimodal trigger conditions are met, the model output is safe Y(t) = 1. The trigger conditions are: the IMU detects standing / sitting movement; the HR returns to the normal range, the EDA drops, and the posture returns to the standing state.

[0063] The system does not immediately alarm when it detects a "fall state". If it judges it as a "suspected fall" for several consecutive seconds, and it is found that the elderly person has not moved, or the heartbeat has not recovered, and the skin conductance value is still high, it is confirmed to be a "fall".

[0064] Calculate the fall judgment credibility score for each type of modal data:

[0065] Quantify the intensity of posture changes in the image; if a large torso drop or posture twist is detected, a higher video modality confidence Conf_video(t) is assigned;

[0066] Based on the acceleration peak, angular velocity variation range, and stable stationary time; if the impact peak is detected to be greater than 3g or the roll angular velocity is greater than 200° / s, and is accompanied by a stationary state, the IMU modal confidence Conf_IMU(t) increases;

[0067] Determine whether the heart rate drops / rises suddenly, whether the EDA slope changes suddenly, or whether it falls below the safety line, and assign the physiological modal confidence Conf_bio(t).

[0068] The above modal confidence factors are weighted and combined to form the overall judgment score Score_fall(t). The calculation formula is as follows:

[0069] Score_fall(t)=w1×Conf_video(t)+w2×Conf_IMU(t)+w3×Conf_bio(t), where w1, w2, and w3 are modal weight parameters. The output range of Score_fall(t) is between 0 and 1, indicating the overall credibility. When the value exceeds the set threshold θ_fall, the official fall confirmation signal Fall_confirm(t)=1 is output.

[0070] The present invention calculates the credibility scores Conf_video(t), Conf_IMU(t), and Conf_bio(t) for the three types of modal features respectively, then calculates the consistency score Score_fall(t) according to the set weights, and sets the fall confirmation threshold θ_fall. This allows the consistency analysis mechanism between modalities to effectively avoid interference from extreme outliers, thereby enhancing the system's comprehensive judgment ability for fall events in complex states.

[0071] By analyzing the key posture change parameters in the current frame and continuous time windows, including the torso tilt angle change Δθ, the falling velocity v_drop, and the posture flip angle α_flip, the action risk intensity in the video mode is calculated.

[0072] The physiological signal mutation detection package calculates the difference between the current HR and the average heart rate over the past 30 seconds. If ΔHR ≥ 20 bpm, it is determined to be intense exercise or a stress response. EDA slope detection uses the first-order derivative ΔEDA(t) / Δt. If the slope is greater than a set threshold (e.g., 0.05 μS / s), it indicates physiological stress or a fall. The recovery judgment mechanism includes HR returning to [60-100] bpm; EDA returning to a steady-state fluctuation range (e.g., <0.01 μS / s); and the video posture switching from horizontal to standing.

[0073] If any of the following conditions are met: the trunk inclination angle changes Δθ>60°, the posture switches from vertical to horizontal, or a free fall feature is detected (e.g., the falling speed increases), then set Conf_video(t) to high confidence (e.g., 0.7 to 0.95).

[0074] Specifically, the acceleration and angular velocity characteristics within the sliding window are used to determine the impact intensity and the post-fall immobility. If any of the following conditions are met: the impact peak a_peak > 3g, the angular velocity peak ω_peak > 200° / s, or the IMU remains immobile for a long time after the fall (no significant fluctuation ≥ 2 seconds), then Conf_IMU(t) is set to medium-high confidence (e.g., 0.6-0.9).

[0075] Specifically, the degree of stress response of the autonomic nervous system is evaluated by analyzing the following physiological signal fluctuation characteristics: sudden increase / decrease in heart rate (HR) (≥20bpm), a sharp drop in HRV (autonomic nerve suppression), EDA sudden increase slope ΔEDA / Δt>threshold, EDA or HR falling below the set "physiological safety line". If the above characteristics are detected, it means that the physiological mode has a stress response, and the confidence of the physiological mode is improved.

[0076] Confidence calculation code:

[0077] import numpy as np

[0078] from scipy.special import expit#sigmoid

[0079] def sigmoid(x):

[0080] return expit(x)

[0081] def compute_confidences(video,imu,bio):

[0082] #Video Mode

[0083] delta_theta=video.get('delta_theta',0)

[0084] v_drop=video.get('v_drop',0)

[0085] conf_video=sigmoid(delta_theta / 90+v_drop / 2.0)

[0086] #IMU mode

[0087] a_peak=imu.get('acc_peak',0)

[0088] gyro_peak=imu.get('gyro_peak',0)

[0089] is_static=imu.get('is_static',0)

[0090] conf_imu=sigmoid(a_peak / 5.0+gyro_peak / 300+is_static)

[0091] #Physiological Mode

[0092] delta_hr=abs(bio.get('delta_hr',0))

[0093] delta_eda=bio.get('eda_slope',0)

[0094] hrv_drop=bio.get('hrv_drop',0)

[0095] conf_bio=sigmoid(delta_hr / 30+delta_eda / 0.5+hrv_drop)

[0096] return conf_video,conf_imu,conf_bio

[0097] #Confidence weighted combination

[0098] def compute_score_fall(conf_video,conf_imu,conf_bio,w1=0.4,w2=0.4,w3=0.2):

[0099] score=w1*conf_video+w2*conf_imu+w3*conf_bio

[0100] return score;

[0101] If the fall is confirmed (Fall_confirm(t) = 1), but any of the following conditions are observed later, it is considered a false alarm: the IMU detects that the body turns from horizontal to vertical; the heart rate and EDA physiological data return to the normal range; the video posture is re-identified as normal activity state walking and standing upright; the fall mark is revoked and Fall_confirm(t) = 0 is output.

[0102] At the current moment t, all of the following conditions are met: Fall_confirm(t) = 1 has been output; the state has lasted for ≥ 3 seconds; the modal confidence consistency score Score_fall(t) ≥ θ_fall; the alarm mechanism is activated and the alarm information is pushed to the bound family mobile phone APP and the nursing system terminal.

[0103] Determine whether the alarm code is triggered:

[0104] import time,json,base64

[0105] from datetime import datetime

[0106] #Simulate system status record cache

[0107] fall_state_history = [3,3,3] #Classification results in the last three seconds

[0108] fall_confirm=1

[0109] score_fall=0.86

[0110] theta_fall=0.75

[0111] #Determine whether to trigger an alarm

[0112] def should_trigger_alarm(fall_confirm,fall_state_history,score_fall,

[0113] Push structured data packets, including timestamp t, video screenshots, IMU trajectory maps, and physiological data.

[0114] The alarm mechanism is automatically triggered after the modal score and state persistence conditions are met, and a structured data packet containing timestamps, video snapshots, IMU trajectory maps and physiological signal maps is pushed to family members and nursing terminals. The false positive cancellation logic is set for later state reversal identification. The alarm and closed-loop response mechanism can effectively improve the efficiency of event processing, thereby ensuring the safety of the elderly and improving the response decision-making quality of guardians.

[0115] Furthermore, the present invention constructs a multimodal input sequence by fusing camera images, inertial data and physiological parameter data, and uses a Bi-LSTM network combined with the previous and next temporal behavior characteristics to accurately identify the current activity status of the elderly; when a fall candidate or confirmation state is detected, the consistency score value Score_fall(t) is further calculated based on the confidence of each modality, which can improve the accuracy of fall recognition and improve the reliability of fall recognition.

[0116] Furthermore, after identifying a fall, the present invention triggers an alarm mechanism when the modal confidence score Score_fall(t) is greater than a preset threshold θ_fall, and pushes the alarm to family members and nursing terminals. At the same time, it has the functions of false alarm cancellation and stand-up recovery judgment, which can avoid false alarms and improve the response efficiency and timeliness of rescue after the elderly fall.

[0117] Some of the data in the above formulas are dimensionless and numerically calculated. Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0118] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A method for accurately monitoring elderly falls based on multimodal data fusion, characterized in that: The following steps are involved: S1. Deploy fixed-point cameras to collect continuous frame image streams. Use wearable IMU devices to collect acceleration and angular velocity signals; collect heart rate (HR) and galvanic skin response (EDA) parameter data. S2. Synchronize all modal data with unified timestamps to construct X_input(t). Then, perform feature analysis on the collected image data, inertial data, and physiological data, perform weighted fusion based on multimodal features, and output the fused feature vector F_fusion(t). S3. Identify the current state of the elderly based on the fusion feature vector, and use the Bi-LSTM network to classify F_fusion(t). The identification results include safety, fall post-selection, and fall; S4. Based on the confidence output results of each modal feature, a consistency scoring function Score_fall(t) is constructed. When Score_fall(t) is greater than or equal to the set threshold θ_fall, a fall confirmation signal Fall_confirm(t) is output. When Score_fall(t) is lower than the threshold, a delayed secondary recognition is performed. If the user's vital signs recover after recognition or the IMU detects standing up, the confirmation signal is canceled. S5. After the fall is confirmed, the alarm mechanism is triggered and the data is pushed to the family terminal and the nursing APP.

2. The method for accurately monitoring falls of the elderly based on multimodal data fusion according to claim 1 is characterized in that: The step 1 includes installing a wide-angle camera in the elderly's activity area, the camera collecting continuous image frame stream data, and connecting the wearable IMU device to the main control unit via Bluetooth; the IMU device collects three-axis acceleration values ​​Accel(t) = [Ax, Ay, Az] and three-axis angular velocity values ​​Gyro(t) = [Gx, Gy, Gz] in real time; and downsampling to 1Hz through sliding average, and using a physiological monitoring instrument to collect heart rate HR and skin electrodermal response EDA physiological data.

3. The method for accurately monitoring falls of the elderly based on multimodal data fusion according to claim 1 is characterized in that: The second step is to use the visual modality Video_frame as the main clock every second, and align the other modal data by linear interpolation; construct the standard input vector X_input(t): X_input(t)=[Video_frame(t),IMU_vector(t),Bio_vector(t)] Among them, Video_frame(t) is the image frame stream data; IMU_vector(t) is the combination of the three-axis acceleration value and the three-axis angular velocity value collected by the IMU device; Bio_vector(t) is the physiological data collected by the physiological monitoring instrument.

4. The method for accurately monitoring falls of the elderly based on multimodal data fusion according to claim 1 is characterized in that: The synchronized X_input(t) data enters feature extraction; For the video_frame(t) image frame, a 3D-CNN structure is used to extract spatiotemporal motion features. A posture time series vector P(t:t+k) is constructed to encode the body tilt angle, falling speed, and balance axis offset features. The video feature vector F_video(t) is output. Construct a sliding window IMU_seq(t-k+1→t) of length k for the IMU_vector(t)=[Ax,Ay,Az,Gx,Gy,Gz] data; identify motion impact peak, posture flip, and direction switching features through the GRU structure; and output the inertial feature vector: F_IMU(t).

5. The method for accurately monitoring falls of the elderly based on multimodal data fusion according to claim 1 is characterized in that: Extract the HR change rate (ΔHR / HRV), EDA slope, and decrease amplitude from the Bio_vector(t)=[HR,HRV,EDA,ΔEDA] data; and use a lightweight fully connected network to extract key signal combination features; Output physiological feature vector F_bio(t); Finally, the above three types of feature vectors are spliced ​​together to output the fused feature vector F_fusion(t)=TransformerFusion(F_video(t),F_IMU(t),F_bio(t)).

6. The method for accurately monitoring falls of the elderly based on multimodal data fusion according to claim 1 is characterized in that: The step three specifically uses a bidirectional long short-term memory network Bi-LSTM to capture the impact, acceleration, displacement before the fall and the static features after the fall; the model inputs the fusion feature vector F_fusion(t), and the model outputs the category label set Y(t)∈{1: safe, 2: fall candidate, 3: confirmed fall}; The model's decision strategy is to meet two of the following multimodal trigger conditions, marking the case where the model output is Y(t) = 2 as a "fall candidate state." The trigger conditions are: a change in video posture angle Δθ > 60°, a transient trunk roll, a sudden change in physiological HR ±20 bpm, or an EDA slope greater than a set threshold, indicating a stress response after a fall. If two of the following multimodal trigger conditions are met, the model outputs a formal fall (Y(t) = 3). The trigger conditions are: the posture remains horizontal for more than 3 seconds; the IMU shows almost no change, indicating stillness; and the EDA remains high. If two of the following multimodal trigger conditions are met, the model output is safe Y(t) = 1. The trigger conditions are: the IMU detects standing / sitting movement; the HR returns to the normal range, the EDA drops, and the posture returns to the standing state.

7. The method for accurately monitoring falls of the elderly based on multimodal data fusion according to claim 1 is characterized in that: Calculate the fall judgment credibility score for each type of modal data: Quantify the intensity of posture changes in the image; if a large torso drop or posture twist is detected, a higher video modality confidence Conf_video(t) is assigned; Based on the acceleration peak, angular velocity variation range, and stable stationary time; if the impact peak is detected to be greater than 3g or the roll angular velocity is greater than 200° / s, and is accompanied by a stationary state, the IMU modal confidence Conf_IMU(t) increases; Determine whether the heart rate drops / rises suddenly, whether the EDA slope changes suddenly, or whether it falls below the safety line, and assign the physiological modal confidence Conf_bio(t).

8. The method for accurately monitoring falls of the elderly based on multimodal data fusion according to claim 1 is characterized in that: The above modal confidence factors are weighted and combined to form the overall judgment score Score_fall(t). The calculation formula is as follows: Score_fall(t)=w1×Conf_video(t)+w2×Conf_IMU(t)+w3×Conf_bio(t), where w1, w2, and w3 are modal weight parameters. The output range of Score_fall(t) is between 0 and 1, indicating the overall credibility. When the value exceeds the set threshold θ_fall, the official fall confirmation signal Fall_confirm(t)=1 is output.

9. The method for accurately monitoring falls of the elderly based on multimodal data fusion according to claim 1, characterized in that: If the fall is confirmed (Fall_confirm(t) = 1), but any of the following conditions are observed later, it is considered a false alarm: the IMU detects that the body turns from horizontal to vertical; the heart rate and EDA physiological data return to the normal range; the video posture is re-identified as normal activity state walking and standing upright; the fall mark is revoked and Fall_confirm(t) = 0 is output.

10. The method for accurately monitoring elderly falls based on multimodal data fusion according to claim 1, characterized in that: At the current moment t, all of the following conditions are met: Fall_confirm(t) = 1 has been output; the state has lasted for ≥ 3 seconds; the modal confidence consistency score Score_fall(t) ≥ θ_fall; the alarm mechanism is activated and the alarm information is pushed to the bound family mobile phone APP and the nursing system terminal.

Images