An old person fall monitoring and early warning method and system based on multi-source information fusion

Abstract

The present application relates to the technical field of intelligent monitoring, in particular to an old people fall monitoring and early warning method and system based on multi-source information fusion, the system comprising a visual perception module, a wearable sensing module, a central processing unit and an early warning module. The method is as follows: collecting human posture video stream, human waist three-axis acceleration and three-axis angular velocity, foot pressure, heart rate and blood oxygen physiological data in the monitoring area; respectively analyzing visual posture information, inertial sensing and foot pressure data, and extracting key motion features; comprehensively analyzing the features of the three types of information sources, especially using the MediaPipe framework to efficiently analyze human posture, combining with wearable sensor data, adopting a multi-modal feature fusion strategy based on attention mechanism, improving the accuracy of multi-source information fusion, so as to accurately judge the stability of the current posture of the old people and the fall risk level, and timely trigger the early warning, improving the accuracy and safety of the monitoring.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring technology, specifically to a method and system for monitoring and early warning of falls in the elderly based on multi-source information fusion. Background Technology

[0002] With the accelerating aging of the global population, falls among the elderly have become a major social problem threatening their health and safety. Current fall prevention technologies have several limitations: environmental sensor-based monitoring systems have blind spots; wearable sensor-based detection schemes (such as the intelligent detection method for fall detection disclosed in Chinese patent CN202110542777.0) are prone to false alarms and missed alarms during slow falls or complex posture transitions; early multi-sensor fusion technologies (such as the fall detection method and system based on wearable sensors and video surveillance disclosed in Chinese patent CN114067436A) still have shortcomings in feature fusion efficiency and accuracy; pure computer vision solutions (such as the real-time fall detection method and system based on Mediapipe skeleton points disclosed in Chinese patent CN117075734A) can capture posture with high accuracy but are easily affected by changes in lighting, privacy issues, and viewpoint obstruction. Furthermore, most existing systems only issue alarms after a fall occurs, lacking proactive warning capabilities. Fall prediction methods based on motion analysis (such as the one disclosed in Chinese patent CN120392077A, which describes a fall prediction method, system, early warning device, and medium based on motion analysis) often struggle to efficiently capture target features due to their hierarchical processing of motion patterns and time-series modeling. Therefore, developing a fall prevention system that can integrate multi-source information and provide highly reliable, non-intrusive, real-time risk warnings is crucial. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a method and system for monitoring and warning of falls in the elderly based on multi-source information fusion. It adopts the MediaPipe framework to achieve efficient human posture estimation and improves the accuracy and reliability of multimodal feature fusion through its optimized multi-sensor fusion mechanism, effectively overcoming the limitations of traditional single-sensor detection.

[0004] The technical solution adopted by this invention to solve its technical problem is: A method for monitoring and early warning of falls in the elderly based on multi-source information fusion includes the following steps: Step 1: Collect key points of the human skeleton to form visual skeleton temporal data; collect triaxial acceleration and triaxial angular velocity, plantar pressure, heart rate and blood oxygen physiological data of the human waist to form multi-source wearable motion data, and use Kalman filtering to denoise the wearable motion data; Step 2: Extract features from the preprocessed multi-source data to form multimodal primary features: For visual skeletal temporal data, calculate its spatial-temporal features, such as the head-hip height difference, spinal tilt angle, and the rate of change of distance and angle between joints; for wearable motion data, calculate multi-dimensional temporal feature vectors including synthetic acceleration (SMV) and synthetic angular velocity (SV); for physiological and stress data, extract its trend and distribution features. Step 3: Build a Transformer model to perform deep representation learning on the multimodal primary features. Specifically, design a convolutional embedding layer for the visual skeleton temporal data and a linear embedding layer for the wearable motion data. Map the visual skeleton temporal data and wearable motion data to a high-dimensional feature space to form feature vectors for each modality. Step 4: Employ a cross-modal attention mechanism to perform weighted fusion of modal features from different sensors using the softmax function: calculate the correlation score between each modal feature and the global feature, and normalize the weights using the softmax function to ensure priority for key point features; concatenate the weighted feature vectors of each modality into a unified feature vector, input it into the Graph Convolutional Network (GCN) model, process the fused features using the GCN model, and output the fused global feature vector; Step 5: Input the fused global feature vector into the pre-trained support vector machine classification model to evaluate the state of the monitored object and predict events; Step 6: Set a judgment threshold. If the threshold is exceeded, a fall is judged, and a local alarm is triggered. The alarm information and location data are uploaded to the cloud platform and mobile phone via the WiFi module, providing real-time feedback capability.

[0005] The aforementioned technical solution, by introducing the MediaPipe human posture recognition framework and deeply fusing it with wearable sensor data, not only improves the processing efficiency of visual perception, but also provides a high-quality data foundation for multi-source information fusion due to its accurate posture estimation capability, thereby optimizing the overall accuracy of the fusion algorithm. It also solves the problems of existing fall detection methods based on single sensors, such as monitoring blind spots, susceptibility to environmental interference, low accuracy under complex postures, lack of proactive early warning capabilities, and difficulty in achieving highly reliable, non-intrusive real-time risk warnings. This invention, by fusing visual perception with wearable inertial, pressure, and physiological sensor data, and using an information fusion algorithm, achieves comprehensive analysis of the elderly's postural stability, early prediction of high-risk fall states, and accurate detection of fall events, and can immediately trigger an early warning mechanism, ensuring the safety and health of the elderly.

[0006] Furthermore, in step 1, the specific steps for collecting key points of the human skeleton include: Step 1.1, Video Data Acquisition: Real-time capture of indoor video stream using a monocular RGB camera; Step 1.2, Key Point Extraction: Input the video stream into the MediaPipe model to obtain the two-dimensional coordinates of key points of the human skeleton in real time; Step 1.3, Coordinate Normalization: Normalize the coordinates of key points to eliminate the influence of resolution differences; Step 1.4, Definition of Fall Features: Detect head height and hip height; distance between hip and head; spinal tilt angle.

[0007] Furthermore, the extraction process of the key points of the human skeleton includes: It adopts MediaPipe's lightweight detector-tracker architecture. The detector is used to locate the human body in the first frame, and the tracker updates the key point coordinates in subsequent frames. The extracted key points cover the core parts of the body, including the nose, hip, knee, and ankle. If the tracking confidence is less than 0.7, re-enable detector initialization.

[0008] Furthermore, the logic for determining the fall characteristics: A frame is marked as a candidate frame for falling if it meets the following condition: head height < hip height; A fall candidate frame is triggered when the following condition is met: the Euclidean distance between the hip and the head is greater than 50% of the height. When five consecutive frames are candidate frames for falling, the classification model is triggered to make a decision.

[0009] Furthermore, step 1, which involves collecting physiological information such as triaxial acceleration and triaxial angular velocity signals of the human waist, pressure on the soles of the feet, heart rate, and blood oxygenation, includes: Wearable sensors are integrated into the waist to collect triaxial acceleration and triaxial angular velocity data of the human waist, and simultaneously calculate pitch angle deviation using the quaternion method. Pressure sensors are integrated into the sole of the shoe to collect pressure data from the left and right soles of the feet; Heart rate and blood oxygen sensors acquire heart rate and blood oxygen data.

[0010] Furthermore, in step 1, the modal features include: temporal correlation features of skeletal points output from visual data, motion pattern features output from wearable data, trend features output from physiological data, and gait features output from stress data.

[0011] Furthermore, in step 4, when the inertial sensor synthesized acceleration exceeds the threshold and the visual skeletal hip-knee angle change is greater than the threshold, the corresponding modal weight is increased to 0.6-0.7, the pressure data weight is set to 0.1-0.2, and the physiological data weight is set to 0.1-0.2 to assist in verifying the authenticity of the fall.

[0012] Furthermore, in step 4, the formula for calculating the normalized weights of the softmax function is: ; in, , S i To calculate the raw scores obtained for "correlation between each modal feature and the 'fall detection' task", P i These are the normalized modal weights.

[0013] Furthermore, in step 6, a fall detection threshold is set, which is met when: in, P 惯性 Scoring of inertial modal features; P 视觉 For visual modality feature scores > 0.3; P 生理 Physiological modality feature score > 0.1; P 压力 If the pressure modal characteristic score is >0.1, it is determined to be a fall event; otherwise, the device's built-in buzzer indicates normal activity. ; The penalty factor C for the SVM classifier is 10, and the kernel function parameter g is 0.1.

[0014] This invention also provides another technical solution: A fall monitoring and early warning system for the elderly based on multi-source information fusion, used to implement the fall monitoring and early warning method for the elderly based on multi-source information fusion, the system comprising: The visual perception module includes several cameras deployed in the monitoring area for real-time acquisition of human posture video streams; Wearable sensing modules include various sensors fixed to the elderly's clothing or shoe soles to synchronously acquire parameters such as human motion acceleration, posture angle, heart rate, blood oxygen, and pressure. The various sensors include visual sensors, inertial sensors, physiological sensors, and plantar pressure sensors. Each sensor has an independent data acquisition frequency adjustment function, which can be dynamically adjusted according to the monitoring scenario to ensure the integrity and timeliness of the acquired data. The central processing unit is connected to the visual perception module and the wearable sensing module respectively, and performs inertial data conversion, pressure data conversion, visual data conversion, feature extraction based on self-supervised learning, multimodal feature fusion, decision analysis, etc. The early warning module, connected to the central processing unit, is used to immediately trigger an alarm when a high risk of falling is determined or a fall has occurred, including local and remote early warnings.

[0015] Technical effects of the present invention: Compared to existing technologies, this invention employs the MediaPipe framework for posture recognition, significantly improving the reliability of video analysis. Compared to the traditional OpenPose algorithm, MediaPipe's lightweight detector-tracker architecture achieves real-time processing at 30FPS at 1280×720 resolution, exhibiting strong robustness, especially in scenes with loose clothing and changing lighting conditions. Measured keypoint stability is >96%. Its keypoint skeletal model eliminates device resolution differences through normalized coordinates and, combined with multi-dimensional spatial features such as head-hip height difference and spinal tilt angle, effectively distinguishes falls from similar postures like sitting / squatting. Furthermore, it requires only a single RGB camera for deployment, significantly reducing monitoring hardware costs. This invention's dual-modal fusion of video and sensors overcomes the limitations of single-sensor detection. Video analysis can compensate for missed detections by sensors in static limb states, while the waist sensor, through quaternion-based calculation of pitch angle deviation and SMV / SV dynamic thresholds, accurately captures the impact characteristics of rapid falls. This addresses the pain points of camera obstruction (e.g., furniture blocking the view) or the inability to monitor private areas like bathrooms, improving the accuracy and security of monitoring. Attached Figure Description

[0016] Figure 1 This is a block diagram of the elderly fall monitoring and early warning system based on multi-source information fusion according to the present invention. Figure 2 This is a flowchart of the fall monitoring and early warning method for the elderly based on multi-source information fusion, as described in this invention. Figure 3 This is a skeletal diagram of key human body points for the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0018] Example 1: like Figure 1As shown in the figure, this embodiment relates to a fall monitoring and early warning system for the elderly based on multi-source information fusion, including a visual perception module, a wearable sensing module, a central processing unit, and an early warning module. The modules interact with each other via a high-speed data bus. The visual perception module is deployed in the monitoring area to collect real-time video streams of human posture. The wearable sensing module is fixed to the elderly person's clothing or shoe soles to simultaneously acquire parameters such as human motion acceleration, posture angle, heart rate, blood oxygen, and pressure. The central processing unit is connected to both the visual perception module and the wearable sensing module, and is used to analyze visual information, inertial sensing, physiological sensing, and pressure sensing data, extract key motion features, and fuse these four types of features for comprehensive analysis to determine posture stability and fall risk level. The early warning module is connected to the central processing unit and is used to immediately trigger an alarm when a high fall risk is detected or a fall occurs.

[0019] 1. Visual perception module The visual perception module deploys a 1280×720 resolution RGB camera to capture indoor video streams in real time at a fixed frame rate of 30 FPS to ensure motion continuity. The camera is installed at a height of 1.5 meters with a 15° downward angle, covering the entire range of motion of the human body. The video stream is directly input into the MediaPipe framework. The first frame uses a detector to initialize the human body position, and subsequent frames update key points through a tracker. When the tracking confidence is <0.7, the detector is reinitialized, reducing the computational load and ensuring real-time performance.

[0020] The MediaPipe algorithm was used to extract the coordinates of 33 key points on the human body. Its lightweight Blaze architecture enabled end-to-end acceleration, outputting the 2D coordinates (X, Y) of the 33 key points, covering core areas such as the nose (0), hips (23-24), knees (25-26), and ankles (27-32). The key point skeleton diagram is shown below. Figure 3 As shown, line segments represent joint connections, and points represent joint points. The order of marking key points on the human body is as follows: Table 1. Order of marking key points on the human body .

[0021] To eliminate resolution differences, the coordinates are normalized according to the formula: Where W and H are the image width and height, respectively; after normalization, the coordinates are mapped to the range of [0, 1] to adapt to different camera resolutions.

[0022] The action classification module transforms normalized keypoints into spatial-temporal features, which are then input into a binary classification model: fall / non-fall. The input layer of the action classification module consists of 33 normalized keypoint coordinates, totaling 66 dimensions. Its network structure is a fully connected layer with 256 nodes → Dropout → Softmax classifier. The training dataset for the action classification module includes fall actions such as leaning forward, sideways, and backward, as well as regular actions such as walking, sitting, and squatting, totaling over 10,000 data points. The optimizer used is Adam, and the loss function is cross-entropy. ; Where L is the loss value, and M is the total number of categories (2). c For index variables when c When the value is 1, it is marked as not falling. c When the value is 2, it is marked as a fall. y c For the sample to belong to c The true label, p c To predict the type of the sample in the model c The probability of.

[0023] Calculate head and hip height: Detect the difference between the Y-coordinate of the head and the Y-coordinate of the feet (head height) d head ) d head = y head - y toot Detect the difference between the Y-coordinate of the hip and the Y-coordinate of the foot (hip height) d hip ) d hip = y hip - y toot , y head The Y-coordinate of the head y hip The Y-coordinate of the hip y toot The Y-coordinate of the foot is used. If the head height is less than the hip height, it is marked as a candidate frame for falling.

[0024] Calculate the Euclidean distance between the hip and the head: , x head The X coordinate of the head. x hip The X-coordinate of the hip is used; if the Euclidean distance is greater than 50% of the height, a fall candidate frame is triggered.

[0025] Calculate the spinal tilt angle: Calculate the angle λ based on key points 11-12 to distinguish between bending over and falling; ; in, x neck The X coordinate of the neck. y neck The Y-coordinate of the neck.

[0026] 2. Wearable sensing module The wearable sensing module integrates an MPU6050 six-axis accelerometer and angular velocity sensor, a MAX30102 heart rate and blood oxygen sensor, and an RX-M9 plantar pressure sensor to ensure that the elderly do not experience significant restriction during daily activities. Specifically, the MPU6050 sensor collects triaxial acceleration (Ax, Ay, Az) and triaxial angular velocity (Wx, Wy, Wz) data at a sampling frequency of 100Hz, with sampling ranges set to ±16g (acceleration) and ±2000° / s (angular velocity), respectively, and simultaneously calculates the pitch angle deviation using the quaternion method; the MAX30102 heart rate and blood oxygen sensor collects the elderly's heart rate and blood oxygen saturation data at a sampling frequency of 50Hz, with measurement ranges of 30-250bpm (heart rate) and 70%-100% (blood oxygen), respectively, and enables ambient light suppression function to avoid interference from indoor and outdoor light on measurement accuracy. The RX-M9 pressure sensor samples data from the left and right foot pressure sensors at 100Hz. With the addition of these pressure sensors, gait data can be combined to further analyze balance during high angular velocities and large turning angles. The inertial sensor is fixed to the elderly person's waist via a flexible, breathable strap corresponding to the lumbar spine, close to the body's center of gravity. The strap length is adjustable, and the pressure sensor is located under the insole.

[0027] The Min Max normalization algorithm is used to map all raw data collected by sensors to the [0, 1] interval, eliminating the impact of differences in data magnitude on subsequent model training. The specific normalization formula is as follows: ;in, x The original data, x min , x max These are the sensor or hardware limits corresponding to the data.

[0028] 3. Central Processing Unit The central processing unit performs inertial data conversion, stress data conversion, visual data conversion, feature extraction based on self-supervised learning, multimodal feature fusion, and decision analysis.

[0029] 3.1 Inertial Data Conversion: Based on the triaxial acceleration and angular velocity data collected by the MPU6050, calculate the composite acceleration (SMV) and composite angular velocity (SV): ; ; in, a x , a y , a z The normalized triaxial acceleration data, w x , w y , w z These are the normalized triaxial angular velocity data.

[0030] 3.2 Pressure Data Conversion: Calculate the resultant force for the forefoot and heel of both feet separately. F r , F l This serves as the basis for gait analysis during the experiment.

[0031] 3.3 Visual Data Conversion: 33 key skeletal points of the elderly were extracted using the MediaPipe human pose estimation algorithm, and the absolute coordinates of the skeletal points were converted into relative features. Shoulder-hip distance: Calculate the Euclidean distance between the midpoint of the left and right shoulders and the midpoint of the hip. Knee-ankle angle change rate: The ratio of the difference between the angles of the lines connecting the knee and ankle in adjacent frames to the time interval, reflecting the rate of change in lower limb posture.

[0032] 3.4 Feature Extraction Based on Self-Supervised Learning An autoencoder model based on Transformer is constructed, consisting of an encoder and a decoder. Convolutional embedding layers are designed for temporal data of visual skeletons (with a time window of 10 frames), and linear embedding layers are designed for wearable motion data, mapping the raw data to a high-dimensional feature space. Then, based on a single-modal data reconstruction task (randomly masking 30%-50% of data segments to train the model's recovery ability), the model transitions to a multi-modal joint reconstruction task (simultaneously masking visual skeleton points and inertial data) to accelerate model convergence. A contrastive learning loss function is introduced: feature similarity is calculated and differences are amplified for normal activity data (walking, sitting, bending over) and simulated fall data (forward tilting, backward imbalance, sideways fall) of the same elderly person, enhancing the model's sensitivity to fall behavior features.

[0033] Encoder: A 3-layer convolutional embedding layer is designed for visual skeleton temporal data, mapping 2D skeleton point coordinates to 128-dimensional feature vectors; a 3-layer linear embedding layer is designed for wearable motion data, integrating SMV, SV, and... F r , F lThe time series data is mapped to a 128-dimensional feature vector.

[0034] Decoder: Corresponding to the encoder structure, the visual branch design has 3 deconvolutional layers, and the wearable branch design has 3 linear layers, which are used to recover the masked visual skeletal points, inertial and pressure data, respectively.

[0035] Training parameter configuration: The model uses the Adam optimizer, with an initial learning rate of 1e-4 and a learning rate decay strategy of cosine annealing (decaying to 0.8 times the previous epoch every 10 epochs); the loss function is MSE loss, the training batch size is set to 32, and the total training epochs are set to 100.

[0036] Visual modal reconstruction training: The visual skeleton temporal data is randomly masked with a masking ratio of 30%-50%. The model recovers the hip and knee coordinates of the masked frame by comparing the unmasked skeleton points with the skeleton change patterns in adjacent frames. During training, the training is considered complete when the visual skeleton point recovery error is ≤5 pixels.

[0037] Inertial modal reconstruction training: Wearable motion data is randomly masked, with a masking ratio of 30%-50%. The model recovers the values ​​of the masked sampling points by analyzing the trends of the unmasked SMV and SV data. When the inertial data recovery error is ≤0.05, the training of this stage is considered complete.

[0038] Synchronous masking: After single-modal reconstruction training is completed, the visual skeleton temporal data and wearable motion data are simultaneously masked. Visual branch: Mask the hip and knee bone points in frame 37 of the 10-frame window.

[0039] Inertial branch: Synchronously mask the SMV and SV data for the corresponding time window.

[0040] During the masking process, ensure that the visual frames and inertial sampling points are strictly aligned in time.

[0041] Joint reconstruction training: The model combines the unmasked visual shoulder bone points and inertial three-axis angular velocities to simultaneously recover the coordinates and inertial data of the masked visual bone points.

[0042] Dataset Construction: Collect 500 sets of normal activity data and 500 sets of simulated fall data from the same elderly person. Normal activity data: including walking, sitting, and bending over.

[0043] Simulated fall data includes forward tilting (tilt angle > 80°), backward imbalance (imbalance angle > 70°), and side fall (side tilt angle > 85°).

[0044] Loss function calculation: In the autoencoder model training process, a contrastive learning loss function is introduced: ; Where sim() is the cosine similarity function. f i For the current sample features, f i+ For positive samples of the same category, f j Features for negative samples of different categories. τ The temperature parameter is used to increase the feature similarity difference between "fall data" and "normal activity data" through this loss function, thereby improving the model's feature sensitivity to fall behavior by more than 30%.

[0045] 3.5 Multimodal Feature Fusion Correlation score calculation: With the trained autoencoder portion fixed, preprocessed multimodal data is input into the encoder to obtain feature vectors for each modality (visual data outputs temporal correlation features of skeletal points, wearable data outputs motion pattern features, physiological data outputs trend features, and stress data outputs gait features). The correlation score between each modality feature and the global features is then calculated, and the softmax function is used to normalize the correlation scores for each modality. ; in, , S i To calculate the raw scores obtained for "correlation between each modal feature and the 'fall detection' task", P i These are the normalized modal weights.

[0046] GCN Model Construction: Using the visual human skeleton connected graph as the topology, and inertial motion features, physiological features, and pressure features as node attributes, a GCN model is constructed.

[0047] Feature fusion process: The weighted feature vectors of each modality are concatenated into a unified feature vector, which is then input into the GCN model. The GCN uses graph convolution operations to mine the synergistic correlations between "skeleton posture, movement intensity, physiological response, and gait features," outputting a 256-dimensional fused global feature vector. During feature fusion, a cross-modal attention mechanism is employed. This mechanism not only considers the correlations between different modal features at the same time but also analyzes the temporal series correlations of features at different times, further enhancing the sensitivity of the fused features to changes in the state of the monitored object.

[0048] 3.6 Decision Analysis The global feature vector output by GCN is input into a pre-trained support vector machine classifier, and a fall detection threshold is set: when the "inertial modality feature score" is satisfied...P 惯性 >0.5+ visual modality feature score P 视觉 >0.3+ Physiological modality feature score P 生理 >0.1+ pressure mode feature score P 压力 If the value is greater than 0.1, it is determined as a "fall event"; otherwise, the built-in buzzer of the device determines it as "normal activity". ; ; The penalty factor C for the SVM classifier is 10, and the kernel function parameter g is 0.1.

[0049] This embodiment can update the parameters of the classification or regression model in real time using an incremental learning algorithm based on newly collected sensor data, adapting to the dynamic changes in the monitoring scenario and ensuring decision accuracy. When performing state assessment and event prediction, it combines Bayesian inference methods to quantify the uncertainty of the model output results, providing users with decision confidence information.

[0050] 4. Early warning module Dual-channel data is uploaded to the cloud platform via WiFi / MQTT protocol, and an alarm is triggered if either module detects a fall.

[0051] When the system determines it to be a "fall event", it immediately triggers a two-level warning: Local alert: The alert module has a built-in buzzer (volume ≥80dB) and a red LED warning light (1Hz flashing) to alert the elderly or people nearby; if the "cancel" button is not pressed and held within 15 seconds, a remote alert will be automatically activated. Users can manually cancel false alarms within 10 seconds. Remote alert: The fall information is uploaded to the cloud platform and the mobile phone of the preset caregiver via WiFi module. If the elderly does not press and hold the "cancel alarm" button on the wearable device within 15 seconds, the system will automatically call the emergency contact and play a preset voice message: "The elderly person XX is suspected of falling. Please check on them in time."

[0052] By fusing dual-path data through cloud-based "OR logic" for decision-making, the fall detection recall rate has been increased to 99.2%, and the false alarm rate has been reduced to below 0.5%.

[0053] The local wearable device features a buzzer alarm with a 10-second manual cancellation function, ensuring emergency response while preventing false alarms from interfering with the elderly's activities. The cloud platform simultaneously pushes GPS location to the family's app and pops up reminders for caregivers. Combined with real-time video stream verification, it forms a complete "detection-alarm-verification-rescue" response chain, solving the problem of monitoring delays caused by insufficient caregiver manpower. The system supports monitoring of multiple environmental parameters such as DHT11 temperature and humidity and GY-906 body temperature, and can be expanded for early warning of abnormal health conditions in the elderly, achieving synergistic effects between fall prevention and health management, and deeply adapting to the closed-loop monitoring mechanism of nursing home scenarios.

[0054] Example 2: like Figure 2 As shown, this embodiment relates to a method for monitoring and early warning of falls in the elderly based on multi-source information fusion, implemented based on the system described in Embodiment 1. The method includes: Step 1: Data Acquisition: Collect key points of the human skeleton to form visual skeleton temporal data; collect triaxial acceleration and triaxial angular velocity, plantar pressure, heart rate and blood oxygen physiological data of the human waist to form multi-source wearable motion data, and use Kalman filtering to denoise the wearable motion data; Step 2: Extract features from the preprocessed multi-source data to form multimodal primary features: For visual skeletal temporal data, calculate its spatial-temporal features, such as the head-hip height difference, spinal tilt angle, and the rate of change of distance and angle between joints; for wearable motion data, calculate multi-dimensional temporal feature vectors including synthetic acceleration (SMV) and synthetic angular velocity (SV); for physiological and stress data, extract its trend and distribution features. Step 3: Data mapping and generation of feature vectors for each modality: Build a Transformer model to perform deep representation learning on the primary features of the multimodal system. Specifically, design a convolutional embedding layer for the visual skeleton temporal data and a linear embedding layer for the wearable motion data. Map the visual skeleton temporal data and wearable motion data to a high-dimensional feature space to form feature vectors for each modality. Step 4: Multimodal Feature Fusion: A cross-modal attention mechanism is adopted to perform weighted fusion of modal features from different sensors using the softmax function: the correlation score between each modal feature and the global feature is calculated, and the weights are normalized using the softmax function to ensure priority for key point features; the weighted feature vectors of each modality are concatenated into a unified feature vector, which is then input into the Graph Convolutional Network (GCN) model. The GCN model processes the fused features and outputs the fused global feature vector. Step 5: Classification and State Assessment: Input the fused global feature vector into the pre-trained support vector machine classification model to assess the state of the monitored object and predict events. Step 6: Judgment Mechanism and Early Warning: Set a judgment threshold. If the threshold is exceeded, a fall is judged, a local alarm is triggered, and the alarm information and location data are uploaded to the cloud platform and mobile terminal via WiFi module, with real-time feedback capability.

[0055] The core of this invention lies in achieving accurate, real-time detection and alarm for falls in the elderly through the collaborative work of wearable sensors and a vision module. The sensor component uses a wearable device integrating an MPU6050 six-axis accelerometer / angular velocity sensor, an RX-M9 pressure sensor, and a MAX30102 heart rate / blood oxygen sensor. The inertial sensor is fixed to the elderly person's lumbar spine using a flexible, breathable strap with adjustable length to ensure no significant restriction during daily activities, thereby continuously collecting data on the body's motion acceleration, angular velocity, and physiological indicators. A foot pressure sensor is placed under the insole. The vision module uses a monocular RGB camera as its core, constructing a real-time human motion recognition and fall detection system based on the MediaPipe algorithm. It captures video streams in real-time in indoor environments (such as homes and nursing homes), extracts the coordinates of 33 key skeletal points, and then builds a Transformer architecture autoencoder model. A convolutional embedding layer is designed for the visual skeletal temporal data, and a linear embedding layer is designed for the wearable motion data, mapping the raw data to a high-dimensional feature space. The correlation scores between each modality feature and the global feature are calculated, and the weights are normalized using the Softmax function to ensure priority for key features. Then, a Graph Convolutional Network (GCN) is used to further process the fused features. The processed fused feature vector is input into a pre-trained Support Vector Machine (SVM) classification model, and a judgment threshold is set. If the threshold is exceeded, a fall is detected. If a fall is detected, a local alarm is triggered, and the alarm information and location data are uploaded to the cloud platform and mobile device via a WiFi module. The entire system achieves efficient and low-latency fall detection by seamlessly integrating sensor data and visual analysis results, reducing redundant classification logic, and aiming to improve the reliability and timeliness of safety protection for the elderly.

[0056] The above-described specific embodiments are merely specific examples of the present invention. The patent protection scope of the present invention includes, but is not limited to, the above-described specific embodiments. Any appropriate changes or modifications made by a person skilled in the art that conform to the claims of the present invention should fall within the patent protection scope of the present invention.

Claims

1. A method for monitoring and early warning of falls in the elderly based on multi-source information fusion, characterized in that, Includes the following steps: Step 1: Collect key points of the human skeleton to form visual skeleton temporal data; collect triaxial acceleration and triaxial angular velocity, plantar pressure, heart rate and blood oxygen physiological data of the human waist to form multi-source wearable motion data, and use Kalman filtering to denoise the wearable motion data; Step 2: Extract features from the preprocessed multi-source data to form multimodal primary features: For visual skeleton temporal data, calculate its spatial-temporal features, including head-hip height difference, spinal tilt angle, and rate of change of distance and angle between joints; For wearable motion data, calculate multi-dimensional time-domain feature vectors including synthetic acceleration and synthetic angular velocity; for physiological and stress data, extract their trend and distribution characteristics. Step 3: Build a Transformer model to perform deep representation learning on the multimodal primary features. Specifically, design a convolutional embedding layer for the visual skeleton temporal data and a linear embedding layer for the wearable motion data. Map the visual skeleton temporal data and wearable motion data to a high-dimensional feature space to form feature vectors for each modality. Step 4: Employ a cross-modal attention mechanism to perform weighted fusion of modal features from different sensors using the softmax function: calculate the correlation score between each modal feature and the global feature, and normalize the weights using the softmax function to ensure priority for key point features; concatenate the weighted feature vectors of each modality into a unified feature vector, input it into the Graph Convolutional Network (GCN) model, process the fused features using the GCN model, and output the fused global feature vector; Step 5: Input the fused global feature vector into the pre-trained support vector machine classification model to evaluate the state of the monitored object and predict events; Step 6: Set a judgment threshold. If the judgment threshold is exceeded, a fall is judged, a local alarm is triggered, and the alarm information and location data are uploaded to the cloud platform and mobile phone via the WiFi module.

2. The method for monitoring and early warning of falls in the elderly based on multi-source information fusion according to claim 1, characterized in that, In step 1, the specific steps for collecting key points of the human skeleton include: Step 1.1, Video Data Acquisition: Real-time capture of indoor video stream using a monocular RGB camera; Step 1.2, Key Point Extraction: Input the video stream into the MediaPipe model to obtain the two-dimensional coordinates of key points of the human skeleton in real time; Step 1.3, Coordinate Normalization: Normalize the coordinates of key points; Step 1.4, Definition of Fall Features: Detect head height and hip height; distance between hip and head; spinal tilt angle.

3. The method for monitoring and early warning of falls in the elderly based on multi-source information fusion according to claim 2, characterized in that, The extraction process of key points of the human skeleton includes: It adopts MediaPipe's lightweight detector-tracker architecture. In the first frame, the detector is used to locate the human body, and in subsequent frames, the tracker updates the coordinates of key points. If the tracking confidence is less than 0.7, re-enable detector initialization.

4. The method for monitoring and early warning of falls in the elderly based on multi-source information fusion according to claim 2, characterized in that, The logic for determining the fall characteristics: A frame is marked as a candidate frame for falling if it meets the following condition: head height < hip height; A fall candidate frame is triggered when the following condition is met: the Euclidean distance between the hip and the head is greater than 50% of the height. When five consecutive frames are candidate frames for falling, the classification model is triggered to make a decision.

5. The method for monitoring and early warning of falls in the elderly based on multi-source information fusion according to claim 1, characterized in that, Step 1, which involves collecting triaxial acceleration and triaxial angular velocity signals of the human waist, pressure on the soles of the feet, heart rate, and blood oxygenation physiological information, includes the following steps: Wearable sensors are integrated into the waist to collect triaxial acceleration and triaxial angular velocity data of the human waist, and simultaneously calculate pitch angle deviation using the quaternion method. Pressure sensors are integrated into the sole of the shoe to collect pressure data from the left and right soles of the feet; Heart rate and blood oxygen sensors acquire heart rate and blood oxygen data.

6. The method for monitoring and early warning of falls in the elderly based on multi-source information fusion according to claim 1, characterized in that, In step 1, the modal features include: temporal correlation features of skeletal points output from visual data, motion pattern features output from wearable data, trend features output from physiological data, and gait features output from stress data.

7. The method for monitoring and early warning of falls in the elderly based on multi-source information fusion according to claim 1, characterized in that, In step 4, when the inertial sensor synthesized acceleration exceeds the threshold and the visual skeleton hip-knee angle change is greater than the threshold, the corresponding modal weight is increased to 0.6-0.7, the pressure data weight is set to 0.1-0.2, and the physiological data weight is set to 0.1-0.

2.

8. The method for monitoring and early warning of falls in the elderly based on multi-source information fusion according to claim 1, characterized in that, In step 4, the formula for calculating the normalized weights of the softmax function is as follows: ； in, , S i The raw scores are used to calculate the correlation between each modal feature and the fall detection task. P i These are the normalized modal weights.

9. The method for monitoring and early warning of falls in the elderly based on multi-source information fusion according to claim 1, characterized in that, In step 6, a fall detection threshold is set, which is met when: ； in, P 惯性 Scoring of inertial modal features; P 视觉 For visual modality feature scores > 0.3; P 生理 Physiological modality feature score > 0.1; P 压力 If the pressure modal characteristic score is >0.1, it is determined to be a fall event; otherwise, the device's built-in buzzer indicates normal activity. ； The penalty factor C for the SVM classifier is 10, and the kernel function parameter g is 0.

1.

10. A fall monitoring and early warning system for the elderly based on multi-source information fusion, characterized in that, The system is implemented based on the method described in any one of claims 1-9, and includes: The visual perception module includes several cameras deployed in the monitoring area for real-time acquisition of human posture video streams; Wearable sensing modules include various sensors fixed to the elderly's clothing or shoe soles to synchronously acquire human motion acceleration, posture angle, heart rate, blood oxygen, and pressure; the various sensors include visual sensors, inertial sensors, physiological sensors, and plantar pressure sensors; The central processing unit is connected to the visual perception module and the wearable sensing module respectively, and performs inertial data conversion, pressure data conversion, visual data conversion, feature extraction based on self-supervised learning, multimodal feature fusion, and decision analysis. The early warning module, connected to the central processing unit, is used to immediately trigger an alarm when a high risk of falling is determined or a fall has occurred, including local and remote early warnings.

Images