An intelligent old person fall detection and early warning method and system based on multi-modal perception

Abstract

The application discloses a kind of based on multimodal perception's intelligent old person fall detection and early warning method and system, belong to health care field, method includes: synchronous acquisition motion data and video frame sequence, obtains multimodal original data set by Kalman filtering and network time protocol alignment;Adopt posture estimation network to extract skeletal key point coordinates, carry out feature level splicing to inertia feature and posture feature, obtain fusion feature vector;Using sliding window extracts time-frequency feature, input double-flow convolutional neural network to obtain fall risk probability;Establish individualized Gaussian mixture model baseline, dynamically adjust decision threshold by Mahalanobis distance, and according to environmental illumination intensity adaptive adjustment double-flow network weight coefficient;Finally, adopt three-level grading strategy to determine early warning grade and execute corresponding instruction.The application improves the accuracy and robustness of fall detection through multimodal feature fusion, individualized adaptive calibration and illumination compensation mechanism, and is suitable for home care monitoring scene.

Description

Technical Field

[0001] This invention belongs to the field of smart elderly care and intelligent health monitoring, and in particular relates to a method and system for intelligent fall detection and early warning for the elderly based on multimodal perception. Background Technology

[0002] With the accelerating aging of the global population, falls at home have become a core safety issue that urgently needs to be addressed in the fields of smart elderly care and intelligent health monitoring. Currently, fall detection technologies mainly fall into two categories: wearable sensor-based solutions collect motion data using accelerometers and gyroscopes worn on the waist, and identify fall events using threshold judgment or machine learning methods. These solutions offer advantages such as strong privacy protection and are unaffected by ambient light, but when relying solely on a single inertial data source, they struggle to effectively distinguish falls from sudden, stumbling movements such as sitting down, misstepping, or rapid squatting, leading to a high false alarm rate. Visual sensor-based solutions use indoor cameras to collect key points or contour information of human posture for fall detection, providing a direct view of posture changes. However, they are heavily dependent on ambient lighting conditions, and their detection performance significantly decreases in complex scenarios such as low light, shadows, or human occlusion. To overcome the limitations of single-modal approaches, existing technologies attempt multi-modal fusion solutions, but most solutions remain at the decision-level fusion stage, failing to fully preserve the correlation information between cross-modal features.

[0003] Existing fall detection solutions generally employ a uniform and fixed decision strategy, failing to fully consider the significant individual differences among elderly individuals in terms of exercise habits, physical condition, and activity patterns. This makes it difficult to balance the false alarm rate and false negative rate when the system is deployed across different users. Especially for elderly individuals with strong mobility, the inertial signal characteristics generated by their daily vigorous movements may be similar to the fall height, making it difficult for fixed threshold solutions to ensure detection accuracy for this group. Regarding alarm mechanisms, most systems have a single response method, lacking both the ability to provide differentiated warnings based on risk levels and the establishment of a system-wide self-optimization closed loop based on individual feedback. Furthermore, existing solutions are insufficient in environmental adaptation; they lack effective cross-modal dynamic compensation mechanisms for visual modal performance degradation in complex environments such as low light, making it difficult to meet the robustness requirements of home monitoring scenarios.

[0004] Based on the above reasons, this invention designs an intelligent method and system for detecting and warning of falls in the elderly based on multimodal perception. Summary of the Invention

[0005] The purpose of this invention is to solve the problems in the prior art, and to propose an intelligent fall detection and early warning method and system for the elderly based on multimodal perception.

[0006] This invention first discloses an intelligent fall detection and early warning method for the elderly based on multimodal perception, comprising the following steps: S1: Data acquisition and preprocessing: Motion data of the monitored object is acquired through wearable inertial sensors, video frame sequences are acquired through indoor vision sensors, the inertial data is filtered and denoised, and the dual-modal data is time-aligned to obtain a time-synchronized multimodal raw dataset. S2: Feature extraction and fusion: Based on the video frame sequence, human skeleton key points are detected, posture key point coordinates are extracted, and the inertial feature vector and posture key point coordinate vector are fused at the feature level to obtain a multimodal fusion feature vector. S3: Fall risk identification: The multimodal fusion feature vector is segmented using a sliding window method to extract time-frequency features. The extracted features are then input into a dual-stream neural network model for fall risk classification to obtain a preliminary fall risk probability. S4: Adaptive calibration, using an individualized baseline motion model to adaptively calibrate the current detection results, calculating the Mahalanobis distance between the current feature window and the baseline model, and dynamically adjusting the decision threshold based on the Mahalanobis distance if the Mahalanobis distance exceeds a preset confidence interval; simultaneously acquiring ambient light intensity data and dynamically adjusting the weight coefficients of each feature stream in the dual-stream neural network model based on the ambient light intensity. S5: Early warning decision-making. Based on the dynamically adjusted judgment threshold and weighting coefficient, the adjusted fall risk probability is recalculated and compared with the adjusted judgment threshold to determine the final early warning level and execute the corresponding early warning instructions.

[0007] In the above method, the data acquisition and preprocessing steps include: The inertial measurement unit, fixed to the waist of the monitored object, collects triaxial acceleration data and triaxial angular velocity data at a first sampling frequency to form an inertial data stream. A wide-angle vision sensor fixed to the indoor ceiling is used to acquire a video frame sequence at a second frame rate. The video frame sequence is then preprocessed to obtain a video data stream. The Kalman filter algorithm is used to perform state estimation and noise filtering on the inertial data stream to eliminate sensor drift error and obtain the filtered inertial dataset. Based on the network time protocol, timestamps are added to the inertial data stream and the video data stream. Using the inertial data time axis as a reference, an interpolation algorithm is used to resample and align the video frame features to obtain a time-synchronized multimodal raw dataset.

[0008] In the above method, the feature extraction and fusion steps include: Based on the video data stream, a pose estimation model with a lightweight convolutional neural network architecture is used to perform human detection on each frame of the image, extract the two-dimensional pixel coordinates of a preset number of skeletal key points, and obtain a set of key point coordinates. The filtered inertial feature vector is concatenated with the normalized attitude key point coordinate vector to obtain the multimodal fusion feature vector.

[0009] In the above method, the fall risk identification step includes: The multimodal fusion feature vector is segmented using a sliding window method to obtain a window sequence; Within each time window, extract time-domain statistical features and frequency-domain energy distribution features to construct a time-frequency feature vector sequence; The time-frequency feature vector sequence is input into the first feature stream of the dual-stream neural network model, and the posture key point feature sequence is input into the second feature stream of the dual-stream neural network model. The fall risk probability is calculated by weighted calculation method through the feature fusion layer to obtain the preliminary fall risk probability distribution.

[0010] In the above method, the adaptive calibration step involves using an individualized baseline motion model to adaptively calibrate the current detection result, including: During the system initialization phase, normal activity data of the monitored object are collected within a preset time period. A Gaussian mixture model is used to model the distribution of motion signals, and an individualized baseline motion model is established through parameter estimation. During the real-time detection phase, the Mahalanobis distance between the current feature window and the individualized baseline motion model is calculated. If the Mahalanobis distance exceeds the critical value of a preset confidence interval, the decision threshold is dynamically increased based on the difference between the Mahalanobis distance and the critical value.

[0011] In the above method, the adaptive calibration step, which dynamically adjusts the weight coefficients of each feature stream in the dual-stream neural network model based on the ambient light intensity, includes: The system acquires real-time illuminance data from an indoor illuminance sensor. When the real-time illuminance data is lower than a preset light threshold, it determines that the system is in a low-light scene and performs a weight adjustment operation, increasing the weight of the first feature stream and decreasing the weight of the second feature stream.

[0012] In the above method, the early warning decision-making step includes: The adjusted probability of fall risk is recalculated based on the dynamically adjusted judgment threshold and weighting coefficient. The adjusted fall risk probability is compared with the adjusted judgment threshold, and a graded judgment method is used to determine the warning level, and the corresponding warning instruction is executed.

[0013] In the above method, the method further includes: Repeated early warning requests from the same monitored subject within a preset time window are merged; when the guardian confirms the cancellation of the alarm within a preset waiting time, the event sample is marked as a false alarm and stored in the sample buffer pool for use in optimizing and updating the individualized baseline motion model.

[0014] Secondly, the present invention provides an intelligent fall detection and early warning system for the elderly based on multimodal perception, used in the above method, comprising: The data acquisition module is configured to acquire motion data of the monitored object through a wearable inertial sensor and video frame sequences through an indoor vision sensor. The data preprocessing module is configured to filter and denoise the inertial data and perform time alignment on the dual-modal data to obtain a time-synchronized multimodal raw dataset. The feature extraction and fusion module is configured to detect key points of human skeleton based on the video frame sequence, extract the coordinates of posture key points, and perform feature-level fusion of the inertial feature vector and the posture key point coordinate vector to obtain a multimodal fusion feature vector. The fall risk identification module is configured to use a sliding window method to segment the multimodal fusion feature vector, extract time-frequency features, and input the time-frequency features into a dual-stream neural network model for fall risk classification to obtain a preliminary fall risk probability. The adaptive calibration module is configured to adaptively calibrate the current detection result using an individualized baseline motion model, calculate the Mahalanobis distance between the current feature window and the baseline model, and dynamically adjust the decision threshold based on the Mahalanobis distance; at the same time, it acquires ambient light intensity data and dynamically adjusts the weight coefficients of each feature stream in the dual-stream neural network model based on the ambient light intensity. The early warning decision module is configured to recalculate the adjusted fall risk probability based on the dynamically adjusted judgment threshold and weight coefficient, compare it with the adjusted judgment threshold, determine the final early warning level, and execute the corresponding early warning instruction.

[0015] In the above system, the adaptive calibration module includes: The baseline modeling unit is configured to collect normal activity data of the monitored object within a preset time period during the system initialization phase, and to use a Gaussian mixture model to model the distribution of motion signals to establish an individualized baseline motion model. The deviation measurement unit is configured to calculate the Mahalanobis distance between the current feature window and the individualized baseline motion model during the real-time detection phase. If the Mahalanobis distance exceeds the critical value of the preset confidence interval, the decision threshold is dynamically increased based on the difference between the Mahalanobis distance and the critical value. The light adaptive unit is configured to acquire real-time illuminance data from an indoor illuminance sensor. When the real-time illuminance data is lower than a preset light threshold, it determines that the current scene is a low-light scene and performs a weight adjustment operation, increasing the weight of the first feature stream and decreasing the weight of the second feature stream.

[0016] The beneficial effects of this invention are as follows: 1. This solution simultaneously acquires multimodal data using wearable inertial sensors and visual sensors, and performs data stitching and fusion at the feature layer. Compared to decision-layer fusion, this method preserves cross-modal correlation information between abrupt acceleration changes and rapid drops in center of mass height. Kalman filtering combined with NTP time alignment ensures data quality and temporal consistency, enabling the dual-stream convolutional neural network to simultaneously utilize the high-frequency motion dynamics features of inertial signals and the geometric change features of posture sequences, thereby improving the classification accuracy of falls and everyday violent movements.

[0017] 2. This solution establishes an individualized baseline movement model, using a Gaussian mixture model to characterize the multi-component feature distribution of the monitored individual's normal activities. Mahalanobis distance is used to quantify the deviation of the current movement pattern from the individual's baseline. When the deviation exceeds the confidence interval, the decision threshold is dynamically increased. This mechanism solves the problem that fixed thresholds cannot adapt to the differences in movement habits among different users, effectively reducing the probability of active elderly people's daily vigorous movements being misjudged as falls, while maintaining high sensitivity to real fall events.

[0018] 3. This solution uses ambient light intensity as a continuous physical quantity to establish a quantitative mapping relationship between light intensity and the weight coefficients of the dual-flow network. In low-light scenarios, it automatically increases the inertial flow weight and decreases the visual flow weight to compensate for the degradation of visual modality performance. This adaptive mechanism differs from existing technologies that adjust weights based on qualitative scene types, enabling the system to maintain stable fall detection capabilities under varying lighting conditions. A three-level graded early warning strategy triggers differentiated responses based on risk probability, and a continuous window confirmation mechanism filters out instantaneous misjudgments, reducing the false alarm rate. Attached Figure Description

[0019] Figure 1 This is a flowchart of an intelligent fall detection and early warning method for the elderly based on multimodal perception disclosed in this invention.

[0020] Figure 2 This is an architecture diagram of an intelligent fall detection and early warning method for the elderly based on multimodal perception disclosed in this invention.

[0021] Figure 3 This is a flowchart of the multimodal feature fusion process in an intelligent fall detection and early warning method for the elderly based on multimodal perception disclosed in this invention. Detailed Implementation

[0022] To facilitate understanding of this application and to make the aforementioned objectives, features, and advantages of this application more apparent, a detailed description of specific embodiments of this application is provided below in conjunction with the accompanying drawings. Numerous specific details are set forth in the following description to provide a thorough understanding of this application, and preferred embodiments are shown in the accompanying drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application. This application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified. In the description of this application, "several" means at least one, such as one, two, etc., unless otherwise explicitly specified. It should be noted that when an element is referred to as being "fixed to" another element, it can be directly attached to the other element or there may be an intervening element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible implementations. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is only for describing particular implementations and is not intended to limit the scope of this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0023] Reference Figure 1-3 A smart fall detection and early warning method for the elderly based on multimodal perception is described below: First, data acquisition and preprocessing steps are performed. A six-axis inertial measurement unit, such as the MPU-6050 model, is fixed to the waist of the monitored person to acquire three-axis acceleration data A=[ax,ay,az] and three-axis angular velocity data G=[gx,gy,gz] at a first sampling frequency of 100Hz. The inertial data is transmitted in real time to an edge computing unit, such as the NVIDIA Jetson Nano, via Bluetooth Low Energy protocol.

[0024] Simultaneously, a wide-angle vision sensor, such as a Hikvision DS-2CD series camera, fixed approximately 2.8 meters above the ground on the indoor ceiling, is used to acquire a 1920×1080 resolution video frame sequence V={v1,v2,…,vn} at a second frame rate of 30fps. Lens distortion correction and brightness normalization preprocessing are then performed on the video frame sequence. A Kalman filter algorithm is used for state estimation and noise filtering of the inertial data stream. The state update equation is:

[0025] Where K k For Kalman gain, Z k H represents the observation matrix, and the process noise covariance Q and measurement noise covariance R are preset as diagonal matrices based on sensor characteristics to eliminate sensor drift error. Millisecond-level timestamps are added to both the inertial and video data streams using the Network Time Protocol (NTP). Using the inertial data time axis as a reference, a cubic spline interpolation algorithm is employed to resample and align the video frame features, resulting in a time-synchronized multimodal raw dataset. Subsequently, feature extraction and fusion steps are performed: based on the preprocessed video data stream, a lightweight convolutional neural network architecture pose estimation model, such as MoveNetLightning, is used to perform human detection on each frame, extracting the two-dimensional pixel coordinates of 17 COCO-formatted skeletal keypoints. , j=1,…,K.

[0026] The filtered inertial feature vector I=[ax,ay,az,gx,gy,gz] and the normalized attitude keypoint coordinate vector P=[x1,y1,…,x17,y17] are concatenated using the vector concatenation operation F=Concat(I,P) to obtain a multimodal fusion feature vector with a dimension of 40. Next, the fall risk identification step is performed: the multimodal fusion feature vector is segmented using a sliding window method, with a window length W set to 200 sampling points (corresponding to 2 seconds) and a step size S set to 50 sampling points (corresponding to 0.5 seconds), resulting in a window sequence {F1,F2,…,Fm}. Temporal statistical features are extracted within each time window, including the mean μ, standard deviation σ, maximum value Xmax, minimum value Xmin, and zero-crossing rate ZCR. A fast Fourier transform algorithm is used to perform spectral analysis on the signal within the window, calculating the energy distribution in the 0-2Hz, 2-5Hz, and 5-15Hz frequency bands. Transform the time-domain feature vector With frequency domain energy eigenvectors A 24-dimensional time-frequency feature vector sequence is constructed by concatenating the two streams. This time-frequency feature vector sequence is then input into the first feature stream of a two-stream neural network model, while the pose keypoint feature sequence is input into the second feature stream. Both the first and second feature streams each contain three one-dimensional convolutional layers (kernel size 3, channel numbers 32, 64, and 128 respectively). Each layer is followed by batch normalization and max pooling. After global average pooling, a 128-dimensional feature vector is obtained. The outputs of the two feature streams are concatenated and then passed through a fully connected layer and a Softmax classifier to output three probability distributions: normal, near fall, and fall. A weighted fusion method is used to calculate the initial fall risk probability.

[0027] Where α and β are preset weighting coefficients, α+β=1. Then, the adaptive calibration step is performed: during the system initialization phase, normal activity data of the monitored object are collected for 7 consecutive days, and the motion signal distribution is modeled using a Gaussian mixture model (GMM) with a mixture component K=5. The parameters are estimated using the EM algorithm. , , An individualized baseline motion model is established, and during the real-time detection phase, the Mahalanobis distance between the current feature window and the individualized baseline motion model is calculated.

[0028] Where μ and Σ are the parameters of the nearest neighbor Gaussian component, if the Mahalanobis distance D exceeds the critical value Dc corresponding to the 95% confidence interval, the first preset decision threshold Th1 is dynamically increased to: Th1'=Th1+λ·(D-Dc) Where λ is the adjustment coefficient, for example, 0.05. At the same time, the real-time illuminance data L of the indoor illuminance sensor is obtained. If L is lower than the preset light threshold Lth, it is determined that the current scene is a low light scene, and the weight adjustment is performed: α'=α+Δα, β'=β-Δα, Δα is taken as 0.15 for example, the weight of the first feature flow (inertial flow) is increased and the weight of the second feature flow (visual flow) is decreased.

[0029] Finally, the early warning decision-making step is executed: based on the dynamically adjusted threshold Th1' and weighting coefficients α' and β', the adjusted fall risk probability is recalculated.

[0030] Compare R' with the adjusted decision threshold Th1'. If R'<Th2 (e.g., 0.5), it is determined as low risk, triggering the sound and light prompt of the local wearable device; if Th2≤R'<Th1', it is determined as medium risk, and a push notification is sent to the guardian terminal; if R'≥Th1' and this condition is satisfied for 10 consecutive seconds, it is determined as high risk, and local sound and light alarm, voice call and emergency platform reporting are performed synchronously, so as to determine the final fall detection control instruction. The above method can significantly improve the accuracy and robustness of fall detection and reduce the false alarm rate through multi-modal data fusion, dual-stream neural network classification, individualized baseline calibration and illumination adaptive weight adjustment.

[0031] Further, in the present method, the data acquisition and preprocessing step may be specifically implemented as follows: a six-axis inertial measurement unit fixed to the waist of the monitored subject acquires triaxial acceleration data A=[ax,ay,az] and triaxial angular velocity data G=[gx,gy,gz] at a first preset sampling frequency, e.g., 100Hz, the acceleration measurement range is set to ±16g, the angular velocity measurement range is set to ±2000° / s, the data precision is 16 bits, and the inertial data is transmitted to the edge computing unit via a wireless communication protocol such as Bluetooth Low Energy 5.0. A wide-angle vision sensor fixed to the indoor top acquires a video frame sequence V={v1,v2,…,vn} at a second preset frame rate, e.g., 30fps, with a field of view of 120°, which can cover a monitoring area of about 25 square meters. Lens distortion correction is performed on the video frame sequence, for example, using the Brown-Conrady distortion model, the correction parameters k1 and k2 are obtained according to lens calibration, and histogram equalization is used for brightness normalization preprocessing. A Kalman filter algorithm is used for state estimation and noise filtering of the inertial data stream, and the state update equation is:

[0032] where K k is the Kalman gain, Z k is the observed value, H is the observation matrix, the process noise covariance Q and the measurement noise covariance R are preset according to the noise characteristics of the sensor, for example, Q=diag([0.001,0.001,0.001,0.01,0.01,0.01]), R=diag([0.1,0.1,0.1,0.5,0.5,0.5]), and the initial state covariance P0 is set as an identity matrix, so as to eliminate the sensor drift error. Millisecond-level timestamps are respectively attached to the inertial data stream and the video data stream based on the Network Time Protocol, with the timestamp accuracy better than 5ms. Taking the time axis of the inertial data as the reference, the cubic spline interpolation algorithm is used to resample and align the video frame features, and the interpolation formula is:

[0033] This approach ensures that the time deviation of the synchronized bimodal data is less than 10ms, meeting the time resolution requirements for fall events, thus obtaining a time-synchronized multimodal raw dataset. This implementation effectively suppresses sensor noise through Kalman filtering and guarantees the temporal consistency of the multimodal data through network time protocols and spline interpolation, providing a high-quality data foundation for subsequent fusion.

[0034] In one feasible embodiment, the specific process of feature extraction and fusion is as follows: Based on the preprocessed video data stream, a pose estimation model with a lightweight convolutional neural network architecture, such as MoveNetLightning, is used to perform human detection on each frame of the image. The input resolution is set to 192×192, and the inference speed can reach 50fps. Two-dimensional pixel coordinates of a preset number of K skeletal key points are extracted. Let j = 1, ..., K, where K is 17, including the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. After obtaining the set of key point coordinates, calculate the height of the human body's center of mass:

[0035] in Preset weighting coefficients for each keypoint, for example, hip weight 0.25, knee weight 0.15, ankle weight 0.10. Calculate the centroid height change rate between adjacent frames:

[0036] The kinematic features of the posture are obtained. If the absolute value of the rate of change of the center of mass height exceeds a preset descent rate threshold, such as -0.8 / s, the current frame is marked as a posture abnormal frame. The rate of change of each joint angle θ_ij is further calculated to obtain the posture abnormality feature identifier. The filtered inertial feature vector I=[ax,ay,az,gx,gy,gz] and the normalized posture keypoint coordinate vector P=[x1,y1,…,x17,y17] are concatenated by the vector operation F=Concat(I,P) to obtain the multimodal fusion feature vector with a dimension of 6+34=40. This feature-level fusion method preserves the correlation information between cross modes, such as the cooperative mode of acceleration mutation and rapid descent of the center of mass height, and has a stronger expressive power than decision-level fusion.

[0037] Preferably, the sliding window segmentation method in the fall risk identification step is implemented as follows: Based on the multimodal fusion feature vector, the system is segmented with a window length W = 200 sampling points (corresponding to 2 seconds, sampling rate 100Hz) and a step size S = 50 sampling points (corresponding to 0.5 seconds, window overlap rate 75%) to obtain a window sequence {F1, F2, ..., Fm}, where m is the total number of windows. Within each time window Fi, temporal statistical features are extracted, including the mean μ = ΣX / N, standard deviation σ = √(Σ(X-μ)² / N), maximum value Xmax, minimum value Xmin, and zero-crossing rate ZCR, to obtain a temporal feature vector Ti. The signal within each time window Fi is subjected to spectral analysis using a Fast Fourier Transform algorithm, with the number of transform points... Given a frequency resolution Δf≈0.39Hz, calculate the energy distribution across three frequency components: 0-2Hz (low-frequency motion), 2-5Hz (dominant frequency of human movement), and 5-15Hz (impact frequency). The frequency domain energy eigenvector Ei is obtained.

[0038] If the peak value of the composite acceleration amplitude Amag = √(ax² + ay² + az²) within the time window exceeds a preset impact threshold, such as 24.5 m / s² (approximately 2.5 g), and the absolute value of the rate of change of the center of mass height ΔH exceeds a preset descent threshold, such as 1.5 / s, then the window is marked as a suspected fall window. The time-domain feature vector Ti and the frequency-domain energy feature vector Ei are concatenated to construct a time-frequency feature vector sequence, which is used as input to the subsequent two-stream neural network. By capturing the impact amplitude through time-domain features and distinguishing the frequency patterns of falls from daily activities through frequency-domain features, dual-threshold pre-screening can effectively reduce the computational load on the subsequent neural network.

[0039] Furthermore, the specific implementation of the two-stream neural network processing is as follows: Based on the time-frequency feature vector sequence, the first feature stream of the two-stream convolutional neural network is input. This first feature stream contains three one-dimensional convolutional layers, each with a kernel size of 3 and channel numbers of 32, 64, and 128 respectively. The convolution operation is represented as follows: Y=ReLU(Conv1D(X,W)+b) Each layer is followed by batch normalization and max pooling with a pooling kernel size of 2 to extract temporal motion pattern features of the inertial signal. After global average pooling, the inertial flow feature vector H1 with a dimension of 128 is obtained. Based on the posture keypoint feature sequence, the second feature stream of the dual-stream convolutional neural network is input. This second feature stream contains three one-dimensional convolutional layers with the same architecture, with the same number of channels: 32, 64, and 128. Temporal posture change features are extracted, and after global average pooling, the visual flow feature vector H2 with a dimension of 128 is obtained. The inertial flow feature vector H1 and the visual flow feature vector H2 are concatenated through a feature fusion layer to obtain a 256-dimensional fusion vector. This fusion vector is then passed through two fully connected layers with 128 and 64 nodes, respectively. Each layer is followed by a Dropout layer with a Dropout rate of 0.4.

[0040] Finally, the Softmax classifier outputs the probability distributions for three categories: normal, near fall, and fall. The final fall risk probability is calculated using a weighted calculation method. ,in The fall category probability of the first feature stream. Let α be the fall category probability of the second feature stream, α+β=1, with the default α=0.6 and β=0.4.

[0041] This dual-stream architecture fully leverages the complementary advantages of inertial and visual modalities. The inertial stream offers high temporal resolution and is unaffected by illumination, while the visual stream provides pose geometry information. Weighted fusion balances the contributions of the two feature streams. Model training employs the Adam optimizer with a learning rate of 0.001, a batch size of 32, and a cross-entropy loss function. After 200 epochs of training, the model achieves an accuracy exceeding 96%, with a single inference time of approximately 15ms, meeting the requirements for real-time deployment at edge devices.

[0042] Therefore, based on the preliminary fall risk probability R, the specific implementation of determining the warning level using a three-level grading judgment method is as follows: if R < Th2, where Th2 is the second preset threshold, it is determined as a low risk level, the acousto-optic prompt module of the local wearable device is triggered, and a first-level warning instruction is obtained. If Th2 ≤ R < Th1, where Th1 is the first preset threshold, it is determined as a medium risk level, an alarm message is sent to the preset guardian terminal, the message content includes the current location area and the risk probability value, and a response timer with a preset waiting duration is started, so as to obtain a second-level warning instruction. If R ≥ Th1 and this condition is continuously met within the preset continuous detection time window Tc, and the proportion of windows in which R ≥ Th1 is met exceeds the preset confirmation proportion threshold, it is determined as a high risk level, and local acousto-optic alarm, voice call initiation to the guardian terminal, and reporting of the real-time position coordinates (x, y) of the monitored object and the fall confidence R to the emergency service platform are synchronously executed, so as to obtain a third-level warning instruction. Duplicate alarm requests for the same area within the preset time window are merged, and when the guardian confirms to cancel the alarm within the waiting duration, the event sample is marked as a false positive and stored in the sample buffer pool, so as to generate a hierarchical warning execution scheme. This three-level grading strategy balances false positive control and response timeliness, the continuous window confirmation mechanism can filter instantaneous misjudgments, and the response timer and false positive marking support subsequent model optimization.

[0043] Specifically, the differentiated response to fall risk using the three-level hierarchical warning strategy can be implemented in the following manner: The preset threshold parameters are configured as follows: the second preset threshold Th2 = 0.5, the first preset threshold Th1 = 0.8, the continuous detection time window Tc = 10 seconds, corresponding to 20 sliding windows with a step length of 0.5 seconds, and the guardian response waiting duration T wait = 60 seconds.

[0044] Taking R = 0.804 calculated in the previous step as an example for the judgment process: first, it is judged whether R ≥ Th1, and the condition is satisfied; the system starts the continuous detection confirmation mechanism to continuously monitor the fall risk probability within the subsequent time window of Tc = 10 seconds, and assume that the sequence of R values of each window in the subsequent 10 seconds is: [0.804, 0.812, 0.798, 0.825, 0.789, 0.803, 0.815, 0.791, 0.806, 0.818] The average value R avg = 0.806, in which 8 windows satisfy the condition of R ≥ Th1, accounting for 80%, which exceeds the preset confirmation proportion threshold of 70%. At this time, the system determines it as a high risk level and triggers the third-level warning.

[0045] Wherein, the execution content of the third-level warning can be implemented according to the following contents: (1) The local wearable device plays a warning audio at a volume of 85 dB and flashes a red LED; (2) Initiates a voice call to the guardian terminal such as a preset mobile phone number through the VoIP protocol; (3) Reports a JSON data packet to, for example, the 120 emergency center interface through the HTTPS API: The content of the data packet can be set in the following format: {"user_id":"U20231001","timestamp":"2024-01-15T14:32:18Z","location":{"x":5.2,"y":3.8,"room":"bedroom"},"fall_confidence":0.806,"alert_level":3}.

[0046] If R=0.65, a second-level alert is triggered: push a notification to the guardian terminal through Firebase Cloud Messaging, with the content optionally selected as: "[Fall Risk Reminder] The elderly person is currently in the bedroom, and a moderate fall risk (confidence 65%) is detected, please pay attention." At the same time, a 60-second response timer is started. If the guardian clicks the "Confirm Safety" button, the alarm is canceled and the feature vector and label of the sample are stored in the local SQLite sample buffer pool.

[0047] If R=0.35<Th2, only a first-level alert is triggered: the wearable device plays a prompt tone at low volume and vibrates for 3 seconds to remind the elderly to pay attention to safety.

[0048] In a preferred embodiment, the specific implementation of the adaptive calibration step is as follows: During the system initialization phase, normal activity data of the monitored subject within a preset number of consecutive days, such as 7 days, is collected, including daily actions such as walking, sitting down, standing up, getting in and out of bed, totaling about 600,000 feature window samples. A Gaussian Mixture Model (GMM) is used to model the distribution of 24-dimensional time-frequency feature vectors, the number of mixture components K=5, which is determined by selection via the Bayesian Information Criterion (BIC). Assuming the baseline model parameters for a certain elderly person are: Component 1 (slow walking) π1=0.35, μ1=[feature mean vector], Σ1=[covariance matrix]; Component 2 (fast walking) π2=0.20; Component 3 (sitting posture) π3=0.25; Component 4 (standing) π4=0.15; Component 5 (transition action) π5=0.05 The EM algorithm is used to iteratively optimize the parameters, and the parameter estimation formula is:

[0049] in which is a mixing coefficient, is a mean vector, and is a covariance matrix, an individualized baseline motion model is established, and the model comprises a plurality of components to characterize the different activity types described above. In the real-time detection stage, the Mahalanobis distance between the current feature window and the individualized baseline motion model is calculated:

[0050] in which and are parameters of the Gaussian component closest to the current feature.

[0051] If the Mahalanobis distance D exceeds the critical value Dc of the preset confidence interval, the first preset threshold Th1 is dynamically adjusted upward to Th1'=Th1+λ·(D-Dc), where λ is an adjustment coefficient. The adjustment makes the threshold upward adjustment amplitude positively correlated with the deviation degree, so as to avoid misjudging the normal strenuous exercise of the elderly as a fall. At the same time, the real-time illuminance data L of the indoor illuminance sensor is acquired. If L<Lth, where Lth is a preset illuminance threshold, it is determined that the current scene is a low-light scene, and at this time, the accuracy of visual pose estimation decreases. A weight adjustment formula is adopted:

[0052]

[0053] where Δα is a parameter, then the weight of the first feature stream characterizing the inertial stream is adjusted upward, and the weight of the second feature stream characterizing the visual stream is adjusted downward. The fall risk probability is recalculated according to the dynamically adjusted threshold and weight coefficient:

[0054] and compared with the adjusted threshold Th1' to determine the final fall detection control instruction. The adaptive calibration mechanism solves the problem of false alarms caused by differences in exercise habits among different users through individualized baseline modeling and Mahalanobis distance measurement, and solves the problem of modal reliability changes of the same user under different environmental conditions through illumination-driven weight adjustment. The synergistic effect of the two significantly improves the robustness of the system in different scenes.

[0055] Specifically, assuming that the current feature window corresponds to the morning exercise activity of the elderly, the Mahalanobis distance from the baseline model is calculated:

[0056] in which and is a parameter of the nearest neighbor component, the preset 95% confidence interval critical value Dc=3.5, and the current D=4.2>Dc, indicating that this motion pattern deviates from the normal baseline.

[0057] The threshold is dynamically adjusted, which can be calculated with reference to the following formula: Th1'=Th1+λ·(D-Dc)=0.8+0.05×(4.2-3.5)=0.8+0.035=0.835 The adjustment coefficient λ=0.05, and the upward adjustment amplitude is positively correlated with the deviation degree. This adjustment can prevent the normal strenuous exercise of the elderly person from being misjudged as a fall.

[0058] In the adaptive ambient light adjustment, assuming that the current bedroom illuminance L=35 lux < Lth, and the preset light threshold Lth=50 lux at this time, the system determines that it is in a low-light scene, the accuracy of visual pose estimation decreases, then the weights are adjusted: Δα=0.15, so α'=0.6+0.15=0.75, β'=0.4-0.15=0.25.

[0059] Meanwhile, the fall risk probability is recalculated: assuming the current =0.78, =0.72, then: R'=α'·P1+β'·P2=0.75×0.78+0.25×0.72=0.585+0.18=0.765 Compare with the adjusted threshold: R'=0.765<Th1'=0.835, the risk level is determined to be medium, rather than the high risk level when a fixed threshold is used.

[0060] Finally, the present method may further comprise a false positive sample feedback optimization mechanism: within a preset time window, for example 5 minutes, repeated warning requests for the same monitored object in the same area are merged to avoid alarm storms. When the guardian confirms to cancel the alarm through the terminal within a preset waiting duration, for example 60 seconds, the feature vector of the event sample and the label marked as false positive are stored in the sample buffer pool. When the number of samples in the buffer pool reaches a preset threshold, for example 1000, incremental model training is triggered, and the elastic weight consolidation or experience replay method is adopted to update the parameters of the Gaussian mixture model in the individualized baseline motion model, as well as fine-tune the weights of the dual-stream neural network model. This mechanism enables the system to continuously adapt to changes in the motion pattern of the monitored object, such as the decline of mobility with age, so as to maintain detection accuracy for a long time.

[0061] As is known from common technical knowledge, this invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative and not exhaustive. All modifications within the scope of this invention or its equivalents are included in this invention.

Claims

1. A method for intelligent fall detection and early warning in the elderly based on multimodal perception, characterized in that, The method includes: S1: Data acquisition and preprocessing: Motion data of the monitored object is acquired through wearable inertial sensors, video frame sequences are acquired through indoor vision sensors, the inertial data is filtered and denoised, and the dual-modal data is time-aligned to obtain a time-synchronized multimodal raw dataset. S2: Feature extraction and fusion: Based on the video frame sequence, human skeleton key points are detected, posture key point coordinates are extracted, and the inertial feature vector and posture key point coordinate vector are fused at the feature level to obtain a multimodal fusion feature vector. S3: Fall risk identification: The multimodal fusion feature vector is segmented using a sliding window method to extract time-frequency features. The extracted features are then input into a dual-stream neural network model for fall risk classification to obtain a preliminary fall risk probability. S4: Adaptive calibration, using an individualized baseline motion model to adaptively calibrate the current detection results, calculating the Mahalanobis distance between the current feature window and the baseline model, and dynamically adjusting the decision threshold based on the Mahalanobis distance if the Mahalanobis distance exceeds a preset confidence interval; simultaneously acquiring ambient light intensity data and dynamically adjusting the weight coefficients of each feature stream in the dual-stream neural network model based on the ambient light intensity. S5: Early warning decision-making. Based on the dynamically adjusted judgment threshold and weighting coefficient, the adjusted fall risk probability is recalculated and compared with the adjusted judgment threshold to determine the final early warning level and execute the corresponding early warning instructions.

2. The method according to claim 1, characterized in that, The data acquisition and preprocessing steps include: The inertial measurement unit, fixed to the waist of the monitored object, collects triaxial acceleration data and triaxial angular velocity data at a first sampling frequency to form an inertial data stream. A video frame sequence is acquired at a second frame rate using a wide-angle vision sensor fixed to the indoor ceiling. The video frame sequence is then preprocessed to obtain a video data stream. The Kalman filter algorithm is used to perform state estimation and noise filtering on the inertial data stream to eliminate sensor drift error and obtain the filtered inertial dataset. Based on the network time protocol, timestamps are added to the inertial data stream and the video data stream. Using the inertial data time axis as a reference, an interpolation algorithm is used to resample and align the video frame features to obtain a time-synchronized multimodal raw dataset.

3. The method according to claim 1, characterized in that, The feature extraction and fusion steps include: Based on the video data stream, a pose estimation model with a lightweight convolutional neural network architecture is used to perform human detection on each frame of the image, extract the two-dimensional pixel coordinates of a preset number of skeletal key points, and obtain a set of key point coordinates. The filtered inertial feature vector is concatenated with the normalized attitude key point coordinate vector to obtain the multimodal fusion feature vector.

4. The method according to claim 1, characterized in that, The fall risk identification steps include: The multimodal fusion feature vector is segmented using a sliding window method to obtain a window sequence; Within each time window, extract time-domain statistical features and frequency-domain energy distribution features to construct a time-frequency feature vector sequence; The time-frequency feature vector sequence is input into the first feature stream of the dual-stream neural network model, and the posture key point feature sequence is input into the second feature stream of the dual-stream neural network model. The fall risk probability is calculated by weighted calculation method through the feature fusion layer to obtain the preliminary fall risk probability distribution.

5. The method according to claim 1, characterized in that, The adaptive calibration step involves using an individualized baseline motion model to adaptively calibrate the current detection results, including: During the system initialization phase, normal activity data of the monitored object are collected within a preset time period. A Gaussian mixture model is used to model the distribution of motion signals, and an individualized baseline motion model is established through parameter estimation. During the real-time detection phase, the Mahalanobis distance between the current feature window and the individualized baseline motion model is calculated. If the Mahalanobis distance exceeds the critical value of a preset confidence interval, the decision threshold is dynamically increased based on the difference between the Mahalanobis distance and the critical value.

6. The method according to claim 1 or 5, characterized in that, In the adaptive calibration step, the weight coefficients of each feature stream in the dual-stream neural network model are dynamically adjusted according to the ambient light intensity, including: The system acquires real-time illuminance data from an indoor illuminance sensor. When the real-time illuminance data is lower than a preset light threshold, it determines that the system is in a low-light scene and performs a weight adjustment operation, increasing the weight of the first feature stream and decreasing the weight of the second feature stream.

7. The method according to claim 1, characterized in that, The early warning decision-making steps include: The adjusted probability of fall risk is recalculated based on the dynamically adjusted judgment threshold and weighting coefficient. The adjusted fall risk probability is compared with the adjusted judgment threshold, and a graded judgment method is used to determine the warning level, and the corresponding warning instruction is executed.

8. The method according to claim 1, characterized in that, The method further includes: Repeated early warning requests from the same monitored subject within a preset time window are merged; when the guardian confirms the cancellation of the alarm within a preset waiting time, the event sample is marked as a false alarm and stored in the sample buffer pool for use in optimizing and updating the individualized baseline motion model.

9. A smart fall detection and early warning system for the elderly based on multimodal perception, characterized in that, The system includes: The data acquisition module is configured to acquire motion data of the monitored object through a wearable inertial sensor and video frame sequences through an indoor vision sensor. The data preprocessing module is configured to filter and denoise the inertial data and perform time alignment on the dual-modal data to obtain a time-synchronized multimodal raw dataset. The feature extraction and fusion module is configured to detect key points of human skeleton based on the video frame sequence, extract the coordinates of posture key points, and perform feature-level fusion of the inertial feature vector and the posture key point coordinate vector to obtain a multimodal fusion feature vector. The fall risk identification module is configured to use a sliding window method to segment the multimodal fusion feature vector, extract time-frequency features, and input the time-frequency features into a dual-stream neural network model for fall risk classification to obtain a preliminary fall risk probability. The adaptive calibration module is configured to adaptively calibrate the current detection result using an individualized baseline motion model, calculate the Mahalanobis distance between the current feature window and the baseline model, and dynamically adjust the decision threshold based on the Mahalanobis distance; at the same time, it acquires ambient light intensity data and dynamically adjusts the weight coefficients of each feature stream in the dual-stream neural network model based on the ambient light intensity. The early warning decision module is configured to recalculate the adjusted fall risk probability based on the dynamically adjusted judgment threshold and weight coefficient, compare it with the adjusted judgment threshold, determine the final early warning level, and execute the corresponding early warning instruction.

10. The system according to claim 9, characterized in that, The adaptive calibration module includes: The baseline modeling unit is configured to collect normal activity data of the monitored object within a preset time period during the system initialization phase, and to use a Gaussian mixture model to model the distribution of motion signals to establish an individualized baseline motion model. The deviation measurement unit is configured to calculate the Mahalanobis distance between the current feature window and the individualized baseline motion model during the real-time detection phase. If the Mahalanobis distance exceeds the critical value of the preset confidence interval, the decision threshold is dynamically increased based on the difference between the Mahalanobis distance and the critical value. The light adaptive unit is configured to acquire real-time illuminance data from an indoor illuminance sensor. When the real-time illuminance data is lower than a preset light threshold, it determines that the current scene is a low-light scene and performs a weight adjustment operation, increasing the weight of the first feature stream and decreasing the weight of the second feature stream.

Images