Fall behavior determination and early warning method and system based on multi-modal data

Abstract

The application discloses a fall behavior determination and early warning method and system based on multi-modal data, and belongs to the technical field of intelligent monitoring and health safety. The method comprises the following steps: a Jetson Nano processor processes a video stream acquired by a camera to obtain a human body posture detection result; a millimeter wave radar acquires human body motion state information, determines the motion state according to a mutation value, and obtains a human body motion state detection result; a blood oxygen bracelet collects human physiological data, identifies sudden physiological abnormalities, and obtains physiological abnormality data; according to the posture detection result, the motion state detection result and the physiological abnormality data, different levels of signal states are sent to a cloud platform; the cloud platform receives and fuses the signal states, adopts fall behavior grading early warning based on the fused multi-dimensional information, and triggers a grading alarm mechanism. The problems of low data utilization rate and single response means are solved, and the accuracy and adaptability of fall monitoring are improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent monitoring and health safety technology, and in particular to a fall behavior determination and early warning method and system based on multimodal data. Background Art

[0002] Falling is an involuntary physical behavior caused by obstruction of human joint movement. As the aging of my country's population continues to deepen, falls have become one of the major risk factors endangering public health, and related physical injury cases are showing a significant upward trend. Clinical research data show that human injuries and subsequent health problems caused by falls are largely attributed to the failure to obtain proper treatment and medical assistance in a timely manner after the injury occurs. Therefore, the development of accurate and efficient human fall behavior detection technology is of great practical significance for shortening emergency response time and reducing the severity of injuries.

[0003] In recent years, traditional fall detection technologies have relied on single sensors or machine learning algorithms, making them difficult to adapt to complex environments and diverse scenarios. Solutions that rely on single-modality data collected by video or wearable devices suffer from inaccurate detection and high rates of false positives and omissions. While camera technology can identify human posture, it is significantly affected by light and occlusion, raising privacy concerns. Millimeter-wave radar offers all-weather monitoring capabilities, but its ability to recognize fine movements is limited. Wearable devices, on the other hand, suffer from issues such as insufficient comfort. Summary of the Invention

[0004] In response to the above-mentioned deficiencies in the existing technology, the present application provides a fall behavior determination and warning method and system based on multimodal data, which solves the problems of existing fall detection technology's insufficient adaptability to complex environments, low data utilization, and single response means.

[0005] In order to achieve the above-mentioned invention objectives, the technical solutions adopted in this application are:

[0006] First aspect:

[0007] This application provides a fall behavior determination and warning method based on multimodal data, including:

[0008] S1: Obtains video streams and uses the Jetson Nano processor to analyze human posture in the video stream to obtain human posture detection results. Based on the human posture detection results, different levels of signal status are sent to the cloud platform;

[0009] S2: Use millimeter-wave radar to obtain human motion state information, and determine human motion state information based on the human motion state information to obtain human motion state detection results. Based on the human motion state detection results, different levels of signal status are sent to the cloud platform;

[0010] S3: Use the blood oxygen wristband to obtain human physiological data, identify sudden physiological abnormalities, obtain physiological abnormality data, and send different levels of signal status to the cloud platform based on the physiological abnormality data;

[0011] S4: The cloud platform is used to synchronize and fuse the received signal states of different levels, and a graded warning of fall behavior is performed based on the fused data. Based on the threshold method, a graded alarm mechanism is triggered according to the fall behavior warning level.

[0012] Furthermore, the use of the Jetson Nano processor to analyze the human posture in the video stream to obtain human posture feature information includes:

[0013] A1: Use the NMS algorithm to filter the video images of overlapping objects in the video images captured by the camera;

[0014] A2: Use the YOLOv11-Pose human key point detection algorithm to detect targets in the video stream, and obtain target detection results including the coordinates of the target box center point, the length and width of the target box, the target detection result, key point information including key point coordinates and confidence, target type and confidence;

[0015] A3: Use the DeepSort target tracking algorithm to fuse the target ID and target box information into the target detection result to obtain the fused target detection result;

[0016] A4: Construct a human posture determination algorithm based on the key point information in the fused target detection results. The algorithm determines the changes in human posture according to the relative positions of the key points, and obtains the human posture detection results.

[0017] Furthermore, the method of determining the change of human posture according to the relative positions of key points to obtain human posture detection information includes:

[0018] B1: Extract key points corresponding to the shoulders, hips, knees, and ankles in the video image based on the key point information in the fused target detection results;

[0019] B2: Calculate the angles between the hips, knees, and ankles based on the extracted key points corresponding to the hips, knees, and ankles;

[0020] B3: When the angles between the hip, knee, and ankle exceed the normal range of 70° to 150°, a signal 1 is sent to the cloud platform, and the relative offset angles of the shoulder and hip are calculated based on the extracted key points corresponding to the shoulder and hip.

[0021] B4: When the relative deviation angle between the shoulder and hip exceeds 35°, calculate the degree of elbow flexion;

[0022] B5: When the elbow bending degree does not exceed 60 degrees, the confidence of the detection frame of the YOLOv11-Pose human key point detection algorithm is enhanced;

[0023] B6: Use the YOLOv11-Pose human key point detection algorithm with enhanced confidence to detect key points. Based on the extracted key points corresponding to the shoulders and hips, calculate the inclination angles of the shoulders and hips.

[0024] B7: Obtain the human posture detection result based on the calculated inclination angles of the shoulders and hips.

[0025] Furthermore, the method of sending different levels of signal status to the cloud platform according to the human posture detection result includes:

[0026] When the tilt angle is between 35° and 50° and the body posture returns to normal within 1.5 seconds, it is determined that the human body has a mild fall and signal 1 is sent to the cloud platform; when the tilt angle is between 50° and 70° and does not recover for more than 2 seconds, it is determined that the human body has a moderate fall and signal 2 is sent to the cloud platform; when the tilt angle is greater than 70° and does not recover for 3 seconds, it is determined that the human body has a severe fall and signal 3 is sent to the cloud platform.

[0027] Furthermore, the S2 includes:

[0028] S201: The millimeter wave radar transmits a millimeter wave signal and receives a reflected wave, processes the echo signal to extract the distance, speed, and angle information of the target, and constructs a three-dimensional motion trajectory of the human body;

[0029] S202: Based on the three-dimensional motion trajectory of the human body, when the radar beam illuminates the human target, the micro-Doppler characteristics caused by breathing and heartbeat are extracted, the radial velocity component is calculated using the phase change between consecutive frames, and the human body motion speed is obtained through the Doppler frequency shift;

[0030] S203: performing differential processing on N consecutive frames of velocity data using a sliding window, and combining it with a Kalman filter to eliminate noise interference, thereby obtaining the acceleration of the center of mass of the human body;

[0031] S204: Determine the human body motion state information based on the obtained speed and acceleration, and obtain a human body motion state detection result;

[0032] S205: Send different levels of signal status to the cloud platform according to the human body motion status detection result.

[0033] Furthermore, the method of sending different levels of signal status to the cloud platform according to the human body motion status detection result includes:

[0034] When the speed exceeds 2m / s, a mild fall is determined and signal 1 is sent to the cloud. If the acceleration exceeds 3m / s², a moderate fall is determined and signal 2 is sent to the cloud. When a sudden speed change > 8m / s² and an acceleration > 9.8m / s² for more than 0.5 seconds is detected, a severe fall is determined and signal 3 is sent immediately.

[0035] Furthermore, in S3, different levels of signal status are sent to the cloud platform according to the physiological abnormality data, including:

[0036] When the blood oxygen saturation is detected to be lower than 90%, it is determined that a moderate fall has occurred and signal 2 is sent to the cloud; if the blood oxygen saturation is lower than 85%, it is determined that a severe fall has occurred and signal 3 is sent to the cloud; when the blood oxygen saturation is maintained at 95%-98% and the heart rate variability index is normal, it is determined that a mild fall has occurred and signal 1 is sent to the cloud.

[0037] Furthermore, the S4 specifically includes:

[0038] The cloud platform receives signal status from different detection devices, starts the cloud decision-making method, counts the status output of the detection equipment, selects the signal status with the most occurrences as the preliminary decision, and performs a graded fall behavior warning based on the preliminary decision. Based on the threshold method, if the graded fall behavior warning is a low-risk warning, a local voice alarm is used; if the graded fall behavior warning is a medium- or high-risk warning, a three-level alarm is used: SMS alarm, phone alarm, and APP alarm.

[0039] Furthermore, if the status outputs of the statistical detection devices are different, a weighted decision process is adopted, including:

[0040] C1: assign different weights to the number of signal states corresponding to the detection equipment;

[0041] C2: Obtain the fall determination result score based on the assigned weight and the number of signal states corresponding to the detection device;

[0042] C3: Provide graded alarms based on the fall assessment results.

[0043] Second aspect:

[0044] This application provides a fall behavior determination and early warning system based on multimodal data, including:

[0045] The user terminal device includes several night vision rotatable cameras deployed high in the house, a Jetson Nano processor, a millimeter-wave radar, and a blood oxygen wristband. The Jetson Nano processor receives the video captured by the night vision rotatable camera via a local area network connection and the RTMP video transmission protocol. The millimeter-wave radar collects human motion status information, and the blood oxygen wristband collects human physiological data.

[0046] a cloud platform that receives and stores multimodal data transmitted by the user terminal device, fuses and analyzes the multimodal data, determines fall behavior using a fall behavior determination model, and triggers an alarm module based on the fall risk level;

[0047] The alarm module includes a local voice alarm module, a text message and mobile phone APP push module, and an emergency contact notification module. The alarm module triggers different alarms based on the fall risk level determined by the cloud platform.

[0048] The beneficial effects of this application are:

[0049] The present application provides a method and system for determining and warning of fall behavior based on multimodal data. Through the fusion of multimodal data such as posture, radar, and blood oxygen data and simple decision-making logic, it can achieve rapid and accurate detection of fall events, thereby improving the reliability and practicality of detection. At the same time, the simplified data processing method also reduces the data transmission volume and processing complexity, and can achieve effective fall detection and alarm control. It can be widely used in public scenarios such as nursing homes, hospitals, and communities for accurate determination and timely warning of human fall behavior, and has significant engineering practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0051] Figure 1 This is a flowchart of a method for fall behavior determination and early warning based on multimodal data provided in an embodiment of the present application.

[0052] Figure 2 A schematic diagram of the structure of a fall behavior determination and early warning system based on multimodal data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0054] Example 1:

[0055] The present invention provides a method for determining and warning of falling behavior based on multimodal data. Figure 1 , Figure 1 The figure shows a flow chart of a method for determining and warning a fall behavior based on multimodal data provided by an embodiment of the present application, including:

[0056] S1: Obtain the video stream and use the Jetson Nano processor to analyze the human posture in the video stream to obtain the human posture detection results. Based on the human posture detection results, different levels of signal status are sent to the cloud platform.

[0057] In one embodiment of the present application, the video stream can be obtained by collecting real-time monitoring videos of the user's living area through a night vision rotatable camera. In the video encoding link, the H264 or H265 transmission format and the RTMP or STSP video transmission protocol are adopted. With the help of Mediamtx, a small server is built to transmit the collected video in the form of streaming media between different local area networks. The terminal for receiving and processing the video is a Jetson Nano processor. The user can adjust the angle of the camera through remote operation according to the actual scene requirements.

[0058] A deep learning runtime environment, including YOLOv11-pose and DeepSort, was built on the Jetson Nano development board. The relevant operating parameters, including the detection confidence threshold (Conf-detection), tracking confidence threshold (Conf-track), maximum number of detected targets (Max-classes), and video storage method and duration, were set. The model performance was optimized through video testing. The Tensor RT tool was used to compress the deployed YOLOv11 model method, change its inference method, set the single image inference size, and adjust the inference mode to half-precision inference. This improved the inference speed while ensuring detection accuracy, and completed image or video testing and verification.

[0059] Furthermore, the use of the Jetson Nano processor to analyze the human posture in the video stream to obtain human posture feature information includes:

[0060] A1: Use the NMS algorithm to filter the video images of overlapping objects in the video images captured by the camera;

[0061] A2: Use the YOLOv11-Pose human key point detection algorithm to detect targets in the video stream, and obtain target detection results including the coordinates of the target box center point, the length and width of the target box, the target detection result, the key point information including the key point coordinates and confidence, the target type and confidence:

[0062]

[0063]

[0064] in, and is the coordinate of the center point of the target frame, and is the length and width of the target box, is the key point information including key point coordinates and confidence, For the target species, is the confidence level, 、 is the coordinate of the center point of the key point, For a single key point, is the confidence of the key point, For a single target, Output results for target detection;

[0065] A3: Targets in continuous videos may be lost due to reality or NMS filtering. The DeepSort target tracking algorithm is introduced. It takes the video image and the target box and confidence output by YOLOv11-Pose as input, and fuses the target ID and target box information into the target detection result to obtain the fused target detection result. :

[0066]

[0067] in, It is a single target after the fusion of YOLOv11-Pose human key point detection algorithm and DeepSort target tracking algorithm. Represents the information fusion process between DeepSort processing and YOLOv11-Pose output results;

[0068] A4: To identify basic human movements, we constructed a human posture determination algorithm to identify five states: standing, falling, lying down, sitting, and key point loss due to visual loss. Taking into account factors such as algorithm operation, we used a manual feature extraction method. Based on the coordinates of the 17 key points in the fused target detection results, we determined changes in human posture according to the relative positions of the key points, and obtained the human posture detection results.

[0069] Furthermore, the method of determining the change of human posture according to the relative positions of key points to obtain human posture detection information includes:

[0070] B1: Extract key points corresponding to the shoulders, hips, knees, and ankles in the video image based on the key point information in the fused target detection results;

[0071] B2: Calculate the angles between the hips, knees, and ankles based on the extracted key points corresponding to the hips, knees, and ankles;

[0072] B3: When the angles between the hip, knee, and ankle exceed the normal range of 70° to 150°, a signal 1 is sent to the cloud platform, and the relative offset angles of the shoulder and hip are calculated based on the extracted key points corresponding to the shoulder and hip.

[0073] B4: When the relative deviation angle between the shoulder and hip exceeds 35°, calculate the degree of elbow flexion;

[0074] B5: When the elbow bending degree does not exceed 60 degrees, the confidence of the detection frame of the YOLOv11-Pose human key point detection algorithm is enhanced;

[0075] B6: Use the YOLOv11-Pose human key point detection algorithm with enhanced confidence to detect key points. Based on the extracted key points corresponding to the shoulders and hips, calculate the inclination angles of the shoulders and hips.

[0076] B7: Obtain the human posture detection result based on the calculated inclination angles of the shoulders and hips.

[0077] Furthermore, the method of sending different levels of signal status to the cloud platform according to the human posture detection result includes:

[0078] When the tilt angle is between 35° and 50° and the body posture returns to normal within 1.5 seconds, it is determined that the human body has a mild fall and signal 1 is sent to the cloud platform; when the tilt angle is between 50° and 70° and does not recover for more than 2 seconds, it is determined that the human body has a moderate fall and signal 2 is sent to the cloud platform; when the tilt angle is greater than 70° and does not recover for 3 seconds, it is determined that the human body has a severe fall and signal 3 is sent to the cloud platform.

[0079] In one embodiment of the present application, a severity grading and false detection control strategy based on multi-joint features is employed to determine human falls. The 17 key points detected in the fused target detection results focus on the geometric relationships and dynamic changes formed by the shoulders (key points 5-6), hips (key points 11-12), knees (key points 13-14), and ankles (key points 15-16). These key points were selected because the shoulders and hips are relatively stable during daily activities and can accurately reflect changes in torso posture. The relative position of the knees and hips can effectively characterize leg motion, which is particularly important for distinguishing falls from similar postures such as sitting or lying down.

[0080] The formula for calculating the hip-knee-ankle angle is:

[0081]

[0082] Where, 、 are the key point coordinates of the left hip, 、 are the key point coordinates of the right hip, 、 is the key point coordinate of the left knee, 、 are the coordinates of the right knee key point.

[0083] When the hip-knee-ankle angle When the angle exceeds the normal range of 70° to 150°, a signal 1 is first sent to the cloud, and then the relative offset angle between the shoulder and hip is calculated. The calculation formula for the relative offset angle between the shoulder and hip is as follows:

[0084]

[0085] Where, 、 is the key point coordinate of the left shoulder, 、 is the key point coordinate of the right shoulder, where the relative offset angle between the shoulder and the hip is Determine whether it is greater than 35°. If it is greater than 35°, calculate the degree of elbow flexion. The calculation formula is as follows:

[0086]

[0087] Where, 、 is the key point coordinate of the left elbow, 、 are the key point coordinates of the right elbow.

[0088] When the elbow bend does not exceed 60 degrees, the confidence of the YOLOv11-Pose human key point detection algorithm is enhanced;

[0089] The YOLOv11-Pose human key point detection algorithm with enhanced confidence is used to detect key points. Based on the extracted key points corresponding to the shoulders and hips, the inclination angles of the shoulders and hips are calculated using the following formula:

[0090]

[0091] According to the obtained tilt angle The size of triggers different levels of signals: when When the body angle is between 35° and 50° and the body posture returns to normal within 1.5 seconds, signal 1 is sent. When the angle is between 50° and 70° and it does not recover for more than 2 seconds, signal 2 is sent. If the angle is greater than 70° and remains unrecovered for 3 seconds, a signal 2 is sent to the cloud. Furthermore, this embodiment performs posture consistency detection between consecutive frames, improving judgment accuracy through cumulative analysis of multi-frame temporal features. It also filters out transient abnormal movements (such as bending or sitting down), thereby significantly reducing false detection rates while improving detection sensitivity.

[0092] In one embodiment of the present application, a human posture determination algorithm is constructed to identify five states: standing, falling, lying down, sitting, and key point loss due to visual loss. After integrating the overlapping targets filtered by NMS, the YOLOv11-Pose human key point detection algorithm is used to detect the targets, and DeepSort target tracking and behavior recognition with corresponding IDs are assigned to form a complete detection process; three-level privacy protection is simultaneously deployed: supporting dynamic trajectories of key points with preset static backgrounds, mean blurring, and human mosaics to achieve privacy protection combining dynamic visualization with static scenes.

[0093] The pose detection results output by YOLO11-pose-TensorRT are standardized and encapsulated and transmitted to the cloud in real time via a TLS 1.3 encrypted channel. Data preprocessing includes anomaly filtering, coordinate normalization, and feature encoding. The transmission process supports breakpoint resuming and keyframe caching. Bidirectional encryption and dynamic key exchange mechanisms ensure data security and integrity.

[0094] S2: Use millimeter-wave radar to obtain human motion state information, and determine human motion state information based on the human motion state information to obtain human motion state detection results. According to the human motion state detection results, different levels of signal states are sent to the cloud platform.

[0095] Furthermore, the S2 includes:

[0096] S201: The millimeter wave radar transmits a millimeter wave signal and receives a reflected wave, processes the echo signal to extract the distance, speed, and angle information of the target, and constructs a three-dimensional motion trajectory of the human body;

[0097] S202: Based on the three-dimensional motion trajectory of the human body, when the radar beam illuminates the human target, the micro-Doppler characteristics caused by breathing and heartbeat are extracted, the radial velocity component is calculated using the phase change between consecutive frames, and the human body motion speed is obtained through the Doppler frequency shift;

[0098] S203: performing differential processing on N consecutive frames of velocity data using a sliding window, and combining it with a Kalman filter to eliminate noise interference, thereby obtaining the acceleration of the center of mass of the human body;

[0099] S204: Based on the obtained velocity and acceleration, determine the human body motion state information according to the sudden change in velocity and the abnormal peak value of acceleration, determine the human body motion state information, and obtain a human body motion state detection result;

[0100] S205: Send different levels of signal status to the cloud platform according to the human body motion status detection result.

[0101] Furthermore, according to the human body motion state detection result, different levels of signal states are sent to the cloud platform, including:

[0102] When the speed exceeds 2m / s, a mild fall is determined and signal 1 is sent to the cloud. If the acceleration exceeds 3m / s², a moderate fall is determined and signal 2 is sent to the cloud. When a sudden speed change > 8m / s² and an acceleration > 9.8m / s² for more than 0.5 seconds is detected, a severe fall is determined and signal 3 is sent immediately.

[0103] In one embodiment of the present application, a millimeter-wave radar is hung on the roof 2.5 meters above the ground. The speed and acceleration of the human body are calculated by transmitting millimeter-wave signals and receiving the time difference and Doppler frequency shift of the reflected waves. The echo signal is first processed to extract the distance, speed and angle information of the target, and the three-dimensional motion trajectory of the human body is constructed.

[0104] When the radar beam illuminates a human target, the system first extracts the micro-Doppler characteristics caused by breathing and heartbeat, calculates the radial velocity component using the phase change between consecutive frames, and then obtains the human body's motion speed using the Doppler frequency shift formula. The calculation formula is as follows:

[0105]

[0106] Where, is the wavelength, is the frequency shift.

[0107] For acceleration calculation, the system uses a sliding window to perform differential processing on the velocity data of N consecutive frames, combines Kalman filtering to eliminate noise interference, and finally outputs the three-dimensional acceleration vector of the human body's center of mass. , the calculation formula is as follows:

[0108]

[0109] Where, For speed, For time, is the derivative operator.

[0110] When the speed exceeds 2m / s, signal 1 is sent to the cloud. If the acceleration exceeds 3m / s 2 When a free fall feature is detected (e.g., a sudden change in velocity > 8 m / s and an acceleration > 9.8 m / s² for more than 0.5 seconds), a signal is sent to the cloud immediately.

[0111] S3: Use the blood oxygen bracelet to obtain human physiological data, identify sudden physiological abnormalities, obtain physiological abnormality data, and send different levels of signal status to the cloud platform based on the physiological abnormality data.

[0112] Furthermore, in S3, different levels of signal status are sent to the cloud platform according to the physiological abnormality data, including:

[0113] When the blood oxygen saturation is detected to be lower than 90%, it is determined that a moderate fall has occurred and signal 2 is sent to the cloud; if the blood oxygen saturation is lower than 85%, it is determined that a severe fall has occurred and signal 3 is sent to the cloud; when the blood oxygen saturation is maintained at 95%-98% and the heart rate variability index is normal, it is determined that a mild fall has occurred and signal 1 is sent to the cloud.

[0114] In one embodiment of the present application, a blood oxygen wristband is used to collect human blood oxygen saturation and heart rate variability indicators through a photoplethysmography (PPG) sensor, and the data is transmitted to the cloud in a hierarchical manner based on a multi-level threshold mechanism. The wristband has built-in dual-wavelength light-emitting diodes (660nm red light and 940nm infrared light) and a high-sensitivity photodetector. After completing the self-test calibration during the device initialization phase, it continuously collects the absorption characteristics of human tissue for light of different wavelengths. By real-time analysis of the ratio of the AC component to the DC component in the PPG signal, blood oxygen saturation ( ) is calculated as follows:

[0115]

[0116] in, 、 is the equipment calibration factor, and are the AC and DC components of red light, and are the AC and DC components of infrared light respectively.

[0117] During the data collection process, an improved three-finger digital filtering algorithm is used to eliminate motion noise and ambient light interference, ensuring stable and reliable measurement results both at rest and during exercise. When the blood oxygen saturation is detected to be below 90%, the system sends signal 2 to the cloud. If the blood oxygen saturation drops further below 85%, signal 3 is immediately transmitted. Under normal monitoring conditions, when the blood oxygen saturation remains between 95% and 98% and the heart rate variability index is normal, the bracelet only transmits signal 1. By dynamically adjusting the data transmission volume and frequency, this design significantly reduces network load and device energy consumption while ensuring the timely transmission of signals for key physiological indicators.

[0118] S4: The cloud platform is used to synchronize and fuse the received signal states of different levels, and a graded warning of fall behavior is performed based on the fused data. Based on the threshold method, a graded alarm mechanism is triggered according to the fall behavior warning level.

[0119] Furthermore, the S4 specifically includes:

[0120] The cloud platform receives signal status from different detection devices, starts the cloud decision-making method, counts the status output of the detection equipment, selects the signal status with the most occurrences as the preliminary decision, and performs a graded fall behavior warning based on the preliminary decision. Based on the threshold method, if the graded fall behavior warning is a low-risk warning, a local voice alarm is used; if the graded fall behavior warning is a medium- or high-risk warning, a three-level alarm is used: SMS alarm, phone alarm, and APP alarm.

[0121] Furthermore, if the status outputs of the statistical detection devices are different, a weighted decision process is adopted, including:

[0122] C1: assign different weights to the number of signal states corresponding to the detection equipment;

[0123] C2: Obtain the fall determination result score based on the assigned weight and the number of signal states corresponding to the detection device;

[0124] C3: Provide graded alarms based on the fall assessment results.

[0125] In one embodiment of the present application, when receiving signal states from different terminal devices, a cloud-based decision-making method is initiated, counting the state inputs from the three sensors and selecting the state with the most occurrences as the preliminary decision. If the cloud platform receives and counts only one occurrence of signal 1, no warning is issued. If the cloud platform receives and counts two or three occurrences of signal 1 from different detection devices, a low-risk warning is triggered, via voice alert. If the cloud platform receives and counts two or three occurrences of signal 2 from different detection devices, a medium-risk warning is triggered, via text message alert. If the cloud platform receives and counts two or three occurrences of signal 3 from different detection devices, a high-risk warning is triggered, via phone call and app alert. If a tie occurs, a weighted decision process is entered. During this weighted decision-making process, the millimeter-wave radar is first assigned a weight of 0.4, the blood oxygen wristband a weight of 0.3, and the posture detection a weight of 0.3. The final score is the sum of each terminal device's state multiplied by its corresponding weight. Because millimeter-wave radar can clearly detect rapid deceleration and sharp tilts in real time and is easy to identify, it is given the highest weight. Radar status is weighted the highest, ensuring the decisive role of motion data; blood oxygen status is second, supplementing the reliability of vital signs; and posture status is the lowest, serving as a secondary verification method. A threshold score of <1.8 triggers a voice alarm. If 1.8 ≤ Score <2.5, an app and SMS alert are triggered. A score ≥2.5 triggers an emergency phone call. To reduce false alarm rates, a dynamic adjustment mechanism is implemented. If a sensor has a high false alarm rate, such as posture detection, its weight can be temporarily reduced (for example, to 0.2) to improve overall system reliability. For example, if posture detection's false alarm rate increases due to environmental interference or frequent user bending, its weight can be reduced to minimize its influence on the final decision.

[0126] Example 2:

[0127] This application provides a fall behavior determination and early warning system based on multimodal data, which can be found in Figure 2 , Figure 2 A schematic diagram of a fall behavior determination and early warning system based on multimodal data provided in an embodiment of the present application includes:

[0128] The user terminal device includes several night vision rotatable cameras deployed high in the house, a Jetson Nano processor, a millimeter-wave radar, and a blood oxygen wristband. The Jetson Nano processor receives the video captured by the night vision rotatable camera via a local area network connection and the RTMP video transmission protocol. The millimeter-wave radar collects human motion status information, and the blood oxygen wristband collects human physiological data.

[0129] a cloud platform that receives and stores multimodal data transmitted by the user terminal device, fuses and analyzes the multimodal data, determines fall behavior using a fall behavior determination model, and triggers an alarm module based on the fall risk level;

[0130] The alarm module includes a local voice alarm module, a text message and mobile phone APP push module, and an emergency contact notification module. The alarm module triggers different alarms based on the fall risk level determined by the cloud platform.

[0131] In one embodiment of the present application, a camera is set at a higher place in the house to monitor the user's posture and perform key point detection, target tracking, and human posture judgment on the human body based on a deep learning algorithm; wherein the millimeter wave radar is set inside the house at the same height as the camera, one to two meters away.

[0132] This application provides a method and system for determining and warning of fall behavior based on multimodal data. By leveraging advanced biosensing technology, computer perception technology, deep learning algorithms, and cloud technology, the system utilizes a wristband to monitor an individual's blood oxygen, heart rate, and other physiological indicators in real time, along with millimeter-wave radar data on the body's motion amplitude and speed. By analyzing these parameters, the system can determine a person's motion and physical state. It also uses a camera to perform real-time detection of individual posture results. By combining a multi-source information fusion mechanism with intelligent algorithms, the system is applicable to a variety of complex environments, including home-based elderly care, public activity areas, and medical institutions. The data from the three sensors is converted into simple 1, 2, or 3 states and transmitted as input to the cloud, which synthesizes these states to control different alarm modes. Through the fusion of multimodal data and simple decision-making logic, the system achieves rapid and accurate detection of fall events, improving the reliability and practicality of the system. At the same time, the simplified data processing method also reduces the data transmission volume and processing complexity, while achieving effective fall detection and alarm control. According to different fall detections, different levels of alarms are activated. While maintaining detection accuracy, the false alarm rate is reduced, and the adaptability in different scenarios is improved, effectively reducing the risk of personal injury to users due to falls.

[0133] It should be noted that those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of this application, and it should be understood that the scope of protection of this application is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in this application without departing from the essence of this application, and such variations and combinations are still within the scope of protection of this application.

Claims

1. A fall behavior determination and early warning method based on multimodal data, characterized in that: include: S1: Obtain a video stream and use the Jetson Nano processor to analyze the human posture in the video stream to obtain the human posture detection result. Based on the human posture detection result, different levels of signal status are sent to the cloud platform. The human posture includes the tilt angle and the body posture recovery time. S2: Utilizing millimeter-wave radar to obtain human motion state information, and determining human motion state information based on the human motion state information to obtain a human motion state detection result, and sending different levels of signal status to the cloud platform based on the human motion state detection result; wherein the human motion state information includes speed and acceleration; S3: Using the blood oxygen wristband to obtain human physiological data, identify sudden physiological abnormalities, obtain physiological abnormality data, and send different levels of signal status to the cloud platform based on the physiological abnormality data; wherein the human physiological data includes blood oxygen saturation and heart rate variability indicators; S4: The cloud platform is used to synchronize and fuse the received signal states of different levels. Based on the fused data, a graded warning of fall behavior is issued. Based on the threshold method, a graded alarm mechanism is triggered according to the fall behavior warning level. According to the human posture detection results, different levels of signal status are sent to the cloud platform, including: When the tilt angle is between 35° and 50° and the body posture returns to normal within 1.5 seconds, it is determined that the person has a mild fall and a signal 1 is sent to the cloud platform. When the tilt angle is between 50° and 70° and the body posture does not return to normal for more than 2 seconds, it is determined that the person has a moderate fall and a signal 2 is sent to the cloud platform. When the tilt angle is greater than 70° and does not return to normal for 3 seconds, it is determined that the person has a severe fall and a signal 3 is sent to the cloud platform. According to the human body motion state detection results, different levels of signal status are sent to the cloud platform, including: When the speed exceeds 2m / s, it is determined that there is a mild fall and signal 1 is sent to the cloud platform. If the acceleration exceeds 3m / s², it is determined that there is a moderate fall and signal 2 is sent to the cloud platform. When a sudden change in speed > 8m / s and the acceleration lasts > 9.8m / s² for more than 0.5 seconds, it is determined that there is a severe fall and signal 3 is sent immediately. In S3, different levels of signal status are sent to the cloud platform based on physiological abnormality data, including: When the blood oxygen saturation is detected to be lower than 90%, it is determined that a moderate fall has occurred and signal 2 is sent to the cloud platform. If the blood oxygen saturation is lower than 85%, it is determined that a severe fall has occurred and signal 3 is sent to the cloud platform. When the blood oxygen saturation is maintained at 95%-98% and the heart rate variability index is normal, it is determined that a mild fall has occurred and signal 1 is sent to the cloud platform. Said S4 specifically includes: The cloud platform receives signal states from different detection devices, activates the cloud decision-making method, and counts the state outputs of the detection devices. If the state outputs of the detection devices are the same, the signal state with the most occurrences is selected as the preliminary decision. Based on the preliminary decision, a graded fall behavior warning is performed. Based on the threshold method, if the graded fall behavior warning is a low-risk warning, a local voice alarm is used; if the graded fall behavior warning is a medium-to-high-risk warning, a three-level alarm is used: SMS alarm, phone alarm, and APP alarm. If the status outputs of the statistical detection equipment are different, a weighted decision process is used, including: C1: assign different weights to the number of signal states corresponding to the detection equipment; C2: Obtain the fall determination result score based on the assigned weight and the number of signal states corresponding to the detection device; C3: Provide graded alarms based on the fall assessment results.

2. The fall behavior determination and early warning method based on multimodal data according to claim 1 is characterized in that: The method of using the Jetson Nano processor to analyze the human posture in the video stream to obtain human posture feature information includes: A1: Use the NMS algorithm to filter the video images of overlapping objects in the video images captured by the camera; A2: Use the YOLOv11-Pose human key point detection algorithm to detect targets in the video stream, and obtain target detection results including the coordinates of the target box center point, the length and width of the target box, the target detection result, key point information including key point coordinates and confidence, target type and confidence; A3: Use the DeepSort target tracking algorithm to fuse the target ID and target box information into the target detection result to obtain the fused target detection result; A4: Construct a human posture determination algorithm based on the key point information in the fused target detection results. The algorithm determines the changes in human posture according to the relative positions of the key points, and obtains the human posture detection results.

3. The fall behavior determination and early warning method based on multimodal data according to claim 2 is characterized in that: The method of determining the change of human posture according to the relative positions of key points to obtain human posture detection information includes: B1: Extract key points corresponding to the shoulders, hips, knees, and ankles in the video image based on the key point information in the fused target detection results; B2: Calculate the angles between the hips, knees, and ankles based on the extracted key points corresponding to the hips, knees, and ankles; B3: When the angles between the hip, knee, and ankle exceed the normal range of 70° to 150°, a signal 1 is sent to the cloud platform, and the relative offset angles of the shoulder and hip are calculated based on the extracted key points corresponding to the shoulder and hip. B4: When the relative deviation angle between the shoulder and hip exceeds 35°, calculate the degree of elbow flexion; B5: When the elbow bending degree does not exceed 60 degrees, the confidence of the detection frame of the YOLOv11-Pose human key point detection algorithm is enhanced; B6: Use the YOLOv11-Pose human key point detection algorithm with enhanced confidence to detect key points. Based on the extracted key points corresponding to the shoulders and hips, calculate the inclination angles of the shoulders and hips. B7: Based on the calculated inclination angles of the shoulders and hips, the human body posture detection result is obtained.

4. The fall behavior determination and early warning method based on multimodal data according to claim 1 is characterized in that: The S2 includes: S201: The millimeter wave radar transmits a millimeter wave signal and receives a reflected wave, processes the echo signal to extract the distance, speed, and angle information of the target, and constructs a three-dimensional motion trajectory of the human body; S202: Based on the three-dimensional motion trajectory of the human body, when the radar beam illuminates the human target, the micro-Doppler characteristics caused by breathing and heartbeat are extracted, the radial velocity component is calculated using the phase change between consecutive frames, and the human body motion speed is obtained through the Doppler frequency shift; S203: performing differential processing on N consecutive frames of velocity data using a sliding window, and combining it with a Kalman filter to eliminate noise interference, thereby obtaining the acceleration of the center of mass of the human body; S204: Determine the human body motion state information based on the obtained speed and acceleration, and obtain a human body motion state detection result; S205: Send different levels of signal status to the cloud platform according to the human body motion status detection result.

5. A system for fall behavior determination and early warning method based on multimodal data according to any one of claims 1 to 4, characterized in that: include: The user terminal device includes several night vision rotatable cameras deployed high in the house, a Jetson Nano processor, a millimeter-wave radar, and a blood oxygen wristband. The Jetson Nano processor receives the video captured by the night vision rotatable camera via a local area network connection and the RTMP video transmission protocol. The millimeter-wave radar collects human motion status information, and the blood oxygen wristband collects human physiological data. a cloud platform that receives and stores multimodal data transmitted by the user terminal device, fuses and analyzes the multimodal data, determines fall behavior using a fall behavior determination model, and triggers an alarm module based on the fall risk level; The alarm module includes a local voice alarm module, a text message and mobile phone APP push module, and an emergency contact notification module. The alarm module triggers different alarms based on the fall risk level determined by the cloud platform.

Images