Fall behavior judgment and early warning method and system based on multi-modal data

Abstract

The invention discloses a tumble behavior judgment and early warning method and system based on multi-modal data, and belongs to the technical field of intelligent monitoring and health safety, and the method comprises the steps: processing a video stream obtained by a camera through a Jeson Nano processor, and obtaining a human body posture detection result; the millimeter-wave radar obtains human body motion state information, judges a motion state according to the sudden change value and obtains a human body motion state detection result; the blood oxygen bracelet collects human body physiological data and recognizes sudden physiological abnormalities to obtain physiological abnormality data; according to the posture detection result, the motion state detection result and the physiological anomaly data, sending different levels of signal states to a cloud platform; and the cloud platform receives and fuses the signal states, and based on the fused multi-dimensional information, adopts falling behavior grading early warning and triggers a grading alarm mechanism. The problems of low data utilization rate and single response means are solved, and the accuracy and adaptability of fall monitoring are improved.

Description

Technical Field

[0001] This application relates to the technical field of intelligent monitoring and health safety, and particularly to a fall behavior determination and warning method and system based on multi-modal data. Background Art

[0002] Falling is an involuntary body behavior caused by the obstruction of human joint movement. With the continuous deepening of the aging degree of China's population, falling has become one of the important risk factors endangering public health, and the number of related physical injury cases shows a significant upward trend. Clinical research data shows that the physical injuries and subsequent health problems caused by falling are largely attributed to the failure to obtain proper treatment and medical assistance in a timely manner after the occurrence of the injury event. Therefore, developing accurate and efficient human fall behavior detection technology has important practical significance for shortening the emergency response time and reducing the severity of injuries.

[0003] In recent years, traditional fall detection technologies mostly rely on single sensors or machine learning algorithms, and it is difficult to meet the requirements of complex environments and multi-scenarios. The solutions that rely on single-modal data collected by video or wearable devices have problems such as inaccurate detection, high false alarm and missed alarm rates. Although camera technology can recognize human postures, it is significantly affected by light and occlusion, and raises privacy concerns; millimeter-wave radar has all-weather monitoring capabilities, but the recognition of fine movements is limited; wearable devices have problems such as insufficient wearing comfort. Summary of the Invention

[0004] Aiming at the above deficiencies in the prior art, a fall behavior determination and warning method and system based on multi-modal data provided by this application solves the problems of insufficient adaptability to complex environments, low data utilization rate, and single response means in existing fall detection technologies.

[0005] In order to achieve the above invention purpose, the technical solution adopted by this application is as follows: First aspect: This application provides a fall behavior determination and warning method based on multi-modal data, including: S1: Obtain a video stream, and use a Jeson Nano processor to analyze the human posture in the video stream to obtain a human posture detection result, and send signal states of different levels to the cloud platform according to the human posture detection result; S2: Use a millimeter-wave radar to obtain human motion state information, and determine the human motion state information according to the human motion state information to obtain a human motion state detection result, and send signal states of different levels to the cloud platform according to the human motion state detection result; S3: Use a blood oxygen bracelet to obtain human physiological data, identify sudden physiological abnormalities to obtain physiological abnormality data, and send signal states of different levels to the cloud platform according to the physiological abnormality data; S4: Use the cloud platform to perform time synchronization and fusion processing on the received signal states of different levels, perform fall behavior classification and early warning based on the fused data, and trigger a classification alarm mechanism based on the threshold method according to the fall behavior early warning level.

[0006] Further, the method for analyzing the human body posture in the video stream by using the Jeson Nano processor to obtain human body posture feature information includes: A1: Use the NMS algorithm to filter the video images of overlapping targets in the video images collected by the camera; A2: Use the YOLOv11-Pose human key point detection algorithm to detect the targets in the video stream, and obtain the target detection results including the center point coordinates of the target box, the length and width of the target box, the target detection results, the key point information including key point coordinates and confidence, the target type and confidence; A3: Use the DeepSort target tracking algorithm to fuse the target ID and target box information into the target detection results to obtain the fused target detection results; A4: Construct a human body posture determination algorithm, and based on the key point information in the fused target detection results, judge the change of the human body posture according to the relative position of the key points to obtain the human body posture detection results.

[0007] Further, the method for judging the change of the human body posture according to the relative position of the key points to obtain the human body posture detection information includes: B1: According to the key point information in the fused target detection results, extract the key points corresponding to the shoulders, hips, knees and ankles in the video image; B2: Based on the key points corresponding to the extracted hips, knees and ankles, calculate the angles between the hips, knees and ankles; B3: When the angles between the hips, knees and ankles exceed the normal range of 70° to 150°, send signal 1 to the cloud platform, and based on the key points corresponding to the extracted shoulders and hips, calculate the relative offset angle between the shoulders and hips; B4: When the relative offset angle between the shoulders and hips exceeds 35°, calculate the degree of elbow bending; B5: When the degree of elbow bending does not exceed 60°, enhance the confidence of the detection box of the YOLOv11-Pose human key point detection algorithm; B6: Use the YOLOv11-Pose human key point detection algorithm with enhanced confidence to detect key points, and based on the key points corresponding to the extracted shoulders and hips, calculate the tilt angle between the shoulders and hips; B7: Obtain the human body posture detection results according to the calculated tilt angle between the shoulders and hips.

[0008] Further, according to the human body posture detection result, sending signal states of different levels to the cloud platform includes: When the tilt angle is between 35° and 50° and the body posture returns to normal within 1.5 seconds, it is determined that there is a mild fall behavior of the human body, and signal 1 is sent to the cloud platform; when the tilt angle is between 50° and 70° and does not recover for more than 2 seconds, it is determined that there is a moderate fall behavior of the human body, and signal 2 is sent to the cloud platform; when the tilt angle is greater than 70° and does not recover for 3 seconds continuously, it is determined that there is a severe fall behavior of the human body, and signal 3 is sent to the cloud platform.

[0009] Further, the S2 includes: S201: Transmitting millimeter-wave signals by the millimeter-wave radar and receiving reflected waves, processing the echo signals to extract the distance, speed, and angle information of the target, and constructing the three-dimensional motion trajectory of the human body; S202: According to the three-dimensional motion trajectory of the human body, when the radar beam irradiates the human body target, extracting the micro-Doppler features caused by breathing and heartbeat, calculating the radial velocity component using the phase change between consecutive frames, and obtaining the human body motion speed through Doppler frequency shift; S203: Performing differential processing on the velocity data of consecutive N frames using a sliding window, and combining Kalman filtering to eliminate noise interference to obtain the acceleration of the human body centroid; S204: Based on the obtained speed and acceleration, determining the human body motion state information to obtain the human body motion state detection result; S205: According to the human body motion state detection result, sending signal states of different levels to the cloud platform.

[0010] Further, according to the human body motion state detection result, sending signal states of different levels to the cloud platform includes: When the speed exceeds 2 m / s, it is determined that there is a mild fall behavior, and signal 1 is sent to the cloud; if the acceleration exceeds 3 m / s², it is determined that there is a moderate fall behavior, and signal 2 is sent to the cloud; when a speed mutation > 8 m / s and the acceleration continuously > 9.8 m / s² for more than 0.5 seconds are detected, it is determined that there is a severe fall behavior, and signal 3 is sent immediately.

[0011] Further, in S3, according to the physiological abnormal data, sending signal states of different levels to the cloud platform includes: When the detected blood oxygen saturation is lower than 90%, it is determined that there is a moderate fall behavior, and signal 2 is sent to the cloud; if the blood oxygen saturation is lower than 85%, it is determined that there is a severe fall behavior, and signal 3 is sent to the cloud; when the blood oxygen saturation is maintained between 95% and 98% and the heart rate variability index is normal, it is determined that there is a mild fall behavior of the human body, and signal 1 is sent to the cloud.

[0012] Further, step S4 specifically includes: The cloud platform receives the signal states from different detection devices, starts the cloud decision-making method, counts the status outputs of the detection devices, selects the signal state with the most occurrences as the preliminary decision, conducts fall behavior classification and early warning based on the preliminary decision, and based on the threshold method, if the fall behavior classification and early warning is a low-risk early warning, local voice alarm is adopted; if the fall behavior classification and early warning is a medium-high risk early warning, a three-level alarm of SMS alarm, phone alarm, and APP alarm is adopted.

[0013] Further, if the status outputs of the statistical detection devices are different, a weighted decision-making process is adopted, including: C1: Assign different weights to the number of signal states corresponding to the detection devices; C2: Based on the assigned weights and the number of signal states corresponding to the detection devices, obtain the fall determination result score; C3: Conduct classification and alarm according to the fall determination result score.

[0014] Second aspect: This application provides a fall behavior determination and early warning system based on multi-modal data, including: A user terminal device, including several night vision rotatable cameras deployed at high places in the house, a Jeson Nano processor, a millimeter wave radar, and a blood oxygen bracelet. The Jeson Nano processor receives the videos collected by the night vision rotatable cameras through connecting to the local area network and adopting the RTMP video transmission protocol. The millimeter wave radar collects human motion state information, and the blood oxygen bracelet collects human physiological data; A cloud platform, which receives and stores the multi-modal data transmitted by the user terminal device, fuses and analyzes the multi-modal data, conducts fall behavior judgment through a fall behavior determination model, and triggers an alarm module based on the fall risk level; An alarm module, including a local voice alarm module, an SMS and mobile APP push module, and a module for notifying emergency contacts. The alarm module triggers different alarms based on the fall risk level determined by the cloud platform.

[0015] The beneficial effects of this application are: A fall behavior determination and early warning method and system based on multi-modal data provided by this application realizes fast and accurate detection of fall events through the fusion of multi-modal data such as posture, radar, and blood oxygen data and simple decision-making logic, improves the reliability and practicality of detection. At the same time, the simplified data processing method also reduces the data transmission volume and processing complexity, and can also realize effective fall detection and alarm control. It can be widely applied to the accurate determination and timely early warning of human fall behaviors in public scenarios such as nursing homes, hospitals, and communities, and has significant engineering practicality. Brief Description of the Drawings

[0016] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can also obtain other embodiments based on these drawings.

[0017] Figure 1 It is a method flow chart of a fall behavior determination and early warning method based on multi-modal data provided by an embodiment of the present application.

[0018] Figure 2 It is a schematic structural diagram of a fall behavior determination and early warning system based on multi-modal data provided by an embodiment of the present application. Detailed Embodiments

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art based on the present application belong to the scope of protection of the present application.

[0020] Embodiment 1: The embodiment of the present application provides a fall behavior determination and early warning method based on multi-modal data. This method can be referred to Figure 1 , Figure 1 As shown in the method flow chart of a fall behavior determination and early warning method based on multi-modal data provided by an embodiment of the present application, it includes: S1: Obtain a video stream, analyze the human posture in the video stream using a Jeson Nano processor to obtain a human posture detection result, and send signal states of different levels to the cloud platform according to the human posture detection result.

[0021] In an embodiment of the present application, the video stream can be obtained by a night vision rotatable camera to collect the monitoring video of the user's living area in real time. In the video encoding link, the H264 or H265 transmission format and the RTMP or STSP video transmission protocol are adopted, and a small server is built with Mediamtx to transmit the collected video in the form of a streaming media between different local area networks. The terminal for receiving and processing the video is a Jeson Nano processor, and the user can adjust the angle of the camera through remote operation according to the actual scenario requirements.

[0022] Build the deep learning running environments such as YOLOv11-pose and DeepSort on the Jeson Nano development board, and set relevant running parameters. The main parameters include the detection confidence threshold (Conf-detection), the tracking confidence threshold (Conf-track), the maximum number of detected targets (Max-classes), as well as the video saving method and duration, etc. Optimize the model performance through video testing and parameter tuning; at the same time, use the Tensor RT tool to compress the deployed YOLOv11 model method, change its inference method, set the single-image inference size, and adjust the inference mode to half-precision inference. Under the condition of ensuring detection accuracy, improve the inference speed, and complete the picture or video test verification.

[0023] Further, analyze the human pose in the video stream using the Jeson Nano processor to obtain human pose feature information, including: A1: Use the NMS algorithm to filter the video images of overlapping targets in the video images collected by the camera; A2: Adopt the YOLOv11-Pose human key point detection algorithm to detect the targets in the video stream, and obtain the target detection results including the center point coordinates of the target box, the length and width of the target box, the target detection results, the key point information including the key point coordinates and confidence, the target category and confidence:

[0024]

[0025] Among them, and are the center point coordinates of the target box, and are the length and width of the target box, is the key point information including the key point coordinates and confidence, is the target category, is the confidence, 、 are the coordinates of the center point of the key point, is a single key point, is the confidence of the key point, is a single target, is the target detection output result; A3: In the continuous video, the target may be lost due to reality or NMS filtering. Introduce the DeepSort target tracking algorithm, take the video image and the target box and confidence output by YOLOv11-Pose as the input, fuse the target ID and target box information into the target detection result, and obtain the fused target detection result :

[0026] Among them, is a single target after the fusion of the YOLOv11-Pose human key point detection algorithm and the DeepSort object tracking algorithm, represents the information fusion processing process between the output result of YOLOv11-Pose after being processed by DeepSort; A4: To perform human basic action recognition, a human pose determination algorithm is constructed to recognize five states: standing, falling, lying down, sitting, and the loss of key points caused by visual loss. Due to factors such as algorithm operation, the manual feature extraction method is adopted. Based on the information such as the coordinates of 17 key points in the fused object detection result, the human pose change is judged according to the relative position of the key points to obtain the human pose detection result.

[0027] Furthermore, the judging of the human pose change according to the relative position of the key points to obtain the human pose detection information includes: B1: According to the key point information in the fused object detection result, extract the key points corresponding to the shoulders, hips, knees, and ankles in the video image; B2: Based on the key points corresponding to the extracted hips, knees, and ankles, calculate the angles between the hips, knees, and ankles; B3: When the angles between the hips, knees, and ankles exceed the normal range of 70° to 150°, send signal 1 to the cloud platform, and based on the key points corresponding to the extracted shoulders and hips, calculate the relative offset angle between the shoulders and the hips; B4: When the relative offset angle between the shoulders and the hips exceeds 35°, calculate the elbow bending degree; B5: When the elbow bending degree does not exceed 60°, enhance the confidence of the detection frame of the YOLOv11-Pose human key point detection algorithm; B6: Use the YOLOv11-Pose human key point detection algorithm with enhanced confidence to detect key points, and based on the key points corresponding to the extracted shoulders and hips, calculate the inclination angle between the shoulders and the hips; B7: According to the calculated inclination angle size between the shoulders and the hips, obtain the human pose detection result.

[0028] Furthermore, sending different levels of signal states to the cloud platform according to the human pose detection result includes: When the tilt angle is between 35° and 50° and the body posture returns to normal within 1.5 seconds, it is determined that the human body has a mild fall behavior, and signal 1 is sent to the cloud platform; when the tilt angle is between 50° and 70° and does not recover for more than 2 seconds, it is determined that the human body has a moderate fall behavior, and signal 2 is sent to the cloud platform; when the tilt angle is greater than 70° and does not recover for 3 seconds, it is determined that the human body has a severe fall behavior, and signal 3 is sent to the cloud platform.

[0029] In an embodiment of the present application, when determining the human fall behavior, a severity grading and false detection control strategy based on multi-site joint features is adopted. Through the 17 key points detected in the fused object detection results, the geometric relationship and dynamic changes formed by the shoulders (key points 5-6), hips (key points 11-12), knees (key points 13-14), and ankles (key points 15-16) are focused on. The reason for selecting these key parts is that the shoulders and hips are relatively stable in daily activities and can accurately reflect the trunk posture changes, while the relative position of the knees and hips can effectively represent the leg movement state, which is particularly important for distinguishing falls from similar postures such as sitting and lying.

[0030] Among them, the calculation formula for the hip-knee-ankle angle is:

[0031] In the formula, 、 are the key point coordinates of the left hip, 、 are the key point coordinates of the right hip, 、 are the key point coordinates of the left knee, 、 are the key point coordinates of the right knee.

[0032] When the hip-knee-ankle angle exceeds the normal range of 70° to 150°, first send signal 1 to the cloud, and then calculate the relative offset angle between the shoulder and the hip. The calculation formula for the relative offset angle between the shoulder and the hip is as follows:

[0033] In the formula, 、 are the key point coordinates of the left shoulder, 、 are the key point coordinates of the right shoulder. Among them, the relative offset angle between the shoulder and the hip is judged whether it is greater than 35°. If it is greater than 35°, calculate the elbow bending degree. The calculation formula is as follows:

[0034] Wherein, and are the key point coordinates of the left elbow, and are the key point coordinates of the right elbow.

[0035] When the elbow bending degree does not exceed 60°, the confidence of the YOLOv11-Pose human key point detection algorithm is enhanced; Use the enhanced-confidence YOLOv11-Pose human key point detection algorithm to detect key points. Based on the extracted key points corresponding to the shoulders and hips, calculate the tilt angle between the shoulders and the hips. The calculation formula is as follows:

[0036] According to the obtained tilt angle of size, trigger different levels of signals: When is between 35° - 50° and the body posture returns to normal within 1.5 seconds, send signal 1. When is between 50° - 70° and has not recovered for more than 2 seconds, send signal 2. And when is greater than 70° and has not recovered for 3 seconds, then send signal 2 to the cloud. In addition, this embodiment also performs pose consistency detection between consecutive frames, improves the determination accuracy through the cumulative analysis of multi-frame temporal features, and filters instantaneous abnormal actions (such as bending down, sitting down) at the same time, thereby greatly reducing the false detection rate while enhancing the detection sensitivity.

[0037] In an embodiment of the present application, a human pose determination algorithm is constructed to identify five states: standing, falling, lying down, sitting down, and key point loss caused by visual loss. After integrating the objects filtered by NMS for overlapping, use the YOLOv11-Pose human key point detection algorithm to detect objects, DeepSort object tracking and behavior recognition with corresponding IDs assigned, to form a complete detection process; synchronously deploy three-level privacy protection: support the dynamic trajectory of key points, mean blur, and human body mosaic of the preset static background, and realize the privacy protection combining dynamic visualization and static scenarios.

[0038] Standardize and encapsulate the pose detection results output by YOLO11-pose-TensorRT, and transmit them to the cloud in real time through the TLS 1.3 encrypted channel. Data preprocessing includes anomaly filtering, coordinate normalization, and feature encoding. The transmission process supports breakpoint resumption and key frame caching, and adopts a two-way encryption and dynamic key exchange mechanism to ensure the security and integrity of the data.

[0039] S2: Obtain human body motion state information using a millimeter-wave radar, determine the human body motion state information based on the human body motion state information to obtain a human body motion state detection result, and send signal states of different levels to the cloud platform according to the human body motion state detection result.

[0040] Further, the S2 includes: S201: Transmit millimeter-wave signals using the millimeter-wave radar and receive reflected waves, process the echo signals to extract the distance, speed, and angle information of the target, and construct a three-dimensional human body motion trajectory; S202: According to the three-dimensional human body motion trajectory, when the radar beam irradiates the human body target, extract the micro-Doppler features caused by breathing and heartbeat, calculate the radial velocity component using the phase change between consecutive frames, and obtain the human body motion speed through Doppler frequency shift; S203: Perform differential processing on the speed data of consecutive N frames using a sliding window, and combine Kalman filtering to eliminate noise interference to obtain the acceleration of the human body centroid; S204: Based on the obtained speed and acceleration, determine the human body motion state information according to the sudden change of speed and the abnormal peak of acceleration, and obtain a human body motion state detection result; S205: Send signal states of different levels to the cloud platform according to the human body motion state detection result.

[0041] Even further, the sending signal states of different levels to the cloud platform according to the human body motion state detection result includes: When the speed exceeds 2 m / s, it is determined that there is a mild fall behavior, and signal 1 is sent to the cloud; if the acceleration exceeds 3 m / s², it is determined that there is a moderate fall behavior, and signal 2 is sent to the cloud; when it is detected that the speed sudden change > 8 m / s and the acceleration continuously > 9.8 m / s² for more than 0.5 seconds, it is determined that there is a severe fall behavior, and signal 3 is sent immediately.

[0042] In an embodiment of the present application, the millimeter-wave radar is hung 2.5 meters above the ground on the roof, and the speed and acceleration of the human body are calculated through the time difference and Doppler frequency shift of transmitting millimeter-wave signals and receiving reflected waves. First, the echo signals are processed to extract the distance, speed, and angle information of the target, and a three-dimensional human body motion trajectory is constructed.

[0043] When the radar beam irradiates the human body target, the system first extracts the micro-Doppler features caused by breathing and heartbeat, calculates the radial velocity component using the phase change between consecutive frames, and then obtains the human body motion speed through the Doppler frequency shift formula. The calculation formula is as follows:

[0044] In the formula, is the wavelength, is the frequency shift amount.

[0045] For acceleration calculation, the system uses a sliding window to perform differential processing on the velocity data of consecutive N frames, combines Kalman filtering to eliminate noise interference, and finally outputs the three-dimensional acceleration vector of the human body centroid , and the calculation formula is as follows:

[0046] In the formula, is the velocity, is the time, is the derivative operator.

[0047] When the velocity exceeds 2 m / s, signal 1 is sent to the cloud. If the acceleration exceeds 3 m / s 2 , signal 2 is sent to the cloud. When the characteristics of free fall are detected (such as a sudden change in velocity > 8 m / s and the acceleration continuously > 9.8 m / s² for more than 0.5 seconds), signal 3 is immediately sent.

[0048] S3: Use a blood oxygen bracelet to obtain human physiological data, identify sudden physiological abnormalities, obtain physiological abnormality data, and send different levels of signal states to the cloud platform according to the physiological abnormality data.

[0049] Furthermore, in S3, sending different levels of signal states to the cloud platform according to the physiological abnormality data includes: When it is detected that the blood oxygen saturation is lower than 90%, it is determined that there is a moderate fall behavior, and signal 2 is sent to the cloud; if the blood oxygen saturation is lower than 85%, it is determined that there is a severe fall behavior, and signal 3 is sent to the cloud; when the blood oxygen saturation is maintained at 95% - 98% and the heart rate variability index is normal, it is determined that there is a mild fall behavior in the human body, and signal 1 is sent to the cloud.

[0050] In an embodiment of the present application, a blood oxygen bracelet is used to collect the human blood oxygen saturation and heart rate variability index through a photoplethysmography (PPG) sensor, and based on a multi-level threshold mechanism, data is transmitted to the cloud in a hierarchical manner. The bracelet is built-in with dual-wavelength light-emitting diodes (660 nm red light and 940 nm infrared light) and a high-sensitivity photodetector. After self-check calibration is completed during the device initialization stage, the absorption characteristics of different wavelengths of light by human tissues are continuously collected. By analyzing the ratio of the AC component to the DC component in the PPG signal in real time, the calculation formula for the blood oxygen saturation ( ) is as follows:

[0051] Among them, , are the device calibration coefficients, and are the AC and DC components of red light respectively, and are the AC and DC components of infrared light respectively.

[0052] During the acquisition process, an improved three-finger digital filtering algorithm is used to eliminate motion noise and ambient light interference, ensuring stable and reliable measurement results in both resting and moving states. When the blood oxygen saturation is detected to be lower than 90%, the system sends signal 2 to the cloud. If the blood oxygen saturation further drops below 85%, signal 3 is immediately sent. In the normal monitoring state, when the blood oxygen saturation is maintained between 95% - 98% and the heart rate variability index is normal, the smart bracelet only transmits signal 1. This design significantly reduces network load and device power consumption while ensuring the timely transmission of key physiological indicators by dynamically adjusting the data transmission volume and frequency.

[0053] S4: Use the cloud platform to synchronize and fuse the received signal states of different levels, perform fall behavior classification and early warning based on the fused data, and trigger a classification alarm mechanism based on the threshold method according to the fall behavior early warning level.

[0054] Furthermore, the S4 specifically includes: The cloud platform receives the signal states from different detection devices, starts the cloud decision-making method, counts the state outputs of the detection devices, selects the signal state with the most occurrences as the preliminary decision, performs fall behavior classification and early warning based on the preliminary decision, and based on the threshold method, if the fall behavior classification and early warning is a low-risk warning, use local voice alarm; if the fall behavior classification and early warning is a medium-high risk warning, use a three-level alarm of text message alarm, phone call alarm, and APP alarm.

[0055] Even further, if the counted state outputs of the detection devices are different, a weighted decision-making process is adopted, including: C1: Assign different weights to the number of signal states corresponding to the detection devices; C2: Based on the assigned weights and the number of signal states corresponding to the detection devices, obtain the fall determination result score; C3: Perform classification alarm according to the fall determination result score.

[0056] In one embodiment of the present application, when receiving signal states from different terminal devices, the cloud decision-making method is started, the state inputs of three sensors are counted, and the state with the most occurrences is selected as the preliminary decision. When the cloud platform receives and counts that signal 1 appears only once, no warning is triggered; if the cloud platform receives and counts two or three signal 1s from different detection devices, a low-risk warning is triggered and an audible alarm is given; if the cloud platform receives and counts two or three signal 2s from different detection devices, a medium-risk warning is triggered and an SMS alarm is sent; if the cloud platform receives and counts two or three signal 3s from different detection devices, a high-risk warning is triggered and an alarm is given via phone call and APP. If there is a tie, the weighted decision-making process is entered. During the weighted decision-making process, first, a weight of 0.4 is assigned to the millimeter-wave radar, the weight of the blood oxygen bracelet is 0.3, and the weight of the posture detection is 0.3. The final score is the sum of the states of each terminal device multiplied by their corresponding weights. Since the millimeter-wave radar can clearly capture the rapid deceleration and severe tilt of the human body in real time and is easy to identify, the weight of the millimeter-wave radar is the largest. The weight of the radar state is the highest to ensure the decisive role of the motion data; the blood oxygen state is the second to supplement the reliability of the vital signs; the posture state is the lowest and serves as an auxiliary verification means. When the threshold Score < 1.8, an audible alarm is triggered; if 1.8 ≤ Score < 2.5, an alarm is triggered via APP and SMS; when Score ≥ 2.5, an emergency phone call alarm is triggered. To reduce the false alarm rate, a dynamic adjustment mechanism is adopted. If a certain sensor has a high false alarm rate, such as the posture detection, its weight can be temporarily reduced (such as reduced to 0.2) to improve the overall reliability of the system. When the false alarm rate of the posture detection increases due to environmental interference or frequent bending of the user, its weight is reduced to minimize its impact on the final decision.

[0057] Embodiment 2: The present application provides a fall behavior determination and warning system based on multimodal data. This system can be seen Figure 2 , Figure 2 which is a schematic structural diagram of a fall behavior determination and warning system based on multimodal data provided by an embodiment of the present application, including: User terminal device, including several night vision rotatable cameras deployed at high places in the house, a Jeson Nano processor, a millimeter-wave radar, and a blood oxygen bracelet. The Jeson Nano processor receives the video collected by the night vision rotatable cameras through connecting to the local area network and using the RTMP video transmission protocol. The millimeter-wave radar collects human motion state information, and the blood oxygen bracelet collects human physiological data; Cloud platform, which receives and stores the multimodal data transmitted by the user terminal device, fuses and analyzes the multimodal data, judges the fall behavior through a fall behavior determination model, and triggers the alarm module based on the fall risk level; The alarm module includes a local voice alarm module, a text message and mobile APP push module, and a module for notifying emergency contacts. The alarm module triggers different alarms based on the fall risk level determined by the cloud platform.

[0058] In an embodiment of the present application, the camera is set at a higher position in the house to monitor the user's posture, and perform key point detection, target tracking, and human posture judgment on the human body according to the deep learning algorithm; wherein the millimeter wave radar is set at the same height as the camera in the house, one to two meters away.

[0059] A method and system for fall behavior determination and early warning based on multi-modal data provided by the present application, through advanced biosensing technology, computer perception technology, deep learning algorithm, and cloud technology, uses a smart bracelet to monitor physiological indicators such as blood oxygen and heart rate of an individual in real time, as well as the movement amplitude and speed parameters of the human body collected by the millimeter wave radar. Through the analysis of the parameter indicators, the system can judge the movement state and physical state of the human body. At the same time, the camera is used to detect the individual posture results in real time, and a multi-source information fusion mechanism is combined with an intelligent algorithm, so that the system is applicable to a variety of complex environments, covering various scenarios such as home care for the elderly, public activity areas, and medical institutions. And the data of the three sensors are converted into simple states of 1, 2, or 3 as inputs and transmitted to the cloud, and the cloud controls different alarm methods based on these states. Through the fusion of multi-modal data and simple decision logic, rapid and accurate detection of fall events is achieved, improving the reliability and practicality of the system. At the same time, the simplified data processing method also reduces the data transmission volume and processing complexity, while being able to achieve effective fall detection and alarm control. According to different fall detections, different levels of alarms are activated, while maintaining detection accuracy, reducing the false alarm rate, improving the adaptability in different scenarios, and effectively reducing the risk of personal injury suffered by users due to fall behavior.

[0060] It should be noted that those of ordinary skill in the art will realize that the embodiments described here are to help readers understand the principles of the present application, and it should be understood that the protection scope of the present application is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations that do not depart from the essence of the present application based on the technical revelations disclosed in the present application, and these deformations and combinations are still within the protection scope of the present application.

Claims

1. A fall behavior determination and warning method based on multimodal data, characterized in that, Including: S1: Obtain a video stream, analyze the human pose in the video stream using a Jeson Nano processor to obtain a human pose detection result, and send signal states of different levels to the cloud platform according to the human pose detection result; S2: Use a millimeter-wave radar to obtain human motion state information, determine the human motion state information based on the human motion state information to obtain a human motion state detection result, and send signal states of different levels to the cloud platform according to the human motion state detection result; S3: Use a blood oxygen bracelet to obtain human physiological data, identify sudden physiological abnormalities to obtain physiological abnormality data, and send signal states of different levels to the cloud platform according to the physiological abnormality data; S4: The cloud platform synchronizes and fuses the received signal states of different levels in time, performs fall behavior classification warning based on the fused data, and triggers a classification alarm mechanism based on the threshold method according to the fall behavior warning level.

2. The fall behavior determination and warning method based on multi-modal data according to claim 1, characterized in that The analysis of the human pose in the video stream using the Jeson Nano processor to obtain human pose feature information includes: A1: Use the NMS algorithm to filter the video images of overlapping targets in the video images collected by the camera; A2: Use the YOLOv11-Pose human key point detection algorithm to detect the targets in the video stream to obtain a target detection result including the center point coordinates of the target box, the length and width of the target box, the target detection result, the key point information including key point coordinates and confidence, the target category, and the confidence of the target detection result; A3: Use the DeepSort target tracking algorithm to fuse the target ID and target box information into the target detection result to obtain a fused target detection result; A4: Construct a human pose determination algorithm, based on the key point information in the fused target detection result, judge the human pose change according to the relative position of the key points to obtain the human pose detection result.

3. The fall behavior determination and warning method based on multimodal data according to claim 2, characterized in that The judgment of the human pose change according to the relative position of the key points to obtain the human pose detection information includes: B1: According to the key point information in the fused target detection result, extract the key points corresponding to the shoulders, hips, knees, and ankles in the video image; B2: Based on the key points corresponding to the extracted hips, knees, and ankles, calculate the angles between the hips, knees, and ankles; B3: When the angles between the hips, knees, and ankles exceed the normal range of 70° to 150°, send signal 1 to the cloud platform, and based on the key points corresponding to the extracted shoulders and hips, calculate the relative offset angle between the shoulders and hips; B4: When the relative offset angle between the shoulders and hips exceeds 35°, calculate the degree of elbow bending; B5: When the degree of elbow bending does not exceed 60°, enhance the confidence of the detection box of the YOLOv11-Pose human key point detection algorithm; B6: Use the YOLOv11-Pose human key point detection algorithm with enhanced confidence to detect key points, and based on the key points corresponding to the extracted shoulders and hips, calculate the tilt angle between the shoulders and hips; B7: Obtain the human pose detection result according to the calculated tilt angle between the shoulders and hips.

4. The fall behavior determination and warning method based on multi-modal data according to claim 3, wherein, According to the human body posture detection results, send signal states of different levels to the cloud platform, including: When the tilt angle is between 35° and 50° and the body posture returns to normal within 1.5 seconds, it is determined that there is a mild fall behavior of the human body, and signal 1 is sent to the cloud platform; when the tilt angle is between 50° and 70° and does not recover for more than 2 seconds, it is determined that there is a moderate fall behavior of the human body, and signal 2 is sent to the cloud platform; when the tilt angle is greater than 70° and does not recover for 3 seconds continuously, it is determined that there is a severe fall behavior of the human body, and signal 3 is sent to the cloud platform.

5. The fall behavior determination and warning method based on multimodal data according to claim 1, wherein The S2 includes: S201: Transmit millimeter-wave signals by the millimeter-wave radar and receive the reflected waves, process the echo signals to extract the distance, speed and angle information of the target, and construct the three-dimensional motion trajectory of the human body; S202: According to the three-dimensional motion trajectory of the human body, when the radar beam irradiates the human body target, extract the micro-Doppler features caused by breathing and heartbeat, calculate the radial velocity component using the phase change between consecutive frames, and obtain the human body motion speed through the Doppler frequency shift; S203: Use a sliding window to perform differential processing on the speed data of consecutive N frames, and combine Kalman filtering to eliminate noise interference to obtain the acceleration of the human body centroid; S204: Based on the obtained speed and acceleration, determine the human body motion state information to obtain the human body motion state detection result; S205: According to the human body motion state detection results, send signal states of different levels to the cloud platform.

6. The fall behavior determination and warning method based on multi-modal data according to claim 5, characterized in that, According to the human body motion state detection results, send signal states of different levels to the cloud platform, including: When the speed exceeds 2m / s, it is determined that there is a mild fall behavior, and signal 1 is sent to the cloud; if the acceleration exceeds 3m / s², it is determined that there is a moderate fall behavior, and signal 2 is sent to the cloud; when a speed mutation > 8m / s and the acceleration continuously > 9.8m / s² for more than 0.5 seconds is detected, it is determined that there is a severe fall behavior, and signal 3 is sent immediately.

7. The fall behavior determination and warning method based on multimodal data according to claim 1, wherein In S3, according to the physiological abnormality data, send signal states of different levels to the cloud platform, including: When the detected blood oxygen saturation is lower than 90%, it is determined that there is a moderate fall behavior, and signal 2 is sent to the cloud; if the blood oxygen saturation is lower than 85%, it is determined that there is a severe fall behavior, and signal 3 is sent to the cloud; when the blood oxygen saturation is maintained between 95% and 98% and the heart rate variability index is normal, it is determined that there is a mild fall behavior of the human body, and signal 1 is sent to the cloud.

8. The fall behavior determination and warning method based on multi-modal data according to claim 1, characterized in that The S4 specifically includes: The cloud platform receives the signal states from different detection devices, starts the cloud decision-making method, counts the state outputs of the detection devices, selects the signal state with the most occurrences as the preliminary decision, performs fall behavior classification and early warning according to the preliminary decision, and based on the threshold method, if the fall behavior classification and early warning is a low-risk early warning, local voice alarm is adopted; if the fall behavior classification and early warning is a medium-high risk early warning, a three-level alarm of SMS alarm, telephone alarm and APP alarm is adopted.

9. The fall behavior determination and warning method based on multimodal data according to claim 8, characterized in that If the counted state outputs of the detection devices are different, a weighted decision-making process is adopted, including: C1: Assign different weights to the number of signal states corresponding to the detection devices; C2: Obtain a fall determination result score based on the assigned weight and the number of signal states corresponding to the detection device; C3: Conduct hierarchical alarm according to the fall determination result score.

10. A fall behavior determination and early warning system based on multimodal data, characterized in that, It includes: A user terminal device, including several night vision rotatable cameras deployed at high places in the house, a Jeson Nano processor, a millimeter wave radar, and a blood oxygen bracelet. The Jeson Nano processor receives the video collected by the night vision rotatable cameras through connecting to a local area network and adopting the RTMP video transmission protocol. The millimeter wave radar collects human body movement state information, and the blood oxygen bracelet collects human body physiological data; A cloud platform, which receives and stores the multi-modal data transmitted by the user terminal device, fuses and analyzes the multi-modal data, conducts fall behavior judgment through a fall behavior determination model, and triggers an alarm module based on the fall risk level; An alarm module, including a local voice alarm module, a text message and mobile APP push module, and a module for notifying emergency contacts. The alarm module triggers different alarms based on the fall risk level determined by the cloud platform.

Images