Treadmill anti-falling system based on multi-modal fusion detection and control method thereof

Abstract

The invention relates to the field of running machine anti-falling braking and injury protection, in particular to a running machine anti-falling system based on multi-modal fusion detection and a control method thereof.A multi-modal sensing module is adopted for collecting user motion data in real time, and three-level trigger judgment is conducted by means of an inertial sensor set and a visual recognition unit; a 4G Cat.1 communication module of the system transmits alarm information with a timestamp, attitude data and an image thumbnail to a remote management terminal in real time; the self-checking diagnosis unit automatically checks the zero drift of the sensor, the focusing precision of the camera and the quality of a communication link. Through multi-dimensional detection, the false alarm rate is reduced to 1% or below, the response time is shortened by 40% compared with that of a traditional scheme, low-power-consumption operation is supported (the standby current is smaller than 10 mA), a mainstream treadmill protocol is also compatible, and the exercise safety and the equipment reliability are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of treadmill anti-fall braking and injury protection, and specifically to a treadmill anti-fall system based on multimodal fusion detection and a control method thereof, which is suitable for emergency braking of the treadmill at the moment the treadmill user falls to reduce fall injuries. Background Art

[0002] Treadmills are a common piece of fitness equipment, widely used in homes and gyms. They simulate the running environment, providing a convenient way for people to exercise. However, while using a treadmill, users may lose their balance and fall due to various reasons, such as improper speed adjustment, physical fatigue, or lack of concentration. Once a fall occurs, the treadmill belt continues to run at high speed, potentially causing serious secondary injuries such as fractures, sprains, and abrasions, which can even be life-threatening.

[0003] Although existing treadmills are constantly improving in function and design, they still have shortcomings in fall protection. Although some technologies have attempted to address this problem, such as Chinese patent 202310699031.X, which discloses a method of setting a vibration sensor on the handrail. When the user loses balance and holds the handrail, the sensor triggers the emergency brake device; Chinese patent 202121612481.3 discloses a method of using a fall emergency stop structure and an elastic buffer protection structure. When the user falls, the pressure sensor is used to control the treadmill to stop suddenly, and an elastic protective belt is used for buffering; or advanced sensor technology and pattern recognition algorithms are used to achieve real-time monitoring and early warning of treadmill user's fall behavior [Xu Weijun. Research on anti-fall recognition of human motion posture on treadmill [D]. Shanghai Jiaotong University, 2015. DOI: 10.27307 / d.cnki.gsjtu.2015.000255.], these technologies still have certain limitations. For example, the sensitivity and accuracy of the sensor are not high enough, which may lead to false triggering or delayed triggering, and lack of follow-up rescue after the fall, which seriously affects the user experience. Summary of the Invention

[0004] In response to the above-mentioned known shortcomings of existing treadmills, the present invention provides a treadmill anti-fall system and its control method based on multimodal fusion detection. Through real-time monitoring and intelligent judgment, it can achieve accurate recognition and rapid response to falling actions, solve the problems of high misjudgment rate, response delay and lack of rescue linkage in traditional anti-fall technology, effectively reduce the risk of secondary injury to users and improve safety. The present invention uses multimodal fusion detection technology, integrating sensors and visual recognition means. The sensor monitors abnormal postures in real time. Once an abnormality is found, it triggers the camera, uses the YOLOv8 model to confirm the image, and then uses the OpenPose algorithm to extract the key points of the human body to accurately judge whether it has fallen. This multi-dimensional and mutually confirmed detection method significantly improves the accuracy of the system's judgment of a person's fall, effectively reduces misjudgment, and improves the performance of the anti-fall system; the constructed "detection-judgment-shutdown" closed-loop system can complete the response process within 0.5 seconds. Compared with the traditional emergency stop button, it can respond to falls more quickly, shut down in time, and reduce the probability of secondary injury. , providing efficient and timely safety protection; after determining a fall, it not only shuts down quickly, but also triggers a buzzer to alert surrounding people, and uses the Internet of Things module to send an alarm message to the management personnel to ensure that the fallen person can get treatment quickly, reduce the risk of secondary bacterial infection of the wound, and comprehensively improve the safety and reliability of use, creating a safer sports environment; in the long run, although the technical application is more complicated, it reduces unnecessary equipment operations caused by misjudgment, avoids frequent deceleration and shutdowns to the treadmill motor and other core components of the damage, helps to extend the overall service life of the treadmill, to a certain extent balances the initial technology investment cost, and achieves a good balance between cost and service life.

[0005] The present invention achieves the above-mentioned object through the following technical solutions: A treadmill anti-fall system based on multimodal fusion detection, comprising:

[0006] The multimodal perception module consists of an inertial sensor group and a visual recognition unit. The inertial sensor group includes at least one accelerometer and gyroscope to collect user motion posture data in real time. The visual recognition unit is configured as a triggerable high-definition camera and is coupled with the YOLOv8 object detection model. Both the inertial sensor group and the visual recognition unit are connected to the data processing module, which transmits the collected sensor data and video data to the data processing module for further analysis and processing.

[0007] The data processing module uses a data fusion algorithm to collaboratively analyze sensor data and visual data. It uses the OpenPose algorithm processing unit to extract the coordinates of 17 key points of the human body for posture analysis. The data fusion unit and the OpenPose algorithm unit are both connected to the multimodal perception module, receiving data from the inertial sensor group and the visual recognition unit. The data processing module transmits the processed results to the execution control module for subsequent control decisions;

[0008] An execution control module includes an emergency stop interface and an audible and visual alarm device interconnected with the treadmill controller. The emergency stop interface is connected to the treadmill's motor control system, and the audible and visual alarm device is directly connected to the execution control module. The execution control module receives control instructions from the data processing module and executes corresponding actions through the emergency stop interface and the audible and visual alarm device.

[0009] The remote communication module integrates the Internet of Things transmission unit, and the system response delay does not exceed 500ms. The Internet of Things transmission unit is connected to the data processing module, receives the processed data and transmits it remotely. At the same time, the remote communication module is also connected to the execution control module to ensure that the alarm information can be sent to the management personnel in time in an emergency.

[0010] The inertial sensor group monitors the acceleration change of the user's center of mass at a sampling frequency of 100 Hz. When the acceleration vector modulus exceeds the preset threshold of 4 m / s 2 When , the visual recognition unit is triggered to start image acquisition.

[0011] The visual recognition unit is configured with a 120° wide-angle lens and an infrared fill light device. After being triggered, it continuously collects 5 seconds of video stream data at a frame rate of 30fps, and realizes human body area positioning and background separation through the YOLOv8 model.

[0012] The OpenPose algorithm processing unit is configured to analyze the three-dimensional coordinates of 17 key points of the human body in real time and establish a key point motion trajectory model. When it is detected that the height of the head key point drops by more than 1.2 meters within 0.3 seconds and the knee joint bending angle exceeds 120 degrees, it is determined to be a fall event.

[0013] The emergency stop interface uses RS-485 or CAN bus protocol to communicate with the treadmill controller. After the emergency stop command is sent, the system completes motor power off and running belt braking within 200ms.

[0014] The IoT transmission unit uses the Air724UG communication module, which supports 4G Cat.1 data transmission. The alarm information includes event timestamps, user posture analysis data, and on-site image thumbnails.

[0015] The system sets up a three-level early warning mechanism: when the sensor detects a level one abnormality, a visual review is initiated; when a level two risk is confirmed, a shutdown instruction is preloaded; and when it is finally determined to be a level three dangerous state, a full system linkage response is executed.

[0016] The data processing module is equipped with a timing alignment unit, which uses the Kalman filter algorithm to synchronize the sensor data with the visual data, establishes a decision model for multimodal feature fusion, and reduces the misjudgment rate to below 1%.

[0017] The system is equipped with a self-diagnosis unit, which automatically checks the sensor accuracy, camera focus function and communication link status when it is first started up every day. In case of abnormality, the status indicator light will prompt maintenance needs.

[0018] A control method for a treadmill anti-fall system based on multimodal fusion detection comprises the following steps:

[0019] (1) The multimodal perception module monitors the user's motion posture data in real time. When the inertial sensor group detects that the acceleration vector modulus exceeds a preset threshold, the visual recognition unit is triggered to start image acquisition;

[0020] (2) The data processing module performs collaborative analysis of sensor data and visual data, and extracts the coordinates of key points of the human body for posture analysis through the OpenPose algorithm;

[0021] (3) When it is detected that the height of the key point of the head drops by more than 1.2 meters within 0.3 seconds and the knee bending angle exceeds 120 degrees, it is determined to be a fall event, and the execution control module sends an emergency stop command, triggers the sound and light alarm device at the same time, and sends an alarm message to the management personnel through the remote communication module.

[0022] The remarkable effects of the present invention are:

[0023] 1. Multimodal collaborative detection improves safety and reliability. Through the dual verification mechanism of inertial sensors and visual recognition, accurate detection is achieved with the false alarm rate reduced to less than 1%. The sensor group monitors acceleration mutations in real time (threshold 4m / s 2 ), combined with the morphological analysis of falling movements by the visual unit (such as sudden drop in head height and changes in joint angles), it effectively distinguishes normal movement interference (such as jumping and sudden stops) from real falling events, and avoids false triggering of a single sensor due to environmental vibration or user habits. The hierarchical response mechanism ensures the efficiency of emergency handling. The system adopts a three-level early warning linkage control: the first-level early warning (sensor trigger) starts visual review, the second-level early warning (OpenPose key point analysis) preloads the shutdown command, and the entire process of shutdown, alarm, and remote notification is completed within 0.5 seconds after the third-level confirmation. Compared with the traditional single threshold trigger solution, the response time is shortened by more than 40%, and the command preloading technology is used to avoid communication delays, ensuring that the running belt braking is completed within 200ms, minimizing the risk of secondary injury to the user.

[0024] 2. Full-scenario adaptability and data integrity: The vision unit, equipped with a wide-angle lens and infrared fill light, accurately captures human outlines in dim environments or complex backgrounds. Alarm information transmitted by the IoT module includes timestamps, posture analysis data, and image thumbnails, providing a multi-dimensional chain of evidence for accident tracing. Furthermore, the system supports 4G Cat.1 network communication, with alarm information transmission latency of less than 2 seconds, meeting the management needs of various scenarios such as gyms and homes.

[0025] Algorithm optimization reduces hardware resource consumption. Using the lightweight YOLOv8 model and the OpenPose keypoint tracking algorithm, the system achieves sub-10W operation on embedded devices and maintains a stable 30fps video stream processing frame rate. A timing alignment unit performs Kalman filtering on multimodal data, reducing redundant computations and enabling smooth system operation on mainstream ARM Cortex-A53 processors, reducing hardware deployment costs.

[0026] 3. Adaptive maintenance ensures long-term stability. The built-in self-diagnosis unit automatically checks the sensor zero drift (accuracy error <0.05g), camera focus accuracy (focus deviation <5% of the frame) and communication link quality (packet loss rate <0.1%) every day. In the event of an abnormality, a three-color LED indicator will indicate the specific fault module (for example, red indicates sensor abnormality, and yellow indicates communication failure). Maintenance personnel can conduct targeted maintenance, and the equipment availability rate is increased to 99.9%.

[0027] 4. Compatibility expansion and energy saving and environmental protection: Through the RS-485 / CAN bus protocol, it is compatible with more than 90% of treadmill brands on the market, without the need to modify the original motor control system; the Internet of Things module supports low power mode (standby current <10mA), and data transmission is activated only when the alarm is triggered. The annual power consumption is less than 2kWh, which meets the green design standards. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic diagram of the overall architecture of the treadmill anti-fall system based on multimodal fusion detection described in the present invention.

[0029] Figure 2 This is a multimodal data fusion flow chart of the treadmill anti-fall system based on multimodal fusion detection described in the present invention.

[0030] Figure 3 This is a timing diagram of the emergency stop control of the treadmill anti-fall system based on multimodal fusion detection described in the present invention.

[0031] Figure 4 This is a workflow diagram of the diagnostic unit of the treadmill anti-fall system based on multimodal fusion detection described in the present invention.

[0032] The symbols in the figure are explained as follows: multimodal perception module 10, data processing module 11, execution control module 12, remote communication module 13, sensor group 101 and triggerable camera 102, data fusion unit 103, OpenPose algorithm unit 104, emergency stop interface 105, sound and light alarm device 106, Internet of Things unit 107, S202 to S205 are steps of an embodiment of the present invention, S301 to S305 are a time flow of the emergency stop control implementation of the present invention, and S401 to S404 are steps of an embodiment of the self-diagnosis unit of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of this invention is further described in detail below through specific embodiments.

[0034] See also Figure 1 The treadmill anti-fall system based on multimodal fusion detection of the present invention comprises a multimodal sensing module 10, a data processing module 11, an execution control module 12 and a remote communication module 13. The specific structure and connection relationship are as follows:

[0035] The multimodal perception module 10 includes a sensor group 101 and a triggerable camera 102. The sensor group 101 uses a high-precision inertial sensor group (such as the MPU6050) with a sampling frequency of 100 Hz, which is used to monitor the acceleration and angular velocity data of human motion in real time. The triggerable camera 102 uses a high-definition camera (such as the Sony IMX series) with a quick start function, which can quickly enter video capture mode after the sensor is triggered.

[0036] The data processing module 11 includes a data fusion unit 103 and an OpenPose algorithm unit 104. The data fusion unit 103 pre-processes the sensor data using an adaptive filtering algorithm to eliminate noise interference and ensure data accuracy and reliability. The OpenPose algorithm unit 104 deploys the OpenPose human pose estimation algorithm to process video frames captured by the camera in real time and extract key human point information, including coordinate and angle data for parts such as the head and knee joints.

[0037] The execution control module 12 includes an emergency stop interface 105 and an audible and visual alarm device 106. The emergency stop interface 105 is connected to the treadmill's motor control system and can quickly send a stop command when a fall risk is detected, ensuring that the treadbelt is fully braked within 500ms. The audible and visual alarm device 106, equipped with a tweeter and a high-brightness LED light, emits audible and visual alarm signals when a fall risk is detected, alerting nearby personnel.

[0038] The remote communication module 13 is built based on the Internet of Things unit 107. The Internet of Things unit 107 uses a 4G Cat.1 communication module and uses the MQTT protocol to push alarm information including timestamps, posture data, image thumbnails, etc. to the cloud platform in real time, realizing convenient remote monitoring and management.

[0039] Example 2

[0040] See also Figure 2 This embodiment is an implementation method of the multimodal data fusion process of the present invention. The multimodal data fusion process is divided into three levels of trigger logic: Level 1 trigger: The system monitors the acceleration data collected by the sensor group in real time. When a sudden acceleration change (threshold of 4m / s) is detected 2 ) S202, the first-level response is triggered and the state of second-level trigger preparation is entered. Second-level trigger: After receiving the first-level trigger signal, the camera quickly starts the video acquisition function S203, and captures a 5-second video clip at a frame rate of 30fps for subsequent visual review. Third-level judgment: The data processing module performs OpenPose key point analysis S204 on the captured video frames, focusing on detecting the drop in head height and changes in knee joint angles. If the head height difference ΔH reaches or exceeds 1.2 meters within 0.3 seconds, and the knee joint angle θ is greater than or equal to 120°, it is determined to be a fall risk and enters the execution control stage. Branch logic: If the acceleration does not exceed the threshold, the system returns to the initial monitoring state S205 and continues to monitor the sensor data in real time.

[0041] The present invention detects key points of the human body through the OpenPose algorithm and implements fall judgment based on the key point information. Normal running posture detection: In the normal running state, the OpenPose algorithm extracts the connection line of the key points of the human body and marks the coordinate positions of the head and knee joints. By analyzing the relative positions and motion trajectories of the key points, a reference model of the normal running posture is established. Fall process detection: During the fall process, the trajectory of the key points changes significantly. The head height drops sharply from 1.8 meters in the normal running state to 0.6 meters, and the head height change curve shows a rapid downward trend. At the same time, the knee joint angle measurement diagram shows the angle change of the line connecting the femur and tibia. The knee joint angle θ may reach or exceed 120° at the moment of falling. Judgment logic formula: The present invention adopts the following judgment logic formula: head height difference ΔH ≥ 1.2 meters (within 0.3 seconds), knee joint angle θ ≥ 120°. When the above two conditions are met at the same time, it is determined that a fall event has occurred and the system enters the emergency response stage.

[0042] Example 3

[0043] See also Figure 3, this embodiment is the emergency stop control implementation method of the present invention. The emergency stop control timing diagram of the present invention describes in detail the whole process from sensor triggering to complete braking of the running belt. In the key nodes of the time axis, at T0 (0ms), the sensor detects a sudden change in acceleration and triggers a first-level response S301, while outputting a high-level pulse signal; then at T1 (50ms), the camera starts video acquisition to record the fall process S302, during which it receives trigger signals from T0 to T1 and maintains a high-level activation state; at T2 (300ms), the OpenPose algorithm completes the key point analysis and outputs the fall judgment result S303; at T3 (350ms), the execution control module sends a shutdown instruction in a standard protocol format to the treadmill motor control system via the CAN bus S304; finally, at T4 (500ms), the running belt is completely braked and stopped S305, and the sound and light alarm device is triggered at the same time. During the entire process, the high-level pulse signal generated by the sensor at time T0 serves as an event trigger mark. The camera maintains a high level during T0-T1, indicating that the video acquisition function continues to operate. The CAN bus command waveform sent at time T3 complies with the standard communication protocol to ensure fast and reliable command transmission.

[0044] The present invention uses the Internet of Things unit to achieve real-time transmission of alarm information to the remote management terminal. The data packet covers a timestamp that records the specific time when the fall event occurred to facilitate subsequent retrospective analysis, acceleration and angular velocity data collected by the sensor, and posture data of key point coordinates and angle information extracted by the OpenPose algorithm, as well as image thumbnails generated by extracting key frames from video clips collected by the camera to intuitively display the scene of the fall. In terms of communication, a 4G Cat.1 communication module with low power consumption, high bandwidth and the ability to meet real-time data transmission requirements is adopted, and combined with the MQTT protocol that can ensure data transmission reliability and stability, efficient data transmission is achieved. The alarm information starts from the local device, is transmitted to the cloud platform via the 4G base station, and is then forwarded by the cloud platform to management terminals such as mobile phone APP and monitoring center. The entire transmission can be completed in a short time to ensure that management personnel receive alarm information in a timely manner and take countermeasures.

[0045] Example 4

[0046] See also Figure 4This embodiment illustrates the implementation method of the self-diagnosis unit of the present invention. The self-diagnosis unit of the present invention can monitor the operating status of key system components in real time to ensure system stability and reliability. The system self-diagnosis process is implemented through three key detection steps: the inertial sensor group regularly performs zero-point calibration to detect zero-point drift (S401). When the drift value exceeds the 0.05g threshold, maintenance mode is triggered; the camera verifies focus accuracy by shooting a standard test pattern (S402). If the frame deviation exceeds 5%, a focus anomaly is determined; and the communication link detects packet loss rate by regularly sending test data packets (S403). When the packet loss rate exceeds the 0.1% threshold, network troubleshooting is triggered. The self-diagnosis unit uses a three-color LED light for status indication (S404): red indicates a serious fault requiring immediate repair, yellow indicates a minor anomaly and recommended maintenance, and green indicates a healthy system. This self-diagnosis mechanism is combined with human fall detection technology that provides real-time monitoring, rapid judgment, and emergency response. Remote alarms and intelligent self-diagnosis are implemented through the Internet of Things, significantly improving system security and intelligence.

[0047] The above descriptions and drawings are specific embodiments of the present invention, and the scope of all rights of the present invention shall be subject to the appended claims. Any changes or modifications that can be easily conceived by those skilled in the art of the present invention may be included in the scope of the claims defined by the present invention.

Claims

1. A treadmill anti-fall system based on multimodal fusion detection, characterized in that: include: The multimodal perception module consists of an inertial sensor group and a visual recognition unit. The inertial sensor group includes at least one accelerometer and gyroscope to collect user motion posture data in real time. The visual recognition unit is configured as a triggerable high-definition camera and is coupled with the YOLOv8 object detection model. Both the inertial sensor group and the visual recognition unit are connected to the data processing module, which transmits the collected sensor data and video data to the data processing module for further analysis and processing. The data processing module uses a data fusion algorithm to collaboratively analyze sensor data and visual data. It uses the OpenPose algorithm processing unit to extract the coordinates of 17 key points of the human body for posture analysis. The data fusion unit and the OpenPose algorithm unit are both connected to the multimodal perception module, receiving data from the inertial sensor group and the visual recognition unit. The data processing module transmits the processed results to the execution control module for subsequent control decisions; An execution control module includes an emergency stop interface and an audible and visual alarm device interconnected with the treadmill controller. The emergency stop interface is connected to the treadmill's motor control system, and the audible and visual alarm device is directly connected to the execution control module. The execution control module receives control instructions from the data processing module and executes corresponding actions through the emergency stop interface and the audible and visual alarm device. The remote communication module integrates the Internet of Things transmission unit, and the system response delay does not exceed 500ms. The Internet of Things transmission unit is connected to the data processing module, receives the processed data and transmits it remotely. At the same time, the remote communication module is also connected to the execution control module to ensure that the alarm information can be sent to the management personnel in time in an emergency.

2. The treadmill anti-fall system according to claim 1, characterized in that: The inertial sensor group monitors the acceleration change of the user's center of mass at a sampling frequency of 100 Hz. When the acceleration vector modulus exceeds the preset threshold of 4 m / s 2 When , the visual recognition unit is triggered to start image acquisition.

3. The treadmill anti-fall system according to claim 1, characterized in that: The visual recognition unit is configured with a 120° wide-angle lens and an infrared fill light device. After being triggered, it continuously collects 5 seconds of video stream data at a frame rate of 30fps, and realizes human body area positioning and background separation through the YOLOv8 model.

4. The treadmill anti-fall system according to claim 1, characterized in that: The OpenPose algorithm processing unit is configured to analyze the three-dimensional coordinates of 17 key points of the human body in real time and establish a key point motion trajectory model. When it is detected that the height of the head key point drops by more than 1.2 meters within 0.3 seconds and the knee joint bending angle exceeds 120 degrees, it is determined to be a fall event.

5. The treadmill anti-fall system according to claim 1, characterized in that: The emergency stop interface uses RS-485 or CAN bus protocol to communicate with the treadmill controller. After the emergency stop command is sent, the system completes motor power off and running belt braking within 200ms.

6. The treadmill anti-fall system according to claim 1, characterized in that: The IoT transmission unit uses the Air724UG communication module, which supports 4G Cat.1 data transmission. The alarm information includes event timestamps, user posture analysis data, and on-site image thumbnails.

7. The treadmill anti-fall system according to claim 1, characterized in that: The system sets up a three-level early warning mechanism: when the sensor detects a level one abnormality, a visual review is initiated; when a level two risk is confirmed, a shutdown instruction is preloaded; and when it is finally determined to be a level three dangerous state, a full system linkage response is executed.

8. The treadmill anti-fall system according to claim 1, characterized in that: The data processing module is equipped with a timing alignment unit, which uses the Kalman filter algorithm to synchronize the sensor data with the visual data, establishes a decision model for multimodal feature fusion, and reduces the misjudgment rate to below 1%.

9. The treadmill anti-fall system according to claim 1, characterized in that: The system is equipped with a self-diagnosis unit, which automatically checks the sensor accuracy, camera focus function and communication link status when it is first started up every day. In case of abnormality, the status indicator light will prompt maintenance needs.

10. A control method for a treadmill anti-fall system based on multimodal fusion detection, characterized in that: The following steps are involved: (1) The multimodal perception module monitors the user's motion posture data in real time. When the inertial sensor group detects that the acceleration vector modulus exceeds a preset threshold, the visual recognition unit is triggered to start image acquisition; (2) The data processing module performs collaborative analysis of sensor data and visual data, and extracts the coordinates of key points of the human body for posture analysis through the OpenPose algorithm; (3) When it is detected that the height of the key point of the head drops by more than 1.2 meters within 0.3 seconds and the knee bending angle exceeds 120 degrees, it is determined to be a fall event, and the execution control module sends an emergency stop command, triggers the sound and light alarm device at the same time, and sends an alarm message to the management personnel through the remote communication module.

Images