Intelligent anti-falling system based on multi-mode perception

Abstract

The invention provides an intelligent anti-falling system based on multi-mode perception, and relates to the technical field of medical monitoring, a mattress is firstly divided into a back area, a middle area and a leg area, a pressure monitoring threshold value is set, and the pressure of each area, a user state and an electromyographic signal are obtained; when the pressure of each area is greater than a monitoring threshold value, the pressure of the middle area is maximum and the user is identified to be awake for the first time, entering an on-bed preparation period, analyzing whether the lower limb joint movement is completed or not during the period, ending the period and counting the duration when the pressure of the back and leg areas is less than the threshold value and the pressure of the middle area is greater than the threshold value, and judging to enter a bedside adaptation period; during the period, whether neck rotation, natural drooping of the two feet and stretching of the upper limbs are completed or not is judged, when the pressure of each area is smaller than a threshold value, the period is ended, the duration is counted, then when it is confirmed that the two feet stand front and back through knee coordinates, the standing stable period is started, and the duration of the period, the synchronous broadcasting duration and the corresponding action conditions are counted.

Description

Technical Field

[0001] This invention relates to the field of medical monitoring technology, specifically to an intelligent fall prevention system based on multimodal perception. Background Technology

[0002] Existing fall prevention technologies mainly include several implementation schemes: First, video surveillance solutions, which use visual recognition technology to monitor human posture and movement trajectory, and can intuitively capture the moment of a fall; second, millimeter-wave radar / ultrasound solutions, which have the advantage of non-contact monitoring and can penetrate clothing and other obstructions, making them suitable for scenarios with high privacy protection requirements; third, pressure sensing solutions, which use pressure sensors deployed on mattresses and the floor to detect changes in human position and provide preliminary warnings for actions such as getting up or getting out of bed; and fourth, wearable device solutions, which rely on accelerometers, gyroscopes, etc., to monitor human movement and can actively alarm when a fall is triggered. All of the above-mentioned existing technologies can play a certain role in fall prevention and early warning in specific scenarios, providing basic protection for the safety of the elderly.

[0003] However, existing technologies still have significant room for improvement: On the one hand, some single-sensor fall prevention solutions rely on data from a single sensor for judgment, making them susceptible to environmental interference and resulting in a high false alarm rate. Related research data shows that the false alarm rate is generally over 32%, affecting the actual user experience. On the other hand, medical research indicates that orthostatic hypotension is one of the core causes of falls during the process of getting out of bed in the elderly. Following the standard getting-out procedure clearly recommended by professional materials—"lying flat for 30 seconds, sitting up for 30 seconds, standing for 30 seconds"—can effectively reduce the risk of falls caused by orthostatic hypotension. However, most existing fall prevention patents only focus on real-time monitoring or post-fall alarms, without explicitly addressing the accurate detection and guidance of the execution of this standard getting-out procedure. This fails to prevent the risk of falls caused by improper getting out of bed from the source.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent fall prevention system based on multimodal perception to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A multimodal perception-based intelligent fall prevention system includes:

[0008] The motion determination module is used to divide the mattress into three areas: back, middle, and legs. It acquires the pressure of each area in real time and sets corresponding monitoring thresholds. It monitors the user's status and electromyography signals in real time. It uses infrared sensing technology combined with a deep learning model to obtain the coordinates of the knee, foot, and eye. Based on the knee and foot coordinates, it determines whether the feet have been naturally lowered. Based on the eye coordinates, it determines whether the neck has been rotated. Based on the knee coordinates, it determines whether the feet are standing forward and backward. Based on the electromyography signals, it determines whether the upper limb extension movement has been completed.

[0009] The pressure analysis module is used to determine the start of the bed preparation period when the pressure in each area is greater than the monitoring threshold, the pressure in the middle area is the greatest, and the user is identified as awake for the first time. During the bed preparation period, the pressure in the leg area is analyzed to determine whether the lower limb joint movement is completed. When the pressure in the back and leg areas is less than the corresponding monitoring threshold, and the pressure in the middle area is greater than the corresponding monitoring threshold, the bed preparation period is determined to end and the duration of the period is calculated.

[0010] The broadcast prompt module is used to determine whether the bedside adaptation period has ended after the bed preparation period, and to determine whether the neck rotation, natural descent of both feet, and upper limb extension movements have been completed. When the pressure in each area is less than the corresponding monitoring threshold, the bedside adaptation period is determined to have ended and the duration of this period is recorded. When the bedside adaptation period ends and the feet are standing forward and backward, the standing stability period is determined to have ended. When the feet are not standing forward and backward, the standing stability period is determined to have ended and the duration of this period is recorded. The module broadcasts prompts on the duration of the bed preparation period, the bedside adaptation period, and the standing stability period, as well as whether the corresponding movements have been completed.

[0011] Furthermore, the method of dividing the mattress into three areas—back, middle, and leg—is as follows:

[0012] Using the length of the mattress as the longitudinal direction, and starting from the end of the mattress closest to the headboard, divide the mattress longitudinally into back, middle, and leg areas according to the user's height. The back area corresponds to the shoulders and back to the waist, the middle area corresponds to the hips, and the leg area corresponds to the thighs to the feet.

[0013] Furthermore, the method for obtaining the coordinates of the knee, foot, and eye using infrared sensing technology combined with a deep learning model is as follows:

[0014] Infrared thermal imaging sample data for model training is obtained. The sample data includes infrared images of the user's knees, feet and eyes, and the temperature differences of each part are preserved. All sample images are uniformly cropped to a preset fixed size, and the areas of the knees, feet and eyes are marked with rectangles. The eye area is marked with the pupil and the bright area around the eye, the knee area is marked with the joint and the surrounding muscle thermal area, and the foot area is marked with the sole and heel thermal area.

[0015] A deep learning target detection model is used, and the above-annotated infrared thermal imaging sample data is input for training. The two-dimensional infrared thermal imaging device is installed at a preset fixed angle, specifically, the two-dimensional infrared thermal imaging device is at a 45°±5° downward angle to the mattress plane, and the installation height is 1.5m-1.8m above the mattress plane. The center pixel coordinates of the knee, foot, and eye regions are obtained by combining the deep learning target detection model, and are used as the output knee, foot, and eye coordinates.

[0016] Furthermore, the method for determining whether a natural foot drop has been completed based on knee and foot coordinates is as follows:

[0017] A pre-set distance stability threshold for natural foot descent is defined as the actual knee-to-foot distance when the feet are naturally descent in a standard sitting posture. Based on the knee and foot coordinates, the coordinate ordinate values ​​of the knee and foot on the same side on both the left and right sides are extracted, and the absolute value of the difference between the two is calculated to obtain the lateral knee-to-foot distance on one side. When the lateral knee-to-foot distance on both the left and right sides is less than the distance stability threshold, the natural foot descent is considered complete.

[0018] Furthermore, the method for determining whether neck rotation has been completed based on eye coordinates is as follows:

[0019] Two eye coordinates are collected in real time, and the mean of the two eye coordinates is calculated at the same time to serve as the center coordinates of the eyes. A time series dataset of the center coordinates is constructed, and the maximum and minimum values ​​of the horizontal values ​​of the center coordinates are extracted from it. The absolute value of the difference between the maximum and minimum values ​​is calculated as the torsion angle.

[0020] A preset torsion angle deviation threshold is set. When the torsion angle exceeds the preset torsion angle deviation threshold, the calibration of neck rotation is completed.

[0021] Furthermore, the method for determining whether the stance is with feet staggered based on knee coordinates is as follows:

[0022] A preset forward and backward offset limit threshold is set. The vertical axis values ​​of the center pixel coordinates of the left and right knees are extracted in real time. The absolute value of the difference between the two knee vertical axis values ​​is calculated as the real-time forward and backward offset. When the real-time forward and backward offset exceeds the preset threshold for the first time, the stance is marked as a stance with both feet forward and backward. After that, when the real-time forward and backward offset does not exceed the preset threshold for the first time, the stance is marked as a stance with neither feet forward and backward.

[0023] Furthermore, the method for determining whether an upper limb extension movement has been completed based on electromyographic signals is as follows:

[0024] Continuously collect electromyographic signals from the upper limbs, set a fixed sliding time window with a duration of 1-3 seconds, and set the sliding step size to 1 / 2 of the duration. Calculate the root mean square value of all electromyographic signals within each window.

[0025] The root mean square value of electromyography (EMG) within 3 seconds before entering the bed preparation period is used as the reference value of EMG in the resting state. The EMG activity conditions are set. The specific setting rule is: if the difference between the root mean square value of EMG signal in a window and the reference value of EMG in the resting state accounts for no less than 15% of the reference value of EMG in the resting state, then the window is judged to meet the EMG activity conditions.

[0026] The root mean square (RMS) values ​​of electromyography (EMG) signals from each window are extracted to construct an upper limb EMG signal sequence. When the RMS values ​​of EMG signals in two or more consecutive windows of the upper limb EMG signal sequence increase sequentially and all meet the EMG activity conditions, it is determined to be an upward trend of EMG. When the RMS values ​​of EMG signals in two or more consecutive windows decrease sequentially after this upward trend and all meet the EMG activity conditions, it is determined to be the completion of one upper limb extension exercise. A threshold for the cumulative number of upper limb extension exercises is preset. When the number of upper limb extension exercises reaches this threshold, it is determined that the upper limb extension exercise is completed.

[0027] Furthermore, the method for analyzing pressure in the leg area during the preparatory period in bed to determine whether lower limb joint movement has been completed is as follows:

[0028] During the bed preparation period, real-time pressure data of the leg area is continuously collected. A fixed-duration sliding time window is set, which ranges from 1 to 3 seconds. The sliding step size is set to 1 / 2 of the duration. The average value of all pressure data is calculated within each sliding time window.

[0029] The pressure change condition is set, and the specific setting rule is: if the relative error between the average pressure data in a window and the average pressure data in the previous window is not less than 5%, then this window is judged to meet the pressure change condition.

[0030] The average pressure data in each window is extracted to construct a leg pressure sequence. When the average pressure data in two or more consecutive windows in the pressure sequence increases sequentially and all meet the pressure change conditions, it is determined to be an upward trend. When the average pressure data in two or more consecutive windows after the upward trend decreases sequentially and all meet the above pressure change conditions, it is determined to be a completed lower limb joint flexion and extension movement. A threshold for the cumulative number of lower limb joint flexion and extension movements is preset. When the number of lower limb joint flexion and extension movements reaches the threshold, the lower limb joint activity is determined to be completed.

[0031] Furthermore, the method for simultaneously broadcasting and prompting the duration of the bed preparation period, bedside adaptation period, and standing stabilization period, as well as the corresponding actions and completion status, is as follows:

[0032] When the bed preparation period is determined to have begun, the current duration and appropriate duration of the bed preparation period will be announced in real time. At the same time, it will prompt that lower limb joint movements need to be completed and announce whether the movement has been completed.

[0033] When it is determined that the bedside adaptation period has begun, the current duration and appropriate duration of the bedside adaptation period will be broadcast in real time. At the same time, it will prompt you to complete neck rotation, natural descent of both feet, and upper limb extension exercises, and broadcast whether each exercise has been completed.

[0034] When the standing stability period is determined to have been entered, the current duration and appropriate duration of the standing stability period are broadcast in real time. At the same time, it prompts that the two feet need to be stood up in front of the other foot, and broadcasts whether the action has been completed.

[0035] The appropriate duration for the bed preparation period, bedside adaptation period, and standing stability period should not be less than 30 seconds.

[0036] Compared with the prior art, the beneficial effects of the present invention are:

[0037] This invention integrates pressure sensing for determining body position and lower limb movement, capturing upper limb electromyographic activity, and infrared sensing for obtaining joint coordinates to achieve posture recognition. Through multi-dimensional data complementarity and verification, it breaks through the single-sensor solution and achieves more accurate monitoring. Subsequently, by addressing the standardized getting-out process not covered by existing technologies, it strictly controls the duration of each stage of the bed preparation period, bedside adaptation period, and standing stability period, as well as the completion of corresponding actions, to guide users to perform each step of the operation in a standardized manner. From a physiological adaptation perspective, it avoids the risk of falls caused by orthostatic hypotension in advance. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the overall system structure of the present invention;

[0039] Figure 2 This is a schematic diagram of the pressure on the leg area of ​​the present invention;

[0040] Figure 3 This is a time-series diagram of the abscissa of the midpoint of both eyes in this invention;

[0041] Figure 4 This is a schematic diagram showing the absolute value of the difference between the values ​​on each side of the vertical axis of the present invention;

[0042] Figure 5 This is a schematic diagram of the root mean square value of the electromyography signal in this invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0044] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0045] Example:

[0046] Please see Figures 1 to 5 The present invention provides a technical solution:

[0047] A multimodal perception-based intelligent fall prevention system includes:

[0048] The motion determination module is used to divide the mattress into three areas: back, middle, and legs. It acquires the pressure of each area in real time and sets corresponding monitoring thresholds. It monitors the user's status and electromyography signals in real time. It uses infrared sensing technology combined with a deep learning model to obtain the coordinates of the knee, foot, and eye. Based on the knee and foot coordinates, it determines whether the feet have been naturally lowered. Based on the eye coordinates, it determines whether the neck has been rotated. Based on the knee coordinates, it determines whether the feet are standing forward and backward. Based on the electromyography signals, it determines whether the upper limb extension movement has been completed.

[0049] The body support and movement trajectory when getting up follow a clear pattern: the upper body's main support area is from the shoulders and back to the waist; the buttocks are the core force point when lying down; and the lower body's key area for force exertion and movement is from the thighs to the feet. To adapt to the body proportions of elderly people of different heights and ensure that the areas correspond precisely to different parts of the body, the mattress is divided into back, middle, and leg areas based on the user's height, with the length of the mattress as the longitudinal direction and the end of the mattress closest to the headboard as the starting point. The back area corresponds to the shoulders and back to the waist, the middle area corresponds to the buttocks, and the leg area corresponds to the thighs to the feet. The pressure of each area is acquired in real time, and corresponding monitoring thresholds are set, such as 6-10 kPa. This eliminates slight pressure interference from non-human objects such as clothing and books on the bed, while also determining whether the user's body has movement characteristics. For example, the middle area has the greatest pressure when lying down, the leg pressure changes first when getting up, and the back pressure gradually disappears. This can accurately determine the user's stage of getting up and provide a basis for subsequent intervention.

[0050] The user's state can be either awake or not. The technology of using multi-sensor data fusion and preset algorithms to monitor whether the user is asleep and to collect electromyographic signals is relatively mature. For example, Huawei bracelets use data features from accelerometers and heart rate sensors to accurately determine whether the user is awake. Bracelets with electromyographic sensing modules collect weak electrical signals during muscle contraction through electrode pads that contact the skin of the upper limb. Combined with preset electromyographic signal feature algorithms, they can directly obtain upper limb electromyographic signals, thus completing the collection of the user's awake state and electromyographic signals.

[0051] Elderly people often get up in low-light environments. Infrared thermal imaging can clearly identify key parts such as the knees, feet, and eyes by detecting the temperature difference between the human body and the environment. It is not affected by light and can penetrate thin clothing, avoiding recognition failure caused by clothing obstruction. Therefore, infrared thermal imaging sample data is obtained for model training. The sample data includes infrared images of the user's knees, feet, and eyes, and retains the temperature differences of each part. All sample images are uniformly cropped to a preset fixed size, and the areas of the knees, feet, and eyes are marked with rectangles. The eye area is marked with the pupil and the bright area around the eye, the knee area is marked with the joint and the surrounding muscle thermal area, and the foot area is marked with the sole and heel thermal area.

[0052] A deep learning object detection model is adopted, which can be YOLO, which is good at object detection and segmentation. The model is trained by inputting the above-mentioned labeled infrared thermal imaging sample data. The two-dimensional infrared thermal imaging device is installed at a preset fixed angle, specifically, the two-dimensional infrared thermal imaging device is at a 45°±5° downward angle to the mattress plane, and the installation height is 1.5m-1.8m above the mattress plane. This angle can completely cover the entire longitudinal area of ​​the mattress, adapt to the common bedroom ceiling height, and achieve a balance between clarity and field of view. The upper left corner of the imaging plane of the two-dimensional infrared thermal imaging device is used as the origin of the coordinate system. The horizontal direction to the right along the imaging plane is the horizontal axis of the pixel coordinate system, and the vertical direction downward along the imaging plane is the vertical axis of the pixel coordinate system. The coordinate measurement unit is pixels. Combined with the deep learning object detection model, the center pixel coordinates of the knee, foot, and eye areas are used as the output knee, foot, and eye coordinates, providing a calculation basis for subsequent action discrimination.

[0053] A pre-set distance stability threshold for naturally drooping feet is defined, corresponding to an actual distance of 15-20cm, which is the actual knee-to-foot distance when both feet are naturally drooping in a standard sitting posture. Based on the knee and foot coordinates, the coordinate ordinate values ​​of the knee and foot on the same side on both sides are extracted, and the absolute value of the difference between the ordinate values ​​on each side is calculated to obtain the front-to-back distance of the knee and foot on one side. When the front-to-back distance of the knee and foot on both sides is less than the distance stability threshold, it is determined that the natural drooping of both feet is completed, and the completion of this action is recognized, indicating that the user is ready to stand up and the balance state is initially stable.

[0054] Two eye coordinates are collected in real time, and the average value of the two eye coordinates is calculated at the same time to serve as the center coordinates of the eyes. A time series dataset of center coordinates is constructed, and the maximum and minimum values ​​of the horizontal direction of the center coordinates of the eyes are extracted. The torsion angle is calculated by the absolute value of the difference between the maximum and minimum values. A torsion angle deviation threshold is preset, which should not be less than 30°, corresponding to a left and right rotation range of not less than 60°. When the torsion angle is greater than the preset torsion angle deviation threshold, the neck rotation is calibrated as complete. The completion of the neck rotation movement proves that the elderly have no dizziness. At this time, moving can effectively reduce the risk of orthostatic hypotension and prevent imbalance caused by getting up hastily.

[0055] A preset forward and backward offset limit threshold is set, which corresponds to an actual distance of 10-12cm. The vertical axis values ​​of the center pixel coordinates of the left and right knees are extracted, and the absolute value of the difference between the vertical axis values ​​of the two knees is calculated as the real-time forward and backward offset. When the real-time forward and backward offset exceeds the preset threshold for the first time, the stance is marked as a two-footed stance. The front and back foot distribution can improve the stability of the center of gravity and maintain the stability of the standing state. Only when the two-footed stance has been completed, when the real-time forward and backward offset does not exceed the preset threshold for the first time, the stance is marked as a non-two-footed stance, thereby completing the determination of the end of the subsequent standing stability period. That is, standing with the two feet in front of the other for a certain period of time is one of the core processes of fall prevention.

[0056] Electromyography (EMG) signals from the upper limbs were continuously collected, and a fixed sliding time window was set, ranging from 1 to 3 seconds, to match the force cycle of upper limb extension movements in the elderly. The complete process from muscle exertion to relaxation takes about 2 to 3 seconds, and the sliding step length is set to 1 / 2 of the time window. The root mean square (RMS) value of all EMG signals is calculated within each window to ensure the continuity of the time-series data. The reason for calculating the RMS value is that EMG signals are alternating signals, i.e., signals with superimposed positive and negative signals. The RMS value can quantify the effective intensity of muscle activity within the window, which is an industry standard practice for EMG signal processing.

[0057] The root mean square value of electromyography (EMG) within 3 seconds before entering the bed preparation period is used as the baseline value of resting EMG. Since there are individual differences in the muscle state of the elderly, setting a fixed baseline value uniformly will lead to judgment bias. The first 3 seconds of the bed preparation period are considered to be the stage when the user has just woken up and has not yet started to exert force. Taking the root mean square value of this period as the baseline can accurately match the individual resting EMG level of the user. The EMG activity conditions are set. The specific setting rule is: if the difference between the window root mean square value and the resting EMG baseline value is not less than 15% of the resting EMG baseline value, then the window is judged to meet the EMG activity conditions.

[0058] The root mean square (RMS) values ​​of each window are extracted to construct an upper limb electromyography (EMG) signal sequence. The rise or fall of the RMS value of a single window may be due to random noise or instantaneous movement. Therefore, when the RMS value of two or more consecutive windows in the upper limb EMG signal sequence increases sequentially and the adjacent windows meet the EMG activity conditions, it is determined to be an upward trend of EMG to avoid misjudgment by a single window. When the RMS value of two or more consecutive windows after this upward trend decreases sequentially and the adjacent windows meet the EMG activity conditions, it is determined to be an upper limb extension movement. A threshold for the cumulative number of upper limb extension movements is preset, and this value must be no less than 2. When the number of upper limb extension movements reaches this threshold, the upper limb extension movement is determined to be completed. By judging multiple repetitive movements, it is confirmed that the user actively and intentionally completes the preparatory movement, rather than an accidental behavior.

[0059] In the guidelines for fall intervention techniques for the elderly, upper limb extension, natural foot drop, neck rotation, and standing with feet staggered are the core guidelines for preventing falls in high-risk scenarios for the elderly. These movements are transformed into quantifiable monitoring indicators to reduce the probability of falls during the waking-up phase.

[0060] The pressure analysis module is used to determine the start of the bed preparation period when the pressure in each area is greater than the monitoring threshold, the pressure in the middle area is the greatest, and the user is identified as awake for the first time. During the bed preparation period, the pressure in the leg area is analyzed to determine whether the lower limb joint movement is completed. When the pressure in the back and leg areas is less than the corresponding monitoring threshold, and the pressure in the middle area is greater than the corresponding monitoring threshold, the bed preparation period is determined to end and the duration of the period is calculated.

[0061] When resting in bed, the buttocks in the middle area of ​​the mattress are the core pressure point, and the pressure is the greatest. The shoulders and back in the back area and the thighs in the legs area are also in contact with the mattress, and the pressure is greater than the corresponding monitoring threshold. Only when wakefulness is detected for the first time does it mean that the user may start the process of getting up. This avoids misjudging invalid movements in sleep, such as turning over, as a sign of getting up. It provides a clear time starting point for subsequent monitoring of lower limb joint activities. Therefore, when the pressure in each area is greater than the monitoring threshold, the pressure in the middle area is the greatest, and wakefulness is detected for the first time, it is determined that the user has entered the bed preparation period.

[0062] Table 1 shows the real-time pressure values ​​of the mattress back, middle, and leg areas at 40 different monitoring times. It also clarifies the pressure monitoring thresholds for each area and determines whether the mattress has entered the pre-bed preparation period, generating a pre-bed preparation period determination data table.

[0063] Table 1 Data Table for Determining Bedtime Preparation Period

[0064]

[0065]

[0066] During the bed preparation period, real-time pressure data of the leg area is continuously collected. A fixed-duration sliding time window is set, which ranges from 1 to 3 seconds. The sliding step size is set to 1 / 2 of the duration. The average value of all pressure data is calculated within each sliding time window.

[0067] The pressure change conditions are set, and the specific setting rules are as follows: if the relative error between the average value of a window and the average value of the previous window is not less than 5%, then this window is judged to meet the pressure change conditions, and then it is identified as an effective pressure change when actively flexing and extending the lower limb. The range of motion is sufficient and belongs to the active preparatory behavior of getting up. Avoiding unintentional leg movements as lower limb joint activities. The average values ​​of each window are extracted to construct a leg pressure sequence. When two or more consecutive average values ​​in the pressure sequence increase sequentially and the adjacent average values ​​meet the pressure change conditions, it is judged as an upward trend, excluding random fluctuations. When two or more consecutive average values ​​after the upward trend decrease sequentially and the adjacent average values ​​meet the above pressure change conditions, it is judged as a lower limb joint flexion and extension movement. A threshold for the cumulative number of lower limb joint flexion and extension movements is preset. This value must be not less than 2. When the number of lower limb joint flexion and extension movements reaches this threshold, the lower limb joint activity is judged to be completed. By judging multiple repeated movements, it is identified as an active and intentional completion of preparatory movements, rather than random behavior. Table 2 shows 40 sets of data for judging whether the lower limb joint activity was completed according to the above conditions.

[0068] Table 2 Lower Limb Joint Movement Completion Assessment Table

[0069]

[0070]

[0071] like Figure 2 As shown, in the monitoring of pressure in each window of the leg area, the unit of pressure is kPa. Each number corresponds to a real-time updated sliding time window, and the duration of the sliding time window is 1 second. The user's preparation time in bed exceeds 40 seconds. The first 40 sets of data are analyzed. The user gradually and actively performs flexion and extension movements of the lower limb joints. The amplitude of the leg muscle force gradually increases with each flexion and extension. Therefore, the pressure in the early window number 1-10 is in a continuous rising stage. In the subsequent window number 11-40, it gradually stabilizes. Then, the number of flexion and extension movements is counted to determine whether the lower limb joint activity is completed. For example, in number 3 and 4, the trend is obviously rising and the relative error is not less than 5%, which meets the pressure change condition. Therefore, it is judged that 1 flexion and extension movement is completed. In number 8 and 9, the trend is obviously rising and the relative error is not less than 5%, which meets the pressure change condition. Therefore, it is judged that 1 flexion and extension movement is completed.

[0072] When a user sits up, their upper body leaves the mattress, their legs move to the edge of the bed, while their buttocks remain on the mattress. This objectively captures the process of getting up. Therefore, when the back and leg areas are below the corresponding monitoring threshold, the preparatory period in bed is determined to be over, and the duration of this period is recorded.

[0073] The broadcast prompt module is used to determine whether the bedside adaptation period has been entered after the bed preparation period, and to determine whether the neck rotation, natural descent of both feet, and upper limb extension movements have been completed. When the pressure in each area is less than the corresponding monitoring threshold, the bedside adaptation period is determined to be over and the duration of this period is recorded. When the bedside adaptation period is over and the feet are standing in a forward and backward position, the standing stability period is determined to be entered. When the feet are not standing in a forward and backward position, the standing stability period is determined to be over and the duration of this period is recorded. The broadcast prompts the duration of the bed preparation period, the bedside adaptation period, and the standing stability period, as well as whether the corresponding movements have been completed.

[0074] The bedside adaptation period begins upon completion of the preparatory phase in bed, corresponding to the transition from lying down to sitting on the edge of the bed. This follows the natural process of getting up for older adults and is a critical risk point in the transition from sitting to standing. During this period, neck rotation, natural dangling of the feet, and upper limb extension verify whether older adults have followed the guidelines to complete adequate preparations before standing, avoiding hasty standing that could lead to imbalance. The bedside adaptation period ends when the pressure in each area is below the corresponding monitoring threshold, and the duration of this period is recorded. At this point, combined with the recommended forward and backward standing movements, balance adaptation has been completed, and the stage of walking is ready to begin. The standing stability period ends when the elderly are not in a forward and backward standing position, and the duration of this period is recorded. This aligns with the core requirement of standing stably before moving, effectively reducing the risk of falls.

[0075] The guidelines for fall intervention techniques for the elderly point out that the risk of falls in scenarios such as getting up and standing up is mainly due to rapid changes in body position, insufficient preparation, and inadequate balance adaptation. The division into the bed preparation period, the bedside adaptation period, and the standing stability period transforms the abstract requirements of the guidelines into concrete behavioral stages. In order to provide real-time prompts and guidance to the elderly to complete the steps according to the requirements of the guidelines, voice prompts are used to remind them.

[0076] When the bed preparation period is determined to have started, the current duration and appropriate duration of the bed preparation period are broadcast in real time, and the current duration is broadcast in real time. At the same time, the lower limb joint movements need to be completed, and the completion of the movement is broadcast.

[0077] When it is determined that the bedside adaptation period has begun, the system will provide real-time prompts on the current duration and appropriate duration of the bedside adaptation period, and will also prompt the user to complete neck rotation, natural foot drop, and upper limb extension exercises, and announce whether each exercise has been completed.

[0078] When the standing stability period is determined to have started, the system will broadcast the current duration and appropriate duration of the standing stability period in real time, broadcast the current duration in real time, prompt the user to complete the standing with both feet in front and behind, and broadcast whether the action has been completed.

[0079] like Figures 3-5 As shown, each sequence number corresponds to a real-time updated sliding time window. Figure 3 In the horizontal coordinates of the center points of both eyes, after each monitoring window data acquisition is completed, the maximum and minimum values ​​of the horizontal coordinates of the center points of both eyes in the current center coordinate time series dataset are calculated and updated in real time. The maximum value in this dataset is maintained at 162 and the minimum value is maintained at 128, with a difference of 34, indicating that the neck rotation movement has been completed.

[0080] Figure 4 As the user is just starting to get up from a lying position, their lower limbs are in a transitional state from curled to extended. The half of the body gradually tilts towards the edge of the bed, and the spatial position of the knees and feet changes frequently. The vertical axis coordinates mapped to the infrared thermal imaging pixel coordinate system also fluctuate accordingly. The knee position is fixed as the body leans forward. When the feet gradually leave the bed surface support and hang down below the edge of the bed, the coordinate values ​​show a downward trend as the spatial position moves down. In window numbers 10-20, the user's feet are completely hanging down below the edge of the bed, and the spatial position of the knees and feet no longer changes significantly. The absolute values ​​of the difference between the vertical axes of the left knee and left foot and the absolute values ​​of the difference between the vertical axes of the right knee and right foot are both less than the distance stability threshold, and the natural hanging of the feet has been completed.

[0081] During the bedside adaptation period, electromyographic (EMG) signals of the upper limbs were continuously collected. A fixed sliding time window of 1 second was set. The duration of this bedside adaptation period exceeded 40 seconds. The first 40 sets of data were analyzed. As the user gradually transitioned to active arm raising and extending movements, the root mean square (RMS) values ​​of the EMG signals gradually increased in the first 30 seconds. During the last 10 seconds, the extended posture was maintained. Therefore, in sequences 30-40, the corresponding RMS values ​​of the EMG signals were at a relatively high level with slight fluctuations. Figure 5 When the root mean square value of the electromyography signal shows an upward or downward trend and the adjacent average values ​​all meet the pressure change condition, the upper limb extension movement is completed when the cumulative number of times reaches the threshold.

[0082] As aging individuals experience decreased vascular elasticity and weakened balance, changes in body position, such as from lying down to sitting up or from sitting up to standing, can easily lead to orthostatic hypotension and dizziness. Therefore, the preparation period in bed, the adaptation period at the bedside, and the stabilization period in standing should all be no less than 30 seconds. This timeframe is set as the minimum safe duration for the body to adapt to changes in body position, ensuring that the elderly can fully complete each movement without rushing. However, due to physiological decline, some elderly individuals may not be able to complete the necessary movements or adapt to changes in body position within the 30-second baseline due to physiological degeneration. In practical applications, this timeframe can be flexibly extended according to the individual circumstances of the elderly. For example, elderly individuals with significant orthostatic hypotension may need more than 30 seconds to relieve dizziness and vertigo symptoms. Extending the timeframe allows sufficient time for blood pressure regulation and reduces the risks associated with changes in body position.

[0083] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0084] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0085] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0086] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An intelligent fall prevention system based on multimodal perception, characterized in that, include: The motion determination module is used to divide the mattress into three areas: back, middle, and legs. It acquires the pressure of each area in real time and sets corresponding monitoring thresholds. It monitors the user's status and electromyography signals in real time. It uses infrared sensing technology combined with a deep learning model to obtain the coordinates of the knee, foot, and eye. Based on the knee and foot coordinates, it determines whether the feet have been naturally lowered. Based on the eye coordinates, it determines whether the neck has been rotated. Based on the knee coordinates, it determines whether the feet are standing forward and backward. Based on the electromyography signals, it determines whether the upper limb extension movement has been completed. The pressure analysis module is used to determine the start of the bed preparation period when the pressure in each area is greater than the monitoring threshold, the pressure in the middle area is the greatest, and the user is identified as awake for the first time. During the bed preparation period, the pressure in the leg area is analyzed to determine whether the lower limb joint movement is completed. When the pressure in the back and leg areas is less than the corresponding monitoring threshold, and the pressure in the middle area is greater than the corresponding monitoring threshold, the bed preparation period is determined to end and the duration of the period is calculated. The broadcast prompt module is used to determine whether the bedside adaptation period has ended after the bed preparation period, and to determine whether the neck rotation, natural descent of both feet, and upper limb extension movements have been completed. When the pressure in each area is less than the corresponding monitoring threshold, the bedside adaptation period is determined to have ended and the duration of this period is recorded. When the bedside adaptation period ends and the feet are standing forward and backward, the standing stability period is determined to have ended. When the feet are not standing forward and backward, the standing stability period is determined to have ended and the duration of this period is recorded. The module broadcasts prompts on the duration of the bed preparation period, the bedside adaptation period, and the standing stability period, as well as whether the corresponding movements have been completed.

2. The intelligent fall prevention system based on multimodal perception according to claim 1, characterized in that: The method for dividing a mattress into three zones—back, middle, and legs—is as follows: Using the length of the mattress as the longitudinal direction, and starting from the end of the mattress closest to the headboard, divide the mattress longitudinally into back, middle, and leg areas according to the user's height. The back area corresponds to the shoulders and back to the waist, the middle area corresponds to the hips, and the leg area corresponds to the thighs to the feet.

3. The intelligent fall prevention system based on multimodal perception according to claim 1, characterized in that: The method for obtaining the coordinates of the knee, foot, and eye using infrared sensing technology combined with a deep learning model is as follows: Infrared thermal imaging sample data for model training is obtained. The sample data includes infrared images of the user's knees, feet and eyes, and the temperature differences of each part are preserved. All sample images are uniformly cropped to a preset fixed size, and the areas of the knees, feet and eyes are marked with rectangles. The eye area is marked with the pupil and the bright area around the eye, the knee area is marked with the joint and the surrounding muscle thermal area, and the foot area is marked with the sole and heel thermal area. A deep learning target detection model is used, and the above-annotated infrared thermal imaging sample data is input for training. The two-dimensional infrared thermal imaging device is installed at a preset fixed angle, specifically, the two-dimensional infrared thermal imaging device is at a 45°±5° downward angle to the mattress plane, and the installation height is 1.5m-1.8m above the mattress plane. The center pixel coordinates of the knee, foot, and eye regions are obtained by combining the deep learning target detection model, and are used as the output knee, foot, and eye coordinates.

4. The intelligent fall prevention system based on multimodal perception according to claim 3, characterized in that: The method for determining whether natural foot drop has been completed based on knee and foot coordinates is as follows: A pre-set distance stability threshold for natural foot descent is defined as the actual knee-to-foot distance when the feet are naturally descent in a standard sitting posture. Based on the knee and foot coordinates, the coordinate ordinate values ​​of the knee and foot on the same side on both the left and right sides are extracted, and the absolute value of the difference between the two is calculated to obtain the lateral knee-to-foot distance on one side. When the lateral knee-to-foot distance on both the left and right sides is less than the distance stability threshold, the natural foot descent is considered complete.

5. The intelligent fall prevention system based on multimodal perception according to claim 3, characterized in that: The method for determining whether neck rotation has been completed based on eye coordinates is as follows: Two eye coordinates are collected in real time, and the mean of the two eye coordinates is calculated at the same time to serve as the center coordinates of the eyes. A time series dataset of the center coordinates is constructed, and the maximum and minimum values ​​of the horizontal values ​​of the center coordinates are extracted from it. The absolute value of the difference between the maximum and minimum values ​​is calculated as the torsion angle. A preset torsion angle deviation threshold is set. When the torsion angle exceeds the preset torsion angle deviation threshold, the calibration of neck rotation is completed.

6. The intelligent fall prevention system based on multimodal perception according to claim 3, characterized in that: The method for determining whether the stance is with feet staggered based on knee coordinates is as follows: A preset forward and backward offset limit threshold is set. The vertical axis values ​​of the center pixel coordinates of the left and right knees are extracted in real time. The absolute value of the difference between the two knee vertical axis values ​​is calculated as the real-time forward and backward offset. When the real-time forward and backward offset exceeds the preset threshold for the first time, the stance is marked as a stance with both feet forward and backward. After that, when the real-time forward and backward offset does not exceed the preset threshold for the first time, the stance is marked as a stance with neither feet forward and backward.

7. The intelligent fall prevention system based on multimodal perception according to claim 1, characterized in that: The method for determining whether an upper limb extension movement has been completed based on electromyographic signals is as follows: Continuously collect electromyographic signals from the upper limbs, set a fixed sliding time window with a duration of 1-3 seconds, and set the sliding step size to 1 / 2 of the duration. Calculate the root mean square value of all electromyographic signals within each window. The root mean square value of electromyography (EMG) within 3 seconds before entering the bed preparation period is used as the reference value of EMG in the resting state. The EMG activity conditions are set. The specific setting rule is: if the difference between the root mean square value of EMG signal in a window and the reference value of EMG in the resting state accounts for no less than 15% of the reference value of EMG in the resting state, then the window is judged to meet the EMG activity conditions. The root mean square (RMS) values ​​of electromyography (EMG) signals from each window are extracted to construct an upper limb EMG signal sequence. When the RMS values ​​of EMG signals in two or more consecutive windows of the upper limb EMG signal sequence increase sequentially and all meet the EMG activity conditions, it is determined to be an upward trend of EMG. When the RMS values ​​of EMG signals in two or more consecutive windows decrease sequentially after this upward trend and all meet the EMG activity conditions, it is determined to be the completion of one upper limb extension exercise. A threshold for the cumulative number of upper limb extension exercises is preset. When the number of upper limb extension exercises reaches this threshold, it is determined that the upper limb extension exercise is completed.

8. The intelligent fall prevention system based on multimodal perception according to claim 2, characterized in that: The method for analyzing pressure in the leg area during the pre-operative preparation period in bed to determine whether lower limb joint movement has been completed is as follows: During the bed preparation period, real-time pressure data of the leg area is continuously collected. A fixed-duration sliding time window is set, which ranges from 1 to 3 seconds. The sliding step size is set to 1 / 2 of the duration. The average value of all pressure data is calculated within each sliding time window. The pressure change condition is set, and the specific setting rule is: if the relative error between the average pressure data in a window and the average pressure data in the previous window is not less than 5%, then this window is judged to meet the pressure change condition. The average pressure data in each window is extracted to construct a leg pressure sequence. When the average pressure data in two or more consecutive windows in the pressure sequence increases sequentially and all meet the pressure change conditions, it is determined to be an upward trend. When the average pressure data in two or more consecutive windows after the upward trend decreases sequentially and all meet the above pressure change conditions, it is determined to be a completed lower limb joint flexion and extension movement. A threshold for the cumulative number of lower limb joint flexion and extension movements is preset. When the number of lower limb joint flexion and extension movements reaches the threshold, the lower limb joint activity is determined to be completed.

9. The intelligent fall prevention system based on multimodal perception according to claim 1, characterized in that: The method for simultaneously broadcasting and prompting the duration of the bed preparation period, bedside adaptation period, and standing stabilization period, as well as the corresponding actions and completion status, is as follows: When the bed preparation period is determined to have begun, the current duration and appropriate duration of the bed preparation period will be announced in real time. At the same time, it will prompt that lower limb joint movements need to be completed and announce whether the movement has been completed. When it is determined that the bedside adaptation period has begun, the current duration and appropriate duration of the bedside adaptation period will be broadcast in real time. At the same time, it will prompt you to complete neck rotation, natural descent of both feet, and upper limb extension exercises, and broadcast whether each exercise has been completed. When the standing stability period is determined to have been entered, the current duration and appropriate duration of the standing stability period are broadcast in real time. At the same time, it prompts that the two feet need to be stood up in front of the other foot, and broadcasts whether the action has been completed. The appropriate duration for the bed preparation period, bedside adaptation period, and standing stability period should not be less than 30 seconds.

Images