Fall monitoring system and method for elderly at home based on feature fusion

Abstract

The invention discloses a household elderly falling monitoring system and method based on feature fusion, and relates to the technical field of falling monitoring. The technical key points are as follows: firstly, the reasonability and effectiveness of an obtained detection result are guaranteed to obtain first gait image recognition of the home elderly to estimate the posture of the home elderly, the improved OpenPose network and CoordConv are utilized to replace VGG-16 into EfficientNet, the first gait of the home elderly is accurately monitored, the balance influence proportion is calculated for different gaits, and the balance influence proportion is calculated for the different gaits. According to the method, the probability that the old people have falling behaviors is judged to generate different risk signals, falling behaviors and postures under medium and high risk signals are further analyzed in a targeted mode, through deflection analysis, changes of human body postures are monitored in real time, characteristics of different falling postures are understood, different falling postures are independently and accurately modeled, and a second falling coefficient is generated; the tumble degree is accurately identified, and a foundation is laid for optimization of a subsequent model.

Description

Technical Field

[0001] The present invention relates to the technical field of fall monitoring, and in particular to a system and method for monitoring falls of elderly people at home based on feature fusion. Background Art

[0002] Falls among the elderly at home refer to the situation where the elderly lose balance and fall to the ground or other lower surfaces while standing, walking or in other postures in the home environment due to various internal and external factors (such as physical health status, home environment, accidents, etc.); falls may cause varying degrees of injuries, including bruises, fractures, head trauma, etc., and may even be life-threatening.

[0003] The causes of falls are usually related to physiological changes in the elderly (such as decreased muscle strength, weakened coordination, decreased vision, etc.), psychological factors (such as fear, anxiety) and the safety of the home environment (such as slippery floors, obstacles, etc.); on the one hand, the existing fall monitoring technology usually relies on wearable devices, such as smart bracelets, belts, etc., which require users to actively wear them. For elderly people or patients with limited mobility, wearing these devices is not practical and may cause inconvenience or discomfort in wearing, affecting the quality of life of users; on the other hand, traditional fall monitoring usually installs surveillance cameras to collect video data and uses image recognition algorithms to analyze the posture of people. The system can detect a person's sudden fall or loss of balance from a standing position by using the accelerometer to detect the person's posture and movement trajectory. When it detects a person's sudden fall from a standing position or loss of balance, an alarm can be triggered. Although a certain recognition effect can be achieved in a certain scenario, it is difficult for a single feature to fully characterize the rich information of the fall action, and the recognition rate in complex scenarios needs to be improved. For example, when the elderly are in the kitchen or bathroom, if they only rely on the accelerometer, different actions (such as quickly squatting to collect garbage or kneeling) may be mistaken for falls. For example, if the elderly use crutches to support their body during a fall, the crutches may mislead the sensor to judge the fall action, resulting in false alarms or missed alarms, affecting the recognition accuracy of the system. Summary of the invention

[0004] 1. Technical issues to be resolved In view of the deficiencies in the prior art, the present invention provides a system and method for monitoring falls of elderly people at home based on feature fusion, which solves the problems raised in the background technology.

[0005] (II) Technical solution To achieve the above objectives, the present invention is implemented through the following technical solutions: In a first aspect, the present application provides a method for monitoring falls of elderly people at home based on feature fusion, the method comprising: Obtain target data of elderly people at home, including personal parameters, video parameters, environmental parameters, crutch parameters, and vibration parameters; The SSIM technology is used to obtain the key frame images of the video parameters, and the image recognition method is used to extract the limb movement characteristics of the key frame images. Combined with personal parameters, the first step is obtained; based on the first step, the judgment and analysis mechanism is triggered to determine whether there is a record of the use of the crutch. If there is no record of use, the initial training model is constructed; On the contrary, the environmental parameters and crutch parameters are deeply analyzed to obtain the environmental assessment coefficient and the crutch assessment coefficient, calculate the proportion of the balance impact on the first step, and generate high-risk, medium-risk or low-risk signals accordingly; Secondary extraction of limb movement features to identify abnormal key points, determine falling behavior, and obtain the first fall coefficient; based on medium or high risk signals, determine the falling posture, and implement adjustment and correction strategies to calculate the second fall coefficient.

[0006] Furthermore, in the process of calculating the second fall coefficient, it also includes: Based on the initial training model, several blank connection layers are added, and the blank connection layers are trained with the data sets corresponding to the first fall coefficient and the second fall coefficient. The peak, mean, standard deviation and root mean square of the data set are extracted, and the first fall coefficient and the second fall coefficient are integrated as classification features to obtain the fusion features, and the fully connected layer is adjusted forward by the connection layer to obtain the fall recognition model.

[0007] Further, the step of obtaining the first posture includes: Image recognition: The pose of key frame images is estimated based on the improved OpenPose network, and CoordConv is introduced to perceive the spatial position information of the image. VGG-16 is replaced by EfficientNet to obtain the 2D position information of the key points of the human body and map it to the 3D position space. The activity and angle of the key points are marked and recorded as limb movement features. Cascade processing: Input the limb movement characteristics into the preset human body dynamics model, output the speed differential value and step frequency, and cascade the speed differential value, step frequency and limb movement characteristics to generate the first step.

[0008] Furthermore, the environmental parameters include ground convexity, temperature, humidity and illumination, and the crutch parameters include force uniformity, screw loosening rate and strength. The step of deeply analyzing the environmental parameters and crutch parameters includes: Extract the set parameter range of the environmental parameters in the database, obtain the set temperature range, humidity range and illumination range, and set the formula to calculate the environmental assessment coefficient en: ; In the formula, bulge represents the ground convexity, tem represents temperature, hum represents humidity, and lur represents illumination. ] indicates the set temperature range, [hums, ] indicates the set humidity range, [lurs, ] represents the set illumination interval, β1 and β2 represent weight correction coefficients, g1, g2 and g3 are the preset weights of temperature, humidity and illumination offset ratio, and g1+g2+g3=1; Set the formula to calculate the crutch evaluation coefficient cru: ; Wherein, uof represents force uniformity, slr represents screw loosening rate, str represents strength, α1, α2 and α3 all represent weight correction coefficients, and α1+α2+α3=1.

[0009] Furthermore, the first posture includes normal walking, fast walking and stationary standing, and the step of calculating the balance influence ratio on the first posture includes: When standing still: call up the environmental assessment coefficient and the crutch assessment coefficient, and set the formula to obtain the balance impact ratio pro: ; In the formula, μ1 and μ2 are weight correction coefficients, and both μ1 and μ2 are greater than 0; When walking normally: call up the environmental assessment coefficient and the crutch assessment coefficient, combine the obtained body fat percentage and shoe size length, and set the formula to output the balance impact ratio pro: ; In the formula, wei represents the body fat percentage, jz represents the critical value of the body fat percentage, len represents the shoe size length, [lens, ] indicates the set shoe size range, , , , are weight correction coefficients, and , , , All are greater than 0; When walking fast: call up the environmental assessment coefficient and the crutch assessment coefficient, combine the step frequency and speed differential value, and set the formula to output the balance impact ratio pro: ; In the formula, cad represents the step frequency, ved represents the speed differential value, represents a constant, , , , are weight correction coefficients, and , , , Both are greater than 0.

[0010] Furthermore, the balance impact ratio is compared and analyzed with the set first ratio interval, second ratio interval, and third ratio interval: When the balance impact ratio is in the first ratio interval, a high risk signal is generated; When the balance impact ratio is in the second ratio interval, a medium risk signal is generated; When the balance impact ratio is in the third ratio interval, a low risk signal is generated; Among them, the first proportion interval is larger than the second proportion interval, which is larger than the third proportion interval.

[0011] Furthermore, if responding to a low risk signal: One-time judgment: draw a circle with any key point as the center and circumscribe a cube to obtain several boundary points of the position comparison area, and calculate the intersection and union ratio of the position comparison area and the standard position comparison area; When the intersection-union ratio is less than the first threshold, no processing is performed; On the contrary, it is determined that there is a fall and a fall signal is generated. At the first moment after the fall, it is determined whether the height of the rectangular frame surrounded by the key points is greater than the first multiple of the height of the rectangular frame at the moment of the fall. If it is greater, the fall signal is discarded; On the contrary, the key point coordinates and key coordinate changes from the initial frame to the end frame are extracted to obtain the falling rate and falling direction of the key point, and the vertical axis is used as the baseline to obtain the angle between the falling direction and the baseline, and the formula is set to calculate the first fall coefficient fall1; If responding to a medium or high risk signal: Secondary judgment: After the first judgment, calculate the rotation angle of the key point relative to the coordinate axis during the descent process, compare and analyze the rotation angle with the set standard angle range, and determine the type of falling posture; extract the key points associated with the falling posture, and perform deflection analysis to execute the corresponding adjustment and correction strategy to obtain the second fall coefficient; among them, the falling postures include: falling sideways, falling forward to the ground, and falling on your back.

[0012] Furthermore, the step of adjusting and correcting the strategy includes: Side fall: Calculate the difference between the waist, hips, knees and ankles corresponding to the key points obtained through deflection analysis, and set the formula to obtain the second fall coefficient fall2: ; Where, reg1 represents the control correction coefficient, and reg1>0, dif1 represents the difference obtained by deflection analysis of the corresponding key points of the waist, hips, knees and ankles; Fall forward to the ground: Get the type of floor you fall forward to the ground, and retrieve vibration parameters based on the floor type, including stiffness coefficient, damping coefficient, and natural vibration frequency, to calculate the vibration factor. Combined with the difference obtained through deflection analysis of the corresponding key points of the head and shoulders, set the formula to obtain the second fall coefficient fall2: ; Where vib represents the vibration factor, reg2, Both represent the control correction coefficient, and reg2, All are greater than 0, dif2 represents the difference between the key points of the head and shoulders obtained through deflection analysis; Falling backwards on your back: Extract facial expression features based on key frame images, use SVM to identify emotional labels, and assign corresponding weights and scores. Combine the duration of facial expressions to obtain expression factors, and combine the difference obtained by deflection analysis of the corresponding key points of the head and shoulders to set the formula to obtain the second fall coefficient fall2: ; Where exf represents the expression factor, reg3, are all control correction coefficients, and reg3, Both are greater than 0.

[0013] In a second aspect, the present application provides a fall monitoring system for elderly people at home based on feature fusion, the system comprising: The data collection module collects target data of the elderly at home, including personal parameters, video parameters, environmental parameters, crutch parameters and vibration parameters; wherein the personal parameters at least include body fat percentage and shoe size length; The data processing module uses SSIM technology to obtain the key frame images of video parameters, uses image recognition methods to extract the limb movement characteristics of the key frame images, and combines personal parameters to obtain the first posture; based on the first posture, the judgment and analysis mechanism is triggered to determine whether there is a use record of the crutch. If there is no use record, an initial training model is constructed based on the target data; conversely, the environmental parameters and crutch parameters are deeply analyzed to obtain the environmental assessment coefficient and the crutch assessment coefficient, and the balance impact ratio on the first posture is calculated, and a high-risk, medium-risk or low-risk signal is generated accordingly; The adjustment and correction module extracts limb movement features for the second time to identify abnormal key points, determine the falling behavior, and obtain the first falling coefficient; combined with medium or high risk signals, the falling posture is determined, and the adjustment and correction strategy is implemented to calculate the second falling coefficient; The model optimization module adds several blank connection layers based on the initial training model, trains the blank connection layers with the data sets corresponding to the first fall coefficient and the second fall coefficient, extracts the peak, mean, standard deviation and root mean square of the data set, and integrates the first fall coefficient and the second fall coefficient as classification features to obtain fusion features, and adjusts the fully connected layer forward from the connection layer.

[0014] (III) Beneficial effects The present invention provides a system and method for monitoring falls of elderly people at home based on feature fusion, which has the following beneficial effects: 1. The present invention uses image recognition and combines multiple network structures, including the improved OpenPose network, CoordConv, and replacing VGG-16 with EfficientNet, to ensure the rationality and effectiveness of the detection results, so as to obtain the first step of the elderly at home, which is convenient for subsequent analysis. For the three gaits of normal walking, fast walking, and static standing, the balance influence ratio is calculated based on significant features, and the risk of falling of the elderly at home is calculated, which effectively improves the accuracy and reliability of the analysis of the possibility of falling under different gaits, and facilitates the effectiveness of the subsequent analysis of the degree of falling; 2. Based on the deflection analysis of different falling postures under different risk signals, the present invention extracts the key points associated with the falling postures, and calculates the deflection angle of the key point change position relative to the coordinate axis, so as to monitor the changes of human body posture in real time; the present invention also separately models the three falling postures of falling on the back, falling forward to the ground, and falling on the back, calculates and generates the second falling coefficient, understands the characteristics of different falling postures, and provides an analysis basis for subsequent model training; 3. The present invention establishes an initial training model based on whether there is a record of crutch usage, adds several blank connection layers based on the initial training model, takes the initial training model as the starting point, adjusts the features of the initial training model, and conducts targeted training on the new connection layer, thereby significantly reducing the time and computing resources required to train the model from scratch; extracts the peak value, mean, standard deviation, root mean square, first fall coefficient, and second fall coefficient of the data set, and integrates them as classification features to enhance the input features and improve the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of a method for monitoring falls of elderly people at home according to an exemplary embodiment; Figure 2 is a schematic diagram of steps of a method for monitoring falls of elderly people at home according to an exemplary embodiment; Figure 3 The figure is a module diagram of a system for monitoring falls of elderly people at home according to an exemplary embodiment. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] Example 1

[0018] The embodiment of the present invention provides a method for monitoring falls of elderly people at home based on feature fusion; Figure 1 is a flow chart of a method for monitoring falls of elderly people at home according to an exemplary embodiment. Figure 2 is a schematic diagram of steps of a method for monitoring falls of elderly people at home according to an exemplary embodiment; please refer to Figure 1 - Figure 2 , the method comprising: S1. Obtain target data of elderly people at home, including personal parameters, video parameters, environmental parameters, crutch parameters, and vibration parameters; S2, using SSIM technology to obtain key frame images of video parameters, using image recognition methods to extract limb movement features of key frame images, combined with personal parameters, to obtain the first step; The specific steps of using SSIM technology to obtain key frame images of video parameters include: Obtain video parameters, perform differential processing on the video parameters at the same time interval to obtain several differential images; compare the similarities of adjacent differential images in the several differential images, and when the similarity value is less than the preset similarity threshold, it means that a large change has occurred between the two frames, and the frame image of the current frame is used as the key frame image; wherein the time interval is a custom parameter; it should be noted that in order to avoid selecting frames with small repeated changes in adjacent frames, a time window mechanism can be introduced; for example: set a time window, if the SSIM of a certain frame and the previous frames are all lower than the threshold, it can be used as a key frame with a higher probability; Steps to get your first stance include: Image recognition: The pose of key frame images is estimated based on the improved OpenPose network, and CoordConv is introduced to perceive the spatial position information of the image. VGG-16 is replaced by EfficientNet to obtain the 2D position information of the key points of the human body and map it to the 3D position space. The activity and angle of the key points are marked and recorded as limb movement features. Specific: First, VGG16 is used to extract features, and then the OpenPose network is passed through multiple stages. The number of convolutional layers increases and is replaced by EfficientNet to further optimize the network structure. The improved OpenPose network replaces some convolutional layers with a depth-separable convolutional network type based on the OpenPose human skeleton information recognition network to reduce the amount of calculation and speed up the processing of the model. CoordConv adds two channels with coordinate information on the basis of conventional convolution after inputting the feature map, so that the convolution operation can perceive the spatial position information of the input image, and can learn translation invariance according to different task requirements, optimize the generalization ability of the entire model, and thus improve the detection accuracy. In addition, posture behaviors can usually be regarded as continuous sequences. LSTM is introduced, and its forget gate stores a series of key frame images that determine the fall in posture behaviors, so as to facilitate the subsequent analysis of fall behaviors. Fall and related behavior data are also extracted from the URFall dataset to enrich the dataset, reduce training time, and protect user privacy. Cascade processing: Input the limb movement characteristics into the preset human body dynamics model, output the speed differential value and step frequency, and cascade the speed differential value, step frequency and limb movement characteristics to generate the first step, including normal walking, fast walking and stationary standing; It should be noted that in the human body dynamics model, the personal parameters of the elderly at home are also used as the input parameters of the model; the personal parameters include age, gender, illness, leg length, body fat percentage and shoe size length; the linear relationship between the personal parameters of the elderly at home and the speed differential value and step frequency is established by numerical fitting method, and the gait changes of the elderly at home are calculated according to the final speed differential value and step frequency, and then the final speed differential value and step frequency are cascaded with the limb movement characteristics, which retains the information of all original features, and with the increase of information, it may help the model improve its performance when processing complex tasks, especially when there is no strong correlation between the features, to define the type of gait; In general, the first step of the elderly at home is obtained through image recognition, so as to ensure the rationality and effectiveness of the test results. In the subsequent analysis process, since the falling behavior of the elderly at home may cause different degrees of falling due to the gait type, if only a single gait is maintained for subsequent testing, it is easy to cause large detection errors; S3. Based on the first step, trigger the judgment and analysis mechanism to determine whether there is a usage record of the crutch: If there is no usage record, an initial training model is constructed based on the target data, and step S5 is executed; If there is a record of crutch use, the environmental parameters and crutch parameters are deeply analyzed to obtain the environmental assessment coefficient and the crutch assessment coefficient, calculate the proportion of the balance impact on the first step, and generate high risk, medium risk or low risk signals; The step of determining whether there is a usage record of the crutch includes: Define usage records: clarify what is the "usage record" of the crutch. It is necessary to obtain whether the crutch has been assembled and used. Basic information such as assembly time, purchase date, usage location, and maintenance history can be recorded; Select the record storage method: Database: Use the database to store records for easy data management and query; File system: Use the file system to store records, such as creating a text file named with the crutch serial number to record relevant information; Label: Put a label on the crutch to record relevant information; Automatic judgment method: Different judgment methods can be selected according to different recording methods: Database query: query the database through the crutch serial number or other identification information to determine whether the crutch has been installed, used or repaired; File system query: query the file system through the crutch serial number or other identification information to determine whether there is a corresponding file; Label recognition: use image recognition technology to identify the label on the crutch and obtain relevant information; Manual inspection: Visual inspection: Check the crutch for signs of wear, scratches, contamination, etc., which are usually signs of use; Odor and cleanliness: Check whether the crutch has a used odor (such as sweat) or signs of cleaning (such as water stains); The above are some ways to determine whether the crutch is being used. I will not elaborate on them again. Set up the plan based on the actual situation. The steps for calculating the proportion of the equilibrium influence on the first step include: Environmental parameters include ground convexity, temperature, humidity and illumination. The set parameter range of environmental parameters in the database is extracted to obtain the set temperature range, humidity range and illumination range, and the formula is set to calculate the environmental assessment coefficient: ; In the formula, en represents the environmental assessment coefficient, bulge represents the ground convexity, tem represents temperature, hum represents humidity, lur represents illumination, [tems, ] indicates the set temperature range, [hums, ] indicates the set humidity range, [lurs, ] represents the set illumination interval, β1 and β2 represent weight correction coefficients, g1, g2 and g3 are the preset weights of temperature, humidity and illumination offset ratio, and g1+g2+g3=1; The following is an explanation of the nouns and associated formulas involved in environmental parameters: The convexity of the indoor floor is measured by a laser radar sensor or an ultrasonic sensor, the temperature of the home environment is measured by a temperature sensor, the humidity of the home environment is measured by a humidity sensor, and the illuminance of the home environment is measured by an illuminance sensor. The above sensors are not drawn in the figure and are adaptively installed in the home environment; the convexity of the ground refers to the unevenness of the ground surface, which is usually used to describe the undulations of the ground. The more convex the ground is, the more significant the impact on the risk of falling of the elderly, especially when the elderly wear ill-fitting shoes or have an unstable gait; the temperature indicates the degree of coldness or hotness of the home environment. Low temperature may cause muscle stiffness caused by coldness, affect balance, and increase the risk of slipping; high temperature may cause physical strength decline or dehydration, affecting reaction ability; high humidity environment makes the ground slippery, increasing the risk of slipping, especially in places with water sources such as bathrooms and kitchens, the probability of elderly people slipping will increase; illuminance indicates the brightness of the light in the home environment. Insufficient light can easily lead to limited vision, increase cognitive impairment of the home environment, and cause falls when walking; in general, the larger the value of the environmental assessment coefficient, the higher the risk of slipping of the elderly at home; Crutch parameters include force uniformity, screw loosening rate, and strength; The crutch assessment coefficient is calculated according to the formula: ; In the formula, cru is the crutch assessment coefficient, uof is the force uniformity, slr is the screw loosening rate, str is the strength, α1, α2 and α3 are all weight correction coefficients, and α1+α2+α3=1, and α1>α2, α1>α3; The following is an explanation of the nouns and associated formulas involved in the crutch parameters: The force uniformity indicates the balance of the force direction of the crutch. The larger the value, the more balanced the force is. The screw loosening rate is a factor that affects the structural stability of the crutch. The higher the screw loosening rate, the worse the stability of the crutch. The strength of the crutch is directly related to the maximum load it can withstand. The greater the strength, the higher the durability of the crutch. The greater the force uniformity, the smaller the screw loosening rate, and the greater the strength, the larger the value corresponding to the crutch evaluation coefficient, indicating that the stability of the crutch used by the elderly at home is better. In different states: When standing still: call up the environmental assessment coefficient and the crutch assessment coefficient, and set the formula to obtain the balance impact ratio pro: ; In the formula, μ1 and μ2 are weight correction coefficients, and both μ1 and μ2 are greater than 0; Formula explanation: When the values ​​of the environmental assessment coefficient and the crutch assessment coefficient are small, the value of the balance impact ratio is also small, indicating that the probability of the elderly at home falling is smaller; when the values ​​of the environmental assessment coefficient and the crutch assessment coefficient are large, the value of the balance impact ratio is also large, indicating that the probability of the elderly at home falling is greater; When walking normally: call up the environmental assessment coefficient and the crutch assessment coefficient, combine the body fat percentage and shoe size length, and set the formula to output the balance impact ratio pro: ; In the formula, wei represents the body fat percentage, jz represents the critical value of the body fat percentage, len represents the shoe size length, [lens, ] indicates the set shoe size range, , , , are weight correction coefficients, and , , , All are greater than 0; Formula explanation: The appropriateness of shoe size affects the stability of gait. Shoes that are too large will lead to instability of the soles of the feet and unnatural gait. Shoes that are too small will squeeze the feet, causing pain and discomfort, which may increase the risk of falls for the elderly. When the body fat percentage reaches the critical value, the possibility of falling will increase with every 1% increase in fat body weight, which will reduce the body's flexibility and balance ability, thereby increasing the risk of falling, and ultimately leading to an increase in the balance impact ratio. When walking fast: call up the environmental assessment coefficient and the crutch assessment coefficient, combine the step frequency and speed differential value, and set the formula to output the balance impact ratio pro: ; In the formula, cad represents the step frequency, ved represents the speed differential value, Represents a constant, which is used to adjust the influence of the speed differential value in the logarithmic term and keep the result of the logarithmic term positive. , , , are weight correction coefficients, and , , , All are greater than 0; Formula explanation: Cadence is an indicator to measure the number of steps per unit time. The faster the cadence, the larger the corresponding value, the larger the speed differential value, the larger the number of items will be, which will lead to an increase in the proportion of balance influence; Setting a first ratio interval, a second ratio interval and a third ratio interval of the balance influence ratio, wherein the first ratio interval is larger than the second ratio interval and larger than the third ratio interval; When the balance impact ratio is in the first ratio interval, it is determined that the balance stability of the elderly at home is low and they are prone to falling, and a high-risk signal is generated; When the balance impact ratio is in the second ratio interval, it is determined that the balance stability of the elderly at home is moderate, the probability of falling is average, and a medium risk signal is generated; When the balance impact ratio is in the third ratio interval, it is determined that the balance stability of the elderly at home is relatively high and they are not likely to fall, and a low-risk signal is generated at the same time; Specifically, this embodiment adopts: Multi-gait analysis: The solution designs three gaits: normal walking, fast walking, and stationary standing. It also captures the significant features of specific types of gaits. For example, when walking fast, the step frequency and speed differential value of the first gait are combined to capture the significant features of different gaits in a more targeted manner, thus avoiding the errors caused by directly performing fall analysis under a single gait. Interval comparison: The program designed an interval comparison to analyze the probability of falls among elderly people at home; In general, this scheme effectively improves the accuracy and reliability of fall possibility analysis under different gaits through technical means such as multi-gait analysis and interval comparison, which facilitates the effectiveness of subsequent fall degree analysis; S4. Based on several risk signals, extract limb movement features for a second time and map them to 3D position space to identify abnormal key points, determine fall behavior, obtain a first fall coefficient, and execute an adjustment and correction strategy to obtain a second fall coefficient; The step of identifying abnormal key points includes: In response to low risk signals: One-time judgment: draw a circle with any key point as the center and circumscribe a cube to obtain 8 boundary points of the position comparison area, including: Vr1 (xa+r, ya+r, za+r), Vr2 (xa-r, ya+r, za+r), Vr3 (xa+r, ya-r, za+r), Vr4 (xa-r, ya-r, za+r), Vr5 (xa+r, ya+r, za-r), Vr6 (xa-r, ya+r, za-r), Vr7 (xa+r, ya-r, za-r), Vr8 (xa-r, ya-r, za-r), calculate the intersection and union ratio of the position comparison area and the standard position comparison area; It should be noted that the key points of the limb motion feature mapping are composed of 33 key nodes across the torso, namely, the head, shoulders, waist, hips, knees and ankles, including the nose, the inner side of the left eye, the left eye, the outer side of the left eye, the inner side of the right eye, the right eye, the outer side of the right eye, the left ear, the right ear, the left side of the mouth, the right side of the mouth, the left shoulder, the right shoulder, the left elbow joint, the right elbow joint, the left wrist, the right wrist, the first knuckle of the left little finger, the first knuckle of the right little finger, the first knuckle of the left index finger, the first knuckle of the right index finger, the second knuckle of the left thumb, the second knuckle of the right thumb, the left hip joint, the right hip joint, the left knee, the right knee, the left ankle, the right ankle, the left heel, the right heel, the left toe and the right toe; When the intersection-union ratio is less than the first threshold, no processing is performed; On the contrary, a fall is determined and a fall signal is generated. At the first moment after the fall, it is determined whether the height of the rectangular frame surrounded by the key points is greater than the first multiple of the height of the rectangular frame at the moment of the fall, and the first multiple is usually 0.8; If it is greater, the fall signal is eliminated; On the contrary, the key point coordinates and key coordinate changes from the initial frame to the end frame are extracted to obtain the descent rate and descent direction of the key point, and the vertical axis is used as the baseline to obtain the angle between the descent direction and the baseline, and the formula is set to calculate the first fall coefficient; The formula is: ; Where fall1 represents the first fall coefficient, V down represents the falling rate, θ represents the angle, It represents the conversion coefficient, which is preset through a large amount of data measurement. When the descent rate is faster and the cosine value of the angle is higher, the value of the first fall coefficient is larger, indicating that the severity of the fall is greater; In addition, an important feature of falling behavior is the significant change in the aspect ratio of the human body. In this embodiment, the smallest circumscribed rectangular frame surrounded by all key points is taken as an example, and the ratio of its width to height is set as the aspect ratio; for example, when standing or walking normally, the aspect ratio of the human body is usually significantly less than 1; during squatting, the aspect ratio gradually approaches 1; and in the case of falling behavior, the aspect ratio is significantly greater than 1. The change in the aspect ratio reflects the drastic change in the human body posture and is an important indicator for identifying falling behavior; When an elderly person at home falls, by correlating the changes in the previous and next frames, it is possible to more accurately distinguish between falling and non-falling behaviors, thereby improving the accuracy and reliability of recognition. For example, in 2D space, when a fall occurs, the position change in the y-axis direction will be significantly greater than the position change in the x-axis direction. By monitoring continuous frames, the y-axis shows a sharp change. In response to a medium or high risk signal: Second judgment: After one determination, the rotation angle of the key point relative to the coordinate axis during the descent process is calculated, and the rotation angle is compared with the set standard angle range; the angle relative to the X axis is recorded as the first angle, the angle relative to the Y axis is recorded as the second angle, and the angle relative to the Z axis is recorded as the third angle; When the first angle and the second angle do not satisfy the first angle interval, and the third angle does not satisfy the second angle interval, it is determined to be a front-to-back fall; when the third angle is close to -90°, it is determined to be a front fall toward the ground; when the third angle is close to 90°, it is determined to be a back fall; When the first angle and the second angle do not satisfy the third angle interval, and the third angle does not satisfy the fourth angle interval, it is determined to be a side fall; In addition, the falling posture can be further refined by combining key points: Select two key points, such as the left shoulder and the left hip joint, calculate the first vector between the two, assume that the Y axis is the direction of the fall (not involved in the selection within the plane), and calculate the projection of the first vector on the XZ plane; record each frame of the descent process, and repeatedly calculate the first vector and its projection on the XZ plane; use the dot product and cross product of the vectors to calculate the angle between the two vectors; without considering the error range, set the angle within [150°, 210°] to determine it as a back fall; if the angle is within [60°, 120°] or [-120°, -60°], it is determined as a left or right side fall; if the angle is within [-30°, 30°], it is determined as a forward fall toward the ground; Extract key points associated with the falling posture, and perform deflection analysis to execute corresponding adjustment and correction strategies to obtain a second falling coefficient; The steps of deflection analysis include: The plane where the key points are located is divided into 8 directions, the deflection angle of the key point change position relative to the coordinate axis is calculated, and the difference between the deflection angle and the corresponding standard deflection threshold is calculated. When the difference is within the set gradient range, no processing is performed; otherwise, it is marked as a deviation, and the difference between the deviation ratio and the standard deviation ratio is calculated to execute the corresponding adjustment and correction strategy; the calculation formula of the difference is: difference = deviation ratio - standard deviation ratio; Side fall: Calculate the difference between the waist, hips, knees and ankles corresponding to the key points obtained by deflection analysis, and set the formula to obtain the second fall coefficient. The formula is: ; Wherein, fall2 represents the second fall coefficient when falling sideways, exp(·) represents the exponential function, reg1 represents the control correction coefficient, and reg1>0, dif1 represents the difference obtained by deflection analysis of the corresponding key points of the waist, hips, knees and ankles; Falling forward to the ground: Get the type of floor you fall forward to the ground, and retrieve vibration parameters based on the floor type, including stiffness coefficient, damping coefficient, and natural vibration frequency, to calculate the vibration factor. Combined with the difference obtained through deflection analysis of the corresponding key points of the head and shoulders, get the second fall coefficient. The formula is: ; Where, vib represents the vibration factor, k represents the stiffness coefficient, ξ represents the damping coefficient, Ω=2πf0, f0 represents the natural vibration frequency; It should be noted that when falling forward, the head, shoulders and other parts of the body first touch the ground, which usually results in a relatively small contact area and a high impact force. Due to the forward shift of the body's center of gravity, the floor may be subjected to a large instantaneous impact force, especially when the faller falls forward at a fast speed. The impact force may be very concentrated, which may cause the floor to vibrate significantly, especially when the floor material is soft or the structure is not stable enough; The formula for the second fall coefficient is: ; Where, fall2 represents the second fall coefficient for falling toward the ground, reg2, Both represent the control correction coefficient, and reg2, All are greater than 0, dif2 represents the difference between the key points of the head and shoulders obtained through deflection analysis; Falling backwards on your back: Extract facial expression features based on key frame images, use SVM to identify emotional labels, and assign corresponding weights and scores. Combine the duration of facial expressions to obtain expression factors, and combine the difference obtained by deflection analysis of corresponding key points of the head and shoulders to obtain the second fall coefficient. The formula is: ; Where, fall2 represents the second fall coefficient after falling on one's back, exf represents the expression factor, reg3, are all control correction coefficients, and reg3, All are greater than 0; It should be noted that during the backward fall, the camera will capture the facial expressions of the human body and identify emotional labels, including frowning, squinting, opening the mouth, and twisting. Different emotional labels represent different degrees of pain and are defined by assigning weights. For example, frowning, assigned a weight of 0.1 < squinting, assigned a weight of 0.2 = opening the mouth, assigned a weight of 0.2 < twisting, assigned a weight of 0.3. The specific values ​​of the weights are set according to the actual situation. Example: Data 1: Assuming that during the backward fall, the camera captured 2 seconds of opening the mouth and 1 second of squinting, then the expression factor: exf=0.2*2*2+0.1*1*1=0.9; Data 2: Assuming that during the backward fall, the camera captures 1 second of frowning, 2 seconds of opening the mouth, and 1 second of squinting, then the expression factor: exf=0.1*1*1+0.2*2*2+0.1*1*1=1.0; Comparing data 1 and data 2, 1.0>0.9, the pain level of data 2 is greater than that of data 1; Specifically, this embodiment adopts: Deflection analysis: Extract key points associated with the falling posture, and calculate the deflection angle of the key point change position relative to the coordinate axis to monitor the changes in human posture in real time. When the elderly fall in different postures or on different surfaces, the amplitude of the angle change may be different. Calculating the deviation range can not only help determine whether it is a normal fall process and avoid false alarms or missed alarms, but also make the fall monitoring system more stable in different environments and improve the adaptability of the system. At the same time, by calculating the angles and deviations of key nodes in the fall process, the risk of possible injury during the fall can be initially quantified. Multi-posture analysis: The solution designs three falling postures: falling on your back, falling forward to the ground, and falling on your back. It also models each falling posture individually and accurately, and calculates and generates a second fall coefficient to reflect the action stage and severity of the fall. In general, by adopting the above technical solutions, real-time monitoring of the falling process is achieved, the occurrence of falling events is accurately identified, and accurate modeling is performed based on the falling posture, the characteristics of different falling postures are understood, and an analytical basis is provided for subsequent model training; S5. Add several blank connection layers based on the initial training model, train the blank connection layers with the data sets corresponding to the first fall coefficient and the second fall coefficient, and integrate the peak value, mean, standard deviation, root mean square, first fall coefficient and second fall coefficient of the data sets as classification features, adjust the fully connected layer forward from the connection layer to achieve model optimization, and obtain a fall recognition model; Specific: When there is no record of the use of crutches by elderly people at home, the initial training model is directly constructed based on the obtained target data, which can be regarded as a "rough training". Then, the target data with the use record of the crutches is obtained, and the preliminary training model is adjusted and optimized, which can be regarded as a "detailed training". By adding several blank connection layers based on the initial training model and taking the initial training model as the starting point, the time and computing resources required for training the model from scratch can be significantly reduced. The target data (with the use record of the crutches) is input into the initial training model again, the general features learned by the pre-training model are fully utilized, and the feature representation of specific tasks is learned on this basis. Through targeted training on the new connection layer, the peak value, mean, standard deviation, root mean square, first fall coefficient and second fall coefficient of the data set are extracted and integrated as classification features to enhance the input features, fuse the features and accelerate the convergence process of the model; in the process of training the blank connection layer, the full connection layer is adjusted forward from the blank connection layer to achieve model optimization, and the optimized model is marked as a fall recognition model; when the target data of the elderly at home is collected in real time in the future and input into the fall recognition model, the fall behavior and degree of the elderly at home are identified.

[0019] Example 2

[0020] The embodiment of the present invention provides a fall monitoring system for elderly people at home based on feature fusion; Figure 3 is a schematic diagram of a module of a fall monitoring system for elderly people at home according to an exemplary embodiment; Figure 3 The system includes: a data acquisition module, a data processing module, an adjustment and correction module and a model optimization module, and the data acquisition module, the data processing module, the adjustment and correction module and the model optimization module are communicatively connected; The data collection module collects target data of the elderly at home, including personal parameters, video parameters, environmental parameters, crutch parameters and vibration parameters; wherein the personal parameters at least include body fat percentage and shoe size length; The data processing module uses SSIM technology to obtain the key frame images of video parameters, uses image recognition methods to extract the limb movement characteristics of the key frame images, and combines personal parameters to obtain the first posture; based on the first posture, the judgment and analysis mechanism is triggered to determine whether there is a use record of the crutch. If there is no use record, an initial training model is constructed based on the target data; conversely, the environmental parameters and crutch parameters are deeply analyzed to obtain the environmental assessment coefficient and the crutch assessment coefficient, and the balance impact ratio on the first posture is calculated, and a high-risk, medium-risk or low-risk signal is generated accordingly; The adjustment and correction module extracts limb movement features for the second time to identify abnormal key points, determine the falling behavior, and obtain the first falling coefficient; combined with medium or high risk signals, the falling posture is determined, and the adjustment and correction strategy is implemented to calculate the second falling coefficient; The model optimization module adds several blank connection layers based on the initial training model, trains the blank connection layers with the data sets corresponding to the first fall coefficient and the second fall coefficient, extracts the peak, mean, standard deviation and root mean square of the data set, and integrates the first fall coefficient and the second fall coefficient as classification features to obtain fusion features, and adjusts the fully connected layer forward from the connection layer to achieve model optimization.

[0021] In the application, the several formulas involved are all calculated by removing the dimensions and taking their numerical values. The formula is a formula obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The formula is set by technical personnel in this field according to actual conditions.

[0022] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. A person of ordinary skill in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0023] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0024] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for monitoring falls of elderly people at home based on feature fusion, characterized in that: The method comprises: Obtain target data of elderly people at home, including personal parameters, video parameters, environmental parameters, crutch parameters, and vibration parameters; The SSIM technology is used to obtain the key frame images of the video parameters, and the image recognition method is used to extract the limb movement characteristics of the key frame images. Combined with personal parameters, the first step is obtained; based on the first step, the judgment and analysis mechanism is triggered to determine whether there is a record of the use of the crutch. If there is no record of use, the initial training model is constructed; On the contrary, the environmental parameters and crutch parameters are deeply analyzed to obtain the environmental assessment coefficient and the crutch assessment coefficient, calculate the proportion of the balance impact on the first step, and generate high-risk, medium-risk or low-risk signals accordingly; Secondary extraction of limb movement features to identify abnormal key points, determine falling behavior, and obtain the first fall coefficient; based on medium or high risk signals, determine the falling posture, and implement adjustment and correction strategies to calculate the second fall coefficient.

2. The method for monitoring falls of elderly people at home based on feature fusion according to claim 1, characterized in that: The process of calculating the second fall coefficient also includes: Based on the initial training model, several blank connection layers are added, and the blank connection layers are trained with the data sets corresponding to the first fall coefficient and the second fall coefficient. The peak, mean, standard deviation and root mean square of the data set are extracted, and the first fall coefficient and the second fall coefficient are integrated as classification features to obtain the fusion features, and the fully connected layer is adjusted forward by the connection layer to obtain the fall recognition model.

3. The method for monitoring falls of elderly people at home based on feature fusion according to claim 1, characterized in that: Steps to get your first stance include: Image recognition: The pose of key frame images is estimated based on the improved OpenPose network, and CoordConv is introduced to perceive the spatial position information of the image. VGG-16 is replaced by EfficientNet to obtain the 2D position information of the key points of the human body and map it to the 3D position space. The activity and angle of the key points are marked and recorded as limb movement features. Cascade processing: Input the limb movement characteristics into the preset human body dynamics model, output the speed differential value and step frequency, and cascade the speed differential value, step frequency and limb movement characteristics to generate the first step.

4. The method for monitoring falls of elderly people at home based on feature fusion according to claim 1, characterized in that: The environmental parameters include ground convexity, temperature, humidity and illumination, and the crutch parameters include force uniformity, screw loosening rate and strength; The steps for in-depth analysis of environmental parameters and crutch parameters include: Extract the set parameter range of the environmental parameters in the database, obtain the set temperature range, humidity range and illumination range, and set the formula to calculate the environmental assessment coefficient en: ； In the formula, bulge represents the ground convexity, tem represents temperature, hum represents humidity, and lur represents illumination. ] indicates the set temperature range, [hums, ] indicates the set humidity range, [lurs, ] represents the set illumination interval, β1 and β2 represent weight correction coefficients, g1, g2 and g3 are the preset weights of temperature, humidity and illumination offset ratio, and g1+g2+g3=1; Set the formula to calculate the crutch evaluation coefficient cru: ; Wherein, uof represents force uniformity, slr represents screw loosening rate, str represents strength, α1, α2 and α3 all represent weight correction coefficients, and α1+α2+α3=1.

5. The method for monitoring falls of elderly people at home based on feature fusion according to claim 1, characterized in that: The first posture includes normal walking, fast walking and static standing, and the step of calculating the balance influence ratio on the first posture includes: When standing still: call up the environmental assessment coefficient and the crutch assessment coefficient, and set the formula to obtain the balance impact ratio pro: ； In the formula, μ1 and μ2 are weight correction coefficients, and both μ1 and μ2 are greater than 0; When walking normally: call up the environmental assessment coefficient and the crutch assessment coefficient, combine the obtained body fat percentage and shoe size length, and set the formula to output the balance impact ratio pro: ； In the formula, wei represents the body fat percentage, jz represents the critical value of the body fat percentage, len represents the shoe size length, [lens, ] indicates the set shoe size range, , , , are weight correction coefficients, and , , , All are greater than 0; When walking fast: call up the environmental assessment coefficient and the crutch assessment coefficient, combine the step frequency and speed differential value, and set the formula to output the balance impact ratio pro: ； In the formula, cad represents the step frequency, ved represents the speed differential value, represents a constant, , , , are weight correction coefficients, and , , , Both are greater than 0.

6. The method for monitoring falls of elderly people at home based on feature fusion according to claim 5, characterized in that: Compare and analyze the balance impact ratio with the set first ratio interval, second ratio interval, and third ratio interval: When the balance impact ratio is in the first ratio interval, a high risk signal is generated; When the balance impact ratio is in the second ratio interval, a medium risk signal is generated; When the balance impact ratio is in the third ratio interval, a low risk signal is generated; Among them, the first proportion interval is larger than the second proportion interval, which is larger than the third proportion interval.

7. The method for monitoring falls of elderly people at home based on feature fusion according to claim 6, characterized in that: If responding to a low risk signal: One-time judgment: draw a circle with any key point as the center and circumscribe a cube to obtain several boundary points of the position comparison area, and calculate the intersection and union ratio of the position comparison area and the standard position comparison area; When the intersection-union ratio is less than the first threshold, no processing is performed; On the contrary, it is determined that there is a fall and a fall signal is generated. At the first moment after the fall, it is determined whether the height of the rectangular frame surrounded by the key points is greater than the first multiple of the height of the rectangular frame at the moment of the fall. If it is greater, the fall signal is discarded; On the contrary, the key point coordinates and key coordinate changes from the initial frame to the end frame are extracted to obtain the falling rate and falling direction of the key point, and the vertical axis is used as the baseline to obtain the angle between the falling direction and the baseline, and the formula is set to calculate the first fall coefficient fall1; If responding to a medium or high risk signal: Secondary determination: After the primary determination, the rotation angle of the key point relative to the coordinate axis during the descent process is calculated, and the rotation angle is compared and analyzed with the set standard angle range to determine the type of falling posture; The key points associated with the falling posture are extracted and deflected and analyzed to execute the corresponding adjustment and correction strategy to obtain the second falling coefficient; wherein the falling posture includes: falling sideways, falling forward to the ground, and falling backward on your back.

8. The method for monitoring falls of elderly people at home based on feature fusion according to claim 7, characterized in that: The steps to implement the remediation strategy include: Side fall: Calculate the difference between the waist, hips, knees and ankles corresponding to the key points obtained through deflection analysis, and set the formula to obtain the second fall coefficient fall2: ; Where, reg1 represents the control correction coefficient, and reg1>0, dif1 represents the difference obtained by deflection analysis of the corresponding key points of the waist, hips, knees and ankles; Fall forward to the ground: Get the type of floor you fall forward to the ground, and retrieve vibration parameters based on the floor type, including stiffness coefficient, damping coefficient, and natural vibration frequency, to calculate the vibration factor. Combined with the difference obtained through deflection analysis of the corresponding key points of the head and shoulders, set the formula to obtain the second fall coefficient fall2: ； Where vib represents the vibration factor, reg2, Both represent the control correction coefficient, and reg2, All are greater than 0, dif2 represents the difference between the key points of the head and shoulders obtained through deflection analysis; Falling backwards on your back: Extract facial expression features based on key frame images, use SVM to identify emotional labels, and assign corresponding weights and scores. Combine the duration of facial expressions to obtain expression factors, and combine the difference obtained by deflection analysis of the corresponding key points of the head and shoulders to set the formula to obtain the second fall coefficient fall2: ； Where exf represents the expression factor, reg3, are all control correction coefficients, and reg3, Both are greater than 0.

9. A fall monitoring system for elderly people at home based on feature fusion, characterized in that: The system comprises: The data collection module collects target data of the elderly at home, including personal parameters, video parameters, environmental parameters, crutch parameters and vibration parameters; wherein the personal parameters at least include body fat percentage and shoe size length; The data processing module uses SSIM technology to obtain the key frame images of video parameters, uses image recognition methods to extract the limb movement features of the key frame images, and combines personal parameters to obtain the first posture; based on the first posture, the judgment and analysis mechanism is triggered to determine whether there is a use record of the crutch. If there is no use record, an initial training model is constructed based on the target data; conversely, the environmental parameters and crutch parameters are deeply analyzed to obtain the environmental assessment coefficient and the crutch assessment coefficient, and the balance impact ratio on the first posture is calculated, and a high-risk, medium-risk or low-risk signal is generated accordingly; The adjustment and correction module extracts limb movement features for the second time to identify abnormal key points, determine the falling behavior, and obtain the first falling coefficient; combined with medium or high risk signals, the falling posture is determined, and the adjustment and correction strategy is implemented to calculate the second falling coefficient; The model optimization module adds several blank connection layers based on the initial training model, trains the blank connection layers with the data sets corresponding to the first fall coefficient and the second fall coefficient, extracts the peak, mean, standard deviation and root mean square of the data set, and integrates the first fall coefficient and the second fall coefficient as classification features to obtain fusion features, and adjusts the fully connected layer forward from the connection layer.

Images