Human fall detection method and system based on sensor and image processing fusion

Abstract

The present invention relates to a human fall detection method and system based on sensor and image processing fusion, the method comprises obtaining sensor data of a human sensor group; extracting features of the sensor data, and inputting the features into a trained Bayesian classifier to obtain a classification result; judging whether the classification result is a fall, if not, executing step S1; if yes, establishing a lower limb real-time model based on Unity according to human joint point information, and extracting key frames of the lower limb real-time model at the moment of falling; judging whether a fall is made according to the key frames by using an image recognition neural network, if not, executing step S1, and outputting fall information if a fall is made. Compared with the prior art, the present invention has the advantages of more accurate detection results and the like.

Description

Technical Field

[0001] The present invention relates to the field of human body detection, and in particular to a human body fall detection method and system based on the fusion of sensor and image processing. Background Art

[0002] With the aging of the population, the protection and monitoring of the elderly are becoming more and more important in society. In fact, most diseases of the elderly, including stroke, fracture, cerebral hemorrhage, etc., are caused by careless falls. If the elderly fall accidentally and the rescuers do not come to treat them in time, the final result may be a tragedy. It is regrettable that with the increasing number of empty-nest elderly people, the devices for elderly people's falls and alarms are not yet popular, and there are more and more unfortunate incidents due to untimely treatment. When the elderly fall, they often fall into unconscious coma or cannot move due to fractures. They cannot ask for help in this state. In this case, the only solution to the problem is to let the device automatically detect falls and alarm.

[0003] In existing detection methods, researchers mostly use no more than two sensors attached to certain parts of the body (thighs, stomach, knees, soles of feet). The sensor signals received are single and do not have the global modeling function of lower limb gait. In addition, the commonly used threshold method is difficult to distinguish between regular gait behaviors such as going up and down stairs, jumping, squatting, etc. and falls when judging falls. It is easy to generalize them and make misjudgments. Therefore, existing fall detection methods cannot correctly judge the human body fall situation. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a human fall detection method and system based on the fusion of sensor and image processing.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A human fall detection method based on sensor and image processing fusion includes the following steps:

[0007] S1, obtaining sensing data of a human body sensor group;

[0008] S2, extract the features of the sensor data and input them into the trained Bayesian classifier to obtain the classification results;

[0009] S3, determine whether the classification result is a fall, if not, execute step S1; if yes, build a lower limb real-time model based on Unity according to the human joint information, and extract the key frame of the lower limb real-time model at the moment of falling;

[0010] S4, using an image recognition neural network to determine whether the person has fallen according to the key frame, if not, executing step S1, if the person has fallen, outputting the fall information;

[0011] The training process of the Bayesian classifier is as follows:

[0012] The features of the sensor data with fall classification labels and no-fall classification labels are taken as input and the classification labels are taken as output to train a Bayesian classifier.

[0013] Furthermore, the sensing data feature extraction step in step S2 is as follows:

[0014] S21, filtering the sensor data through a Kalman filter to eliminate random deviations and obtain primary filtered data;

[0015] S22, setting a threshold, removing data exceeding the threshold in the primary filtering data, and obtaining secondary filtering data;

[0016] S23. Calculate the standard deviation, mean and covariance of the secondary filtered data as data features.

[0017] Furthermore, the filtering steps of the Kalman filter are as follows:

[0018] S211, obtaining the initial value and initial confidence of the sensor data, and combining the covariance matrix and sampling time of the noise to obtain the observation value;

[0019] S212: Update the covariance matrix and the sensor data values ​​according to the observed values.

[0020] Furthermore, the sensor data includes accelerometer data, gravimeter data and magnetometer data.

[0021] Furthermore, in step S4, the steps of establishing the image recognition neural network are as follows:

[0022] A large amount of image information with fall classification labels and non-fall classification labels is input into the neural network, and the classification labels are used as output to train the image recognition neural network.

[0023] Furthermore, in step S3, the specific steps of establishing the lower limb real-time model based on Unity are as follows:

[0024] S31, obtaining the world coordinates of several nodes of the original human body joints, and converting them into posture quaternions in the world coordinate system;

[0025] S32, according to the IMU data in the sensor data, obtain the angular velocity information in the world coordinate system, and calculate the attitude quaternion at the next moment in combination with the attitude quaternion;

[0026] S33. Update the joint node information of the lower limb real-time model according to the posture quaternion at the next moment.

[0027] Furthermore, the calculation expression of the posture quaternion at the next moment is as follows:

[0028]

[0029] Among them, q t Represents the quaternion of the next moment’s attitude, q t-1 Represents the current attitude quaternion, ω t represents angular velocity information, and Δt represents unit time.

[0030] A human fall detection system based on sensor and image processing fusion includes a data acquisition module and a PC processing module:

[0031] The data acquisition module is used to obtain the sensor information of sensors at various nodes of the body, aggregate and transmit it to the PC processing module after time synchronization;

[0032] The PC processing module performs the following steps:

[0033] A1. Extract the features of sensor information and input them into the trained Bayesian classifier to obtain the classification results;

[0034] A2. Determine whether the classification result is a fall. If not, continue to receive data from the data acquisition module. If yes, build a lower limb real-time model based on Unity according to the human joint information, and extract the key frame of the lower limb real-time model at the moment of falling.

[0035] A3. Use the image recognition neural network to determine whether the person has fallen based on the key frames. If the person has not fallen, continue to receive data from the data acquisition module. If the person has fallen, output the fall information.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] 1. The present invention first obtains the information of the sensor and determines the fall at the data level; if it is determined to be a fall, a 3D model is established through the human body node coordinates, and the fall is further determined at the image level. The sensor and image processing are integrated, which not only ensures the sensitivity of recognition, but also avoids misjudgment caused by changes in the sensor when going up and down stairs, thereby improving the accuracy of fall detection; and at the image processing level, the 3D model image is obtained through sensor information and human body node coordinates, which can still achieve effective judgment in dim light, and has a wider application scenario.

[0038] 2. The present invention uses a Bayesian classifier to process sensor data and makes decision classification based on the maximum a posteriori probability, thereby reducing sensor errors and further improving the accuracy of fall detection.

[0039] 3. The present invention can realize fall detection by only installing sensors on the human body and connecting to a PC computer, which is low-cost and easy to promote.

[0040] 4. The present invention only judges falls by extracting key frames during image processing. Compared with the traditional image recognition method that needs to judge each frame, the detection efficiency of the present invention is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the process of the present invention.

[0042] Figure 2 This is a principle flow chart of the Bayesian classifier of the present invention.

[0043] Figure 3 This is a hardware structure diagram of the system of the present invention.

[0044] Figure 4 Schematic diagram of wearable sensors on the human body. DETAILED DESCRIPTION

[0045] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0046] This embodiment provides a human fall detection method based on sensor and image processing fusion, such as Figure 1 As shown, the following steps are included:

[0047] Step S1, acquiring sensing data of a human body sensor group.

[0048] Step S2: extract the features of the sensor data and input them into the trained Bayesian classifier to obtain the classification result.

[0049] Step S3, determine whether the classification result is a fall, if not, execute step S1; if yes, establish a lower limb real-time model based on Unity according to the human body joint information, and extract the key frame of the lower limb real-time model at the moment of falling.

[0050] Step S4: Use the image recognition neural network to determine whether the person has fallen down based on the key frame. If the person has not fallen down, execute step S1. If the person has fallen down, output the fall down information:

[0051] In this embodiment, the sensor data of the sensor group includes the three-axis data of the accelerometer, gravimeter, and magnetometer. The data of all sensor groups are collected through the aggregation node, and the data is initially calibrated and corrected. Figure 4 shown.

[0052] Specifically, the accelerometer is calibrated, and the calibration steps are as follows:

[0053] The three axes of the gyroscope are x, y, and z. Assume that the z-axis is upward and the xy plane is placed flat on the plane. The zero point error of the z-axis is b, the temperature proportional factor is s, and the gravity acceleration is g. The z-axis acceleration at this time is Z up ; Invert the z-axis, the z-axis acceleration is Z down , then we have the following formula:

[0054] Z up =b-(1+s)g

[0055] Z down =b+(1+s)g

[0056] Keep the sensor still and horizontal, and calibrate the z-axis zero bias b and temperature proportional factor s; calibrate the y-axis and z-axis in the same way; and then use the above formula to calculate the true value of the accelerometer.

[0057] The magnetometer is calibrated and rotated slowly for two weeks to obtain the three-axis magnetic field data. i The three-axis magnetic field data is (x i ,y i , z i ). Use MATLAB software to draw a 3D stereo image of all points. The final data should be in the shape of an ellipsoid. Take the long axis of the ellipsoid and take the coordinates of the highest and lowest points of the axis, recorded as (x max ,y max , z max ) and (x min ,y min , z min ), then the zero point compensation value of the magnetometer is (z max -z min ) / 2.

[0058] In this embodiment, step S2 can be specifically expanded as follows:

[0059] Step S21, filtering the sensor data through a Kalman filter to eliminate random deviations and obtain primary filtered data.

[0060] Step S22: Set a threshold value, remove data exceeding the threshold value in the primary filtered data, and obtain secondary filtered data.

[0061] Step S23: Calculate the standard deviation and mean of the three-axis data of the accelerometer, gravimeter, and magnetometer, as well as the covariance between the two as data features.

[0062] In this embodiment, the method of using the Kalman filter for filtering is as follows:

[0063] Step S211, obtain the initial value and initial confidence of the three-axis data of the accelerometer, gravimeter, and magnetometer, and obtain the observation value by combining the covariance matrix and sampling time of the noise.

[0064] Step S212: Update the covariance matrix and the sensor data values ​​according to the observed values.

[0065] In this embodiment, a Bayesian classifier is selected as a classification model for fall determination. The classification principle of the Bayesian classifier is as follows: Figure 2 As shown in the figure, the posterior probability of the object is calculated by using the Bayesian formula through the prior probability of the object and the class conditional probability density function, that is, the probability that the object belongs to a certain class, and the class with the largest posterior probability is selected as the class to which the object belongs. It is based on the Bayesian theorem and estimates the posterior probability by training a large number of samples. It is an important random pattern classification method.

[0066] First, the training process of the Bayesian classifier is as follows:

[0067] The training samples and the number of classes are selected as the input of the Bayesian classifier. The training samples include 50 nine-axis gyroscope data of normal walking, sprinting, and falling events of adult males aged 18-30 within two seconds. The number of classes is 3, which includes 3 categories: normal walking, accelerated trotting, and falling. The training samples are first subjected to feature extraction, and the mean and standard deviation of the three axes of the accelerometer, gravimeter, and magnetometer are extracted as 6 feature data. 20 sets of feature values ​​are extracted within two seconds, and the conditional probability estimate of the training samples is obtained based on the 50*20 feature values ​​of the known categories.

[0068] After training the Bayesian classifier, input the test sample to be classified to obtain the posterior probability. According to the maximum posterior probability, the category of the test sample to be classified is determined.

[0069] The specific expressions of the classification are as follows:

[0070] Let x = {a1, a2, a3, a4, a5, a6} be an item to be classified, and each a is a characteristic attribute of x, where a1 represents the mean of the three axes of the accelerometer, a2 represents the mean of the three axes of the gravimeter, a3 represents the mean of the three axes of the magnetometer, a4 represents the standard deviation of the three axes of the accelerometer, a5 represents the standard deviation of the three axes of the gravimeter, and a6 represents the standard deviation of the three axes of the magnetometer. The human gait category set is D = {y1, y2, y3}, where y1 represents normal walking, y2 represents accelerated trotting, and y3 represents falling.

[0071] First, calculate the posterior probability P(y1|x), P(y2|x), ..., P(y3|x). The calculation steps are as follows:

[0072] Find a set of items to be classified with known classification, which is the training sample set. According to the feature values ​​and categories of the known samples, the conditional probability estimates of each feature attribute under each category are statistically obtained. That is:

[0073] P(a1|y1),P(a2|y1),P(a3|y1),P(a4|y1),P(a5|y1),P(a6|y1);

[0074] P(a1|y2),P(a2|y2),P(a3|y2),P(a4|y2).P(a5|y2),P(a6|y2);

[0075] P(a1|y3),P(a2|y3),P(a3|y3),P(a4|y3).P(a5|y3),P(a6|y3).

[0076] By default, each feature attribute is conditionally independent, then according to Bayes' theorem, there is the following expression:

[0077]

[0078] Where P(x) is a constant, compare P(y i |x) is to compare P(x|y) i )P(y i ), and because each feature attribute is conditionally independent, we have:

[0079]

[0080] If P(y k |x)=max{P(y1|x),P(y2|x),P(y3|x)}, then x∈y k . Bayesian classifier P(y k |x)=max{P(y1|x),P(y2|x),P(y3|x)} can be transformed into the following expression:

[0081] P(x|y k )P(y k )=max{P(x|y1)P(y1),P(x|y2)P(y2),P(x|y3)P(y3)}

[0082] In this embodiment, the specific steps of establishing a real-time model of lower limbs based on Unity are as follows:

[0083] First, a 3D model of the human lower limb skeleton is established, the world coordinates of the original five nodes (waist and abdomen, left knee, right knee, left ankle, and right ankle) are obtained, and then converted into posture quaternions in the world coordinate system.

[0084] Get IMU data, that is, angular velocity ω in the world coordinate system t , the quaternion expression of the attitude at the next moment can be obtained as follows:

[0085]

[0086] Among them, q t Represents the quaternion of the next moment’s attitude, q t-1 Represents the current attitude quaternion, ω t Indicates angular velocity information.

[0087] According to the posture quaternion at the next moment, the joint node information of the lower limb real-time model is updated.

[0088] In this embodiment, if a fall is determined by the Bayesian classifier, the image content of the key frame of the lower limb real-time model will be captured, and the image will be used to determine whether the person has fallen. If the person has fallen, the information will be sent to the information processing base station for an alarm.

[0089] In this embodiment, the steps for establishing the image recognition neural network are as follows:

[0090] A large amount of image information with fall classification labels and non-fall classification labels is input into the neural network, and the classification labels are used as output to train the image recognition neural network.

[0091] This embodiment also provides a human fall detection system based on sensor and image processing fusion, such as Figure 3 As shown, it includes data acquisition module and PC processing module:

[0092] The data acquisition module is used to obtain the sensing information of sensors at various nodes of the body, aggregate and transmit it to the PC processing module after time synchronization.

[0093] Specifically, the data collection module can be divided into a data acquisition submodule, a data aggregation submodule and a data transmission submodule.

[0094] The data acquisition submodule (IMU sensor group), mainly the nine-axis gyroscope jy901s, is used to measure the three-axis and three-circle data of the accelerometer, gravimeter, and magnetometer of each body node, and is attached to the data aggregation module.

[0095] The data aggregation submodule uses the stm32f103 development board as an aggregation node to aggregate data from different nodes and perform time synchronization. It is installed on the thigh, calf, and above the belly.

[0096] The data transmission submodule uses the ESP86 IoT development board to connect with the stm32f103, acts as the client of the mqtt protocol for the server to subscribe, and is attached to the data aggregation module.

[0097] The PC processing module calls a computer program to perform the following steps:

[0098] Step A1: extract the features of the sensor information and input them into the trained Bayesian classifier to obtain the classification results.

[0099] Step A2: determine whether the classification result is a fall. If not, continue to receive data from the data acquisition module; if so, build a lower limb real-time model based on Unity according to the human joint information, and extract the key frame of the lower limb real-time model at the moment of falling.

[0100] Step A3: Use the image recognition neural network to determine whether the person has fallen based on the key frames. If the person has not fallen, continue to receive data from the data acquisition module. If the person has fallen, output the fall information.

[0101] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by the processor, the human fall detection based on sensor and image processing fusion as mentioned in the embodiment of the present invention can be implemented, and any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0102] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A human fall detection method based on sensor and image processing fusion, characterized in that: The following steps are involved: S1, obtaining sensing data of a human body sensor group; S2, extract the features of the sensor data and input them into the trained Bayesian classifier to obtain the classification results; S3, determine whether the classification result is a fall, if not, execute step S1; if yes, build a lower limb real-time model based on Unity according to the human joint information, and extract the key frame of the lower limb real-time model at the moment of falling; S4, using an image recognition neural network to determine whether the person has fallen according to the key frame, if not, executing step S1, if the person has fallen, outputting the fall information; The training process of the Bayesian classifier is as follows: Taking the features of the sensor data with the fall classification labels and the non-fall classification labels as input and the classification labels as output, a Bayesian classifier is trained; The sensing data feature extraction steps in step S2 are as follows: S21, filtering the sensor data through a Kalman filter to eliminate random deviations and obtain primary filtered data; S22, setting a threshold, removing data exceeding the threshold in the primary filtering data, and obtaining secondary filtering data; S23, calculating the standard deviation, mean and covariance of the secondary filtering data as data features; In step S3, the specific steps of establishing the lower limb real-time model based on Unity are as follows: S31, obtaining the world coordinates of several nodes of the original human body joints, and converting them into posture quaternions in the world coordinate system; S32, according to the IMU data in the sensor data, obtain the angular velocity information in the world coordinate system, and calculate the attitude quaternion at the next moment in combination with the attitude quaternion; S33, updating the joint node information of the lower limb real-time model according to the posture quaternion at the next moment; The calculation expression of the attitude quaternion at the next moment is as follows: Among them, q t Represents the quaternion of the next moment’s attitude, q t-1 Represents the current attitude quaternion, ω t represents angular velocity information, and Δt represents unit time.

2. According to claim 1, a human fall detection method based on sensor and image processing fusion is characterized in that: The filtering steps of the Kalman filter are as follows: S211, obtaining the initial value and initial confidence of the sensor data, and combining the covariance matrix and sampling time of the noise to obtain the observation value; S212: Update the covariance matrix and the sensor data values ​​according to the observed values.

3. The human fall detection method based on sensor and image processing fusion according to claim 1 is characterized in that: The sensor data includes accelerometer data, gravimeter data and magnetometer data.

4. The human fall detection method based on sensor and image processing fusion according to claim 1 is characterized in that: In step S4, the steps of establishing the image recognition neural network are as follows: A large amount of image information with fall classification labels and non-fall classification labels is input into the neural network, and the classification labels are used as output to train the image recognition neural network.

5. A human fall detection system based on sensor and image processing fusion, characterized in that: Including data acquisition module and PC processing module: The data acquisition module is used to obtain the sensor information of sensors at various nodes of the body, aggregate and transmit it to the PC processing module after time synchronization; The PC processing module performs the following steps: A1. Extract the features of sensor information and input them into the trained Bayesian classifier to obtain the classification results; A2. Determine whether the classification result is a fall. If not, continue to receive data from the data acquisition module. If yes, build a lower limb real-time model based on Unity according to the human joint information, and extract the key frame of the lower limb real-time model at the moment of falling. A3. Use the image recognition neural network to determine whether the person has fallen according to the key frame. If the person has not fallen, continue to receive data from the data acquisition module. If the person has fallen, output the fall information. The sensing data feature extraction steps in step A1 are as follows: S21, filtering the sensor data through a Kalman filter to eliminate random deviations and obtain primary filtered data; S22, setting a threshold, removing data exceeding the threshold in the primary filtering data, and obtaining secondary filtering data; S23, calculating the standard deviation, mean and covariance of the secondary filtering data as data features; The specific steps to build a real-time model of the lower limbs based on Unity are as follows: S31, obtaining the world coordinates of several nodes of the original human body joints, and converting them into posture quaternions in the world coordinate system; S32, according to the IMU data in the sensor data, obtain the angular velocity information in the world coordinate system, and calculate the attitude quaternion at the next moment in combination with the attitude quaternion; S33, updating the joint node information of the lower limb real-time model according to the posture quaternion at the next moment; The calculation expression of the attitude quaternion at the next moment is as follows: Among them, q t Represents the quaternion of the next moment’s attitude, q t-1 Represents the current attitude quaternion, ω t represents angular velocity information, and Δt represents unit time.

Images