Real-time fall detection method based on depth image sequences

Abstract

The present invention discloses a real-time fall detection method based on a depth image sequence. The present invention uses a depth image sequence as input, trains a target detection neural network, and outputs the positioning information and posture classification of the human target in each frame of the image; then, a sliding time window is used to match and track the human target, perform a time series analysis on the human posture, calculate the human posture height and its descent speed, imbalance time and other features; finally, the time series features are analyzed to make a judgment on whether the human body has fallen. The present invention can be applied to fall detection in single-person scenes and multi-person scenes, and has achieved advanced performance in indicators such as accuracy, precision, and recall rate; only depth images are collected, and user privacy protection is good; the algorithm complexity is low, and it can be deployed on a low-power and low-computing terminal device for real-time operation.

Description

Technical Field

[0001] The present invention relates to the fields of three-dimensional vision, human posture tracking, abnormal behavior event detection, deep learning, and more particularly to a real-time fall detection method based on depth image sequences. Background Art

[0002] With the aging population, elderly care has become a hot topic in society. Statistics show that falls are the leading cause of injury and even death among people over 65. While falls are difficult to predict, timely medical attention can effectively reduce the injury and mortality rates associated with falls. Therefore, a fall detection system with high accuracy, strong robustness, and real-time performance has significant application value.

[0003] Currently, mainstream fall detection methods fall into three main categories: wearable sensor-based fall detection methods, distributed environmental sensor-based fall detection methods, and computer vision-based fall detection methods. Wearable sensor-based methods require users to wear devices such as accelerometers on certain parts of their bodies. The devices analyze sensor data to determine whether the user has fallen. However, prolonged wear can cause discomfort and inconvenience in daily life. Furthermore, elderly people often suffer from memory loss, making them prone to forgetting to wear or losing their devices. Environmental sensor-based methods require the pre-deployment of complex sensor networks at the target location, primarily pressure and sound sensors. This leads to high deployment and maintenance costs, limiting their application scenarios. Among computer vision-based methods, most current algorithms use RGB data as input and process time series information using complex convolutional neural networks or recurrent neural networks. However, the acquisition of RGB images can infringe on user privacy. While these algorithms offer high accuracy, their computational complexity makes real-time detection difficult to guarantee.

[0004] Artificial intelligence technology has developed rapidly in recent years, and hardware computing power has continued to increase. Some complex detection networks that achieve high accuracy can also achieve real-time results on computer platforms. However, actual applications often use edge processing or centralized data transmission to servers for processing. The former requires the hardware to be as small as possible and capable of integration into depth cameras, while high-performance computer platforms are generally larger. The latter needs to meet the high concurrency requirements of multiple data inputs simultaneously. Therefore, while complex networks can improve accuracy, they are difficult to achieve widespread practical application in real life. Summary of the Invention

[0005] In order to solve the problems in the background technology, the present invention provides a real-time fall detection method based on depth image sequences.

[0006] The technical solution of the present invention is achieved by the following steps:

[0007] Step 1) Collect depth image sequences containing various fall actions and non-fall actions in indoor scenes, fuse them with the public fall dataset to obtain a fused dataset, draw a circumscribed rectangular box for each target human body in each depth image in the fused dataset, and annotate the human posture category to obtain a fall detection dataset based on depth images;

[0008] Step 2) Building a real-time object detection network model for detecting human positions in depth images and identifying human posture features, and training the network using gradient descent using the depth image-based fall detection dataset from step 1) until convergence;

[0009] Step 3) The depth image sequence to be detected is input into the real-time target detection network model. The model outputs the rectangular frame positioning information of each target person in each image and the probability that the corresponding human posture belongs to each posture category. The posture category with the highest probability is selected as the posture judgment of the person in the corresponding rectangular frame for subsequent time series analysis;

[0010] Step 4) For each human body rectangle obtained by the target detection network model, a horizontal sampling line with a width of one pixel is selected, and the top pixels within the bounding box are sampled. Then, the pixels on the sampling line are merged into several superpixels using a region growing algorithm. A number of candidate height values ​​are obtained using a coordinate back-projection transformation. According to the imaging principle, the maximum value among the candidate values ​​is the target posture height;

[0011] Step 5) Using a sliding time window of set length and stride, matching and tracking the human targets detected in each frame image within each time window is performed to determine whether the human posture change process meets the necessary conditions for falling;

[0012] Step 6) For each time window that satisfies the fall condition in step 5), Kalman filtering and Kalman smoothing are performed on the posture height and descent speed data of each human target in the current time window to obtain the optimal estimate of the posture height and descent speed of each target;

[0013] Step 7) combines the human posture detection results obtained in step 3) to extract the maximum value of the posture height descent speed when the human target is in the transition posture, compares it with the preset threshold, and combines the transition posture duration to determine whether the target falls within the time window.

[0014] The step 1) is specifically as follows:

[0015] According to human kinesiology and medical knowledge, the dynamic changes of human posture during a fall are divided into three necessary stages: the normal stage, the imbalance stage, and the falling stage.

[0016] Normal stage: refers to the short period before the fall, when the target still maintains normal behavior, such as walking, standing, sitting, etc.

[0017] Unbalanced stage: refers to the time when the fall occurs, when the target loses control of their body balance and the body's center of gravity is in an uncontrollable downward state;

[0018] Falling stage: refers to the target's body center of gravity has stopped falling, with the buttocks, back, legs or chest and other parts of the body in contact with the ground over a large area, and the target is sitting, lying or lying on the ground;

[0019] Human body posture categories are labeled using the following classification criteria:

[0020] Normal posture: The upper body remains upright, the body's only contact surface with the ground is the feet, the center of gravity height changes remain stable, and the body is in the state of standing, walking, and sitting; corresponding to the normal stage of falling

[0021] Transition posture: The center of gravity of the human body is in the descending stage; corresponding to the imbalance stage of falling.

[0022] Landing posture: The human body contacts the ground in a lying, prone, or sitting position. The contact area with the ground is not limited to the feet, and the center of gravity height changes remain stable; this corresponds to the falling stage.

[0023] Maintaining a stable center of gravity height means that the center of gravity fluctuation does not exceed 1 / 2 of the distance from the knee to the heel.

[0024] The step 2) is specifically as follows:

[0025] The real-time object detection network model uses the YOLOv4 network model. The input is a fall detection dataset based on depth images. The output is the coordinate information of each human object bounding box in each frame image and the probability that the human posture belongs to each category. The loss function is calculated based on this information.

[0026] The real-time object detection network model is iteratively trained using the gradient descent method until convergence.

[0027] The step 3) is specifically as follows:

[0028] 3-1) Offline testing on public datasets:

[0029] A depth map sequence of known length from a public dataset is input into a real-time object detection network model. The model outputs the location information of each human target rectangle and the probability that the corresponding human posture belongs to each posture category. The posture category with the highest probability is selected as the posture judgment of the human body in the corresponding rectangle for subsequent time series analysis.

[0030] 3-2) Real-time online testing:

[0031] The depth image sequence acquired in real time by the depth camera is input into the real-time target detection network model. The model outputs the positioning information and posture judgment of each human target rectangle in real time for subsequent timing analysis.

[0032] The step 4) is specifically as follows:

[0033] 4.1) For each human body rectangle in the depth image, select a horizontal sampling line with a single pixel width:

[0034] Construct the image coordinate system O-uv, the equation of the horizontal sampling line is v = b, and the calculation formula of the parameter b is:

[0035] b=top+(bot-top)*p%

[0036] Where top is the v-axis parameter of the horizontal line at the top of the rectangular box, boh is the v-axis parameter of the horizontal line at the bottom of the rectangular box, and p% is the ratio of the distance between the horizontal sampling line and the top to the height of the rectangular box (set by yourself).

[0037] 4.2) Apply the region growing algorithm to merge the pixels on the horizontal sampling line into several superpixels:

[0038] Sequentially scan the pixels on the horizontal sampling line and calculate the absolute value of the difference between the depth value of the current pixel and the mean depth value of the current superpixel. The current superpixel is the superpixel to which the previous pixel of the current pixel belongs. If the absolute value is less than the given threshold ε1, the current pixel is added to the current superpixel. Otherwise, the current pixel is included in the new superpixel (the first pixel on the horizontal sampling line is the first new superpixel).

[0039] 4.3) Calculate the true height above the ground corresponding to the mean depth of each superpixel through coordinate back-projection transformation and add it to the candidate attitude height value set;

[0040]

[0041] Among them, height is the candidate height value; ave is the mean depth value of the current superpixel; u0, f x Both are intrinsic parameters of the camera, and their physical meanings are the horizontal pixel difference between the center pixel coordinate of the image and the origin pixel coordinate of the image coordinate system O-uv, and the horizontal focal length of the camera; H and θ are extrinsic parameters of the camera, and their physical meanings are the height of the camera in the world coordinate system and the depression angle of the camera, respectively.

[0042] 4.4) Apply the region growing algorithm to weight and merge the attitude height candidate values ​​that differ within a given threshold ε2 into new candidate values;

[0043] 4.5) Select the largest height candidate value from the height candidate values ​​updated in step 4.4) as the posture height of the human body in the current rectangular frame.

[0044] The step 5) is specifically as follows:

[0045] The input depth image sequence is segmented using a sliding time window of set length and stride (stride is less than length), and the following detection is performed on all depth image sequences within each time window:

[0046] Matching and tracking are performed on each human target detected in each frame of the depth image sequence to determine whether the posture change process of the human target meets the necessary conditions for falling: if the posture of the human body in the current time window undergoes the transformation process of normal posture-transition posture-landing posture in sequence, the necessary conditions for falling are met; otherwise, the necessary conditions for falling are not met, and it is determined that the human target has not fallen in the current time window.

[0047] The step 6) is specifically as follows:

[0048] 6-1) For each time window that meets the fall condition in step 5), calculate the state transition equation based on the basic motion equation:

[0049] z(t)=z(t-1)+v z (t-1)

[0050] v z (t) = v z (t-1)

[0051] Among them, z(t), v z (t) are the posture height and descent speed of the human target at time t, z(t-1), v z (t-1) are the posture height and descent speed of the human target at time t-1 respectively; due to the short time, the descent process is regarded as uniform linear motion, and the state transfer matrix A is obtained as follows:

[0052]

[0053] The Kalman filter system state signal at each moment is obtained according to the following Kalman filter state transfer equation:

[0054] X k+1 =AX k +W k

[0055] Among them, X k+1 , X k are the Kalman filter system state signals at time k+1 and k respectively, under the condition that the measurement values ​​at the current time and the previous time are known; W kis the system noise, which satisfies Gaussian distribution;

[0056] Among them, the measured values ​​are attitude height and descent speed;

[0057] 6-2) Based on the Kalman filter system state signal obtained in step 6-1), under the condition that the measurement values ​​in the current time window are known, the Kalman smoothing algorithm is applied to calculate the Kalman smoothed system state signal X′ at each moment k ;

[0058] 6-3) Obtain the optimal estimate Y of the attitude height and descent speed at each moment according to the following measurement equation: k :

[0059] Y k =CX′ k +V k

[0060] Among them, Y k is the measurement signal of the system at time k under the condition that the measurement values ​​of the current time and the previous time are known; C is the measurement matrix, which is a second-order unit matrix; X′ k is the Kalman smoothed system state signal processed in step 6-2); V k is the observation noise, which satisfies the Gaussian distribution.

[0061] The step 7) is specifically as follows:

[0062] For each time window that meets the fall condition, the duration t when the human target is in the transition posture is obtained based on step 3). off Based on step 6), the optimal estimate of the descent speed at each moment in the current time window is obtained, and the maximum value of the optimal estimate is taken as the maximum speed v of the attitude height descent. off ;

[0063] t off The set imbalance stage duration threshold t th , v off With the falling speed threshold v th1 and v th2 (v th1 <v th2 ) to compare and determine the status of each human target in the current time window:

[0064] 7-1) When v is satisfied off <v th1 , indicating that although the human body has a falling-like movement, the speed of the height drop is small enough, and it is judged to be safe;

[0065] 7-2) When v is satisfied th1 <v off <vth2 And t off >t th , indicating that although the human body has a falling-like movement and the posture height drops at a high speed, the imbalance phase lasts long enough, indicating that the human body's transition posture is under control, but there is still danger, and the judgment is "Caution Needed";

[0066] 7-3) When v is satisfied off >v th2 , or v th1 <v off <v th2 And t off <t th , indicating that the height of the human body is decreasing in an uncontrollable state and is judged as a fall.

[0067] The beneficial effects of the present invention are:

[0068] This invention effectively improves the accuracy of fall detection and reduces misjudgments of similar fall behaviors, such as lying down to find something or lying down voluntarily. It has a wide range of applications and can be applied to fall detection in both single-person and multi-person scenarios, achieving advanced performance in indicators such as accuracy, precision, and recall. Only depth images are collected, providing good user privacy protection. It also utilizes a low-complexity target detection network, which can run in real time on low-power, low-computing-power embedded terminal devices, expanding the application market for intelligent fall detection systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is an overall flow chart of the fall detection function implemented in the present invention;

[0070] Figure 2 Three examples of target postures that have been marked;

[0071] Figure 3 This is a relationship diagram of the modules in the ROS system;

[0072] Figure 4 is the grayscale value of each pixel on the sampling line in the depth map;

[0073] Figure 5 Schematic diagram of the ToF camera layout;

[0074] Figure 6 Flowchart of the fall detection algorithm. DETAILED DESCRIPTION

[0075] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0076] In particular, in this embodiment, the depth image acquisition device uses a ToF (Time-of-Flight) depth camera, and the target detection network uses a YOLOv4 detection network. The above selection is only a preferred embodiment of the present invention and is not intended to limit the present invention.

[0077] The present invention first refines the falling behavior and establishes the necessary conditions for the occurrence of a falling event:

[0078] The fall behavior to be detected is broken down into three stages based on relevant medical literature: the normal stage, the imbalance stage, and the final fall stage. The specific definitions of each stage are as follows:

[0079] 1) Normal stage: refers to the short period before the fall, when the target still maintains normal behavior, such as walking, standing, sitting, etc.

[0080] 2) Unbalanced stage: refers to the stage when the fall occurs, when the target loses control of his or her body balance and the center of gravity of the body is in an uncontrollable downward state;

[0081] 3) Falling stage: refers to the target's body center of gravity has stopped falling, with the buttocks, back, legs or chest in contact with the ground over a large area, such as sitting on the ground or lying on the ground.

[0082] The second is the specific implementation process of this method.

[0083] like Figure 1 As shown, the real-time fall detection method based on depth image sequence of the present invention includes the following steps:

[0084] Step 1: Use a ToF camera to capture depth image sequences containing various fall and non-fall actions. At the same time, collect relevant public datasets, including the TST Fall Detection dataset V2. Combined with the above three-stage human posture classification, classify the human posture in each depth image frame and annotate the rectangular box of the human outline to construct a fall detection dataset based on depth images.

[0085] The annotation process of each depth image in the fall detection dataset is as follows:

[0086] 1.1) Label Determination: Based on the above-mentioned detailed description of the fall phase, human posture can be divided into the following three categories: normal posture, transition posture, and landing posture;

[0087] 1.1.1) Normal Posture: This corresponds to the first stage of a fall, the normal stage, where the subject is in a standing, sitting, or other position with the upper body upright.

[0088] 1.1.2) Transitional Position: This corresponds to the second stage of a fall—the off-balance phase. This refers to a position in which the subject's center of gravity is in the downward phase of the fall, but the majority of the body is not in contact with the ground. In particular, normal movements of the upper body, such as bending over or flexing, are also considered this position.

[0089] 1.1.3) Landing Posture: This corresponds to the third stage of a fall, the falling stage, and refers to the posture in which the target body's buttocks, back, chest, legs, etc., are in contact with the ground over a large area, such as sitting, lying down, or prone on the ground.

[0090] 1.2) Convert the acquired video into an image sequence frame by frame, preprocess each image, and then calibrate each target human body in the image according to the above classification.

[0091] The main operations of preprocessing are: filling the holes in the depth map through the morphological erosion algorithm, removing the depth image noise with a Gaussian filter, calculating the scaling factor, and projecting the 16-bit ToF depth map to 8 bits so that all pixel values ​​in its channel fall within the range of 0 to 255.

[0092] 1.2.1) Calculate the scaling factor α:

[0093]

[0094] Among them I dmax is the maximum depth value in the ToF depth map, I dmin The minimum non-zero depth value in the ToF depth map.

[0095] The projected 8-bit image can be calculated using the following formula:

[0096] I n =(I d -I dmin )*α

[0097] Among them I d is the pixel value in the original 16-bit ToF depth image, I n is the pixel value in the projected image.

[0098] Thus, the preprocessed depth image is obtained, and each pixel value falls within the interval [0,255].

[0099] 1.3) According to the definition, mark each target in the depth image.

[0100] The results after annotation are as follows: Figure 2 .

[0101] Step 2: Build a YOLOv4 real-time detection network for human target detection and human posture features, and use the dataset created in step 1 to train the above network using the gradient descent method.

[0102] The model input is a ToF depth image, and outputs the positioning information of each human target bounding box in each image and the probability that the human posture belongs to each category. The loss function is calculated based on this, and the network is iteratively trained using the gradient descent method until convergence.

[0103] Step 3: Apply the network model obtained in step 3 for inference, input the ToF depth image sequence, and output the posture and positioning information of the human target in each frame of the image;

[0104] The network is tested in two modes: offline testing using a public dataset and online real-time testing using a ToF camera and data processing.

[0105] 3.1) Offline Test Mode: The model inputs a depth map sequence of known length from a public dataset and outputs the location information corresponding to one or more bounding boxes of the target, along with the probability of the object belonging to each class within the box. The class with the highest probability is selected as the pose judgment for the object within the box. After obtaining all detection results, the temporal information processing method described later is applied to implement fall detection and judgment.

[0106] 3.2) Real-time online test mode: Based on the ROS (Robot Operation System) framework, the ToF camera module, model detection module, and timing information processing module are established. For the specific module relationship diagram and data flow diagram, see Figure 3 The ToF camera is connected to the PC to obtain the depth map data captured by the ToF camera in real time. The ROS framework publishes topics through the ToF camera module, and the model detection module subscribes to the corresponding topics to complete the depth image data transmission. The YOLOv4 model in the model detection module infers the incoming data, determines the positioning information of each target bounding box and the probability that the object in the box belongs to each category, and selects the category with the highest probability as the posture judgment of the object in this box. The above detection results are also transmitted to the timing information processing module by publishing topics and subscribing to topics. After the detection results are processed by the above module, a judgment is made on whether the target has fallen;

[0107] Step 4: Apply the region growing algorithm to obtain the pixels belonging to the top of the human body in the depth map. Combined with the camera imaging principle, the coordinate back projection transformation is used to obtain the approximate value of the human body height.

[0108] For each bounding box obtained by network detection, a horizontal sampling line with a single pixel width is selected to sample the pixels within the bounding box. The pixels on the sampling line are then merged into several superpixels using the region growing algorithm. Combined with the camera inverse projection transformation, several pose height candidate values ​​are obtained. According to the imaging principle, the maximum value among the candidate values ​​is the target pose height, such as Figure 5 As shown;

[0109] The specific implementation of step 4 above is as follows:

[0110] 4.1) In the pixel coordinate system O-uv, calculate the parameter b of the equation v=b for the horizontal sampling line:

[0111] b=top+(bot-top)*p%

[0112] Where top is the v-axis parameter of the horizontal line at the top of the rectangular box, bot is the v-axis parameter of the horizontal line at the bottom of the rectangular box, and p% is the ratio of the distance between the horizontal sampling line and the top to the height of the rectangular box.

[0113] 4.2) Apply the region growing algorithm to merge the pixels on the sampling line into several superpixels, and record the mean depth value of each superpixel and the number of pixels it contains:

[0114] The pixels on the sampling line are operated from left to right. The difference between the depth value of the current pixel and the mean depth value of the current superpixel is calculated, and compared with the given threshold ε1, and a judgment is made as to whether the current pixel belongs to the current superpixel.

[0115] 4.2.1) The current pixel belongs to the current superpixel: Update the mean depth value of the current superpixel and the number of pixels it contains:

[0116] l′=l+1

[0117]

[0118] Where ave is the mean superpixel depth value before the update, l is the number of pixels in the current superpixel before the update, and cur is the depth value of the current pixel. ave′,l′ are both the values ​​after the current update. At the end of this update round, the updated values ​​ave′,l′ are used as the new ave,l.

[0119] 4.2.2) The current pixel does not belong to the current superpixel: the current pixel is assigned to a new superpixel;

[0120] 4.3) Calculate the true height value corresponding to each superpixel depth candidate value through coordinate back-projection transformation and record it into the pose height candidate value set:

[0121]

[0122] S′=S∪(height,l)

[0123] Where S is the candidate value set of attitude height before updating, S′ is the value after this round of updating, and at the end of this round of updating, the updated value S′ is used as the new S. height is the candidate value of attitude height, u0, f x Both are internal parameters of the ToF camera, and their physical meanings are the horizontal pixel difference between the center pixel coordinate and the origin pixel coordinate of the image, and the horizontal focal length of the ToF camera. H and θ are external parameters of the ToF camera, and their physical meanings are the height of the ToF camera in the world coordinate system and the depression angle of the ToF camera. The specific ToF camera layout is as follows: Figure 6 As shown;

[0124] 4.4) Apply the region growing algorithm to weight and merge the height candidate values ​​that differ within the given threshold ε2 into new candidate values:

[0125] L′=L+l

[0126]

[0127] where h new is the weighted combined target pose height candidate value before updating, L is the corresponding weight (number of pixels) before updating, h cur is the candidate value of the current posture height, and l is the weight (number of pixels) of the candidate value of the current posture height. new , L′ are the values ​​after the current round of update. In the next round of update, the updated value h′ of this round will be new , L′ as the new h new , L;

[0128] 4.5) Select the maximum value of the candidate posture height values, which is the target human posture height;

[0129] Step 5: Segment the continuous time information by setting time windows, and perform a detection in each time window. Perform matching tracking on the human targets detected in each frame image within the window, and then determine whether the posture change of each target in the time window meets the necessary conditions for falling:

[0130] 5.1) Track multiple targets detected within the time window;

[0131] For two adjacent frames, the number of targets detected in the previous frame and the next frame is calculated respectively. Based on the number, it is determined whether any targets have entered or left the ToF camera's field of view. The Euclidean distance between the center of the bounding box of each target in the previous frame and the center of the bounding box of each target in the next frame is calculated in the pixel coordinate system. The resulting distance matrix is ​​the number of targets in the previous frame * the number of targets in the next frame. The Hungarian algorithm is applied to match the targets in the previous and next frames, that is, to track the targets.

[0132] 5.2) Determine the posture change of each target within the time window:

[0133] 5.2.1) If the target's posture changes from normal posture to transition posture to landing posture, this indicates that the target has sequentially gone through the normal phase, the imbalance phase, and the falling phase, thus meeting the necessary conditions for a fall.

[0134] 5.2.2) If the target's posture does not undergo a transition from normal posture to transition posture to landing posture, then the target does not meet the necessary conditions for falling, and the target has not experienced a fall event within the current time window;

[0135] 5.2.3) Calculate the average detection height of the target in the last 10 frames in the time window

[0136] Let h th is the preset target attitude height threshold, if This indicates that the target still maintains a relatively high altitude at the end of the time window, and the target has not fallen within the current time window.

[0137] At this point, some non-fall behaviors have been detected, and targets that have been judged as non-falls do not require subsequent calculations.

[0138] Step 6: Combined with the attitude height detection results of step 4, the attitude height and descent speed information of the target in this time window are obtained, and the Kalman filter and Kalman smoothing algorithm are used to best estimate the attitude height and descent speed of the target in each frame;

[0139] Each target is judged in turn, and the target's attitude height and descent speed information are linearly filtered through Kalman filtering and Kalman smoothing within the time window to obtain the optimal estimate of the target's attitude height and descent speed. Then, combined with the detection conclusion of step 3, the maximum descent speed of the target in the transition attitude is obtained;

[0140] 6-1) For each time window that meets the fall condition in step 5), calculate the state transition equation based on the basic motion equation:

[0141] z(t)=z(t-1)+v z (t-1)

[0142] v z (t) = v z (t-1)

[0143] Among them, z(t), v z (t) are the posture height and descent speed of the human target at time t, z(t-1), v z(t-1) are the posture height and descent speed of the human target at time t-1 respectively; due to the short time, the descent process is regarded as uniform linear motion, and the state transfer matrix A is obtained as follows:

[0144]

[0145] The Kalman filter system state signal at each moment is obtained according to the following Kalman filter state transfer equation:

[0146] X k+1 =AX k +W k

[0147] Among them, X k+1 , X k are the Kalman filter system state signals at time k+1 and k respectively, under the condition that the measurement values ​​at the current time and the previous time are known; W k is the system noise, which satisfies Gaussian distribution;

[0148] Among them, the measured values ​​are attitude height and descent speed;

[0149] 6-2) Based on the Kalman filter system state signal obtained in step 6-1), under the condition that the measurement values ​​in the current time window are known, the Kalman smoothing algorithm is applied to calculate the Kalman smoothed system state signal X′ at each moment k ;

[0150] 6-3) Obtain the optimal estimate Y of the attitude height and descent speed at each moment according to the following measurement equation: k :

[0151] Y k =CX′ k +V k

[0152] Among them, Y k is the measurement signal of the system at time k under the condition that the measurement values ​​of the current time and the previous time are known; C is the measurement matrix, which is a second-order unit matrix; X′ k is the Kalman smoothed system state signal processed in step 6-2); V k is the observation noise, which satisfies the Gaussian distribution.

[0153] Step 7: Combined with the posture detection results of step 3, the maximum descent speed of the target in the transition posture in the result of step 6 is obtained, and compared with the pre-set threshold to determine whether the target falls within the time window.

[0154] 7.1) Record the duration of the target's imbalance phase (t) as it goes through the normal phase, imbalance phase, and fall phase. off , and obtain the descent speed of the target at the corresponding moment in the transition posture, and record the maximum value as v off ;

[0155] 7.2) According to the set imbalance stage duration threshold t th With the falling speed threshold v th1 , v th2 (v th1 <v th2 ), the fall situations of each target within the time window are divided into three categories;

[0156] 7.2.1) Safety: Satisfy v off <v th1 ,Although there is a similar behavior of falling, the falling speed is small enough, and the target is judged to be safe within the current time window;

[0157] 7.2.2) Note: Satisfy v th1 <v off <v th2 And t off >t th Although there is a similar behavior to falling and a certain descent speed, the imbalance phase lasts for a long time (indicating that the target body can control the downward movement). The current time window of the target is judged as requiring attention. That is, although there is no fall incident at present, the current posture is still dangerous and requires the attention of relevant personnel;

[0158] 7.2.3) Falling: One of the following two conditions is met: v off >v th2 (speed is greater than a given threshold), or v th1 <v off <v th2 And t off <t th (There is a certain descending speed and the duration of the imbalance phase is short, indicating that the target's downward movement is uncontrollable), the target is judged to have fallen in the current time window.

[0159] Based on this, the detection of falling behavior is completed.

[0160] Under the hardware conditions of i7-9700 CPU and GeForce RTX 2080Ti graphics card, Ubuntu16.04lts+cuda 10.1+ROS PC environment was built, and the ToF camera used in the experiment was Microsoft Kinect V2. Based on the above environment, offline experiments and online experiments were designed to verify the feasibility, real-time and accuracy of the present invention. The offline test was carried out on the public dataset TST Fall detection dataset V2. The experimental results showed that the method achieved an accuracy of 97.92%, a sensitivity of 97.5% and a specificity of 98.3%, with an operating frame rate of 45Hz. In the real-time online test, both the single-person scene and the multi-person scene achieved an accuracy of 100%, and the operating frame rate was 40Hz. The above experiments show that the present invention not only effectively improves the accuracy of fall recognition, but also ensures the real-time operation of the system.

[0161] In summary, the solution of the present invention first refines the event of falling, determines its necessary conditions, and then inputs the depth map captured by the ToF camera into the pre-trained YOLOv4 model. The model outputs the posture information of each target in the depth map and the position information of its bounding box. The subsequent time series information processing module further processes and refines the network detection results: combining the depth map information, the camera imaging principle and the network detection results, the target posture height is obtained, and the maximum speed of the posture height drop when in the unbalanced stage is determined. This is used as the criterion for judging whether a fall has occurred. This solution has a wide range of application scenarios. While ensuring user privacy and real-time system operation, it also has high detection accuracy and exhibits good detection performance.

[0162] It should be noted that the above embodiments can be freely combined as needed, and this embodiment is only a preferred embodiment of the present invention. Without departing from the principle of the present invention, the present invention can be improved and modified in a number of ways, and these improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A real-time fall detection method based on depth image sequence, characterized in that: The following steps are involved: Step 1) Collect a depth image sequence containing various fall actions and non-fall actions, fuse it with a public fall dataset to obtain a fused dataset, draw a circumscribed rectangular box for each target human body in each depth image in the fused dataset, and annotate the human posture category to obtain a fall detection dataset based on depth images; Step 2) Building a real-time object detection network model for detecting the position of a human body in a depth image and identifying human posture features, and training the network model using the depth image-based fall detection dataset from step 1) using a gradient descent method until convergence; Step 3) The depth image sequence to be detected is input into the real-time target detection network model. The model outputs the rectangular frame positioning information of each target person in each image and the probability that the corresponding human posture belongs to each posture category. The posture category with the highest probability is selected as the posture judgment of the person in the corresponding rectangular frame; Step 4) For each human body rectangle obtained by the target detection network model, a horizontal sampling line with a width of one pixel is selected, and the top pixels within the bounding box are sampled. Then, the pixels on the sampling line are merged into several superpixels using a region growing algorithm. A number of candidate height values ​​are obtained using a coordinate back-projection transformation. According to the imaging principle, the maximum value among the candidate values ​​is the target posture height; Step 5) Using a sliding time window of set length and stride, matching and tracking the human targets detected in each frame image within each time window is performed to determine whether the human posture change process meets the necessary conditions for falling; Step 6) For each time window that satisfies the fall condition in step 5), Kalman filtering and Kalman smoothing are performed on the posture height and descent speed data of each human target in the current time window to obtain the optimal estimate of the posture height and descent speed of each target; Step 7) combines the human posture detection results obtained in step 3) to extract the maximum value of the posture height descent speed when the human target is in the transition posture, compares it with the preset threshold, and combines the transition posture duration to determine whether the target falls within the time window.

2. The real-time fall detection method based on depth image sequence according to claim 1, characterized in that: In the step 1), the human body posture categories are marked by the following classification standards: Normal posture: The upper body remains upright, the body's only contact surface with the ground is the feet, the center of gravity height changes remain stable, and the body is in the state of standing, walking, and sitting; Transition posture: the body's center of gravity is in the descending stage; Landing posture: The human body contacts the ground in a lying, prone or sitting position. The contact area with the ground is not limited to the feet, and the height change of the center of gravity remains stable.

3. The real-time fall detection method based on depth image sequence according to claim 1, characterized in that: The step 2) is specifically as follows: The real-time object detection network model uses the YOLOv4 network model. The input is a fall detection dataset based on depth images. The output is the coordinate information of each human object bounding box in each frame image and the probability that the human posture belongs to each category. The real-time object detection network model is iteratively trained using the gradient descent method until convergence.

4. The real-time fall detection method based on depth image sequence according to claim 1, characterized in that: The step 3) is specifically as follows: 3-1) Offline testing on public datasets: A depth map sequence of known length from a public dataset is input into a real-time object detection network model. The model outputs the location information of each human target rectangle and the probability that the corresponding human posture belongs to each posture category. The posture category with the highest probability is selected as the posture judgment of the human body in the corresponding rectangle for subsequent time series analysis. 3-2) Real-time online testing: The depth image sequence acquired in real time by the depth camera is input into the real-time target detection network model. The model outputs the positioning information and posture judgment of each human target rectangle in real time for subsequent timing analysis.

5. The real-time fall detection method based on depth image sequence according to claim 1, characterized in that: The step 4) is specifically as follows: 4.1) For each human body rectangle in the depth image, select a horizontal sampling line with a single pixel width: Construct the image coordinate system O-uv, the equation of the horizontal sampling line is v = b, and the calculation formula of the parameter b is: b=top+(bot-top)*p% Where top is the v-axis parameter of the horizontal line at the top of the rectangular box, bot is the v-axis parameter of the horizontal line at the bottom of the rectangular box, and p% is the ratio of the distance between the horizontal sampling line and the top to the height of the rectangular box. 4.2) Apply the region growing algorithm to merge the pixels on the horizontal sampling line into several superpixels: Sequentially scan the pixels on the horizontal sampling line and calculate the absolute value of the difference between the depth value of the current pixel and the mean depth value of the current superpixel. The current superpixel is the superpixel to which the previous pixel of the current pixel belongs. If the absolute value is less than the given threshold ε1, the current pixel is added to the current superpixel; otherwise, the current pixel is included in the new superpixel. 4.3) Calculate the true height above the ground corresponding to the mean depth of each superpixel through coordinate back-projection transformation and add it to the candidate attitude height value set; Among them, height is the candidate height value; ave is the mean depth value of the current superpixel; u0, f x Both are intrinsic parameters of the camera, and their physical meanings are the horizontal pixel difference between the center pixel coordinate of the image and the origin pixel coordinate of the image coordinate system O-uv, and the horizontal focal length of the camera; H and θ are extrinsic parameters of the camera, and their physical meanings are the height of the camera in the world coordinate system and the depression angle of the camera, respectively. 4.4) Apply the region growing algorithm to weight and merge the attitude height candidate values ​​that differ within a given threshold ε2 into new candidate values; 4.5) Select the largest height candidate value from the height candidate values ​​updated in step 4.4) as the posture height of the human body in the current rectangular frame.

6. The real-time fall detection method based on depth image sequence according to claim 1, characterized in that: The step 5) is specifically as follows: The input depth image sequence is segmented using a sliding time window of set length and stride, and the following detection is performed on all depth image sequences within each time window: Matching and tracking are performed on each human target detected in each frame of the depth image sequence to determine whether the posture change process of the human target meets the necessary conditions for falling: if the posture of the human body in the current time window undergoes the transformation process of normal posture-transition posture-landing posture in sequence, the necessary conditions for falling are met; otherwise, the necessary conditions for falling are not met, and it is determined that the human target has not fallen in the current time window.

7. The real-time fall detection method based on depth image sequence according to claim 1, characterized in that: The step 6) is specifically as follows: 6-1) For each time window that meets the fall condition in step 5), calculate the state transition equation based on the basic motion equation: z(t)=z(t-1)+v z (t-1) v z (t)=v z (t-1) Among them, z(t), v z (t) are the posture height and descent speed of the human target at time t, z(t-1), v z (t-1) are the posture height and descent speed of the human target at time t-1 respectively; the descent process is regarded as uniform linear motion, so the state transfer matrix A is obtained as follows: The Kalman filter system state signal is obtained according to the following Kalman filter state transfer equation: X k+1 =AX k +W k Among them, X k+1 , X k are the Kalman filter system state signals at time k+1 and k respectively, under the condition that the measurement values ​​at the current time and the previous time are known; W k is the system noise, which satisfies Gaussian distribution; Among them, the measured values ​​are attitude height and descent speed; 6-2) Based on the Kalman filter system state signal obtained in step 6-1), under the condition that the measurement values ​​in the current time window are known, the Kalman smoothing algorithm is applied to calculate the Kalman smoothed system state signal X′ at each moment k ; 6-3) Obtain the optimal estimate Y of the attitude height and descent speed at each moment according to the following measurement equation: k : Y k =CX′ k +V k Among them, Y k is the measurement signal of the system at time k under the condition that the measurement values ​​of the current time and the previous time are known; C is the measurement matrix, whose value is a second-order unit matrix; X′ k is the Kalman smoothed system state signal processed in step 6-2); V k is the observation noise, which satisfies the Gaussian distribution.

8. The real-time fall detection method based on depth image sequence according to claim 1, characterized in that: The step 7) is specifically as follows: For each time window that meets the fall condition, the duration t when the human target is in the transition posture is obtained based on step 3). off Based on step 6), the optimal estimate of the descent speed at each moment in the current time window is obtained, and the maximum value of the optimal estimate is taken as the maximum speed v of the attitude height descent. off ; t off The set imbalance stage duration threshold t th , v off With the falling speed threshold v th1 and v th2 Compare and judge the status of each human target in the current time window: 7-1) When v is satisfied off <v th1 , indicating that although the human body has a falling-like movement, the speed of the height drop is small enough, and it is judged to be safe; 7-2) When v is satisfied th1 <v off <v th2 And t off >t th , indicating that although the human body has a falling-like movement and the posture height drops at a high speed, the imbalance phase lasts long enough, indicating that the human body's transition posture is under control, but there is still danger, and the judgment is "Caution Needed"; 7-3) When v is satisfied off >v th2 , or v th1 <v off <v th2 And t off <t th , indicating that the height of the human body is decreasing in an uncontrollable state and is judged as a fall.

Images