A human fall detection method and device based on point cloud

Abstract

The application discloses a human body fall detection method and device based on point cloud, and belongs to the field of fall detection. The method specifically comprises the following steps: converting the starting frame of a depth image sequence collected indoors into point cloud, and extracting the gravity direction; combining the texture, optical flow and neural network of the depth image sequence to extract the human body movement area; tracking the human body movement area, converting the area into point cloud frame by frame, and performing denoising and gravity direction correction; clustering and matching the human body movement area point cloud; performing size analysis on the matched point cloud to confirm the human body point cloud; and analyzing the human body point cloud barycenter movement and point cloud pose state to detect whether a fall occurs. The human body fall detection method and device disclosed by the application fully excavate the motion characteristics of the depth image and the three-dimensional physical characteristics of the point cloud, do not require supervision information, and ensure that the application has high interpretability, accuracy, robustness, real-time performance, privacy and practicability.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of fall detection, and more particularly, to a human body fall detection method and device based on point cloud. BACKGROUND

[0002] The world population is aging, and the elderly, especially the elderly living alone, are prone to falls. The use of human fall detection technology can quickly detect fallen elderly people and issue an alarm, thereby achieving effective treatment and reducing the national and family medical burden.

[0003] With the advent of the big data era, algorithms based on neural networks have developed rapidly, GPU processing data power has improved significantly, and target detection and behavior detection technologies have made significant progress, providing a good research foundation for human fall detection.

[0004] The current common fall detection methods mainly include:

[0005] (1) Fall detection based on wearable sensors

[0006] Common fall detection devices based on acceleration sensors include waistbands containing accelerometers, smartphones, smartwatches, smartbands, etc. This type of algorithm calculates the acceleration or speed of the hand and body in time series to determine whether a fall has occurred. In 2014, Wu Zhiqiang et al. proposed a human fall detection system design based on a smartphone. The advantages of this type of algorithm are that there are no privacy issues and no need for fixed devices, but the disadvantages are that the wearer needs to directly or indirectly contact the device. In addition, due to the complexity of human behavior in daily activities, this type of algorithm has a certain false alarm rate.

[0007] (2) Fall detection based on environmental information sensors

[0008] This type of algorithm mainly uses pressure sensors to detect changes in ground pressure to determine whether a fall has occurred. Alwan M et al. distinguished between falls and daily activities by detecting the vibration pattern of the ground when a person falls. This type of algorithm has low sensor costs and does not require wearing, but it requires the installation of more sensors, and since it detects changes in ground pressure rather than directly detecting human falls, it is easily affected by environmental factors such as object falling, and the accuracy is generally less than 90%.

[0009] (3) Fall detection based on wireless signals

[0010] This kind of algorithm is mainly based on the wireless signal of 802.11n protocol and Doppler radar, through principal component analysis (PCA) recurrent neural network (RNN) and other methods to extract the possible human motion features in the channel state information (CSI) of wireless signal and classify, so as to determine whether the fall occurs. The CSI feature is a widely used feature in the field of wireless communication, and the CSI features of different human behaviors have differences in amplitude and phase. For example, Wang GH et al. proposed in 2016 to use a router as a sensor to detect falls while also detecting human sleep status. This kind of algorithm does not need to wear sensors, but the technology is still in its infancy.

[0011] (4) Fall detection based on visual processing

[0012] Fall detection based on visual processing mainly includes early RGB cameras, infrared cameras and recently developed depth camera processing technology. The first type is to classify according to the extracted multi-frame motion information, such as Rougier et al. and Foroughi et al. extract the human region through background modeling, then combine the motion information in the multi-frame image, and input it into the classifier for classification. Here, the motion information is a motion history image (MHI) image generated by superimposing multiple human region frames, and this kind of algorithm is greatly affected by the accuracy of human region segmentation; the second type is to classify according to the change of human contour in the image, such as Ma et al. extract the size ratio change feature of human contour to judge whether a fall occurs, and Mastorakis et al. calculate the speed of the human according to the position and aspect ratio of the human detected in the image to judge whether a fall occurs. This kind of algorithm also depends heavily on the accuracy of human detection; the third type is to judge whether a fall occurs according to the human posture, mainly to extract the human skeleton points, such as T. Xu, Y. Zhou et al. through ground detection, to determine the distance relationship between the skeleton points and the ground to confirm the fall. This kind of algorithm has high requirements for the detection accuracy of skeleton points; the fourth type is to judge whether a fall occurs according to the position of human head and the motion information of human head, such as Rougier et al. first detect the human head in the image sequence, and calculate the speed change of the human head in the horizontal and vertical directions to judge whether a fall occurs. This kind of algorithm is greatly affected by the camera angle; the fifth type is fall detection based on depth image, which uses depth image as input in depth image, manually labels the position of bed surface and ground, uses random forest to obtain human head position and upper body, uses LMNN-based metric learning combined with K-nearest neighbor classifier to classify human head position change, and confirms according to the distance between human upper body position and ground. This kind of method depends on human head detection, and some holes in the depth image will cause interference.

[0013] In summary, the current fall detection algorithm needs more samples for training, and the collection of a large number of fall detection samples in actual situations requires a lot of manpower, financial resources and material resources, and the collected samples are difficult to cover all fall detection scenarios; and these algorithms have problems such as contact-type devices, dependence on human or head detection, accuracy limited by light, etc. In addition, since real fall experiments cause great harm to the elderly, most of the existing methods are simulated on a data set, and the real fall sample data set is limited in size and has some differences from the simulated data, which is also a problem that needs to be considered in the use of some existing data-driven fall detection methods. SUMMARY

[0014] In view of the above defects or improvement needs of the prior art, the present application provides a human fall detection method and device based on point cloud, which realizes unsupervised fall detection in three-dimensional point cloud to solve the technical problem of low accuracy of existing fall detection algorithms.

[0015] To achieve the above-mentioned purpose, the present application provides a human fall detection method based on point cloud, comprising:

[0016] S1 three-dimensional conversion: converting the starting frame of the real-time collected human depth image sequence in the indoor scene into a three-dimensional space point cloud in the actual physical size, analyzing the point cloud to obtain the angle a between the Z-axis of the current coordinate system and the gravity direction of the actual scene;

[0017] S2 region detection: obtaining the human potential frame of the initial frame based on the texture features of the depth image, the optical flow map features and the neural network respectively, and merging the three obtained human potential frames to obtain the human motion region;

[0018] S3 region tracking: tracking the human motion region of the depth image sequence frame by frame, and converting the tracked human motion region into a three-dimensional human motion region point cloud in the physical size, denoising the human motion region point cloud, and correcting the Z-axis with a as the rotation angle to make the Z-axis parallel to the gravity direction;

[0019] S4 point cloud clustering: clustering the human motion region point cloud frame by frame to obtain different category point clouds of each frame;

[0020] S5 point cloud matching: matching the different category point clouds of each frame frame by frame to obtain continuous instance point clouds of different categories;

[0021] S6 point cloud analysis: performing size analysis on the matched instance point cloud to confirm the human point cloud, analyzing the center of gravity and pose change of the human point cloud, and determining whether a fall occurs.

[0022] Further, step S2 comprises:

[0023] S2.1 Obtaining the human latent bounding box through depth image texture features: The initial frame F... raw After the depth value is normalized to 0-255, it is converted into a grayscale-like image F. grey Based on this grayscale image, the potential bounding boxes for the human body are obtained, and after non-maximum suppression, set B is obtained. apprance ={b a1 ,b a2 ,…,b ai ,…,b an}, i∈{1,2,…,n},b ai ∈R 4 , where each box b ai It consists of four coordinates: the horizontal and vertical coordinates of the upper left corner and the horizontal and vertical coordinates of the lower right corner of the two-dimensional depth image;

[0024] S2.2 Obtaining latent human bounding boxes from optical flow map features obtained through optical flow estimation: After obtaining the initial frame optical flow map using an optical flow estimation network, the amplitude of the optical flow map is normalized to the range of 0 to 255. Based on this optical flow map, latent human bounding boxes are obtained and non-maximum suppression is performed to obtain set B. flow ={b f1 ,b f2 ,…,b fi ,…,b fn}, i∈{1,2,…,n},b fn ∈R 4 , where each box b fn It consists of four coordinates: the horizontal and vertical coordinates of the upper left corner and the horizontal and vertical coordinates of the lower right corner of the two-dimensional depth image;

[0025] S2.3 Obtaining the human latent bounding box through a neural network: in the grayscale image F grey The set B is obtained by acquiring the potential bounding boxes of the human body through a neural network and then performing nonmaximum suppression. human ={b h1 ,b h2 ,…,b hi ,…,b hn}, i∈{1,2,…,n},b hn ∈R4, where each box b hn It consists of four coordinates: the horizontal and vertical coordinates of the upper left corner and the horizontal and vertical coordinates of the lower right corner of the two-dimensional depth image;

[0026] S2.4 Merge the potential human bounding boxes obtained in steps S2.1 to S2.3 to obtain the final human motion region.

[0027] Further, step S1 includes:

[0028] S1.1, based on the correspondence between the two-dimensional depth image and the three-dimensional actual physical size of the point cloud, the starting frame of the depth image sequence is converted into a three-dimensional point cloud, and the coordinate correspondence adopted is:

[0029]

[0030] where x, y, z are the xyz coordinate values of the corresponding point cloud, Dx, Dy are the row and column coordinates of the depth image, Dz is the depth value of the depth image at <Dx, Dy>, fx, fy, cx, cy are the camera intrinsic parameters;

[0031] S1.2, using the coordinate relationship in the actual point cloud space, the normal vector of each point is calculated, the points with the same normal vector direction are classified into a plane, and all point cloud planes in the scene are obtained;

[0032] S1.3, removing the point cloud planes with a number of points less than a preset threshold, and determining the point cloud planes of the ground, wall and ceiling according to the value size of the center of gravity of the remaining point cloud planes in the Z axis and the angle of the normal vector;

[0033] S1.4, based on the point cloud and normal vector set of the ground, wall and ceiling, the camera pitch angle a is estimated by an iterative method;

[0034] S1.5, correcting the Z axis direction of the point cloud according to the camera pitch angle a, so that it is parallel to the gravity direction, determining the Z axis value of the ground according to the distribution of the Z axis direction coordinate value, and obtaining the ground plane equation according to the Z axis value of the ground.

[0035] Further, the iterative method for estimating the camera pitch angle a is realized by minimizing the energy function, and the specific form of the energy function is:

[0036]

[0037] <θcα,G>

[0038] Where Cg, Cw and C are the ground, wall and ceiling point cloud sets, Ng, Nw and Nc are the number of points of the ground, wall and ceiling planes, θgα, θwα and θcα are the angles between the normal vectors of the ground, wall and ceiling after the point cloud is rotated by a along the pitch angle direction and the gravity direction, and G is the gravity direction.

[0039] Further, step S4 comprises:

[0040] S4.1, unsupervised clustering of human motion area point cloud frame by frame to obtain different category point cloud sets of each frame;

[0041] Further, the unsupervised clustering of different category point clouds in step S4.1 specifically comprises:

[0042] S4.1.1 preset the distance metric value esp and the minimum number of neighborhood points min_sam;

[0043] S4.1.2 take the points in the point cloud whose number of neighborhood points within the distance metric value esp is greater than the minimum number of neighborhood points min_sam as initial core points, and obtain a core point set;

[0044] S4.1.3 traverse the core point set, and classify according to the following rules: classify the core points P1 within the radius esp of the core point P0 which has not been classified into a class together with P0, and at the same time, classify all unclassified points within the radius esp of P1 into the class, until there is no traversable core point within the radius esp of all core points in the class, and the clustering of the class is ended; other unclassified core points are set into new classes according to the above rules and clustered until all points are classified into corresponding classes.

[0045] S4.2 further clean the point cloud set obtained by clustering, and remove the point cloud set whose number is less than a preset threshold and the point cloud set whose center of gravity distance from the ground plane is greater than another preset threshold.

[0046] Further, step S6 includes:

[0047] S6.1 performing size analysis on the matched instance point cloud to confirm the human instance point cloud, and obtaining a human instance point cloud sequence;

[0048] S6.2 calculating the center of gravity descent speed of the human instance point cloud sequence based on the ground plane equation;

[0049] S6.3 calculating the proportion change of the number of points in the human instance point cloud sequence whose distance from the ground plane is within a preset threshold to the total number of points in the human instance point cloud sequence based on the ground plane equation, and obtaining an instance point cloud pose change feature;

[0050] S6.4 combining the center of gravity descent speed and the pose change feature, and determining that a fall occurs when the center of gravity descent speed is greater than a preset threshold and the pose change feature is greater than another preset threshold.

[0051] Further, the point cloud denoising processing of step S3 includes:

[0052] S3.1 searching K points of each point in the point cloud, calculating the distance between the point and the K points, and calculating the standard deviation of the distance, and removing the points whose distance exceeds a preset multiple of the standard deviation.

[0053] Further, step S5 includes:

[0054] S5.1 initializing an instance set, wherein the instance set includes m category point clouds of the starting frame;

[0055] S5.2, calculate the ratio of the intersection and union volume of each category point cloud in each frame and the three-dimensional bounding box of each instance point cloud in the instance set, if the value of 3DIOU is greater than the threshold value, the category point cloud in this frame is regarded as the same instance point cloud in time sequence; if the 3DIOU of the category point cloud in this frame and all instance point clouds in the instance set is less than the threshold value, the category point cloud in this frame is added to the instance set as a new category instance point cloud.

[0056] In another aspect, corresponding to the above-mentioned human body fall detection method based on point cloud, the application also provides a human body fall detection device based on point cloud, comprising a depth camera and a computing platform, wherein the depth camera is used to collect a depth image sequence of a human body in an indoor scene, the computing platform comprises a memory and a processor, the memory is used to store program instructions, and the processor is used to realize the above-mentioned human body fall detection method based on point cloud when executing the program instructions.

[0057] Overall, compared with the prior art, the above technical solutions conceived by the application can achieve the following beneficial effects:

[0058] 1. The application uses depth images collected by a low-cost depth camera as data input objects, and calculates human motion region features based on depth image sets, extracts human motion regions through image texture, optical flow characteristics and neural networks, only converts three-dimensional space point clouds in the human motion region, reduces the point cloud processing region, has stronger timeliness and accuracy, greatly reduces the calculation amount and processing cost, and the accuracy is not affected by the application light, has no biological invasion, protects the user privacy, has strong interpretability and high robustness;

[0059] 2. The application implements unsupervised fall detection technology in three-dimensional space, converts the fall detection problem into an unsupervised time sequence multi-instance point cloud matching and spatial domain physical motion analysis problem, does not need to be labeled information training, and greatly reduces the human, material and financial investment of data labeling. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is a flowchart of a human body fall detection method based on point cloud provided by an embodiment of the application;

[0061] Figure 2 is an indoor arrangement schematic diagram of a human body fall detection device based on point cloud provided by an embodiment of the application;

[0062] Figure 3 is a schematic diagram of two-dimensional depth image to three-dimensional point cloud provided by an embodiment of the application;

[0063] Figure 4 is a schematic diagram of human motion region obtained by combining depth image texture, optical flow and neural network provided by an embodiment of the application;

[0064] Figure 5 is a human body point cloud center of gravity change situation and human body pose change situation diagram provided by the embodiment of the application when the human body falls in an indoor scene; DETAILED DESCRIPTION

[0065] In order to make the purpose, technical scheme and advantages of the application more clear and understandable, the application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application. In addition, the technical features involved in each embodiment of the application described below can be combined with each other as long as they do not conflict with each other.

[0066] Figure 1 is a flowchart of a human body fall detection method based on point cloud provided by the embodiment of the application, and the method comprises the following steps:

[0067] S1 three-dimensional conversion: converting the starting frame of the real-time collected human body depth image sequence in an indoor scene into a three-dimensional space point cloud in actual physical size, analyzing the point cloud to obtain the angle a between the Z axis of the current coordinate system and the gravity direction of the actual scene;

[0068] S2 region detection: obtaining the human body potential frame of the initial frame based on the texture features of the depth image, the optical flow map features and the neural network respectively, and merging the three obtained human body potential frames to obtain the human body motion region;

[0069] S3 region tracking: tracking the human body motion region of the depth image sequence frame by frame, and converting the tracked human body motion region into a three-dimensional human body motion region point cloud in physical size, denoising the human body motion region point cloud, and correcting the Z axis with a as the rotation angle to make the Z axis parallel to the gravity direction;

[0070] S4 point cloud clustering: clustering the human body motion region point cloud frame by frame to obtain different category point clouds of each frame;

[0071] S5 point cloud matching: matching the different category point clouds of each frame frame by frame to obtain continuous instance point clouds of different categories;

[0072] S6 point cloud analysis: performing size analysis on the matched instance point cloud to confirm the human body point cloud, analyzing the center of gravity and pose change of the human body point cloud, and thus judging whether a fall occurs.

[0073] Specifically, as shown in the drawings, Figure 2 the sequence depth image collected by the application follows the following principles in order to express the three-dimensional human body point cloud coordinates as completely as possible, so that the human body information is as sufficient as possible and the plane equation fitting error is as small as possible:

[0074] a The optimal measurement range of the depth camera is about 1m-4m, so the distance between the human body and the depth camera during sample collection is about 2m to 4m;

[0075] b The depth camera should be placed at a downward angle of view with an angle of about 30° to 45° with the horizontal plane, and the height from the ground is about 2.5m;

[0076] c The target moves normally in a natural state, and the human body region in the area where it may fall is as much as possible to collect sufficient points;

[0077] As shown in Figure 3 , step S1 includes:

[0078] S1.1 Based on the coordinate correspondence relationship between the two-dimensional depth image and the point cloud in the three-dimensional actual physical size, the starting frame of the depth image sequence is converted into a three-dimensional point cloud, and the coordinate correspondence relationship used is:

[0079]

[0080] Where x, y, z are the xyz coordinate values of the corresponding point cloud points, Dx, Dy are the row and column coordinates of the depth image, Dz is the depth value of the depth image at <Dx, Dy>, fx, fy, cx, cy are the camera intrinsic parameters;

[0081] S1.2 Using the coordinate relationship in the actual point cloud space, the normal vector of each point is calculated, the points with the same normal vector direction are classified into a plane, and all point cloud planes in the scene are obtained;

[0082] S1.3 Remove the point cloud planes with a point number less than a predetermined threshold, and determine the point cloud planes of the ground, wall and ceiling according to the value size of the center of gravity of the remaining point cloud planes in the Z axis and the angle of the normal vector;

[0083] S1.4 Since the camera itself has a certain pitch angle when acquiring images, it is necessary to estimate the pitch angle of the point cloud and rotate the point cloud so that the Z axis direction is parallel to the gravity direction. Based on the point cloud and normal vector set of the ground, wall and ceiling, the camera pitch angle a is estimated by iteration.

[0084] Wherein, the iterative estimation of the camera pitch angle a is realized by minimizing the energy function. The specific form of the energy function is:

[0085]

[0086] <θcα,G>

[0087] Where Cg, Cw and C are the ground, wall and ceiling point cloud sets, Ng, Nw and Nc are the number of points on the ground, wall and ceiling planes, and θgα, θwα and θcα are the angles between the normal vectors of the ground, wall and ceiling after the point cloud is rotated by α along the pitch angle direction and the gravity direction, which will change with α, and G is the gravity direction. Optimizing the objective function is to find the optimal α value to minimize the energy value. After correction, the Z-axis direction should be parallel to the gravity direction.

[0088] S1.5 Correct the point cloud Z-axis direction according to the camera pitch angle α to make it parallel to the gravity direction. After correcting the point cloud Z-axis direction, calculate the distribution of the Z-axis direction coordinate values. Since the ground points will have a peak value in the Z-axis distribution, the Z-axis value of the ground plane can be obtained by threshold segmentation methods such as entropy segmentation, and the ground plane equation is finally determined.

[0089] As shown in Figure 4 , step S2 includes:

[0090] S2.1 Obtain the human potential frame through the depth image texture feature: normalize the depth value to 0-255, and convert it to a gray-like image F raw . On the basis of this gray image, use potential target extraction technology (such as edgebox, bing++ etc.) to obtain the potential frame and perform non-maximum suppression to obtain the set B grey = {b apprance , b a1 , …, b a2 , …, b ai , …, b an}, i ∈ {1, 2, …, n}, b ai ∈ R 4 , where each frame b an is composed of four coordinates: the upper left corner horizontal and vertical coordinates and the lower right corner horizontal and vertical coordinates on the two-dimensional depth image.

[0091] S2.2 Obtain the human potential frame through the optical flow estimation technology: obtain the optical flow map of the initial frame using the optical flow estimation network, normalize the amplitude of the optical flow map to the range of 0 to 255, and on the basis of this optical flow map, use potential target extraction technology (such as edgebox, bing++ etc.) to obtain the potential frame and perform non-maximum suppression to obtain the set B flow = {b f1 , b f2 , …, b fi , …, b fn}, i ∈ {1, 2, …, n}, b fn ∈ R 4 , where each frame b fnThe four coordinates of the upper left corner and the lower right corner of the two-dimensional depth map;

[0092] S2.3 Obtain the human potential frame on the current two-dimensional image through the neural network and perform non-maximum suppression to obtain a set B grey human h1 h2 hi hn} i∈{1,2,…,n}, b hn ∈R4, wherein each frame b hn The four coordinates of the upper left corner and the lower right corner of the two-dimensional depth map;

[0093] S2.4 Merge the potential frames obtained in steps S2.1 to S2.3 to obtain the final human motion region frame.

[0094] Further, the point cloud denoising processing in step S3 includes:

[0095] S3.1 Search K points in the neighborhood of each point in the point cloud, calculate the distance between the point and its K neighborhood points, and calculate the standard deviation of the distance, and remove the points whose distance exceeds the preset multiple standard deviation.

[0096] Further, step S4 includes:

[0097] S4.1 Use a density-based spatial clustering algorithm to unsupervisedly cluster the human motion region point cloud frame by frame to obtain different category point cloud sets of each frame;

[0098] The unsupervised clustering of different category point clouds includes:

[0099] S4.1.1 Pre-set distance measurement value esp and neighborhood point number minimum value min_sam;

[0100] S4.1.2 Take the points in the point cloud whose neighborhood point number is greater than the neighborhood point number minimum value min_sam within the distance measurement value esp as initial core points to obtain a core point set;

[0101] S4.1.3 Traverse the core point set and classify according to the following rules: classify the core points P1 within the radius esp of the core point P0 that have not been classified into the same category as P0, and at the same time, classify all unclassified points within the radius esp of P1 into this category, until there are no traversable core points within the radius esp of all core points in this category, and the clustering of this category ends; other unclassified core points are classified into new categories according to the above rules and clustered until all points are classified into corresponding categories.

[0102] ​​​​​S4.2 Further clean the obtained point cloud set, remove the point cloud set whose number is less than a preset threshold and the point cloud set whose center of gravity distance from the ground plane is greater than another preset threshold, the effect is to remove non-human point cloud.

[0103] Further, step S5 includes:

[0104] S5.1 Initialize an instance set, the instance set including m category point clouds of the starting frame;

[0105] S5.2 Calculate the ratio of the intersection and union volume of each category point cloud in each frame and the three-dimensional bounding box of each instance point cloud in the instance set 3DIOU frame by frame, if the value of 3DIOU is greater than a threshold, the category point cloud in this frame is regarded as the same instance point cloud in time sequence; if the 3DIOU of the category point cloud in this frame and all instance point clouds in the instance set is less than the threshold, the category point cloud in this frame is added to the instance set as a new category instance point cloud; repeat the above process until the traversal of all instance point clouds is completed.

[0106] As shown in Figure 5 , step S6 includes:

[0107] S6.1 Size analysis is performed on the matched instance point cloud to confirm the human instance point cloud, and a human instance point cloud sequence is obtained;

[0108] S6.2 The center of gravity descent speed of the human instance point cloud sequence is calculated based on the ground plane equation;

[0109] S6.3 The proportion change of the number of points in the human instance point cloud sequence within a preset threshold distance from the ground plane to the total number of points in the human instance point cloud sequence is calculated based on the ground plane equation, and an instance point cloud pose change feature is obtained;

[0110] S6.4 The center of gravity descent speed and the pose change feature are combined, when the center of gravity descent speed is greater than a preset threshold and the pose change feature is greater than another preset threshold, it is determined that a fall occurs.

[0111] Figure 5 A schematic diagram of a continuous motion superimposed on a segment of continuous frame human point cloud, the human center of gravity change speed is represented by the center of gravity descent speed of each frame point cloud, and the human pose is represented by the proportion of the number of point clouds of the human point cloud within 40 cm height from the ground to the number of points of the frame human point cloud. It can be seen that from the starting frame, the human body is in a standing position, the proportion is small, and after falling, the human body is in a lying posture, the proportion is large, which better reflects the change of human body pose.

[0112] On the other hand, corresponding to the above-mentioned point cloud-based human fall detection method, please refer to Figure 2As shown, the application also provides a human body fall detection device based on point cloud, comprising a depth camera and a computing platform, wherein the depth camera is used to collect a depth image sequence of a human body in an indoor scene, the computing platform comprises a memory and a processor, the memory is used to store program instructions, and the processor is used to implement the aforementioned human body fall detection method based on point cloud when executing the program instructions.

[0113] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A human fall detection method based on point cloud, characterized in that, The method comprises the steps of: S1 three-dimensional conversion: converting the starting frame of the real-time collected human body depth image sequence in an indoor scene into a three-dimensional space point cloud in actual physical size, analyzing the point cloud to obtain the angle α between the Z axis of the current coordinate system and the direction of the actual scene gravity; S2 region detection: obtaining the human body potential frame of the initial frame based on the texture features of the depth image, the optical flow map features and the neural network respectively, and merging the three obtained human body potential frames to obtain the human body motion region; S3 region tracking: tracking the human body motion region of the depth image sequence frame by frame, and converting the tracked human body motion region into a three-dimensional human body motion region point cloud in physical size, denoising the human body motion region point cloud, and correcting the Z axis with α as the rotation angle to make the Z axis parallel to the gravity direction; S4 point cloud clustering: clustering the human body motion region point cloud frame by frame to obtain different category point clouds of each frame; S5 point cloud matching: matching different category point clouds of each frame frame by frame to obtain continuous instance point clouds of different categories; S6 point cloud analysis: performing size analysis on the matched instance point cloud to confirm the human body point cloud, analyzing the center of gravity and pose change of the human body point cloud, and determining whether a fall occurs.

2. The human fall detection method based on point cloud according to claim 1, wherein, The step S2 comprises: S2.1 Obtain human potential frame by depth image texture feature: obtain initial frame F raw After the depth value is normalized to 0-255, it is converted into a gray-like image F grey After obtaining the human potential frame and performing non-maximum suppression on the basis of the gray image, a set B is obtained apprance ={b a1 ,b a2 ,…,b ai ,…,b an}, i∈{1,2,…,n}, b ai ∈R 4 , wherein each frame b ai is composed of four coordinates of the upper left corner and the lower right corner of the two-dimensional depth image. S2.2 Obtain the human potential frame by the optical flow estimation technology: after obtaining the optical flow map of the initial frame by the optical flow estimation network, normalize the amplitude of the optical flow map to the interval of 0 to 255, obtain the human potential frame on the basis of the optical flow map, and after non-maximum suppression, obtain the set B flow = {b f1 ,b f2 ,…,b fi ,…,b fn}, i ∈ {1, 2, …, n}, b fn ∈ R 4 , wherein each frame b fn is composed of four coordinates of the upper left corner and the lower right corner of the two-dimensional depth image. S2.3 Obtain human potential frame through neural network: in the gray scale F grey After obtaining the human potential frame through the neural network and performing non-maximum suppression, a set B is obtained human = {b h1 ,b h2 ,…,b hi ,…,b hn}, i ∈ {1, 2, …, n}, b hn ∈ R4, wherein each frame b hn is composed of four coordinates of the upper left corner and the lower right corner of the two-dimensional depth image. S2.4 merging the human body potential frames obtained in steps S2.1 to S2.3 to obtain the final human body motion region.

3. The human fall detection method based on point cloud according to claim 1, wherein, The step S1 comprises: S1.1 converting the starting frame of the depth image sequence into a three-dimensional point cloud based on the coordinate correspondence relationship between the two-dimensional depth image and the three-dimensional point cloud in the actual physical size, and the coordinate correspondence relationship adopted is: Wherein x, y, z are the xyz coordinate values of the points of the point cloud, Dx, Dy are the row and column coordinates of the depth image, Dz is the depth value of the depth image at <Dx, Dy>, and fx, fy, cx, cy are the camera parameters; S1.2 calculating the normal vector of each point using the coordinate relationship in the actual point cloud space, grouping the points with the same normal vector direction into a plane, and obtaining all point cloud planes in the scene; S1.3 removing the point cloud planes with a point number less than a preset threshold, and determining the point cloud planes of the ground, wall and ceiling according to the center of gravity of the remaining point cloud planes on the Z axis and the value size and normal vector angle; S1.4 estimating the camera pitch angle α by an iterative method based on the point cloud and normal vector set of the ground, wall and ceiling; S1.5 correcting the Z axis direction of the point cloud according to the camera pitch angle α so that it is parallel to the gravity direction, determining the Z axis value of the ground according to the distribution of the Z axis direction coordinate value, and obtaining the ground plane equation according to the Z axis value of the ground.

4. The human fall detection method based on point cloud according to claim 3, wherein, The iterative method for estimating the camera pitch angle α is realized by minimizing the energy function, and the specific form of the energy function is: Wherein, Cg, Cw and C are the ground, wall and ceiling point cloud sets respectively, Ng, Nw and Nc are the plane point numbers of the ground, wall and ceiling respectively, θgα, θwα and θcα are the angles between the normal vectors of the ground, wall and ceiling after the point cloud is rotated by α along the pitch angle direction and the gravity direction respectively, and G is the gravity direction.

5. The human fall detection method based on point cloud according to claim 3, wherein, The step S4 comprises: S4.1 unsupervised clustering the human body motion region point cloud frame by frame to obtain different category point cloud sets of each frame; S4.2 Further clean the clustered point cloud set, remove the point cloud set whose number is less than a preset threshold and the point cloud set whose center of gravity distance from the ground plane is greater than another preset threshold.

6. The human fall detection method based on point cloud according to claim 5, wherein, The step S4.1 of unsupervised clustering of different categories of point clouds specifically includes the following steps: S4.1.1 preset distance metric value esp and neighborhood point number minimum value min_sam; S4.1.2 take the points in the point cloud whose neighborhood point number within the distance metric value esp is greater than the neighborhood point number minimum value min_sam as initial core points to obtain a core point set; S4.1.3 traverse the core point set and classify according to the following rules: classify the core points P1 within the radius esp of the core point P0 which has not been classified into the same category as P0, and at the same time, classify all unclassified points within the radius esp of P1 into this category, until there is no traversable core point within the radius esp of all core points in this category, the clustering of this category is ended; other unclassified core points are classified into new categories according to the above rules and clustered until all points are classified into corresponding categories.

7. The human fall detection method based on point cloud according to claim 3, wherein, The step S6 includes: S6.1 perform size analysis on the matched instance point cloud to confirm the human instance point cloud and obtain a human instance point cloud sequence; S6.2 calculate the center of gravity descent speed of the human instance point cloud sequence based on the ground plane equation; S6.3 calculate the proportion change of the number of points in the human instance point cloud sequence within a preset threshold distance from the ground plane to the total number of points in the human instance point cloud sequence based on the ground plane equation to obtain an instance point cloud pose change feature; S6.4 combine the center of gravity descent speed and the pose change feature, and when the center of gravity descent speed is greater than a preset threshold and the pose change feature is greater than another preset threshold, determine that a fall occurs.

8. The human fall detection method based on point cloud according to claim 1, wherein, The denoising processing in the step S3 includes: S3.1 search K points in the neighborhood of each point in the point cloud, calculate the distance between the point and its K neighborhood points, and calculate the standard deviation of the distance, and remove the points whose distance exceeds the preset multiple standard deviation.

9. The human fall detection method based on point cloud according to any one of claims 1-8, wherein, The step S5 includes: S5.1 initialize an instance set, the instance set including m category point clouds of the starting frame; S5.2 calculate the 3DIOU of the intersection and union volume of the three-dimensional bounding box of each category point cloud in each frame and each instance point cloud in the instance set, if the 3DIOU value is greater than a threshold, the category point cloud in this frame is regarded as the same instance point cloud in time sequence; if the 3DIOU of the category point cloud in this frame and all instance point clouds in the instance set is less than the threshold, the category point cloud in this frame is added to the instance set as a new category instance point cloud. 10.A human fall detection device based on point cloud, characterized in that, The application includes a depth camera and a computing platform, wherein the depth camera is used to collect a depth image sequence of a human body in an indoor scene in real time, the computing platform includes a memory and a processor, the memory is used to store program instructions, and the processor is used to execute the program instructions to realize the point cloud-based human body fall detection method in claims 1-9.

Images