A fall detection method based on graph convolutional network of human key points

Through the graph convolution network method based on key points of the human body, human posture characteristics are extracted and classified, and fall judgments are made in combination with timing information, the problem of insufficient mining of human skeleton and posture information in the existing technology is solved, and high accuracy and low Complexity fall detection.

CN114387666BActive Publication Date: 2024-11-12DALIAN UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111620249.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-28
Publication Date
2024-11-12
Estimated Expiration
2041-12-28

AI Technical Summary

Technical Problem

The existing fall detection technology, especially vision-based methods, lacks sufficient exploration of human skeleton and posture information, resulting in poor detection results and the inability to effectively identify fall situations in the elderly.

Method used

The graph convolution network method based on human body key points is adopted, and the human body key points are extracted through the pose estimation calculation method, the pose key points data set is constructed, and the double-flow graph convolution network is designed for pose feature extraction and classification, and the fall judgment is made based on timing information.

Benefits of technology

It improves the robustness of posture characteristics and the accuracy of fall detection, enhances the generalization ability of detection algorithms, and has the characteristics of low computing complexity, high recognition accuracy and easy deployment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114387666B_ABST
    Figure CN114387666B_ABST
Patent Text Reader

Abstract

The present invention provides a graph convolution network fall detection method based on human key points, which belongs to the field of computer vision technology. The present invention designs a dual-stream graph convolution network to model human posture in the form of a graph, fully mines and utilizes the posture information of the human body, improves the robustness of posture features, and is beneficial to subsequent posture classification; at the same time, it utilizes time series information, combines the detection results of the previous frame and the current frame to make a joint judgment of the fall, and enhances the generalization ability of the fall detection algorithm; the fall detection method described in the present invention has the characteristics of low computational complexity, high recognition accuracy, easy deployment and migration.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and relates to a fall detection method based on a graph convolutional network of human key points. Background Art

[0002] With the progress of society and the development of science and technology, people's living environment and social medical security have been greatly improved, which has generally extended human life expectancy. One problem brought about by the extension of human life expectancy is population aging. With the increase in the elderly population, the issue of health care for the elderly has also become prominent, attracting widespread attention from the society.

[0003] Statistics from the World Health Organization show that falls are one of the main causes of accidental or unintentional injury deaths around the world. Among the elderly, 23% to 40% of deaths due to accidental injuries are caused by falls. If the fall of a person can be detected in time and an early warning is issued, the injured person can be treated in time and lives can be saved. Therefore, the research on real-time fall detection methods has important practical significance and market value.

[0004] At present, there are some corresponding fall detection technologies, which are mainly divided into three methods: wearable device-based, environment-based, and vision-based. Wearable devices use sensors such as gyroscopes and accelerometers, which need to invade the human body and cannot guarantee all-weather detection. The environment-based fall detection method requires the deployment of multiple sets of sensors in the places where people move, and the data changes in the environment are obtained by the equipment for fall detection. Its migration ability is poor and the cost is high. The vision-based detection solution combines computer vision, image processing and other technologies for detection. It uses cameras for image acquisition and can be easily embedded in the monitoring system. It has low cost, strong migration ability, and can guarantee all-weather detection. Therefore, the vision-based solution has a wider economic significance. However, the current vision-based fall detection solution mainly considers information such as changes in the contour of the human body and changes in the movement of the human body, and lacks sufficient mining of the human skeleton and posture information. Summary of the invention

[0005] In order to solve the problem of fall detection, the present invention fully exploits the human skeleton posture information and proposes a fall detection method based on human key point graph convolutional network. The overall flow chart is as follows: Figure 1 shown.

[0006] The technical solution of the present invention is:

[0007] A fall detection method based on a graph convolutional network of human key points comprises the following steps:

[0008] S1. Human body key point extraction

[0009] Based on the posture estimation algorithm, key points of the human body in the surveillance image are extracted to construct a posture key point dataset.

[0010] S2. Posture feature extraction and classification

[0011] A dual-stream graph convolutional network is designed to extract posture features of human key points. The dual-stream graph convolutional network consists of a graph convolutional network and a fully connected network; the graph convolutional network extracts geometric structure information of human key points, and the fully connected network extracts more global information from the key points. The posture features extracted by the graph convolutional network and the fully connected network are spliced ​​and fused, and sent to the classifier to complete posture classification, and the posture state of the current pedestrian is obtained.

[0012] S3. Fall judgment

[0013] Set up a posture classification result queue, set the risk factor based on the timing information and classification results, and determine whether the current pedestrian has fallen.

[0014] Furthermore, in step S1, a posture estimation algorithm is used to extract key point information of the human body, and data collection is performed on various postures to obtain key point information corresponding to different posture images, and construct the key point data set into a posture key point data set; the specific process is as follows:

[0015] S11, classifying postures into four types of postures: standing, bending, sitting and lying, and collecting image data of the four types of postures;

[0016] S12. Based on the posture estimation algorithm, the key point information corresponding to each type of posture image is obtained to construct a posture key point data set.

[0017] Furthermore, in step S12, the posture estimation algorithm obtains the coordinate information of 17 key points of the human body. Taking the nose key point as an example, its coordinate information is p nose =[x 1 ,y 1 ], cascade the coordinate information of all key points to obtain the posture vector P = [p nose ,p neck ,…,p lef t ankle =[x 1 ,y 1 ,…,x 17 ,y 17 ] to construct a pose key point dataset.

[0018] Furthermore, in step S2: the posture vector P extracted in step S1 is converted into [p nose ,p neck ,…,p left_ankle] Input the dual-stream graph convolutional network, extract the geometric structure features and global features of the posture from different dimensions, and concatenate and fuse the obtained geometric structure features and global features, and input them into the classifier for the final posture classification; the specific process is as follows:

[0019] S21. When sending the posture vector into the graph convolutional network, the posture vector needs to be constructed into a skeleton graph first, and then the unstructured skeleton graph is represented into a structured data form. The specific process is as follows:

[0020] First, the posture vectors P are connected to form an undirected graph, and the connection is as follows: nose-left eye, nose-right eye, left eye-right eye, left eye-left ear, right eye-right ear, left ear-left shoulder, right eye-right shoulder, left shoulder-left elbow, right shoulder-right elbow, left shoulder-right elbow, left elbow-left wrist, right elbow-right wrist, left shoulder-left hip, right shoulder-right hip, left hip-left knee, right hip-right knee, left knee-left ankle, right knee-right ankle; connect in this way to obtain the human skeleton graph G, which has 17 nodes and 19 edges.

[0021] Secondly, the skeleton graph G is structured and represented. For the i-th node in the skeleton graph G, its node feature is represented by its coordinate vector p i =[x i ,y i ] indicates that for the undetected default nodes, their node features are filled with 0 vectors, and the feature vectors p of the 17 key points are 1 ,…,p 17 Arrange them in rows and get the feature matrix H about the skeleton graph G. The connection relationship of the nodes is represented by the adjacency matrix A. The i-th row and j-th column of the adjacency matrix A stores the connection information between the i-th node and the j-th node of the skeleton graph, which is expressed as

[0022] At this point, the feature matrix H and the adjacency matrix A have performed structured data representation on the human skeleton graph G.

[0023] S22. Build a graph convolution network to extract features from the skeleton graph. When updating the features of each node in the graph convolution, the connected nodes will be aggregated and updated based on the connection information of the nodes, and then the updated features will be input into the nonlinear activation function to enhance the nonlinear ability of the network. At the same time, after multiple layers of stacking, the deep network has a deeper receptive field. Each node can perceive more feature information of its neighboring nodes and pass this information to the next layer. By stacking multiple layers of graph convolution layers, the geometric structure information of human body posture can be fully mined. Define the l-th layer graph convolution function as follows:

[0024]

[0025] Where W(l) is the weight matrix of the convolutional network layer l, H (l) is the input feature matrix of the lth layer, A is the adjacency matrix, ReLU is the nonlinear activation function, and D is the diagonal matrix of the skeleton graph node degree.

[0026] After the feature matrix H of the skeleton graph G is updated by multi-layer graph convolution, the output feature matrix H final , H final All row vectors are averaged and pooled to obtain the posture feature vector

[0027] S23, based on the extracted key point P of the human nose nose =[x 1 ,y 1 ] and ankle key point P ankle =[x 17 ,y 17 ], estimate the approximate height of the human body h = max(|x 1 -x 17 |,|y 1 -y 17 |), normalize the posture vector extracted in step S1 to height Get the normalized posture vector P norm .

[0028] S24, stack two layers of fully connected layers to build a fully connected network, and normalize the posture vector P norm Perform feature extraction to obtain the posture feature vector

[0029] S25, the posture feature vector extracted by the fully connected network in step S24 and the posture feature vector extracted by the graph convolutional network in step S22 Perform cascade splicing to obtain the fused feature vector F. It indicates that feature concatenation is performed by channel; finally, the feature F is input into a classifier consisting of two fully connected layers, and finally normalized by the softmax function to output the predicted posture category.

[0030] Furthermore, in step S3, a fall judgment is made based on the previous detection result, and the judgment process is as follows: a result queue with a length of T frames is set to store the posture classification results of T frames, and the current queue is cyclically updated according to each frame of the video stream. The threshold coefficient α is set, and the sum of the number of frames of the "lying" posture in the result queue with a length of T frames is T lie , when T lie When >αT, a fall is judged.

[0031] Beneficial effects of the present invention:

[0032] 1) A dual-stream graph convolutional network is designed to model human posture in the form of a graph, fully mining and utilizing the human posture information, improving the robustness of posture features, and facilitating subsequent posture classification.

[0033] 2) The timing information is utilized to combine the detection results of the previous frame and the current frame to make a joint judgment on the fall, thereby enhancing the generalization ability of the fall detection algorithm.

[0034] 3) The fall detection method has the characteristics of low computational complexity, high recognition accuracy, and easy deployment and migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flowchart of the present invention.

[0036] Figure 2 Schematic diagram of the human skeleton connection method.

[0037] Figure 3 This is the framework diagram of the dual-stream graph convolutional network. DETAILED DESCRIPTION

[0038] The specific implementation of the present invention is further explained below in conjunction with the examples and drawings, but is not intended to limit the present invention.

[0039] Figure 1 The overall flow chart of the graph convolutional network fall detection method based on human key points includes the following steps:

[0040] S1. Human body key point extraction

[0041] Based on the posture estimation algorithm, key points of the human body in the surveillance image are extracted to construct a posture key point dataset.

[0042] S2. Posture feature extraction and classification

[0043] A dual-stream graph convolutional network is designed to extract posture features of human key points. The dual-stream graph convolutional network consists of a graph convolutional network and a fully connected network; wherein the graph convolutional network extracts features of the extracted human key points to obtain graph structure features, which are spliced ​​with the posture features extracted by the fully connected network and sent to the classifier for posture classification to obtain the current posture state of the pedestrian.

[0044] S3. Fall judgment

[0045] Set up a posture classification result queue, set the risk factor based on the timing information and classification results, and determine whether the current pedestrian has fallen.

[0046] Among them, S1 extracts key points of pedestrians in the image based on the posture estimation algorithm, as follows: First, the daily activity postures of the human body are divided into four categories, namely standing, bending, sitting and lying. Then, the Raspberry Pi camera is used to collect data of the four types of postures in the experimental simulation environment and the real indoor environment to obtain image data of the four types of postures; at the same time, for the image data of each type of posture, the posture estimation algorithm is used to extract key points of pedestrians in the image to obtain a data set of key points of human postures. The key points of the human body are the left ear, right ear, left eye, right eye, nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and right ankle, a total of 17 key points. The posture estimation algorithm extracts the coordinate information of each key point in the image. Specifically, taking the nose key point as an example, its coordinate information is p nose =[x 1 ,y 1 ]; Fill the undetected key points with 0, and then concatenate the coordinate information of all key points into a posture vector P = [p nose ,p neck ,…,p left_ankle ]=[x 1 ,y 1 ,...,x 17 ,y 17 ].

[0047] Among them, S2 performs posture classification on the human body posture information extracted by S1. Figure 3 The graph convolution posture classification network designed by the present invention is demonstrated. The network is a dual-stream graph convolution network, which includes a graph convolution network and a fully connected network. It fully utilizes the structural representation ability of graph convolution and the global feature extraction ability of full connection to model the geometric spatial relationship of the key points of the human body.

[0048] For graph convolutional networks, the input is graph structure data. First, the posture vector P of the human body needs to be constructed into an undirected graph data structure. The details are as follows: Figure 2 The human skeleton graph G is connected in a way to form a human skeleton graph G, which has 17 nodes and 19 edges. For the i-th node in the skeleton graph G, its node feature is represented by its coordinate vector p i =[x i ,y i ] represents; for the undetected default nodes, their node features are filled with 0 vectors, and the features of the edges in the skeleton graph are left blank, resulting in a feature matrix H with a dimension of 17×2 for the skeleton graph G. The connection relationship of the nodes is represented by the adjacency matrix A At this point, the feature matrix H and the adjacency matrix A complete the structured data representation of the human skeleton graph.

[0049] Graph convolution has the characteristics of information transmission and aggregation. The update of the current node feature in the graph convolution network comes from the feature aggregation of other nodes connected to the node. For each node, the average pooling of the features of all adjacent nodes is calculated and then multiplied by the weight matrix W. (l) Update the feature of the node, and then input the updated feature into the nonlinear activation function to enhance the nonlinear ability of the network; define the l-th layer graph convolution function as

[0050]

[0051] Where W (l) is the weight matrix of the convolutional network layer l, H (l) is the input feature matrix of the lth layer, ReLU is the nonlinear activation function, and D is the diagonal matrix of node degrees.

[0052] When doing matrix multiplication, the adjacency matrix A will change the scale range of the eigenvector, so it needs to be normalized. Complete the normalization of the adjacency matrix A. In the initial state, the feature dimension of the nodes in the graph is small, with only two dimensions: x and y coordinates. Therefore, when stacking graph convolution layers, it is necessary to map them to a higher-dimensional space to learn more robust features. Specifically, 4 layers of graph convolution layers are stacked, and the feature dimensions of the nodes in each layer are 64, 128, 256, and 256. After passing through 4 layers of graph convolution networks, the dimension of the feature matrix H is 17×256. At the same time, after multiple layers of stacking, the deep network has a deeper receptive field, and each node can perceive more feature information of its neighboring nodes and pass this information to the next layer.

[0053] In order to obtain the feature expression of the entire graph structure, the global average pooling is performed on each node feature in the graph to obtain the posture feature vector Its dimension is 1×256.

[0054] For the fully connected network, the input is the posture vector P. Due to the size difference of the human body and the distance between the human body and the camera, the scale of the key point information is inconsistent, so it needs to be normalized. The normalization method is as follows: Based on the extracted key point P of the human nose nose =[x 1 ,y 1 ] and ankle key point P ankle =[x 17 ,y 17 ], estimate the approximate height of the human body h = max(|x 1 -x 17 |,|y 1 -y 17|), normalize the posture vector extracted in step S1 to height Then the normalized posture vector P norm After two fully connected layers and nonlinear activation functions, the new posture feature vector is mapped to the 256-dimensional space. Then Pose feature vector extracted with graph convolutional network Perform cascade splicing to obtain the posture feature expression F of the pedestrian. It means that the features are concatenated by channel; finally, the feature F is input into a classifier consisting of two fully connected layers, where the two fully connected layers have 256 nodes and 4 nodes respectively, and then normalized by softmax to output the predicted posture category.

[0055] Among them, in S3, the above steps obtain the posture detection results of a single frame. In order to determine whether a fall has occurred, it is necessary to use the previous detection results combined with the current frame for joint judgment. The judgment process is as follows: First, set a result queue with a length of T frames to store the posture classification results of T frames, and cyclically update the current queue according to each frame of the video stream. Set the threshold coefficient α, and the number of frames of the "lying" posture in the result queue with a length of T frames is T lie , when T lie When >αT, a fall is judged.

Claims

1. A fall detection method based on graph convolutional network of human key points, characterized in that: The method comprises the following steps: S1. Human body key point extraction Based on the posture estimation algorithm, the key points of the human body in the monitoring image are extracted to build a posture key point dataset; S2. Posture feature extraction and classification A dual-stream graph convolution network is designed to extract posture features of key points of the human body; the dual-stream graph convolution network is composed of a graph convolution network and a fully connected network; wherein the graph convolution network extracts geometric structure information of key points of the human body, and the fully connected network extracts global information of the key points; the posture features extracted by the graph convolution network and the fully connected network are spliced ​​and fused, and sent to the classifier to complete posture classification, so as to obtain the posture state of the current pedestrian; the specific process of step S2 is as follows: S21. When the posture vector is sent to the graph convolutional network, the posture vector is first constructed into a skeleton graph, and then the unstructured skeleton graph is represented into a structured data form; the specific process is as follows: First, the posture vectors P are connected to form an undirected graph, and the connection is as follows: nose-left eye, nose-right eye, left eye-right eye, left eye-left ear, right eye-right ear, left ear-left shoulder, right eye-right shoulder, left shoulder-left elbow, right shoulder-right elbow, left shoulder-right elbow, left elbow-left wrist, right elbow-right wrist, left shoulder-left hip, right shoulder-right hip, left hip-left knee, right hip-right knee, left knee-left ankle, right knee-right ankle; connect in this way to get the human skeleton graph G, which has 17 nodes and 19 edges; Secondly, the skeleton graph G is structured to represent data: for the i-th node in the skeleton graph G, its node feature is represented by its coordinate vector p i =[x i ,y i ] indicates that for the undetected default nodes, their node features are filled with 0 vectors, and the feature vectors p1,…,p of the 17 key points are 17 Arrange them in rows and get the feature matrix H of the skeleton graph G. The connection relationship of the nodes is represented by the adjacency matrix A. The i-th row and j-th column of the adjacency matrix A stores the connection information between the i-th node and the j-th node of the skeleton graph, which is expressed as So far, the feature matrix H and the adjacency matrix A have performed structured data representation on the human skeleton graph G; S22. Build a graph convolution network to extract features from the skeleton graph. When updating the features of each node in the graph convolution, the connected nodes will be aggregated and updated based on the connection information of the nodes, and then the updated features will be input into the nonlinear activation function to enhance the nonlinear ability of the network. At the same time, after multiple layers of stacking, the deep network has a deeper receptive field. Each node can perceive more feature information of its neighboring nodes and pass this information to the next layer. By stacking multiple layers of graph convolution layers, the geometric structure information of the human body posture can be fully mined. The definition of the l-th layer graph convolution function is as follows: Among them, W (l) is the weight matrix of the convolutional network layer l, H (l) is the input feature matrix of the lth layer, A is the adjacency matrix, ReLU is the nonlinear activation function, and D is the diagonal matrix of the skeleton graph node degree; After the feature matrix H of the skeleton graph G is updated by multi-layer graph convolution, the output feature matrix H final , H final All row vectors are averaged and pooled to obtain the posture feature vector S23, based on the extracted key point P of the human nose nose =[x1,y1] and ankle key point P ankle =[x 17 ,y 17 ], estimate the approximate height of the human body h = max(|x1-x 17 |,|y1-y 17 |), normalize the posture vector extracted in step S1 to height Get the normalized posture vector P norm ; S24, stack two layers of fully connected layers to build a fully connected network, and normalize the posture vector P norm Perform feature extraction to obtain the posture feature vector S25, the posture feature vector extracted by the fully connected network in step S24 and the posture feature vector extracted by the graph convolutional network in step S22 Perform cascade splicing to obtain the fused feature vector F. Indicates feature concatenation by channel; finally, the feature F is input into a classifier consisting of two fully connected layers, and finally normalized by the softmax function to output the predicted posture category; S3. Fall judgment Set up a posture classification result queue, set the risk factor based on the timing information and classification results, and determine whether the current pedestrian has fallen.

2. According to claim 1, a graph convolutional network fall detection method based on human key points is characterized in that: The specific process of step S1 is as follows: S11, classifying postures into four types of postures: standing, bending, sitting and lying, and collecting image data of the four types of postures; S12, based on the posture estimation algorithm, the key point information corresponding to each type of posture image is obtained, and a posture key point data set is constructed; specifically, the posture estimation algorithm obtains the coordinate information of 17 key points of the human body. Taking the nose key point as an example, its coordinate information is p nose = [x1, y1], cascade the coordinate information of all key points to get the posture vector P = [p nose ,p neck ,…,p left_ankle ]=[x1,y1,...,x 17 ,y 17 ] to construct a pose key point dataset.

3. The fall detection method based on graph convolutional network of human key points according to claim 1 or 2, characterized in that: In step S3, a fall judgment is made according to the previous detection result, and the judgment process is as follows: a result queue with a length of T frames is set to store the posture classification results of T frames, and the current queue is cyclically updated according to each frame of the video stream; a threshold coefficient α is set, and the sum of the number of frames of the 'lying' posture in the result queue with a length of T frames is T lie , when T lie When >αT, a fall is judged.