A fall behavior detection method based on bat-gcn

Abstract

The application discloses a fall behavior detection method based on BAT-GCN, specifically comprising the following steps: adopting a TransPose model to frame a video into T pictures at a frame rate of 30 frames per second as input images, and then abstracting each key point of each person in each picture into a two-dimensional coordinate to construct a skeleton network sequence; performing data normalization on the skeleton network, taking a BAT-GCN model composed of 10 layers of space-time graph convolution operators as a fall detection algorithm framework of a behavior classification model, outputting the probability of predicting 60 action categories after calculation, and taking the one with the highest probability as the action prediction of the skeleton network sequence. The application has less calculation amount and higher detection precision, and meets the requirement of real-time detection of falling actions on low-power hardware devices.

Description

Technical Field

[0001] This invention belongs to the field of posture estimation, and in particular relates to a fall behavior detection method based on BAT-GCN. Background Technology

[0002] A fall is defined as a sudden, involuntary, and unintentional change in body position that results in a fall to the ground or another lower surface. Studies have shown that the fall rate within two years among people aged 60 and above in my country is 23.4%. Compared to previous results using data from the first three periods of the CHARLS study (19.28%, 20.83%, 19.41%, and 20.8%), the fall rate shows an upward trend, and the likelihood and danger of falls increase with age.

[0003] Therefore, if falls can be predicted or detected accurately and promptly, or if falls can be prevented by providing timely assistance, the risk of falls among the elderly will be greatly reduced. Currently, methods for detecting falls among the elderly, both domestically and internationally, can be broadly categorized into three types: 1) Environmental device-based detection methods. These methods detect falls based on the environmental noise generated when a person falls, such as sensing changes in object pressure and sound. However, this method is easily affected by environmental noise, greatly increasing the false alarm rate, and therefore is difficult to implement in daily life. 2) Wearable device-based detection methods. These devices sense information about body posture or movement, and then process this information through algorithms to determine whether a fall has occurred. However, wearing such devices increases the physical burden on the elderly and may be intrusive to some users sensitive to their physical privacy, adding additional psychological burden. 3) Computer vision-based detection methods. This method uses visual sensors (such as RGB cameras, infrared cameras, depth cameras, and camera array-based 3D methods) to record video signals of human activity within a monitored area. It then uses a human joint detection model to extract the skeletal network sequence of the human activity process from the video. Finally, it employs a behavior classification model based on an improved Graph Convolutional Network (GCN) to rapidly analyze the motion patterns through dynamic skeleton modalities, thereby accurately identifying whether the current human behavior constitutes a fall. Using this method for fall detection achieves real-time, efficient, and accurate detection, and it requires no wearable devices for the elderly, thus relieving them of physical burden.

[0004] Currently, research on fall detection methods based on computer vision mainly focuses on the following two aspects: keypoint estimation models and GCN-based behavior classification models.

[0005] Previous studies both domestically and internationally have proposed many joint estimation models, such as Openpose, AlphaPose, and TransPose, which have achieved good results in the field of joint estimation. Therefore, we can directly use one of these models for joint estimation. However, this paper chooses the TransPose model, which has an accuracy of 75.8% on the COCO 2017 validation set and a computational cost of only 21.8 GFLOPS, for joint estimation.

[0006] GCN-based behavior classification models utilize spatial graph convolution to extract features from each skeleton network at each time step, and employ temporal convolutional networks (TCNs) to capture the dynamic changes of the entire skeleton sequence over time, such as ST-GCN. ST-GCN, a classic skeleton-based behavior recognition model proposed at AAAI 2018, significantly improves the accuracy of skeleton-based behavior recognition tasks by applying graph convolution to topologically structured human skeleton data and using spatiotemporal graph convolution to extract spatiotemporal features. ST-GCN was the first model to use a skeleton network based on GCN for action recognition.

[0007] Since the release of ST-GCN, several methods have been developed based on it, improving the accuracy of action recognition for skeletal networks by enhancing feature extraction or optimizing model structure. For example... Figure 1 As shown on the left, ST-GCN employs a single-label partitioning strategy on a single frame, meaning it captures only the physical dependencies between adjacent joints based on a fixed skeleton graph, which may ignore implicit joint dependencies. Figure 1 As shown on the right, to capture richer dependencies, many models have already optimized ST-GCN's single-frame partitioning strategy using attention mechanisms. For example, self-attention mechanisms can more effectively capture the characteristics of the global field of view, enhancing the correlation between non-physically connected joints while maintaining the physical dependencies of existing joint connections. This strengthens the coherence of action features between frames, making the model's action representation of the skeletal network sequence more obvious, and ultimately improving the accuracy of action prediction. However, self-attention mechanisms in GCN suffer from low computational efficiency and high computational resource consumption, making them unsuitable for real-time detection on low-power devices. Although ST-GCN has only 2.0M parameters, making it suitable for low-power devices, its accuracy is not high enough to accurately identify fall behavior. Summary of the Invention

[0008] In order to use a fall detection model with less computation and higher detection accuracy to meet the requirement of real-time detection of fall actions on low-computing-power hardware devices, this invention provides a fall behavior detection method based on BAT-GCN.

[0009] This invention provides a fall detection method based on BAT-GCN, which uses the TransPose model as a human joint detection model and the BAT-GCN model as a behavior classification model. The specific framework of this fall detection algorithm is as follows:

[0010] First, the TransPose model extracts T frames from the video at a rate of 30 frames per second, using these T frames as input images. Then, it abstracts each keypoint of each person in each image into a two-dimensional coordinate (x, y), sorts them by keypoint number, and forms a sequence, which is the skeleton network. The T frames then contain the P values ​​of each person and the M values ​​of each C value. in C in =2-dimensional keypoints, forming T×Pe×M×C in The dimensional skeleton network is the skeleton network sequence; if no people are detected in a frame, then the skeleton network of these people is set to zero; this skeleton network sequence is the input of BAT-GCN.

[0011] Then, T×Pe×M×C in The dimensional skeleton network undergoes data normalization, meaning that in BAT-GCN, it is first transformed into Pe C's. in The feature sequence of dimension ×T×N is used as the input sequence f in Where N represents the number of joint connections; thus, a 1×M dimensional keypoint sequence is transformed into a 1×N dimensional joint connection sequence; the BAT-GCN model consists of 10 layers of temporal-space graph convolution operators ST-GCN Blocks. Specifically, β-GCN is responsible for performing C×N convolution operations in the spatial domain, and TCN is responsible for performing C×T convolution operations in the temporal domain; the number of output channels in the first three layers is C. out The number of output channels is 64 for the first three layers, 128 for the next three layers, and 256 for the last four layers. TCN is a one-dimensional convolution in the time dimension with a kernel size of 1×9. The calculation process can be represented by formula (1):

[0012] (1)

[0013] Where t represents the position of the one-dimensional convolution kernel weights, TCN shares the weights of the joint connections N in the spatial dimension, Td(i) represents the features in the time dimension, i represents the position of the features in the time dimension, and Ke(t) represents the temporal convolution kernel.

[0014] For the feature sequence output by the temporal-space graph convolution operator, the global average pooling operation layer performs average pooling operations on the temporal and spatial domains, outputting C. out The sequence is divided into 1×1 dimensions; finally, it passes through a fully connected layer, which outputs the probability of predicting 60 action categories. The one with the highest probability is taken as the action prediction for the skeleton network sequence.

[0015] Furthermore, each person has 17 key points, corresponding to the following body parts: 0: nose, 1: left eye, 2: right eye, 3: left ear, 4: right ear, 5: left shoulder, 6: right shoulder, 7: left elbow, 8: right elbow, 9: left wrist, 10: right wrist, 11: left hip, 12: right hip, 13: left knee, 14: right knee, 15: left ankle, 16: right ankle.

[0016] Furthermore, there are 17 joint connections, namely: [1, 3], [1, 0], [2, 4], [2, 0], [0, 5], [0, 6], [5, 7], [7, 9], [6, 8], [8, 10], [5, 11], [6, 12], [11, 12], [11, 13], [13, 15], [12, 14], [14, 16].

[0017] Furthermore, the β-GCN model adds a β attention mechanism, which uses a learnable parameter matrix β∈N×N to enhance the connection between each keypoint. The β-GCN model is constructed as follows, and its single-frame partitioning strategy model formula is as follows:

[0018] (2)

[0019] The calculation formula for the β attention mechanism is shown in formula (3):

[0020] (3)

[0021] in, and Indicates the positions of the i-th and j-th elements. A dimensional vector, used to represent the features and relevance of each keypoint; W Q W is the feature mapping of the i-th set of key points. K W is the feature mapping of the j-th key point, with initial values ​​following a random normal distribution; α These are the weighting coefficients, with an initial value of 1.

[0022] The beneficial technical effects of this invention are as follows:

[0023] (1) The BAT-GCN of the present invention uses an attention mechanism that is more efficient and less computationally complex than the self-attention mechanism, namely the β-attention attention mechanism, which enhances the correlation between joint connections, thereby improving the action classification accuracy and computational efficiency of the GCN model.

[0024] (2) This invention combines the TransPose model and the BAT-GCN model to create a new type of artificial intelligence fall detection system. Attached Figure Description

[0025] Figure 1 This is a comparison chart of ST-GCN and BAT-GCN.

[0026] Figure 2 This is a schematic diagram of the TransPose+ BAT-GCN fall detection algorithm framework of the present invention.

[0027] Figure 3 A schematic diagram illustrating the annotation of key points in the skeleton.

[0028] Figure 4 This is a diagram of the BAT-GCN model architecture of the present invention.

[0029] Figure 5 This is a structural diagram of the β-GCN model of the present invention.

[0030] Figure 6 Comparison of BAT-GCN training results (2D / 3D). Detailed Implementation

[0031] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0032] This invention discloses a fall detection method based on BAT-GCN, which uses the TransPose model as a human joint detection model and the BAT-GCN model as a behavior classification model. The specific structure of this fall detection algorithm framework is as follows: Figure 2 As shown, specifically:

[0033] like Figure 2 As shown, firstly, the TransPose model extracts T frames from the video at a rate of 30 frames per second, using these as input images. Then, it abstracts each keypoint of each person in each image into a two-dimensional coordinate (x, y), sorting them by keypoint number to form a sequence, such as: [(x0, y0), (x1, y1), ......, (x... 16 , y 16 [], which is the skeleton network, then the Pe of the T-frame image has M Cs. in C in=2-dimensional keypoints, forming T×Pe×M×C in The skeleton network, or skeleton network sequence, is used to determine the dimensionality of the data. If no individuals are detected in a frame, their skeleton networks are set to zero. This skeleton network sequence is the input to BAT-GCN. Based on this skeleton network sequence, BAT-GCN can determine 60 action categories, one of which is a fall, and the remaining 59 are everyday behaviors. If a fall is detected, the system issues an alarm.

[0034] like Figure 3 As shown, each person has 17 key points, corresponding to the following body parts: 0: nose, 1: left eye, 2: right eye, 3: left ear, 4: right ear, 5: left shoulder, 6: right shoulder, 7: left elbow, 8: right elbow, 9: left wrist, 10: right wrist, 11: left hip, 12: right hip, 13: left knee, 14: right knee, 15: left ankle, 16: right ankle.

[0035] like Figure 3 As shown, there are 17 joints: [1, 3], [1, 0], [2, 4], [2, 0], [0,5], [0, 6], [5, 7], [7, 9], [6, 8], [8, 10], [5, 11], [6, 12], [11, 12], [11,13], [13, 15], [12, 14], [14, 16].

[0036] like Figure 4 As shown, T×Pe×M×C in The dimensional skeleton network undergoes data normalization, meaning that in BAT-GCN, it is first transformed into Pe C's. in The feature sequence of dimension ×T×N is used as the input sequence f in Where N represents the number of joint connections; thus, a 1×M dimensional keypoint sequence is transformed into a 1×N dimensional joint connection sequence; the BAT-GCN model consists of 10 layers of temporal-space graph convolution operators ST-GCN Blocks. Specifically, β-GCN is responsible for performing C×N convolution operations in the spatial domain, and TCN is responsible for performing C×T convolution operations in the temporal domain; the number of output channels in the first three layers is C. out The number of output channels is 64 for the first three layers, 128 for the next three layers, and 256 for the last four layers. TCN is a one-dimensional convolution in the time dimension with a kernel size of 1×9. The calculation process can be represented by formula (1):

[0037] (1)

[0038] Where t represents the position of the one-dimensional convolution kernel weights, TCN shares the weights of the joint connections N in the spatial dimension, Td(i) represents the features in the time dimension, i represents the position of the features in the time dimension, and Ke(t) represents the temporal convolution kernel.

[0039] For the feature sequence output by the temporal-space graph convolution operator, the Global Average Pooling layer performs average pooling operations on the temporal and spatial domains, outputting C. out The sequence is divided into 1×1 dimensions; finally, it passes through a fully connected layer, which outputs the probability of predicting 60 action categories. The one with the highest probability is taken as the action prediction for the skeleton network sequence.

[0040] Depend on Figure 2 As can be seen, the partitioning strategy used in ST-GCN in the single-frame case is to define the subset as the entire neighborhood set itself. In this strategy, the feature vector of each adjacent node will be inner-producted with the same weight vector.

[0041] The internal connections of joints in a single frame are represented by an adjacency matrix A ∈ N×N and an identity matrix I ∈ N×N representing self-connections. The ST-GCN network using this partitioning strategy can be implemented by the following formula:

[0042]

[0043] Wherein, matrix W s Represents the input feature map f in The feature map is initialized to 1 and consists of learnable parameters.

[0044] In practice, this strategy has a significant drawback: in the case of a single frame, using this strategy is equivalent to calculating the inner product between the weight vector and the average feature vector of all neighboring nodes. This is suboptimal for skeletal sequence classification because this operation may lose local differential properties. Furthermore, since hand and foot movements during a fall are coherent and coordinated, but this strategy only represents the joint connections of keypoints, the β-GCN model of this invention adds a β-attention mechanism, i.e., a learnable parameter matrix β∈N×N, to enhance the connections between each keypoint. The β-GCN model is constructed as follows, and its single-frame partitioning strategy model formula is as follows:

[0045] (2)

[0046] The calculation formula for the β attention mechanism is shown in formula (3):

[0047] (3)

[0048] in, and Indicates the positions of the i-th and j-th elements. A dimensional vector, used to represent the features and relevance of each keypoint; W Q W is the feature mapping of the i-th set of key points. K W is the feature mapping of the j-th key point, with initial values ​​following a random normal distribution; α These are the weighting coefficients, with an initial value of 1.

[0049] like Figure 5 As shown, the calculation process of β-GCN can be visualized according to formulas (2) and (3). Wherein, W Q , , W K W α W s β, A, and I are all learnable parameters. This indicates that the matrix elements are added one by one. Represents the matrix dot product. Residual connection in C... in With C out Different input feature sequences need to undergo dimensionality transformation to enhance the model's generalization ability, allowing the β-GCN model to adapt to more types of action recognition datasets.

[0050] Since the original NTU-RGB+D dataset is a 3D dataset annotated with 25 keypoints, while TransPose outputs a 2D image set with 17 keypoints after image detection, this invention uses both the original NTU-RGB+D dataset and the NTU-RGB+D 2D dataset annotated with 17 keypoints for training to demonstrate the advantages and generalization ability of the BAT-GCN model. The training results are as follows: Figure 6 As shown, the accuracy curve trained on the NTU RGB+D 2D dataset is more stable. However, the model trained on the 3D dataset has a steeper training curve due to the limitations of camera angle capture and the presence of noisy labels in the NTU-RGB+D data, resulting in a relatively larger error. Thanks to the model's well-designed architecture and the reduced noise in the Z-axis direction of the 2D dataset, the model performs more stably on the 2D dataset.

[0051] The accuracy comparison of the model of this invention on the NTU RGB+d 2D and NTU RGB+d 3D datasets is shown in Table 1:

[0052] Table 1. Accuracy Comparison of BAT-GCN

[0053]

[0054] Table 2 shows a comparison of the BAT-GCN model with other models on the NTU RGB+D dataset:

[0055] Table 2. Accuracy Comparison of BAT-GCN

[0056]

[0057] This invention proposes a novel BAT (β-attention) attention mechanism, thereby constructing the BAT-GCN model. This model has only 1.37M parameters, 0.63M fewer than ST-GCN, offering advantages such as low computational resource consumption, high computational efficiency, and meeting real-time detection requirements. It can be used on low-computing-power devices, thus reducing hardware costs. In the NTU RGB+D original dataset, using Top-1 classification accuracy as the metric, the model achieves a test accuracy of 89.12% on the validation set using the cross-target (X-Sub) splitting criterion, an improvement of 7.62% over ST-GCN; and a test accuracy of 94.33% on the validation set using the cross-view (X-View) splitting criterion, an improvement of 6.03% over ST-GCN. Therefore, the proposed BAT-GCN model belongs to a class of lightweight GCN models with very high detection accuracy.

Claims

1. A fall behavior detection method based on BAT-GCN, characterized in that, The fall detection algorithm framework, which uses the TransPose model as the human joint detection model and the BAT-GCN model as the behavior classification model, is as follows: First, the TransPose model extracts T frames from the video at a rate of 30 frames per second, using these T frames as input images. Then, it abstracts each keypoint of each person in each image into a two-dimensional coordinate (x, y), sorts them by keypoint number, and forms a sequence, which is the skeleton network. The T frames then contain the P values ​​of each person and the M values ​​of each C value. in C in =2-dimensional keypoints, forming T×Pe×M×C in The dimensional skeleton network is the skeleton network sequence; if some people in Pe are not detected in a frame, then the skeleton network of these people is set to zero; this skeleton network sequence is the input of BAT-GCN. Then, T×Pe×M×C in The dimensional skeleton network undergoes data normalization, meaning that in BAT-GCN, it is first transformed into Pe C's. in The feature sequence of dimension ×T×N is used as the input sequence f in Where N represents the number of joint connections; thus, a 1×M dimensional keypoint sequence is transformed into a 1×N dimensional joint connection sequence; the BAT-GCN model consists of 10 layers of temporal-space graph convolution operators ST-GCN Blocks. Specifically, β-GCN is responsible for performing C×N convolution operations in the spatial domain, and TCN is responsible for performing C×T convolution operations in the temporal domain; the number of output channels in the first three layers is C. out The number of output channels is 64 for the first three layers, 128 for the next three layers, and 256 for the last four layers. TCN is a one-dimensional convolution in the time dimension with a kernel size of 1×9. The calculation process can be represented by formula (1): （1） Where t represents the position of the one-dimensional convolution kernel weights, TCN shares the weights of joint connections N in the spatial dimension, Td(i) represents the features in the time dimension, i represents the position of the features in the time dimension, and Ke(t) represents the temporal convolution kernel. The β-GCN model adds a β attention mechanism, namely a learnable parameter matrix β∈N×N, to enhance the connection between each keypoint. The model formula for its single-frame partitioning strategy is as follows: （2） The calculation formula for the β attention mechanism is shown in formula (3): （3） in, and Indicates the positions of the i-th and j-th elements. A dimensional vector, used to represent the features and relevance of each keypoint; W Q W is the feature mapping of the i-th set of key points. K W is the feature mapping of the j-th key point, with initial values ​​following a random normal distribution; α These are weighting coefficients, with an initial value of 1; For the feature sequence output by the temporal-space graph convolution operator, the global average pooling operation layer performs average pooling operations on the temporal and spatial domains, outputting C. out The sequence is divided into 1×1 dimensions; finally, it passes through a fully connected layer, which outputs the probability of predicting 60 action categories. The one with the highest probability is taken as the action prediction for the skeleton network sequence.

2. The fall behavior detection method based on BAT-GCN according to claim 1, characterized in that, Each person has 17 key points, corresponding to the following body parts: 0: nose, 1: left eye, 2: right eye, 3: left ear, 4: right ear, 5: left shoulder, 6: right shoulder, 7: left elbow, 8: right elbow, 9: left wrist, 10: right wrist, 11: left hip, 12: right hip, 13: left knee, 14: right knee, 15: left ankle, 16: right ankle.

3. The fall behavior detection method based on BAT-GCN according to claim 2, characterized in that, The joints are 17 in number: [1, 3], [1, 0], [2, 4], [2, 0], [0, 5], [0, 6], [5, 7], [7, 9], [6, 8], [8, 10], [5, 11], [6, 12], [11, 12], [11, 13], [13, 15], [12, 14], [14, 16].

Images