HAR for capturing activity spatial-temporal characteristics based on human body RFID skeleton fine granularity

Abstract

The invention relates to the field of HAR of bound RFID, and provides an HAR method for capturing activity spatio-temporal characteristics based on fine granularity of a human body RFID skeleton. Through a graph attention network (GAT), the problem that important and effective node information cannot be captured due to the fact that a traditional GCN is poor in modeling capacity for the relation between nodes is solved. Through a multi-scale time convolutional network, the problem that LSTM can only model a long-term dependency relationship and lacks the ability to capture local time details is solved. The combination of the two breaks through the limitation of the existing method in the aspect of fine-grained capture of the spatial-temporal characteristics of human activities. In the invention, the RFID with the advantages of privacy protection, identifiability, battery-free maintenance and the like is utilized to perceive human activities. Through an RFID system, a human body RFID skeleton diagram is designed to express human body activities, a human body RFID skeleton space-time diagram convolutional network (BRS-GCN) is provided, and space-time characteristics of the human body RFID skeleton diagram are captured on a fine-grained level, so that human body activity identification is realized. The performance advantages of the BRS-GCN in the aspect of activity identification compared with the existing HARs are verified through experiments.

Description

Technical Field

[0001] The present invention belongs to the HAR field of binding RFID, and in particular relates to a HAR method for capturing activity spatiotemporal features in fine granularity based on a human RFID skeleton. Background Art

[0002] Human activity recognition (HAR) is a technology that identifies and classifies human activities by analyzing and processing sensor data. It has important applications in many fields, including health monitoring, smart homes, security systems, sports analysis, and human-computer interaction.

[0003] Traditional human activity recognition is usually based on machine vision methods, which are limited by line of sight and light, and also pose a threat to user privacy and security. With the development of sensors, wireless sensors are also used in HAR, but this type of method has the disadvantages of difficult battery maintenance and strong invasiveness. These problems have made the existing HAR solutions based on machine vision and sensors controversial. As researchers have a deeper understanding of wireless signals, some wireless communication technologies such as WIFI are used for human activity recognition, effectively overcoming the related problems of existing solutions and achieving good results. However, they do not have the target object marking characteristics and cannot solve the problem of target object identity recognition.

[0004] RFID, with its tag-sensing characteristics, can effectively solve the three core key issues in the field of IoT: identity recognition, location positioning and state perception of target objects, and stands out among the existing human activity recognition solutions. RFID-based HAR includes non-binding RFID HAR and binding RFID HAR.

[0005] The types of activities recognized by the HAR of non-binding RFID are limited and cannot recognize more complex activities. There is edge fuzziness, which makes it impossible to perceive human activities in a fine-grained manner. In addition, the orientation between the human body and the tag will seriously affect the ability to perceive activities. Compared with sensor-based HAR, the HAR of binding RFID has the outstanding advantages of no power supply and maintenance, lightweight and low intrusion. At the same time, the identifiable advantage of RFID can also be used to identify active individuals and active parts. Therefore, the present invention studies the HAR of binding RFID that has privacy protection, easy identification of active individuals, low intrusion, and no battery maintenance. There were few early studies on HAR based on binding RFID, and most of them used traditional machine learning such as SVM and DTW as recognition methods. These HARs have limited modeling capabilities for complex activities, cannot effectively capture the spatiotemporal characteristics of activities, and are more difficult to model nonlinear problems. Summary of the invention

[0006] In order to overcome the problem that traditional GCN has weak modeling ability for the relationship between nodes and cannot capture important and effective node information, and LSTM can only model long-term dependencies and lacks the ability to capture local temporal details, the present invention proposes a HAR method based on human RFID skeleton which combines a graph attention network (GAT) with a multi-scale temporal convolutional network.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] According to the present invention, a HAR method for capturing fine-grained spatiotemporal features of activities based on a human RFID skeleton comprises the following steps:

[0009] First, deploy RFID tags to the key skeleton nodes of the subjects to collect the subjects' motion data.

[0010] Second, the motion data of each activity is preprocessed (data alignment and normalization).

[0011] Third, we build an RFID skeleton-based activity recognition model (BRS-GCN). (The multi-scale temporal convolution module captures the skeleton temporal features, and the graph attention module captures the skeleton spatial features.)

[0012] Fourth, use the constructed HAR model to train and test the model on the collected data set.

[0013] Fifth, save the final model and model parameters, and analyze the results.

[0014] Preferably, the tag response feature data involved in the collected subject's action includes not only RSSI (received signal strength), but also DF (Doppler) and phase. We use DF, RSSI and Phase at the same time to form a set of tag response feature data, and the t-th group of tag response feature data of 16 nodes constitutes the t-th human body RFID skeleton, which is used to represent the spatial relationship of human activities at time t. The combination of three feature data provides richer activity features, which is conducive to improving the classification accuracy of the model. The actions include: standing up, standing until squatting, squatting, squatting to standing up, standing until bending down, standing and keeping bending down, bending down to standing up, swinging arms and walking in big steps, swinging arms and walking in small steps, walking without swinging arms and walking in big steps, walking without swinging arms and walking in small steps, swinging arms and marching in place, walking without swinging arms and marching in place, sitting down, sitting quietly, sitting and bending down, sitting and keeping bending down, sitting to standing up, lying down, from lying down to sitting up, a total of 21 human activities.

[0015] As a preferred method, since the multi-tag reading mechanism of the RFID system may have tag occlusion and tag collision problems, the number of tag response records obtained by the reader for each tag is uncertain. In order to adapt to the calculation of the model and complete data alignment, we need to unify the tag response feature data length of the 16 nodes of the human RFID skeleton to h. Insufficient data is filled with zero values. In order to speed up the training of the model and improve the recognition accuracy of the model, we use the maximum absolute value of DF, RSSI and Phase collected for each activity to normalize the tag response feature data. The calculation formula is as follows: new_DF = DF / abs(DF), new_RSSI = RSSI / abs(RSSI), new_Phase = Phase / abs(Phase)

[0016] As a preference, we use the constructed RFID skeleton activity graph as the input of our model (BRS-GCN).

[0017] The Multi-Scale Temporal Convolutional Module (MSTC) in BRS-GCN is used to capture skeleton temporal features.

[0018] MSTC mainly consists of three layers: the first layer uses CNN with convolution kernels of 6×6 and 1×1 for convolution operations. The second and third layers are multi-scale temporal convolutions.

[0019] The multi-scale temporal convolution consists of 6 branches, each of which has a CNN with a convolution kernel of 1×1 to reduce the channel dimension. The first four branches contain four temporal convolutions with different dilation rates, and the dilation rates are 1, 2, 3 and 4 respectively.

[0020] This is followed by a MaxPool operation and a 1×1 convolution to implement the residual connection.

[0021] The outputs of the six branches are concatenated, and a 1×1 convolution is used to increase the channel dimension. The strides of the convolution kernels of the second and third layers of multi-scale temporal convolution are 1 and 2, respectively.

[0022] The graph attention module in BRS-GCN uses a multi-head attention network (GAT) to capture skeleton space features.

[0023] The graph attention module consists of a three-layer GAT network. The first layer contains three parallel GATs, which are then connected in series with two layers of GAT to obtain richer spatial features.

[0024] The calculation process of the multi-head attention network (GAT) is as follows: For each attention head k, in order to calculate the attention weights between nodes, we first need to calculate the adjacent node pairs<i,j> The node feature vector is linearly transformed to obtain the transformed feature vector: Then we can calculate the attention coefficient between node i and node j:

[0025]

[0026] Among them, || represents the splicing operation, is the weight parameter matrix of the linear transformation in the kth attention head, is the attention weight vector in the kth attention head. (i) ,N (i) represents the first-order neighborhood of node i. We use softmax() to normalize the attention weights:

[0027]

[0028] Function U() is a nonlinear activation function LeakyReLU(). Finally, we use the normalized attention weights to update the feature representation of node i, and concatenate the updated feature representations of all attention heads to obtain the final feature vector h of node i. i ′, the calculation formula is as follows:

[0029]

[0030] Where σ is the nonlinear activation function ReLU().

[0031] As a preferred method, the human activity dataset (CWNU-RDA) has the largest number of human activities in the field of binding RFID human activity recognition. The dataset has 21 human activities, including 11,018 activity data samples.

[0032] As a preference, we saved the best trained model and its parameters, analyzed the recognition effect of the model for each activity (i.e., analysis of the confusion matrix) when the model reached the best recognition accuracy (98.31%), and compared it with other models.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] (1) This paper designs a HAR method for fine-grained capture of activity spatiotemporal features based on human RFID skeletons. This method uses a model that combines a graph attention module and a multi-scale temporal convolution module. In the feature extraction process, the multi-scale temporal convolution module is used to extract long-term and short-term temporal information from the RFID skeleton, and the graph attention module is used to extract spatial features from the RFID skeleton to capture richer node relationships and node information. At the same time, the captured spatiotemporal features are used to perform activity classification and recognition tasks.

[0035] (2) The recognition accuracy of this method on the data set we collected can reach 98.31%, surpassing most human activity recognition methods based on binding RFID. In addition, we use multi-scale CNN to capture temporal features, which has fewer parameters than LSTM and a simpler model structure. Therefore, the model has the advantages of faster operation speed, less training time, and higher recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a human RFID skeleton.

[0037] Figure 2 This is the overall framework diagram of the model.

[0038] Figure 3 It is the multi-scale temporal convolution module in the overall framework diagram.

[0039] Figure 4 is the confusion matrix of the model. DETAILED DESCRIPTION

[0040] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0041] Reference Figure 1-3 The HAR method shown in the figure is based on the fine-grained capture of activity spatiotemporal features of the human RFID skeleton. This method uses a model that combines a graph attention module and a multi-scale temporal convolution module. In the feature extraction process, the multi-scale temporal convolution module is used to extract long-term and short-term temporal information from the RFID skeleton, and the graph attention module is used to extract spatial features from the RFID skeleton to capture richer node relationships and node information. At the same time, the captured spatiotemporal features are used to classify and identify activities. The specific process includes the following:

[0042] like Figure 1As shown in the figure, during the data collection process, we designed a human RFID skeleton. Based on the skeleton, we deployed RFID tags on the skeleton nodes of each subject to obtain key data of human activities. Every time the RFID system successfully reads a tag, the data collection system will obtain a tag response record u, which includes five values: electronic product code (EPC), timestamp (Timestamp), Doppler frequency (Doppler Frequency: DF), received signal strength (RSSI), and phase (Phase). The EPC value is the number m of the skeleton node, that is, the tag m (m∈[0,15]), and the three values ​​of DF, RSSI, and Phase together constitute a set of tag response characteristic data used to express human activities. During the duration of an activity (assuming 5 seconds), each tag generates at most h groups of tag response characteristic data. The tth (t∈[0,h-1]) group of tag response characteristic data of all 16 node tags constructs the tth human RFID skeleton. There is a sequential time relationship between the h groups of human RFID skeletons, that is, they constitute a human RFID skeleton sequence, and each human RFID skeleton structure fully expresses the spatial relationship of human activities. Therefore, h groups of human RFID skeletons jointly construct an RFID skeleton activity graph that can fully express human activities. We collected a total of 21 human activities. The data collected from all activities constitute our human activity dataset (CWNU-RDA).

[0043] In the data preprocessing stage, we align the label response characteristic data of each active node, that is, fill the nodes with less than h sets of label response characteristic data with zero values, and normalize each set of label response characteristic data using the maximum absolute value of DF, RSSI, and Phase.

[0044] like Figure 2 As shown in Figure 2, we built a human RFID skeleton spatiotemporal graph convolutional network (BRS-GCN). In the model construction stage, we used multi-scale CNN to extract temporal features from skeleton nodes and used graph attention network to extract spatial features from skeletons. The multi-scale temporal convolutional layer (MSTC) is shown in Figure 2. Figure 3As shown in the figure. MSTC mainly consists of three layers: the first layer is to increase the channel dimension and reduce the time dimension. CNNs with convolution kernels of 6×6 and 1×1 are used for convolution operations respectively. The second and third layers are multi-scale temporal convolutions. Multi-scale temporal convolutions include 6 branches, each of which has a CNN with a convolution kernel of 1×1 to reduce the channel dimension. The first four branches contain four temporal convolutions with different dilation rates, and the dilation rates are 1, 2, 3 and 4 respectively. Different dilation rates can adjust different receptive fields to perceive actions of different time lengths. Then there is the MaxPool operation and the 1×1 convolution to implement the residual connection. The outputs of the six branches are concatenated, and then a 1×1 convolution is used to increase the channel dimension. The difference between the multi-scale temporal convolutions in the second and third layers is that the stride of the convolution kernel is inconsistent. The stride of the second layer is 1, and the stride of the third layer is 2. Convolution kernels with different step lengths can capture the temporal characteristics of human activities at different time scales. Small step convolution kernels can capture local temporal details and fast movements, while large step convolution kernels can extract overall motion trends and long-term temporal dependencies. We designed a three-layer graph attention network as follows: Figure 2 As shown in the figure. In the first layer, we use three parallel GAT networks to enrich the convolution results. The second and third layers of GAT networks are to obtain different degrees of information aggregation results. In order to prevent the problem of low recognition accuracy caused by excessive data smoothing during the information aggregation process of the multi-layer GAT network, we splice the output of each layer of the graph attention network as the output of the spatial feature extraction module. When the data is input into the model, batch processing is performed first. After the spatial feature extraction, pooling and softmax() are used for activity classification.

[0045] During the training process, we divided all the activity data samples (11018) in the human activity dataset into a training set (8825 activity samples), a validation set (552 activity samples) and a test set (1641 activity samples) in a ratio of 8:0.5:1.5. The experimental setting details are as follows: The initialization parameters of the model are the system default values. The number of training rounds (Epoch) is 55. The initial learning rate (LR) is set to 0.003, and it decreases by 0.1 every 20 epochs. The batch size (Bs) is set to 256, the initial node vector size (h) is set to 192, and the feature dimension (Nc) is set to 512. The number of channels in each layer of the multi-scale temporal convolution layer (MSTC) is: 16, 36, 48 respectively. During the training process, we use the cross entropy loss function (CrossEntropy Loss), and use the Adam optimizer to update the model parameters according to the loss to achieve the optimal value. The model was trained for a total of 55 epochs, and achieved a recognition accuracy of 98.31% at epoch=42.

[0046] After training, the model parameters are saved. And our experimental results are as follows:

[0047] Figure 4 It shows that when the model reaches the maximum recognition accuracy of 98.31%, the recognition accuracy of the model for each activity. As can be seen from the figure, the recognition accuracy of the model for 7 of the activities (including activities labeled 2, 3, 11, 15, 18, 19, and 20) can reach 100%, and the recognition accuracy for other activities is greater than 90%, and even the recognition accuracy of many activities is close to 99%. This further illustrates the effectiveness and high accuracy of our model recognition. Among them, the activities with the lowest recognition accuracy are: "arm swinging and walking with big steps" (7: 92%), "arm swinging and walking with small steps" (8: 97%), "no arm swinging and walking with big steps" (9: 96%), and "no arm swinging and walking with small steps" (10: 97%). These are all activities with high similarity. 7% of the activities in "arm swinging and walking with big steps" are identified as "arm swinging and walking with small steps". 3% of the activities in "arm swinging and walking with small steps" are identified as "arm swinging and walking with big steps". 3% of the activities in "long steps without arm swinging" were identified as "small steps without arm swinging", and 1% of the activities were identified as "long steps with arm swinging". 2% of the activities in "small steps without arm swinging" were identified as "long steps without arm swinging", and 1% of the activities were identified as "small steps with arm swinging". These phenomena can show that the similarity between activities has a negative impact on the classification accuracy of the model. However, the recognition accuracy of the model we proposed for similar activities is mostly higher than 95%, which is enough to show that our model is reliable and highly accurate.

[0048] In addition, we evaluate the recognition performance of several deep learning and skeleton-based human activity recognition models on our dataset (CWNU-RDA). These models have difficulty in identifying important adjacent nodes and capturing local temporal features, resulting in low classification accuracy and high model complexity. In contrast, the BR-GCN proposed in this paper achieves the highest classification accuracy with fewer parameters. The comparison results are shown in Table 1.

[0049] Table 1. Classification model comparison experimental results

[0050] Classification accuracy (%) Parameter size (MB) GCN 73.09 2.74 LSTM 93.45 14.53 ST-GCN 97.24 11.82 CTR-GCN 97.55 5.74 BRS-GCN (Ours) 98.31 6.17

[0051] The present invention and its embodiments are described schematically above, and the description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by it and designs a structural method and an embodiment similar to the technical solution without creativity without departing from the purpose of the invention, they shall all fall within the protection scope of the present invention.

Claims

1. A HAR method for capturing fine-grained spatiotemporal features of activities based on human RFID skeleton, comprising the steps of: (1), RFID tags are deployed to the key skeleton nodes of the subjects to collect the subjects' motion data. (2),Preprocess the motion data of each activity (data alignment and normalization). (3) Construct an RFID skeleton-based activity recognition model (BRS-GCN). (The multi-scale temporal convolution module captures the skeleton temporal features, and the graph attention module captures the skeleton spatial features.) (4) Use the constructed HAR model to train and test the model on the collected dataset. (5) Save the final model and model parameters, and analyze the results.

2. A HAR method for capturing fine-grained spatiotemporal features of activities based on human RFID skeleton as described in claim 1, characterized in that : The tag response feature data involved in the subject's action collected in step (1) includes not only RSSI (received signal strength), but also DF (Doppler) and phase. We use DF, RSSI and Phase at the same time to form a set of tag response feature data. The t-th group of tag response feature data of 16 nodes constitutes the t-th human body RFID skeleton, which is used to represent the spatial relationship of human activities at time t. The combination of the three feature data provides richer activity features, which is conducive to improving the classification accuracy of the model. The actions include: standing up, standing until squatting, squatting, squatting to standing up, standing until bending down, standing and keeping bending down, bending down to standing up, swinging arms and walking in big steps, swinging arms and walking in small steps, walking without swinging arms and walking in big steps, walking without swinging arms and walking in small steps, swinging arms and marching in place, marching in place without swinging arms, sitting down, sitting quietly, sitting and bending down, sitting and keeping bending down, sitting to standing up, lying down, lying down to sitting up, a total of 21 human activities.

3. A HAR method for capturing fine-grained spatiotemporal features of activities based on human RFID skeleton as described in claim 1, characterized in that : In step (2), due to the problem of tag occlusion and tag collision in the multi-tag reading mechanism of the RFID system, the number of tag response records obtained by the reader for each tag is uncertain. In order to adapt to the calculation of the model and complete data alignment, we need to unify the length of the tag response feature data of the 16 nodes of the human RFID skeleton to h. Insufficient data is filled with zero values. In order to speed up the training of the model and improve the recognition accuracy of the model, we use the maximum absolute value of DF, RSSI and Phase collected for each activity to normalize the tag response feature data. The calculation formula is as follows: new_DF=DF / abs(DF), new_RSSI=RSSI / abs(RSSI, new_Phase=Phase / abs(Phase).

4. A HAR method for capturing activity spatiotemporal features based on fine-grained human RFID skeleton as described in claim 1, characterized in that: in step (3), we use the constructed RFID skeleton activity graph as the input of the model (BRS-GCN). 4.1, the multi-scale temporal convolution module (MSTC) in BRS-GCN is used to capture skeleton temporal features. 4.1.1, MSTC mainly consists of three layers: the first layer uses CNN with convolution kernels of 6×6 and 1×1 for convolution operations. The second and third layers are multi-scale temporal convolutions. 4.1.2, the multi-scale temporal convolution consists of 6 branches, each of which has a CNN with a convolution kernel of 1×1 to reduce the channel dimension. The first four branches contain four temporal convolutions with different dilation rates, and the dilation rates are 1, 2, 3 and 4 respectively. 4.1.3, followed by the MaxPool operation and a 1×1 convolution to implement the residual connection. 4.1.4, the outputs of the six branches are concatenated, and a 1×1 convolution is used to increase the channel dimension. The stride of the convolution kernel of the second and third layers of multi-scale temporal convolution is 1 and 2 respectively. 4.2, the graph attention module in BRS-GCN uses a multi-head attention network (GAT) to capture skeleton space features. 4.2.1, the graph attention module consists of a three-layer GAT network. The first layer contains three parallel GATs, which are then connected in series with two layers of GAT to obtain richer spatial features. 4.2.2, the calculation process of the multi-head attention network (GAT) is as follows: For each attention head k, in order to calculate the attention weights between nodes, we first need to calculate the adjacent node pairs<i,j> The node feature vector is linearly transformed to obtain the transformed feature vector: Then we can calculate the attention coefficient between node i and node j: in, || represents the concatenation operation. is the weight parameter matrix of the linear transformation in the kth attention head, is the attention weight vector in the kth attention head. (i) ,N (i) represents the first-order neighborhood of node i. We use softmax() to normalize the attention weights: Function U() is a nonlinear activation function LeakyReLU(). Finally, we use the normalized attention weights to update the feature representation of node i, and concatenate the updated feature representations of all attention heads to obtain the final feature vector h of node i. i ′, the calculation formula is as follows: Where σ is the nonlinear activation function ReLU().

5. A HAR method for capturing fine-grained spatiotemporal features of activities based on human RFID skeleton as claimed in claim 1, characterized in that: The human activity dataset (CWNU-RDA) used in step (4) has the largest number of human activities in the field of binding RFID human activity recognition. This dataset has 21 human activities, including 11,018 activity data samples.

6. A HAR method for capturing fine-grained spatiotemporal features of activities based on human RFID skeleton as claimed in claim 1, characterized in that: In step (5), we saved the best trained model and its parameters, analyzed the recognition effect of the model for each activity (i.e., the analysis of the confusion matrix) when the model reached the best recognition accuracy (98.31%), and compared it with other models.

Images