Human body behavior recognition method combining graph convolutional network and time convolutional network based on attention mechanism

Abstract

The invention discloses a human behavior recognition method combining a graph convolutional network and a time convolutional network based on an attention mechanism. The method comprises the following steps: collecting CSI information of various human behaviors in various scenes; cSI amplitude information is extracted, the CSI amplitude is preprocessed, firstly, an abnormal value is removed by using Hampel, then data denoising is realized through discrete wavelet transform, and finally normalization is performed; converting the preprocessed CSI amplitude data into a CSI feature image, and constructing and dividing a data set; a training set is used as input of a GTA-net model, the model firstly uses a graph convolutional network to extract spatial features between CSI feature images, then uses a time convolutional network fused with an attention mechanism to extract a sequential relationship of the features, and further processes and classifies the sequential relationship. The method solves the problem of insufficient precision in the field of human behavior detection, has high recognition accuracy, is suitable for various scenes, and provides a new solution for human behavior recognition.

Description

Technical Field

[0001] The present invention belongs to the field of wireless sensing technology, and specifically relates to a human behavior recognition method using a temporal convolutional network that combines a graph convolutional network with an attention mechanism. Background Art

[0002] With the rapid development of Internet of Things technology, people's lives are gradually moving towards a new era of intelligence, and concepts such as smart cities, smart communities, and smart homes are gradually becoming a reality. Human-computer interaction can achieve information transmission between people and computers through computer input and output devices, thereby completing effective dialogue between people and machines and promoting the deep integration of the physical world and the information world. The key lies in the way of perception, the transition from subjective perception, sensors to sensor networks, and gradually achieving a deeper and more natural human-computer interaction.

[0003] Human behavior recognition aims to use different sensor devices to simulate human visual perception ability, analyze human behavior status, and realize intelligent interpretation of complex dynamic scenes. Depending on whether the user needs to wear a sensing device, the sensing method can be divided into contact and non-contact sensing. Contact sensing technology mainly uses various sensors built into wearable sensing devices such as accelerometers, gyroscopes, pressure sensors, etc. to realize the perception of human movement and behavior, while the latter mainly realizes non-contact perception through computer vision technology or various wireless radio frequency signals. Contact human action recognition relies on sensor devices worn by users. However, in practical applications, users of wearable sensors may face the problem of inconvenience in wearing. Non-contact methods mainly include vision-based and radio frequency-based methods. Vision-based methods rely on line of sight and lighting conditions. Obstacles or dim environments will hinder accurate detection, and there are privacy violations. Radio frequency-based methods realize the perception, recognition and detection of the human body by analyzing the wireless signals received by the receiver, thanks to its non-contact, non-line-of-sight and privacy protection characteristics. Because WiFi has natural advantages in cost, ease of use, and universality, and can achieve non-line-of-sight, non-invasive, and privacy-protecting perception, contactless human behavior and motion perception technology based on WiFi signals has gradually become a research hotspot in the field.

[0004] Wang et al. proposed WiHear technology, in which directional antennas are used to capture CSI changes caused by oral movements. WiHear has an average detection accuracy of 91% when a single sentence does not exceed 6 words, and an average detection accuracy of 74% when no more than 3 people speak at the same time; however, this application is only used to monitor spoken language. Arshad et al. proposed Wi-Chase technology based on CSI for human behavior recognition. Wi-Chase makes full use of all available subcarriers in WiFi signals and combines their phase and amplitude changes. The recognition accuracy for the walking category fails to reach 91%, and only a single scenario is considered. Sigg et al. proposed a hardware device USRP to collect RSSI signals from WiFi devices. They used RSSI signals to identify four activities: standing, walking, lying down, and crawling, with an accuracy of only 80%. Yang et al. developed a human behavior recognition framework based on CSI signals. They first proposed an algorithm that can automatically select antennas based on their sensitivity to various activities. Secondly, two signal enhancement techniques are used, namely enhancing active signals and weakening inactive signals. The average accuracy reaches 96.82%, but only a single scenario is considered, and the performance is insufficient. Damodaran et al. proposed two algorithms: SVM classification and LSTM recurrent neural network. SVM requires complex preprocessing, while LSTM directly processes the data after wavelet denoising. Experiments show that when the data is small, the accuracy of LSTM (97.33%) is higher than that of SVM. However, only four human activities were collected, which is very different from the real environment. Chen et al. proposed an attention-based bidirectional long short-term memory network (ABLSTM) to identify human activities using CSI signals, with an accuracy of nearly 97.3% on public and self-made data sets. Yan et al. proposed an adaptive activity segmentation algorithm (AACA), which can distinguish different activities based on the relationship between body movement amplitude and movement amplitude. The average accuracy is only 94.20%. Shang et al. proposed a deep learning model that uses CSI signals combined with LSTM-CNN. The average result of their model on public data sets is 94.14%. Pritam et al. proposed a DL architecture LSTM to automatically extract features. When using CSI amplitude and phase signals at the same time, the accuracy reaches 92.29%, while the accuracy decreases when using phase signals alone. Alsaify et al. proposed a method to extract and select important features. Finally, the feature is used to train a support vector machine (SVM) classifier to identify different execution activities. This method only achieves an accuracy of 91.27% and requires a lot of preprocessing, resulting in low performance. Summary of the invention

[0005] The purpose of the present invention is to provide a human behavior recognition method that combines a graph convolutional network and a temporal convolutional network based on an attention mechanism, which solves the problems of the existing human behavior recognition methods having a single applicable scenario range and low detection accuracy.

[0006] The technical solution adopted by the present invention is: combining a human behavior recognition method with a graph convolutional network and a time convolutional network based on an attention mechanism, building a WiFi detection area with multiple antennas, a single transmitter and a single receiver, and collecting CSI data of different human behaviors; extracting CSI amplitude information from the original CSI data, preprocessing the CSI amplitude data, first using Hampel to remove outliers, then implementing data denoising through discrete wavelet transform, and finally normalizing the data; converting the preprocessed CSI amplitude data into a CSI feature map, constructing and dividing a data set; inputting the training set constructed by the CSI feature map into the human behavior recognition model GTA-net for training, extracting effective feature information of human behavior from it and classifying it; the specific operations are as follows: Step 1: Build a MIMO-OFDM WiFi detection area with multiple antennas, single transmitter and single receiver, and design experimental scenarios to collect CSI data of different human behaviors; Step 2, extracting CSI amplitude information from the CSI data collected in step 1, and preprocessing the CSI amplitude data; Step 3, converting the CSI amplitude data preprocessed in step 2 into a CSI feature map, constructing and dividing the data set into a training set, a validation set, and a test set; Step 4: Input the CSI feature map converted in step 3 into the human behavior recognition model GTA-net. First, use the graph convolution network GCN to extract the spatial features of the CSI feature map. Then, input the extracted effective features into the temporal convolution network TCN fused with the attention mechanism, and extract the temporal relationship of the features to complete the training of human behavior recognition. Save the model parameters and obtain the optimized human behavior recognition model. Step 5: Input the test set into the human behavior recognition model GTA-net in step 4 for classification, and finally obtain the prediction result.

[0007] The present invention is also characterized in that: Step 1 is as follows: Step 1.1, build an experimental platform for collecting CSI data. The receiving end and the transmitting end are each equipped with three antennas. The Dell desktop is used as the receiver. It needs to be equipped with an Intel5300 network card and a Linux system, and the Linux 802.11nCSI-Tool tool is configured. The transmitter uses a commercial standard 2.4GHz WiFi b / g / n TP-Link router, and the packet frequency is set to 100Hz. The WiFi signal operating frequency is IEEE 802.11n and 2.4 GHz, and the bandwidth is 40MHz, which can capture the CSI of the complex matrix; Step 1.2: Design and arrange the experimental scene, collect CSI signals of different human behaviors in indoor and corridor scenes under line-of-sight paths, set the distance between the person and the transmitter and receiver to 1 meter, and place the person in the center of the transmitter and receiver.

[0008] Step 2 specifically includes the following steps: Step 2.1, collect the CSI data of different human behaviors under the line-of-sight path through step 1, obtain a file with the suffix dat, and read the file using the read_bf_file function based on the Matlab software platform; among them, the CSI data is a N A two-dimensional matrix of rows; CSI data matrix It is expressed as: (1) is the number of antennas at the transmitting end, is the number of antennas at the receiving end, The transmitting antenna The first antenna to the receiving antenna The CSI amplitude data of the root antenna, It is expressed as: (2) is the number of subcarriers in the CSI signal, on the Intel 5300 wireless network card is 30; from step 1.1, we know that the receiving end and the transmitting end are each equipped with three antennas, so the received signal contains 9 data streams, and the data contained in each data packet is explained as follows:

[0009] Step 2.2, select all subcarriers of the three receiving antennas on any transmitting antenna and calculate the original amplitude; where the collected CSI data is a complex matrix, which is expressed as , the CSI amplitude can be obtained by the complex calculation formula , the calculation formula is as follows: (4) Step 2.3, preprocess the CSI amplitude calculated in step 2.2. First, use the Hampel test to remove outliers. The outliers are identified by calculating the median and median absolute deviation of each data point, and these outliers are replaced by the local median. Then, the discrete wavelet transform is used for denoising. Finally, linear normalization is used to map the data to the standard range of 0 to 255.

[0010] Step 3 is as follows: Step 3.1, each action is composed of CSI data on multiple transmit and receive antenna links, and the CSI amplitude data on each transmit and receive antenna link is regarded as an independent CSI feature map; these maps are similar to a channel in an image, such as the red, green or blue channel in an RGB image, where the number of channels corresponds to the number of transmit and receive antenna links; specifically, the columns in the CSI feature map represent the CSI amplitude data on the transmit and receive antenna links, and the rows represent samples collected at different time points; Step 3.2, select one transmitting antenna and three receiving antennas to form three receiving links, combine the CSI feature graphs constructed on the three receiving links, and combine them into a W*30*3 structure, where W is the number of data packets divided by seconds, 100 packets / second; 30 is the number of subcarriers of each receiving link, and 3 is the combination of the amplitudes of the data obtained on the three receiving links, and construct a data set; Step 3.3, divide the data set obtained in step 3.2 according to the ratio of training set: validation set: test set = 7:1.5:1.5.

[0011] The human behavior recognition model GTA-net described in step 4 is constructed by integrating the graph convolutional network GCN and the time convolutional network TCN based on the attention mechanism; The GCN network contains two convolutional layers, the number of input channels is 90, the output of the first convolutional layer includes 64 channels, then, the ReLu activation function is used after the first convolution layer, and the dropout technique is used to implement random inactivation of neurons to prevent overfitting; the number of output channels of the second convolutional layer is 52, the output of the second convolutional layer is used as the input of the TCN network, the loss function of the GCN network is the cross entropy loss, the optimizer is SGD, the learning rate is 7.5e-5, the number of iterations is 500, and the batch size is 16; The TCN network includes multiple time blocks, attention blocks and fully connected layers. Each time block contains an expansion convolution block. The convolution block is composed of a convolution layer, an activation layer, and a Dropout layer. The number of input and output features of each time block is determined by the channel number list; the expansion rate of the time block increases with the increase of blocks. The attention block improves the model's ability to focus on important information by giving different weights to different parts of the input sequence. This attention mechanism combined with TCN can further improve the model's ability to understand and process time series data.

[0012] The channel number list of the TCN network is [52, 52, 52, 72, 72], and the input data tensor includes batch size, number of features, and time series length. The batch size, number of features, and time series length are [16, 52, 100] respectively.

[0013] Step 4 is as follows: Step 4.1: Use the training set divided in step 3.3 as the input of the human behavior recognition model GTA-net. GTA-net first uses the graph convolutional network GCN to extract the spatial features of the CSI feature map; Step 4.2: Input the spatial features of the CSI feature graph in step 4.1 into the temporal convolutional network TCN fused with the attention mechanism to extract the temporal relationship of the features. The specific extraction method is as follows: First, use the permute() function to adjust the tensor of the input data, then traverse each layer in the list and calculate the expansion rate of each layer; the number of input channels of the first layer is the number of output channels of the GCN network; the number of input channels of subsequent layers uses the number of output channels of the previous layer, and the number of output channels of each layer is set according to the value in the channel number list; Next, create a time block and pass in the following parameters: kernel size kernel_size is 3, drop_out rate is 0.2, attention mechanism dimension attention_size is 16; expansion rate dilation and the corresponding padding padding The calculation formula is as follows: (5) (6) in, i It is the layer index, and its specific value range is 0~4; According to whether the number of input channels and output channels of each time block is the same, a 1x1 convolution layer is created to convolve the input to adjust the dimension, the output of the convolution block is added to the adjusted input, and activated by the activation function relu; the input features are then mapped to the same number of features as the dimension of the attention mechanism, and then the Tanh activation function is used to map the number of features to a single output feature, the attention weight is calculated, and then the fully connected layer outputs the features to obtain the final classification result, that is, the corresponding human behavior is detected.

[0014] Step 5 is as follows: Input the test set obtained by step 3.3 into the human behavior recognition model obtained by step 4 for testing, realize classification, and finally obtain the prediction result.

[0015] The beneficial effects of the present invention are as follows: the human behavior recognition method proposed by the present invention, which combines a graph convolutional network and a time convolutional network based on an attention mechanism, aims at the problems of the existing human behavior recognition methods having a single applicable scenario range and low detection accuracy. The method includes preprocessing the original CSI. First, the CSI amplitude data is extracted from the original CSI data, and then Hampel is used to detect and remove outliers. Discrete wavelet transform is then used for denoising. Finally, linear normalization is used to map the data to a standard range (0 to 255). The preprocessed CSI amplitude is converted into a CSI feature map, a data set is constructed and divided, and the training set is input into the human behavior recognition method model GTA-net. GTA-net first uses the graph convolution network GCN to extract the spatial features of the CSI feature image, and then uses the temporal convolution network TCN with an integrated attention mechanism to extract the temporal relationship of the features. It can effectively process and analyze data with complex temporal and graph structural features, thereby improving the accuracy of recognition. In-depth evaluation shows that this method can cope with a variety of application scenarios and is superior to traditional human behavior recognition methods. The minimum accuracy of the model in indoor environments and corridor scenes reached 98.30% and 99.32% respectively; the accuracy of the model in various scenarios has reached more than 98%, and the best can reach 99.83%. The present invention further solves the problems of single scene range and low detection accuracy, making up for the shortcomings of traditional models. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a framework diagram of the human behavior recognition model GTA-net of the present invention; Figure 2 (a) CSI feature map when a person is walking; Figure 2 (b) CSI feature map when a person is in a lying position; Figure 3 It is a structural diagram of the human behavior recognition model based on deep learning of the present invention; Figure 4 (a) is the confusion matrix of the detection results in the indoor scene; Figure 4 (b) is the confusion matrix of the detection results in the corridor scene. DETAILED DESCRIPTION

[0017] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments so that the advantages and features of the present invention can be more easily understood by those skilled in the art.

[0018] Example 1 Combining the human behavior recognition method of graph convolutional network and temporal convolutional network based on attention mechanism, a WiFi detection area with multiple antennas, single transmitter and single receiver is built to collect CSI data of different human behaviors; CSI amplitude data is extracted from the original CSI data, and the CSI amplitude data is preprocessed. First, Hampel is used to remove outliers, and then data denoising is achieved through discrete wavelet transform, and finally the data is normalized; the preprocessed CSI amplitude data is converted into CSI feature graph, and the data set is constructed and divided; the training set constructed by the CSI feature graph is input into the human behavior recognition model GTA-net for training, and the effective feature information of human behavior is extracted and classified.

[0019] Example 2 Step 1: Build a MIMO-OFDM WiFi detection area with multiple antennas, single transmitter and single receiver, and design experimental scenarios to collect CSI data of different human behaviors; Step 2, extracting CSI amplitude information from the CSI data collected in step 1, and preprocessing the CSI amplitude data; Step 3, converting the CSI amplitude data preprocessed in step 2 into a CSI feature map, constructing and dividing the data set into a training set, a validation set, and a test set; Step 4: Input the CSI feature map converted in step 3 into the human behavior recognition model GTA-net. First, use the graph convolution network GCN to extract the spatial features of the CSI feature map. Then, input the extracted effective features into the temporal convolution network TCN fused with the attention mechanism, and extract the temporal relationship of the features to complete the training of human behavior recognition. Save the model parameters and obtain the optimized human behavior recognition model. Step 5: Input the test set into the human behavior recognition model GTA-net in step 4 for classification, and finally obtain the prediction result.

[0020] Example 3 The present invention combines a graph convolutional network and a human behavior recognition method based on a time convolutional network with an attention mechanism, builds a WiFi detection area with multiple antennas, a single transmitter and a single receiver, collects CSI data of different human behaviors; extracts CSI amplitude data from the original CSI data, and preprocesses the CSI amplitude data. First, Hampel is used to detect and remove outliers. Then, discrete wavelet transform is used for denoising. Finally, linear normalization is used to map the data to a standard range (0 to 255). The preprocessed CSI amplitude is converted into a CSI feature map, a data set is constructed and divided, and the training set is input into the human behavior recognition method model GTA-net. GTA-net first uses a graph convolutional network GCN to extract the spatial features of the CSI feature image, and then uses a time convolutional network TCN with an attention mechanism to extract the temporal relationship of the features. It can effectively process and analyze data with complex temporal and graph structure features, thereby improving the accuracy of recognition. During the training process, the model parameters with the best recognition rate for the verification set are saved, thereby constructing a model with superior classification performance. Finally, the data in the test set is input into the best recognition model trained in step 4 to achieve classification, and finally obtain the prediction result. Experimental verification shows that the invention has a high accuracy rate in various data sets and various scenarios, making up for the defects brought by traditional models.

[0021] Example 4 The present invention combines a graph convolutional network with a human behavior recognition method based on a temporal convolutional network with an attention mechanism, such as Figure 1 As shown, the specific steps include: Step 1: Build a MIMO-OFDM WiFi detection area with multiple antennas, single transmitters and single receivers, and design experimental scenarios to collect CSIs of different queue numbers. Step 1 specifically includes: Step 1.1, build an experimental platform for collecting CSI. The receiving end and the transmitting end are each equipped with three antennas. The Dell desktop is used as the receiver. The Intel5300 network card and Linux system must be installed, and the Linux 802.11nCSI-Tool tool must be configured. The transmitter uses a commercial standard 2.4GHz WiFi b / g / n TP-Link router, and the packet frequency is set to 100Hz. The WiFi signal operating frequency is IEEE 802.11n and 2.4 GHz, and the bandwidth is 40MHz, which can capture the CSI of the complex matrix; Step 1.2: Design and arrange the experimental scene, collect CSI signals of different human behaviors in indoor and corridor scenes under line-of-sight paths, set the distance between the person and the transmitter and receiver to 1 meter, and place the person in the center of the transmitter and receiver.

[0022] Step 2 specifically includes the following steps: Step 2.1, collect the CSI of different human behaviors under the line-of-sight path through step 1, and obtain a file with the suffix dat. Use the read_bf_file function based on the Matlab software platform to read the file; among them, the CSI data is a N A two-dimensional matrix of rows, N is the total number of packets; CSI data matrix It is expressed as: (1) is the number of antennas at the transmitting end, is the number of antennas at the receiving end, The transmitting antenna The first antenna to the receiving antenna CSI data of the root antenna, It is expressed as: (2) is the number of subcarriers in the CSI signal, on the Intel 5300 wireless network card is 30; as shown in step 1.1, the receiving end and the transmitting end are each equipped with three antennas, so the received signal contains 9 data streams, and the data contained in each data packet can be explained as follows:

[0023] Step 2.2, select all subcarriers of three receiving antennas on a certain transmitting antenna and calculate the original amplitude; wherein the collected CSI data is a complex matrix, which is expressed as , the CSI amplitude can be obtained by the complex calculation formula , the calculation formula is as follows: (4) Step 2.3, preprocess the CSI amplitude calculated in step 2.2. First, use Hampel to detect and remove outliers. It identifies outliers by calculating the local median and median absolute deviation, and replaces these outliers with the local median to reduce their impact on data analysis. Next, use discrete wavelet transform for denoising. This method separates noise and useful signals by decomposing the signal into different frequency levels. Finally, use linear normalization to map the data to a standard range (0 to 255) so that different data sets have the same scale for easy comparison and analysis.

[0024] Step 3, convert the preprocessed CSI amplitude data obtained in step 2 into a CSI feature image according to the following steps, and construct and divide the data set; Step 3 is as follows: Step 3.1, each action is composed of channel state information (CSI) data on multiple transmit and receive antenna links. We regard the CSI data on each transmit and receive antenna link as an independent CSI feature sub-graph; these sub-graphs are similar to a channel in an image, such as the red, green or blue channel in an RGB image, where the number of channels corresponds to the number of transmit and receive antenna links. Specifically, the columns in the constructed feature image represent the CSI data on the transmit and receive antenna links, and the rows represent samples collected at different time points.

[0025] Figure 2 (a) and Figure 2 (b) are CSI feature graphs under different behaviors. Figure 2 (a) is the CSI feature graph when a person is walking, and Figure 2 (b) is the CSI feature graph when a person is lying down.

[0026] Step 3.2, select one transmitting antenna and three receiving antennas to form three receiving links, combine the CSI feature sub-images constructed on the three receiving links, and combine them into a W*30*3 structure, where W is the number of data packets divided by seconds (100 packets / second), 30 is the number of subcarriers of each receiving link, and 3 is the combination of the amplitudes of the data obtained on the three receiving links to construct a data set; Step 3.3, divide the data set obtained in step 3.2 according to the ratio of training set: validation set: test set = 7:1.5:1.5.

[0027] Step 4: Input the CSI graph structure data converted in step 3 into the human behavior recognition model GTA-net (such as Figure 3 The model parameters with the best recognition rate on the validation set are saved, thereby building a model with superior classification performance. Step 4 is as follows: Step 4.1, use the training set divided in step 3.3 as the input of the human behavior recognition model GTA-net. GTA-net first uses the graph convolutional network GCN to extract the spatial features of the CSI feature image, and then uses the temporal convolutional network TCN with an integrated attention mechanism to extract the temporal relationship of the features. It can effectively process and analyze data with complex temporal and graph structure features, and is suitable for the CSI feature image graph structure time series prediction and classification tasks faced by this patent.

[0028] Step 4.2, the GCN network specifically includes two convolutional layers, the number of input channels is 90, the output of the first convolutional layer includes 64 channels, then, the ReLu activation function is used after the first convolution layer, and the dropout technology is used to implement random inactivation of neurons to prevent overfitting, the number of output channels of the second convolutional layer is 52, and the output of the second convolutional layer is used for the subsequent TCN network. The loss function of the GCN network model is cross entropy loss, the optimizer is SGD, the learning rate is 7.5e-5, the number of iterations is 500, and the batch size is 16; Step 4.3, input the data output from step 4.1 into the TCN network. In the TCN network, there are multiple temporal blocks TemporalBlock, each of which contains an expansion convolution layer. The number of input and output features of each temporal block is determined by the channel number list num_channels, followed by weight normalization, ReLU activation function and Dropout layer. The expansion rate of the temporal block increases with the increase of blocks, specifically, where i is the index of the temporal block. This design allows the model to capture information at different time scales while maintaining computational efficiency. In addition, the TCN architecture also includes an attention block AttentionBlock, which improves the model's ability to focus on important information by giving different weights to different parts of the input sequence. This attention mechanism combined with TCN can further improve the model's ability to understand and process time series data. Specifically, the permute() function is first used to adjust the shape of the input data tensor to [batch size, number of features, time series length]. Specifically, the size of the tensor is (16, 52, 100). The channel number list num_channels of the TCN network is [52, 52, 52, 72, 72], and then each layer in the num_channels list is traversed. Calculate the dilation rate of each layer, which is 2 to the power of the layer index i. The number of input channels of the first layer is the number of output channels of the GCN network (that is, 52), and the number of input channels of subsequent layers uses the number of output channels of the previous layer. The number of output channels of each layer is set according to the value in the channel number list num_channels. Next, create a temporal block (TemporalBlock) instance and pass in the following parameters: the convolution kernel size kernel_size is 3, the drop rate drop_out is 0.2, the attention mechanism dimension attention_size is 16, as well as the calculated dilation rate and the corresponding padding. The calculation formula is as follows: (5) (6) In the temporal block, the convolutional blocks with normalized weights are combined into a convolutional layer, an activation layer, and a dropout layer. In addition, depending on whether the number of input channels and output channels of each temporal block is the same, a 1x1 convolutional layer may be created for the input. x Perform convolution to adjust the dimension. Add the output of the convolution block to the adjusted input and return the final result through the activation function relu. Use the attention layer after the time block, which contains a linear layer to map the input features to the same number of features as the attention mechanism dimension attention_size, and then use the Tanh activation function and another linear layer to map the number of features to a single output feature for calculating the attention weight, followed by a fully connected layer that maps 72 input features to 10 output features for the final classification result, that is, detecting the corresponding human behavior.

[0029] Step 5, input the test set into the human behavior recognition model in step 4 for classification, and finally obtain the prediction result; Step 5 specifically includes: Step 5.1, input the test set obtained in step 3.3 into the human behavior recognition model obtained in step 4 for testing, realize classification, and finally obtain the prediction result.

[0030] Example 5 like Figure 3 As shown, the human behavior recognition model GTA-net of the present invention is constructed by integrating the graph convolutional network GCN and the time convolutional network TCN based on the attention mechanism; The GCN network contains two convolutional layers, the number of input channels is 90, and the output of the first convolutional layer includes 64 channels. Subsequently, the ReLu activation function is used after the first convolution layer, and the dropout technique is used to implement random inactivation of neurons to prevent overfitting; the number of output channels of the second convolutional layer is 52, and the output of the second convolutional layer is used as the input of the TCN network. The loss function of the GCN network is cross entropy loss, the optimizer is SGD, the learning rate is 7.5e-5, the number of iterations is 500, and the batch size is 16; The TCN network consists of multiple time blocks, attention blocks and fully connected layers. Each time block contains an expanded convolution block. The convolution block is composed of a convolution layer, an activation layer, and a Dropout layer. The number of input and output features of each time block is determined by the channel number list.

[0031] The channel number list of the TCN network is [52, 52, 52, 72, 72], and the input data tensor includes batch size, number of features, and time series length. The batch size, number of features, and time series length are [16, 52, 100] respectively.

[0032] Example 6 Figure 4 (a) and Figure 4 (b) are the confusion matrices of the detection results in indoor and corridor scenes, respectively. It can be observed from Figure 4 (a) that the main diagonal is almost close to 1, but there are still slight recognition errors in the detection of running, picking up, squatting, jumping, drinking water, etc., with an accuracy rate of 99.63%; it can be observed from Figure 4 (b) that there are only slight recognition errors when detecting walking behavior, and all other situations are predicted accurately, with an accuracy rate of 99.83%, which is higher than the indoor environment. This is because the multipath environment in the laboratory is complex, while the corridor environment is relatively open, and the multipath effect is not as good as indoors, so the recognition rate is slightly higher.

[0033] Table 1 Comparison of the accuracy of the training set and test set of the method of the present invention in all scenarios.

[0034]

[0035] Table 2 Comparison results of recognition rates of different data sets under the GTA-net model proposed in this invention.

[0036]

[0037] The above is only one embodiment of the present invention. For those skilled in the art, several improvements and modifications made to the present invention without departing from the principle of the present invention should also be regarded as within the protection scope of the present invention.

Claims

1. A human behavior recognition method combining a graph convolutional network and a temporal convolutional network based on an attention mechanism, characterized in that: Build a multi-antenna single transmitter-single receiver WiFi detection area to collect CSI data of different human behaviors; extract CSI amplitude information and pre-process the CSI amplitude data. First, use Hampel to remove outliers, then use discrete wavelet transform to achieve data denoising, and finally normalize the data; Convert the preprocessed CSI amplitude data into CSI feature maps, and construct and divide the data set; The training set constructed by the CSI feature map is input into the human behavior recognition model GTA-net for training, from which the effective feature information of human behavior is extracted and classified.

2. The human behavior recognition method combining a graph convolutional network and a temporal convolutional network based on an attention mechanism according to claim 1, characterized in that: The specific operations are as follows: Step 1: Build a MIMO-OFDM WiFi detection area with multiple antennas, single transmitter and single receiver, and design experimental scenarios to collect CSI data of different human behaviors; Step 2, extracting CSI amplitude data from the CSI data collected in step 1, and preprocessing the CSI amplitude data; Step 3, converting the CSI amplitude data preprocessed in step 2 into a CSI feature map, constructing and dividing the data set into a training set, a validation set, and a test set; Step 4: Input the CSI feature map converted in step 3 into the human behavior recognition model GTA-net. First, use the graph convolution network GCN to extract the spatial features of the CSI feature map. Then, input the extracted effective features into the temporal convolution network TCN fused with the attention mechanism, and extract the temporal relationship of the features to complete the training of human behavior recognition. Save the model parameters and obtain the optimized human behavior recognition model. Step 5: Input the test set into the human behavior recognition model GTA-net in step 4 for classification, and finally obtain the prediction result.

3. The human behavior recognition method combining a graph convolutional network and a temporal convolutional network based on an attention mechanism according to claim 2, characterized in that: Step 1 is as follows: Step 1.1, build an experimental platform for collecting CSI data. The receiving end and the transmitting end are each equipped with three antennas. The Dell desktop is used as the receiver. It needs to be equipped with an Intel5300 network card and a Linux system, and the Linux 802.11nCSI-Tool tool is configured. The transmitter uses a commercial standard 2.4GHz WiFi b / g / n TP-Link router, and the packet frequency is set to 100Hz. The WiFi signal operating frequency is IEEE 802.11n and 2.4 GHz, and the bandwidth is 40MHz, which can capture the CSI of the complex matrix; Step 1.2: Design and arrange the experimental scene, collect CSI signals of different human behaviors in indoor and corridor scenes under line-of-sight paths, set the distance between the person and the transmitter and receiver to 1 meter, and place the person in the center of the transmitter and receiver.

4. The human behavior recognition method combining a graph convolutional network and a temporal convolutional network based on an attention mechanism according to claim 3, characterized in that: Step 2 specifically includes the following steps: Step 2.1, collect the CSI data of different human behaviors under the line-of-sight path through step 1, obtain a file with the suffix dat, and read the file using the read_bf_file function based on the Matlab software platform; among them, the CSI data is a N A two-dimensional matrix of rows; CSI data matrix It is expressed as: （1） is the number of antennas at the transmitting end, is the number of antennas at the receiving end, The transmitting antenna The first antenna to the receiving antenna The CSI amplitude data of the root antenna, It is expressed as: （2） is the number of subcarriers in the CSI signal, on the Intel 5300 wireless network card is 30; from step 1.1, we know that the receiving end and the transmitting end are each equipped with three antennas, so the received signal contains 9 data streams, and the data contained in each data packet is explained as follows: Step 2.2, select all subcarriers of the three receiving antennas on any transmitting antenna and calculate the original amplitude; where the collected CSI data is a complex matrix, which is expressed as , the CSI amplitude can be obtained by the complex calculation formula , the calculation formula is as follows: （4） Step 2.3, preprocess the CSI amplitude calculated in step 2.

2. First, use the Hampel test to remove outliers. The outliers are identified by calculating the median and median absolute deviation of each data point, and these outliers are replaced by the local median. Then, the discrete wavelet transform is used for denoising. Finally, linear normalization is used to map the data to the standard range of 0 to 255.

5. The human behavior recognition method combining a graph convolutional network and a temporal convolutional network based on an attention mechanism according to claim 4, characterized in that: Step 3 is as follows: Step 3.1, each action is composed of CSI data on multiple transmit and receive antenna links, and the CSI amplitude data on each transmit and receive antenna link is regarded as an independent CSI feature graph; the columns in the CSI feature graph represent the CSI amplitude data on the transmit and receive antenna links, and the rows represent samples collected at different time points; Step 3.2, select one transmitting antenna and three receiving antennas to form three receiving links, combine the CSI feature graphs constructed on the three receiving links, and combine them into a W*30*3 structure, where W is the number of data packets divided by seconds, 100 packets / second; 30 is the number of subcarriers of each receiving link, and 3 is the combination of the amplitudes of the data obtained on the three receiving links, and construct a data set; Step 3.3, divide the data set obtained in step 3.2 according to the ratio of training set: validation set: test set = 7:1.5:1.

5.

6. The human behavior recognition method combining a graph convolutional network and a temporal convolutional network based on an attention mechanism according to claim 5, characterized in that: The human behavior recognition model GTA-net described in step 4 is constructed by integrating the graph convolutional network GCN and the time convolutional network TCN based on the attention mechanism; The GCN network contains two convolutional layers, the number of input channels is 90, and the output of the first convolutional layer includes 64 channels. Subsequently, the ReLu activation function is used after the first convolution layer, and the dropout technique is used to implement random inactivation of neurons to prevent overfitting; the number of output channels of the second convolutional layer is 52, and the output of the second convolutional layer is used as the input of the TCN network. The loss function of the GCN network is cross entropy loss, the optimizer is SGD, the learning rate is 7.5e-5, the number of iterations is 500, and the batch size is 16; The TCN network consists of multiple time blocks, attention blocks and fully connected layers. Each time block contains an expanded convolution block. The convolution block is composed of a convolution layer, an activation layer, and a Dropout layer. The number of input and output features of each time block is determined by the channel number list.

7. The human behavior recognition method combining a graph convolutional network and a temporal convolutional network based on an attention mechanism according to claim 6, characterized in that: The channel number list of the TCN network is [52, 52, 52, 72, 72], and the input data tensor includes batch size, number of features and time series length, and the batch size, number of features and time series length are [16, 52, 100] respectively.

8. The human behavior recognition method combining a graph convolutional network and a temporal convolutional network based on an attention mechanism according to claim 7, characterized in that: Step 4 is as follows: Step 4.1: Use the training set divided in step 3.3 as the input of the human behavior recognition model GTA-net. GTA-net first uses the graph convolutional network GCN to extract the spatial features of the CSI feature map; Step 4.2: Input the spatial features of the CSI feature graph in step 4.1 into the temporal convolutional network TCN fused with the attention mechanism to extract the temporal relationship of the features. The specific extraction method is as follows: First, use the permute() function to adjust the tensor of the input data, then traverse each layer in the list and calculate the expansion rate of each layer; the number of input channels of the first layer is the number of output channels of the GCN network; the number of input channels of subsequent layers uses the number of output channels of the previous layer, and the number of output channels of each layer is set according to the value in the channel number list; Next, create a time block and pass in the following parameters: kernel size kernel_size is 3, dropout rate drop_out is 0.2, attention mechanism dimension attention_size is 16; expansion rate dilation and the corresponding padding padding The calculation formula is as follows: （5） （6） in, i It is the layer index, and its value range is 0~4; According to whether the number of input channels and output channels of each time block is the same, a 1x1 convolution layer is created to convolve the input to adjust the dimension, the output of the convolution block is added to the adjusted input, and activated by the activation function relu; the input features are then mapped to the same number of features as the dimension of the attention mechanism, and then the Tanh activation function is used to map the number of features to a single output feature, the attention weight is calculated, and then the fully connected layer outputs the features to obtain the final classification result, that is, the corresponding human behavior is detected.

9. The human behavior recognition method combining a graph convolutional network and a temporal convolutional network based on an attention mechanism according to claim 8, characterized in that: Step 5 is as follows: Input the test set obtained by step 3.3 into the human behavior recognition model obtained by step 4 for testing, realize classification, and finally obtain the prediction result.

Images