A human behavior recognition method based on fusion attention mechanism

Abstract

The present invention relates to a method for human behavior recognition based on a fusion attention mechanism. A convolutional network module is used to learn the spatial features inside multi-source heterogeneous sensor signals, and the feature vectors of each sensor are fused. The correlation between sensor data is learned through a sensor channel attention module. The output of the convolutional network module is input into a recurrent neural network module to obtain the relationship between time series. A self-attention mechanism is used to capture the key parts in the input sequence, and the input weight of each part is determined. An attention gate is used to filter the noise part, and the part with a higher attention score is retained. Finally, a feature vector of the entire time series is formed to obtain a behavior classification result. The attention mechanism is introduced to separate the intra-sensor relationship, the inter-sensor relationship and the behavior time series relationship under different human behaviors, thereby improving the accuracy of human action classification. Compared with the prior art, the present invention has the advantages of high accuracy, strong interpretability, good extensibility and the like.

Description

Technical Field

[0001] The present invention relates to a human behavior recognition technology, and in particular to a human behavior recognition method based on a fusion attention mechanism. Background Art

[0002] In human behavior recognition technology, more attention is paid to the most frequently performed actions of the human body in daily life, which represent the basic movement units of the human body, such as walking, standing, standing still, running, going up and down stairs, etc. However, the diversity and complexity of behavior have higher requirements for the fine-grained distinction of human behavior. Existing studies mostly use accelerometer sensors to collect user data, but the research of a single sensor is difficult to distinguish the differences in behavior patterns caused by subject differences and inter-class similarities. Subject differences refer to the differences between individual users due to physiological characteristics, such as age and gender; inter-class similarities refer to the behavior patterns with only slight differences in similar behavior categories. For example, when watching TV, different people have very different differentiated behavior patterns; when running and walking, the sensor data of different people show similarities. Due to the influence of the above two major differences, the fine-grained distinction of human behavior is affected to varying degrees by sensors such as accelerometers, gyroscopes, and magnetometers.

[0003] Existing research has mainly explored how to obtain the relationship between sensor modalities from the following perspectives: (1) Early fusion: Initially fuse all data as model input, and obtain recognition results after feature engineering; (2) Sensor fusion: Consider each modality separately, fuse different modalities for classification; or process each sensor axis signal separately. (3) Axis fusion: Design a specific neural network for each univariate time series of each feature, integrate the feature representation of all channels into the final classification network, and eliminate the interference between axes of different sensors. (4) Shared parameter fusion: Based on axis fusion, neural networks of different tasks can use shared parameters to extract common features, and then train unique parameters based on the characteristics of different tasks, allowing for flexible complexity distribution to handle human behaviors of different complexity levels.

[0004] Traditional modal fusion methods are prone to losing the detailed correlation within each modality. Designing models for individual sensor axis signals will ignore the interaction between sensors. It also has high research costs. The accuracy and interpretability of human behavior recognition need to be further improved. Traditional modal fusion methods will mix in other sensor noises that are not the main behavior characteristics during the classification process and produce errors. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a human behavior recognition method based on a fusion attention mechanism, which has the advantages of high accuracy, strong interpretability, and good scalability.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A human behavior recognition method based on a fusion attention mechanism, comprising:

[0008] Collect human posture data obtained by multi-source heterogeneous sensors on wearable devices, form a data set, and perform preprocessing;

[0009] Input the data set into the trained deep behavior recognition model to obtain the human behavior classification results;

[0010] Wherein, the deep behavior recognition model includes:

[0011] The sensor channel attention module is used to extract the functional relationship features between multiple sensor channels;

[0012] The convolutional network module is used to obtain the modal features of each sensor feature channel at the same time according to the functional relationship features between sensor channels;

[0013] A recurrent neural network module is used to extract temporal features based on the time series of modal features;

[0014] The temporal attention module is used to obtain the attention weights at different moments in the same time slice according to the time features, and weight the time features according to the attention weights to obtain the attention features;

[0015] The fully connected layer behavior judgment module is used to obtain the probability distribution of human behavior through the fully connected layer according to the attention characteristics, and take the human behavior with the highest probability as the human behavior classification result within the time slice.

[0016] Furthermore, the specific process of the pretreatment includes:

[0017] Filter the data set;

[0018] The synthetic minority oversampling technique is used to handle the imbalanced data in the dataset;

[0019] The fixed-length sliding window method is used to segment the data set.

[0020] Furthermore, the calculation formula of the synthetic minority oversampling technique is:

[0021] c=a+rand(0,1)*|ab|

[0022] Among them, a is the original sample point data, b is another sample data that is the nearest neighbor to a under the Euclidean distance, c is the sample data obtained by synthetic sampling, and rand(0,1) represents a random number between 0 and 1.

[0023] Furthermore, the specific process of the filtering process includes:

[0024] The dataset was denoised by median filtering and third-order Butterworth low-pass filtering.

[0025] Furthermore, the overlapping degree of the sliding windows in the fixed-length sliding window method is 50%.

[0026] Furthermore, the sensor channel attention module is the SE channel attention module in SENet.

[0027] Furthermore, the convolutional network module is CNN, Resnet or depth-separable convolution.

[0028] Furthermore, the recurrent neural network module is LSTM or GRU.

[0029] An electronic device comprises a memory and a processor, wherein the memory stores a computer program and the processor calls the program instructions to execute the human behavior recognition method.

[0030] A computer-readable storage medium includes a computer program, wherein the computer program can be executed by a processor to implement the human behavior recognition method.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) The present invention uses a convolutional network module to learn the spatial features inside multi-source heterogeneous sensor signals, fuses the feature vectors of each sensor, learns the correlation between sensor data through a sensor channel attention module, inputs the output of the convolutional network module into a recurrent neural network module to learn the temporal relationship, uses a self-attention mechanism to capture the key parts of the input sequence, determines the input weight of each part, uses an attention gate to filter the noise part, retains the part with a higher attention score, and finally forms the feature vector of the entire time series to obtain the behavior classification result. The present invention introduces an attention mechanism to separate the intra-sensor relationship, inter-sensor relationship, and behavior temporal relationship under different human behaviors, which is manifested in the channel and time attention separation of sensor features in the data, thereby improving the classification accuracy of human actions;

[0033] (2) The deep behavior recognition model of the present invention uses two attention modules for the multi-channel features of the sensor to obtain the attention of each sensor channel and the attention on the time series, and separates the channel and spatial changes of the sensor features. It is interpretable, and the behavior recognition for wearable sensor data can reduce the limit on the number of sensors. With interpretable methods such as SHAP, LIME or gradient methods, with the help of channel and time attention between sensors, the cause of classification from the perspective of the model can be better visualized, which can better make people understand the reasons for the classification of the model, save deployment time and cost, and can also study the reasons for misclassified samples;

[0034] (3) The present invention can identify behavior using only wearable sensor devices, without the need for a complex set of tracking equipment, and has good autonomy;

[0035] (4) The present invention can be applied to multi-source heterogeneous sensor data by only modifying the input feature number, without redesigning the network structure, and has good scalability;

[0036] (5) The components of each model structure in the deep behavior recognition model of the present invention can be replaced. For example, the convolution module can be a traditional CNN, Resnet, deep separable convolution, etc., and the loop module can be LSTM or GRU, which has strong development potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the structure of the sensor channel attention module;

[0038] Figure 2 Schematic diagram of the basic structure of a recurrent neural network;

[0039] Figure 3 It is a schematic diagram of the structure of a bidirectional time series recurrent neural network;

[0040] Figure 4 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0041] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0042] Example 1

[0043] A human behavior recognition method based on fusion attention mechanism, such as Figure 4 , including the following steps:

[0044] S1, collect human posture data obtained by multi-source heterogeneous sensors on wearable devices to form a data set;

[0045] S2, preprocessing the data set;

[0046] S3. Input the preprocessed data set into the trained deep behavior recognition model for detecting human behavior to obtain the human behavior classification result.

[0047] In step S1, multi-source heterogeneous sensors are built into the wearable device and placed in the left and right pockets. Built-in accelerometers and built-in gyroscopes are used to collect data samples. The sampling frequency is 50 Hz, 50 sample data per second, and the original data has a total of 6 data channels, namely X-axis acceleration, Y-axis acceleration, Z-axis acceleration, X-axis gyroscope, Y-axis gyroscope, and Z-axis gyroscope.

[0048] Step S2 includes the following sub-steps:

[0049] S201, filtering the data set;

[0050] S202, using synthetic minority oversampling technology to process the unbalanced data in the data set;

[0051] S203, using a fixed-length sliding window method to perform signal segmentation on the data set.

[0052] The specific process of sub-step S201 includes:

[0053] The original data is processed by median filtering to smooth the mutation signals of the six data channels in time. The high-frequency part of the original signal is filtered using a third-order Butterworth low-pass filter after median filtering. The cut-off frequency of the filter is 20 Hz. The formula of the square of the amplitude to frequency of the third-order Butterworth low-pass filter is as follows:

[0054]

[0055] Where n = 3, indicating the use of a third-order Butterworth filter, ω c The cut-off frequency is 20Hz, |H(ω)| 2 ^2 is the value of the passband edge. Another 3rd order low pass Butterworth filter with a frequency of 0.3 Hz is used to separate the XYZ axis accelerometer signal into XYZ axis body acceleration and XYZ axis gravity acceleration.

[0056] The specific process of sub-step S202 includes:

[0057] The synthetic oversampling technique is used on the dataset. The amount of data for different mobile behaviors in the dataset varies, which causes the model results to be biased towards the class with more data. Therefore, it is necessary to process the unbalanced class. The difference between the considered sample and its nearest neighbor is taken to generate a synthetic sample. The difference is multiplied by a random number between 0 and 1 and then added to the considered sample. The calculation formula of the synthetic minority oversampling technique is:

[0058] c=a+rand(0,1)*|ab|

[0059] Among them, a is the original sample point data, b is another sample data that is the nearest neighbor to a under the Euclidean distance, c is the sample data obtained by synthetic sampling, and rand(0,1) represents a random number between 0 and 1.

[0060] The specific process of sub-step S203 includes:

[0061] The fixed-length sliding window method is used to perform signal segmentation on the data set. There are differences in the duration of movement behavior, and a sliding window with an overlap of 50% is used to obtain a more comprehensive information representation. First, the data set processed in step S201 is segmented, with 2.56 seconds (128 data points) as a piece, and a sliding window with a fixed width of 50% data overlap is constructed (128 data per window, 64 data per movement), each piece includes 128 data points, and each data point has 9 channels of data, namely XYZ-axis body acceleration, XYZ-axis gravity acceleration and XYZ-axis gyroscope; secondly, for the sampled data generated in step S202, it is segmented according to its time sequence, and the parameters used for segmentation are consistent with the segmentation parameters of the original data. If the sampled data is less than one data piece, this part of the data is discarded. If the data sequence is not enough to separate 128 data points, this part of the data is discarded.

[0062] In step S3, the deep behavior recognition (AT-HCG Learning) model includes a sensor channel attention module, a convolutional network module, a recurrent neural network module, a temporal attention module and a fully connected layer behavior judgment module.

[0063] Sensor channel attention module: Figure 1The sensor channel attention module uses the SE channel attention module in SENet. The input data is 9 channels and 128*1 feature input. The linear layer (liner) is used to linearly increase the dimension of the 9-channel data. The general dimension is increased to an integer multiple of 8 or 16 to help better obtain the interaction relationship between channels. The global pooling layer (Global Pooling) uses average pooling to encode the single channel feature of the entire time slice into a global feature. FC-1 plays a dimensionality reduction role, reducing the dimension of the dimension-upgraded features, and FC-2 plays a dimensionality increase role, increasing the dimension of the reduced features to the linear layer to generate feature numbers. Sigmoid is used to convert the activation value of the FC-2 layer to the range of 0 to 1, and multiply it with the channel feature value of the linear layer so that the model has the ability to distinguish the features of each channel.

[0064] Convolutional network module: Perform 1*1 convolution on the 128 k-dimensional features output by the sensor channel attention module, where the k dimension is the number of feature outputs of the feature attention mechanism in 4.1 above. For each convolution layer, the 1*1 convolution is point-by-point convolution, and the convolution kernel size is 1*1*k. For the structure between multiple convolution layers, the convolution layers are connected in a CNN manner.

[0065] Recurrent neural network module: For the 128 m-dimensional features generated by the convolutional network module, where the m dimension is the number of feature outputs of the feature convolution module in 4.2 above, the time features are extracted through the recurrent neural network module according to their time sequence. Figure 2 and Figure 3 As shown in the figure, the recurrent neural network module uses a recurrent neural network (RNN) constructed with a bidirectional time series. The recurrent neural network module can be implemented through RNN networks such as LSTM and GRU. The initial transfer weights are all 0. The output features and transfer weights of the network at time t, that is, the behavior category probability, are obtained by the input features at time t and the transfer weights of the RNN at time t-1. For the 128 m-dimensional features generated by the convolution layer, 128 output features with time order are generated according to the time series.

[0066] Temporal attention module: Calculates the 128*6-dimensional human behavior data output by the recurrent neural network module and determines its temporal attention. It uses a fully connected layer and a softmax layer to obtain the attention weights at different times in the same time slice, and multiplies its attention weights with the 128*6-dimensional sensor data to obtain 128*6-dimensional features with attention.

[0067] Fully connected layer behavior judgment module: Based on the 128*6-dimensional features with attention output by the temporal attention module, the final 6-dimensional classification probability is obtained through multiple layers of fully connected layers. The 6-dimensional classification probability is processed into a behavior probability distribution with a sum of 1 using softmax, and the behavior with the largest behavior probability in the probability distribution is taken as the behavior result of this time slice.

[0068] Example 2

[0069] In this embodiment, for each convolution layer in the convolutional network module, the 1*1 convolution is a grouped convolution, and the convolution kernel size is 1*1*(k1+k2+...+kn). The rest is the same as Example 1.

[0070] Example 3

[0071] In this embodiment, for the structure between multiple convolutional layers in the convolutional network module, the layers are connected in a ResNet-like residual form, and the rest is the same as in Embodiment 1.

[0072] Example 4

[0073] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor calls the program instructions to execute the human behavior recognition method described in Example 1.

[0074] Example 5

[0075] A computer-readable storage medium includes a computer program, wherein the computer program can be executed by a processor to implement the human behavior recognition method described in Example 1.

[0076] Embodiment 1, embodiment 2 and embodiment 3 propose a human behavior recognition method, electronic device and computer medium based on a fusion attention mechanism. For wearable sensor data, the inter-modal relationship under the intra-sensor relationship, inter-sensor relationship and time series relationship is studied to explore the deep representation of the original sample data and improve the classification accuracy of fine-grained human behavior. The convolutional network module is used to learn the spatial characteristics of different sensor signals, and the feature vector representations of each sensor are fused to learn the internal relationship of the sensor. The channel attention mechanism is added to learn the axis correlation of sensor data. The output of the network is input into a recurrent neural network to learn the time series relationship. The self-attention mechanism is used to capture the key parts of the input sequence, determine the input weight of each part, use the attention gate to filter the noise part, retain the part with a higher attention score, and finally form the feature vector of the entire time series to obtain the fine-grained recognition division of human behavior.

[0077] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A human behavior recognition method based on fusion attention mechanism, characterized in that: include: Collect human posture data obtained by multi-source heterogeneous sensors on wearable devices, form a data set, and perform preprocessing; Input the data set into the trained deep behavior recognition model to obtain the human behavior classification results; Wherein, the deep behavior recognition model includes: The sensor channel attention module is used to extract the functional relationship features between multiple sensor channels; The convolutional network module is used to obtain the modal features of each sensor feature channel at the same time according to the functional relationship features between sensor channels; A recurrent neural network module is used to extract temporal features based on the time series of modal features; The temporal attention module is used to obtain the attention weights at different moments in the same time slice according to the time features, and weight the time features according to the attention weights to obtain the attention features; The fully connected layer behavior judgment module is used to obtain the probability distribution of human behavior through the fully connected layer according to the attention characteristics, and take the human behavior with the highest probability as the human behavior classification result within the time slice. The sensor channel attention module adopts the SE channel attention module in SENet. The input data is 9 channels and 128*1 size feature input. The linear layer is used to linearly increase the dimension of the 9-channel data to an integer multiple of 8 or 16 to obtain the interaction relationship between channels. The global pooling layer uses average pooling to encode the single channel features of the entire time slice into a global feature. FC-1 is used to reduce the dimension of the features after dimension increase, and FC-2 is used to increase the dimension of the reduced features to the linear layer to generate feature numbers. Sigmoid converts the activation value of the FC-2 layer to the range of 0~1, and multiplies it with the channel feature value of the linear layer to make the model discriminate the features of each channel.

2. According to claim 1, a method for human behavior recognition based on fusion attention mechanism is characterized in that: The specific process of the pretreatment includes: Filter the data set; The synthetic minority oversampling technique is used to handle the imbalanced data in the dataset; The fixed-length sliding window method is used to segment the data set.

3. According to claim 2, a method for human behavior recognition based on fusion attention mechanism is characterized in that: The calculation formula of the synthetic minority oversampling technique is: Among them, is the original sample point data, is another sample data with the nearest neighbor under the Euclidean distance, is the sample data obtained by synthetic sampling, and represents a random number between 0 and 1.

4. According to claim 2, a method for human behavior recognition based on fusion attention mechanism is characterized in that: The specific process of the filtering process includes: The dataset was denoised by median filtering and third-order Butterworth low-pass filtering.

5. The method for human behavior recognition based on fusion attention mechanism according to claim 2 is characterized in that: The overlapping degree of the sliding windows in the fixed-length sliding window method is 50%.

6. The method for human behavior recognition based on fusion attention mechanism according to claim 1, characterized in that: The sensor channel attention module is the SE channel attention module in SENet.

7. The method for human behavior recognition based on fusion attention mechanism according to claim 1, characterized in that: The convolutional network module is CNN, Resnet or depth-separable convolution.

8. The method for human behavior recognition based on fusion attention mechanism according to claim 1, characterized in that: The recurrent neural network module is LSTM or GRU.

9. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and the processor calls the program instructions to execute the human behavior recognition method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: It comprises a computer program, which can be executed by a processor to implement the human behavior recognition method as described in any one of claims 1-8.

Images