Human activity recognition method based on feature fusion and attention mechanism

Abstract

The application discloses a human activity recognition method based on feature fusion and attention mechanism, wearing an embedded device containing a gyroscope and an accelerometer on the waist of a human body, obtaining a sensor data set and performing data preprocessing; a neural network model composed of a feature fusion module, an attention module and a classifier module is constructed; the feature fusion module contains a gyroscope convolutional neural network model and an accelerometer convolutional neural network model introducing a DenseNet multi-scale feature fusion principle, and the gyroscope and accelerometer feature vectors output by the two models are input into the attention module after position coding, and finally the classifier outputs the probability of each activity mode, and the activity mode with the maximum probability is taken as the recognition result of the model; the trained neural network model is used for human activity recognition. The application has high recognition accuracy and enhances the robustness of the neural network model in human activity recognition application.

Description

Technical Field

[0001] This invention belongs to the field of human activity recognition, and uses data generated by accelerometer and gyroscope sensors during activity to identify the ongoing activities of the human body. Background Technology

[0002] With the widespread application of deep learning, human activity recognition technology based on wearable devices has been greatly improved and has been widely used in healthcare, sports and fitness, smart homes, and many other fields. However, on the one hand, most existing algorithms ignore the feature differences of heterogeneous sensors and different depths of convolutional layers in deep convolutional networks, resulting in insufficient extraction of spatial features from sensor data and low activity recognition accuracy. On the other hand, due to the shortcomings of conventional recurrent neural networks in extracting features from long-term sequences, the models fail to extract sufficient contextual features from sensor data, resulting in low accuracy in recognizing complex activities.

[0003] Feature fusion refers to jointly modeling multiple datasets that differ in properties and features to generate a new dataset. Feature fusion can extract the most differentiated information from multiple original feature sets, eliminating model performance degradation caused by differences in the properties of the original feature sets, as well as redundant information caused by the similarity between the original feature sets. Research on human activity recognition often utilizes different sensors, such as accelerometers, gyroscopes, and magnetometers. Data collected by different sensors contains information with different focuses, and this information can often complement each other to obtain more advantageous features. Therefore, feature fusion from different sensors achieves better results than using a single sensor, improving the accuracy of activity recognition.

[0004] Currently, most models used for human activity recognition are convolutional neural networks (CNNs) or recurrent neural networks (RNNs). CNNs have significant advantages in extracting features from the original data space, but deep CNNs also have drawbacks. In image recognition, in deep CNNs, the receptive field becomes clearer with increasing convolutional layers, and semantic features become clearer, but image resolution also gradually decreases. After multiple layers of convolutional operations, many detailed features become blurred. Shallow neural networks have small receptive fields and strong ability to express detailed features, but at the same time, their semantic meaning is weaker. This also applies to sensor-based human activity recognition, which also uses deep convolutional networks. Therefore, how to efficiently integrate high-level and low-level features, taking advantage of their strengths and discarding their weaknesses, is of great significance for improving the performance of deep CNNs and increasing the accuracy of activity recognition.

[0005] Convolutional neural networks (CNNs) can effectively utilize receptive fields to extract local features from text sequences and concatenate these key local information to obtain the semantic information of the entire sequence. However, CNNs can only extract spatial information from text sequences, and the size of the convolutional kernel can limit their ability to extract text sequence features. In contrast, recurrent neural networks (RNNs) can extract temporal features from text sequences, but they suffer from the vanishing gradient problem on long text sequences, severely limiting their feature extraction capabilities. Furthermore, RNNs operate sequentially, resulting in long training times and unidirectional feature extraction. Transformers offer significant improvements to address these issues. They abandon traditional feature extraction methods and fully employ a self-attention mechanism for feature extraction. By encoding the input data at positions, they effectively solve the problems of parallelization and unidirectional feature extraction, fully extracting the global dependencies between vectors. Currently, sensor data is mostly long sequences; applying multi-head attention mechanisms to human activity recognition can effectively extract the contextual relationships within the sequence, improving activity recognition accuracy. Summary of the Invention

[0006] In order to overcome the shortcomings of the existing technology in human activity recognition, which is insufficient in extracting the spatial and temporal features of the original data and thus has low recognition accuracy, this invention provides a human activity recognition method based on feature fusion and attention mechanism. This method first performs feature fusion on heterogeneous sensor data to initially extract the spatial features in the data, then extracts the global dependencies of the data through a multi-head attention model, and finally inputs the feature vector extracted by the model into a classifier for human activity recognition.

[0007] To address the aforementioned technical problems, this invention proposes a human activity recognition method based on feature fusion and attention mechanisms. This method involves wearing an embedded device containing multiple sensors, including a gyroscope and an accelerometer, on the waist of the body. The method includes the following steps:

[0008] Step 1: Obtain the dataset and perform data preprocessing, including:

[0009] During human activity, triaxial data from gyroscopes and accelerometers are collected in real time, and the data are labeled with activity patterns to form a dataset. Missing and outlier values ​​in the dataset are removed, and polynomial interpolation is used to unify the frequency of the data. The data is divided into sliding windows, and the activity patterns of the collected objects in the same window are the same. Zero-mean standardization is used to standardize the data and reduce the differences in the units of measurement between the data.

[0010] Step 2, Construction of the Neural Network Model

[0011] The neural network model comprises a feature fusion module, an attention module, and a classifier module connected in series; wherein:

[0012] 2-1) The feature fusion module includes a gyroscope convolutional neural network model and an accelerometer convolutional neural network model, used to extract spatial features from the preprocessed gyroscope and accelerometer data. The gyroscope and accelerometer convolutional neural networks incorporate the multi-scale feature fusion principle of the DenseNet network. The outputs of the gyroscope and accelerometer convolutional neural network models are respectively the gyroscope feature vector F. A and accelerometer eigenvector F B ;

[0013] 2-2) The multi-head attention module includes three sub-models and one feedforward neural network model. The three sub-models are the gyroscope multi-head attention sub-model, the accelerometer multi-head attention sub-model, and the convolutional fusion sub-model.

[0014] The multi-head attention module will use the gyroscope feature vector F A and accelerometer eigenvector F B First, the vector undergoes layer normalization, then trigonometric function position encoding, and finally embeds positional information to obtain the feature vector F. A’ and eigenvector F B’ ; respectively the feature vector F A’ and eigenvector F B’ The number of vectors is divided in a 2:1 ratio, where the feature vector F A’ Two-thirds of the data is fed into the gyroscope multi-head attention sub-model, and the feature vector F B’ Two-thirds of the data is fed into the multi-head attention sub-model of the accelerometer; the feature vector F A’ and eigenvector F B’ The remaining 1 / 3 of the data is concatted and then input into the convolutional fusion sub-model; the output data of the three sub-models are concatted on each channel to obtain the fused feature vector E. f1 ;

[0015] Fusion feature vector E f1 With eigenvector F A’ and eigenvector F B’ Perform a residual operation and input the result into the feedforward neural network model; the output of the feedforward neural network model is then multiplied by the feature vector F. A’ and eigenvector F B’ After performing residual operations, the output of the feedforward neural network is the feature vector E. f2 ;

[0016] 2-3) The aforementioned feature vector E f2 After layer normalization, the data is input into the classifier, which outputs the probability of each activity mode. The activity mode with the highest probability is taken as the result of model recognition.

[0017] Step 3: Train the neural network model and optimize the network parameters.

[0018] Input the preprocessed dataset from step one into the neural network model constructed in step two to obtain the final recognition result, compare it with the label after one-hot encoding, and apply the loss function and gradient descent method to optimize the neural network model to obtain the optimal parameters of the neural network model.

[0019] Step four: Use the trained neural network model to recognize human activities.

[0020] An embedded device containing a gyroscope and an accelerometer is worn on the waist of the human body to collect data in real time. The collected data is preprocessed in real time according to the preprocessing method described in step one and then input into the trained neural network model for activity recognition. The neural network model outputs the probability of each activity recognition and takes the highest probability as the activity recognition result.

[0021] Compared with the prior art, the beneficial effects of the present invention are:

[0022] This invention constructs models for gyroscope and accelerometer data separately and introduces the DenseNet feature fusion principle to fuse shallow and deep features of convolutional neural networks at multiple scales, fully capturing the spatial features of gyroscope and accelerometer data. Simultaneously, by introducing a multi-head attention mechanism, it fully extracts the global dependencies between gyroscope and accelerometer data. This invention can effectively extract spatial features and global dependencies beneficial to activity recognition from gyroscope and accelerometer data. The accuracy for recognizing complex activities is improved by approximately 10% and 5% compared to a single traditional convolutional neural network model and a hybrid network model of convolutional neural networks and recurrent neural networks, respectively. It also enhances the robustness of neural network models in human activity recognition applications. Attached Figure Description

[0023] Figure 1 This is a diagram showing the overall structure of the neural network model constructed in this invention;

[0024] Figure 2 This is a flowchart of the neural network model training process in this invention;

[0025] Figure 3 This is a schematic diagram of the accelerometer convolutional neural network model in this invention;

[0026] Figure 4This is a schematic diagram of the gyroscope convolutional neural network model in this invention;

[0027] Figure 5 This is a graph showing the impact of the sliding segmentation window size on recognition accuracy. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the following embodiments are by no means intended to limit the present invention.

[0029] This invention proposes a human activity recognition method based on feature fusion and attention mechanisms, such as... Figure 1 As shown, firstly, an embedded device containing multiple sensors, including a gyroscope and an accelerometer, is worn on the waist of the human body. The method mainly includes: acquiring sensor datasets and performing data preprocessing; constructing a neural network model consisting of a feature fusion module, an attention module, and a classifier module connected in series; training the neural network, optimizing network parameters, and finally using the trained neural network model to recognize human activities. The specific details of each step are described below:

[0030] Step 1: Acquire sensor datasets and perform data preprocessing.

[0031] 1-1) Eighteen activity patterns were selected as labels for the experimental data. These included simple daily activities such as "standing," "sitting," and "lying down," as well as complex and transitional activities such as "playing ping-pong," "push-ups," "sitting-standing," and "lying-sitting." A Xiaomi 10s was used as the data collector, with the sensor frequency set to 100Hz. Real-time data was collected from the gyroscope and accelerometer during human activity. Each action lasted for more than one minute, and 3000 samples were collected for each action. The activity patterns for each time period were recorded, and the collected data were labeled using the activity patterns as tags.

[0032] 1-2) Remove missing and outlier values ​​from the dataset. Remove irrelevant data from the first and last seconds of each activity sequence. Then, use polynomial interpolation to unify the data to 100Hz.

[0033] 1-3) Divide the data into sliding windows, determine the size of the sliding window, and ensure that the activity patterns of the objects collected within the same window are the same. In this embodiment, the window size is set to 300, meaning that each time window contains 3 seconds of sensor data (e.g., Figure 5 As shown in the image, the window overlap rate is set to 50%.

[0034] 1-4) Zero-difference normalization was used to standardize the data range, reducing the dimensional differences between data points. After this process, a total of 207,500 subsamples were generated. 60% of the data was used as the training set, 20% as the validation set, and 20% as the test set to obtain the input dataset.

[0035] Step 2, Neural Network Model Construction

[0036] like Figure 1 As shown, the neural network model constructed in this invention consists of a feature fusion module, an attention module, and a classifier module connected in series. The feature fusion module comprises a gyroscope convolutional neural network model (DeA-HAR) and an accelerometer convolutional neural network model (DeG-HAR) arranged side-by-side to extract spatial features from the preprocessed gyroscope and accelerometer data. The attention module consists of a gyroscope multi-head attention sub-model, an accelerometer multi-head attention sub-model, a convolutional fusion sub-model, and a feedforward neural network model. The classifier module mainly consists of fully connected layers and a softmax function.

[0037] The gyroscope and accelerometer data processed in step one are then... The structure is input into the gyroscope convolutional network model and the accelerometer convolutional network model for spatial feature extraction. 16 represents the batch size of the dataset as 16, 300 represents the dataset window size as 300, and 3 represents that each sensor of the gyroscope and accelerometer has three axes: x, y, and z.

[0038] 2-1) The gyroscope convolutional neural network model and the accelerometer convolutional neural network model adopt the DenseNet network structure and introduce the multi-scale feature fusion principle of the DenseNet network to fuse the low-level features extracted by the low-level convolutional neural network and the high-level features extracted by the high-level convolutional neural network. The gyroscope feature vector F is extracted by the gyroscope convolutional neural network and the accelerometer convolutional neural network respectively. A and accelerometer eigenvector F B .

[0039] In this embodiment, the accelerometer convolutional network model is as follows: Figure 3As shown, it mainly consists of three convolutional layers and an intermediate batch normalization layer. Each convolutional layer uses a one-dimensional convolution with a kernel size of 3 and a kernel count of 16. The input to each convolutional layer is the result of concatenating the outputs of the previous convolutional layers and the initial input data of the model. In the convolutional layers, the padding parameter is set to 'same' to ensure that the output of each convolutional layer has the same values ​​in all dimensions except for the number of channels. The model input is... The structure outputs the feature vector as:

[0040]

[0041] In this embodiment, the formula for the concat operation is as follows. Where is the fused feature vector, and x... i and y i Each feature vector has its own channel, k is the convolution kernel corresponding to each channel, * represents the convolution operation, and C is the number of channels.

[0042]

[0043] The specific data flow process is as follows: After preprocessing, the accelerometer sensor data generates feature vector F1. Feature vector F1 undergoes operation C1 to generate feature vector F2. Feature vector F2 and F1 are concatenated and then processed by C2 to generate feature vector F3. Feature vector F3 is then concatenated with F1 and F2, processed by C3 to generate feature vector F4. F4 is further processed by C4 to generate feature vector F5. F5 is the feature vector F extracted from the accelerometer sensor data. A .

[0044] The gyroscope convolutional neural network model mentioned above, such as Figure 4 As shown, it consists of 5 convolutional layers and an intermediate batch normalization layer. Each convolutional layer uses a one-dimensional convolution with a kernel size of 3 and a kernel count of 32. The model input is... The output is 2.

[0045] The signal processing of a gyroscope sensor is similar to that of an accelerometer. The input signal C1 is the preprocessed feature vector F1 of the original signal. The feature vectors input to C2, C3, C4, and C5 are the fusion of the feature vectors output from all previous bottleneck layers and F1, respectively. C1 is a one-dimensional convolutional layer that performs initial feature extraction from the original signal. C2, C3, C4, and C5 are bottleneck modules. Unlike the bottleneck in the accelerometer convolutional neural network model, the output of each convolutional layer in a gyroscope is a feature vector with 32 channels. The output of the gyroscope convolutional neural network model is the feature vector F1. B .

[0046] The output F of the feature fusion module A and F B As input to the attention module.

[0047] 2-2) The multi-head attention module includes three sub-models and one feedforward neural network model. The three sub-models are the gyroscope multi-head attention sub-model (MSA). gyr ), Multi-head attention sub-model of accelerometers (MSA) acc The system employs a multi-head attention sub-model for gyroscopes and a multi-head attention sub-model for accelerometers to extract global dependency features between tokens within a time window from data from each sensor. The multi-head attention sub-model for gyroscopes and accelerometers is used to extract spatial correlation features between the gyroscopes and accelerometers, and the feedforward neural network model is used to further extract deep semantic information from the data.

[0048] The multi-head attention module will use the gyroscope feature vector F A and accelerometer eigenvector F B First, layer normalization is applied, then positional encoding using trigonometric functions is performed to embed positional information, resulting in the feature vector F. A’ and eigenvector F B’ .

[0049] In this embodiment, the output vectors of the gyroscope and accelerometer models are encoded using trigonometric functions, as shown in the formula below. `pos` is the position index of the token in the sequence, `i` represents the dimension, and `PE` represents the feature vector at that position. Trigonometric function position encoding assigns positional information to the data.

[0050]

[0051] The feature vector FA 'and eigenvector F B The numbers are divided in a 2:1 ratio, where the feature vector F A Two-thirds of the data is fed into the gyroscope multi-head attention sub-model, and the feature vector F B Two-thirds of the data is fed into the multi-head attention sub-model of the accelerometer; the feature vector F A 'and eigenvector F B The remaining 1 / 3 of the data is concatted and then input into the convolutional fusion sub-model; the output data of the three sub-models are concatted on each channel to obtain the fused feature vector E. f1 .

[0052] The gyroscope multi-head attention sub-model and the accelerometer multi-head attention sub-model use the following attention function. Here, Q, K, and V are all vectors of dimension d. Each key-value pair is weighted and summed to output vector Q. The weight r(Q,K) in the formula is calculated based on the compatibility function.

[0053] Attention(Q,K,V)=r(Q,K)×V. (3)

[0054]

[0055] Fusion feature vector E f1 With eigenvector F A 'and eigenvector F B Perform residual operations and input the results into the feedforward neural network model; the feedforward neural network model consists of two fully connected layers and one dropout layer, and ReLU is selected as the activation function. The output of the feedforward neural network model is then multiplied by the feature vector F. A 'and eigenvector F B Perform residual operations; the output of this feedforward neural network model is the feature vector E. f2 .

[0056] 2-3) The fused feature vector E f2 After layer normalization, the data is input into the classifier module. The classifier module classifies activities based on the features extracted from all previous networks. The classifier outputs the probability of each activity mode, and the activity mode with the highest probability is taken as the result of model recognition.

[0057] In this embodiment, the classifier mainly consists of a fully connected layer and a normalization function Softmax. Its main function is to identify 18 activities and output the identification probability of each activity. The Softmax formula is shown below, where Zi is the output value of the i-th node and C is the number of output nodes.

[0058]

[0059] The classifier outputs the probability of each activity mode, and the activity mode with the highest probability is taken as the result of model recognition.

[0060] Step 3: Train the neural network, iterate and optimize the model.

[0061] The preprocessed dataset collected in step one is input into the neural network model constructed in step two to obtain the final recognition result. This result is then compared with the one-hot encoded labels. The loss function and gradient descent method are applied to optimize the neural network model, yielding its optimal parameters. Figure 2 As shown, the specific process includes:

[0062] 3-1) Obtain the sensor dataset and preprocess it.

[0063] 3-2) Initialize the parameters of the model created in step 2, setting the maximum loss value to L' and the maximum number of iterations (epochs) of the model.

[0064] 3-3) Input the gyroscope data and accelerometer data obtained in step one into the gyroscope convolutional neural network model and the accelerometer convolutional neural network model respectively, and start model training.

[0065] 3-4) Obtain a unique output for each data sample input.

[0066] 3-5) Calculate the loss value using the cross-entropy loss function. The formula for the cross-entropy loss function is as follows. Where y is the actual label value after one-hot encoding (1 for positive classes, 0 for negative classes). It is the predicted probability value (y·∈(0,1)), which represents the difference between the true sample label and the predicted probability.

[0067]

[0068] 3-6) Determine the magnitude of the loss value L and L'. If L is greater than L', continue with step 3-3 if the model iteration count has not reached the epoch. If the epoch has been reached, store the model parameters and end the training. If the loss value L is less than L', save the model parameters and stop the training.

[0069] Step four: Use the trained neural network model to recognize human activities.

[0070] An embedded device containing a gyroscope and an accelerometer is worn on the waist of the human body. The sampling frequency of the gyroscope and accelerometer is set to 300Hz to collect data in real time. The collected data is preprocessed in real time (according to the preprocessing method in step one above) and then input into the trained neural network model for activity recognition. The neural network model outputs the probability of each activity recognition and takes the highest probability as the activity recognition result.

[0071] Although the present invention has been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many modifications under the guidance of the present invention without departing from the spirit of the present invention, and these modifications are all within the protection scope of the present invention.

Claims

1. A method for human activity recognition based on feature fusion and attention mechanisms, comprising wearing an embedded device containing multiple sensors on the waist of the human body, said sensors including a gyroscope and an accelerometer, characterized in that, The method includes the following steps: Step 1: Obtain the dataset and perform data preprocessing, including: 1-1) During human activity, real-time data from the gyroscope and accelerometer are collected, and the data is tagged with the activity mode. 1-2) Remove missing and outlier values ​​from the dataset and use polynomial interpolation to unify the frequency of the data; 1-3) Divide the data into sliding windows, and ensure that the activity patterns of the objects being collected within the same window are the same; 1-4) Zero-mean standardization is used to standardize the data and reduce the differences in units between data. Step 2, Construction of the Neural Network Model The neural network model comprises a feature fusion module, a multi-head attention module, and a classifier module connected in series; wherein: 2-1) The feature fusion module includes a gyroscope convolutional neural network model and an accelerometer convolutional neural network model, used to extract spatial features from the preprocessed gyroscope and accelerometer data. The gyroscope and accelerometer convolutional neural networks incorporate the multi-scale feature fusion principle of the DenseNet network. The outputs of the gyroscope and accelerometer convolutional neural network models are respectively the gyroscope feature vector F. A and accelerometer eigenvector F B ; 2-2) The multi-head attention module includes three sub-models and one feedforward neural network model. The three sub-models are the gyroscope multi-head attention sub-model, the accelerometer multi-head attention sub-model, and the convolutional fusion sub-model. The multi-head attention module will use the gyroscope feature vector F A and accelerometer eigenvector F B First, the vector undergoes layer normalization, then trigonometric function position encoding, and finally embeds positional information to obtain the feature vector F. A’ and eigenvector F B’ ; respectively the feature vector F A’ and eigenvector F B’ The number of vectors is divided in a 2:1 ratio, where the feature vector F A’ Two-thirds of the data is fed into the gyroscope multi-head attention sub-model, and the feature vector F B’ Two-thirds of the data is fed into the multi-head attention sub-model of the accelerometer; the feature vector F A’ and eigenvector F B’ The remaining 1 / 3 of the data is concatted and then input into the convolutional fusion sub-model; the output data of the three sub-models are concatted on each channel to obtain the fused feature vector E. f1 ; Fusion feature vector E f1 With eigenvector F A’ and eigenvector F B’ Perform a residual operation and input the result into the feedforward neural network model; the output of the feedforward neural network model is then multiplied by the feature vector F. A’ and eigenvector F B’ After performing residual operations, the output of the feedforward neural network is the feature vector E. f2 ; 2-3) The aforementioned feature vector E f2 After layer normalization, the data is input into the classifier, which outputs the probability of each activity mode. The activity mode with the highest probability is taken as the result of model recognition. Step 3: Train the neural network model and optimize the network parameters. Input the preprocessed dataset from step one into the neural network model constructed in step two to obtain the final recognition result, compare it with the label after one-hot encoding, and apply the loss function and gradient descent method to optimize the neural network model to obtain the optimal parameters of the neural network model. Step four: Use the trained neural network model to recognize human activities. An embedded device containing a gyroscope and an accelerometer is worn on the waist of the human body to collect data in real time. The collected data is preprocessed in real time according to the preprocessing method described in step one and then input into a trained neural network model for activity recognition. The neural network model outputs the probability of each activity recognition and takes the highest probability as the activity recognition result.

Images