Fast activity recognition method based on multi-path parallel MLP mixer architecture

Abstract

This invention discloses a fast activity recognition method based on a multi-path parallel MLP mixer architecture. First, multi-dimensional sequence data is used as input. An embedding module encodes the multi-dimensional data to obtain a feature sequence. This feature sequence is then fed into multiple MLP mixer branches, where features are mixed along the time, channel, and frequency domain dimensions, respectively. In each branch, after passing through multiple MLP mixers, global pooling is used to aggregate the feature sequence along the time dimension into a feature vector. Finally, the feature vectors obtained from each branch are concatenated and sent to a classification head for activity recognition. Compared with existing deep learning-based activity recognition methods, this invention achieves more accurate activity recognition with fewer model parameters, less floating-point computation, and faster inference speed, meeting the requirements of high precision and high efficiency in practical applications.

Description

Technical Field

[0001] This invention belongs to the field of activity recognition technology, specifically relating to a fast activity recognition method based on a multi-path parallel MLP mixer architecture. Background Technology

[0002] Sensor-based activity recognition aims to identify the activity state of an active subject, such as a human, animal, bird, or other active entity, based on data collected from various sensors including accelerometers, gyroscopes, and magnetometers. With the rapid advancement of sensor technology and the Internet of Things (IoT), wearable sensor-based activity recognition has found widespread application, such as Human Activity Recognition (HAR) in healthcare and smart homes, and animal and bird activity recognition for monitoring and protection. Human activity recognition, in particular, has become a research hotspot. Sensor-based activity recognition methods typically segment sensor data into time-series segments and classify the time-series data into predefined activity categories, such as running, jumping, walking, and sitting. Due to differences in sensors, the presence of noise signals, activity categories, and variations in the measured object, activity recognition remains a challenging research topic. Various existing machine learning algorithms, such as SVM, KNN, and random forests, have been used for activity recognition and have achieved relatively good performance. However, these shallow learners require feature engineering on the sensor data, which relies on domain-specific knowledge. Later, deep learning was widely applied to sensor-based activity recognition, as it can automatically learn feature representations. Various architectures of convolutional neural networks (CNNs) and hybrid models of CNNs and recurrent neural networks (RNNs) have been proposed and proven to improve the accuracy of activity recognition.

[0003] Recently, research has focused on the attention mechanism of deep learning-based HAR models, which can help the model focus on more useful information in features, thereby further improving recognition accuracy. Inspired by the success of Transformer and “Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Advances in neural information processing systems, 2017, 30.” in natural language processing and computer vision, the paper “Saif Mahmud, MTTonmoy, Kishor Kumar Bhaumik, AMRahman, MAAmin, M.Shoyaib, Muhammad Asif Hossain Khan, and A.Ali. Human activity recognition from wearable sensor data using self-attention. In ECAI 2020-24th European Conference on Artificial Intelligence, 29 August-8” proposes a self-attention-based neural network that combines sensor modal attention and global temporal attention.

[0004] These deep learning-based activity recognition methods are continuously improving their accuracy. However, their models are typically complex and time-consuming, making it difficult to meet the high efficiency requirements of real-world applications, especially online activity recognition applications that require real-time feedback. Therefore, it is necessary to develop high-efficiency and high-accuracy deep activity recognition models. Summary of the Invention

[0005] To address the technical problems existing in the aforementioned deep learning-based activity recognition methods, this invention proposes a fast activity recognition method based on a multi-path parallel MLP mixer architecture.

[0006] The technical solution adopted in this invention is: a fast activity recognition method based on a multi-path parallel MLP mixer architecture, the specific steps of which are as follows:

[0007] S1. Take multidimensional sequence data as input and encode the multidimensional data through an embedding module to obtain a feature sequence;

[0008] S2. Send the feature sequence to multiple MLP mixer branches to mix the features along the time, channel, and frequency domain dimensions, respectively;

[0009] S3. In each branch, after passing through multiple MLP mixers, global pooling is used to aggregate the feature sequences into feature vectors along the time dimension.

[0010] S4. The feature vectors obtained in step S3 are concatenated and sent to a classification head for activity recognition.

[0011] Furthermore, step S1 is specifically as follows:

[0012] The multidimensional sequence data acquired by the sensor is truncated into a window of length L as input, and an embedding module is used to encode the multidimensional data at each time step to obtain a feature sequence. It serves as the input for multiple subsequent parallel MLP mixer branches.

[0013] in, Let C represent the real number field, and let C represent the length of the feature.

[0014] The MLP mixer used is a position-based fully connected network, with each fully connected element acting independently at each position of the last dimension of the feature tensor. Each branch of the MLP mixer has a similar structure.

[0015] Furthermore, step S2 is specifically as follows:

[0016] In the time dimension branch, MLP takes each column (channel) of the feature matrix X as independent input and mixes the features from different times in each channel.

[0017] In the channel dimension branch, MLP takes each row (time step) of the feature matrix X as independent input and mixes the features from different channels at each time step.

[0018] In the frequency domain branch, there are several ways to construct frequency domain features: perform a one-dimensional fast Fourier transform along the time or channel dimension, or perform fast Fourier transforms in two dimensions successively, and then use the real part, imaginary part, magnitude or other quantities after the Fourier transform as the frequency domain feature Z, which has the same shape as X; MLP can mix frequency domain features along the time dimension or channel dimension.

[0019] Furthermore, in step S2, the combination of MLP mixer branches and the adjustment of the number of MLP mixers are as follows:

[0020] The choice of which branches to combine depends on the difficulty of the task and the efficiency requirements, including time-based, channel-based, or one or more frequency domain branches.

[0021] The number of MLP mixers stacked on the branch is determined by the difficulty of the task and the efficiency requirements.

[0022] The beneficial effects of this invention are as follows: The method of this invention first takes multidimensional sequence data as input, encodes the multidimensional data through an embedding module to obtain a feature sequence, then sends the feature sequence to multiple MLP mixer branches to mix features along the time, channel, and frequency domain dimensions respectively. In each branch, after passing through multiple MLP mixers, global pooling is used to aggregate the feature sequence along the time dimension into a feature vector. Finally, the feature vectors obtained from each branch are concatenated and sent to a classification head for activity recognition. Compared with existing deep learning-based activity recognition methods, the method of this invention achieves more accurate activity recognition with fewer model parameters, floating-point operations, and faster inference speed, meeting the requirements of high precision and high efficiency in practical applications. Attached Figure Description

[0023] Figure 1 This is a flowchart of a fast activity recognition method based on a multi-path parallel MLP mixer architecture according to the present invention.

[0024] Figure 2 This is a diagram of the activity recognition neural network structure of a three-way parallel MLP mixer in an embodiment of the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] like Figure 1 The flowchart of a fast activity recognition method based on a multi-path parallel MLP mixer architecture is shown below. The specific steps are as follows:

[0027] S1. Take multidimensional sequence data as input and encode the multidimensional data through an embedding module to obtain a feature sequence;

[0028] S2. Send the feature sequence to multiple MLP mixer branches to mix the features along the time, channel, and frequency domain dimensions, respectively;

[0029] S3. In each branch, after passing through multiple MLP mixers, global pooling is used to aggregate the feature sequences into feature vectors along the time dimension.

[0030] S4. The feature vectors obtained in step S3 are concatenated and sent to a classification head for activity recognition.

[0031] In this embodiment, step S1 is specifically as follows:

[0032] like Figure 2As shown, the multidimensional sequence data collected by the sensor is truncated into a window of length L as input. The embedding module first linearly encodes the data at each time step, and then adds position embedding (PE) to obtain a feature sequence. It serves as the input for multiple subsequent parallel MLP mixer branches.

[0033] in, Let C represent the real number field, and let C represent the length of the feature.

[0034] The MLP used in this embodiment is also a position-based fully connected network, with each fully connected layer acting independently at each position of the last dimension of the feature tensor. The MLP mixer in each branch has the same structure, which includes two position-based fully connected layers (linear transformation layers), a nonlinear activation function, residual connections, and layer normalization (LayerNorm).

[0035] The formal representation of an MLP mixer is as follows:

[0036] Y out =LayerNorm(Y in +ρ(Y in W1)W2)

[0037] Among them, Y in and Y out Let W1 and W2 represent the input and output feature tensors of the MLP, respectively, which have the same dimension. W1 and W2 represent the weight matrices of two linear transformations, and ρ represents the nonlinear activation function, which is GELU in this embodiment. This method of performing transformations first and then layer normalization is called PostNorm, and vice versa is called PreNorm.

[0038] In this embodiment, step S2 is specifically as follows:

[0039] like Figure 2 As shown, this embodiment uses an activity recognition neural network with a three-way parallel MLP mixer to mix features along the time, channel, and frequency domain dimensions.

[0040] In the time dimension branch, the input feature matrix X is first transposed to obtain X. T Y, as the input of the first MLP mixer in this branch in The MLP mixer blends features from different moments; it operates independently on each channel and shares parameters across all channels. For all MLP mixers in this branch, there are... and L′ represents the adjustable width of the hidden layer.

[0041] In the channel dimension branch, the MLP mixer mixes the characteristics of different channels. It runs independently at each time step but shares parameters across all time steps. For the first MLP mixer in this branch, set Y... in =X. For all MLP mixers in this branch, there are and C′ represents the adjustable hidden layer width.

[0042] In the frequency domain branch, a one-dimensional Fast Fourier Transform (FFT) is performed along the channel dimension, and the real part of the Fourier transform result is taken as the feature:

[0043] Z = R(f c (X))

[0044] Where R represents a function taking the real part of a complex number, f c This represents a one-dimensional fast Fourier transform along the channel.

[0045] In the frequency domain branch, the MLP mixer mixes the frequency domain characteristics of different channels. For the first MLP mixer, Y is set... in =Z. For all MLP mixers in this branch, the shape of the weight matrix is ​​the same as that of the channel branch.

[0046] The number N of MLP mixers stacked on the branch is determined by the difficulty of the subsequent experimental tasks.

[0047] In this embodiment, step S3 is specifically as follows:

[0048] Global average pooling is used to aggregate feature sequences into feature vectors along the time dimension.

[0049] In this embodiment, step S4 is specifically as follows:

[0050] The activity recognition classification head includes a fully connected layer (FC Layer) and a Softmax operation.

[0051] This embodiment compares the method of the present invention with existing methods on four commonly used public HAR datasets (including PAMAP2, OPPORTUNITY, USC-HAD, and SKODA). The comparison methods include: a one-dimensional convolutional network TCN along the time dimension, a network DeepConvLSTM combining CNN and LSTM, a self-attention-based model SANet, a dual-attention model DanHAR, and a triple-attention model TriCDA.

[0052] The dataset is as follows:

[0053] The PAMAP2 (Activity Monitoring of Older Adults 2) dataset was collected from nine participants performing 12 daily activities (including “walking,” “lying down,” and “standing”) and exercises (including “watching TV,” “using a computer,” “working,” and “driving a car”). Three inertial measurement units (IMUs), accelerometers, gyroscopes, and magnetometers were placed on each participant's hand, chest, and ankle, and raw sensor data was collected at 100 Hz. See: “Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th international symposium on wearable computers, pages 108-109. IEEE, 2012.” This data includes labeled human activities performed by nine subjects, and the dataset contains 12 categories of human activities.

[0054] The OPPORTUNITY dataset comprises labeled data from wearable and environmental sensors for four subjects, used to simulate public kitchen activities. Each subject was asked to perform five natural kitchen activities and one “practice” activity, for a total of 24 activities. Sensor recordings were collected using inertial sensors placed at 12 body positions. (See: Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Kilian...) Gerhard Paul Lukowicz, David Bannach, Gerald Pirkl, Alois Ferscha, et al. Collecting complex activity datasets in highly rich networked sensor environments. In 2010 Seventh International Conference on Networked Sensing Systems (INSS), pages 233-240. IEEE, 2010. For fair comparison, this example uses a subset from a recent Opportunity Challenge, which includes 17 intermediate activity classes and one empty activity class, at a sampling rate of 30 Hz.

[0055] The USC-HAD dataset contains six readings acquired from accelerometer and gyroscope sensors via motion node devices. This dataset was created by seven male and seven female subjects with well-defined body specifications and ages. See: “MiZhang and Alexander A Sawchuk. Usc-had: A daily activity dataset for ubiquitous activity recognition using wearable sensors. In Proceedings of the 2012 ACM conference on ubiquitous computing, pages 1036-1043, 2012.” The sensor data was sampled at 100Hz, and each time step in the dataset is labeled with one of 12 activity classes. Due to the varying sensor placement and activity classes, USC-HAD is a challenging dataset.

[0056] The SKODA dataset is a dataset that tracks 10 operational gestures / activities performed in automotive maintenance scenarios. See also: "Thomas Stiefmeier, Daniel Roggen, Georg Ogris, Paul Lukowicz, and Gerhard..." Wearable activity tracking in car manufacturing. IEEE Pervasive Computing, 7(2):42-50, 2008. This dataset contains accelerometer readings taken from different locations on a subject's arm and labeled as one of 10 activity categories or an empty category, with a sampling rate of approximately 98 Hz.

[0057] In the training-test split settings of the four datasets mentioned above, this embodiment constructs the test set by merging the validation set and test set used in the SANet work, making the evaluation more challenging than the settings in SANet.

[0058] Since the datasets involved in the experiment have different sampling frequencies, adjusting the frequency through downsampling helps in reasonable performance comparisons. In this embodiment, PAMAP2, USC HAD, and SKODA were downsampled to close to 30Hz, consistent with the OPPORTUNITY dataset.

[0059] For HAR tasks, segmenting the sensor time series into a series of windows, with fixed window lengths and overlapping sliding windows being widely used methods. The window size is the sample sequence length considered when constructing the classification feature representation. The label for the activity type is determined by majority voting on the samples within the window. To date, there is still no clear consensus in academia regarding the optimal window length for various HAR tasks. Online activity recognition focuses on real-time data collection and activity identification, which is essential for certain applications (e.g., patient monitoring). Therefore, this embodiment uses a small window length in experiments to quickly identify activities. A 1-second window size was used for PAMAP2, OPPORTUNITY, and USC-HAD, and a 1.5-second window size was used for SKODA, all with 50% overlapping sliding windows.

[0060] For the PAMAP2, OPPORTUNITY, and SKODA datasets, this embodiment stacks two MLP mixers on each branch, while for USC-HAD, it stacks three MLP mixers. The feature dimension is set to 96, and the number of neurons in the hidden layer of the MLP mixer is set to four times its input.

[0061] This embodiment trains the model by minimizing the cross-entropy loss using mini-batch gradient descent, with a batch size of 128. The Adam optimizer with a cosine annealing learning rate is used, and the initial learning rate is set to 0.001. All experiments are implemented using the PyTorch framework and Python, and run on a server equipped with an Intel Xeon E5-2678 v3 CPU, 128GB of RAM, and an NVIDIA RTX2080 Ti GPU.

[0062] This embodiment uses the macro-average F1 score and accuracy as metrics to evaluate the activity identification method. The macro-average F1 score (or macro F1 score) is the arithmetic mean of the F1 scores for all activity classes. It can be represented as follows:

[0063]

[0064] Where C represents the number of activity categories, Precision i and Recall i Let represent the precision and recall for the i-th activity category, respectively. This metric treats all categories equally, regardless of the number of instances. The precision formula is as follows:

[0065]

[0066] Where N represents the number of test samples, f(x) i ) indicates that the activity recognition model is applied to sample x. iPredicted activity category labels, y i Indicates the true label, This indicates an indicator function that returns 1 if the predicted label matches the true label, and 0 otherwise.

[0067] The method of the present invention was then evaluated on four public datasets, compared with existing methods in terms of F1 score and accuracy, and the results are presented in Table 1.

[0068] Table 1

[0069]

[0070] The method of this invention was compared with existing technologies on four public datasets using F1 score (%, first term) and accuracy (%, second term). Best results are marked in bold, and second-best results are marked in underline.

[0071] As shown in Table 1, the method of this invention significantly outperforms the second-best method in F1 score and accuracy on the OPPORTUNITY, USC-HAD, and SKODA datasets, especially on the two challenging datasets, OPPORTUNITY and USC-HAD. In PAMAP2, the method of this invention achieves an F1 score of 99.89% and an accuracy of 99.90%, both very close to the best performance. Among existing methods, TriCDA and DanHAR are leading methods, achieving good performance on PAMAP2, USC-HAD, and SKODA, but their performance on the OPPORTUNITY dataset is still inferior to that of simple TCN.

[0072] This embodiment also compares the model size and inference speed. The most commonly used metrics for comparing the size of different models are the number of model parameters and floating-point operations (FLOPs). The number of parameters and FLOPs of the model of the present invention and the comparison model are shown in Table 2, and the inference time (milliseconds) is shown in Table 3.

[0073] Table 2

[0074]

[0075] The table includes data on model parameters (first item) and FLOPs (second item). Best results are marked in bold, and second-best results are marked with an underline.

[0076] Table 3

[0077] Model PAMAP2 OPPORTUNITY USC-HAD SKODA TCN 11.9 13.7 14.1 14.0 DeepConvLSTM 1.3 1.3 1.3 1.7 SANet 2.6 2.9 2.6 2.5 DanHAR 5.0 4.7 4.9 4.2 TriCDA 5.7 6.0 5.8 5.6 Ours <![CDATA[ 2.0 ]]> <![CDATA[ 2.2 ]]> <![CDATA[ 2.9 ]]> <![CDATA[ 2.1 ]]>

[0078] The numbers in the table represent inference time (milliseconds). The best result is marked in bold, and the second-best result is marked in underline.

[0079] Based on the experimental results in Tables 2 and 3, the self-attention-based model SANet and the model proposed in this invention have significantly fewer parameters and fewer FLOPs than existing state-of-the-art methods TriCDA and DanHAR, with inference time only about half that of the latter. On different datasets, the model proposed in this invention uses only 16%–37% of the parameters and 6%–69% of the FLOPs of these two models.

[0080] Further ablation experiments were conducted on the challenging OPPORTUNITY and USC-HAD datasets to investigate the effectiveness of each component in the embodiments of the present invention, and some components were removed from the network to verify their respective contributions. First, a model with only time and channel dimension branches was implemented, including versions with pre-normalization (PreNorm) without positional embedding (PE), post-normalization (PostNorm) without PE, and PostNorm plus PE (i.e., the approach in this embodiment). Results show that using PostNorm and PE improves performance. Finally, a frequency domain branch was added to obtain a three-branch version, and the results were compared between performing Fast Fourier Transform (FFT) in the time dimension and in the channel dimension. Results show that adding the frequency domain branch further improves performance, and performing FFT in the channel dimension yields better results.

[0081] The impact of different components of the model is shown in Table 4, where the data are F1 score (%, first item) and accuracy (%, second item).

[0082] Table 4

[0083] Model OPPORTUNITY USC-HAD Time+Channel(PreNorm w\o PE) 66.59,90.52 56.58,56.36 Time+Channel(PostNorm w\o PE) 67.86,91.23 59.70,58.54 Time+Channel 68.61,91.28 62.34,61.15 Time + Channel + Frequency(time) 68.82,91.35 64.02,63.93 Time+Channel+Frequence(channel) 69.18,91.74 67.44,70.14

[0084] In this embodiment, the TCN is specifically referenced from: Bai S, Kolter JZ, Koltun V. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 1803.01271, 2018.

[0085] For details on DeepConvLSTM, please refer to: Francisco Javier and Daniel Roggen. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors,16(1):115,2016.

[0086] For SANet, see specifically: Saif Mahmud, M.T.H. Tonmoy, Kishor Kumar Bhaumik, A.M. Rahman, M.A. Amin, M. Shoyaib, Muhammad Asif Hossain Khan, and A. Ali. Human activity recognition from wearable sensor data using self-attention. In ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August - 8 September 2020, Santiago de Compostela, Spain, 2020.

[0087] For DanHAR, see specifically: Wenbin Gao, Lei Zhang, Qi Teng, Jun He, and Hao Wu. Danhar: Dual attention network for multimodal human activity recognition using wearable sensors. Applied Soft Computing, 111: 107728, 2021.

[0088] For TriCDA, see specifically: Yin Tang, Lei Zhang, Qi Teng, Fuhong Min, and Aiguo Song. Triple cross - domain attention on human activity recognition using wearable sensors. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(5): 1167 - 1176, 2022.

[0089] In summary, to address the limitations of existing deep learning-based activity recognition methods in terms of limited accuracy and high computational complexity, this invention first segments the sequence data collected by sensors before feeding the segmented data into the network of this invention for rapid activity recognition. The method is evaluated on four public benchmark HAR datasets. Experimental results show that the proposed method achieves more accurate activity recognition with fewer model parameters, less floating-point operations, and faster inference speed. Therefore, the method of this invention better meets the requirements of high precision and high efficiency in practical applications.

[0090] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A fast activity recognition method based on a multi-path parallel MLP mixer architecture, the specific steps of which are as follows: S1. Take multidimensional sequence data as input and encode the multidimensional data through an embedding module to obtain a feature sequence; S2. Send the feature sequence to multiple MLP mixer branches to mix the features along the time, channel, and frequency domain dimensions, respectively; In the time dimension branch, MLP uses the feature matrix Each column is used as independent input, and features from different times on each channel are mixed; In the channel dimension branch, MLP uses the feature matrix Each line is used as independent input, and features from different channels at each time step are mixed. In the frequency domain branch, there are several ways to construct frequency domain features: performing a one-dimensional Fast Fourier Transform (FFT) along the time or channel dimension, or performing FFTs sequentially in two dimensions, and then using the real part, imaginary part, magnitude, or other quantities obtained from the Fourier transform as frequency domain features. It and They have the same shape; MLP blends frequency domain features along the time dimension or channel dimension; S3. In each branch, after passing through multiple MLP mixers, global pooling is used to aggregate the feature sequences into feature vectors along the time dimension. S4. The feature vectors obtained in step S3 are concatenated and sent to a classification head for activity recognition.

2. The fast activity recognition method based on a multi-path parallel MLP mixer architecture according to claim 1, characterized in that, The specific steps of S1 are as follows: The multidimensional sequence data acquired by the sensor is truncated into a window of length L as input, and an embedding module is used to encode the multidimensional data at each time step to obtain a feature sequence. As input to multiple subsequent parallel MLP mixer branches; in, Let C represent the real number field, C represent the length of the feature, and L represent the length of the time window. The MLP mixer used is a position-based fully connected network, with each fully connected element acting independently at each position of the last dimension of the feature tensor. Each branch of the MLP mixer has a similar structure.

3. The fast activity recognition method based on a multi-path parallel MLP mixer architecture according to claim 1, characterized in that, In step S2, the combination of MLP mixer branches and the adjustment of the number of MLP mixers are as follows: The choice of which branches to combine depends on the difficulty of the task and the efficiency requirements; these branches can be time-based, channel-based, or based on one or more frequency domains. The number of MLP mixers stacked on the branch is determined by the difficulty of the task and the efficiency requirements.

Images