Enhanced human body behavior recognition method and system fusing space-time convolution and multi-head attention mechanism

Abstract

The invention discloses an enhanced human body behavior recognition method and system fusing space-time convolution and a multi-head attention mechanism, and the method comprises the steps: collecting a multi-position wearing human body behavior data set, and carrying out the preprocessing, and obtaining the processed data; based on the processed data, constructing a human body behavior recognition model based on a multi-head attention mechanism and a convolutional neural network; and completing human body behavior recognition by using the human body behavior recognition model. According to the method, an innovative multi-head space-time attention module is constructed, and key time sequence features and a space mode of a cross-sensor channel can be adaptively focused, so that the capability of extracting valuable information from complex sensor data is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of human behavior recognition, and in particular to an enhanced human behavior recognition method and system integrating spatiotemporal convolution and multi-head attention mechanism. Background Art

[0002] As one of the core applications of smart wearable technology, wearable human activity recognition (HAR) integrates sensors such as accelerometers, gyroscopes, heart rate monitors, and combines advanced machine learning and deep learning technologies to achieve comprehensive behavior recognition from daily tasks to complex health monitoring. This technology not only helps in daily life assistance, chronic disease management, and rehabilitation training, but also shows great potential in abnormal behavior monitoring, industrial safety, human-machine collaboration, and humanoid robot design.

[0003] According to the wearing position of the sensor, wearable human behavior recognition can be divided into two categories: single-position wearable and multi-position wearable. Single-position wearable sensor systems, such as smartphones or smart watches, are easy to carry, but are mainly suitable for basic functions such as step counting and heart rate monitoring. In contrast, multi-position wearable sensor systems can capture more detailed motion information through sensor arrays distributed throughout key parts of the body, significantly improving the accuracy and reliability of complex behavior pattern recognition, especially in the fields of gait analysis, posture recognition and rehabilitation assessment. The data collected by multi-position wearable sensors are usually presented in the form of matrices or tensors, which include both continuous changes in the time dimension and cross-part linkage in the spatial dimension. Therefore, the effective extraction of spatiotemporal features has become the key to improving the performance of human behavior recognition models. Traditional machine learning models, such as support vector machines, random forests or decision trees, usually rely on domain knowledge and manually extracted features. These models only perform well on specific recognition tasks and often lack the ability to recognize complex behaviors. The emergence and application of deep learning technology has accelerated the development of human behavior recognition. Deep learning models, such as Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM), and Attention Mechanism, can automatically extract features from raw sensor data, capture differences between data, and improve recognition accuracy. LSTMs are good at processing the dependencies of time series data, but the limitation of sequential processing affects processing efficiency and training speed; Convolutional Neural Networks can process the entire input data in parallel, and because of their advantages in strong local feature extraction, parameter sharing, and multi-channel processing capabilities, they are widely used in the process of human behavior recognition. However, classic Convolutional Neural Networks tend to process all input data or features uniformly, which sometimes leads to a lack of differentiation in the contribution of sensor information from different body parts, especially in multi-position wear environments. The attention mechanism solves this problem by allowing the model to dynamically focus on the most relevant parts of the input data for the recognition task. This selective focus enhances the processing of key information while minimizing the interference of irrelevant inputs.

[0004] In order to further optimize the performance of the human behavior recognition model, researchers cleverly combined the attention mechanism with the convolutional neural network, enabling the model to dynamically focus on key data, effectively suppress the interference of irrelevant information, and significantly improve the modeling effect and recognition accuracy of spatiotemporal features. At the same time, the introduction of the multi-head attention mechanism further enhances the robustness of the model to noise and abnormal data, allowing the model to capture a wider range of relevant features, making it more adaptable to inconsistencies in the data. On the other hand, two-dimensional convolutional neural networks also show unique advantages in constructing hierarchical features and gradually parsing behavioral patterns from simple to complex, which is crucial for accurately identifying and classifying complex human behaviors. However, when processing complex sensor data, existing technologies often find it difficult to simultaneously and efficiently capture the temporal dynamics and spatial correlation of data, resulting in limited performance in identifying complex behaviors. Therefore, there is an urgent need for a method that can comprehensively extract features in the temporal dimension and the cross-sensor spatial dimension to improve the accuracy and efficiency of data processing and analysis. Summary of the invention

[0005] To solve the technical problems in the above background, the present invention constructs a model MhsTANet for human activity recognition using multi-position wearable sensors, and utilizes the spatiotemporal multi-head attention mechanism and convolutional neural network to extract and fuse features from time series sensor data in the time and space dimensions to simultaneously and efficiently capture the temporal dynamics and spatial correlation of the data, thereby achieving the purpose of improving the accuracy and efficiency of data processing and analysis.

[0006] To achieve the above object, the present invention provides an enhanced human behavior recognition method integrating spatiotemporal convolution and multi-head attention mechanism, the steps comprising:

[0007] Collect multi-position wearable human behavior data sets and pre-process them to obtain processed data;

[0008] Based on the processed data, a human behavior recognition model based on a multi-head attention mechanism and a convolutional neural network is constructed;

[0009] The human behavior recognition model is used to complete human behavior recognition.

[0010] Preferably, the wearable human behavior data set is preprocessed using a sliding window technique with a window overlap rate of 50% and Min-Max normalization to obtain the processed data.

[0011] Preferably, the constructed human behavior recognition model includes a spatiotemporal attention part and a convolutional neural network part;

[0012] The spatiotemporal attention part is used to extract features from the processed data;

[0013] The convolutional neural network part is used for feature fusion.

[0014] Preferably, the spatiotemporal attention part includes: a temporal multi-head attention module and a spatial multi-head attention module; both the temporal multi-head attention module and the spatial multi-head attention module introduce multi-head attention to emphasize important features and reduce noise;

[0015] The temporal multi-head attention module is used to capture temporal features;

[0016] The spatial multi-head attention module is used to capture the spatial relationship between sensor channels.

[0017] Preferably, the temporal multi-head attention module is designed to capture the temporal dependencies within a single sensor channel; by focusing on key moments in the time series, the temporal multi-head attention module highlights and gives higher weight to important information, which is crucial for accurately identifying behaviors with obvious temporal patterns;

[0018] The spatial multi-head attention module focuses on the spatial dependencies between different sensor channels and identifies the relationships between data from different body parts for understanding complex behaviors involving multiple body parts.

[0019] Preferably, after feature extraction is completed, the extracted features are fused using the convolutional neural network part, and the combined output of the temporal multi-head attention module and the spatial multi-head attention module is fused through a two-dimensional convolution layer, batch normalization, ReLU and maximum pooling layer. The calculation process is expressed as:

[0020] C = MP{Relu[BN(conv2d(A))]}

[0021] Among them, C represents the generated feature map.

[0022] The present invention also provides an enhanced human behavior recognition system integrating spatiotemporal convolution and multi-head attention mechanism, the system is used to implement the above method, including: an acquisition module, a construction module and a recognition module;

[0023] The acquisition module is used to collect multi-position wearable human behavior data sets and perform preprocessing to obtain processed data;

[0024] The building module is used to build a human behavior recognition model based on a multi-head attention mechanism and a convolutional neural network based on the processed data;

[0025] The recognition module is used to complete human behavior recognition using the human behavior recognition model.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention proposes to construct an innovative multi-head spatiotemporal attention module that can adaptively focus on key time series features and spatial patterns across sensor channels, thereby significantly improving the ability to extract valuable information from complex sensor data. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 A schematic diagram of a method flow chart of an embodiment of the present invention;

[0030] Figure 2 The sensor settings and joint angle visualization overview of the MPJA-HAD dataset in an embodiment of the present invention; (a) the wearing position and the joint position from which the angle value is derived; (b) the angle value definition at the neck and left wrist joints in the MPJA-HAD dataset;

[0031] Figure 3 The figure is a visualization overview of the joint rotation direction and joint angle of the MPWHAR dataset in an embodiment of the present invention; (a) is the joint layout; (b) is the definition of the neck angle value in the MPWHAR dataset;

[0032] Figure 4 A schematic diagram of the MhSTANet model structure according to an embodiment of the present invention;

[0033] Figure 5 A schematic diagram of a multi-head attention architecture according to an embodiment of the present invention;

[0034] Figure 6 It is a schematic diagram of the accuracy curve of the MhSTANet model on the PAMAP2, OPPORTUNITY and MHEALTH datasets in an embodiment of the present invention;

[0035] Figure 7 1 is the performance of MhSTANet on the MPWHAR and MPJAHAD datasets in an embodiment of the present invention; wherein, (a) represents the accuracy curve on the MPWHAR dataset; (b) represents the confusion matrix on the MPWHAR dataset; (c) represents the accuracy curve of the MPJA-HAD dataset; (d) represents the confusion matrix of the MPJA-HADA dataset;

[0036] Figure 8The attention output visualization of the temporal multi-head attention module (TA) and the spatial multi-head attention module (SA) on the PAMAP2 dataset in an embodiment of the present invention; (a) is the output visualization of the 0th attention head of TA; (b) is the output visualization of the 1st attention head of TA; (c) is the output visualization of the 0th attention head of SA; (d) is the output visualization of the 1st attention head of SA;

[0037] Fig. 9 The MhSTANet model in the embodiment of the present invention is deployed on Raspberry Pi;

[0038] Fig.10 The figure is a schematic diagram of the result after testing 200 inference times on a Raspberry Pi 4B in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] Embodiment 1

[0042] like Figure 1 FIG. 1 is a schematic diagram of the method flow of this embodiment, and the steps include:

[0043] S1. Collect multi-position wearable human behavior datasets and preprocess them to obtain processed data.

[0044] In this embodiment, the multi-position wearable human activity dataset includes two datasets: MPWHAR (Multi-PositionWearable HumanActivity Recognition dataset) and MPJA-HAD (Multi-Position JointAngle Dataset for HumanActivity).

[0045] In this embodiment, the data sets MPJA-HAD and MPWHAR come from the same data collection process (both collected using wearable sensors). However, they are derived and classified into two different data sets based on different criteria. The data were collected from 9 college student volunteers aged between 18 and 27. During the data collection process, 15 inertial motion capture sensors named CAPTIV (T-SENS MOTION) were used to collect human behavior data. Each inertial motion capture sensor unit includes a 3-axis accelerometer, a 3-axis gyroscope and a 3-axis magnetometer. These inertial motion capture sensors are worn on 15 body parts, including the head, chest, arms, legs, lower back and feet. This embodiment defines six daily activities in the MPJA-HAD and MPWHAR data sets: walking, standing, sitting, lying down, running and skipping rope. Each activity is sampled at a frequency of 32Hz and lasts for 3 minutes.

[0046] like Figure 2 As shown in (a), the specific positions corresponding to the angles of 14 body joints in the MPJA-HAD dataset are shown from the front, marked with circles and numbered from 1 to 14: neck (3), left shoulder (3), right shoulder (3), left elbow (2), right elbow (2), chest (3), left wrist (2), right wrist (2), left hip (3), right hip (3), left knee (2), right knee (2), left ankle (2) and right ankle (2). The numbers in brackets indicate the number of channels for each joint, and the different number of channels reflects the range of motion of different joints. For example, Figure 2 As shown in (b), the neck joint can perform flexion and rotation in two directions, while the movement of the left wrist joint is limited to flexion in two directions. This difference reflects the biomechanical properties and structural limitations of the joints, which are determined by their functional requirements and range of motion capabilities. The joint angle values ​​of each joint are recursively derived from sensors at adjacent positions. The number of angle value channels at each joint is indicated in brackets after the corresponding joint name, so the MPJA-HAD dataset has a total of 34 channels.

[0047] The MPWHAR dataset follows different motion definitions and is described using a 3D digital virtual human model corresponding to the human body, such as Figure 3(a). The dotted circles indicate the positions of the 15 joints measured. Compared to MPJA-HAD, MPWHAR includes an additional joint, the root hip joint, which is a virtual joint that serves as the central reference point of the 3D digital virtual human. Unlike real human joints, all joints of the 3D digital virtual human are not constrained by biological structures and can rotate in three directions. Therefore, each joint has three data channels: X_rotation, Y_rotation, and Z_rotation, and the reference coordinate system (X, Y, Z) is centered on the root hip joint. Therefore, the MPWHAR dataset contains 45 rotation angle channels for 15 joints. Figure 3 (b) shows the definition of neck angle values ​​in the MPWHAR dataset.

[0048] After the multi-position wearable human behavior data set is collected, it is preprocessed to obtain processed data; specifically, the data is preprocessed using a sliding window technique with a window overlap rate of 50%, and Min-Max normalization is applied to improve the network's convergence speed and feature extraction capabilities. This method helps the model more effectively capture local spatial features and temporal changes in behavioral data, enhances feature extraction, and enables the network to better identify dynamic changes in time series data, thereby improving the performance of human behavior recognition tasks.

[0049] S2. Based on the processed data, a human behavior recognition model based on multi-head attention mechanism and convolutional neural network is constructed.

[0050] In this embodiment, the constructed human behavior recognition model is called MhSTANet model, and its overall structure is as follows: Figure 4 As shown in the figure, the MhSTANet model mainly consists of two parts: spatiotemporal attention and convolutional neural network (CNN); the spatiotemporal attention part is used to extract features from processed data, and the convolutional neural network part is used for feature fusion.

[0051] The spatiotemporal attention part mainly consists of two modules: the temporal multi-head attention module (capturing temporal features) and the spatial multi-head attention module (capturing the spatial relationship between sensor channels). Both modules introduce multi-head attention to emphasize important features and reduce noise. Features from these branches are connected and fused through convolutional neural network blocks to capture complex patterns across dimensions.

[0052] The attention mechanism has shown strong capabilities in processing time series and perceiving spatial dimensions, making it very suitable for sensor data processing. In this context, this embodiment applies the attention mechanism to effectively process time series features from various sensor axes and better extract spatial features generated by the interaction between different sensors. The core concepts of the attention mechanism are as follows:

[0053] Q=XW q ,K=XW k ,V=XW v (1)

[0054]

[0055] Among them, * represents the transposition operation, the input data First, by combining three learnable weight matrices and Multiply them to perform linear transformation into query vector (Q), key vector (K) and value vector (V); d k Represents the dimension size of the embedding space; H represents the vector height and W represents the vector width; Represents the field of real numbers.

[0056] The multi-head attention architecture in the MhSTANet model is as follows Figure 5 As shown in the figure, N represents the number of attention heads. In each attention head, the input sample data X is transformed into Q after the corresponding linear transformation in formula (1). i , K i and V i , which are both two-dimensional tensors Each attention head independently calculates the attention score and weighted sum to capture the characteristics of the input data from different angles. Then the outputs of these attention heads are concatenated and linearly transformed to obtain the final output:

[0057] h i =Attention(Q i ,K i ,V i ) (3)

[0058] O=f(Q i ,K i ,V i )=Concat(h1,...,h N )W o (4)

[0059] in, is the output transformation matrix, and the width of the N attention heads after splicing is Nd k . You can adjust W o The dimension of the output The dimension size of , to facilitate subsequent feature fusion and other operations; D represents the latitude after connection.

[0060] like Figure 4As shown in the figure, in MhSTANet, two different attention mechanisms are applied to effectively extract spatiotemporal features from the processed data. These two attention mechanisms are implemented by two parallel multi-head attention modules. The temporal multi-head attention module (TA) aims to capture the temporal dependencies within a single sensor channel. By focusing on the key moments in the time series, the temporal multi-head attention module can highlight and give higher weights to important information, which is crucial for accurately identifying behaviors with obvious temporal patterns. The spatial multi-head attention module (SA) focuses on the spatial dependencies between different sensor channels. It aims to identify the relationships between data from different body parts, which is critical for understanding complex behaviors involving multiple body parts. The input of the temporal multi-head attention module is transposed as the input of the spatial multi-head attention module, allowing MhSTANet to analyze behavioral data from a spatial perspective.

[0061] One of the time windows into which the original data is divided after preprocessing It can be expressed as:

[0062] Where S represents the number of sensor channels and T represents the time step. First, before X is input into the temporal multi-head attention module and the spatial multi-head attention module, it is processed into X t =X T and X S =X T Then, they are respectively passed through two independent multi-head attention operations to obtain the output of the temporal multi-head attention module and the spatial multi-head attention module. The calculation process is as follows:

[0063]

[0064] in, Represents the output of the temporal multi-head attention module and the spatial multi-head attention module respectively; tensor and tensors is done by applying the respective learnable matrix to the input; and is the output transformation matrix; d k represents the dimension of the embedding space; N represents the number of attention heads.

[0065] The output of the spatial multi-head attention module is transposed to ensure that the output of the temporal multi-head attention module and the spatial multi-head attention module have the same dimension size in the time step when the feature is concatenated. After the feature is concatenated, a linear transformation is applied. This process also adds an extra dimension to prepare for the next step of feature fusion based on convolutional neural networks. The expression of this process is:

[0066] A=Linear{Concat[Ot ,(O s ) * ]}(8)

[0067] Among them, * transpose operation, D is the concatenated dimension; A is the combined output of temporal and spatial multi-head attention, which effectively integrates the features extracted from the two dimensions.

[0068] After feature extraction is completed, the convolutional neural network is used to fuse the extracted features. Figure 4 As shown in Figure 1, the combined output A of the temporal and spatial multi-head attention modules passes through a two-dimensional convolutional layer (Conv2d), batch normalization (BN), ReLU, and a maximum pooling layer (Maxpooling). The calculation process can be expressed as:

[0069] C=MP{Relu[BN(conv2d(A))]}(9)

[0070] Where C represents the feature map generated after the above operations. This feature fusion step integrates the temporal and spatial features extracted by the multi-head attention mechanism, enabling the model to effectively capture complex patterns in the data.

[0071] S3. Use the human behavior recognition model to complete human behavior recognition.

[0072] Embodiment 2

[0073] The stability and superiority of the model of the present invention will be described in detail below in conjunction with this embodiment.

[0074] This embodiment uses five data sets to test and evaluate the model of the present invention. Specifically including: MPWHAR, MPJA-HAD, PAMAP2, OPPORTUNITY and MHEALTH. Among them, the MPWHAR data set and the MPJA-HAD data set are collected through wearable sensors, while PAMAP2, OPPORTUNITY and MHEALTH are three public multi-position wearable human behavior recognition data sets. The difference from the public multi-position wearable human behavior recognition data sets (PAMAP2, OPPORTUNITY, MHELTH) is that MPWHAR and MPJA-HAD represent human behavior through joint angle data. Table 1 summarizes the detailed information of these data sets, including the number of subjects, behavior categories, sampling frequency, and number of sensor channels.

[0075] Table 1

[0076]

[0077] The testing and evaluation of the MhSTANet model were implemented on a server equipped with an NVIDIA RTX 3090 GPU using the PyTorch framework. The above datasets are divided into training and test sets in a ratio of 7:3. To improve the efficiency of the experiment, the samples are input into the model in batches, with a batch size of 64. The model uses the Adam optimizer and the cross entropy loss function to update the training parameters and minimize the loss. The maximum number of iterations is set to 100 and the learning rate is set to 0.0001.

[0078] This embodiment uses four indicators: Precision, Recall, F1-score and Accuracy to conduct a comprehensive performance evaluation of the model. The formulas of these indicators are as follows:

[0079]

[0080] Wherein, TP, FN, FP and TN represent true positive examples, false positive examples, false negative examples and true negative examples, respectively. In addition, this embodiment uses parameter quantity and FLOPs (floating point operations) to measure the complexity and computational efficiency of the model.

[0081] Table 2

[0082]

[0083] Table 2 shows the experimental results of MhSTANet when the number of attention heads is 1, 2, 4, and 8 on five datasets: OPPORTUNITY, MHEALTH, PAMAP2, MPWHAR, and MPJA-HAD. The results in the table show that when the number of attention heads is 4, MhSTANet exhibits the highest recognition accuracy, reaching 97.78% on PAMAP2, 91.71% on OPPORTUNITY, 99.78% on MHEALTH, 99.97% on MPWHAR, and 99.85% on MPJA-HAD. Although the performance is still high when the number of attention heads is 8, the recognition accuracy is still slightly reduced. The test results show that when the number of attention heads is 4, it provides the best balance between accuracy and performance for MhSTANet, so the number of attention heads is set to 4 in subsequent experiments.

[0084] Table 3 shows the accuracy, precision, recall, and F1-score of MhSTANet on the PAMAP2, OPPORTUNITY, MHEALTH, MPWHAR, and MPJA-HAD datasets. The average and standard deviation of each metric across three runs are reported. As shown in Table 3, MhSTANet performs well on all datasets, with the average accuracy, precision, recall, and F1-score on the PAMAP2 dataset being approximately 97.60%, reflecting strong stability and reliability. On the OPPORTUNITY dataset, the accuracy and recall of MhSTANet exceed 91.67%, demonstrating its adaptability to complex multi-sensor data. For the MHEALTH dataset, the model performs well in all metrics, approaching 99.66%, highlighting its superior performance in behavior recognition. The results on the MPWHAR dataset show that all four metrics are close to 99.89%. Similarly, on the MPJA-HAD dataset, the average of the model reaches around 99.82%, further verifying the robustness and generalization of MhSTANet in different sensor datasets. Overall, the excellent performance of MhSTANet on five datasets confirms its effectiveness in processing multi-channel and multi-sensor data.

[0085] Table 3

[0086]

[0087] Figure 6 The accuracy curves of the validation set of MhSTANet trained for 100 generations on the PAMAP2, OPPORTUNITY, and MHEALTH datasets are shown. It can be seen that MhSTANet achieves fast convergence, converging after about 20 generations on the PAMAP2 and MHEALTH datasets, and converging after about 50 generations on the OPPORTUNITY dataset. This demonstrates the learning efficiency and adaptability of MhSTANet on various datasets. The results confirm that the model can achieve consistent high performance on different datasets within a relatively short training period.

[0088] Figure 7 The performance of MhSTANet on the datasets MPWHAR and MPJA-HAD collected in this embodiment is shown, and the accuracy curves of the training set and the validation set and the confusion matrix on the test dataset are shown. The accuracy curve shows that convergence is achieved in only 5 generations, which is faster than Figure 6 The convergence speed on the three public datasets shown is much faster. Therefore, the curve shows that MhSTANet has higher stability after convergence on the datasets MPWHAR and MPJA-HAD collected in this embodiment. Figure 7 (a) and Figure 7As shown in (c), the close agreement between the accuracy curves of the training set and the validation set indicates excellent model performance, with MPWHAR and MPJA-HAD having the least overfitting. Figure 7 The confusion matrices in (b) and (d) show that the number of correctly predicted samples for the six activities on both datasets is very high, with only very few samples being incorrectly predicted.

[0089] In order to determine the contribution of different modules in the proposed MhSTANet model to its overall performance, ablation experiments were conducted. This involves systematically removing or changing key modules and observing the impact on the model performance. The results are detailed in Table 4, showing the impact of different modules in MhSTANet on several different performance metrics on the PAMAP2, OPPORTUNITY, MHEALTH, MPJA-HAD, and MPWHAR datasets. The temporal multi-head attention module (TA) alone achieved moderate performance on all datasets, with the highest accuracy of 96.59% on the MHEALTH dataset and the lowest accuracy of 88.49% on PAMAP2. The performance of the spatial attention module (SA) alone was slightly lower than that of TA. Combining TA and SA (TA+SA) improved the performance, with an accuracy of up to 97.36% on MHEALTH. Further integration with convolutional neural networks, such as SA+CNN and TA+CNN, improved the accuracy, with TA+CNN achieving 99.36% accuracy on the MHEALTH dataset. The complete model MhSTANet (TA+SA+CNN) performed best and achieved the highest accuracy on all datasets, reaching 99.78% and 99.85% recognition accuracy on the MHEALTH and MPJA-HAD datasets, respectively.

[0090] Table 4

[0091]

[0092] Precision values ​​follow a similar trend to accuracy, with TA+SA+CNN consistently outperforming the combination. While TA and SA individually perform moderately, their combination (TA+SA) and their integration with CNN significantly improve precision. Recall also highlights the superior performance of the ensemble of TA+SA+CNN, achieving the highest recall across all datasets. TA and SA individually perform poorly in capturing true positives, but their integration with CNN significantly improves recall. The F1-score reflects the overall balance between precision and recall, with TA+SA+CNN having the highest F1-score, indicating its robustness and effectiveness in balancing precision and recall. TA+CNN and SA+CNN also perform well, but not as well as the combined performance of TA+SA+CNN. Regardless of the model configuration, consistently high performance (over 99.80%) is demonstrated on both the MPWHAR and MPJAHAD datasets, which can be attributed to the high quality of the datasets and their relevance to the human action recognition task.

[0093] The ablation results show that each module of the MhSTANet model (TA, SA, and CNN) contributes greatly to it. When TA, SA, and CNN are integrated together, they have the best performance, highlighting the importance of combining temporal and spatial attention mechanisms with convolutional neural networks to enhance feature extraction in human action recognition tasks. This integration enables MhSTANet to achieve excellent performance on a variety of datasets, thus confirming its robustness and generalization.

[0094] In order to further explore the role of TA and SA, this example Figure 8 In the paper, the attention output of the two attention heads on the PAMAP2 dataset is visualized. In this example, the attention output heat map of two attention heads is used as an example for analysis. Figure 8 In (a) and (b), the TA output shows rich and diverse temporal features between different heads, indicating that TA is able to emphasize the key time steps in the sequence and fully capture the temporal dependencies. Figure 8 (c) and (d) show the output of SA, which focuses on the spatial dependencies between different sensor channels. The heatmaps show less diversity, indicating that SA focuses on fewer but more important spatial relationships. These observations are consistent with the results of ablation studies, where combining TA and SA can improve performance because TA provides extensive temporal features while SA adds critical spatial information. The fusion of these features produces a more robust and comprehensive feature representation, thereby improving the overall performance of the MhSTANet model.

[0095] In order to evaluate the computational efficiency of MhSTANet, this embodiment calculates the number of parameters and FLOPs (floating point operations), as shown in Table 5. The accuracy of MhSTANet on the PAMAP2 dataset is as high as 97.60%, while maintaining significantly lower model parameters (0.17M) and FLOPs (1.98M). The computational cost using the MHEALTH, OPPORTUNITY, MPWHAR, and MPJA-HAD datasets is also kept at a low level. Overall, Table 5 shows that MhSTANet is highly efficient and is very suitable for real-time human behavior recognition on mobile devices.

[0096] Table 5

[0097]

[0098] In order to evaluate the real-time deployment performance, this example deploys MhSTANet on a resource-limited Raspberry Pi 4B motherboard equipped with a 1.5GHz quad-core Cortex-A72 64-bit processor. Fig. 9 As shown in FIG. 1 , this embodiment deploys the MhSTANet model trained on the PAMAP2 dataset onto a Raspberry Pi device. Window data is input into the Raspberry Pi for behavior recognition. The remote desktop interface displays the recognized behavior category, classification probability, and inference time for real-time monitoring. Fig.10 The inference time statistics for 200 randomly selected samples are shown, and the inference time for most behaviors is clustered around 29.53 milliseconds.

[0099] Embodiment 3

[0100] This embodiment also provides an enhanced human behavior recognition system that integrates spatiotemporal convolution and multi-head attention mechanism, including: an acquisition module, a construction module and a recognition module; the acquisition module is used to collect multi-position worn human behavior data sets and preprocess them to obtain processed data; the construction module is used to construct a human behavior recognition model based on the multi-head attention mechanism and convolutional neural network based on the processed data; the recognition module is used to use the human behavior recognition model to complete human behavior recognition.

[0101] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. An enhanced human behavior recognition method integrating spatiotemporal convolution and multi-head attention mechanism, characterized in that the steps include: Collect multi-position wearable human behavior data sets and pre-process them to obtain processed data; Based on the processed data, a human behavior recognition model based on a multi-head attention mechanism and a convolutional neural network is constructed; The human behavior recognition model is used to complete human behavior recognition.

2. The enhanced human behavior recognition method integrating spatiotemporal convolution and multi-head attention mechanism according to claim 1 is characterized in that: The wearable human behavior data set is preprocessed using a sliding window technique with a window overlap rate of 50% and Min-Max normalization to obtain the processed data.

3. The enhanced human behavior recognition method integrating spatiotemporal convolution and multi-head attention mechanism according to claim 1 is characterized in that: The constructed human behavior recognition model includes a spatiotemporal attention part and a convolutional neural network part; The spatiotemporal attention part is used to extract features from the processed data; The convolutional neural network part is used for feature fusion.

4. The enhanced human behavior recognition method integrating spatiotemporal convolution and multi-head attention mechanism according to claim 3 is characterized in that: The spatiotemporal attention part includes: a temporal multi-head attention module and a spatial multi-head attention module; both the temporal multi-head attention module and the spatial multi-head attention module introduce multi-head attention to emphasize important features and reduce noise; The temporal multi-head attention module is used to capture temporal features; The spatial multi-head attention module is used to capture the spatial relationship between sensor channels.

5. The enhanced human behavior recognition method integrating spatiotemporal convolution and multi-head attention mechanism according to claim 4 is characterized in that: The temporal multi-head attention module aims to capture the temporal dependencies within a single sensor channel; By focusing on key moments in the time series, the temporal multi-head attention module highlights and gives higher weight to important information, which is crucial for accurately identifying behaviors with obvious temporal patterns; The spatial multi-head attention module focuses on the spatial dependencies between different sensor channels and identifies the relationships between data from different body parts for understanding complex behaviors involving multiple body parts.

6. The enhanced human behavior recognition method integrating spatiotemporal convolution and multi-head attention mechanism according to claim 4 is characterized in that: After feature extraction is completed, the extracted features are fused using the convolutional neural network part. The combined output of the temporal multi-head attention module and the spatial multi-head attention module is fused through a two-dimensional convolutional layer, batch normalization, ReLU, and maximum pooling layer. The calculation process is expressed as: C = MP{Relu[BN(conv2d(A))]} Among them, C represents the generated feature map.

7. An enhanced human behavior recognition system integrating spatiotemporal convolution and multi-head attention mechanism, the system is used to implement the method according to any one of claims 1 to 6, characterized in that: include: Acquisition module, construction module and recognition module; The acquisition module is used to collect multi-position wearable human behavior data sets and perform preprocessing to obtain processed data; The building module is used to build a human behavior recognition model based on a multi-head attention mechanism and a convolutional neural network based on the processed data; The recognition module is used to complete human behavior recognition using the human behavior recognition model.

Images