Method for optimizing human activity recognition based on multi-branch space-time interaction network

Abstract

A method for optimizing human activity recognition based on a multi-branch space-time interaction network comprises the following steps: (1) data preprocessing: processing wearable sensor data to generate synthetic diffusion data; (2) feature extraction: establishing a deep convolutional network model, namely a multi-branch space-time interaction network, and performing feature extraction on the diffusion data to obtain space-time feature data; (3) training the model: introducing loss function configuration, distributing different weights for each loss function, adjusting the weights of the loss functions according to precision indexes when processing a big data set, and updating parameters by using an Adam optimizer; and (4) identifying the spatial-temporal characteristic data in the step (2) through the model in the step (3), and finally obtaining a classification result of each motion. Through the structure, the invention provides an efficient and accurate method for optimizing human activity identification based on the multi-branch space-time interaction network.

Description

Technical Field

[0001] The present invention belongs to the field of human activity recognition, and in particular relates to a method for optimizing human activity recognition based on a multi-branch spatiotemporal interaction network. Background Art

[0002] Human activity recognition refers to the automatic recognition of various human behaviors, including sitting, walking, running, hand movements, etc., by analyzing the interaction with the environment. This recognition is crucial for applications such as security, elderly care, and behavior prediction. It merges physical activities with the behavioral context to create a context-aware framework for accurate recognition. Advances in wearable sensors and mobile devices have driven the development of human activity recognition systems, thereby improving the accuracy of classifying activities such as walking, jogging, and sitting.

[0003] Deep learning can automatically extract complex features from massive amounts of biomedical data without the need for manual feature engineering, enhancing its practicality in areas such as genomics, drug discovery, and medical imaging, thereby maximizing the use of this data. In addition, deep learning models can perform multi-level nonlinear feature transformations through representation learning, overcoming the limitations of large data sets and achieving groundbreaking results in areas such as healthcare, education, and security. The rapid adoption of this technology in the field of health informatics is due to its predictive power, its ability to obtain semantic interpretations from input data, and its ability to automatically optimize high-level features, showing its potential for application in various fields in the future.

[0004] Deep learning has significantly enhanced human action recognition by enabling automatic high-level feature extraction, thereby optimizing performance in various fields such as healthcare, surveillance, and daily activity monitoring. Deep neural networks extract complex features from massive data, improving the classification accuracy of human action recognition systems. The fusion of deep learning techniques such as convolutional neural networks and gated recurrent units allows for precise feature extraction and comprehensive model architectures, resulting in high classification accuracy on various datasets. In addition, a perception extraction network consisting of three convolutional blocks and a relational network is proposed to explore global relations and local features in human activity recognition. By effectively coordinating 3D-CNN, 2D-CNN, and 1D-CNN, an integrated architecture is proposed, which further enhances the feature extraction efficiency compared to traditional CNN methods. The introduction of residual blocks in traditional CNNs effectively solves the gradient vanishing problem in deep network training. This connection method allows information to bypass one or more layers directly, thereby achieving a deeper network structure and higher quality feature extraction. Residual connections have also been applied to human action recognition, linking the spatial features extracted by 1D-CNN with a bidirectional long short-term memory network, enhancing the model's ability to understand complex temporal patterns.

[0005] Although existing HAR methods have achieved certain success in some application scenarios, there are still many problems and challenges:

[0006] Data scarcity and difficulty in labeling: A large amount of sensor data is not labeled, resulting in a scarcity of training set data, which affects the generalization ability and accuracy of the model. The manual labeling process of sensor data is time-consuming and laborious, and requires the participation of domain experts, which is costly. In addition, the number of samples between different activities is often unbalanced, further increasing the difficulty of model training.

[0007] Insufficient feature extraction: Traditional feature extraction methods usually rely on manual design, which makes it difficult to fully capture the complex features in the data, resulting in the model being unable to fully utilize the useful information in the data. The limitations of traditional methods are particularly obvious when facing high-dimensional and multimodal data.

[0008] Unstable model training: During model training, traditional loss functions (such as cross entropy loss) perform poorly when dealing with unbalanced data, which can easily lead to overfitting or underfitting of the model. In practical applications, data imbalance and noise are often difficult to avoid, which further affects the stability and reliability of the model.

[0009] Lack of effective extraction and utilization of spatiotemporal features: Human activity data has obvious spatiotemporal characteristics, which traditional methods often ignore, resulting in insufficient performance of the model in capturing the temporal changes and spatial distribution of activities. Effectively extracting and utilizing spatiotemporal features is the key to improving HAR performance. Summary of the invention

[0010] The present invention proposes a method for optimizing human activity recognition based on a multi-branch spatiotemporal interaction network, which solves the technical problems existing in the prior art, such as data scarcity and difficulty in labeling, insufficient feature extraction, unstable model training, and lack of effective extraction and utilization of spatiotemporal features.

[0011] The technical solution adopted by the present invention is as follows: a gait data generation method based on a generative gait network: the steps are as follows:

[0012] (1) Data preprocessing: Using the unsupervised statistical feature-guided diffusion model SF-DM, a large amount of unlabeled wearable sensor data is processed to generate synthetic diffusion data;

[0013] Step 1.1: Sample a batch of data x of size m from the training dataset X;

[0014] Step 1.2: Calculate statistical characteristics: mean μ, standard deviation σ, Z score Z and skewness γ;

[0015] Step 1.3: Connect all statistical features into a vector f;

[0016] Step 1.4: Generate a random noise ∈ with the same shape as the input data from the real data, and use the noise scale β and diffusion step t to generate the noise data

[0017]

[0018] Step 1.5: Noisy data The statistical features f are input into SF-DM and the hyperparameter η of the Adam optimizer is used. E , by minimizing the reconstruction loss L rec To pre-train SF-DM, the reconstruction loss L rec for:

[0019]

[0020]

[0021] Step 1.6: Sample a batch (x, y) from the training dataset X and the corresponding activity labels Y;

[0022] Step 1.7: Calculate statistical characteristics: mean μ, standard deviation σ, Z score Z and skewness γ;

[0023] Step 1.8: Connect all system features in 1.7 into a vector f;

[0024] Step 1.9: Generate a random noise vector ω with the same shape as the SF-DM input;

[0025] Step 1.10: Use Adam optimizer with learning rate η C , by minimizing the synthetic loss L syn To train SF-DM, we can get the generated data:

[0026]

[0027]

[0028] Step 1.11: Sample a batch of data (x, y) of size m from the generated dataset X and the corresponding activity label Y;

[0029] Step 1.12: Use Adam optimizer with a learning rate of η C , by minimizing the true loss L real To fine tune SF-DM:

[0030]

[0031]

[0032] Step 1.13: Sample a batch (x, y) from the training dataset X and the corresponding activity labels Y and import it into the fine-tuned SF-DM.

[0033] Step 1.14: Generate diffusion data and export.

[0034] (2) Feature extraction: Establish a deep convolutional network model - a multi-branch spatiotemporal interaction network, combine the split module with temporal and spatial attention, extract features from the diffusion data, and obtain spatiotemporal feature data;

[0035] Step 2.1: Import the generated dataset into the model and add spatial attention to the signal, perform average pooling and maximum pooling operations on the time axis, and then apply a convolutional layer to generate a spatial attention map Use two pooling operations to aggregate channel information and generate feature maps With feature map Subsequently, a standard convolutional layer is used for concatenation and convolution;

[0036] Step 2.2: Divide the data processed by the convolutional layer in step 2.1 into K basis arrays, each basis array contains C / K channels, the size of the input feature signal is t×C, and the size of each basis array after division is W×(C / K). Each basis array is further divided into R subgroups, each subgroup contains C / (K*R) channels, then the size of each subgroup is t×(C / (K*R)), and the total number of subgroups is G=KR;

[0037] Step 2.3: Perform group transformation on each subgroup The input feature signal passes through a 1×1 convolution layer and then a 3×3 convolution layer. The 3×3 convolution is used to capture larger spatial context information; the intermediate representation of each subgroup is obtained:

[0038] Step 2.4: Summing the elements across multiple partitions to obtain the combined representation of each base array in

[0039] Step 2.5: Then add temporal attention to compress the global spatial information into channel descriptors by generating channel-level statistics. is generated by shrinking the output U by its spatial dimension S, so that the cth element of Z is calculated as:

[0040]

[0041] Step 2.6: Collect channel statistics of the embedding using global average pooling: The cth component is calculated as follows:

[0042] Step 2.7: Parameterize using two fully connected layers with ReLU activations and use channel soft attention to aggregate the weighted basis array representations

[0043]

[0044] in:

[0045] Step 2.8: Concatenate the base array representations in the channel dimension: V = Concat{V 1 ,V 2 ,...V K}Generate the final output Y through shortcut connections.

[0046] (3) Training model: Introduce loss function configuration, which integrates the composite loss function of cross entropy loss, focus loss and label smoothing regularization, so that the model can handle unbalanced data and label noise. The loss function configuration assigns different weights to each loss function. When processing large data sets, adjust the weight of the loss function according to the accuracy index, and use the Adam optimizer to update the parameters;

[0047] Step 3.1: Add a loss function to the network model and define the loss function as an autonomous update loss function, which is composed of: cross entropy loss, focal loss and label smoothing regularization. ce , focal loss Loss fl and label smoothing regularization Loss sl-nl The function formula is as follows:

[0048]

[0049] Step 3.2: Add weights to these three loss functions. Therefore, the specific formula for the autonomous update loss function is as follows:

[0050] Loss total =ω0Loss sl-n +ω1Loss fl +ω2Loss ce

[0051] Step 3.3: When training in large batches, the model is prone to overfitting. Therefore, the weight formula is updated outside the Adam optimization algorithm to obtain the current accuracy in each training cycle. The increase in accuracy will appropriately reduce the maximum weight value, so that the model is out of the local optimal situation. The weight calculation formula is as follows:

[0052] ω0=0.5*(1-ω1)

[0053]

[0054] ω2=0.5*(1-ω1)

[0055] Step 3.4: At the end of each batch of training, calculate the total loss function;

[0056] Step 3.5: Use the validation set to evaluate the trained model, calculate the classification accuracy and the value of the central loss function of the model, and adjust the model parameters according to the results;

[0057] Step 3.6: Use the test set to evaluate the final model, calculate the classification accuracy of the model and the value of the central loss function, and compare the test results with other existing methods to verify the effectiveness.

[0058] (4) The spatiotemporal feature data in (2) are identified through the model in (3), and finally the classification results of each movement are obtained.

[0059] The beneficial effects of the present invention are:

[0060] 1. It solves the problem of a large amount of unlabeled wearable sensor data in traditional datasets, thereby reducing the difficulty of data preprocessing, and the generated synthetic data can be better used for model training.

[0061] 2. The present invention focuses more accurately on the connection between human activities and time and space by combining a multi-branch spatiotemporal interaction network of temporal attention, spatial attention and split modules, thereby performing efficient feature extraction.

[0062] 3. The present invention introduces an automatically updated loss function, which combines cross entropy loss, focal loss and label smoothing regularization, which can enhance the model's attention to minority classes, alleviate the problem of overconfidence of the model, and improve the generalization performance of the model.

[0063] 4. By introducing the weight algorithm, the deficiency of the traditional method that requires manual updating of weights is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 :Multi-branch spatiotemporal interaction network optimization human activity recognition flow chart;

[0065] Figure 2 :Multi-branch spatiotemporal interaction network optimizes the overall model diagram of human activity recognition;

[0066] Figure 3 : Multi-branch spatiotemporal interaction network model diagram;

[0067] Figure 4: Autonomously update the loss function graph;

[0068] Figure 5 : Confusion matrix plot. DETAILED DESCRIPTION

[0069] Gait data generation method based on generative gait network: The steps are as follows:

[0070] (1) Data preprocessing: Using the unsupervised statistical feature-guided diffusion model SF-DM, a large amount of unlabeled wearable sensor data is processed to generate synthetic diffusion data;

[0071] Step 1.1: Sample a batch of data x of size m from the training dataset X;

[0072] Step 1.2: Calculate statistical characteristics: mean μ, standard deviation σ, Z score Z and skewness γ;

[0073] Step 1.3: Connect all statistical features into a vector f;

[0074] Step 1.4: Generate a random noise ∈ with the same shape as the input data from the real data, and use the noise scale β and diffusion step t to generate the noise data

[0075]

[0076] Step 1.5: Noisy data The statistical features f are input into SF-DM and the hyperparameter η of the Adam optimizer is used. E , by minimizing the reconstruction loss L rec To pre-train SF-DM, the reconstruction loss L rec for:

[0077]

[0078]

[0079] Step 1.6: Sample a batch (x, y) from the training dataset X and the corresponding activity labels Y;

[0080] Step 1.7: Calculate statistical characteristics: mean μ, standard deviation σ, Z score Z and skewness γ;

[0081] Step 1.8: Connect all system features in 1.7 into a vector f;

[0082] Step 1.9: Generate a random noise vector ω with the same shape as the SF-DM input;

[0083] Step 1.10: Use Adam optimizer with learning rate η C , by minimizing the synthetic loss L syn To train SF-DM, we can get the generated data:

[0084]

[0085]

[0086] Step 1.11: Sample a batch of data (x, y) of size m from the generated dataset X and the corresponding activity label Y;

[0087] Step 1.12: Use Adam optimizer with a learning rate of η C , by minimizing the true loss L real To fine tune SF-DM:

[0088]

[0089]

[0090] Step 1.13: Sample a batch (x, y) from the training dataset X and the corresponding activity labels Y and import it into the fine-tuned SF-DM.

[0091] Step 1.14: Generate diffusion data and export.

[0092] (2) Feature extraction: Establish a deep convolutional network model - a multi-branch spatiotemporal interaction network, combine the split module with temporal and spatial attention, extract features from the diffusion data, and obtain spatiotemporal feature data;

[0093] Step 2.1: Import the generated dataset into the model and add spatial attention to the signal, perform average pooling and maximum pooling operations on the time axis, and then apply a convolutional layer to generate a spatial attention map Use two pooling operations to aggregate channel information and generate feature maps With feature map Subsequently, a standard convolutional layer is used for concatenation and convolution;

[0094] Step 2.2: Divide the data processed by the convolutional layer in step 2.1 into K basis arrays, each basis array contains C / K channels, the size of the input feature signal is t×C, and the size of each basis array after division is W×(C / K). Each basis array is further divided into R subgroups, each subgroup contains C / (K*R) channels, then the size of each subgroup is t×(C / (K*R)), and the total number of subgroups is G=KR;

[0095] Step 2.3: Perform group transformation on each subgroup The input feature signal passes through a 1×1 convolution layer and then a 3×3 convolution layer. The 3×3 convolution is used to capture larger spatial context information; the intermediate representation of each subgroup is obtained:

[0096] Step 2.4: Summing the elements across multiple partitions to obtain the combined representation of each base array in

[0097] Step 2.5: Then add temporal attention to compress the global spatial information into channel descriptors by generating channel-level statistics. is generated by shrinking the output U by its spatial dimension S, so that the cth element of Z is calculated as:

[0098]

[0099] Step 2.6: Collect channel statistics of the embedding using global average pooling: The cth component is calculated as follows:

[0100] Step 2.7: Parameterize using two fully connected layers with ReLU activations and use channel soft attention to aggregate the weighted basis array representations

[0101]

[0102] in:

[0103] Step 2.8: Concatenate the base array representations in the channel dimension: V = Concat{V 1 ,V 2 ,...V K}Generate the final output Y through shortcut connections.

[0104] (3) Training model: Introduce loss function configuration, which integrates the composite loss function of cross entropy loss, focus loss and label smoothing regularization, so that the model can handle unbalanced data and label noise. The loss function configuration assigns different weights to each loss function. When processing large data sets, adjust the weight of the loss function according to the accuracy index, and use the Adam optimizer to update the parameters;

[0105] Step 3.1: Add a loss function to the network model and define the loss function as an autonomous update loss function, which is composed of: cross entropy loss, focal loss and label smoothing regularization. ce , focal loss Loss fland label smoothing regularization Loss sl-nl The function formula is as follows:

[0106]

[0107] Loss fl =-α t (1-p t ) γ log(p t )

[0108]

[0109] Step 3.2: Add weights to these three loss functions. Therefore, the specific formula for the autonomous update loss function is as follows:

[0110] Loss total =ω0Loss sl-nl +ω1Loss fl +ω2Loss ce

[0111] Step 3.3: When training in large batches, the model is prone to overfitting. Therefore, the weight formula is updated outside the Adam optimization algorithm to obtain the current accuracy in each training cycle. The increase in accuracy will appropriately reduce the maximum weight value, so that the model is out of the local optimal situation. The weight calculation formula is as follows:

[0112] ω0=0.5*(1-ω1)

[0113]

[0114] ω2=0.5*((1-ω1)

[0115] Step 3.4: At the end of each batch of training, calculate the total loss function;

[0116] Step 3.5: Use the validation set to evaluate the trained model, calculate the classification accuracy and the value of the central loss function of the model, and adjust the model parameters according to the results;

[0117] Step 3.6: Use the test set to evaluate the final model, calculate the classification accuracy of the model and the value of the central loss function, and compare the test results with other existing methods to verify the effectiveness.

[0118] (4) The spatiotemporal feature data in (2) are identified through the model in (3), and finally the classification results of each movement are obtained.

[0119] Embodiment 1:

[0120] We use the current mainstream open source datasets PAMAP2 and WISDM for human activity recognition to verify the effectiveness of our overall process. PAMAP2 was released in 2012. It includes data collected from 9 participants performing 18 different sports activities. WISDM is a dataset for HAR research released in 2010. It contains sample data of 51 volunteers performing 6 common daily activities, including standing, sitting, walking, going up and down stairs, and lying down. We loaded our data into other mainstream deep models for comparison. Our test results include two indicators, one of which is accuracy and the other is F1-weighted. In Tables 1 and 2, we named the multi-branch spatiotemporal interaction network MSTI (Multi-branch Spatiotemporal Interaction Network), and the comparison results are shown in Table 1:

[0121] Table 1: Comparison of relevant model performance indicators

[0122]

[0123] We also compared the accuracy of each motion classification of the WISDM dataset with the current mainstream models. The comparison results are shown in Table 2:

[0124] Table 2: Classification results of the WISDM dataset

[0125]

[0126] In order to better observe the category confusion, we plotted the confusion of the MSTI model on the WISDM dataset. Figure 5 shown.

Claims

1. A method for optimizing human activity recognition based on a multi-branch spatiotemporal interaction network, characterized in that: The steps are: (1) Data preprocessing: Using the unsupervised statistical feature-guided diffusion model SF-DM, a large amount of unlabeled wearable sensor data is processed to generate synthetic diffusion data; (2) Feature extraction: Establish a deep convolutional network model - a multi-branch spatiotemporal interaction network, combine the split module with temporal and spatial attention, extract features from the diffusion data, and obtain spatiotemporal feature data; (3) Training model: Introduce loss function configuration, which integrates the composite loss function of cross entropy loss, focus loss and label smoothing regularization, so that the model can handle unbalanced data and label noise. The loss function configuration assigns different weights to each loss function. When processing large data sets, adjust the weight of the loss function according to the accuracy index, and use the Adam optimizer to update the parameters; (4) The spatiotemporal feature data in (2) are identified through the model in (3), and finally the classification results of each movement are obtained.

2. According to claim 1, a method for optimizing human activity recognition based on a multi-branch spatiotemporal interaction network is characterized in that: In the step (1), the specific method is: Step 1.1: Sample a batch of data x of size m from the training dataset X; Step 1.2: Calculate statistical characteristics: mean μ, standard deviation σ, Z score Z and skewness γ; Step 1.3: Connect all statistical features into a vector f; Step 1.4: Generate a random noise ∈ with the same shape as the input data from the real data, and use the noise scale β and diffusion step t to generate the noise data Step 1.5: Noisy data The statistical features f are input into SF-DM and the hyperparameter η of the Adam optimizer is used. E , by minimizing the reconstruction loss L rec To pre-train SF-DM, the reconstruction loss L rec for: Step 1.6: Sample a batch (x, y) from the training dataset X and the corresponding activity labels Y; Step 1.7: Calculate statistical characteristics: mean μ, standard deviation σ, Z score Z and skewness γ; Step 1.8: Connect all system features in 1.7 into a vector f; Step 1.9: Generate a random noise vector ω with the same shape as the SF-DM input; Step 1.10: Use Adam optimizer with learning rate η C , by minimizing the synthetic loss L syn To train SF-DM, we can get the generated data: Step 1.11: Sample a batch of data (x, y) of size m from the generated dataset X and the corresponding activity label Y; Step 1.12: Use Adam optimizer with a learning rate of η C , by minimizing the true loss L real To fine tune SF-DM: Step 1.13: Sample a batch (x, y) from the training dataset X and the corresponding activity labels Y and import them into the fine-tuned SF-DM. Step 1.14: Generate diffusion data and export.

3. The method for optimizing human activity recognition based on a multi-branch spatiotemporal interaction network according to claim 1, characterized in that: In the step (2), the specific method is: Step 2.1: Import the generated dataset into the model and add spatial attention to the signal, perform average pooling and maximum pooling operations on the time axis, and then apply a convolutional layer to generate a spatial attention map Use two pooling operations to aggregate channel information and generate feature maps With feature map Subsequently, a standard convolutional layer is used for concatenation and convolution; Step 2.2: Divide the data processed by the convolutional layer in step 2.1 into K basis arrays, each basis array contains C / K channels, the size of the input feature signal is t×C, and the size of each basis array after division is W×(C / K). Each basis array is further divided into R subgroups, each subgroup contains C / (K*R) channels, then the size of each subgroup is t×(C / (K*R)), and the total number of subgroups is G=KR; Step 2.3: Perform group transformation on each subgroup The input feature signal passes through a 1×1 convolution layer and then a 3×3 convolution layer. The 3×3 convolution is used to capture larger spatial context information; the intermediate representation of each subgroup is obtained: Step 2.4: Summing the elements across multiple partitions to obtain the combined representation of each base array in Step 2.5: Then add temporal attention to compress the global spatial information into channel descriptors by generating channel-level statistics. is generated by shrinking the output U by its spatial dimension S, so that the cth element of Z is calculated as: Step 2.6: Collect channel statistics of the embedding using global average pooling: The cth component is calculated as follows: Step 2.7: Parameterize using two fully connected layers with ReLU activations and use channel-wise soft attention to aggregate the weighted basis array representations in: Step 2.8: Concatenate the base array representations in the channel dimension: V = Concat{V 1 ,V 2 ,...V K }Generate the final output Y through shortcut connections.

4. The method for optimizing human activity recognition based on a multi-branch spatiotemporal interaction network according to claim 1, characterized in that: In the step (3), the specific method is: Step 3.1: Add a loss function to the network model and define the loss function as an autonomous update loss function, which is composed of: cross entropy loss, focal loss and label smoothing regularization. ce , focal loss Loss fl and label smoothing regularization Loss sl-nll The function formula is as follows: Loss fl =-α t (1-p t ) γ log(p t ) Step 3.2: Add weights to these three loss functions. Therefore, the specific formula for the autonomous update loss function is as follows: Loss total =ω0Loss sl-nl +ω1Loss fl +ω2Loss ce Step 3.3: When training in large batches, the model is prone to overfitting. Therefore, the weight formula is updated outside the Adam optimization algorithm to obtain the current accuracy in each training cycle. The increase in accuracy will appropriately reduce the maximum weight value, so that the model is out of the local optimal situation. The weight calculation formula is as follows: ω0=0.5*(1-ω1) ω2=0.5*(1-ω1) Step 3.4: At the end of each batch of training, calculate the total loss function; Step 3.5: Use the validation set to evaluate the trained model, calculate the classification accuracy and the value of the central loss function of the model, and adjust the model parameters according to the results; Step 3.6: Use the test set to evaluate the final model, calculate the classification accuracy of the model and the value of the central loss function, and compare the test results with other existing methods to verify the effectiveness.

Images