Activity identification method based on variational auto-encoder and sequential network

Abstract

The invention discloses an activity identification method based on a variational auto-encoder and a sequential network, and belongs to the technical field of behavior feature identification, and the method comprises the following steps: S1, collecting an original data set of an activity signal, and carrying out the preprocessing of the original data set; s2, extracting potential features by adopting VAE based on the S1, and outputting a reconstruction error; s3, on the basis of the S1, obtaining time dynamic characteristics by adopting CNN-GRU; s4, performing feature fusion by adopting a self-attention mechanism based on results of S2 and S3, performing soft distribution by adopting a deep embedded clustering layer, and performing classification by adopting a classifier to complete model training; and S5, testing, judging whether the input is OOD data or not, and outputting the category. According to the method, the reconstruction error and the output of the deep embedded clustering layer are judged, so that the OOD data are effectively identified and processed, and the robustness and generalization ability of the model are enhanced; through combination of VAE, CNN, GRU and a self-attention mechanism, spatiotemporal features in IMU signals are comprehensively extracted, activity identification accuracy is improved, time sequence information is effectively reserved, and model training efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior feature recognition, and in particular to an activity recognition method based on variational autoencoder and temporal network. Background Art

[0002] With people's increasing attention to health and the widespread popularity of intelligent technology, human activity recognition technology that can monitor and analyze daily activities in real time has become a key tool for improving the quality of life and optimizing health management. Through accurate activity recognition, people can have a more comprehensive understanding of their own movement patterns, scientifically plan exercise time and intensity, and thus effectively prevent chronic diseases and promote physical health. Human activity recognition technology is widely entering multiple fields such as medical rehabilitation, smart home and sports guidance, showing broad application prospects.

[0003] With the popularity of wearable devices, human activity recognition has entered a new stage of development. As the core technology in wearable devices, the inertial measurement unit (IMU) sensor can fully record the dynamic state of the human body and identify various activity patterns by capturing three-axis acceleration, angular velocity and direction information in real time.

[0004] In the prior art, the invention patent with publication number CN117765607A and titled "Activity Recognition Method and System Based on Recursive Graph and CNN-LSTM" discloses an activity recognition method based on recursive graph and CNN-LSTM. The method converts the three-axis acceleration sensing time series data of the sample into a corresponding two-dimensional recursive matrix by adopting an improved recursive graph algorithm, and uses the recursive matrix sample to implement the classification of the CNN-LSTM classification model. However, this method does not take into account the actual Out of Distribution (OOD) problem, such as abnormal patterns or noise in sensor data. When its model encounters OOD data, since these data have not appeared during training, the model has not learned how to handle this type of input, and usually produces wrong predictions or cannot make predictions. Therefore, this method cannot effectively process OOD data, and the accuracy and reliability of the model are low.

[0005] Based on the above defects, an activity recognition method based on variational autoencoder and temporal network is proposed, which can effectively process OOD data. Summary of the invention

[0006] The purpose of the present invention is to provide an activity recognition method based on a variational autoencoder and a temporal network to solve the problems in the background technology.

[0007] To achieve the above object, the present invention provides an activity recognition method based on a variational autoencoder and a temporal network, comprising the following steps:

[0008] S1. Using an inertial measurement unit sensor to collect raw data sets for activity timing signals, preprocessing the raw data sets to obtain training sets and test sets;

[0009] S2, input the training set obtained in S1 into the variational autoencoder, define the loss function to train the variational autoencoder, the encoder in the variational autoencoder is responsible for extracting potential features, the decoder in the variational autoencoder is responsible for reconstructing the data distribution using the potential features, outputting the reconstruction error, and using reparameterization for back propagation;

[0010] S3, for the training set obtained in S1, CNN is used to capture the spatial feature association to obtain local spatiotemporal features, and GRU is used to capture the dynamic evolution of local spatiotemporal features in the time dimension to obtain temporal dynamic features;

[0011] S4, using the self-attention mechanism to fuse the latent features obtained in S2 and the temporal dynamic features obtained in S3 to obtain sample features, using the deep embedding clustering layer to soft-assign the sample features and iteratively train them to obtain the distance between each sample feature and the cluster center, using the classifier to classify the sample features, and completing the model training;

[0012] S5: Input the test set in S1 into the trained model, and determine whether the reconstruction error obtained in S2 and the distance between the sample features in S4 and the nearest cluster center are both greater than the corresponding threshold. If both are greater, it is determined to be OOD data. Otherwise, the category of the test set is output through the classifier in S4.

[0013] Preferably, in S1, the specific process of pretreatment is:

[0014] 1) Data reading: read the original data set and convert it into a Pandas two-dimensional data structure;

[0015] 2) Outlier processing: Based on the results of step 1), remove abnormal null value data;

[0016] 3) Window division: perform fixed-length data segmentation on the result of step 2) to obtain overlapping windows;

[0017] 4) Standardization: The result of step 3) is standardized by Z-score normalization to obtain a high-quality data set with a standard normal distribution, and the high-quality data set is divided into a training set and a test set.

[0018] Preferably, in S2, the loss function is defined as:

[0019] L VAE =-E q(z|x) [logp(x|z)]+D KL(q(z|x)||p(z));

[0020] In the formula, x is the input data, z is the latent variable, and L VAE represents the loss function of the variational autoencoder, E q(z|x) is the expected value of the approximate posterior distribution q(z|x), logp(x|z) is the logarithm of the reconstructed data distribution p(x|z), and D KL (·) denotes the Kullback-Leibler divergence, and p(z) is the prior distribution.

[0021] Preferably, in S2, the potential feature is expressed as:

[0022] q(z|x)=N(z;μ(x),σ 2 (x));

[0023] Where q(z|x) is the Gaussian distribution of the latent variable, i.e., the approximate variational distribution, N(·) represents the normal distribution, μ(x) is the mean of the latent variable, and σ 2 (x) is the variance of the latent variable.

[0024] Preferably, in S2, the output of the decoder is expressed as:

[0025]

[0026] Where q(x|z) is the probability distribution of the decoder output, f(z) is the prior distribution, is the reconstructed variance.

[0027] Preferably, in S3, capturing spatial feature associations includes convolution operations, introducing nonlinearity using activation functions, and batch normalization operations.

[0028] Preferably, the convolution operation is expressed as:

[0029]

[0030] Where Y t is the value of the output feature map at time step t, W i is the i-th convolution kernel, x t+i-1 is the data of the input signal at the t+i-1th moment, and b is the bias term.

[0031] Preferably, the activation function is used to introduce the nonlinear representation as follows:

[0032] ReLU(x)=max(0,x);

[0033] Where ReLU(·) represents the activation function and max(·) represents the maximum value function.

[0034] Preferably, the batch normalization is expressed as:

[0035]

[0036] In the formula, μ is the mean of the current batch of data, σ 2 is the variance of the current batch of data, ε is a small constant, and γ and β are both learning parameters.

[0037] Preferably, in S3, the specific process of capturing the dynamic evolution of local spatiotemporal features in the time dimension is: calculating the values ​​of the update gate, the reset gate and the final hidden state, and determining the information that needs to be remembered and discarded in the local spatiotemporal features.

[0038] Preferably, the update gate is expressed as:

[0039] z t =σ(W z x t +U z h t-1 +b z );

[0040] In the formula, z t is the update gate, σ(·) represents the Sigmoid activation function, W z and U z are the weight matrices of the update gates, x t is the input of the current time step, h t-1 is the hidden state of the previous moment, b z is the bias term for the update gate.

[0041] Preferably, the reset gate is expressed as:

[0042] z r =σ(W r x t +U r h t-1 +b r );

[0043] In the formula, z r To reset the gate, W r and U r are the weight matrices of the update gate, b r is the bias term for the update gate.

[0044] Preferably, the final hidden state is expressed as:

[0045]

[0046] In the formula, h t is the final hidden state, ⊙ represents element-by-element multiplication, is a candidate hidden state;

[0047] Among them, the calculation formula for the candidate hidden state is:

[0048]

[0049] Where tanh(·) represents the hyperbolic tangent function, W h and U h are all weight matrices of candidate hidden states, b r is the bias term of the candidate hidden state, r t is the output of the reset gate.

[0050] Preferably, in S4, the specific process of feature fusion is:

[0051] 1) Scale the dot product result between the query vector and the key vector. The scaling is expressed as:

[0052]

[0053] In the formula, Attention Scores represents the attention score, Q is the query vector, and K T is the transpose of the key vector, d k is the dimension of the key vector;

[0054] 2) Use the Softmax function to normalize the attention score to obtain the attention weight, which is expressed as:

[0055] Attention Weights=Softmax(Attention Scores,dim=-1);

[0056] In the formula, Attentikon Weights represents the attention weight, Softmax(·) represents the normalization process using the Softmax function;

[0057] 3) Apply the attention weight to the value vector to obtain the weighted feature, which is expressed as:

[0058] Attention Output=Attention Weights*V;

[0059] In the formula, Attention Output represents the attention output, and V is the value vector;

[0060] 4) The weighted features are mapped back to the original dimension of the input through a linear layer to obtain sample features, which are expressed as:

[0061] Fused Features=Linear(Attention Output);

[0062] Where, Fused Features is the fused sample features, and Linear(·) represents linear layer processing.

[0063] Preferably, in S4, the specific process of iterative training is:

[0064] 1) Use Student's t distribution to calculate the probability that each sample feature belongs to each cluster;

[0065] 2) Minimize the KL divergence loss between the probability that each sample feature belongs to each cluster and the target distribution.

[0066] Preferably, the calculation formula for the probability that each sample feature data belongs to each cluster is:

[0067]

[0068] In the formula, q ij is the probability that input i belongs to the jth cluster, siml(·) represents the similarity function calculation, z i represents the signal corresponding to input i in the latent space z, w j is the center of the jth cluster, α is a hyperparameter, and k is the number of clusters.

[0069] Therefore, the present invention provides an activity recognition method based on variational autoencoder and temporal network, which has the following beneficial effects:

[0070] (1) By judging and processing the reconstruction error of the variational autoencoder and the output of the deep embedding clustering layer, OOD data can be effectively identified and processed, thereby enhancing the robustness and generalization ability of the model.

[0071] (2) By combining variational autoencoders, CNNs, GRUs, and self-attention mechanisms, the spatiotemporal features in IMU signals are comprehensively extracted, thereby improving the accuracy of activity recognition.

[0072] (3) By learning the probability distribution of the latent space through the variational autoencoder, the time series information can be effectively retained, the high-dimensional IMU signal can be compressed into a low-dimensional latent space, the amount of model calculation can be reduced, and the efficiency of model training can be improved.

[0073] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 This is a model architecture diagram of an embodiment of the present invention;

[0075] Figure 2 It is a schematic diagram of the clustering effect of an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0077] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0078] Example

[0079] like Figure 1 As shown, the present invention provides an activity recognition method based on variational autoencoder and temporal network. The data of accelerometer sensors of 29 subjects from the WISDM dataset are used to test the classification effect and robustness of the model.

[0080] The WISDM (Wireless Sensor Data Mining) dataset is a time series dataset widely used in motion recognition, health monitoring, and intelligent system research. It consists of accelerometer sensor data from smartphones and is used to classify different types of activities (such as walking, running, etc.). The accelerometer in WISDM is used to collect three-dimensional acceleration data. Each data point contains the acceleration value of the accelerometer in three dimensions: x, y, and z. The sampling frequency of each data point is 20Hz, that is, data is collected every 50 milliseconds. The WISDM dataset contains multiple categories of daily activities. The accelerometer data for each activity category is labeled, and the specific activities of this embodiment include:

[0081] ①Walking: normal walking movements;

[0082] ②Jogging: fast running action;

[0083] ③ Sitting: activities of sitting still;

[0084] ④Standing: activities of standing still;

[0085] ⑤Upstairs: Activities involving walking up stairs;

[0086] ⑥Downstairs(downstairs): the activity of going down the stairs.

[0087] Each activity is performed by one or more subjects and data is collected at different times and under different environmental conditions.

[0088] The specific steps are as follows:

[0089] S1. Using an inertial measurement unit sensor to collect raw data sets for activity timing signals, preprocessing the raw data sets to obtain training sets and test sets;

[0090] The specific process of preprocessing is as follows:

[0091] 1) Data reading: read the original data set into the temp array, remove unnecessary separators, and convert the original data set into a NumPy array structure of Pandas two-dimensional data. This two-dimensional structure facilitates subsequent cleaning, analysis, and modeling;

[0092] 2) Outlier processing: In the inertial measurement unit sensor data, there may be null values ​​due to equipment failure or external interference, or data with inconsistent lengths due to the existence of real data. These outliers will have an adverse effect on the training of the model. Therefore, it is necessary to remove the abnormal null value data based on the results of step 1);

[0093] 3) Window division: For the result of step 2), data segmentation of fixed length is performed to obtain overlapping windows. In this embodiment, the window size is 200 and the overlap rate is 50%. Specifically:

[0094] The model needs to learn the features of time series data through a fixed-length time window, so the continuous data is divided into multiple windows of equal length. At the same time, in order to avoid missing important information or the influence of active transition areas when dividing the windows, the data is divided according to a certain overlap rate to a certain extent. This method can effectively increase the amount of data samples and ensure that there is enough context information between windows, which helps to convert long-term signals into fixed-length data suitable for input into the neural network, thereby improving the robustness and generalization ability of the model;

[0095] 4) Standardization: Data usually has different dimensions and ranges, which may have a negative impact on the model training process. Therefore, the results of step 3) are standardized using Z-score normalization to obtain a high-quality data set with a standard normal distribution (mean 0, standard deviation 1). Label encoding is performed to convert the action categories into integer values. All data in the high-quality data set are randomly divided into a training set and a test set in a ratio of 7:3.

[0096] Among them, the standardization process is expressed as:

[0097]

[0098] In the formula, is the data after standardization, x is the input data before standardization, μ is the mean of the input data x before standardization, and σ is the standard deviation of the input data x before standardization. This normalization not only speeds up the convergence of the gradient descent algorithm, but also eliminates the impact of dimensional differences on model learning.

[0099] Through a series of preprocessing operations, the inertial measurement unit sensor data is converted into a consistent and high-quality data set, ensuring that the model can learn the key features of time series data more quickly and accurately, and reduce performance degradation caused by inconsistent data or missing information; these preprocessing operations are not only the basis for data preparation, but also the key to improving model performance and reliability. Through this carefully designed preprocessing process, a solid foundation can be laid for time series classification tasks, making the subsequent model learning process more efficient and accurate.

[0100] S2. Input the training set obtained in S1 into the variational autoencoder, define the loss function to train the variational autoencoder, the encoder in the variational autoencoder is responsible for extracting potential features, which can retain the time series information while compressing the feature dimension, thereby improving the performance of downstream tasks. The decoder in the variational autoencoder is responsible for reconstructing the data distribution using the potential features, outputting the reconstruction error, and using reparameterization for back propagation, specifically:

[0101] The core goal of VAE is to train the model by maximizing the log-likelihood of the input data, log p(x), that is, to hope that the model can generate samples similar to the training data. Since it is difficult to directly calculate p(x), VAE uses variational inference methods to approximate the posterior distribution, thereby optimizing the model parameters, specifically:

[0102] The training objective of the variational autoencoder is to maximize the log-likelihood of the data, log p(x). Since the posterior distribution p(z|x) is difficult to calculate, VAE introduces an approximate variational distribution q(z|x) to simplify the inference process. In the variational autoencoder, resampling aims to sample latent variables from the latent space and generate new samples in this way. VAE samples from the latent space through resampling techniques, allowing the model to generate new data using the latent variables during training. VAE is trained by maximizing the variational lower bound (ELBO), which includes parameterizing the distribution of the latent variables, assuming that the latent variables follow a standard normal distribution, that is, p(z) = N(0,I).

[0103] Specifically, VAE optimizes the model by maximizing the Evidence Lower Bound (ELBO);

[0104] The formula for maximizing the evidence lower bound is:

[0105] logp(x)≥E q(z|x) [logp(x|z)]-D KL (q(z|x)||p(z));

[0106] Among them, E q(z|x) [logp(x|z)] is the reconstruction error, also known as the likelihood term, which is the quality of the model's reconstruction of the data given the latent variables. It measures the difference between the input data and the reconstructed data. The mean square error is usually used as the reconstruction error. D KL (q(z|x)||p(z)) is the Kullback-Leibler divergence, which measures the difference between the approximate posterior distribution, the distribution of the latent variable, and the prior distribution.

[0107] Therefore, the loss function of VAE (the negative of ELBO) is:

[0108] L VAE =-E q(z|x) [logp(x|z)]+D KL (q(z|x)||p(z));

[0109] In the formula, z is the latent variable, L VAE represents the loss function of the variational autoencoder, E q(z|x) is the expected value of the approximate posterior distribution q(z|x), logp(x|z) is the logarithm of the reconstructed data distribution p(x|z), and D KL (·) denotes the Kullback-Leibler divergence, and p(z) is the prior distribution.

[0110] The variational autoencoder (VAE) is based on the classic autoencoder architecture, but combines it with a probabilistic inference method. The autoencoder consists of an encoder and a decoder, where the encoder maps the input to a latent space, while the decoder reconstructs the input from the latent space. VAE is able to generate new samples by learning the probability distribution of the latent space.

[0111] In VAE, the encoder and decoder are responsible for the inference of latent variables and the generation of data respectively. The encoder's task is to extract the probability distribution of latent variables from the input data. Given an input, the encoder will output the Gaussian distribution of the latent variables (latent features);

[0112] In this embodiment, the filter list of the convolution layer in the encoder is set to [32, 64, 128, 256, 512], the padding of each convolution layer is 0, the step size is 1, the convolution kernel size is 3, and the dimension of the latent space is set to 32. Two fully connected layers are defined after the encoder to map the extracted features to 32. The two fully connected layers are used to map the mean and the variance respectively.

[0113] The latent features are expressed as:

[0114] q(z|x)=N(z;μ(x),σ 2 (x));

[0115] Where q(z|x) is the Gaussian distribution of the latent variable, i.e., the approximate variational distribution, N(·) represents the normal distribution, μ(x) is the mean of the latent variable, and σ 2 (x) is the variance of the latent variable. These two parameters describe the distribution in the latent space and are used to generate new samples from the latent space. Because a three-axis motion sensor is used in this dataset, the number of channels of the model's input data is 3, and the length of the input sequence is 200 in the data processing part.

[0116] The task of the decoder is to reconstruct the input data from the latent variables. Given the latent variables, the decoder calculates the reconstructed distribution p(x|z) of the data through a generative network. For continuous data, the decoder outputs the probability distribution of the data, including the mean and variance.

[0117] The probability distribution of the decoder output is expressed as:

[0118]

[0119] Where q(x|z) is the probability distribution of the decoder output, f(z) is the prior distribution, is the reconstructed variance.

[0120] The latent variables are first mapped to the output space of the encoder through a fully connected map, and then passed through the decoder. The filter list of the convolutional layer in the decoder of this embodiment is [512, 256, 128, 64, 32]. The padding of each convolutional layer is 0, the step size is 1, and the convolution kernel size is 3. After passing through the decoder, the latent variables are mapped into signals of the same dimension as the input, that is, the reconstructed data distribution.

[0121] VAE training requires sampling from the posterior distribution q(z|x), but since the sampling operation is not differentiable, it cannot be directly back-propagated. Therefore, VAE uses reparameterization to solve this problem. By transforming the latent variable z∈N(0,I), the sampling operation can be transformed into a deterministic operation, allowing the model to perform back-propagation.

[0122] S3. CNN is used to capture the spatial feature association of the training set obtained in S1 to obtain local spatiotemporal features. CNN extracts local features through convolution operations, and then uses pooling layers for dimensionality reduction and feature selection. GRU is used to capture the dynamic evolution of local spatiotemporal features in the time dimension to obtain temporal dynamic features. GRU is a variant of recurrent neural network (RNN) specifically used to process time series data. It introduces a "gate" mechanism (update gate and reset gate) to enable the model to capture long-term dependencies in long sequences and solve the gradient vanishing problem in traditional RNNs. CNN and GRU are combined to process sequence data with spatial and temporal dependencies.

[0123] The input signal of this embodiment is a three-axis acceleration signal, so the input of CNN is [3, 200]; wherein, capturing spatial feature association includes convolution operation, using activation function to introduce nonlinearity and batch normalization operation, specifically:

[0124] 1) Convolution operation:

[0125] The core of CNN is the convolution layer. In this embodiment, the filter list of the convolution layer is set to [32, 64, 128, 256]. Its function is to extract local features in the input signal through the convolution kernel (filter). Assume that the IMU data input to the convolution layer is a multi-channel time series signal, which is expressed as:

[0126] X=[x1,x2,…x i …x T ];

[0127] In the formula, x i is the inertial measurement unit sensor data at the i-th moment;

[0128] For a convolutional layer, the output feature map can be calculated by the following formula:

[0129]

[0130] Where Y t is the value of the output feature map at time step t, W i is the i-th convolution kernel, x t+i-1 is the data of the input signal at the t+i-1th moment, and b is the bias term;

[0131] Through convolution operations, the model is able to learn features that are locally relevant in space and time, such as correlations between sensors and patterns of signal change within certain time steps.

[0132] 2) Use activation function to introduce nonlinearity:

[0133] After the convolution operation, an activation function (such as ReLU) is used to introduce nonlinearity so that the model can learn more complex features. The ReLU activation function is expressed as:

[0134] ReLU(x)=max(0,x);

[0135] In the formula, ReLU(·) represents the activation function, and max(·) represents the maximum value function;

[0136] 3) Batch normalization operation:

[0137] Batch normalization is a technique for standardizing the input of each layer during training. It can speed up network training, improve model stability, reduce dependence on initialization, and sometimes play a regularization role. Batch normalization normalizes the output of each layer so that its output has zero mean and unit variance, thereby preventing gradient explosion or gradient disappearance. Batch normalization is expressed as:

[0138]

[0139] In the formula, μ is the mean of the current batch of data, σ 2 is the variance of the current batch of data, ε is a small constant added to prevent division by zero errors, and γ and β are both learning parameters updated by back propagation.

[0140] GRU controls the information flow through reset gates and update gates. Given the input and the hidden state of the previous moment, the process of capturing the dynamic evolution of local spatiotemporal features in the time dimension is: calculating the values ​​of the update gate, reset gate, and final hidden state, and determining the information that needs to be remembered and discarded in the local spatiotemporal features. Specifically:

[0141] 1) The update gate controls the degree of integration between the current input and the hidden state at the previous moment. The update gate is expressed as:

[0142] z t =σ(W z x t +U z h t-1 +b z );

[0143] In the formula, z t is the update gate, σ(·) represents the Sigmoid activation function, W z and U z are the weight matrices of the update gates, x t is the input of the current time step, h t-1 is the hidden state of the previous moment, b z is the bias term of the update gate;

[0144] 2) The reset gate determines the proportion of the hidden state information at the previous moment that is forgotten. The reset gate is expressed as:

[0145] z r =σ(W r x t +U r h t-1 +b r );

[0146] In the formula, z r To reset the gate, W r and U r are the weight matrices of the update gate, b r is the bias term of the update gate;

[0147] 4) The candidate hidden state is calculated based on the hidden state at the previous moment controlled by the reset gate. The candidate hidden state is expressed as:

[0148]

[0149] In the formula, is a candidate hidden state, tanh(·) represents the hyperbolic tangent function, W h and U h are all weight matrices of candidate hidden states, b r is the bias term of the candidate hidden state, r t is the output of the reset gate;

[0150] 4) The final hidden state is calculated by updating the gate and the weighted average of the candidate hidden states. The final hidden state is expressed as:

[0151]

[0152] In the formula, h t is the final hidden state, ⊙ represents element-by-element multiplication, is a candidate hidden state;

[0153] The number of hidden layers of GRU is set to 2, the number of hidden state nodes is 256, and it is set to unidirectional, so after passing through the GRU module, all features will be extracted into a state space of [256,1].

[0154] S4, using the self-attention mechanism to fuse the potential features obtained in S2 and the temporal dynamic features obtained in S3 to obtain sample features, using the deep embedding clustering layer to soft-assign the sample features and iteratively train them to obtain the distance between each sample feature and the cluster center, using the classifier to classify the sample features, and completing the model training, specifically:

[0155] 1) Self-attention mechanism:

[0156] The self-attention mechanism is a mechanism that generates weighted outputs by calculating the correlation between input features. It uses three vectors, "query", "key" and "value", to calculate the attention score, thereby determining the relative importance of each input. In the self-attention mechanism, given a sequence of input features, they can be mapped to the query, key and value spaces respectively through linear transformations, specifically:

[0157] Query vector: used to determine the importance of a feature.

[0158] Key vector: used to compare with the query vector and calculate the matching degree.

[0159] Value vector: contains the actual information used to generate the final weighted output.

[0160] In the self-attention mechanism, the similarity between the query vector and the key vector (usually calculated by dot product) determines the relevance between features. By calculating the dot product between each pair of Q (query vector) and K (key vector), a matrix is ​​obtained to represent the similarity between all features.

[0161] ① In order to avoid gradient explosion caused by excessive dot product values, the dot product results are usually scaled. The formula is as follows:

[0162]

[0163] In the formula, Attention Scores represents the attention score, K T is the transpose of the key vector, d k is the dimension of the key vector;

[0164] ②Use Softmax to normalize the attention score so that the sum of the weights of each row is 1. This step converts the attention score into a probability distribution through the Softmax function, indicating the importance of each element in feature fusion, and obtains the attention weight, which is expressed as:

[0165] Attention Weights=Softmax(Attention Scores,dim=-1);

[0166] In the formula, Attention Weights represents the attention weight, and Softmax(·) represents the normalization process using the Softmax function;

[0167] ③By applying the attention weights to the value vector V, we get the weighted features. The calculation formula for this process is:

[0168] Attention Output=Attention Weights*V;

[0169] Where Attention Output represents the attention output; this operation multiplies the attention weight by the value vector to obtain the weighted feature representation.

[0170] ④ Map the weighted features back to the original dimension of the input through a linear layer to obtain the final sample features. The sample features are expressed as:

[0171] Fused Features=Linear(Attention Output);

[0172] Where, Fused Features is the fused sample features, and Linear(·) represents linear layer processing.

[0173] 2) Deep Embedded Clustering Layer:

[0174] The deep embedding clustering method attempts to assign input data to multiple cluster centers, and each data point will get a probability distribution indicating that it belongs to each cluster center, that is, soft assignment. By calculating the similarity between each sample and the cluster center, and outputting a probability distribution indicating that each sample belongs to each cluster based on the Student's t-distribution, this probability distribution represents the "soft" classification result of the sample, that is, the membership of each cluster, rather than a hard division.

[0175] The deep embedding clustering layer will be initialized before training, the cluster centers will be evenly dispersed, and the cluster centers will be set with variable parameters. For this data set, there are 6 categories in total. Based on this prior condition, the number of cluster centers is directly defined as 6. In each training, the output of the attention mechanism module is input into the clustering layer, and the similarity between the sample features output by the attention mechanism and each cluster center is calculated. In this embodiment, the distance between the sample features and each cluster center is calculated by the Euclidean distance. The smaller the distance, the higher the similarity. After calculating the similarity, a student t distribution formula is used to calculate the probability that each sample feature belongs to each cluster, as follows:

[0176]

[0177] In the formula, q ij is the probability that input i belongs to the jth cluster, siml(·) represents the similarity function calculation, z i represents the signal corresponding to input i in the latent space z, w j is the center of the jth cluster, α is a hyperparameter, and k is the number of clusters.

[0178] To iteratively train the temporal clustering layer, it is necessary to reinforce high-confidence predictions and normalize the loss to prevent distortion of the latent representation, so the objective is formulated as minimizing q ij and the target distribution P ij The KL divergence loss between .

[0179] At this point, the entire model has been trained, and the clustering effect of the model after training is as follows: Figure 2 As shown, the following test set is used to test the model:

[0180] S5, input the test set in S1 into the trained model, and determine whether the reconstruction error obtained in S2 and the distance between the sample feature in S4 and the nearest cluster center are both greater than the corresponding threshold. If both are greater, it is determined to be OOD data. Otherwise, the classifier in S4 outputs the category of the test set, which is specifically:

[0181] For the trained model, when testing, the sample is input into the model. First, the reconstruction error of VAE and the distance to the nearest cluster center (cluster center) in the clustering layer are used to determine whether there is OOD data. Only when the reconstruction error of VAE is greater than Th VAE And the distance between cluster centers is greater than Th Cluster It will be judged as OOD, otherwise the category of the test set is directly output through the classifier;

[0182] Among them, Th VAE is the threshold of VAE reconstruction error, Th Cluster is the threshold of the cluster center distance.

[0183] Therefore, the activity recognition method based on the variational autoencoder and the time series network of the present invention can effectively identify and process OOD data by judging and processing the reconstruction error of the variational autoencoder and the output of the deep embedding clustering layer, thereby enhancing the robustness and generalization ability of the model;

[0184] By combining variational autoencoders, CNNs, GRUs, and self-attention mechanisms, we can comprehensively extract spatiotemporal features from IMU signals, thereby improving the accuracy of activity recognition.

[0185] By learning the probability distribution of the latent space through the variational autoencoder, the timing information can be effectively retained, the high-dimensional IMU signal can be compressed into a low-dimensional latent space, the amount of model calculation can be reduced, and the efficiency of model training can be improved.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. An activity recognition method based on variational autoencoder and temporal network, characterized in that: The following steps are involved: S1. Using an inertial measurement unit sensor to collect raw data sets for activity timing signals, preprocessing the raw data sets to obtain training sets and test sets; S2, input the training set obtained in S1 into the variational autoencoder, define the loss function to train the variational autoencoder, the encoder in the variational autoencoder is responsible for extracting potential features, the decoder in the variational autoencoder is responsible for reconstructing the data distribution using the potential features, outputting the reconstruction error, and using reparameterization for back propagation; S3, for the training set obtained in S1, CNN is used to capture the spatial feature association to obtain local spatiotemporal features, and GRU is used to capture the dynamic evolution of local spatiotemporal features in the time dimension to obtain temporal dynamic features; S4, using the self-attention mechanism to fuse the latent features obtained in S2 and the temporal dynamic features obtained in S3 to obtain sample features, using the deep embedding clustering layer to soft-assign the sample features and iteratively train them to obtain the distance between each sample feature and the cluster center, using the classifier to classify the sample features, and completing the model training; S5: Input the test set in S1 into the trained model, and determine whether the reconstruction error obtained in S2 and the distance between the sample features in S4 and the nearest cluster center are both greater than the corresponding threshold. If both are greater, it is determined to be OOD data. Otherwise, the category of the test set is output through the classifier in S4.

2. The activity recognition method based on variational autoencoder and temporal network according to claim 1, characterized in that: In S1, the specific process of preprocessing is: 1) Data reading: read the original data set and convert it into a Pandas two-dimensional data structure; 2) Outlier processing: Based on the results of step 1), remove abnormal null value data; 3) Window division: perform fixed-length data segmentation on the result of step 2) to obtain overlapping windows; 4) Standardization: The result of step 3) is standardized by Z-score normalization to obtain a high-quality data set with a standard normal distribution, and the high-quality data set is divided into a training set and a test set.

3. The activity recognition method based on variational autoencoder and temporal network according to claim 1, characterized in that: In S2, the loss function is defined as: L VAE =-E q(z|x) [logp(x|z)]+D KL (q(z|x)||p(z)); In the formula, x is the input data, z is the latent variable, and L VAE represents the loss function of the variational autoencoder, E q(z|x) is the expected value of the approximate posterior distribution q(z|x), logp(x|z) is the logarithm of the reconstructed data distribution p(x|z), and D KL (·) denotes the Kullback-Leibler divergence, and p(z) is the prior distribution.

4. The activity recognition method based on variational autoencoder and temporal network according to claim 1, characterized in that: In S3, capturing spatial feature associations includes convolution operations, introducing nonlinearity using activation functions, and batch normalization operations.

5. The activity recognition method based on variational autoencoder and temporal network according to claim 1, characterized in that: In S3, the specific process of capturing the dynamic evolution of local spatiotemporal features in the time dimension is: calculating the values ​​of the update gate, the reset gate and the final hidden state, and determining the information that needs to be remembered and discarded in the local spatiotemporal features.

6. The activity recognition method based on variational autoencoder and temporal network according to claim 5, characterized in that: The update gate is expressed as: z t =σ(W z x t +U z h t-1 +b z ); In the formula, z t is the update gate, σ(·) represents the Sigmoid activation function, W z and U z are the weight matrices of the update gate, x t is the input of the current time step, h t-1 is the hidden state of the previous moment, b z is the bias term for the update gate.

7. The activity recognition method based on variational autoencoder and temporal network according to claim 6, characterized in that: The reset gate is represented by: z r =σ(W r x t +U r h t-1 +b r ); In the formula, z r To reset the gate, W r and U r are the weight matrices of the update gate, b r is the bias term for the update gate.

8. The activity recognition method based on variational autoencoder and temporal network according to claim 7, characterized in that: The final hidden state is expressed as: In the formula, h t is the final hidden state, ⊙ represents element-by-element multiplication, is a candidate hidden state; Among them, the calculation formula for the candidate hidden state is: Where tanh(·) represents the hyperbolic tangent function, W h and U h are all weight matrices of candidate hidden states, b r is the bias term of the candidate hidden state, r t is the output of the reset gate.

9. The activity recognition method based on variational autoencoder and temporal network according to claim 1, characterized in that: In S4, the specific process of iterative training is: 1) Use Student's t distribution to calculate the probability that each sample feature belongs to each cluster; 2) Minimize the KL divergence loss between the probability that each sample feature belongs to each cluster and the target distribution.

10. The activity recognition method based on variational autoencoder and temporal network according to claim 9, characterized in that: The calculation formula for the probability that each sample feature data belongs to each cluster is: In the formula, q ij is the probability that input i belongs to the jth cluster, siml(·) represents the similarity function calculation, z i represents the signal corresponding to input i in the latent space z, w j is the center of the jth cluster, α is a hyperparameter, and k is the number of clusters.

Images