Activity recognition method based on variational autoencoder and temporal network

Abstract

The application discloses an activity recognition method based on a variational autoencoder and a time sequence network, and belongs to the technical field of behavior feature recognition, and comprises the following steps: S1, collecting an original data set of activity signals and performing pretreatment; S2, extracting latent features based on S1 by using a VAE, and outputting a reconstruction error; S3, obtaining time dynamic features based on S1 by using a CNN-GRU; S4, performing feature fusion based on the results of S2 and S3 by using a self-attention mechanism, performing soft assignment by using a deep embedding clustering layer, and performing classification by using a classifier, and completing model training; S5, testing, judging whether input is OOD data, and outputting a category. The application effectively recognizes and processes OOD data by judging the reconstruction error and the output of the deep embedding clustering layer, and enhances the robustness and generalization ability of the model; by combining the VAE, the CNN, the GRU and the self-attention mechanism, space-time features in IMU signals are comprehensively extracted, the accuracy of activity recognition is improved, time sequence information is effectively reserved, and the model training efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of behavior feature recognition, in particular to an activity recognition method based on a variational autoencoder and a time series network. BACKGROUND

[0002] With the increasing concern for health and the widespread popularity of intelligent technology, human activity recognition technology that can monitor and analyze daily activities in real time has become a key tool for improving quality of life and optimizing health management. Through accurate activity recognition, people can better understand their own exercise patterns and scientifically plan their exercise time and intensity, thereby effectively preventing chronic diseases and promoting physical health. Human activity recognition technology is widely used in medical rehabilitation, smart homes, and exercise guidance, and has shown great application prospects.

[0003] With the popularity of wearable devices, human activity recognition has entered a new stage of development. Inertial measurement unit (IMU) sensors, as the core technology in wearable devices, can record the dynamic state of the human body by capturing real-time three-axis acceleration, angular velocity, and direction information, and recognize multiple activity patterns.

[0004] In the prior art, an invention patent with publication number CN117765607A and the name of "Activity recognition method and system based on recursive graph and CNN-LSTM" discloses an activity recognition method based on recursive graph and CNN-LSTM. This method converts the three-axis acceleration sensor time series data of the sample into a corresponding two-dimensional recursive matrix by using an improved recursive graph algorithm, and uses the recursive matrix sample to realize the classification of the CNN-LSTM classification model. However, this method does not consider the actual OutofDistribution (OOD) problem, such as abnormal patterns or noise in sensor data. When the model encounters OOD data, the model has not learned how to handle this type of input because these data did not occur during training, and usually produces incorrect predictions or cannot make predictions. Therefore, this method cannot effectively handle OOD data, and the accuracy and reliability of the model are low.

[0005] Based on the above-mentioned defects, an activity recognition method based on a variational autoencoder and a time series network is proposed, which can effectively handle OOD data. SUMMARY

[0006] The purpose of the present application is to provide an activity recognition method based on a variational autoencoder and a time series network to solve the problems in the background art.

[0007] To achieve the above-mentioned purpose, the present application provides an activity recognition method based on a variational autoencoder and a time series network, comprising the following steps:

[0008] S1, collect a raw data set by using an inertial measurement unit sensor for an activity timing signal, pre-process the raw data set to obtain a training set and a test set;

[0009] S2, input the training set obtained in S1 into a variational autoencoder, define a loss function to train the variational autoencoder, the encoder in the variational autoencoder is responsible for extracting latent features, the decoder in the variational autoencoder is responsible for reconstructing the data distribution using the latent features, and outputting a reconstruction error, and using reparameterization for back propagation;

[0010] S3, for the training set obtained in S1, using CNN to capture spatial feature association, obtaining local spatio-temporal features, using GRU to capture the dynamic evolution of local spatio-temporal features in time latitude, obtaining time dynamic features;

[0011] S4, using a self-attention mechanism to fuse the latent features obtained in S2 and the time dynamic features obtained in S3 to obtain sample features, using a deep embedding clustering layer to perform soft assignment on the sample features and iterative training to obtain the distance between each sample feature and the cluster center, using a classifier to classify the sample features, and completing the model training;

[0012] S5, input the test set in S1 into the trained model, judge whether the reconstruction error obtained in S2 and the distance between the sample features in S4 and the nearest cluster center are greater than the corresponding threshold value, and if both are greater than the corresponding threshold value, it is determined as OOD data, otherwise the class of the test set is output through the classifier in S4.

[0013] Preferably, in S1, the specific process of pre-processing is:

[0014] 1) Data reading: reading the raw data set and converting the raw data set into a Pandas two-dimensional data structure;

[0015] 2) Outlier processing: removing abnormal null data based on the result of step 1);

[0016] 3) Window division: performing fixed-length data segmentation on the result of step 2) to obtain overlapping windows;

[0017] 4) Standardization: standardizing the result of step 3) using Z-score normalization to obtain a high-quality data set with standard normal distribution, and dividing the high-quality data set into a training set and a test set.

[0018] Preferably, in S2, the loss function is defined as:

[0019] L VAE =-E q(z|x) [logp(x|z)]+D KL(q(z|x)||p(z));

[0020] where x is the input data, z is the latent variable, L VAE is the loss function of the variational autoencoder, E q(z|x) is the expectation of the approximate posterior distribution q(z|x), logp(x|z) is the log of the reconstruction data distribution p(x|z), D KL (·) is the Kullback-Leibler divergence, and p(z) is the prior distribution.

[0021] Preferably, in the S2, the latent feature is represented as:

[0022] q(z|x) = N(z; μ(x), σ 2 (x));

[0023] where q(z|x) is the Gaussian distribution of the latent variable, i.e., the approximate variational distribution, N(·) represents the normal distribution, μ(x) is the mean of the latent variable, and σ 2 (x) is the variance of the latent variable.

[0024] Preferably, in the S2, the output of the decoder is represented as:

[0025]

[0026] where q(x|z) is the probability distribution of the decoder output, f(z) is the prior distribution, is the reconstruction variance.

[0027] Preferably, in the S3, the capturing of the spatial feature correlation includes a convolution operation, the introduction of nonlinearity using an activation function, and a batch normalization operation.

[0028] Preferably, the convolution operation is represented as:

[0029]

[0030] where Y t is the value of the output feature map at time step t, W i is the i-th convolution kernel, x t+i-1 is the data of the input signal at the t+i-1 time, and b is the bias term.

[0031] Preferably, the introduction of nonlinearity using an activation function is represented as:

[0032] ReLU(x) = max(0, x);

[0033] where ReLU(·) represents the activation function, and max(·) represents the maximum function.

[0034] Preferably, the batch normalization is represented as:

[0035]

[0036] where μ is the mean of the current batch data, σ 2 is the variance of the current batch data, ε is a small constant, and γ and β are both learned parameters.

[0037] Preferably, in the S3, the specific process of capturing the dynamic evolution of the local spatio-temporal features in the time dimension is: calculating the values of the update gate, the reset gate and the final hidden state, and determining the information that needs to be memorized and discarded in the local spatio-temporal features.

[0038] Preferably, the update gate is represented as:

[0039] z t =σ(W z x t +U z h t-1 +b z );

[0040] where z t is the update gate, σ(·) represents the Sigmoid activation function, W z and U z are both weight matrices of the update gate, x t is the input at the current time step, h t-1 is the hidden state at the previous time, and b z is the bias term of the update gate.

[0041] Preferably, the reset gate is represented as:

[0042] z r =σ(W r x t +U r h t-1 +b r );

[0043] where z r is the reset gate, W r and U r are both weight matrices of the update gate, and b r is the bias term of the update gate.

[0044] Preferably, the final hidden state is represented as:

[0045]

[0046] where h t is the final hidden state, ⊙ represents element-wise multiplication, is the candidate hidden state.

[0047] wherein the formula of the candidate hidden state is:

[0048]

[0049] wherein tanh(·) represents the hyperbolic tangent function, W h and U h are weight matrices of the candidate hidden state, b r is a bias term of the candidate hidden state, and r t is the output of the reset gate.

[0050] Preferably, in the S4, the specific process of feature fusion is:

[0051] 1) scaling the dot product result between the query vector and the key vector, the scaling is represented as:

[0052]

[0053] wherein Attention Scores represents the attention score, Q is the query vector, K T is the transpose of the key vector, and d k is the dimension of the key vector;

[0054] 2) normalizing the attention score by using the Softmax function to obtain the attention weight, the normalization is represented as:

[0055] Attention Weights=Softmax(Attention Scores,dim=-1);

[0056] wherein Attention Weights represents the attention weight, and Softmax(·) represents the normalization processing by using the Softmax function;

[0057] 3) applying the attention weight to the value vector to obtain the weighted feature, the weighted feature is represented as:

[0058] Attention Output=Attention Weights*V;

[0059] wherein Attention Output represents the attention output, and V is the value vector;

[0060] 4) mapping the weighted feature back to the original dimension of the input by using the linear layer to obtain the sample feature, the sample feature is represented as:

[0061] Fused Features=Linear(Attention Output);

[0062] In the formula, Fused Features is the fused sample feature, and Linear(·) represents linear layer processing.

[0063] Preferably, in the S4, the specific process of iterative training is as follows:

[0064] 1) The probability of each sample feature belonging to each cluster is calculated using the student t distribution;

[0065] 2) The KL divergence loss between the probability of each sample feature belonging to each cluster and the target distribution is minimized.

[0066] Preferably, the calculation formula of the probability of each sample feature data belonging to each cluster is as follows:

[0067]

[0068] In the formula, q ij is the probability of input i belonging to the jth cluster, siml(·) represents similarity function calculation, z i represents the corresponding signal of input i in the latent space z, w j is the center of the jth cluster, and alpha is a hyperparameter, and k is the number of clusters.

[0069] Therefore, the activity recognition method based on the variational autoencoder and the time sequence network has the following beneficial effects:

[0070] (1) By judging and processing the reconstruction error of the variational autoencoder and the output of the deep embedding clustering layer, OOD data can be effectively recognized and processed, and the robustness and generalization ability of the model are enhanced.

[0071] (2) By combining the variational autoencoder, CNN, GRU and self-attention mechanism, the spatio-temporal features in the IMU signal are comprehensively extracted, thereby improving the accuracy of activity recognition.

[0072] (3) By learning the probability distribution of the latent space through the variational autoencoder, the time sequence information can be effectively preserved, the high-dimensional IMU signal can be compressed into a low-dimensional latent space, the model calculation amount can be reduced, and the model training efficiency can be improved.

[0073] The technical solutions of the present application will be described in further detail below with reference to the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 is a model architecture diagram of an embodiment of the present application;

[0075] Figure 2 is a clustering effect schematic diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0076] The technical solutions of the present application will be further described below by means of the accompanying drawings and examples.

[0077] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments of the present application.

[0078] Embodiments

[0079] As shown in the drawings, the present application provides an activity recognition method based on variational autoencoder and time series network. The data from 29 subjects' accelerometer sensors of the WISDM dataset are used to test the classification effect and robustness of the model. Figure 1

[0080] The WISDM (Wireless Sensor Data Mining) dataset is a widely used time series dataset for action recognition, health monitoring and intelligent system research. It is composed of accelerometer sensor data from smartphones, used to classify different activity types (such as walking, running, etc.). The accelerometer in WISDM is used to collect three-dimensional acceleration data. Each data point contains the acceleration values of the accelerometer in x, y and z dimensions. The sampling frequency of each data point is 20Hz, i.e. data is collected every 50 milliseconds. The WISDM dataset contains multiple categories of daily activities. The accelerometer data for each activity category is labeled, and the specific activities in this embodiment include:

[0081] ① Walking (walking): normal walking action;

[0082] ② Jogging (fast running): fast running action;

[0083] ③ Sitting (sitting): activity of sitting still;

[0084] ④ Standing (standing): activity of standing still;

[0085] ⑤ Upstairs (going upstairs): activity of walking stairs;

[0086] ⑥ Downstairs (going downstairs): activity of going downstairs.

[0087] Each activity is performed by one or more subjects under different time and environmental conditions and data is collected.

[0088] The specific steps are as follows: ​

[0089] S1, collecting a raw data set by using an inertial measurement unit sensor for an activity timing signal, preprocessing the raw data set to obtain a training set and a test set;

[0090] The specific process of preprocessing is as follows:

[0091] 1) Data reading: read the raw data set into the temp array, remove unnecessary delimiters, and convert the raw data set into a Pandas two-dimensional data NumPy array structure. This two-dimensional structure facilitates subsequent cleaning, analysis, and modeling.

[0092] 2) Outlier processing: In the inertial measurement unit sensor data, there may be null values caused by device failure or external interference, or data of inconsistent length caused by actual data. These outliers can adversely affect model training, so based on the results of step 1), remove the abnormal null data.

[0093] 3) Window division: The results of step 2) are divided into fixed-length data to obtain overlapping windows. In this embodiment, the window size is 200, and the overlap rate is 50%, specifically:

[0094] The model needs to learn the features in the time series data through a fixed-length time window, so the continuous data is divided into multiple windows of equal length. To avoid missing important information or affecting the transition area during window division, a certain overlap rate is used to some extent. This approach effectively increases the amount of data samples while ensuring that there is enough context information between windows, which helps to convert long-time signals into fixed-length data suitable for input into neural networks, thereby improving the robustness and generalization ability of the model.

[0095] 4) Standardization: Data usually has different dimensions and ranges, which can negatively affect the training process of the model. Therefore, the results of step 3) are standardized using Z-score normalization to obtain a high-quality data set with a standard normal distribution (mean of 0 and standard deviation of 1). Label encoding converts action categories to integer values, and all data in the high-quality data set is randomly divided into a 7:3 ratio to form a training set and a test set.

[0096] The standardization process is represented as:

[0097]

[0098] In the formula, For the standardized data, x is the input data before standardization, μ is the mean of the input data x before standardization, and σ is the standard deviation of the input data x before standardization; this normalization not only speeds up the convergence of the gradient descent algorithm, but also eliminates the influence of dimensional differences on model learning;

[0099] Through a series of preprocessing operations, the inertial measurement unit sensor data is converted into a consistent and high-quality dataset, ensuring that the model can learn the key features of time series data more quickly and accurately, and reducing the performance degradation caused by inconsistent data or missing information; These preprocessing operations are not only the basis of data preparation, but also the key to improving model performance and reliability. Through this carefully designed preprocessing process, a solid foundation can be laid for time series classification tasks, making the subsequent model learning process more efficient and accurate.

[0100] S2, input the training set obtained in S1 into the variational autoencoder, define the loss function, and train the variational autoencoder; the encoder in the variational autoencoder is responsible for extracting latent features, which can preserve time series information while compressing feature dimensions, thereby improving the performance of downstream tasks; the decoder in the variational autoencoder is responsible for reconstructing the data distribution using latent features, outputting reconstruction errors, and using reparameterization for backpropagation, specifically:

[0101] The core goal of VAE is to train the model by maximizing the log-likelihood of the input data log p(x), that is, the model is expected to generate samples similar to the training data. Since it is difficult to calculate p(x) directly, VAE uses variational inference method to approximate the posterior distribution, thereby optimizing the model parameters, specifically:

[0102] The training goal of variational autoencoder is to maximize the log-likelihood of data log p(x). Since the posterior distribution p(z|x) is difficult to calculate, VAE introduces an approximate variational distribution q(z|x) to simplify the inference process. In the variational autoencoder, resampling aims to sample latent variables from the latent space and generate new samples in this way. VAE samples from the latent space using the resampling technique, so that the model can generate new data using the latent variable during training. VAE is trained by maximizing the variational lower bound (ELBO), which includes parameterizing the latent variable distribution, assuming that the latent variable follows a standard normal distribution, i.e. p(z) = N(0, I).

[0103] Specifically, VAE optimizes the model by maximizing the evidence lower bound (ELBO);

[0104] The calculation formula of maximizing the evidence lower bound is:

[0105] logp(x) ≥ E q(z|x) [logp(x|z)]-D KL (q(z|x)||p(z));

[0106] where E q(z|x) [logp(x|z)] is the reconstruction error also called the likelihood term, which measures the quality of the model's reconstruction of the data given the latent variables, it measures the difference between the input data and the reconstructed data, usually using mean square error as the reconstruction error; D KL (q(z|x)||p(z)) is the Kullback-Leibler divergence, which is used to measure the difference between the approximate posterior distribution and the prior distribution of the latent variables.

[0107] Therefore, the loss function of VAE (the negative of ELBO) is:

[0108] L VAE =-E q(z|x) [logp(x|z)]+D KL (q(z|x)||p(z));

[0109] where z is the latent variable, L VAE denotes the loss function of the variational autoencoder, E q(z|x) is the expected value of the approximate posterior distribution q(z|x), logp(x|z) is the logarithm of the reconstructed data distribution p(x|z), D KL (·) represents the Kullback-Leibler divergence, and p(z) is the prior distribution.

[0110] Variational autoencoder (VAE) is based on the classic autoencoder (Autoencoder) architecture, but combines it with a probabilistic inference method. Autoencoder consists of an encoder and a decoder, the encoder maps the input to the latent space, and the decoder reconstructs the input from the latent space. VAE can generate new samples by learning the probability distribution of the latent space.

[0111] In VAE, the encoder and the decoder are responsible for the inference of the latent variables and the generation of data respectively. The task of the encoder is to extract the probability distribution of the latent variables from the input data. Given an input, the encoder will output a Gaussian distribution of the latent variables (latent features);

[0112] In this embodiment, the filter list of the convolutional layer in the encoder is set to [32, 64, 128, 256, 512], the padding of each convolutional layer is 0, the step is 1, the kernel size is 3, and the dimension of the latent space is set to 32. Two fully connected layers are defined after the encoder to map the extracted features to 32, one of the two fully connected layers is used to map the mean, and the other is used to map the variance.

[0113] The latent feature representation is denoted as:

[0114] q(z|x) = N(z; μ(x), σ 2 (x));

[0115] where q(z|x) is the Gaussian distribution of the latent variable, i.e., the approximate variational distribution, N(·) represents the normal distribution, μ(x) is the mean of the latent variable, and σ 2 (x) is the variance of the latent variable, both of which describe the distribution in the latent space for generating new samples from the latent space; because a three-axis motion sensor is used in this data set, the input data of the model has 3 channels, and the length of the input sequence is 200 in the data processing part.

[0116] The task of the decoder is to reconstruct the input data from the latent variable. Given the latent variable, the decoder calculates the reconstruction distribution p(x|z) of the data through a generative network. For continuous data, the decoder outputs the probability distribution of the data, including the mean and variance, etc.

[0117] The probability distribution of the output of the decoder is denoted as:

[0118]

[0119] where q(x|z) is the probability distribution of the decoder output, f(z) is the prior distribution, is the reconstruction variance.

[0120] The latent variable is first mapped to the output space of a fully connected mapping encoder, and then mapped through the decoder. In the decoder of this embodiment, the filter list of the convolutional layer is [512, 256, 128, 64, 32], the padding of each convolutional layer is 0, the step is 1, and the convolution kernel size is 3. After passing through the decoder, the latent variable is mapped into a signal with the same dimension as the input, i.e., the reconstruction data distribution.

[0121] The training of VAE requires sampling from the posterior distribution q(z|x), but since the sampling operation is not differentiable, it cannot be directly backpropagated. Therefore, VAE uses reparameterization to solve this problem. By letting the latent variable z N(0, I), the sampling operation can be converted into a deterministic operation, so that the model can be backpropagated.

[0122] S3, the training set obtained in S1 is used to capture spatial feature association by CNN to obtain local space-time features, CNN extracts local features through convolution operation, and then uses a pooling layer for dimension reduction and feature selection; GRU is used to capture the dynamic evolution of local space-time features in the time latitude to obtain time dynamic features, GRU is a variant of recurrent neural network (RNN) and is specially used for processing time series data, it introduces a "gate" mechanism (update gate and reset gate) to enable the model to capture long-term dependencies in long sequences and solve the gradient disappearance problem in traditional RNN; CNN and GRU are combined to process sequence data with spatial and temporal dependencies;

[0123] The input signal of the embodiment is a three-axis acceleration signal, so the input of the CNN is [3, 200]; wherein, the capturing of spatial feature association includes convolution operation, introduction of nonlinearity by using an activation function and batch normalization operation, specifically:

[0124] 1) Convolution operation:

[0125] The core of CNN is the convolution layer, and the filter list of the convolution layer in the embodiment is set to [32, 64, 128, 256], which is used to extract local features in the input signal through a convolution kernel (filter). Assuming that the IMU data input into the convolution layer is a multi-channel time series signal, which is represented as:

[0126] X = [x1, x2, … x i …x T ];

[0127] In the formula, x i is the inertial measurement unit sensor data at the i-th moment;

[0128] For a convolution layer, the output feature map can be calculated by the following formula:

[0129]

[0130] In the formula, Y t is the value of the output feature map at time step t, W i is the i-th convolution kernel, x t+i-1 is the data at the t+i-1 moment of the input signal, and b is the bias term;

[0131] Through convolution operation, the model can learn the locally relevant features in space and time, such as the correlation between sensors and the change pattern of signals in some time steps.

[0132] 2) Introducing nonlinearity by using an activation function:

[0133] After the convolution operation, an activation function (such as ReLU) is used to introduce nonlinearity so that the model can learn more complex features. The ReLU activation function is represented as:

[0134] ReLU(x) = max(0, x);

[0135] where ReLU(·) represents the activation function, and max(·) represents the maximum value function.

[0136] 3) Batch normalization operation:

[0137] Batch normalization is a technique that normalizes the input of each layer during training, which can accelerate network training, improve model stability, reduce dependence on initialization, and sometimes act as regularization. Batch normalization normalizes the output of each layer so that its output has zero mean and unit variance, preventing gradient explosion or gradient disappearance. Batch normalization is represented as:

[0138]

[0139] where μ is the mean of the current batch data, σ 2 is the variance of the current batch data, ε is a small constant added to prevent division by zero error, and γ and β are learning parameters updated by backpropagation.

[0140] GRU controls information flow through reset and update gates. Given the input and the previous hidden state, the process of capturing the dynamic evolution of local spatio-temporal features in the time dimension is as follows: calculate the values of the update gate, the reset gate, and the final hidden state, determine the information to be remembered and discarded in the local spatio-temporal features, and the specific process is as follows:

[0141] 1) The update gate controls the degree of combination of the current input and the previous hidden state. The update gate is represented as:

[0142] z t =σ(W z x t +U z h t-1 +b z );

[0143] where z t is the update gate, σ(·) represents the Sigmoid activation function, W z and U z are the weight matrices of the update gate, x t is the input of the current time step, h t-1 is the hidden state of the previous time step, and b z is the bias term of the update gate.

[0144] 2) The reset gate determines the proportion of the hidden state information at the previous time that is forgotten, and the reset gate is represented as:

[0145] z r = sigma(W r x t + U r h t-1 + b r );

[0146] In the formula, z r is the reset gate, W r and U r are weight matrices of the update gate, and b r is a bias term of the update gate;

[0147] 4) The candidate hidden state is calculated based on the hidden state at the previous time controlled by the reset gate, and the candidate hidden state is represented as:

[0148]

[0149] In the formula, h is the candidate hidden state, tanh(·) represents the hyperbolic tangent function, W h and U h are weight matrices of the candidate hidden state, b r is a bias term of the candidate hidden state, and r t is the output of the reset gate;

[0150] 4) The final hidden state is calculated by weighted average of the update gate and the candidate hidden state, and the final hidden state is represented as:

[0151]

[0152] In the formula, h t is the final hidden state, represents element-wise multiplication, is the candidate hidden state;

[0153] The number of hidden layers of the GRU is set to 2, the hidden state node is 256, and it is set to be unidirectional, so after passing through the GRU module, all features are extracted into a state space of [256, 1].

[0154] S4, the latent features obtained by S2 and the time dynamic features obtained by S3 are fused by using a self-attention mechanism to obtain sample features, the sample features are soft-assigned and iteratively trained by using a deep embedding clustering layer to obtain the distance between each sample feature and a cluster center, and the sample features are classified by using a classifier to complete model training, and specifically:

[0155] 1) Self-attention mechanism:

[0156] Self-attention mechanism is a mechanism that generates weighted output by calculating the correlation between input features. It uses three vectors: "Query", "Key" and "Value" to calculate attention scores, which determine the relative importance of each input. In self-attention mechanism, given a sequence of input features, they can be mapped to query, key and value spaces through linear transformation, specifically:

[0157] Query vector: used to determine the importance of a certain feature.

[0158] Key vector: used to compare with query vector to calculate matching degree.

[0159] Value vector: contains actual information, used to generate the final weighted output.

[0160] In self-attention mechanism, the similarity between query vector and key vector (usually calculated by dot product) determines the correlation between features. By calculating the dot product between each pair of Q (query vector) and K (key vector), a matrix representing the similarity between all features is obtained.

[0161] ① To avoid the gradient explosion caused by the large value of dot product, the dot product result is usually scaled, the formula is as follows:

[0162]

[0163] In the formula, Attention Scores represents attention scores, K T is the transpose of key vector, d k is the dimension of key vector;

[0164] ② Use Softmax to normalize the attention scores, so that the sum of weights in each row is 1. This step converts the attention scores into a probability distribution through the Softmax function, representing the importance of each element in feature fusion, and obtains the attention weights, which are represented as:

[0165] Attention Weights = Softmax(Attention Scores, dim = -1);

[0166] In the formula, Attention Weights represents attention weights, and Softmax(·) represents normalization processing using Softmax function;

[0167] ③ By applying attention weights to value vector V, we get the weighted features, and the calculation formula of this process is:

[0168] Attention Output = Attention Weights * V;

[0169] where Attention Output denotes the attention output; this operation multiplies the attention weights with the value vector to obtain a weighted feature representation.

[0170] (4) Map the weighted features back to the original dimension of the input through a linear layer to obtain the final sample feature, denoted as:

[0171] Fused Features = Linear(Attention Output);

[0172] where Fused Features is the fused sample feature, and Linear(·) denotes a linear layer processing.

[0173] 2) Deep embedding clustering layer:

[0174] The deep embedding clustering method attempts to assign input data to multiple cluster centers, and each data point will obtain a probability distribution representing its belonging to each cluster center, i.e., soft assignment. By calculating the similarity between each sample and the cluster center, and based on the Student's t-distribution, a probability distribution representing each sample belonging to each cluster is output, which represents the "soft" classification result of the sample, i.e., the membership degree of each cluster, rather than a hard partition.

[0175] The deep embedding clustering layer is initialized before training, and the cluster centers are uniformly dispersed, and the cluster center is set as a variable parameter. For this data set, there are 6 categories in total, based on this prior condition, the number of cluster centers is directly defined as 6, and in each training, the output of the attention mechanism module is input to the clustering layer, and the similarity between the sample feature output by the attention mechanism and each cluster center is calculated, and in this embodiment, the Euclidean distance is used to calculate the distance between the sample feature and each cluster center, and the smaller the distance, the higher the similarity. After calculating the similarity, a Student's t-distribution formula is used to calculate the probability of each sample feature belonging to each cluster, as follows:

[0176]

[0177] where q ij is the probability of input i belonging to the jth cluster, siml(·) denotes the similarity function calculation, z i denotes the corresponding signal of input i in the latent space z, w j is the center of the jth cluster, and a is a hyperparameter, and k is the number of clusters.

[0178] To iteratively train the temporal clustering layer, high-confidence predictions need to be reinforced and the loss needs to be normalized to prevent distortion of the latent representation, so the objective is formulated to minimize the KL divergence loss between q ij and the target distribution P ij .

[0179] So far, the entire model has been trained, and the clustering effect of the trained model is shown in Figure 2 The following tests the model using the test set:

[0180] S5, input the test set in S1 into the trained model, judge whether the reconstruction error obtained in S2 and the distance between the sample feature in S4 and the nearest cluster center are greater than the corresponding threshold, and if both are greater than the corresponding threshold, it is determined as OOD data, otherwise the class of the test set is output through the classifier in S4, specifically:

[0181] For the trained model, when testing, the sample is input into the model, first judge whether there is OOD data through the reconstruction error of VAE and the distance to the nearest cluster center (cluster center) in the clustering layer, only when the reconstruction error of VAE is greater than Th VAE and the distance of the cluster center is greater than Th Cluster , it will be determined as OOD, otherwise the class of the test set is directly output through the classifier;

[0182] Wherein, Th VAE is the threshold of the reconstruction error of VAE, and Th Cluster is the threshold of the distance of the cluster center.

[0183] Therefore, the activity recognition method based on the variational autoencoder and the time sequence network can effectively recognize and process OOD data and enhance the robustness and generalization ability of the model by judging and processing the reconstruction error of the variational autoencoder and the output of the deep embedding clustering layer.

[0184] By combining the variational autoencoder, CNN, GRU and self-attention mechanism, the spatio-temporal features in the IMU signal can be comprehensively extracted, thereby improving the accuracy of activity recognition.

[0185] By learning the probability distribution of the latent space through the variational autoencoder, the temporal information can be effectively preserved, the high-dimensional IMU signal can be compressed into a low-dimensional latent space, the model calculation amount can be reduced, and the model training efficiency can be improved.

[0186] It should be pointed out finally that the above examples are only used to illustrate the technical solutions of the present application but not to limit it, and although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can still be modified or replaced equivalently, and these modifications or equivalent replacements should not make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. An activity recognition method based on variational autoencoders and temporal networks, characterized in that, Includes the following steps: S1. For the activity timing signal, the inertial measurement unit sensor is used to collect the raw data of the accelerometer sensor. The raw data is preprocessed to obtain the training set and the test set. S2. Input the training set obtained in S1 into the variational autoencoder, define the loss function to train the variational autoencoder. The encoder in the variational autoencoder is responsible for extracting latent features, and the decoder in the variational autoencoder is responsible for reconstructing the data distribution using the latent features and outputting the reconstruction error. Backpropagation is performed using reparameterization. S3. Using CNN to capture spatial feature associations on the training set obtained in S1, local spatiotemporal features are obtained. GRU is used to capture the dynamic evolution of local spatiotemporal features in the time dimension, thus obtaining temporal dynamic features. Capturing spatial feature correlations includes convolution operations, introducing nonlinearity using activation functions, and batch normalization operations; The specific process of capturing the dynamic evolution of local spatiotemporal features in the time dimension is as follows: calculate the values ​​of the update gate, reset gate and final hidden state, and determine the information that needs to be remembered and discarded in the local spatiotemporal features; S4. The latent features obtained in S2 and the time dynamic features obtained in S3 are fused using a self-attention mechanism to obtain sample features. A deep embedding clustering layer is used to softly assign the sample features and iteratively train them to obtain the distance between each sample feature and the cluster center. A classifier is used to classify the sample features to complete the model training. The specific process of iterative training is as follows: 1) Calculate the probability that each sample feature belongs to each cluster using the Student's t-distribution; 2) Minimize the KL divergence loss between the probability of each sample feature belonging to each cluster and the target distribution; S5. Input the test set from S1 into the trained model and determine whether the reconstruction error obtained from S2 and the distance between the sample features in S4 and the nearest cluster center are both greater than the corresponding threshold. If both are greater, it is determined to be OOD data; otherwise, the class of the test set is output through the classifier in S4.

2. The activity recognition method based on variational autoencoder and temporal network according to claim 1, characterized in that, In S1, the specific preprocessing process is as follows: 1) Data Reading: Read the original dataset and convert it into a Pandas two-dimensional data structure; 2) Outlier handling: Based on the results of step 1), remove outlier null values; 3) Window partitioning: The data from step 2) is divided into fixed-length segments to obtain overlapping windows; 4) Standardization: The results of step 3) are standardized by Z-score normalization to obtain a high-quality dataset with a standard normal distribution. The high-quality dataset is then divided into a training set and a test set.

3. The activity recognition method based on variational autoencoder and temporal network according to claim 1, characterized in that, In S2, the loss function is defined as: ； In the formula, For input data, As latent variables, This represents the loss function of the variational autoencoder. For approximate posterior distribution Expected value To reconstruct the data distribution The logarithm of Indicates the Kullback-Leibler divergence. This is the prior distribution.

4. The activity recognition method based on variational autoencoder and temporal network according to claim 1, characterized in that, The update gate is represented as follows: ； In the formula, To update the door, This represents the Sigmoid activation function. and Both are the weight matrices of the update gate. For the current time step input, This is the hidden state from the previous moment. To update the bias term of the gate.

5. The activity recognition method based on variational autoencoder and temporal network according to claim 4, characterized in that, The reset gate is represented as: ； In the formula, To reset the door, and Both are the weight matrices of the update gate. To update the bias term of the gate.

6. The activity recognition method based on variational autoencoder and temporal network according to claim 5, characterized in that, The final hidden state is represented as follows: ； In the formula, This is the final hidden state. This indicates element-wise multiplication. This is a candidate hidden state; The formula for calculating the candidate hidden state is as follows: ； In the formula, Represents the hyperbolic tangent function. and These are all weight matrices for candidate hidden states. The bias term for the candidate hidden state. To reset the output of the gate.

7. The activity recognition method based on variational autoencoder and temporal network according to claim 1, characterized in that, The formula for calculating the probability that each sample feature data belongs to each cluster is as follows: ； In the formula, For input Belongs to the The probability of a cluster. Represents the calculation of the similarity function. Indicates input In potential space The corresponding signal in For the first The center of the cluster, For hyperparameters, The number of clusters.

Images