Explainable human body behavior recognition method based on time-feature decoupling decision path

Abstract

The invention discloses an interpretable human behavior recognition method based on a time-feature decoupling decision path, which ensures standardized input of multi-source time series data, effective capture time dependence of xLSTM variants, classification result correction by a hierarchical decision tree through soft probability and visualization of the decision path, and positioning of key spatio-temporal features by integrating a gradient technology. Through fusion of xLSTM feature extraction, hierarchical decision tree reasoning and integrated gradient attribution, high precision and interpretability of human behavior recognition are realized, the effectiveness of the framework is verified by experimental results, and certain technical support is provided for transparent decision making of scenes such as industrial element universe and the like.

Description

Technical Field

[0001] This invention belongs to the field of behavior recognition technology, specifically relating to an interpretable human behavior recognition method based on time-feature decoupling decision paths. Background Technology

[0002] Industrial Metaverse, as an emerging model integrating digital twins, the Internet of Things, and artificial intelligence, constructs an immersive virtual ecosystem for real-time interaction and human-machine collaboration. Among them, Human Action Recognition (HAR) collects time-series data (such as acceleration and angular velocity) through sensors to capture dynamic behavioral characteristics, becoming the core support for realizing virtual-real interaction.

[0003] Traditional HAR classification methods include the K-nearest neighbor (KNN) algorithm, the Support Vector Machine (SVM) algorithm, and Naive Bayes (Nearest Neighbor (NNN) algorithm). Both Bayes (NB) and Random Forest (RF) algorithms require manual feature extraction and classification, which is inefficient and difficult to handle complex action recognition tasks.

[0004] To address this limitation, deep learning models have been introduced into HAR research, such as Recurrent Neural Networks (RNNs), Long Short-Term Memory Networks (LSTMs), and Transformers, which have significantly improved classification performance through automatic feature extraction. However, significant limitations remain: RNNs are prone to the vanishing gradient problem when processing long-sequence data; while LSTMs optimize memory capacity through gating mechanisms, their serial computation characteristics limit parallel efficiency; and the Transformer's self-attention mechanism lacks direct temporal dependencies and the ability to model continuity in time series. Therefore, an extended LSTM (xLSTM) model has been proposed, which significantly improves the model's storage capacity, computational efficiency, and ability to model dependencies in long-term series by introducing exponential gating and enhanced memory structures.

[0005] However, these models are opaque and difficult to explain, so Explainable Artificial Intelligence (XAI) has become an important research area aimed at improving the transparency of models.

[0006] Traditional decision tree models are widely adopted due to their inherent interpretability, but they usually fail to achieve the performance of deep learning models. In addition, existing interpretability methods usually cannot capture the complex nonlinear relationships between features in high-dimensional space, and the local linear assumption may not reveal the internal model details, such as the SHapley Additive exPlanations (SHAP) method and the Local Interpretable Model-agnostic Explanations (LIME) method.

[0007] To solve these limitations, a Neural-Backed Decision Trees (NBDT) based model is derived, which uses a differentiable decision sequence to replace the final linear layer of the neural network, establishing a global interpretability framework. Unlike the local assumption of SHAP and LIME, NBDT decomposes the complex decision-making process into explicit, step-by-step decision paths, while retaining the high performance of neural networks, showing significant advantages in balancing interpretability and model performance. SUMMARY

[0008] The purpose of the technical solution of the present application is to improve the classification accuracy of multi-dimensional time series tasks while maintaining the interpretability of the model, quantifying the contribution of input features to decision nodes, and thus providing transparent and trustworthy decision insights.

[0009] In order to achieve the above-mentioned purpose of the application, the technical solution of the present application provides an interpretable human behavior recognition method based on time-feature decoupled decision path, which comprises the following steps:

[0010] Based on the public human activity dataset, the corresponding time series samples are obtained by corresponding sliding time window segmentation and standardization preprocessing, and a multi-source time series standardized behavior recognition dataset with training set and validation set is constructed;

[0011] The training set is input into the xLSTM network, compressed into a fixed-length feature vector through a time average pooling layer, and input into a fully connected layer to obtain an original classification category. According to the xLSTM network, the fixed-length feature vector, and the original classification category, a corresponding fully connected layer weight matrix is assumed, and the fully connected layer weight matrix is combined with the original classification category to obtain a standardized matrix through L2 normalization. The original classification category corresponding to the leaf node is defined, and the row vector of the fully connected layer weight matrix is assigned to each leaf node. From the leaf node, the feature vectors of the parent nodes are generated by merging two child nodes in turn from bottom to top, and a hierarchical decision path is obtained. A hybrid loss function is designed by dynamically weighting the classification cross-entropy loss and the soft probability loss of the hierarchical decision path. The training set and the validation set are used to optimize the classification performance and the decision logic interpretability through back propagation. In the early stage, the classification accuracy is focused on, and in the later stage, the hierarchical decision tree construction is strengthened. The early stopping strategy is applied to terminate the training when the validation accuracy does not improve for multiple rounds. The internal nodes represent the intermediate state of the hierarchical decision path, and finally a binary tree structure is obtained to construct a hierarchical decision tree.

[0012] The dot product similarity of the fixed-length feature vector and the child node weight is recursively calculated from the root node of the hierarchical decision tree. The dot product similarity is converted into left and right branch probabilities through Softmax, and the path probability is calculated by multiplying all the transition probabilities on the path. Until the leaf node is reached, the leaf node with the maximum path probability is finally selected as the classification result.

[0013] The integrated gradient technique is used to calculate the contribution value of the fixed-length feature vector based on the intermediate nodes of the hierarchical decision tree and the feature dimension contribution degree. The key time step selection criterion is defined, the time slice with the highest average feature contribution degree is extracted, the average value of the absolute value of the feature contribution degree at each time step is calculated, and the key time step with the highest average contribution degree is selected. The top pre-set number of features with the highest contribution degree are selected in the selected key time step, and the SmoothGrad technique is used for attribution to obtain the attribution result.

[0014] The original classification category, the path probability, the transition probability, the top pre-set number of features, and the corresponding contribution degree are used to generate a decision tree topology map through visual annotation, realize the intuitive visual display of the hierarchical decision path, and realize the interpretable human behavior recognition.

[0015] Preferably, the disclosed human activity dataset is used to describe IMU data, heart rate signals, multi-dimensional sensor feature data, and daily life activity data, including PAMAP2 dataset, OPPORTUNITY dataset, and HAPT dataset.

[0016] Preferably, the window size of the sliding time window of the sliding time window segmentation is 30-1024, the step size is 12.5%-50% of the window size, the standardization preprocessing includes Z-score standardization or normalization, and the standardization preprocessing is divided into a training set and a validation set according to a preset proportion, and Gaussian noise is added to the divided training set, and finally the corresponding time sequence sample is obtained.

[0017] Preferably, the xLSTM network includes a single sLSTM block and a plurality of mLSTM blocks connected through residual stacking.

[0018] Preferably, the sLSTM block controls the input amount of the training set by using exponential operation, and the formula related to the exponential operation is as follows:

[0019]

[0020] In the formula, i t is the input gate for controlling the input amount of new information, represents the weight vector of the input gate, x t represents the input feature vector at time step t, r i represents the hidden state h t-1 at t-1 time step for controlling the contribution of the current input gate, h t-1 represents the hidden state, b i represents the bias term.

[0021] Preferably, the mLSTM block updates the cell state matrix by using matrix memory through matrix outer product, and the formula related to the matrix memory is as follows:

[0022]

[0023] In the formula, C t is the cell state matrix, f t is the forget gate output capable of flexibly adjusting the degree of forgetting previous information, v t and are the value vector and the key vector at t time step, respectively, is the matrix outer product, i t is the input gate for controlling the input amount of new information.

[0024] Preferably, the parent node feature vector formula is as follows:

[0025]

[0026] In the formula, W left and W right are the normalized weight vectors of the left and right child nodes, respectively, W patent represents the aggregated feature vector of the parent node.

[0027] Preferably, the left branch probability, the path probability and the classification result are formulated as follows:

[0028] p left = Softmax(sim(x, W left )), p right = 1-p left

[0029]

[0030] In the formula, p k is a path node set reaching the kth leaf node, p(C k (n) | n) is the transition probability of node n selecting child node C k (n), sim(x, W left ) is the dot product similarity, and p is the classification result.

[0031] Preferably, the integrated gradient technique is used to calculate the contribution value of the fixed-length feature vector based on the intermediate node of the hierarchical decision tree and the feature dimension contribution, including:

[0032] The dot product of the fixed-length feature vector and the node weight is calculated as the node logit value, and the formula is as follows:

[0033] logit n (x) = w n ·x

[0034] In the formula, the logit value represents the dot product of the weight vector of node n and the fixed-length feature vector x.

[0035] The contribution value of the fixed-length feature vector based on the intermediate node is calculated in combination with the logit value.

[0036] Taking a zero tensor as the baseline input, the feature dimension contribution is calculated by integrating the gradient along the straight line path to the input feature x i , and the formula is as follows:

[0037]

[0038] In the formula, x b is the baseline value, IG n (x i ) is the integrated gradient contribution value of feature x i to the decision of node n, x i and x b,i are the i-th dimensional feature value of the actual input and the i-th dimensional feature value of the baseline input.

[0039] Preferably, the average value of the absolute value of the contribution degree of each time step feature is calculated, and the key time steps with the highest average contribution degree are screened out, including:

[0040] For a given feature f in the time segment, the importance score of the time step t corresponding to the given feature f is calculated, and the formula is as follows:

[0041]

[0042] In the formula, F represents the total number of features;

[0043] The formula for screening out the key time steps with the highest average contribution degree is as follows:

[0044]

[0045] Preferably, the SmoothGrad technology is used for attribution to obtain the attribution result, and the formula for selecting the top pre-set number of features with the highest contribution degree in the screened key time steps is as follows:

[0046]

[0047] In the formula, k is the pre-set number;

[0048] The formula for calculating the attribution result is as follows:

[0049]

[0050] In the formula, ∈ j is Gaussian noise with a mean of 0 and a standard deviation of σ, and M represents the number of perturbed samples.

[0051] The technical scheme of the present application proposes an interpretable human behavior recognition method based on time-feature decoupling decision path, which realizes decoupling analysis of time dimension and feature dimension by mapping the input feature contribution value of the time step to the intermediate node of the decision path, adopts a hierarchical decision path to decouple the model reasoning process, and quantifies the contribution degree of each feature by combining dynamic spatio-temporal attribution technology. This not only realizes excellent classification performance on PAMAP2 and other data sets, but also enhances the interpretability of the model. A mixed loss function is introduced for joint optimization, which provides an innovative method for in-depth analysis and understanding of the decision logic of complex time series models. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 An interpretable human behavior recognition framework based on time-feature decoupling decision path is provided for the embodiments of the present application;

[0053] Figure 2To provide the sLSTM variant internal structure in the xLSTM model of the neural network backbone for the embodiment of the application;

[0054] Figure 3 To provide the mLSTM variant internal structure in the xLSTM model of the neural network backbone for the embodiment of the application;

[0055] Figure 4 To provide the single sample hierarchical decision path graph and feature contribution label visualization result graph on the PAMAP2 dataset for the embodiment of the application;

[0056] Figure 5 To provide the single sample hierarchical decision path graph and feature contribution label visualization result graph on the OPPORTUNITY dataset for the embodiment of the application;

[0057] Figure 6 To provide the single sample hierarchical decision path graph and feature contribution label visualization result graph on the HAPT dataset for the embodiment of the application;

[0058] Figure 7 To provide the accuracy rate curve and loss curve of the training set and test set on the PAMAP2 dataset for the embodiment of the application;

[0059] Figure 8 To provide the accuracy rate curve and loss curve of the training set and test set on the OPPORTUNITY dataset for the embodiment of the application;

[0060] Figure 9 To provide the accuracy rate curve and loss curve of the training set and test set on the HAPT dataset for the embodiment of the application;

[0061] Figure 10 To provide the confusion matrix graph on the PAMAP2 dataset for the embodiment of the application;

[0062] Figure 11 To provide the confusion matrix graph on the OPPORTUNITY dataset for the embodiment of the application;

[0063] Figure 12 To provide the confusion matrix graph on the HAPT dataset for the embodiment of the application. DETAILED DESCRIPTION

[0064] The application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not used to limit the scope of the application. Furthermore, it should be understood that those skilled in the art can make various modifications or changes to the application after reading the content of the application, and these equivalent forms also fall within the scope of the appended claims of the application.

[0065] AsFigure 1 As shown, the embodiment of the present application provides an interpretable human behavior recognition method based on time-feature decoupling decision path, comprising the following steps:

[0066] Based on the public human activity dataset, the corresponding time sequence samples are obtained through corresponding sliding time window segmentation and standardization preprocessing, and a multi-source time sequence standardized behavior recognition dataset with training set and validation set is constructed, which is used as input for training, validation and interpretability analysis. The public human activity dataset is used to describe the IMU data, heart rate signal, multi-dimensional sensor feature data and daily life activity data.

[0067] The public human activity dataset covers IMU data, heart rate signal, multi-dimensional sensor feature data and daily life activity data, including PAMAP2 dataset, OPPORTUNITY dataset and HAPT dataset.

[0068] The PAMAP2 dataset, OPPORTUNITY dataset and HAPT dataset are respectively subjected to sliding time window segmentation and standardization preprocessing to generate time sequence samples which can be directly used as input. The standardization preprocessing includes Z-score standardization or normalization, and the standardization preprocessing is divided into training set and validation set according to a preset proportion. Gaussian noise is added to the divided training set, and finally the corresponding time sequence samples are obtained. The steps are as follows:

[0069] IMU data and heart rate signals of 9 subjects on chest, wrist and ankle are obtained from the PAMAP2 dataset, a sliding window with window size 1024 and step length 128 is used to segment the data, and the features are subjected to Z-score standardization. Gaussian noise with standard deviation of 0.01 is added to the divided training set (80% of the total data) to improve the generalization ability and avoid overfitting. The remaining 20% of the data is used as the validation set to monitor the accuracy such as accuracy and F1 score during the training process, and guide the early stopping strategy.

[0070] From the OPPORTUNITY dataset, 113-dimensional sensor features such as accelerometer and gyroscope data are selected, the original labels are converted into 17 categories of actions through label mapping, the window size is set to 50 and the overlap rate is set to 50% to generate samples, the samples are normalized to the interval [0, 1], and the training set and validation set are divided by the average method in the ratio of 8:2.

[0071] The accelerometer and gyroscope data of the waist smartphone in the HAPT dataset are extracted to generate a 561-dimensional feature vector, and 12 labels are included, including 6 basic actions and 6 transition actions. The data is divided by setting the window size to 30, and after Z-score standardization, the training and verification sets are divided in a 7:3 ratio. The training set is used to train the xLSTM network and the hierarchical decision tree to learn the mapping relationship between the time series features and the classification labels. The verification set is used for final evaluation, such as accuracy, F1 score, and explainability analysis, such as trait attribution and decision path visualization, to verify the effectiveness of the framework structure on new data.

[0072] The specific parameter settings of the sliding time window segmentation and standardization preprocessing are shown in Table 1:

[0073]

[0074] The training set of the multi-source time series standardization behavior recognition dataset is uniformly input into the xLSTM backbone network for feature extraction, and then the hierarchical decision tree is used to generate an interpretable path, finally realizing a human activity recognition framework that integrates high-precision classification and transparent decision-making.

[0075] The xLSTM backbone network includes two variants, sLSTM (Scalar LSTM) and mLSTM (Matrix LSTM), as shown in Figure 2 and Figure 3 The sLSTM exponential gating mechanism enhances the ability to capture time series features to build a hybrid architecture neural network backbone, which is beneficial for modifying and storing decisions, as follows:

[0076] c t =ft t c t-1 +i t z t

[0077] In the formula, c t is the cell state at time t that stores long-term memory, f t is the forget gate that determines the degree of forgetting of previous information by using the σ function or exponential operation to realize the degree of forgetting of the history cell state c t-1 , z t is the candidate cell state for updating the cell state c t after being processed by the activation function , f t and c t-1 , i t and z t are multiplied to realize the interaction of gating and state, and complete the cell state update.

[0078]

[0079] wherein, is the inactive forget gate state, r f , b f are the weight, recurrent weight, bias of the forget gate.

[0080]

[0081] wherein, i t is the input gate that controls the input of new information, represents the weight vector of the input gate, the operation of the input feature vector x t at time step t is realized by matrix multiplication, r i represents the corresponding recurrent weight for controlling the contribution of the hidden state h t-1 at time t-1 to the current input gate, b i is the input gate bias term, all of which participate in the forward propagation calculation and affect the regulation of new information input, that is, control the input of the training set.

[0082]

[0083] wherein, is the inactive candidate cell state, r z , b z are the weight, recurrent weight, bias of the candidate cell state, respectively.

[0084] m t = max (log(f t )+m t-1 , log(i t ))

[0085] wherein, m t is a stabilization term to prevent overflow of the exponential operation.

[0086]

[0087] f t ' = exp (log(f t )+m t-1 -m t )

[0088] wherein, i' t , f' t are the optimized input gate and forget gate outputs, which ensure numerical stability and are used for cell state and hidden state calculation subsequently; that is, the state of the input gate without exponentiation.

[0089] n t =f t n t-1 +i t

[0090]

[0091] In the formula, n t To enable the normalized state updated by combining the output of the forget gate and the input gate, used for stable hidden state calculation; h t The final hidden state, obtained through the interaction of cell state, query input, and normalized state, ensures stability and accuracy when handling complex memory state tasks; t For use in controlling the final hidden state h t Output gate o t , It is an inactive output gate state. r o b o Similarly, the weights, recursive weights, and biases of the output gates are also considered.

[0092] By combining the matrix memory of mLSTM to achieve parallel processing, a nonlinear modeling xLSTM deep feature extraction network is formed:

[0093]

[0094] v t =W v x t +b v

[0095] q t =W q x t +b q

[0096] In the formula, k t v t For the introduced key-value pairs, q t For query input, used for indexing, updating, and information retrieval in matrix memory; W k W v W q It is a weight matrix of keys, values, and queries; b k b v b q is the corresponding bias term; d is the input feature dimension, used to normalize the key vector and reduce the impact of variance.

[0097]

[0098] The gating logic of mLSTM is achieved through the weight matrix. W o Parallelization is realized, and high-dimensional time series data processing is adapted.

[0099] n t = f t n t-1 + i t k t

[0100]

[0101] wherein C t The matrix memory stores high-dimensional time series as a cell state matrix, n t , o t , The logic is consistent with sLSTM, and is adapted to the matrix operation scene.

[0102] The xLSTM network is constructed by stacking one sLSTM block and six mLSTM blocks in residual form (xLSTM Block Stack ( Figure 1 ): The second block uses sLSTM to process global dependencies, and the remaining six blocks stack mLSTM blocks to capture local dependencies. Finally, the features are compressed into a fixed-length feature vector through time-average pooling, and input into the fully connected layer to generate the original classification logits, providing a basis for the explainability analysis of the subsequent hierarchical decision tree.

[0103] According to the xLSTM network, the fixed-length feature vector, and the original classification category, it is assumed that the weight matrix of the fully connected layer of the xLSTM network is where D represents the input feature dimension in the fixed-length feature vector, and K represents the number of categories of the original classification category. L2 normalization is performed on the fully connected layer weight matrix to obtain the standardized matrix W', eliminating the difference in numerical range between the original classification categories.

[0104] The original classification category corresponding to the leaf node is defined, and the row vector of the fully connected layer weight matrix is assigned to each category of the leaf node. From the leaf node, the feature vectors of the parent nodes are generated by merging two child nodes from bottom to top, and finally the entire induced hierarchy is formed, i.e. the hierarchical decision path is generated. A hybrid loss function is designed to dynamically weight the classification cross-entropy loss and the soft probability loss of the hierarchical decision path. Through backpropagation, the classification performance and decision logic explainability are optimized simultaneously. In the early training stage, the classification accuracy is focused on, and in the later stage, the decision tree construction is strengthened. Early stopping strategy is applied to terminate training when validation accuracy does not improve for multiple rounds, avoiding overfitting. The internal nodes represent the intermediate state of the decision path, and the hierarchical decision tree with a binary tree structure is obtained. The formula of the parent node feature vector is as follows:

[0105]

[0106] where W left and W right are the normalized weight vectors of left and right child nodes, respectively, and W parent represents the aggregated feature vector of the parent node.

[0107] In the inference process, the fixed-length feature vector obtained after processing the xLSTM network is taken as the input data x, and the root node of the hierarchical decision tree is traversed from the root node, and the dot product similarity of the feature vector and the child node weight is calculated recursively to reflect the matching degree of the feature and the node. The greater the value, the more "important" the branch is:

[0108] sim(x, W left ) = x · W left

[0109] sim(x, W right ) = x · W right

[0110] The dot product similarity is converted into left and right branch probabilities through the Softmax function, and the product of all transition probabilities on the path is calculated as the path probability, as follows:

[0111] p left = Softmax(sim(x, W left )), p right = 1 - p left

[0112]

[0113] where P k is the path node set to reach the kth leaf node, and p(C k (n) | n) is the transition probability of node n to select child node C k (n).

[0114] Until the leaf node is reached, the leaf node with the maximum path probability is finally selected as the final classification result:

[0115]

[0116] The ensemble gradient technique is used for the intermediate nodes in the hierarchical decision tree, and the dot product of the fixed-length feature vector and the node weight is calculated as the node logit value:

[0117] logit n (x) = w n · x

[0118] where the logit value represents the dot product of the weight vector of node n and the fixed-length feature vector x. ​

[0119] The fixed-length feature vector is calculated based on the contribution value of the intermediate node logit value, with a zero tensor as the baseline input, along the baseline value x b to the input feature x i The straight-line path integral gradient is calculated, and the feature dimension contribution is calculated as follows:

[0120]

[0121] In the formula, IG n (x i ) is the integrated gradient contribution value of feature x i to node n decision, the larger the absolute value, the more important the feature, x i and x b,i are the actual input i-dimensional feature value and the baseline input i-dimensional feature value.

[0122] The key time step selection criterion is defined, and the time slice with the highest average feature contribution is extracted. The absolute value of the feature contribution of each time step is calculated for feature f, and the importance score of time step t is calculated as follows:

[0123]

[0124] In the formula, F represents the total number of features.

[0125] The key time step with the highest average contribution is selected as follows:

[0126]

[0127] The top k features with the highest contribution are selected in the filtered time step, and the formula is as follows:

[0128]

[0129] In order to reduce noise and improve the robustness of the attribution process, SmoothGrad technology is used to calculate multiple perturbed versions of the input, and then these versions are averaged to obtain more reliable attribution results. The formula is as follows:

[0130]

[0131] In the formula, ∈ j is Gaussian noise with mean 0 and standard deviation σ, and M represents the number of perturbed samples.

[0132] The original classification categories such as "Ascending stairs" and the path probability Prob value reaching each leaf are displayed on the leaves, the transition probability, the top k features with the highest contribution value and their contribution values and the attribution results constitute the visualization labels of the intermediate nodes, and finally the decision tree topology diagram is generated to realize the intuitive display of the hierarchical decision path, as shown in Figures 4-6 The category corresponding to the leaf with the maximum Prob value is the final classification result of the input data sample, which is displayed in a blue path.

[0133] As shown in Figure 4 , the classification path of the PAMAP2 dataset "Ascending stairs" clearly shows the feature contribution of the key nodes, such as the feature importance of ankle_imu_0 at t = 666 reaching 1.750, and the features of hand_imu_2 and hand_imu_11 also play a key role at this time step.

[0134] As shown in Figure 5 , the classification of the OPPORTUNITY dataset "Close dishwasher" depends on the InertialMeasurementUnit_RUA sensor feature.

[0135] As shown in Figure 6 , the decision path of the HAPT dataset "Walking_upstairs" labels the importance of features such as tBodyAccJerk-Max-1 at t = 26. These path charts show that this method can capture spatiotemporal features related to actions, providing a new idea for transparent decision making.

[0136] As shown in Figure 7 , the training accuracy quickly rises at the beginning and stabilizes at a high level, the validation accuracy has fluctuations but the overall trend is stable, and the training loss and validation loss gradually decrease with the increase of training rounds, reflecting the stability and generalization ability. As shown in Figure 8 , the training curve also presents the similar rules of the PAMAP2 dataset, indicating that the mixed loss function and optimization strategy effectively promote the convergence of the model. As shown in Figure 9 , the training curve also presents the similar rules of the PAMAP2 dataset, indicating that the mixed loss function and optimization strategy effectively promote the convergence of the model.

[0137] In the PAMAP2 dataset, the accuracy of this method is 96.89%, and the F1 score is 96.82%, which is better than Att-ResBiGRU and other methods. According to the confusion matrix, this method accurately identifies most of the samples in the PAMAP2 dataset (as shown in Figure 10As shown in the figure, six categories achieved 100% classification accuracy, accounting for half of the total categories. However, the classification accuracy for the activity "sitting" was relatively low at 79%, which may be attributed to the method being trained more on dynamic activities. Its ability to recognize static activities (such as sitting and standing postures) was weaker, especially when the distinguishing features between these activities were small.

[0138] On the OPPORTUNITY dataset, this method achieves an accuracy of 94.70% and an F1 score of 94.71%, outperforming methods such as xLSTM. Figure 11 As can be seen from the confusion matrix, the method makes accurate predictions for almost all categories, with negligible misclassification. In particular, the prediction accuracy reaches 100% for 75 samples of "Close drawer 3" and 133 samples of "Clean table," demonstrating the method's accurate identification ability for these activities.

[0139] On the HAPT dataset, our method achieves an accuracy of 98.12% and an F1 score of 98.08%, outperforming models such as 1DCNN-LSTM, highlighting the effectiveness of our approach. The confusion matrix of the HAPT dataset (…). Figure 12 The results show that this method achieves high accuracy in recognizing basic postures such as "Walking upstairs," "Walking downstairs," and "Laying," reaching 98.9%, 99.8%, and 99.4%, respectively. However, the classification accuracy for transitional behaviors such as "Stand to sit" and "Lie to stand" is significantly lower. This may be attributed to the unique motion characteristics of these transitional behaviors, making it difficult for the model to accurately identify these target categories. Overall, however, the model performs well in recognizing most activities.

[0140] To address the challenges of black-box behavior and insufficient interpretability in deep learning models, this invention provides an interpretable human behavior recognition method based on time-feature decoupled decision paths. This method integrates the xLSTM model into the NBDT framework for the first time, constructs a hierarchical path with spatiotemporal decoupling characteristics using differentiable decision sequences, thereby revealing the global logic of model decision-making. Furthermore, it introduces ensemble gradient technology to quantify the contribution of input features at the node level in the decision path, constructing an interpretable framework based on hierarchical decision paths and dynamic feature contribution analysis.

[0141] In human behavior recognition tasks, the method provides auxiliary support for the decision-making process of virtual reality interaction through hierarchical path visualization and feature attribution analysis, enhancing the explainability of time series data modeling and providing a reference for users to understand the decision-making mechanism of complex models. The hierarchical decision-making path and node-level explainability method design improves the classification accuracy of multi-dimensional time series tasks while maintaining the explainability of the model. The quantification of input features' contribution to decision nodes provides transparent and trustworthy decision insights, not only bridging the gap between performance and explainability, but also providing new research ideas and technical support for industrial metaverse and related fields.

Claims

1. An interpretable human behavior recognition method based on time-feature decoupling decision paths, characterized in that, Includes the following steps: Based on publicly available human activity datasets, corresponding time-series samples are obtained through corresponding sliding time window segmentation and standardized preprocessing, and a multi-source time-series standardized behavior recognition dataset with training and validation sets is constructed. The training set is input into an xLSTM network, compressed into a fixed-length feature vector by a time-averaged pooling layer, and then input into a fully connected layer to obtain the original classification categories. Based on the xLSTM network, the fixed-length feature vector, and the original classification categories, the weight matrix of the fully connected layer is assumed to correspond to the weight matrix of the original classification categories. The weight matrix of the fully connected layer is then L2 normalized to obtain a standardized matrix. Leaf nodes are defined to correspond to the original classification categories. The row vectors of the fully connected layer weight matrix are assigned to the leaf nodes of each category. Starting from the leaf node, two child nodes are merged sequentially from bottom to top to generate the parent node feature vector, resulting in a hierarchical decision path. A hybrid loss function is designed by dynamically weighting the classification cross-entropy loss and the soft probability loss of the hierarchical decision path. The classification performance and the interpretability of the decision logic are simultaneously optimized by backpropagation using the training and validation sets. Initially, the focus is on classification accuracy, and later the construction of the hierarchical decision tree is strengthened. An early stopping strategy is applied to terminate training when the validation accuracy has not improved after multiple rounds. Internal nodes represent the intermediate states of the hierarchical decision path. Finally, a binary tree structure is obtained to construct a hierarchical decision tree with a tree structure. Starting from the root node of the hierarchical decision tree, the dot product similarity between the fixed-length feature vector and the weights of the child nodes is recursively calculated. The dot product similarity is then converted into left and right branch probabilities using Softmax. The path probability is obtained by multiplying all transition probabilities along the path until the leaf node is reached. Finally, the leaf node with the highest path probability is selected as the classification result. The integrated gradient technique is used to calculate the contribution value of fixed-length feature vectors based on the intermediate nodes of hierarchical decision trees and the contribution of feature dimensions. The key time step selection criteria are defined, the time segments with the highest average feature contribution are extracted, the average absolute value of feature contribution at each time step is calculated, and the key time steps with the highest average contribution are selected. Within the selected key time steps, the top preset number of features with the highest contribution are selected, and the SmoothGrad technique is used for attribution to obtain the attribution results. Based on the original classification category, path probability, transition probability, pre-set number of features, and corresponding contribution, a decision tree topology is generated through visual annotation, enabling an intuitive visualization of hierarchical decision paths and thus achieving interpretable human behavior recognition.

2. The interpretable human behavior recognition method based on time-feature decoupling decision path as described in claim 1, characterized in that, The sliding time window segmentation has a window size of 30-1024 and a step size of 12.5%-50% of the window size. The standardization preprocessing includes Z-score standardization or normalization and dividing the standardization preprocessing into training and validation sets according to a preset ratio. Gaussian noise is added to the divided training set to finally obtain the corresponding time series samples.

3. The interpretable human behavior recognition method based on time-feature decoupling decision path as described in claim 1, characterized in that, The xLSTM network consists of a single sLSTM block and multiple mLSTM blocks connected by residual stacking.

4. The interpretable human behavior recognition method based on time-feature decoupling decision path as described in claim 3, characterized in that, The sLSTM block uses exponential operations to control the input amount of the training set. The relevant formulas for the exponential operations are as follows: In the formula, i t This is the input gate for controlling the amount of new information input in exponential operations. The weight vector representing the input gate, x t r represents the input feature vector at time step t. i This represents the hidden state h used to control time t-1. t-1 The recursive weight h corresponding to the contribution of the current input gate t-1 Represents the hidden state, b i This indicates the bias term.

5. The interpretable human behavior recognition method based on time-feature decoupling decision path as described in claim 3, characterized in that, The mLSTM block uses matrix memory to update the cell state matrix through matrix outer product. The relevant formula for matrix memory is as follows: In the formula, C t Let f be the cell state matrix. t To allow for flexible adjustment of the forget gate output based on the degree of forgetting of previous information, v t and Let be the value vector and key vector at time step t, respectively. For the outer product of matrices, i t This is the input gate for controlling the amount of new information input in exponential operations.

6. The interpretable human behavior recognition method based on time-feature decoupling decision path as described in claim 1, characterized in that, The formula for the parent node's feature vector is as follows: In the formula, W left and W right These are the normalized weight vectors of the left and right child nodes, respectively, W. parent This represents the feature vector after aggregation of the parent node.

7. The interpretable human behavior recognition method based on time-feature decoupling decision path as described in claim 1, characterized in that, The formulas for the left and right branch probabilities, the path probabilities, and the classification results are as follows: In the formula, P k Let p(C) be the set of path nodes leading to the k-th leaf node. k (n)|n) selects a child node C for node n. k The transition probability of (n), sim(x,W) left ) represents the dot product similarity. The classification result is as described.

8. The interpretable human behavior recognition method based on time-feature decoupling decision path as described in claim 1, characterized in that, The contribution value of a fixed-length feature vector based on the intermediate nodes of a hierarchical decision tree and the contribution of its feature dimensions are calculated using ensemble gradient techniques, including: The dot product of a fixed-length feature vector and its node weights is used to calculate the node's logit value, as shown in the following formula: logit n (x)=w n ·x In the formula, the logit value represents the dot product of the weight vector of node n and the fixed-length feature vector x; The contribution value of a fixed-length feature vector based on intermediate nodes is calculated by combining the logit value; Using the zero tensor as the baseline input, along the input feature x i The integral gradient along the straight path is used to calculate the contribution of the feature dimension, as shown in the following formula: In the formula, x b As the baseline value, ID n (x i ) is a feature x i The ensemble gradient contribution value for the decision of node n, x i and x b,i Let i be the i-th eigenvalue of the actual input and the i-th eigenvalue of the baseline input.

9. The interpretable human behavior recognition method based on time-feature decoupling decision path as described in claim 1, characterized in that, The calculation of the mean of the absolute values ​​of the feature contributions of each time step and the selection of the key time steps with the highest average contribution include: For a given feature f in the time segment, calculate the importance score of the time step t corresponding to the given feature f, as follows: In the formula, F represents the total number of features; The formula for selecting the key time step with the highest average contribution is as follows:

10. The interpretable human behavior recognition method based on time-feature decoupling decision path as described in claim 1, characterized in that, In the process of selecting the top preset number of features with the highest contribution within the key time steps after screening, and using SmoothGrad technology for attribution to obtain the attribution results, the formula for selecting the top preset number of features with the highest contribution is as follows: In the formula, k is the preset number; The formula for calculating the attribution result is as follows: In the formula, ∈ j It is Gaussian noise with a mean of 0 and a standard deviation of σ, where M represents the number of perturbed samples.

Images