Human body behavior recognition method based on space-time dynamic attention mechanism of nested structure

Abstract

The invention discloses a human body behavior recognition method based on a nested structure spatio-temporal dynamic attention mechanism. The method comprises the steps of numerical mapping, position coding, spatio-temporal information fusion, behavior feature extraction and behavior activity prediction. According to the technical scheme, the human behavior recognition model of the lightweight nested structure is provided, the recurrent neural network is applied to two dimensions of sensor signals to extract the dynamic and time dependency relationship between the sensors, meanwhile, the dynamic association of the wearable sensors in the space dimension and the time dimension is considered, and the recognition accuracy of the wearable sensors is improved. And a weight sharing technology is introduced, so that the network scale and the reasoning efficiency are greatly reduced.

Description

Technical Field

[0001] The present invention relates to the field of information retrieval technology, and in particular to a human behavior recognition method based on a nested structured spatiotemporal dynamic attention mechanism. Background Art

[0002] Human activity recognition (HAR) refers to the task of identifying various types of daily human activities, which is of great significance in many real-world applications, such as healthcare, smart homes, manufacturing, and fitness tracking. Therefore, many researchers have proposed various HAR methods to achieve ubiquitous, long-term continuous, and accurate monitoring.

[0003] Traditional methods widely use camera-based signals and wireless signals as sensing media to identify human activities. However, these methods are limited by fixed environments, and their high complexity hinders their widespread implementation and continuous perception. To address these problems, a large number of recognition methods based on wearable sensors have been studied, and most of the measurement properties of these sensors are also related to the user's movement.

[0004] Early wearable-based approaches relied heavily on handcrafted features and adopted traditional machine learning methods such as support vector machines (SVM), decision trees (DT), and adaboost to achieve the purpose of classifying human activities. However, these methods require domain-specific knowledge and expertise, making them generally time-consuming and labor-intensive. In addition, handcrafted features may not capture the full complexity and variability of human activities, resulting in limited performance and generalization capabilities.

[0005] Recent methods address these limitations by using deep learning techniques to automatically extract features from complex data structures. For example, Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) are widely used in human action recognition tasks due to their strong spatial and temporal feature extraction capabilities. In order to improve HAR performance, some technicians use attention models to learn spatial and temporal dependencies in data. However, despite achieving high recognition accuracy, these methods face the problem of increased computational complexity due to the excessive number of parameters. Summary of the invention

[0006] Based on the above problems, the present invention proposes a human behavior recognition method based on a nested structured spatiotemporal dynamic attention mechanism. Different behaviors are dominated by different sets of sensors only, and the correlation changes dynamically over time, reflecting the core of the time dependency of spatial relationships. This scheme applies the RNN recurrent neural network to the two dimensions of sensor signals respectively to extract the dynamic and time dependency relationships between sensors. The specific technical scheme is as follows.

[0007] The human behavior recognition method based on the nested structure spatiotemporal dynamic attention mechanism includes the following steps:

[0008] Step S1: numerical mapping, collecting multiple wearable sensor signals, dividing the signals into signal segments of fixed length and performing numerical mapping preprocessing, and projecting the signal segments into a hidden space;

[0009] Step S2: Position encoding, the segment data is input into a lightweight nested structured human behavior recognition model, the model encoding layer encodes the relative position or location of the sensors in the sensor array based on sine and cosine encoding functions, distinguishes sensors according to spatial proximity, and embeds the position encoding information into the mapped segment data;

[0010] Step S3: spatiotemporal information fusion, the human behavior recognition model information fusion module applies two RNN network modules to the time dimension and sensor dimension of the two-dimensional segment data respectively to obtain spatiotemporal features of fixed length;

[0011] Step S4: behavior feature extraction, using the attention mechanism of the human behavior recognition model to calculate the weight value of the spatiotemporal feature vector at each time stamp, the gated recurrent network controls the impact of the spatiotemporal features of each time step on the behavior, and obtains the relevant behavior feature vector based on the behavior time dynamic dependency characteristics;

[0012] Step S5: Behavior activity prediction: Based on the obtained behavior feature vector, the trained model fully connected network predicts the prediction values ​​of different behaviors and identifies human behavior.

[0013] Specifically, the acquisition of spatiotemporal features in step S3 specifically includes: utilizing weight sharing to make the output vector of the RNN network module in the time dimension and the space dimension as the input of the next time step, extracting the spatial correlation relationship between sensors at each time step through continuous iteration, and obtaining a spatiotemporal feature of a fixed length by modeling the time dependency of the spatial correlation at each time step.

[0014] Specifically, the human behavior recognition model information fusion module is a nested structure, which uses minimal parameters and computing resources to extract behavior-related spatiotemporal features from wearable sensor signals, specifically including:

[0015] Given N wearable sensor sample data X∈RTxN , using recurrent neural network layers applied to the time dimension T and spatial dimension N of the input data X respectively;

[0016] At initialization, it is assumed that the network hidden state vector h T and h N The sub-table is the output vector of the recurrent network in the time dimension and space dimension, and we get:

[0017]

[0018] in, and They represent the intermediate vector and weight matrix of the network in the time dimension and space dimension respectively, which are obtained through training.

[0019] Specifically, the output vector of the time dimension is a behavior-related spatiotemporal feature.

[0020] Specifically, the human behavior recognition model attention mechanism combines the attention mechanism and gate control to construct an attention-enhanced RNN network module, and adjusts the importance of the spatiotemporal features extracted at each time step for classification. Specifically, the attention module is used to generate the extracted spatiotemporal features and the attention score corresponding to each timestamp a=(a1,...,a t ,...,a T ), and apply the weight value to the GRU network, expressed as:

[0021] C t =(1-a t )C t-1 +a t C′ t ;

[0022] C′ t =g(W·C t-1 +U·h (t) + b);

[0023] Among them, W and U are the linear transformation parameters of the previous and current time steps, and b is the bias term.

[0024] Specifically, the step S5 also includes: adopting a marginal-based loss function and using a data set to train the lightweight nested structure human behavior recognition model.

[0025] Specifically, the margin-based loss function is as follows: the angle θ between the network weight W and the input feature x is calculated at the network output layer, the accumulated loss error between the vectors is converted into a loss function related to the angle θ, and a penalty value m is added to the angle θ. The margin-based loss function is expressed as:

[0026]

[0027] Beneficial effects of the present invention: The present invention proposes a human behavior recognition method based on a nested structured spatiotemporal dynamic attention mechanism, and applies a recurrent neural network to the two dimensions of sensor signals respectively to extract the dynamic and temporal dependencies between sensors. At the same time, the interaction of spatial dimension and temporal dimension information is achieved through weight sharing, which simplifies the network for spatiotemporal feature extraction. Subsequently, a gating technique with attention scores is used to control how much information is included in the extracted features at each time step. At the same time, in order to enhance intra-class compactness and inter-class differences, the present invention uses a margin-based loss instead of a softmax loss to reduce the misjudgment rate. Compared with traditional methods, the present invention simultaneously considers the dynamic association of wearable sensors in the spatial and temporal dimensions, and introduces weight sharing technology to propose a lightweight network that effectively captures spatiotemporal features, greatly reducing the network scale and reasoning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the human behavior recognition method based on the nested structure spatiotemporal dynamic attention mechanism of the present invention;

[0029] Figure 2 Schematic diagram of a human behavior recognition model with a lightweight nested structure in an embodiment of the present invention;

[0030] Figure 3 Schematic diagram of an attention-enhanced RNN network module in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described with reference to the accompanying drawings.

[0032] In order to achieve high performance in terms of accuracy and computational complexity, this paper proposes a human behavior recognition method based on a nested spatiotemporal dynamic attention mechanism, the process of which is as follows: Figure 1 As shown, the following steps are included:

[0033] Step S1: numerical mapping, collecting multiple wearable sensor signals, dividing the signals into signal segments of fixed length and performing numerical mapping preprocessing, and projecting the signal segments into the hidden space;

[0034] Step S2: Position encoding, the segment data is input into a lightweight nested structured human behavior recognition model, the model encoding layer encodes the relative position or location of the sensors in the sensor array based on sine and cosine encoding functions, distinguishes sensors according to spatial proximity, and embeds the position encoding information into the mapped segment data;

[0035] Step S3: spatiotemporal information fusion, the human behavior recognition model information fusion module applies two RNN network modules to the time dimension and sensor dimension of the two-dimensional segment data respectively to obtain spatiotemporal features of fixed length;

[0036] Step S4: behavior feature extraction, using the attention mechanism of the human behavior recognition model to calculate the weight value of the spatiotemporal feature vector at each time stamp, the gated recurrent network controls the impact of the spatiotemporal features of each time step on the behavior, and obtains the relevant behavior feature vector based on the behavior time dynamic dependency characteristics;

[0037] Step S5: Behavior activity prediction: Based on the obtained behavior feature vector, the trained model fully connected network predicts the prediction values ​​of different behaviors and identifies human behavior.

[0038] In this embodiment, the acquisition of spatiotemporal features in step S3 specifically includes: utilizing weight sharing to make the output vector of the RNN network module in the time dimension and the space dimension the input of the next time step, extracting the spatial correlation relationship between sensors at each time step through continuous iteration, and obtaining a spatiotemporal feature of a fixed length by modeling the time dependency of the spatial correlation at each time step.

[0039] In this embodiment, the human behavior recognition model information fusion module is a nested structure, which uses minimal parameters and computing resources to extract behavior-related spatiotemporal features from wearable sensor signals, specifically including:

[0040] Given N wearable sensor sample data X∈R TxN , using recurrent neural network layers applied to the time dimension T and spatial dimension N of the input data X respectively;

[0041] At initialization, it is assumed that the network hidden state vector h T and h N The sub-table is the output vector of the recurrent network in the time dimension and space dimension, and we get:

[0042]

[0043] in, and They represent the intermediate vector and weight matrix of the network in the time dimension and space dimension respectively, which are obtained through training.

[0044] In this embodiment, the output vector of the time dimension is the spatiotemporal feature related to the behavior.

[0045] In this embodiment, the attention mechanism of the human behavior recognition model combines the attention mechanism and gate control to construct an attention-enhanced RNN network module, and adjusts the importance of the spatiotemporal features extracted at each time step for classification. Specifically, the attention module is used to generate the extracted spatiotemporal features and the attention score corresponding to each timestamp a=(a1,...,a t ,...,a T ), and apply the weight value to the GRU network, expressed as:

[0046] C t =(1-a t )C t-1 +a t C′ t ;

[0047] C′ t =g(W·C t-1 +U·h (t) + b);

[0048] Among them, W and U are the linear transformation parameters of the previous and current time steps, and b is the bias term.

[0049] In this embodiment, step S5 further includes: using a marginal-based loss function and using a data set to train the lightweight nested human behavior recognition model.

[0050] In this embodiment, the margin-based loss function is specifically as follows: the angle θ between the network weight W and the input feature x is calculated at the network output layer, the accumulated loss error between the vectors is converted into a loss function related to the angle θ, and a penalty value m is added to the angle θ, and the margin-based loss function is expressed as:

[0051]

[0052] In one embodiment, the lightweight nested human behavior recognition model architecture is as follows: Figure 2 As shown in the figure, first, the signals collected by multiple sensors are divided into segments of fixed length T. In the subsequent "value mapping", "position encoding", "time-space information fusion", "behavior feature extraction" and "behavior activity prediction", fixed-length behavior feature vectors are extracted from these segments. Then, the behavior category corresponding to the segment is output through the behavior recognition module.

[0053] The specific execution steps are as follows.

[0054] Numerical mapping: Given the preprocessed segments, a 1D CNN layer of size 1×1 is applied to project them into the latent space, thereby achieving better nonlinear representation capabilities.

[0055] Position encoding: Encode the relative position or location of sensors in the sensor array using sine and cosine based encoding functions, allowing the model to distinguish sensors based on their spatial proximity and embed the position encoding information into the mapped segment data in S1.

[0056] Spatiotemporal information fusion: Two RNN networks are applied to the time dimension and sensor dimension (spatial dimension) of the two-dimensional segment data respectively. The weight sharing technology is used to make the output vectors of the RNN in the time dimension and space dimension as the input of the cell in the next time step. Through continuous iteration, the spatial correlation relationship between sensors in each time step can be extracted. By modeling the time dependency of the spatial correlation in each time step, a fixed-length spatiotemporal feature is obtained for subsequent processing.

[0057] Behavioral feature extraction: Figure 3 As shown, the attention model is used to calculate the weight value of the spatiotemporal feature vector obtained in S3 at each timestamp, and the value is input into the gated recurrent network to control the influence of the spatiotemporal features of each time step on the behavior, introduce the temporal dynamic dependency characteristics of the behavior, and obtain the behavior-related feature vector, so as to obtain the behavior activity classification result later.

[0058] Behavior activity prediction: Given the extracted behavior feature vector, the prediction layer predicts the prediction value of different behaviors through a layer of fully connected network. In the present invention, a margin-based loss function is used as a metric to train the network.

[0059] In this embodiment, a nested information fusion module is constructed to efficiently extract behavior-related spatiotemporal features from wearable sensor signals using minimal parameters and computing resources. Specifically, given the sampled data X∈R of N wearable sensors, T x N , the present invention uses a recurrent neural network layer to be applied to the time dimension (T) and space dimension (N) of the input data X, and at the same time, it assumes that the network hidden state vector is h T and h N The subtable is the output vector of the recurrent network in the time dimension and space dimension. At this time, the network can be abbreviated as

[0060]

[0061] and The intermediate vectors and weight matrices of the network in the time dimension and the space dimension are respectively represented, which can be obtained through training. Since human activity recognition is essentially a sequence problem, the method of the present invention uses the output vector of the time dimension as the spatiotemporal feature of the behavior.

[0062] In this embodiment, an attention-enhanced RNN module is constructed, which combines the attention mechanism and gate control technology to adjust the importance of the spatiotemporal features extracted at each time step to achieve the purpose of classification. Specifically, the attention module is used to generate the extracted spatiotemporal features and the attention score corresponding to each timestamp a=(a1,...,a t ,...,a T ), the weight value is applied to the GRU network, and this module is represented as

[0063] C t =(1-a t )C t-1 +a t C′ t

[0064] C′ t =g(W·C t-1 +U·h (t) +b)

[0065] W and U are the linear transformation parameters of the previous and current time steps, and b is the bias term.

[0066] In this embodiment, a marginal-based loss function is designed, and the angle θ between the network weight W and the input feature x is calculated at the network output layer, and the accumulated loss error between the vectors is converted into a loss function related to the angle θ, that is, At the same time, in order to enhance the intra-class compactness and inter-class differences, it adds a penalty value m to the angle θ. At this time, the loss function is expressed as

[0067]

[0068] In this embodiment, the recognition model of the invention is trained using a training set to obtain a trained behavior recognition model.

[0069] The present invention has shown and described the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A human behavior recognition method based on a nested spatiotemporal dynamic attention mechanism, characterized in that: The following steps are involved: Step S1: numerical mapping, collecting multiple wearable sensor signals, dividing the signals into signal segments of fixed length and performing numerical mapping preprocessing, and projecting the signal segments into a hidden space; Step S2: Position encoding, the segment data is input into a lightweight nested structured human behavior recognition model, the model encoding layer encodes the relative position or location of the sensors in the sensor array based on sine and cosine encoding functions, distinguishes sensors according to spatial proximity, and embeds the position encoding information into the mapped segment data; Step S3: spatiotemporal information fusion, the human behavior recognition model information fusion module applies two RNN network modules to the time dimension and sensor dimension of the two-dimensional segment data respectively to obtain spatiotemporal features of fixed length; Step S4: behavior feature extraction, using the attention mechanism of the human behavior recognition model to calculate the weight value of the spatiotemporal feature vector at each time stamp, the gated recurrent network controls the impact of the spatiotemporal features of each time step on the behavior, and obtains the relevant behavior feature vector based on the behavior time dynamic dependency characteristics; Step S5: Behavior activity prediction: Based on the obtained behavior feature vector, the trained model fully connected network predicts the prediction values ​​of different behaviors and identifies human behavior.

2. The human behavior recognition method based on nested structure spatiotemporal dynamic attention mechanism according to claim 1 is characterized in that: The acquisition of spatiotemporal features in step S3 specifically includes: utilizing weight sharing to make the output vector of the RNN network module in the time dimension and the space dimension the input of the next time step, extracting the spatial correlation relationship between sensors at each time step through continuous iteration, and obtaining a spatiotemporal feature of a fixed length by modeling the time dependency of the spatial correlation at each time step.

3. The human behavior recognition method based on nested structure spatiotemporal dynamic attention mechanism according to claim 1 is characterized in that: The human behavior recognition model information fusion module is a nested structure, which uses minimal parameters and computing resources to extract behavior-related spatiotemporal features from wearable sensor signals, specifically including: Given N wearable sensor sample data x∈R TxN , using recurrent neural network layers applied to the time dimension T and spatial dimension N of the input data X respectively; At initialization, it is assumed that the network hidden state vector h T and h N The sub-table is the output vector of the recurrent network in the time dimension and space dimension, and we get: in, and They represent the intermediate vector and weight matrix of the network in the time dimension and space dimension respectively, which are obtained through training.

4. The human behavior recognition method based on nested structure spatiotemporal dynamic attention mechanism according to claim 3 is characterized in that: The output vector of the time dimension is the spatiotemporal feature related to the behavior.

5. The human behavior recognition method based on nested spatiotemporal dynamic attention mechanism according to claim 1 is characterized in that: The human behavior recognition model attention mechanism combines the attention mechanism and gate control to construct an attention-enhanced RNN network module, and adjusts the importance of the spatiotemporal features extracted at each time step for classification. Specifically, the attention module is used to generate the extracted spatiotemporal features and the attention score corresponding to each timestamp a=(a1,...,a t ,...,a T ), and apply the weight value to the GRU network, expressed as: C t =(1-a t )C t-1 +a t C′ t ; C′ t =g(W·C t-1 +U·h (t) +b); Among them, W and U are the linear transformation parameters of the previous and current time steps, and b is the bias term.

6. The human behavior recognition method based on nested structure spatiotemporal dynamic attention mechanism according to claim 1 is characterized in that: The step S5 also includes: adopting a marginal-based loss function and using a data set to train the lightweight nested structure human behavior recognition model.

7. The human behavior recognition method based on nested spatiotemporal dynamic attention mechanism according to claim 6 is characterized in that: The margin-based loss function is specifically as follows: the angle θ between the network weight W and the input feature x is calculated at the network output layer, the accumulated loss error between the vectors is converted into a loss function related to the angle θ, and a penalty value m is added to the angle θ. The margin-based loss function is expressed as:

Images