Human behavior recognition method and system based on self-attention active contrastive coding

Abstract

The present application relates to the technical field of behavior recognition, and especially relates to a human behavior recognition method and system based on self-attention active contrast coding, comprising collecting human position data through a sensor; using a learning framework based on self-attention active contrast coding to learn action representation for classification; fusing a convolutional neural network unit with a self-attention mechanism at an input end to learn local context features and time dynamics features and capture the relationship between time points in an input data sample; measuring the uncertainty of a sample under an inductive consistency prediction algorithm through information quality; and measuring the uncertainty through sample posterior probability using interval sampling measurement. The self-attention active contrast coding method of the present application firstly adds a self-attention technology in a pre-training stage to learn weight coefficients for important representations capable of representing actions; secondly, a difficult sample prone to confusion is mined based on an active learning method and used for training model adjustment.

Description

Technical Field

[0001] This invention relates to the field of behavior recognition technology, and in particular to a method and system for human behavior recognition based on self-attention active contrastive coding. Background Technology

[0002] The increasing use of mobile devices generates massive data streams that can be used to better understand human behavior and potentially have a significant impact on human health management. Sensor-based human activity recognition is a crucial area of ​​human behavior perception, acquiring raw time-series data from embedded sensors to infer complex human activities. However, the sensor data streams from mobile devices are abstract and difficult for humans to interpret. One of the major obstacles to integrating data-driven systems into applications is the difficulty in collecting large amounts of labeled data. The emergence of unsupervised and self-supervised learning techniques, using unlabeled data as an alternative source of training data, has been used to overcome the limitations of a lack of labels; self-supervised learning, a method for training models on unlabeled data, has become an increasingly popular framework for training deep neural networks; and contrastive learning, as a form of self-supervised learning, reduces the cost of labeling large datasets and has recently been widely applied in fields such as computer vision and natural language processing.

[0003] Traditional contrastive learning struggles to directly capture spatiotemporal contextual features from sensors. Therefore, this invention introduces self-attention technology into the encoding part of the contrastive learning model. The self-attention-based contrastive learning model can eliminate repetitive structures, generate high-dimensional features for classification, and capture the importance of sensor patterns and locations in predicting different activity categories. In action recognition using contrastive learning, data selection is a crucial operation for model updates. Selecting information-rich and easily confused samples based on certain criteria facilitates model fine-tuning. Active learning is an effective query strategy in traditional machine learning methods. However, the most effective and theoretically sound active learning algorithms rely on the assumption of limited model class capacity, making it unpopular in deep learning. Based on this limitation, this invention does not introduce active learning into the pre-training process of contrastive learning but instead applies it to the model fine-tuning stage. A multi-criteria information metric based on active learning is used to mine information-rich difficult samples in unlabeled datasets. Summary of the Invention

[0004] To address the shortcomings of existing algorithms, this invention is based on the Active Contrastive Coding Method with Self-Attention (ACSC+Self-Attention). First, self-attention technology is added during the pre-training stage to learn weight coefficients for important representations that can represent actions. Second, an active learning method is used to mine easily confused difficult samples and use them to train and adjust the model.

[0005] The technical solution adopted in this invention is: a human behavior recognition method based on self-attention active contrastive coding, comprising the following steps:

[0006] Step 1: Collect human body position data through sensors and construct a multi-axis data aggregation matrix based on the time step;

[0007] Step 2: Use a self-attention-based active contrastive encoding learning framework to learn action representations for classification;

[0008] Furthermore, specifically including:

[0009] Step 21: Apply two transformations to each example using the enhancement module. The input data X generates two views for each sample in the batch by applying the two transformations. As a positive sample, Negative samples;

[0010] Step 22: The encoder part f(·) based on the neural network will combine the two views. Mapping to a low-dimensional representation yields and

[0011] Furthermore, the encoder employs a CNN-based self-attention encoder;

[0012] Step 23: Using the projection head g(·) of the multilayer perceptron in the hidden layer, map the representation learned by the encoder to the space where the contrastive loss is applied, to obtain z. i =g(h i ) and z j =g(h j );

[0013] Step 24: Define a contrastive loss function for the contrast task, using normalized temperature-scaled cross-entropy loss as the loss function, and using cosine similarity to measure the two augmented instances. and Similarity between them;

[0014] Furthermore, the formula for cosine similarity is:

[0015]

[0016] Where ||·||2 is the l2 norm;

[0017] Furthermore, the loss function is defined as:

[0018]

[0019] In the formula, τ refers to the temperature parameter of softmax.

[0020] Step 3: Integrate a convolutional neural network unit with a self-attention mechanism at the input end to learn local contextual features and temporal dynamic features, and capture the relationship between time points in the input data sample;

[0021] Furthermore, specifically including:

[0022] First, a one-dimensional convolutional neural network is used as the backbone model, including batch normalization and ReLU activation; second, the output of the convolutional neural network is passed through an encoder module based on a self-attention mechanism; finally, the encoded features are flattened.

[0023] Furthermore, specifically including:

[0024] Assuming that after one-dimensional convolution, augmented data is generated at multiple time points T. The characterization is Attention score α t (t = 1, 2, ..., T) represents the result obtained from the convolutional layer. The contribution of the feature representation used subsequently is calculated by formula (5):

[0025]

[0026] in, express The transpose of W u and W v It is the weight matrix of the self-attention module, and the softmax(·) function;

[0027] The resulting feature vector S, encoded by the self-attention mechanism, is represented by the attention score α. t and The linear weighted sum is shown in formula (6):

[0028]

[0029] Assuming that after one-dimensional convolution, augmented data is generated at multiple time points T. The characterization is The output after passing through an attention layer is As shown in formula (7),

[0030]

[0031] Among them, W att It is the weight matrix that constitutes the attention module;

[0032] The output obtained It is used to compare with a trainable parameter matrix u used to represent context information, which is randomly initialized to obtain... The allocation coefficients are then normalized using softmax, as shown in formula (8):

[0033]

[0034] Here, the score function can be any comparison function; the final normalized vector S is all... The weighted sum is shown in formula (9):

[0035]

[0036] The attention score obtained from learning represents the score achieved by the convolutional layer. In terms of contribution to the feature representation used in subsequent layers, the self-attention block uses dot-product-based attention scores to transform the feature values ​​at each time step.

[0037] Step 4: Measure the uncertainty of the sample using the inductive consistency prediction algorithm through information quality; measure the uncertainty using the posterior probability of the sample through interval sampling.

[0038] Furthermore, specifically including:

[0039] Assuming the sample set U is unlabeled d An unlabeled sample X m The corresponding class label is Y m The p-value is used to quantify the information quality of a data instance. The information quality of a sample is defined as:

[0040]

[0041] in, and This represents the first and second largest p-values ​​of the m-th sample;

[0042] The uncertainty of sampling interval is defined as:

[0043] I MS (X m |U d )=P(Y m =l1|X m ;M)-P(Y m=l2|X m ;M) (11)

[0044] Where l1 and l2 represent the most likely class labels predicted under the given model; the final sample X m The uncertainty measure is:

[0045]

[0046] Furthermore, a system for human behavior recognition based on self-attention active contrastive coding includes:

[0047] The task definition module is used for sensors to collect human position data and construct a multi-axis data aggregation matrix based on the time step.

[0048] A learning framework module based on self-attention active contrastive encoding for action representations in classification;

[0049] The self-attention mechanism module is used to learn local contextual features and temporal dynamic features to capture the relationship between time points in the input data sample.

[0050] The query strategy model based on active learning is used to measure the uncertainty of samples by information quality under the inductive consistency prediction algorithm; and to measure uncertainty by using the posterior probability of samples through interval sampling.

[0051] The beneficial effects of this invention are:

[0052] 1. An effective self-attention mechanism is introduced into the feature representation of contrastive learning, which fully considers the local context and temporal dynamics of time series, constructs an effective feature representation for classification, and improves the performance of pre-trained models.

[0053] 2. Considering the time and effort required for sample labeling, a multi-standard information metric based on active learning is adopted to identify and label the samples with the most information in the large-scale dataset, and to update the model with the fewest labeled samples, thereby improving the robustness of the model. Attached Figure Description

[0054] Figure 1 This is a flowchart of the human behavior recognition method based on self-attention active contrastive coding of the present invention;

[0055] Figure 2 These are the comparison results of the contrastive learning and self-attention contrastive learning of this invention on four different datasets;

[0056] Figure 3 This invention relates to the effects of visual contrastive learning and self-attentional contrastive learning on UCI and UTD;

[0057] Figure 4 This is a comparison of the different selection mechanisms of the present invention. Detailed Implementation

[0058] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0059] For data preparation, the ACSC+ self-attention model was applied to four different datasets: MotionSense, UCI, UTD, and WISDM to evaluate its performance.

[0060] For all considered datasets, the signals are segmented into fixed-size windows with 50% overlap. The appropriate window size is a task-specific parameter, tuned or selected based on prior knowledge to improve performance. Next, each dataset is divided into training, validation, and test sets. For example, 20% of the 30 users from the UCI dataset are randomly selected as the test set, 60% of the user data is used to train the network, and 20% is used as the validation set for validation and hyperparameter tuning. Experiments are based on 5x cross-validation results. It is important to emphasize that the same network architecture is used to evaluate model performance on different datasets, highlighting that improvements are actually due to the self-supervised training process.

[0061] like Figure 1 As shown, the human behavior recognition method based on self-attention active contrastive coding includes a task definition module, a learning framework module based on self-attention active contrastive coding, a self-attention mechanism module, and a query strategy module based on active learning.

[0062] Step 1, the steps for defining the task, include:

[0063] Suppose S = (s1, s2, s3, ...) is a set of sensors placed at different locations on the human body, recording data from multiple axes at a fixed sampling frequency (e.g., a three-axis accelerometer or gyroscope generates signals along the x, y, and z axes respectively); in a dataset containing sensor signal records from n axes, the sensor reading at time t is represented as a multidimensional time series x. t As shown in formula (1):

[0064]

[0065] Multi-axis data aggregated into a matrix X at a specific time step T can be represented as:

[0066] X = [x1, x2, ..., x T (6)

[0067] Where X represents the window data in the sequence, and T = 256 represents the window size; the window consists of a fixed number of data points representing the sensor signal at the corresponding timestamp, and the ultimate goal is to determine the window state label.

[0068] Step 2: Use a self-attention-based active contrastive encoding learning framework to learn action representations for classification. The learning framework consists of four parts:

[0069] Step 21, Data Augmentation: Assuming the batch consists of N instances, the augmentation module applies two transformations to each instance, resulting in 2N transformed views for the batch data. The original input data X is first transformed to generate two views for each sample in the batch. Two views obtained from the same instance are called positive samples, while views transformed from different instances are called negative samples.

[0070] Step 22: The encoder part f(·) based on the neural network maps the transformed samples to a low-dimensional representation. There are various choices for the network structure of the encoder design; for example, CNN-based encoders are widely used to learn the representation of visual data; the encoder in this embodiment is built using CNN + self-attention, and maps the data samples generated after data augmentation to the corresponding representation. and

[0071] Step 23: The projection head g(·) is a multilayer perceptron (MLP) with one hidden layer, consisting of two fully connected layers. The MLP maps the representation learned by the encoder to a space where a contrastive loss is applied, thus obtaining z. i =g(h i ) and z j =g(h j The projection head g(·) can output the encoder feature h. i Mapped to the potential high-dimensional space z i Using z i Instead of h i It is helpful to define contrast loss.

[0072] Step 24: Define a contrastive loss function for the contrastive task, using Normalized Temperature Scale Cross-Entropy Loss (NT-Xent) as the loss function, and cosine similarity is used to measure the difference between two augmented instances. and Similarities between them:

[0073]

[0074] Where ||·||² is the L2 norm, and the loss function for a pair of positive samples (i,j) is defined as:

[0075]

[0076] In the formula, z i and z j The characteristic output by the projection head g(·) is τ, which refers to the temperature parameter of softmax.

[0077] When using a pre-trained encoder-generated model M to train a classification model with available labeled data, fine-tuning is the final stage. The projection model is removed, and the encoder is used for feature extraction. It is worth noting that instead of directly labeling the available data for fine-tuning, an active learning query strategy is adopted to select data. The purpose is to mine samples with richer information and reduce the number of labeled samples. At the same time, the encoder parameters are also frozen during the fine-tuning stage.

[0078] Step 3: A convolutional neural network unit with a self-attention mechanism is fused at the input end to adaptively focus on the most important part of the input signal, learn local contextual features and temporal dynamics features, and capture the relationship between time points in the input data samples, which helps to determine the state label. In this work, a one-dimensional convolutional neural network is used as the backbone model, including batch normalization and ReLU activation. The output of the convolutional neural network is then passed through an encoder based on a self-attention mechanism. The main purpose of the self-attention layer is to learn the weight coefficients to obtain the relationship between the input data at different times, thereby weighting and encoding the features extracted by the CNN. The encoded features are then flattened so that they can be nonlinearly mapped through the input projection head.

[0079] Assuming that after one-dimensional convolution, augmented data is generated at multiple time points T. The characterization is Attention score α t (t = 1, 2, ..., T) represents the result obtained from the convolutional layer. The contribution of the feature representation used in subsequent applications can be calculated using formula (5):

[0080]

[0081] in, express The transpose of W u and W v It is the weight matrix of the self-attention module, and the softmax(·) function ensures that the sum of the weights of all attention scores obtained is 1; Equation (5) can also be regarded as an unbiased 2-layer MLP model, {W u W v} represents the coefficients; the final feature vector S encoded by the self-attention mechanism is represented by the attention score α. t and The linear weighted sum is shown in formula (6):

[0082]

[0083] Assuming that after one-dimensional convolution, augmented data is generated at multiple time points T. The characterization is The output after passing through an attention layer is As shown in formula (7),

[0084]

[0085] Among them, W att The weight matrix that constitutes the attention module, and the resulting output It is used to compare with a trainable parameter matrix u used to represent context information, which is randomly initialized to obtain... The allocation coefficients are then normalized using softmax, as shown in formula (8):

[0086]

[0087] The score function can theoretically be any comparison function, and the self-attention weight coefficient α is in the range (0, 1); the final normalized vector S is all... The weighted sum is shown in formula (9):

[0088]

[0089] The attention score obtained from learning represents the score achieved by the convolutional layer. In terms of contribution to the feature representation used in subsequent layers, the self-attention block uses dot-product-based attention scores to transform the feature values ​​at each time step.

[0090] The effect comparison of adding a self-attention mechanism will evaluate the effect of using self-supervised representations with contrastive learning versus adding a self-attention technique; this is done to emphasize the importance of including a self-attention mechanism in the proposed network. Figure 2 The comparison results of three evaluation metrics across four datasets are shown. It can be seen that the accuracy of the contrastive learning method with self-attention is significantly improved across all four datasets. For the WISDM dataset, the recall index decreases slightly, while the accuracy improves significantly. The evaluation metrics for the other three datasets are either increasing or constant. Particularly for the UTD dataset, the method of this invention shows a very high improvement in the F1-Score.

[0091] Figure 2 The comparison results were visualized using t-SNE, showing the UCI and UTD datasets; Figure 3 The results show that contrastive learning with self-attention further reduces intra-class distance and increases inter-class distance; this means that, compared with self-supervised pre-training, internal information can be extracted from signal input without using labels, and the extracted features have inherent advantages.

[0092] Step 4: The query strategy based on active learning includes: ACSC+self-attention employs two query strategies (information quality and margin sampling (MS)). The query strategy based on active learning uncertainty measurement improves the model fine-tuning effect. Information quality measures the uncertainty of samples under the inductive consistency prediction algorithm, while MS measures uncertainty using the posterior probability of samples. The combination of the two results in hard samples containing more information, which can better improve the classification performance of the network.

[0093] Assuming the sample set U is unlabeled d An unlabeled sample X m The corresponding class label is Y m The uncertainty of the sample is measured using information quality and MS metric; the p-value is used to quantify the information quality of the data instance, which is defined as:

[0094]

[0095] in, and This represents the first and second largest p-values ​​of the m-th sample;

[0096] The uncertainty of sampling interval is defined as:

[0097] I MS (X m |U d )=P(Y m =l1|X m ;M)-P(Y m =l2|X m ;M) (11)

[0098] Where l1 and l2 represent the most likely class labels predicted under a given model; I MS The smaller the value, the higher the sample uncertainty, and the final sample X... m The uncertainty measure is:

[0099]

[0100] The impact of using query strategies to select labeled data for training; the effect of using active learning query strategies to select samples with special attributes for model tuning, and comparison with two other methods; the first method is a fully supervised method, while the other two methods are based on self-attention contrastive learning; the second method is to extract labeled samples at random proportions for fine-tuning; the third method is to use active learning query strategies to select labeled samples, extracting 10%, 20%, 40%, 60%, and 100% of active samples from the labeled dataset to verify the performance of the above three models.

[0101] Figure 4 The performance of models trained under different selection mechanisms is shown. Models trained using the active learning strategy outperform those trained using randomly selected proportions; particularly on the UCI, UTD, and WISDM datasets, the active learning strategy significantly outperforms the randomly selected learning strategy. While active learning, using only partial labels, performs somewhat worse than fully supervised learning, it overcomes the label limitation and provides a new approach to deep learning. The addition of active learning can further reduce the number of online labeled samples; the model tends to stabilize when the selected data reaches 60%.

[0102] Furthermore, on the UCI and WISDM datasets, the performance of actively trained models outperforms fully supervised models; this may be related to the limitations of uncertainty and noise in the datasets. In contrast, on the UTD dataset, due to the smaller number of active samples, the training effect of the model increases with the number of selected samples, and the performance reaches its best when the selection reaches 100%.

[0103] Ablation experiments were conducted on both self-attention and active learning components. The results are shown in Table 1. Ablation experiments verified that self-attention and active learning improved the model's performance across all datasets. During active sampling, the MotionSense and WISDM datasets collected 60% of the samples, achieving the best F1 scores, while UTD required collecting all samples due to its smaller sample size. The consistency performance gain across the entire dataset further demonstrates that the active learning query strategy can be integrated with contrastive learning without degrading performance.

[0104] Table 1 Ablation experiments involving self-attention and active learning

[0105]

[0106] Meanwhile, the method of the present invention is compared with existing methods, as shown in Table 2. The existing methods are compared with CNN+LSTM (ConvLSTM)18 and the self-supervised learning representation framework (STF-CSL)12, which uses short-time Fourier transform (STFT) to construct the encoder; and with the method of adding an attention mechanism after the CNN+LSTM11 structure; finally, in order to show the effectiveness of the dataset, the structure of the short-time Fourier transform encoder is also compared; the F1 scores of the results are summarized in Table 2; in terms of F1 score, the method of the present invention is superior to the existing methods, and the improvement obtained on all datasets (especially UTD) is significant.

[0107] Table 2 compares the proposed method with the F1 scores of existing literature.

[0108]

[0109]

[0110] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A human behavior recognition method based on self-attention active contrastive coding, characterized in that, Includes the following steps: Step 1: Collect human body position data through sensors and construct a multi-axis data aggregation matrix based on the time step; Step 2: Use a self-attention-based active contrastive encoding learning framework to learn action representations for classification; Step two specifically includes: Step 21: Apply two transformations to each example using the enhancement module, input data. Two views are generated for each sample in the batch by applying two transformations. ,in, As a positive sample, Negative samples; Step 22: The encoder part based on the neural network Two views Mapping to a low-dimensional representation yields and ; Step 23: Projection head through the multilayer sensor in the hidden layer The representation learned by the encoder is mapped to the space where the contrastive loss is applied, resulting in... and ; Step 24: Define a contrastive loss function for the contrast task, using normalized temperature-scaled cross-entropy loss as the loss function, and using cosine similarity to measure the two augmented instances. Similarity between them; Step 3: Integrate a convolutional neural network unit with a self-attention mechanism at the input end to learn local contextual features and temporal dynamic features, and capture the relationship between time points in the input data sample; Step three specifically includes: First, a one-dimensional convolutional neural network is used as the backbone model, including batch normalization and ReLU activation; second, the output of the convolutional neural network is passed through an encoder module based on a self-attention mechanism; finally, the encoded features are flattened. Step 4: Measure the uncertainty of the sample using the inductive consistency prediction algorithm through information quality; measure the uncertainty using the posterior probability of the sample through interval sampling.

2. The human behavior recognition method based on self-attention active contrastive coding according to claim 1, characterized in that: The encoder in step 22 is a CNN-based self-attention encoder.

3. The human behavior recognition method based on self-attention active contrastive coding according to claim 1, characterized in that, The formula for cosine similarity in step 24 is: in, yes Norm, and For passing through the projection head Output characteristics.

4. The human behavior recognition method based on self-attention active contrastive coding according to claim 1, characterized in that, The loss function is defined as: in, This refers to the temperature parameter in softmax.

5. The human behavior recognition method based on self-attention active contrastive coding according to claim 1, characterized in that, One-dimensional convolutional neural networks specifically include: Assuming that after one-dimensional convolution, multiple time points Upgraded data The characterization is Attention score Represents the result of the convolutional layer The contribution of the feature representation used subsequently is calculated by formula (5): (5) in, express transpose, and It is the weight matrix of the self-attention module. function; The resulting feature vector encoded by the self-attention mechanism Represented as attention score and The linear weighted sum is shown in formula (6): (6) Assuming that after one-dimensional convolution, multiple time points Upgraded data The characterization is The output after passing through an attention layer is As shown in formula (7), (7) in, It is the weight matrix that constitutes the attention module; The output obtained It is used in a trainable parameter matrix to represent contextual information. Random initialization and comparison are obtained The allocation coefficients are then normalized using softmax, as shown in formula (8): (8) in, score The function can be any comparison function; the final normalized vector is obtained. It is all The weighted sum is shown in formula (9): (9) The attention score obtained from learning represents the score achieved by the convolutional layer. In terms of contribution to the feature representation used in subsequent layers, the self-attention block uses dot-product-based attention scores to transform the feature values ​​at each time step.

6. The human behavior recognition method based on self-attention active contrastive coding according to claim 1, characterized in that, Step four specifically includes: Assuming an unlabeled sample set An unlabeled sample The corresponding class tag is , The value is used to quantify the information quality of a data instance. The information quality of a sample is defined as: (10) in, Indicates the first The first and second largest samples value; The uncertainty of sampling interval is defined as: (11) in, These represent the most likely class labels predicted under a given model; the final sample The uncertainty measure is: (12)。 7. A system employing the human behavior recognition method based on self-attention active contrastive coding as described in any one of claims 1-6, characterized in that, include: The task definition module is used for sensors to collect human position data and construct a multi-axis data aggregation matrix based on the time step. A learning framework module based on self-attention active contrastive encoding for action representations in classification; The self-attention mechanism module is used to learn local contextual features and temporal dynamic features to capture the relationship between time points in the input data sample. The query strategy model based on active learning is used to measure the uncertainty of samples by information quality under the inductive consistency prediction algorithm; and to measure uncertainty by using the posterior probability of samples through interval sampling.

Images