Skeleton human behavior recognition method based on multi-view space-time contrast loss

Through end-to-end dual-channel spatiotemporal modeling network and multi-view space-time comparison loss, the problem of insufficient extraction of spatial and temporal features of skeleton human behavior recognition methods in the existing technology is solved, and the ability to distinguish similar actions and generalize models is improved. .

CN119049128APending Publication Date: 2024-11-29HEFEI UNIV

Patent Information

Application Number
CN202411166531.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing skeleton human behavior recognition methods have shortcomings in learning spatial and temporal characteristics, and it is difficult to effectively extract comprehensive spatiotemporal representations of skeleton sequences, resulting in insufficient discrimination ability of similar actions.

Method used

Using an end-to-end dual-channel spatiotemporal modeling network, the spatiotemporal features are learned through two encoders with complementary characteristics of ST-Encoder and TS-Encoder, and the pooled projection module is used to decouple and project features, combining multi-view space-time contrast loss and Cross-entropy loss optimizes the backbone network and improves the expressive ability and generalization ability of action characteristics.

Benefits of technology

It realizes a more comprehensive spatial and temporal representation of skeleton movements, improves the ability to distinguish similar movements and generalizes the model, and enhances the ability to learn complementary information for skeleton movements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049128A_ABST
    Figure CN119049128A_ABST
Patent Text Reader

Abstract

The invention discloses a skeleton human body behavior recognition method based on multi-view space-time comparison loss, and relates to the technical field of computer vision. A recognition network for skeleton human body behavior recognition takes an end-to-end double-path space-time modeling network as a main network; the backbone network comprises two kinds of space-time encoders, i.e., ST-Encoder and TS-Encoder, which have different modeling sequences; performing decoupling and nonlinear projection on two spatio-temporal features obtained by the two spatio-temporal encoders by using a pooling projection module to obtain feature distribution under global spatio-temporal, spatial and time viewing angles; the parameter optimization of the backbone network is realized by combining multi-view space-time contrast loss and cross entropy loss, and the end-to-end double-path space-time modeling network can flexibly learn the comprehensive space-time representation of human skeleton actions by using two encoders with complementary characteristics, so that the distinguishing capability of similar actions is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of computer vision, and in particular to a skeleton human behavior recognition method based on multi-view spatiotemporal contrast loss. Background Art

[0002] In recent years, artificial intelligence (AI) technology has developed rapidly. Computer vision is an important research direction of AI, and its goal is to enable computers to learn to analyze the surrounding environment and establish a visual system similar to that of humans. Computer vision includes many classic downstream research tasks, such as video understanding, video generation, image classification, and object detection. Among the video understanding and analysis tasks, human action recognition (HAR) is a core task with high research value and application scenarios.

[0003] HAR is an active and challenging topic in the field of computer vision. It mainly studies how to automatically extract human motion features from raw video data and perform recognition and classification. Early HAR was mainly based on RGB data. With the continuous development of sensors, data in modalities such as skeleton data, depth image data, and infrared image data have attracted more and more attention from researchers. HAR is widely used in reality, such as intelligent video surveillance systems, human-computer interaction, unmanned driving, motion human analysis, virtual reality, etc. The current intelligent monitoring system in the market is relatively complete. It can recognize the behavior of the human body in the video surveillance and make preset responses, such as fall detection and notification of contacts, dangerous behavior detection and alarm, etc. In recent years, people have paid more and more attention to sports and health issues, which has promoted the application of behavior recognition technology in this field. With efficient and accurate human capture and recognition, human behavior recognition technology helps athletes to scientifically standardize their sports postures, avoid injuries and improve their sports performance. In the medical field, human behavior recognition technology plays an important role in the diagnosis of depression and postoperative rehabilitation. In addition, with the rapid development of artificial intelligence technology, human-computer interaction, unmanned driving and virtual reality have become hot research areas. The research on human behavior recognition methods has also promoted the development of the above research fields to a certain extent.

[0004] Initially, researchers manually designed relevant features of the human skeleton and used machine learning algorithms for behavior recognition. However, methods based on manual design have difficulty in obtaining advanced features and cannot be applied to large datasets, which greatly limits recognition performance and efficiency. As deep learning network models have achieved exciting results on RGB data, many researchers have also applied classic deep learning network models to skeleton human behavior recognition tasks.

[0005] RNN and its variant, long short-term memory network (LSTM), have natural advantages in modeling time series data. For the task of skeleton human behavior recognition, some researchers have tried to model human skeleton sequence data, focusing on the co-occurrence features of joints in the human skeleton to extract spatiotemporal features. In addition, feature enhancement and special network structures are used to improve the performance of the network. Unlike RNN, CNN has the ability to extract high-level features and is good at spatial modeling. On the one hand, 2D CNN has been favored by early researchers due to its advantages such as simple network structure and fast running speed. On the other hand, in order to further explore the spatial and temporal features in skeleton sequence data, 3D CNN models are used for skeleton human behavior recognition. RNN is not sufficient for extracting spatial information and is difficult to train. CNN is good at processing Euclidean data, and the spatial topological structure information of the human body will be lost in the process of constructing the corresponding input form for skeleton sequence data. The skeleton human behavior recognition methods based on RNN and CNN have their own shortcomings. With the development of deep network models, new deep learning networks, such as graph convolutional networks (GCN) and Transformer, are increasingly attracting the attention of researchers.

[0006] GCN has a natural advantage in processing graph-structured data such as human skeletons. Yan et al. innovatively proposed the spatiotemporal graph convolution (ST-GCN) model. For the skeleton human action recognition task, they first designed the spatiotemporal topological graph of the skeleton sequence based on the natural connection of the joints inside the human body and the connection of the same joint between adjacent frames. Then, the spatiotemporal features of the skeleton sequence are extracted by stacking a series of spatiotemporal graph convolution blocks, and the category prediction is performed using the SoftMax classifier. Unlike the local spatial features of the joints extracted in the ST-GCN model, the action structure graph convolution network (AS-GCN) is used. The AS-GCN model can not only capture specific joint dependencies through actions, but also obtain multi-neighborhood joint dependencies by extending the existing skeleton graph to structural connections. The pre-defined human skeleton topological graph cannot reflect the invisible connections between joints, so the researchers tried to mine the connections between non-adjacent joints and construct a sample-adaptive graph structure, and used multiple data streams to train the network model. The above models focus on the spatial modeling of the skeleton graph, but are insufficient for the extraction of temporal features of the skeleton sequence.

[0007] Based on this, the present invention aims to provide a skeleton human behavior recognition method based on multi-view spatiotemporal contrast loss, which can more flexibly learn the comprehensive spatiotemporal representation of human skeleton movements and improve the ability to distinguish similar movements. Summary of the invention

[0008] In order to make up for the shortcomings of the prior art problems, the purpose of the present invention is to provide a skeleton human behavior recognition method based on multi-view spatiotemporal contrast loss, whose end-to-end dual-path spatiotemporal modeling network uses two encoders with complementary characteristics to flexibly learn the comprehensive spatiotemporal representation of human skeleton movements; in addition, the pooling projection module decouples the two complementary action features and projects them into different high-dimensional spaces to enhance the expression ability of the features. The MST-CL is combined with the cross entropy loss, and the parameters of the backbone network are optimized end-to-end, so that the backbone network can better learn the complementary information in the skeleton movements, which is conducive to distinguishing similar movements.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] A skeleton human behavior recognition method based on multi-view spatiotemporal contrast loss, wherein the recognition network of the skeleton human behavior recognition is based on an end-to-end dual-path spatiotemporal modeling network as the backbone network, and the backbone network includes two spatiotemporal encoders, ST-Encoder and TS-Encoder, with different modeling orders. The two encoders cooperate and complement each other to learn and respectively extract comprehensive spatiotemporal representations of human skeleton actions; the two spatiotemporal features obtained by the two spatiotemporal encoders are respectively decoupled and nonlinearly projected using a pooling projection module to obtain feature distribution in the global spatiotemporal, spatial and temporal perspectives;

[0011] The parameter optimization of the backbone network is achieved by combining multi-view spatiotemporal contrast loss with cross entropy loss.

[0012] Furthermore, the ST-Encoder is composed of a series of ST-Blocks stacked together, each ST-Block including channel topology refinement graph convolution and multi-scale time modeling;

[0013] TS-Encoder is composed of a series of TS-Blocks stacked together, which include multi-scale time modeling and channel topology refinement graph convolution;

[0014] The ST-Encoder and TS-Encoder have the same module structure, but the order of multi-scale time modeling and channel topology refinement graph convolution is different.

[0015] Furthermore, the pooling projection module decouples and nonlinearly projects the two spatiotemporal features, respectively, as follows:

[0016] F ST Perform global pooling, spatial pooling, and temporal pooling in turn to obtain and Similarly, for F TS Perform global pooling, spatial pooling, and temporal pooling to obtain and

[0017] F ST is the feature vector extracted by ST-Encoder from the input skeleton sequence data, F ST ∈R B×C×T×N ; F TS is the feature vector extracted by TS-Encoder from the input skeleton sequence data, F TS ∈R B×C×T×N ; B represents Batchsize, C represents the number of channels, T represents the time dimension, and N represents the space dimension.

[0018] Furthermore, the multi-view spatiotemporal contrast loss MST-CL is calculated as follows:

[0019] L MST-CL =L G +L S +L T

[0020]

[0021] in, It means calculating the cosine similarity between the two. The contrast loss tends to be "equal" as the temperature coefficient τ increases, and only focuses on the most difficult negative samples as τ decreases. τ plays a role in adjusting the attention of negative samples.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) First, the present invention proposes an end-to-end dual-path spatiotemporal modeling network. The end-to-end dual-path spatiotemporal modeling network includes two feature encoders, TS-Encoder and ST-Encoder. The two feature encoders have the same component modules but different spatiotemporal modeling orders, forming two encoders with complementary characteristics, which can flexibly learn the comprehensive spatiotemporal representation of human skeletal movements.

[0024] (2) Secondly, the present invention aims at the characteristic that the action features of the same action under different perspectives should have consistent expression. The two complementary features obtained by TS-Encoder and ST-Encoder are decoupled in time and space and linearly projected. Then, the contrast loss of the features of the same action under different perspectives is calculated, and the feature distribution space is optimized, so that the model can learn discriminative spatiotemporal features, thereby improving the generalization ability of the model and facilitating the model to classify similar actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a framework diagram of the identification method of the present invention.

[0026] Figure 2 It is a schematic diagram of the structure of the backbone network of the present invention.

[0027] Figure 3 Schematic diagram of the multi-view spatiotemporal contrast loss calculation process of the present invention.

[0028] Figure 4 This is the confusion matrix diagram of the present invention in NTU RGB+D. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Combination Figure 1-4 , a skeleton human behavior recognition method based on multi-view spatiotemporal contrast loss is specifically described. The present invention proposes an end-to-end dual-path spatiotemporal modeling network (DSTMN) as the backbone network, and designs a multi-view spatiotemporal contrast loss (MST-CL) to ensure the consistency of the representation of skeleton actions under different perspectives, thereby improving the ability to distinguish similar actions. The backbone network includes two spatiotemporal encoders, ST-Encoder and TS-Encoder, with different modeling orders, which can flexibly extract two synergistic and complementary spatiotemporal features. The pooling projection module (PPM) can decouple the two complementary action features and project them into different high-dimensional spaces to enhance the expressive power of the features. MST-CL optimizes the feature distribution space by calculating the contrast loss of action features under time, space and global spatiotemporal perspectives, so that the same action has consistent action features under different perspectives, thereby obtaining more essential and discriminative action features. In addition, the present invention combines MST-CL with cross entropy loss (CEL) to perform end-to-end parameter optimization on the backbone network, so that the backbone network can better learn the complementary information in the skeleton action, which is conducive to distinguishing similar actions.

[0031] 1. Identification Method Framework

[0032] Figure 1The framework diagram of the skeleton human action recognition method based on multi-view spatiotemporal contrast loss proposed by us is shown. First, we construct different representations of skeleton data as the input data of the backbone network. Then, the ST-Encoder and TS-Encoder in the backbone network learn the complementary spatiotemporal features in the skeleton action respectively and calculate the contrast loss after decoupling and projection operations. Finally, the cross entropy loss and multi-view contrast loss are calculated respectively, and then the model parameters are optimized by back propagation.

[0033] 2. Dual-path spatiotemporal modeling network

[0034] At present, all methods based on graph convolutional networks adopt a single spatiotemporal modeling method, so it is difficult to learn the comprehensive spatiotemporal characteristics of skeleton actions. We propose a dual-path spatiotemporal modeling network (DSTMN) framework, whose structure is as follows: Figure 2 As shown in Figure 1. The network adopts a dual-path parallel network structure, which includes two spatiotemporal encoders, ST-Encoder and TS-Encoder. The parameter list of the encoder is [input channel, output channel, step size, adjacency matrix, residual connection]. The parameters of the two encoders are consistent. The specific model parameters are shown in Table 1. Both encoders are composed of a temporal modeling network and a spatial modeling network. Taking ST-Encoder as an example, in order to extract the spatiotemporal features in the skeleton sequence, the present invention sets up a spatial modeling network and a temporal modeling network respectively, and stacks the above two sub-network modules. For spatial modeling and temporal modeling, the present invention adopts the channel topology refinement graph convolution (CTR-GC) and multi-scale temporal modeling (MS-TCN) in the mainstream method CTR-GCN[5] in recent years. The component modules of TS-Encoder are consistent with those of ST-Encoder, but the present invention adjusts the spatiotemporal modeling order and designs two flexible and complementary spatiotemporal modeling methods, thereby forming two encoders with complementary characteristics.

[0035] Table 1 Network structure parameters of ST-Encoder

[0036]

[0037] The input of DSTMN is a multidimensional array. Taking the Joint data stream as an example, that is, Input: 3×T×N, where 3 represents the number of channels, that is, the three dimensions of the joint coordinates, T represents the duration of the action, and N represents the number of joints in the human skeleton.

[0038] 3. Multi-view spatiotemporal contrast loss

[0039] ST-Encoder and TS-Encoder are two excellent and complementary encoders. They extract features from the input skeleton sequence data and obtain two feature vectors F and F respectively. ST ∈RB×C×T×N and F TS ∈R B×C×T×N . Where B represents Batchsize, C represents the number of channels, T represents the time dimension, and N represents the space dimension. The present invention proposes a multi-view spatiotemporal contrast loss (MST-CL) for optimizing the feature space. First, for two spatiotemporal feature vectors F with complementary characteristics ST and F TS , the present invention designs a pooling projection module (PPM) to decouple and nonlinearly project the two spatiotemporal features respectively, so as to obtain the feature distribution under the global spatiotemporal, spatial and temporal perspectives. Specifically, F ST Global pooling, spatial pooling, and temporal pooling are performed in sequence, and we obtain and Similarly, for F TS Perform global pooling, spatial pooling, and temporal pooling to obtain and Specific operations such as Figure 3 (a) shown.

[0040] Then, for a certain sample i, the spatiotemporal features are obtained by ST-Encoder and TS-Encoder respectively. and Since ST-Encoder and TS-Encoder can obtain synergistic and complementary spatiotemporal features for the same sample and have the characteristic of consistent expression, the distribution of the two feature vectors in the feature space is similar. The present invention believes that the same sample should have consistent feature distribution from the three perspectives of global spatiotemporal, spatial and temporal. Therefore, the present invention defines a sample x in a Mini-batch i ∈F ST , the samples are obtained through the pooling projection module PPM Similarly, sample X i ∈F TS Obtained via PPM From a global space-time perspective, is the anchor sample, then the set of positive sample pairs is It contains samples from both spatial and temporal perspectives. The remaining samples are all negative sample pairs. G Considering that the same sample should have the characteristics of consistent expression in the global spatiotemporal perspective, temporal perspective and spatial perspective, that is, in the feature space, the anchor point sample is closer to the positive sample and farther from the negative sample. Therefore, the present invention proposes a multi-view spatiotemporal contrast loss (MST-CL), the principle of which is as follows: Figure 3 (b) is shown, and the calculation formula is as follows.

[0041] L MST-CL=L G +L S +L T

[0042]

[0043] in, Indicates the calculation of the cosine similarity between the two. As the temperature coefficient τ increases, the contrast loss tends to be "equal", and as τ decreases, it only focuses on the most difficult negative samples. τ plays a role in adjusting the attention of negative samples.

[0044] Since ST-Encoder and TS-Encoder have different focuses on spatiotemporal features, the extracted features are naturally complementary, and their distribution in the feature space has certain similarities. In order to maintain the characteristics of this complementary feature, the present invention combines the cross entropy loss function with the contrast loss to form the overall loss function of the present invention. The cross entropy loss function optimizes the network parameters through label data, and the contrast loss function optimizes the network parameters by mining the features in the sample. In order to better utilize the two constraints and achieve a balance between label information and feature information, the present invention introduces a hyperparameter to adjust the importance of the two different loss functions.

[0045] L=L CEL +λL MST-CL

[0046] The present invention uses a network for end-to-end learning, does not require an additional data structure to store negative samples, and increases the flexibility of deployment training. The two-branch network, ST-Encoder and TS-Encoder, encode the same sample, obtain high-level semantic expressions of the same sample with different focuses, and can obtain higher-quality positive and negative sample pairs. Individual discrimination is used as a proxy task to obtain a certain supervisory signal, and supervised training is performed in combination with the cross entropy loss function. The extracted features are further pooled in different ways, such as global pooling, time pooling and space pooling, which further enhances the nonlinear expression of the features. The DSTMN proposed in the present invention can focus on the global spatiotemporal features, single time features and single space features of the samples. Global spatiotemporal features, spatial features and time features are three different perspectives, and the same sample has the characteristics of consistent feature expression under three different perspectives. Increasing the mutual information between the same sample at different perspectives and reducing the correlation with negative samples are the optimization directions of MST-CL. Constructing a feature space with a large sample size as much as possible helps reduce the randomness of the sample and is more conducive to smooth updating of network parameters.

[0047] (IV) Data testing

[0048] (4.1) Dataset Introduction

[0049] The NTU RGB+D dataset [6] was created by Nanyang Technological University, Singapore. The dataset is captured by three Kinect cameras from different angles and contains 56,578 skeleton sequence data, 60 action categories, and 40 subjects. Among the 60 action categories, the first 50 action categories are single-person actions, and the last 10 action categories are two-person actions. Each Kinect camera collects 3D coordinate data of 25 joints of the subject. This dataset provides two evaluation benchmarks, namely cross-subject (X-Sub) and cross-view (X-View). Specifically, under the X-Sub evaluation benchmark, 40,091 skeleton sequence data from 20 subjects will constitute the training set, and the remaining 16,487 skeleton sequence data will constitute the test set. Under the X-View evaluation benchmark, 37,648 skeleton sequence data from camera 2 and camera 3 will constitute the training set, and the remaining 18,932 skeleton sequence data will constitute the test set.

[0050] The NTU RGB+D 120 dataset [7] is an extended version of the NTU RGB+D dataset, which contains 113,945 skeleton sequence data, 120 action categories, and 106 subjects. The difference is that the NTU RGB+D 120 dataset provides two evaluation benchmarks, namely cross-subject (X-Sub) and cross-setting (X-Set). Specifically, under the X-Sub evaluation benchmark, 63,026 skeleton sequence data from 53 subjects will be used to train the network, and the remaining 50,919 skeleton sequence data will be used for testing. The X-Set evaluation benchmark contains 32 camera setting numbers, of which 54,468 skeleton sequence data from even-numbered camera setting numbers will be used to train the network, and the remaining 59,477 skeleton sequence data will be used for testing.

[0051] (4.2) Experimental environment configuration and parameter setting

[0052] Our experiments were conducted on an NVIDIA GeForce RTX4080 GPU, and PyTorch was selected as the deep learning framework. The batch size was set to 16 on the NTU RGB+D and NTU RGB+D 120 datasets. The model was trained for a total of 90 iterations, and the network parameters were updated using stochastic gradient descent (SGD), with a momentum of 0.9 and a weight decay of 0.0001. For the NTU RGB+D dataset, the learning rate was set to 0.05 initially and decayed by 10 times after the 40th and 60th iterations, respectively. For the NTU RGB+D 120 dataset, the learning rate was set to 0.05 and decayed by 10 times after the 50th and 70th iterations, respectively. The hyperparameter τ = 0.07 and the experimental parameter λ = 1. The present invention is consistent with LA-CGN [8], and a multi-stream skeleton data flow is used to integrate the network.

[0053] We will conduct relevant ablation experiments on the NTU RGB+D dataset to verify the effectiveness of the relevant modules in the proposed method. In addition, we verify the recognition performance of the proposed method on two large datasets, NTU RGB+D and NTU RGB+D 120.

[0054] (4.3)MST-CL

[0055] The present invention selects different loss functions and conducts experiments on the NTU RGB+D dataset, and the experimental results are shown in Table 2. In this experiment, the present invention uses four data streams, namely Joint, Bone, Joint-Motion and Bone-Motion. It can be seen from the experimental results that when DSTMN only uses cross entropy as the loss function, it achieves recognition accuracy rates of 92.1% and 95.4% on X-Sub and X-View, respectively. When DSTMN uses the MST-CL proposed in the present invention to optimize the model parameters, it improves by 0.5% and 0.9% on X-Sub and X-View, respectively. The above experiments prove that the MST-CL proposed in the present invention helps the model learn discriminative spatiotemporal features and improves the model's ability to classify similar actions. In addition, for the experimental parameter λ, the present invention sets a series of ablation experiments to select the parameter combination with the best performance, and the experimental results are shown in Table 3. The value of λ controls the importance of MST-CL. When λ = 1, the network performs best, achieving 92.7% and 96.3% recognition accuracy on the X-Sub and X-View benchmarks in the NTU RGB+D dataset, respectively.

[0056] Table 2 Recognition accuracy achieved by DSTMN under different loss functions

[0057]

[0058] Table 3 Different experimental parameter values

[0059]

[0060] The proposed DSTMN method is compared with the most advanced methods in recent years to demonstrate its effectiveness in the skeleton behavior recognition task. The experimental results are shown in Table 4. The proposed method achieves recognition accuracy rates of 96.7% and 93.0% on the NTU RGB+D dataset, and 90.6% and 89.5% on the NTU RGB+D 120 dataset. It is worth noting that FR-Head[9] has a similar concept to the proposed method. This method also designs a contrast loss to improve the model's ability to distinguish similar actions. Compared with this method, the proposed method achieves roughly equivalent performance, demonstrating the flexibility and excellent performance of contrastive learning. In addition, the proposed method improves Info-GCN

[10] and FR-Head[9] by 0.3% and 0.2% respectively on the X-Sub benchmark in NTU RGB+D. At the same time, compared with the benchmark model CTR-GCN[5], the proposed method improves 0.6% on both X-Sub of NTU RGB+D and X-Sub of NTU RGB+D 120. The confusion matrix of the present invention on the NTURGB+D dataset is as follows: Figure 4 shown.

[0061] Table 4 Comparison with state-of-the-art methods on NTU RGB+D and NTU RGB+D 120 datasets

[0062]

[0063] References:

[0064] 1. Yan, S., Y. Xiong, and D. Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. in Proceedings of the AAAIConference on Artificial Intelligence. 2018. New Orleans, LA, United states: AAAIpress.

[0065] 2.Shi,L.,et al.Two-stream adaptive graph convolutional networks forskeleton-based action recognition.in Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition.2019.Long Beach,CA,United states:IEEE Computer Society.

[0066] 3.Liu,Z.Y.,et al.Disentangling and Unifying Graph Convolutions forSkeleton-Based Action Recognition.in Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition.2020.Electr Network.

[0067] 4.Chen,Z.,et al.Multi-Scale Spatial Temporal Graph ConvolutionalNetwork for Skeleton-Based Action Recognition.in Proceedings of the AAAIConference on Artificial Intelligence.2021.Electr Network.

[0068] 5.Chen,Y.,et al.Channel-wise Topology Refinement Graph Convolutionfor Skeleton-Based Action Recognition.in Proceedings ofthe IEEE / CVFInternational Conference on Computer Vision.2021.

[0069] 6.Shahroudy,A.,et al.NTU RGB+D:A large scale dataset for 3D humanactivity analysis.in Proceedings ofthe IEEE / CVF Conference on Computer Visionand Pattern Recognition.2016.Las Vegas,NV,United states:IEEE ComputerSociety.

[0070] 7.Liu,J.,et al.,NTU RGB+D 120:A Large-Scale Benchmark for 3D HumanActivity Understanding.IEEE Transactions on Pattern Analysis and MachineIntelligence,2020.42(10):p.2684-2701.

[0071] 8.Xu,H.,et al.,Language Knowledge-Assisted Representation Learningfor Skeleton-Based Action Recognition.arXiv preprint arXiv:2305.12398,2023.

[0072] 9.Zhou,H.,Q.Liu,and Y.Wang.Learning Discriminative RepresentationsforSkeleton BasedAction Recognition.in Proceedings ofthe IEEE / CVF Conference onComputer Vision and Pattern Recognition.2023.

[0073] 10.Hyung-Gun,C.,et al.InfoGCN:Representation Learningfor HumanSkeleton-based Action Recognition.in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2022.

[0074] 11.Pengfei,Z.,et al.Semantics-Guided Neural Networks for EfficientSkeleton-Based Human Action Recognition.in Proceedings of the IEEE / CVFConference on Computer Vision andPattern Recognition.2020.

[0075] 12. Ke, L., K.-C. Peng, and S. Lyu. Towards to-at spatio-temporal focus for skeleton-based action recognition. in Proceedings of the AAAI Conference on Artificial Intelligence. 2022.

[0076] 13. Liu, Y., et al., Skeleton-based human action recognition via large-kernel attention graph convolutional network. IEEE Transactions onVisualization and Computer Graphics, 2023.29(5):p.2575-2585.

[0077] 14. Song, Y.-F., et al., Constructing stronger and faster baselines for skeleton-based action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.45(2): p.1474-1488.

[0078] What is described above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A skeleton human behavior recognition method based on multi-view spatiotemporal contrast loss, characterized in that: The recognition network of skeleton human behavior recognition is based on an end-to-end dual-path spatiotemporal modeling network as the backbone network. The backbone network includes two spatiotemporal encoders, ST-Encoder and TS-Encoder, with different modeling orders. The two encoders cooperate and complement each other to learn and extract comprehensive spatiotemporal representations of human skeleton actions respectively. The two spatiotemporal features obtained by the two spatiotemporal encoders are decoupled and nonlinearly projected by pooling projection modules respectively to obtain feature distribution in the global spatiotemporal, spatial and temporal perspectives. The parameter optimization of the backbone network is achieved by combining multi-view spatiotemporal contrast loss with cross entropy loss.

2. The skeleton human behavior recognition method based on multi-view spatiotemporal contrast loss according to claim 1 is characterized in that: The ST-Encoder is composed of a series of ST-Blocks stacked together, each of which includes channel topology refinement graph convolution and multi-scale time modeling; TS-Encoder is composed of a series of TS-Blocks stacked together, which include multi-scale time modeling and channel topology refinement graph convolution; The ST-Encoder and TS-Encoder have the same module structure, but the order of multi-scale time modeling and channel topology refinement graph convolution is different.

3. According to the skeleton human behavior recognition method based on multi-view spatiotemporal contrast loss of claim 1, the pooling projection module decouples and nonlinearly projects the two spatiotemporal features respectively, as follows: F ST Perform global pooling, spatial pooling, and temporal pooling in turn to obtain and Similarly, for F TS Perform global pooling, spatial pooling, and temporal pooling to obtain and F ST is the feature vector extracted by ST-Encoder from the input skeleton sequence data, F ST ∈R B×C×T×N ; F TS is the feature vector extracted by TS-Encoder from the input skeleton sequence data, F TS ∈R B×C×T×N ; B represents Batchsize, C represents the number of channels, T represents the time dimension, and N represents the space dimension.

4. According to the skeleton human behavior recognition method based on multi-view spatiotemporal contrast loss of claim 1, the multi-view spatiotemporal contrast loss MST-CL is calculated by the following formula: L MST-CL =L G +L S +L T in, It means calculating the cosine similarity between the two. The contrast loss tends to be "equal" as the temperature coefficient τ increases, and only the most difficult negative samples are focused on as τ decreases. τ plays a role in adjusting the attention of negative samples.

Citation Information

Patent Citations

  • Skeleton human body behavior recognition method based on comprehensive space-time representation

    CN118038539A

Cited By

  • Action anomaly detection method and system, terminal and storage medium

    CN120279601A

  • Abnormal motion detection method, system, terminal and storage medium

    CN120279601B