A Multimodal Dynamic Gesture Recognition Method Based on Lightweight 3D Residual Network and TCN

Abstract

The present invention claims to protect a multi-modal dynamic gesture recognition method based on a lightweight 3D residual network and TCN. First, the original videos in the dataset are sampled and saved in chronological order; then, a lightweight 3D residual network is pre-trained using a large public gesture recognition dataset, and the model weight files are saved; then, RGB‑ The D image sequence is used as the input, and the lightweight 3D residual network and the temporal convolutional network are used as the basic model to extract long-term and short-term spatial and temporal features, and use the attention mechanism to weight and fuse multi-modal information. The RGB and depth (Depth) sequences are respectively input to the same network structure; finally, the fully connected layer is used for classification, the cross-entropy loss function is used to calculate the loss value, and the accuracy and F1Score are used as the evaluation indicators of the network model. The invention can not only achieve higher classification accuracy, but also has the advantages of low parameter quantity.

Description

technical field [0001] The invention belongs to the technical field of video spatiotemporal feature extraction and classification methods, in particular to a lightweight heterogeneous structure for dynamic gesture spatiotemporal feature extraction, which can not only reduce the amount of model parameters, but also ensure model performance. Background technique [0002] Gestures are a common form of human communication. Gesture recognition enables human-computer interaction in a natural way. Gesture recognition aims to understand human movements by extracting features from images or videos, and then classifying or identifying each sample as a specific label. Traditional gesture recognition is mainly based on manually extracted features. Although this method can achieve a good recognition effect, this method relies on the experience of researchers to design features, and the manually extracted features have poor adaptability to dynamic gestures. [0003] With the development...

AI Technical Summary

Problems solved by technology

The combination of lightweight 3D residual network and TCN is expected to solve the problem of generally high complexity of existing methods

Images