Multi-modal dynamic gesture recognition method based on lightweight 3D residual network and TCN

Abstract

The invention discloses a multi-modal dynamic gesture recognition method based on a lightweight 3D residual network and a TCN. Firstly, original videos in a data set are sampled and sorted and storedaccording to a time sequence; then, the lightweight 3D residual network is pre-trained by using a large public gesture recognition data set, and thus storing a weight file of the model; then, the RGB-D image sequence is used as input, the lightweight 3D residual network and the time convolution network are used as basic models to extract long-term and short-term spatial and temporal features, andan attention mechanism is used for weighted fusion of multi-modal information, wherein the RGB sequence and the Depth sequence are respectively input into the same network structure; finally, a full connection layer is used for classification, a cross entropy loss function is used for calculating a loss value, and accuracy and F1Score are used as evaluation indexes of the network model. The invention not only can achieve high classification accuracy, but also has the advantage of low parameter quantity.

Description

technical field [0001] The invention belongs to the technical field of video spatiotemporal feature extraction and classification methods, in particular to a lightweight heterogeneous structure for dynamic gesture spatiotemporal feature extraction, which can reduce the amount of model parameters and ensure model performance. Background technique [0002] Gestures are a common form of human communication. Gesture recognition can enable human-computer interaction in a natural way. Gesture recognition aims to understand human actions by extracting features from images or videos, and then classify or recognize each sample as a specific label. Traditional gesture recognition is mainly based on manually extracted features. Although this method can achieve good recognition results, it relies on the experience of researchers to design features, and manually extracted features are poorly adaptable to dynamic gestures. [0003] With the development of deep learning, end-to-end gestu...

AI Technical Summary

Problems solved by technology

The combination of lightweight 3D residual network and TCN is expected to solve the problem of generally high complexity of existing methods

Images