Dynamic gesture recognition method based on hybrid deep learning model

Abstract

The invention discloses a dynamic gesture recognition method based on a hybrid deep learning model. The dynamic gesture recognition method includes a training phase and a test phase. The training phase includes first, training a CNN based on an image set constituting a gesture video and then extracting spatial features of each frame of the dynamic gesture video sequence frame by frame using the trained CNN; for each gesture video sequence to be recognized, organizing the frame-level features learned by the CNN into a matrix in chronological order; inputting the matrix to an MVRBM to learn gesture action spatiotemporal features that fuse spatiotemporal attributes; and introducing a discriminative NN; and taking the MVRBM as a pre-training process of NN model parameters and network weights and bias that are learned by the MVRBM as initial values of the weights and bias of the NN, and fine-tuning the weights and bias of the NN by a back propagation algorithm. The test phase includes extracting and splicing features of each frame of the dynamic gesture video sequence frame by frame based on CNN, and inputting the features into the trained NN for gesture recognition. The effective spatiotemporal representation of the 3D dynamic gesture video sequence is realized by adopting the technical scheme of the invention.

Description

Technical field [0001] The invention belongs to the field of computer vision and machine learning, and specifically relates to a dynamic gesture recognition method based on a hybrid deep learning model. Background technique [0002] Gesture recognition has important applications in visual communication, human-computer interaction, and augmented reality. However, due to the complexity and variability of gestures and the influence of factors such as illumination and individual changes, the research on gesture recognition is still a challenging problem. Vision-based gesture recognition usually includes two aspects: feature extraction and classifier design. Commonly used classifiers include neural network (NN), hidden Markov model (HMM), etc. Gesture feature representation methods usually include: hand shape, hand center of gravity position, moment feature, scale-invariant feature transform (SIFT) feature, Histogram of Oriented Gradient (HOG) feature, etc. However, manually select...

AI Technical Summary

Problems solved by technology

For high-dimensional data such as 2D and 3D, it is obvious that the vectorization operation will destroy the internal structure of 2D images, 3D videos, etc., resulting in the loss of hidden interactive information in the structure. In the traditional vector variable-based restricted Boltzmann machine research Based on this, Qi Guanglei et al. proposed a matrix variable-based restricted Boltzmann machine (Matrix Variable Restricted Boltzmann Machine, MVRBM) for high-dimensional data such as images. This model can better model 2D data, but the RBM and MVRBM models are unsupervised

In the field of deep learning, another model that has received wide attention is the Convolutional Neural Network (CNN). CNN has been successfully applied to various image analysis and understanding fields such as positioning, detection, and recognition classification. There are also methods based on CNN and Its variant 3DCNN models the time axis of the video sequence for video classification, which is a difficult and complex task with large model parameters, difficult to train, and requires a lot of training data

Images