A Dynamic Gesture Recognition Method Based on Hybrid Deep Learning Model

Abstract

The invention discloses a dynamic gesture recognition method based on a hybrid deep learning model, which includes: in the training stage, first training a CNN based on an image set constituting a gesture video, and then using the trained CNN to extract the spatial features of each frame of a dynamic gesture video sequence frame by frame; For each gesture video sequence to be recognized, the frame-level features learned by CNN are organized into a matrix in time order; the matrix is ​​input to MVRBM to learn the spatiotemporal features of gestures that fuse spatiotemporal attributes; a discriminative NN is introduced; Consider MVRBM as the pre-training process of NN model parameters, use the network weights and offsets learned by MVRBM as the initial values ​​of NN weights and offsets, and fine-tune the NN weights and offsets through the backpropagation algorithm; test In the stage, the features of each frame of the dynamic gesture video sequence are also extracted frame by frame based on CNN and stitched together, and input to the previously trained NN for gesture recognition. By adopting the technical scheme of the invention, effective spatiotemporal representation of 3D dynamic gesture video sequences is realized.

Description

technical field [0001] The invention belongs to the field of computer vision and machine learning, and in particular relates to a dynamic gesture recognition method based on a hybrid deep learning model. Background technique [0002] Gesture recognition has important applications in visual communication, human-computer interaction, augmented reality and other fields. However, due to the complexity and variability of gestures and the influence of factors such as illumination and individual changes, gesture recognition research is still a challenging problem. Vision-based gesture recognition usually includes two aspects: feature extraction and classifier design. Commonly used classifiers include neural network (NN), hidden Markov model (HMM), etc. Gesture feature representation methods usually include: hand shape, hand center of gravity position, moment feature, scale-invariant feature transform (Scale-invariant feature transform, SIFT) feature, histogram of oriented gradien...

AI Technical Summary

Problems solved by technology

For high-dimensional data such as 2D and 3D, it is obvious that the vectorization operation will destroy the internal structure of 2D images, 3D videos, etc., resulting in the loss of hidden interactive information in the structure. In the traditional vector variable-based restricted Boltzmann machine research Based on this, Qi Guanglei et al. proposed a matrix variable-based restricted Boltzmann machine (Matrix Variable Restricted Boltzmann Machine, MVRBM) for high-dimensional data such as images. This model can better model 2D data, but the RBM and MVRBM models are unsupervised

In the field of deep learning, another model that has received wide attention is the Convolutional Neural Network (CNN). CNN has been successfully applied to various image analysis and understanding fields such as positioning, detection, and recognition classification. There are also methods based on CNN and Its variant 3DCNN models the time axis of the video sequence for video classification, which is a difficult and complex task with large model parameters, difficult to train, and requires a lot of training data

Images