A Mute Language Action Recognition Method Based on Two-way 3dCNN Model

Abstract

The invention provides a dumb language action recognition method based on a two-way 3DCNN model, and relates to the technical field of computer vision. The steps of the present invention are as follows: step 1: collect video data sets; step 2: preprocess the video data sets; step 3: set up a 3D convolutional neural network model for extracting gesture local information, and output a set of feature vectors of hands; step 4: Establish a 3D convolutional neural network model that extracts overall global information, and extract a set of overall global feature vectors; Step 5: Establish a two-way 3D convolutional neural network model, and obtain a feature map with local gesture information and global overall information ; Step 6: Obtain the word embedding feature vector, input the feature map and feature vector into the long-short-term memory network that generates dumb sentences, and obtain coherent sentences corresponding to dumb actions through iterative training. This method can correctly capture the gestures and body information of the human body to obtain coherent sentences, and overcomes the problem of only recognizing a single vocabulary.

Description

technical field [0001] The present invention relates to the technical field of computer vision, in particular to a method for recognizing dumb language actions based on a two-way 3DCNN model. Background technique [0002] Mute language is the most natural way for deaf people to communicate and communicate, and it is also an important way for deaf schools to teach and convey ideas. There are currently about 5,500 routine gestures included in Chinese textbooks for the dumb language, each gesture corresponding to a Chinese word. The purpose of dumb language recognition is to provide an effective and accurate mechanism through the computer to translate the dumb language into text or speech to make the communication between deaf-mute and hearing people more convenient and fast. When the deaf-mute communicates with the outside world, if the other party does not understand dumb language, it will cause a lot of trouble. Therefore, many experts and scholars have carried out a series...

AI Technical Summary

Problems solved by technology

[0003] The technical problem to be solved by the present invention is to provide a method for dumb language action recognition based on a two-way 3DCNN model, which can correctly capture gestures and body information of the human body, and also adds time dimension information. Coherent sentences can be obtained, and the customer service can only recognize the problem of a single vocabulary

Images