Video continuous sign language recognition method and system based on grammar classifier

Abstract

The invention discloses a video continuous sign language recognition method and system based on a grammar classifier. The method comprises the following steps: segmenting an acquired original sign language video into a plurality of video segments, performing time-space domain feature extraction on each video segment based on a residual connected three-dimensional convolutional neural network, andperforming context learning on the extracted time-space domain features by using a bidirectional long-short-term memory network to obtain features of the sign language video; performing global poolingon the features of the video by adopting a maximum pooling layer to obtain a feature vector of the original sign language video; based on the feature vector, giving a confidence coefficient score corresponding to each word in the sentence by adopting a word classifier module, and giving a confidence coefficient score of each multi-tuple in the sentence by adopting a tuple classifier module; and determining a sign language recognition result based on the confidence score corresponding to each word in the sentence given by the word classifier module and the confidence score of each multi-tuplein the sentence given by the tuple classifier module. The sign language recognition performance can be improved.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a video continuous sign language recognition method and system based on a grammar classifier. Background technique [0002] Sign language is a bridge of communication between deaf and hearing people. Due to the lack of sign language knowledge of most listeners, there is a large communication barrier between deaf people and listeners. In today's information age, this will cause the loss of education and job hunting for the hearing-impaired. In order to alleviate this phenomenon, more and more researchers are devoting themselves to developing sign language recognition systems. For example, video-based sign language recognition systems aim to translate sign language videos into sequential sign language vocabulary to help listeners understand what deaf people are saying in the videos. In a nutshell, sign language recognition is divided into two categories: sign language re...

AI Technical Summary

Problems solved by technology

[0004] Continuous sign language recognition methods based on the encoder-decoder framework usually cannot effectively circumvent the inherent shortcomings of the encoder-decoder structure: during training, the input of the decoder at each time step is the real vocabulary label; , it is impossible for the decoder to obtain the real label, and it can only use the prediction result of the previous time step as the reference information for the prediction of this step

In addition, encoder-decoder-based continuous sign language recognition methods and connectionist temporal classifier-based sign language recognition methods usually only use recurrent neural networks for global context modeling, but cannot explicitly localize the sign language modeling

Images