Continuous sign language statement recognition method based on modal matching

Abstract

The invention discloses a continuous sign language statement recognition method based on modal matching, and the method comprises the steps: taking a color video of a sign language statement, a key frame of an optical flow image sequence and a target word segment sequence as input, and, through a continuous sign language statement recognition model based on modal matching, matching and aligning the color video of the sign language statement and the key frame fragment sequence of the optical flow image sequence with semantics to obtain a final semantic sequence. The invention discloses a continuous sign language statement recognition model based on modal matching, which uses a lightweight feature extraction network to reduce the parameter quantity, performs task specific training for a sign language data set, and recognizes continuous sign language statements under the condition that the number of labels of samples in the data set is small; and the problem of high dependence on human body posture information in a traditional sign language recognition method is reduced.

Description

technical field [0001] The invention belongs to sign language recognition technology, in particular to a continuous sign language sentence recognition method based on modality matching. Background technique [0002] Sign language recognition uses computer technology to analyze the semantics of sign language used by humans, so that computers can understand sign language and convert sign language, a body language, into easy-to-understand text, voice and other forms of expression. [0003] With the rapid popularization of artificial intelligence, human-computer interaction technology has made human beings feel unprecedented convenience, which is of great significance in promoting the harmonious development of human society. For the first time, a continuous sign language sentence recognition method based on modal matching is proposed to align video and semantics, using color video and its corresponding optical flow video key frame sequence as input, enhancing data representation...

AI Technical Summary

Problems solved by technology

[0004] Sign language recognition based on wearable devices generally uses data gloves, sensors or some motion capture devices to collect hand movement data, extracts effective information from these data, and then models and classifies hand gestures. The recognition method can indeed accurately locate the hand, and the accuracy is high. However, data gloves are expensive, and sign language speakers must wear bulky gloves during demonstrations, which often make sign language demonstrators feel constrained; sign language recognition based on traditional machine learning is usually divided into two categories: Three steps, data preprocessing, feature extraction and modeling recognition, usually use algorithms such as scale invariant feature transformation and gradient direction histogram to design features manually, and then use traditional machine learning classifiers for modeling and recognition, such as SVM, HMM Such as traditional classifiers, sign language recognition based on traditional machine learning requires manual design of feature extraction methods, cannot automatically obtain image features, and relies on manual experience, which is time-consuming and labor-intensive

[0005] The feature extraction network structure is relatively complex, and training on a large-scale sign language data set will have a large amount of parameters and is very time-consuming, so that the task of continuous sign language sentence recognition cannot be completed in a targeted manner

In addition, the output sentence of sign language sentence recognition has the problem that it cannot conform to the daily grammatical relationship when the sentence structure is relatively complex, and it is difficult to train the codec network on a data set that lacks labels, which is not very practical.

Images