Hand model perception-based isolated word sign language identification method

Abstract

The invention discloses a hand model perception-based isolated word sign language identification method, which comprises the following steps that a hand sequence intercepted from a sign language video is converted into a latent semantic representation containing a hand state through a visual encoder; then, a hand model perception decoder works in a model perception mode, the latent semantic representation containing the hand state is mapped into a three-dimensional hand grid, and the position of each hand joint point is obtained; finally, the three-dimensional hand grids are optimized through an inference module, space-time representation of each hand joint point is obtained, classification is conducted, and therefore vocabularies corresponding to the hand sequences are recognized. According to the method, model and data driving can be fused, the hand shape prior is introduced, the recognition accuracy of the system is improved, an intermediate result (i.e., a three-dimensional hand grid) can be visualized, and the interpretability of a framework is enhanced.

Description

technical field [0001] The invention relates to the technical field of sign language recognition, in particular to a hand model-sensed isolated word sign language recognition method. Background technique [0002] According to the statistics of the World Health Organization (WHO) in 2020, there are about 466 million people with hearing impairment in the world, accounting for about 5% of the total global population. Among hearing-impaired people, the most commonly used medium of communication is sign language. As a visual language, sign language has its unique language characteristics. It mainly expresses semantic information through hand-controlled features (hand shape, hand movement and position, etc.) and assisted by fine-grained non-hand-controlled features (expression, lip shape, etc.). [0003] In order to solve the communication gap between hearing people and deaf people, sign language recognition came into being and has been widely researched. Through computer algor...

AI Technical Summary

Problems solved by technology

However, direct data-driven sign language recognition methods have the following problems: limited interpretability; easy to overfit under limited training data

Since the labeling of sign language data requires professional knowledge, the number of samples in each category is relatively small in the existing sign language data sets compared with the action recognition data set. Therefore, the recognition accuracy of the existing schemes needs to be improved

Images