A Sign Language Word Recognition Method Based on Multimodal Hierarchical Information Fusion

Abstract

The invention discloses a sign language word recognition method based on multi-modal hierarchical information fusion. The method includes: taking the key frame sequence of three modes of color video, depth video and skeleton node video as network input, and constructing a dual-stream I3D network extracts color video and depth video features, integrates dual-modal semantic features through feature splicing, and then uses LSTM to construct long-term spatio-temporal features, and uses SoftMax to classify and score; at the same time, use DST-GCN network to extract spatio-temporal features of skeletal node videos, and then use SoftMax performs classification and scoring; finally, the prediction scores of the two SoftMax layers are fused through decision-level fusion to obtain sign language word recognition results. The sign language word recognition method based on multi-modal hierarchical information fusion proposed by the present invention fully utilizes the complementary information of multi-modal data by constructing a hierarchical fusion strategy; by constructing a DST‑GCN network, the spatio-temporal map volume is enhanced Accumulated network time feature extraction ability, thereby improving the accuracy of sign language word recognition.

Description

technical field [0001] The invention belongs to the technical field of sign language recognition, and specifically refers to a sign language word recognition method based on multimodal hierarchical information fusion. Background technique [0002] According to data, the number of deaf-mute people in my country has exceeded 20.8 million, accounting for about 1.69% of the total population of our country. In addition, there are a large number of people suffering from hearing impairment. For them, sign language is their communication medium, but only a few people have mastered sign language and can communicate with it, which makes deaf people have many communication barriers in their lives. [0003] At present, the main solutions are: manual-based translation and sign language users wearing specific equipment on their hands for translation. However, manual translation requires a lot of labor costs, which can only be used in formal occasions, and ordinary people cannot afford it...

AI Technical Summary

Problems solved by technology

However, manual translation requires a lot of labor costs, which can only be used in formal occasions, and ordinary people cannot afford it; the way of wearing equipment will bring equipment burden to sign language users, and the limitation of equipment will affect normal sign language communication

Images