Multi-modal fusion sign language recognition system and method based on graph convolution

Abstract

The invention discloses a multi-modal fusion sign language recognition system and method based on graph convolution. The system comprises a feature extraction module, a feature fusion module, a sequence learning module and an alignment translation module. The method comprises the following steps: 1, respectively extracting color, depth and skeleton features of video frames from a sign language video database by using a convolutional neural network and a graph neural network; 2, combining the multi-modal features, and fusing the network fusion features through a multi-modal sequence; 3, constructing a bidirectional recurrent neural network to perform sequence learning on a series of fused fragment-level features; and 4, aligning the feature sequence through the connected subjective timing sequence classification model, and translating a complete sign language sentence. According to the invention, translation of continuous sign language sentences can be realized, and the accuracy of continuous sign language translation is improved.

Description

technical field [0001] The invention belongs to the field of multimedia information processing, and relates to technologies such as computer vision, natural language processing, and deep learning, and specifically relates to a multi-modal fusion sign language recognition system and method based on graph convolution. Background technique [0002] Able-bodied people can communicate easily using spoken language, while deaf or voiceless people need to convey their thoughts through sign language. Since most able-bodied people lack the foundation of sign language education, there are obstacles in promoting sign language for normal social communication. Technology is good, and sign language recognition technology facilitates the integration of deaf people into society to a certain extent. [0003] Early sign language recognition research focused on discrete sign language recognition, which is essentially a special video classification problem. With the development of video unders...

AI Technical Summary

Problems solved by technology

[0004] Due to the complexity of video data, existing sign language recognition has many disadvantages, especially in the representation and fusion of multi-modal data

When existing sign language recognition methods use data from multiple modal sources, they often ignore the complementary associations between different modalities, and perform violent fusion of them, and the robustness of fusion features is poor; on the other hand, in feature During the learning process, the time and space characteristics in the video data stream are less explored, and the time-varying characteristics of sign language features are not fully utilized, resulting in poor coherence and low accuracy of sign language translation results

Images