Sign language video translation method based on fusion of temporal convolutional network and recurrent neural network

Abstract

The invention discloses a continuous sign language video translation method based on the fusion of time-domain convolutional network and cyclic neural network, comprising the following steps: feature extraction of sign language video and construction of word list; processing of time-domain convolutional network TCN; two-way loop The processing of neural network BGRU; the word mapping process of features; the processing of fusion network FL; the optimization of network model parameters based on the fusion of time-domain convolutional network TCN and bidirectional recurrent neural network BGRU; fusion and decoding of word encoding vectors. The present invention can use network structures from different perspectives to effectively overcome communication barriers caused by inaccurate interpretation of sign language translation, use different networks to study and analyze different forms of data, and further improve the accuracy of sign language translation , to increase the robustness of sign language translation.

Description

technical field [0001] The invention belongs to the technical field of computer vision, and relates to technologies such as pattern recognition, natural language processing, and artificial intelligence. Specifically, it relates to a continuous sign language video translation method based on the fusion of time-domain convolutional networks and cyclic neural networks. Background technique [0002] Sign language is a way for normal people to communicate with deaf-mute people. It usually consists of a series of meaningful movements consisting of the sign language user's body movements, joint movements and facial expressions. However, there are often communication barriers between normal people who have not learned sign language and sign language users. Therefore, how to capture information such as gestures of sign language users and convert them into information that normal people can understand has gradually attracted people's attention. A good sign language translation system...

AI Technical Summary

Problems solved by technology

The disadvantage of these features is that when extracting features, we only pay attention to the information of each video frame, while ignoring the correlation information and change information between frames in continuous video, which will affect the effect of subsequent recognition.

[0005] In the learning process of sequence models, commonly used models include support vector machines, dynamic time warping algorithms, hidden Markov models, and other traditional models. These models are more suitable for recognizing and translating single sign language words, and for continuous and multiple Videos of gestures are difficult to learn, and it is also difficult to learn the semantic correspondence between actions and words, so it is impossible to effectively translate continuous sign language sentences

Images