A sign language video translation method based on the fusion of a time domain convolution network and a cyclic neural network

Abstract

The invention discloses a continuous sign language video translation method based on the fusion of a time domain convolution network and a circulating neural network, which comprises the following steps: feature extraction of the sign language video and construction of a word list; processing of a time domain convolution network TCN; the processing of a bi-directional circulating neural network BGRU; word mapping process of feature; processing of a fusion network FL; optimization of network model parameters based on the fusion of the time domain convolution network TCN and the bi-directional circulating neural network BGRU; fusion and decoding of word coding vectors. The invention can effectively overcome the communication obstacle caused by inaccurate interpretation of sign language translation by utilizing the network structure of different perspectives, learn and analyze different manifestations of data by utilizing different networks, further improve the accuracy of sign language translation, and increase the robustness of sign language translation.

Description

technical field [0001] The invention belongs to the technical field of computer vision, and relates to technologies such as pattern recognition, natural language processing, and artificial intelligence. Specifically, it relates to a continuous sign language video translation method based on the fusion of time-domain convolutional networks and cyclic neural networks. Background technique [0002] Sign language is a way for normal people to communicate with deaf-mute people. It usually consists of a series of meaningful movements consisting of the sign language user's body movements, joint movements and facial expressions. However, there are often communication barriers between normal people who have not learned sign language and sign language users. Therefore, how to capture information such as gestures of sign language users and convert them into information that normal people can understand has gradually attracted people's attention. A good sign language translation system...

AI Technical Summary

Problems solved by technology

The disadvantage of these features is that when extracting features, we only pay attention to the information of each video frame, while ignoring the correlation information and change information between frames in continuous video, which will affect the effect of subsequent recognition.

[0005] In the learning process of sequence models, commonly used models include support vector machines, dynamic time warping algorithms, hidden Markov models, and other traditional models. These models are more suitable for recognizing and translating single sign language words, and for continuous and multiple Videos of gestures are difficult to learn, and it is also difficult to learn the semantic correspondence between actions and words, so it is impossible to effectively translate continuous sign language sentences

Images