End-to-end sign language translation method and system

Abstract

The invention discloses an end-to-end sign language translation method, and belongs to the technical field of sign language translation. The method comprises the following steps: 1) acquiring a sign language video and a corresponding target annotation sequence and target text sequence; 2) establishing a sign language translation model; extracting visual features of the sign language video by a feature extractor and coding the same by a mask coder, and dividing a coding result into three branches for decoding; in the first branch, predicting a word boundary firstly by a boundary predictor, and then predicting an annotation sequence by an auxiliary annotation decoder in combination with an output result of the boundary predictor; in the second branch, linearly mapping the coding result and then using the coding result as the input of a CTC decoder, to generate a prediction annotation sequence; in the third branch, using the coding result as the input of a wait-k decoder, and outputting a prediction text sequence; and 3) carrying out feature extraction and coding on a sign language video to be translated by utilizing the trained sign language translation model, taking a coding result as the input of the wait-k decoder, and generating a prediction text sequence as a translation result.

Description

technical field [0001] The invention relates to the technical field of sign language translation, in particular to an end-to-end sign language translation method and system. Background technique [0002] Sign language is a visual language widely used by about 466 million hearing-impaired people, who use various methods to convey information, such as gestures, movements, mouth shapes, facial expressions, etc. However, it is common for people without hearing impairment to have received sign language education, which makes it difficult for them to understand the meaning of sign language expressions. Sign language translation (SLT) is the use of AI technology to convert sign language videos into spoken language (or text) so that more people can understand it. [0003] The study of sign language translation has a long history. In recent years, with the rise of deep learning, many people are trying to use neural network methods to deal with SLT tasks and have achieved good resul...

AI Technical Summary

Problems solved by technology

Latency is also critical for sign language translation, however, studies done so far in SLT have had to read a full video of sign language to start the translation, which would lead to significant desynchronization between the sign language speaker and the translated text generated by the model

Images