Multi-task learning sign language translation method based on syntax tree

Abstract

The invention discloses a multi-task learning sign language translation method based on a syntax tree, and relates to sign language translation. Comprising the following steps: 1) obtaining a syntax tree of spoken language sentences, and constructing a data set; 2) building a neural network which is mainly divided into an encoder and a decoder; after the encoder obtains the input abstract feature representation, the input abstract feature representation is input into a decoder to be decoded; and 3) predicting a preorder traversal sequence of the syntax tree, the depth of each node of the syntax tree and spoken language sentences. And the translation performance of the model is improved in a multi-task learning mode. The method is not only suitable for a sign language translation process, but also can be used for neural machine translation tasks. And the robustness of translation is better than that of a basic Transform model. In the model decoding process, not only spoken language sentences but also the syntax trees corresponding to the spoken language sentences need to be predicted, and deep information hidden in a training data set can be more fully mined through hard parameter sharing, so that the prediction result of the translation model is more accurate.

Description

technical field [0001] The invention relates to sign language translation, in particular to a syntax tree-based multi-task learning sign language translation method. Background technique [0002] As a relatively special visual language, sign language expresses real semantics through multiple channel information. It can be divided into hand features and non-hand features as a whole: hand features include the shape, position, direction and movement of both hands; non-hand features Hand features are mainly body posture changes, including facial expressions, eyes, mouth, elbows, and torso. Although sign language is completely different from Chinese, English and other languages, it does have a set of language rules. In the task of mutual translation between English and Chinese, it is basically possible to translate one by one from front to back, but sign language translation needs to be understood and translated from the two dimensions of time and space. For most people, sign l...

AI Technical Summary

Problems solved by technology

In the field of sign language translation, although there is work studying simultaneous recognition and translation, no work attempts multi-task learning during translation

Images