Non-specific person sign language translation method and system based on contrast decoupling element learning

Abstract

The invention discloses a non-specific person sign language translation method and system based on contrast decoupling element learning, and belongs to the field of sign language translation. The model is trained by using sign language video-target sentence pairs marked by a plurality of specific sign language persons and unknown sign language persons. In order to adapt to conditions of non-specific people, the invention provides a new framework-contrast decoupling element learning; the framework is improved in the aspects of model architecture and training mode; specific sign language features and specific task features in the encoder are separated through the decoupling module, and information irrelevant to translation tasks is eliminated, which is not considered in the traditional method. In addition, contrast constraints are calculated using the obtained particular task features and the self-attention representations of the generated words to facilitate frame-word alignment, providing complementary information to each other by decoupling and contrast constraints. The model is trained by using a fine-grained strategy, meta-learning is simultaneously carried out in various sign language person adaptive scenes, and an accurate translation result can be provided for unknown sign language persons.

Description

technical field [0001] The invention relates to the field of sign language translation, in particular to a non-specific sign language translation method and system based on contrastive decoupling meta-learning. Background technique [0002] It is always difficult for people without experience in sign language to understand the expressions of sign language speakers. Sign language translation aims to directly convert sign language videos into a natural sentence (such as figure 1 As shown), the challenges for sign language translation mainly lie in temporal dynamic modeling and fine-grained alignment between different modalities. [0003] A common approach is to train an encoder-decoder model with video-sentence pairs labeled by a specific signer. In general, an encoder extracts visual features from raw video data, and a decoder utilizes these features to generate natural words. Most codec models are built on Long Short-Term Memory (LSTM) or Transformer models. Oscar et al. ...

AI Technical Summary

Problems solved by technology

Although annotations have a strong ability to distinguish video segments, however, annotations are difficult to collect due to their special

The purpose of end-to-end sign language translation is to directly translate the original sign language video into natural sentences, but it needs to overcome the problems of model generalization and accuracy

Images