Real-time sign language recognition system based on improved Transform model

Abstract

The invention discloses a real-time sign language recognition system based on an improved Transform model. The real-time sign language recognition system comprises a data preprocessing module, a feature extraction module and a sign language translation module. The data preprocessing module extracts sign language video key frames in real time through an inter-frame difference method, and performs character recognition and image segmentation through a YOLOv5 algorithm; the feature extraction module detects and extracts hand skeleton and mouth shape features by using a Mediape model, and then performs data enhancement and normalization processing to obtain a feature set; the sign language translation module improves a Transform model through knowledge distillation, takes the Transform model as a teacher model, pre-trains CNN and RNN as student models, and receives a feature set for translation; the system can effectively reduce the interference of complex backgrounds and illumination conditions, and remarkably improves the robustness of sign language recognition in practical application.

Images