Sign language recognition method based on space-time attention mechanism

Abstract

The invention discloses a sign language recognition method based on a space-time attention mechanism. The method comprises the following steps: firstly, sampling a sign language video into a continuous sign language sequence with a uniform length as input of a model; and then, inputting the video frame sequence into a spatial attention network formed by 3D residual blocks, so that the network canautomatically pay attention to a salient region in a space. Analyzing the extracted convolution features through a ConvLSTM convolution long-short-term memory network, extracting long-time sequence features, and distributing time attention weights of different video frames to generate feature representation of the video; and finally, the generated feature representation passes through a Softmax classifier, and classification categories are output in a vector form. According to the invention, the interference of redundant information on recognition can be reduced, and the recognition accuracy is improved.

Description

technical field [0001] The invention belongs to the field of image processing and pattern recognition, in particular to a sign language recognition method based on a spatio-temporal attention mechanism. Background technique [0002] The purpose of sign language recognition is to translate sign language into text or voice for output, so as to achieve the purpose of communication between deaf people and normal people, and between deaf people and deaf people. Sign language contains picture information and motion information, so how to effectively extract the spatio-temporal features in sign language is the focus of sign language recognition research. [0003] The traditional sign language recognition framework includes four parts: preprocessing, gesture detection, feature extraction, and classification. Among them, feature extraction and classification are the two key parts. Therefore, researchers focus on designing complex artificial features, while using traditional machine...

AI Technical Summary

Problems solved by technology

However, the general CNN-RNN-based network is lacking in the effectiveness of spatial feature extraction, and cannot highlight the differences in time, so the extracted spatio-temporal features contain a lot of redundant information, and the spatio-temporal correlation is not strong.

Images