3D convolutional neural network sign language identification method integrated with multi-modal data

Abstract

The invention discloses a 3D convolutional neural network sign language identification method integrated with multi-modal data. The 3D convolutional neural network sign language identification methodintegrated with multi-modal data includes the steps: constructing a deep neural network, respectively performing characteristic extraction on an infrared image and a contour image of a gesture from the spatial dimension and the time dimension of a video, and integrating two network output based on different data formats to perform final classification of sign language. The 3D convolutional neuralnetwork sign language identification method integrated with multi-modal data can accurately extract the limb movement track information in two different data formats, can effectively reduce the computing complexity of a model, uses a deep learning strategy to integrate the classification results of two networks, and effectively solves the problem that single classifier encounters error classification because of data loss, so as to enable the model to have relatively higher robustness for illumination and background noise of different scenes.

Description

technical field [0001] The invention belongs to the technical field of educational informatization, and more specifically relates to a dynamic sign language recognition method and system based on a 3D convolutional neural network, which can be applied to somatosensory interaction for special groups of deaf-mute people in the environment of science and technology museums. Background technique [0002] Sign language is the most beneficial tool for the deaf-mute to communicate with each other and for the deaf-mute to communicate with normal people. It is also the most important and natural way for the deaf-mute to obtain information services, participate in social life on an equal footing, and share social material and cultural achievements. At the same time, dynamic sign language has a very high application value in the field of human-computer interaction because of its strong visual effects, vivid and intuitive features. [0003] The existing gesture recognition methods follo...

AI Technical Summary

Problems solved by technology

This method uses RGB-D images to train the deep network model. Due to the relatively large amount of data, the calculation and analysis of data is slow, and the complexity of the algorithm is high. not ideal

Images