Method for converting gestures into Chinese-Tibetan bilingual speech

Abstract

The present invention provides a method of converting gestures into a Chinese-Tibetan bilingual speech. The method comprises the steps that sample data is used for carrying out gesture recognition ongestures to be recognized to obtain meanings of the gestures; the meanings of the gestures are expressed in a Chinese-Tibetan bilingual form to obtain a semantic definition of the gestures, and context-related annotations of the gestures are generated according to the semantic definition of the gestures; a speaker-related Tibetan model or a Mandarin model is obtained through self-adaptive trainingof the speaker and using the training corpus of a specific speaker in Mandarin or Tibetan, and the Tibetan speech or Mandarin speech are synthesized by using a speaker-related Tibetan model or a Mandarin model and the context-sensitive annotations of the gestures. The method can convert the input static gestures and dynamic gestures into Mandarin or Tibetan language, can promote the daily communication activities of speech impaired persons and normal persons, solves the speech output problem in the communication between disabled persons and the normal persons, and can further be applied to the aspects such as computer assisted instruction for deaf and dumb, bilingual broadcasting of TV programs.

Description

technical field [0001] The invention relates to the technical field of gesture recognition and voice conversion, in particular to a method for converting gestures into Chinese-Tibetan bilingual voices. Background technique [0002] At present, technologies such as gesture recognition, natural language understanding and generation, and speech synthesis based on computer vision have become increasingly mature. However, a gesture-to-speech translation system for people with disabilities has not yet been promoted and applied in real life. Therefore, in order to meet the needs of speech-impaired persons, if a research method of gesture-to-speech real-time conversion can be proposed with the help of vision-based gesture recognition technology and increasingly mature speech synthesis technology, it will be able to promote speech-impaired persons and The daily communication activities of normal people effectively solve the problem of voice output in the communication between disabl...

AI Technical Summary

Problems solved by technology

[0004] However, most of the gesture recognition methods in the existing technology are independently researched on the problems of speech synthesis system and gesture recognition. There is a lack of research on the combination of gesture recognition and speech synthesis, and in gesture recognition, there is also a lack of gesture recognition based on deep learning. recognition frame

Images