Sign language translation method and sign language-voice interaction method and system based on MR

Abstract

The invention discloses a sign language translation method and a sign language-voice interaction method and system based on MR. The RGB-D images of the sign language of a sign language user are collected and translated, the virtual character model of the sign language user in a virtual scene reads the sign language paraphrases into audio, and the virtual character model of a voice user demonstrates the sign language actions corresponding to the voice paraphrases. According to the invention, real-time translation of sign language actions is realized; the translation method can be used for interacting with a voice user; the method can be applied to various scenes, especially in doctor-patient communication scenes, doctors and patients can perform barrier-free communication in a familiar language mode, the diagnosis and treatment efficiency is improved, the experience of the two parties is enhanced, the MR technology shows the doctor-patient communication scenes after sign language translation, and the perfect effect of doctor-patient communication of the deaf-mute patients is achieved. And the sign language translation system can be more accurate by using the deep learning technologyto train the model.

Description

technical field [0001] The invention relates to MR technology, in particular to a sign language translation method, an MR-based sign language-voice interaction method and system. Background technique [0002] As a visual language, sign language mainly expresses meaning through the movement of hands and arms supplemented by appropriate facial expressions and mouth shapes. It is an important way for deaf-mute patients to communicate with the outside world. According to the latest statistics, the number of deaf-mute patients in my country has reached 20.57 million, accounting for 1.67% of the total population. However, sign language medical staff in hospitals in my country is very scarce, so it is inconvenient for deaf-mute patients to seek medical treatment and often delays diagnosis. How to improve the quality of life of deaf-mute patients and enable them to better integrate into society is a subject of national key research. [0003] With the further development of MR techn...

AI Technical Summary

Problems solved by technology

According to the latest statistics, the number of deaf-mute patients in my country has reached 20.57 million, accounting for 1.67% of the total population. However, the number of sign language medical staff in our hospitals is very scarce, so it is inconvenient for deaf-mute patients to seek medical treatment and often delays diagnosis

Images