Sign language translation video generation method and system based on deep learning

A technology of sign language translation and deep learning, applied in the field of sign language translation video generation method and system

Active Publication Date: 2020-01-24
TSINGHUA UNIV +1
View PDF5 Cites 16 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, due to the limitation of computing resources and capabilities at this stage, fine video generation is still a challenge
Moreover, sign language video generation needs to integrate the translation-related language model and the generative model of video images, and no related work has been able to achieve this challenge

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Sign language translation video generation method and system based on deep learning
  • Sign language translation video generation method and system based on deep learning
  • Sign language translation video generation method and system based on deep learning

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0028] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, in which the same or similar reference numerals indicate the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to explain the present invention, but should not be construed as limiting the present invention.

[0029] In the embodiments of the present invention, in order to make it more convenient for the hearing impaired to communicate with ordinary people, it is a very critical task for the computer to automatically perform sign language translation and sign language video generation. Different from translation tasks in the usual sense, sign language translation needs to construct a mapping between text and image sequence. The feature space of images, especially sequence images, is very large, and it is qu...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a sign language translation video generation method and system based on deep learning, and the method comprises the steps: carrying out the cyclic training of a variational auto-encoder model and an annular generative adversarial network model through a sign language video of a data set till a preset dimension is reached, and outputting an image low-dimensional feature sequence; training a language translation model through a language text sequence of the data set, and outputting a text low-dimensional feature sequence corresponding to the image low-dimensional featuresequence to construct a mapping relationship between the language text sequence and the sign language video; and based on the mapping relationship, inputting the language text sequence to be tested into the language translation model, mapping an image low-dimensional feature sequence corresponding to the language text sequence to be tested, and decoding the image low-dimensional feature sequence by the video model to generate a corresponding sign language video. According to the method provided by the embodiment of the invention, dimension reduction processing is carried out on the image by constructing the compressed hidden space, the extremely high requirements of a video generation problem on computer storage and calculation capabilities are solved, and translation training can be carried out by using a universal language model.

Description

Technical field [0001] The present invention relates to the technical fields of computer vision and natural language processing, in particular to a method and system for generating sign language translation videos based on deep learning. Background technique [0002] The disabled, especially the hearing impaired, occupy a large proportion in the current society. How to protect the social life of the disabled and facilitate the communication between the deaf-mute and ordinary people has become an important issue. Countries and regions with sound social security attach great importance to the education and employment of disabled people, but communication with deaf-mute people requires relevant sign language education for everyone, which is too expensive. Therefore, if a computer can automatically complete the translation and conversion of sign language and natural language, it will greatly facilitate and improve the daily life of deaf-mute people. [0003] With the continuous develo...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06K9/00G06F40/58
CPCG06V40/28G06V20/49
Inventor 徐枫薄子豪娄昕杨东兰怡娜雍俊海戴琼海
Owner TSINGHUA UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap