Translation model training method, translation method and system for sign language video in specific scene

A translation model and training method technology, applied in the field of video natural language generation, can solve problems such as poor translation accuracy

CN112307778APending Publication Date: 2021-02-02NANJING UNIV OF TECH
8 Cites 2 Cited by

Patent Information

Authority / Receiving Office
CN · China
Current Assignee / Owner
Publication Date
2021-02-02

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention discloses a translation model training method, a translation method and a translation system for a sign language video in a specific scene, and belongs to the field of video natural language generation. According to the invention, during training, a filtering network is trained through the constructed actor-double-critic deep reinforcement learning training architecture, then the keyframe sequence set of the sign language video is screened out through the trained filtering network, and finally the deep sequence self-encoding network is trained based on deep learning. The filtering network removes the space-time redundancy of the video frames, and the depth sequence self-coding network fuses the attention mechanism, so that the trained sign language video translation model has high translation accuracy, and the translation method and system of the sign language video in the specific scene are further provided. And on the basis that the trained translation model has relatively high translation accuracy, sign language video translation is carried out, so that the accuracy of sign language video translation is effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention belongs to the field of video natural language generation, is applied to sign language video recognition, and specifically relates to a translation model training method, a translation method and a system of sign language video in a specific scene. Background technique

[0002] There are about 72 million speech and hearing impaired groups in China. This group uses sign language as a tool to communicate, but sign language has not been widely popularized in the whole society. There are many inconveniences when the speech and hearing impaired groups carry out social activities. The current public environmental facilities and product design often ignore the special needs of this group. Especially in some public places such as stations, airports, and civil service places, it is very difficult for normal people who cannot understand sign language to understand the meaning of sign language. This situation hinders communication and communication...

Examples

Embodiment Construction

[0078] The present invention will be further described below in conjunction with embodiment and accompanying drawing.

[0079] In the present invention, the sign language translation model includes a filter network and a deep sequence autoencoder network, wherein the filter network is used for frame sampling of the video, thereby filtering out key frame sequences; the depth sequence autoencoder network is used for feature extraction, and after encoding -The decoding process completes the translation of the sign language video and generates the content text of the sign language video.

[0080] In the embodiment of the present invention, in order to train the model, different data sets are constructed according to different public places. For example, for the scene of the station, the Chinese sign language data set CSL500 is used for reference to construct the sign language data set of the barrier-free windows of the station. The data set has marked a large number of vocabulary,...