Sign language translation method based on conditional sentence generation and cross-modal rearrangement

By dividing sign language video sequences into fragments, identifying keywords using a three-dimensional convolutional neural network and timing modeler, and generating relevant sentences through a cross-modal rearranger, the existing sign language translation methods are solved with high cost and poor translation performance. The problem is to achieve semantic accuracy and grammatical correct sentence generation, and improve the accuracy and communication convenience of the sign language translation system.

CN112287690BActive Publication Date: 2024-11-26UNIV OF SCI & TECH OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011182427.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-10-29
Publication Date
2024-11-26
Estimated Expiration
2040-10-29

AI Technical Summary

Technical Problem

The existing sign language translation methods require sign language annotation during the learning process. The cost is high and not based on annotation methods, which lead to poor translation performance and poorly generated sentences due to the large gap between modals.

Method used

By dividing sign language video sequences into video clips, using a three-dimensional convolutional neural network to extract timing features, combining a timing modeler to identify keywords, generate smooth sentences, and find the sentences with the highest correlation with the video sequence through a cross-modal rearranger As a translation result.

Benefits of technology

Without the need for sign language experts to annotate the data set, the system masters oral grammar rules, and the generated sentences are semantic, correct and fluent, improving the accuracy of sign language translation and communication convenience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112287690B_ABST
    Figure CN112287690B_ABST
Patent Text Reader

Abstract

The present invention discloses a sign language translation method based on conditional sentence generation and cross-modal rearrangement, comprising: dividing a sign language video sequence into a plurality of video segments, extracting temporal features from a set of video segments by means of a three-dimensional convolutional neural network, and then identifying a set of keywords expressed in the sign language video sequence by means of a temporal modeler; encoding and decoding the keyword set to generate a set of coherent sentences composed of keywords; comparing the set of coherent sentences with the sign language video sequence to find out the sentence with the highest correlation with the sign language video sequence as a sign language translation result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of sign language translation, and in particular to a sign language translation method based on conditional sentence generation and cross-modal rearrangement. Background Art

[0002] Sign language is a very important way of communication for people with hearing impairments. In order to make it easier for these people with disabilities to communicate with normal people who do not understand sign language, a sign language translation system becomes particularly important. A very important part of this system is to enable the computer to accurately perceive a variety of information expressions, including gesture language and facial expressions, and output accurate and fluent spoken sentences for normal people to understand. Sign language uses hand shape changes and arm movements, supplemented by facial expressions and lip movements to express precise semantic information, and in the long-term development, it has standardized grammar, clear semantics and a sound vocabulary system.

[0003] The purpose of sign language translation is to design a certain algorithm through a computer to translate the sign language visual information captured by the camera into fluent spoken sentences. This research can not only promote the development of the field of human-computer interaction, but also provide sign language machine translation for the deaf and hearing-impaired, so that the deaf can communicate with the hearing people more conveniently and facilitate their daily life.

[0004] Sign language translation specifically refers to inputting a video containing complete sign language sentences, identifying the specific meaning expressed, and expressing it in fluent spoken language. Among the existing sign language translation methods, there are two common methods.

[0005] The first method is called a sign language translation algorithm based on sign language annotations. It first performs continuous sign language recognition on the sign language video to identify the sign language words expressed in the video; then, through machine translation and other means, the sign language words are expressed as common spoken language. However, its drawback is that the sign language translation algorithm based on sign language annotations requires the data set to have sign language annotations during the learning process, and the sign language annotations need to be calibrated by experienced sign language experts, which is very costly.

[0006] The second method is a sign language translation algorithm that is not based on sign language annotations. It directly converts sign language videos into spoken sentences through cross-modal mapping. Both methods currently use deep neural networks for data-driven learning. However, the drawback is that sign language translation algorithms that are not based on sign language annotations often have poor translation performance due to the large gap between modalities. And because the data of sign language videos is limited, it is impossible to directly learn the grammatical rules of spoken language, resulting in very unfluent generated spoken sentences. Summary of the invention

[0007] The purpose of the present invention is to provide a sign language translation method based on conditional sentence generation and cross-modal rearrangement, which can effectively improve the translation accuracy and grammatical correctness of a sign language translation system.

[0008] The objective of the present invention is achieved through the following technical solutions:

[0009] A sign language translation method based on conditional sentence generation and cross-modal rearrangement, comprising:

[0010] The sign language video sequence is divided into several video clips, and the temporal features are extracted from the video clip set through a three-dimensional convolutional neural network. Then, the keyword set expressed in the sign language video sequence is identified through a temporal modeler.

[0011] Encode and decode the keyword set to generate a set of coherent sentences consisting of the keywords;

[0012] Compare the fluent sentence set with the sign language video sequence, and find the sentence with the highest correlation with the sign language video sequence as the sign language translation result.

[0013] It can be seen from the technical solution provided by the present invention that it is not necessary for experts to perform sign language annotation standards on the sign language translation data set, and the translation system can master the grammatical rules of spoken language. The sentences generated by this method are not only more semantically accurate, but also grammatically correct and fluent, which greatly facilitates communication between normal people who do not understand sign language and hearing-impaired people. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0015] Figure 1 A schematic diagram of a sign language translation method based on conditional sentence generation and cross-modal rearrangement provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0017] The embodiment of the present invention provides a sign language translation method based on conditional sentence generation and cross-modal rearrangement, which mainly includes:

[0018] 1. Divide the sign language video sequence into several video clips, extract temporal features from the video clip set through a three-dimensional convolutional neural network, and then identify the set of keywords expressed in the sign language video sequence through a temporal modeler.

[0019] 2. Encode and decode the keyword set to generate a set of coherent sentences consisting of keywords.

[0020] 3. Compare the fluent sentence set with the sign language video sequence, and find the sentence with the highest correlation with the sign language video sequence as the sign language translation result.

[0021] like Figure 1 As shown, the above three steps are each implemented by a module, namely a word discriminator, a conditional sentence generator, and a cross-modal rearranger.

[0022] The following introduces the implementation process of this method in combination with these three modules.

[0023] 1. Use the word recognition module to identify the set of keywords expressed in the sign language video.

[0024] The word recognition module contains a three-dimensional convolutional neural network and a time series modeler.

[0025] In the embodiment of the present invention, for a sign language video sequence containing L frames, it is divided into T video segments, each video segment contains w frames, and there are s frames between the segments. In this way, The obtained video clip set is represented as

[0026] Input the video clip set into the 3D convolutional neural network to obtain the temporal features f:

[0027]

[0028] Among them, Ω θ Represents a 3D convolutional neural network with parameters θ.

[0029] The timing modeler includes m first transformer networks, a fully connected layer and a binary classification discriminator.

[0030] The time series features f are input into the m first transformer networks to obtain m corresponding hidden variables ∈, which can be expressed as:

[0031] ∈=(e1,e2…e m )=(r1(f),r2(f),…,r m (f))

[0032] Among them, r is the first transformer network, e is the latent variable obtained by the first transformer network, the subscript is the index of the first transformer network; m is the number of candidate spoken words.

[0033] Then pass the obtained latent variable ∈ through the fully-connected layer and the binary classifier (sigmoidLayer) to obtain the probability of each candidate spoken word appearing in the sign language video sequence:

[0034] p(v i )=sigmoid(w i *e i +b i )

[0035] Among them, sigmoid is the activation function; w i and b i are the weights and bias parameters of the fully connected layer, i = 1, 2, ..., m, for different latent variables e i , weight w i With bias b i are different.

[0036] The probability p(v i ) exceeds the set value of candidate spoken words v i As keywords, a keyword set is obtained.

[0037] During the training phase, the correct labels in the spoken language dataset and the predicted results can be used to make a binary classification loss function, and the word discrimination module can be trained using the stochastic gradient descent method.

[0038] 2. Use the conditional sentence generator to generate smooth spoken sentences using keywords.

[0039] In the embodiment of the present invention, the keyword set S is processed through the word encoding layer (Embedding): H0=Emb(S); and then input into the second transformer network for re-encoding-decoding learning, so as to generate a coherent sentence set composed of keywords through the beam search technology.

[0040] Since the sentence generator does not involve sign language translation, it only generates spoken sentences through spoken words. Therefore, the sentence generator can be pre-trained based on the large amount of spoken language corpus provided by Wikipedia. The input of pre-training is part of the words of the spoken sentence, and the label is the corresponding sentence. After the pre-training is completed, the sentence generator has learned a lot of grammatical rules of spoken language, and the output sentences are grammatically accurate and fluent and easy to read.

[0041] 3. Use the cross-modal rearranger to find the most relevant spoken sentences for the sign language video

[0042] Since the same word set may often generate sentences with completely different semantics, we use fluent spoken sentences to compare with the original sign language video sequence to find the sentence with the highest correlation with the sign language video sequence.

[0043] In the embodiment of the present invention, the transformer neural network (BERT) is used to extract the features of each fluent sentence, the convolutional neural network (CNN) and the third transformer network are used to extract the features of the sign language video sequence, and the cosine distance between the features of each fluent sentence and the features of the sign language video sequence is calculated as the correlation index. The calculation formula is:

[0044]

[0045] Among them, cos(.) means calculating the cosine distance, y Q Represents the features of the sign language video sequence Q, y D represents the features of the fluent sentence D, and ‖.‖ indicates taking the modulus of the features.

[0046] In the embodiment of the present invention, the cross-modal rearranger can be trained in advance by using the ROUGE distance between the generated sentence set and the correct sentence as an evaluation criterion, and finally the sentence with the highest relevance to the video is selected as the final translation result.

[0047] At this point, a sign language video has obtained its corresponding spoken sentence through the above three steps.

[0048] In addition, it should be noted that Figure 1 The keyword set, fluent sentence set, and final sign language translation results shown are all examples and do not constitute a limitation. In addition, the foregoing only roughly describes the three-module training method. Based on such a description, those skilled in the art can complete the training of the three modules in combination with the conventional techniques they have mastered. Moreover, the first, second, and third transformer networks (Transformer) mentioned in the foregoing belong to the same network, and their specific forms may be different, and the specific forms may be implemented by those skilled in the art through conventional techniques as needed.

[0049] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0050] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A sign language translation method based on conditional sentence generation and cross-modal rearrangement, characterized in that: include: The sign language video sequence is divided into several video clips, and the temporal features are extracted from the video clip set through a three-dimensional convolutional neural network. Then, the keyword set expressed in the sign language video sequence is identified through a temporal modeler. Encoding and decoding the keyword set to generate a set of coherent sentences consisting of the keywords, including: processing the keyword set through a word encoding layer, and then inputting it into a second transformer network for encoding-decoding learning, thereby generating a set of coherent sentences consisting of the keywords through a beam search technology; Compare the fluent sentence set with the sign language video sequence, and find the sentence with the highest correlation with the sign language video sequence as the sign language translation result; Among them, the transformer neural network BERT is used to extract the features of each fluent sentence, and the convolutional neural network and the third transformer network are used to extract the features of the sign language video sequence. The cosine distance between the features of each fluent sentence and the features of the sign language video sequence is calculated as the correlation index. The calculation formula is: Among them, cos(.) means calculating the cosine distance, y Q Represents the features of the sign language video sequence Q, y D represents the features of the fluent sentence D, and ‖.‖ indicates taking the modulus of the features.

2. The sign language translation method based on conditional sentence generation and cross-modal rearrangement according to claim 1, characterized in that: The step of dividing the sign language video sequence into a plurality of video segments and extracting temporal features from the video segment set by using a three-dimensional convolutional neural network comprises: A sign language video sequence containing L frames is divided into T video segments, and the resulting video segment set is expressed as Input the video clip set into the 3D convolutional neural network to obtain the temporal features f: Among them, Ω θ Represents a 3D convolutional neural network with parameters θ.

3. The sign language translation method based on conditional sentence generation and cross-modal rearrangement according to claim 1, characterized in that: The keyword set expressed in the sign language video sequence identified by the timing modeler includes: The timing modeler includes m first transformer networks, a fully connected layer and a binary classification discriminator; First, the time series features f are input into the m first transformer networks to obtain m corresponding hidden variables ∈, expressed as: ∈=(e1,e2…e m )=(r1(f),r2(f),…,r m (f)) Where r is the first transformer network, e is the latent variable obtained by the first transformer network, and the subscript is the index of the first transformer network; m is the number of candidate spoken words; Then pass the obtained latent variable ∈ through the fully connected layer and the binary classifier to obtain the probability of each candidate spoken word appearing in the sign language video sequence: p(v i )=sigmoid(w i *e i +b i ) Among them, sigmoid is the activation function; w i and b i are the weights and bias parameters of the fully connected layer; i = 1, 2, …, m; The probability p(v i )Candidate spoken words that exceed the set value are taken as keywords, thereby obtaining a keyword set.