A multi-task learning sign language translation method based on syntax tree

Abstract

A multi-task learning sign language translation method based on syntax tree, involving sign language translation. The method comprises the following steps: 1) obtaining a syntax tree of a spoken sentence and constructing a data set; 2) building a neural network, which is mainly divided into two parts, an encoder and a decoder; after the encoder obtains the abstract feature representation of the input, it is input into the decoder for decoding; 3) predicting the pre-order traversal sequence of the syntax tree, the depth of each node of the syntax tree, and the spoken sentence. The translation performance of the model is improved by multi-task learning. It is not only applicable to the translation process of sign language translation, but also can be used for neural machine translation tasks. It is better than the basic Transformer model in terms of translation robustness. In the model decoding process, not only the spoken sentence is predicted, but also its corresponding syntax tree is predicted, and hard parameters are shared to more fully explore the deep information hidden in the training data set, so that the prediction result of the translation model is more accurate.

Description

Technical Field

[0001] The invention relates to sign language translation, and in particular to a multi-task learning sign language translation method based on a grammar tree. Background Art

[0002] As a special visual language, sign language expresses real semantics through multiple channels of information. It can be generally divided into hand features and non-hand features: hand features include the shape, position, direction and movement of the hands; non-hand features are mainly changes in body posture, including facial expressions, eyes, mouth, elbows, and torso movements. Although the expression form of sign language is completely different from that of Chinese, English and other languages, it does have a set of language rules. In the task of translating between English and Chinese, it can basically be translated from front to back, but sign language translation needs to be understood and translated from two dimensions of time and space. For most people, sign language is very difficult to understand without professional and systematic learning. Therefore, it is a very meaningful thing to enable the deaf and mute to communicate normally with normal people through sign language translation.

[0003] Sign language translation mainly includes two processes: recognition and translation. Recognition is to recognize the sign language video as a sign language vocabulary sequence; translation is to translate the sign language vocabulary sequence into spoken sentences. The sign language vocabulary sequence is an annotation of the sign language video, which marks the actual meaning of different sign language gestures in the sign language video and is the most basic spoken word. Figure 1 As shown, a specific example of German sign language translation is presented to distinguish the relevant concepts in sign language translation.

[0004] In the field of sign language translation, researchers generally believe that recognition has an important impact on translation, so a lot of research work focuses on the recognition process, while research on the translation process is rare. In the field of neural machine translation, a lot of work has proven that incorporating syntax trees into translation models can improve the translation performance of the model, but there is no work studying how to incorporate syntax trees into sign language translation. In the past two years, multi-task learning has become one of the hot topics of research. Without adding additional data, by designing multiple related tasks, the implicit information of the data set itself is fully utilized to assist in improving the generalization ability of each task. In the field of sign language translation, although there is work studying simultaneous recognition and translation, there is no work trying to perform multi-task learning during the translation process. Summary of the invention

[0005] The purpose of the present invention is to introduce multi-task learning in the translation process and propose a more robust multi-task learning sign language translation method based on grammar tree (Multi-task Learning Sign Language Translation Based on Parse Tree, MLSLTBPT). In the model decoding process, not only the spoken sentence is predicted, but also its corresponding grammar tree is predicted, and hard parameters are shared to more fully mine the deep information hidden in the training data set, so that the prediction result of the translation model is more accurate.

[0006] The present invention comprises the following steps:

[0007] 1) Obtain the syntax tree of the spoken sentence and construct the data set;

[0008] 2) Build a neural network;

[0009] 3) Predict the pre-order traversal sequence of the syntax tree, the depth of each node of the syntax tree, and the spoken sentence.

[0010] In step 1), the syntax tree of the spoken sentence is obtained by using the Berkeley syntax analyzer to obtain the pre-order traversal sequence of the syntax tree. Since the pre-order traversal sequence corresponds to many specific structures of the syntax tree, the specific structure of the syntax tree can be restored by using depth information; the data set is constructed to construct the data set required for multi-task learning, the input is the sign language vocabulary sequence, and the output includes the pre-order traversal sequence of the syntax tree, the depth of each node of the syntax tree, and the spoken sentence, a total of three parts;

[0011] The input sign language vocabulary sequence is GLS = {g1, g2, g3, ..., g w}, where g i Represents the i-th word.

[0012] In step 2), the neural network is based on the open source Transformer model and is mainly divided into two parts: encoder and decoder. The sign language vocabulary sequence is first embedded and positionally encoded, and then input into the encoder. The encoder is composed of N identical layers stacked together, each layer consisting of two sub-layers, namely a multi-head attention mechanism and a feedforward neural network, and each sub-layer is added with residual connection and regularization. Assuming the input is x, the output of the sub-layer is expressed as:

[0013] sub-layer-out=LayerNorm(x+(SubLayer(x))) (1)

[0014] The multi-head attention mechanism is evolved from the attention mechanism, which can be expressed in the following form:

[0015]

[0016] Among them, Q, K, and V are three vectors involved in the attention mechanism. Q represents a query, and its similarity with each K is calculated as the weight. Then the weight is normalized using the softmax function, and finally the weight and the corresponding V are weighted summed. The multi-head attention mechanism projects Q, K, and V through h different linear transformations, and then splices the results of different attention mechanisms together:

[0017] MultiHead(Q,K,V)=Concat(head1,...,head h )W O (3)

[0018] Among them, head i It can be expressed as:

[0019] head i =Attention(QW i Q , KW i K , VW i V ) (4)

[0020] The feedforward neural network provides a nonlinear transformation to convert the output of the previous layer of the network into a suitable dimension;

[0021] After the encoder obtains the abstract feature representation of the input, it is input into the decoder for decoding; the decoder is similar to the encoder in structure, except that it also includes a masked multi-head attention mechanism. Since the attention mechanism can see all the words before and after each position, and decoding is a sequential operation process, when the decoder predicts a specific word at the kth time step, it can only see the first k-1 prediction results, so some content must be masked; at the same time, in the multi-head attention mechanism of the decoder, K and V come from the output of the encoder, while Q comes from the previous output of the decoder.

[0022] In step 3), the specific steps of predicting the pre-order traversal sequence of the syntax tree, the depth of each node of the syntax tree, and the spoken sentence may be:

[0023] First, the output of the decoder is converted to the dimension of the vocabulary size through a linear layer, and then the word probability distribution is calculated through the softmax function; for the pre-order traversal sequence of the syntax tree, the depth of each node of the syntax tree, and the spoken sentence, three branches are required for prediction, and the final loss function is:

[0024]

[0025] Among them, α and β are two hyperparameters. Since the nodes and depth of the syntax tree are inseparable, the weights of the loss functions of the two are set to be the same; lp and lt represent the length of the predicted syntax tree and the spoken sentence respectively; They are the probability distributions of the three predicted at a certain time step; p (i) ,y l (i) ,y t (j) They represent the true one-hot vectors of the three respectively; <·, ·> represents the inner product; the final training goal is to minimize L.

[0026] Compared with the prior art, the present invention has the following outstanding advantages and technical effects:

[0027] The present invention is based on the multi-task learning sign language translation of the syntax tree, that is, the multi-task learning is introduced in the translation process. Since the sign language vocabulary sequence is a brief description of the sign language video, it does not have an obvious grammatical structure, so it is impossible to add syntax tree information at the source end of the model, but the target port sentence has a complete grammatical structure. Therefore, at the target end, not only the spoken sentence can be predicted, but also its corresponding syntax tree can be predicted. The present invention improves the translation performance of the model through such a multi-task learning method.

[0028] Since there is no obvious grammatical structure in the sign language vocabulary sequence, after predicting the grammatical tree of the spoken sentence, it can be added to the source end of the model and iterative training can be performed to better utilize the grammatical tree information. Because the deeper the predicted grammatical tree, the greater the difficulty, the present invention can further study the impact of predicting grammatical trees of different depths on model performance.

[0029] Because the translation process of the neural machine translation task is basically the same as that of the sign language translation task, the present invention is not only applicable to the translation process of sign language translation, but can also be used for neural machine translation tasks. Because in the neural machine translation task, both the source and target texts have complete grammatical structures, based on the present invention, grammatical tree information can be added to the source end of the model to further improve the translation performance of the model. Experimental results show that the present invention is better than the basic Transformer model in terms of translation robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The following is a specific example of German Sign Language translation.

[0031] Figure 2 It is a schematic diagram of the overall architecture of the present invention. DETAILED DESCRIPTION

[0032] The following embodiments will further illustrate the present invention in conjunction with the accompanying drawings.

[0033] The embodiment of the present invention includes the following steps:

[0034] 1) Obtain the syntax tree of the spoken sentence and construct a data set; because the syntax tree is to be used, the Berkeley syntax analyzer is used to obtain the syntax tree of the spoken sentence. Because the tool obtains the pre-order traversal sequence of the syntax tree, and this sequence can correspond to many specific structures of the syntax tree, so depth information is also needed to restore the specific structure of the syntax tree. And it is necessary to construct the data set required for multi-task learning. The input is the sign language vocabulary sequence, and the output includes the pre-order traversal sequence of the syntax tree, the depth of each node of the syntax tree, and the spoken sentence.

[0035] 2) Build a neural network. The neural network is based on the open source Transformer model and is mainly divided into two parts: encoder and decoder. The input sign language vocabulary sequence is first embedded and positionally encoded, and then input into the encoder. The encoder is composed of N identical layers stacked together, each layer consists of two sub-layers, namely the multi-head attention mechanism and the feedforward neural network, and each sub-layer is added with residual connection and regularization.

[0036] The multi-head attention mechanism is evolved from the attention mechanism. The multi-head attention mechanism projects the three vectors Q, K, and V involved in the attention mechanism through h different linear transformations, and then splices the results of different attention mechanisms. The feedforward neural network provides nonlinear transformation to convert the output of the previous layer of the network into a suitable dimension.

[0037] After the encoder obtains the abstract feature representation of the input, it is input into the decoder for decoding. The decoder has the same structure as the encoder, but with an additional masked multi-head attention mechanism. In the multi-head attention mechanism of the decoder, K and V come from the output of the encoder, while Q comes from the previous output of the decoder.

[0038] 3) Predict the pre-order traversal sequence of the syntax tree, the depth of each node of the syntax tree, and the spoken sentence. First, the output of the decoder is converted to the dimension of the vocabulary size through a linear layer, and then the word probability distribution is calculated through a softmax function. Since it is necessary to predict the pre-order traversal sequence of the syntax tree, the depth of each node of the syntax tree, and the spoken sentence, three branches need to be set for prediction.

[0039] like Figure 2 As shown, the present invention is based on Transformer and mainly consists of three modules, namely an encoder module, a decoder module and a prediction module.

[0040] 1. Data format

[0041] The input is a sign language vocabulary sequence GLS = {g1, g2, g3, ..., gw}, where g i Represents the i-th word;

[0042] The output consists of three parts: the pre-order traversal sequence of the syntax tree, the depth of each node of the syntax tree, and the spoken sentence.

[0043] 2. Encoder module

[0044] The input sign language vocabulary sequence is first embedded and positionally encoded, and then input into the encoder. The encoder is composed of N identical layers stacked together, each layer consists of two sublayers, namely the multi-head attention mechanism and the feedforward neural network, and each sublayer is added with residual connection and regularization. The output of the sublayer is expressed as

[0045] sub-layer-out=LayerNorm(x+(SubLayer(x)))

[0046] The multi-head attention mechanism is evolved from the attention mechanism, which is expressed in the following form:

[0047]

[0048] Q, K, and V are three vectors involved in the attention mechanism. Q represents a query, and its similarity with each K is calculated as the weight. Then the weight is normalized using the softmax function, and finally the weight and the corresponding V are weighted summed. The multi-head attention mechanism projects Q, K, and V through h different linear transformations, and then concatenates the results of different attention mechanisms:

[0049] MultiHead(Q,K,V)=Concat(head1,...,head h )W O

[0050] Among them head i It can be expressed as:

[0051] head i =Attention(QW i Q , KW i K , VW i V )

[0052] Feedforward neural networks provide a nonlinear transformation that converts the output of the previous layer of the network into a suitable dimension.

[0053] 3. Decoder module

[0054] After the encoder obtains the abstract feature representation of the input, it needs to be input into the decoder for decoding; the decoder has the same structure as the encoder, with an additional masked multi-head attention mechanism, because the attention mechanism can see all the words before and after each position, and decoding is a sequential operation process. When the decoder predicts a specific word at the kth time step, it can only see the first k-1 prediction results, so some content needs to be masked. At the same time, in the multi-head attention mechanism of the decoder, K and V come from the output of the encoder, while Q comes from the previous output of the decoder.

[0055] 4. Prediction Module

[0056] First, the output of the decoder is converted to the dimension of the vocabulary size through a linear layer, and then the word probability distribution is calculated through a softmax function. Since it is necessary to predict the pre-order traversal sequence of the syntax tree, the depth of each node of the syntax tree, and the spoken sentence, three branches are required for prediction, so the final loss function is:

[0057]

[0058] Among them, α and β are two hyperparameters. Since the nodes and depth of the syntax tree are inseparable, the weights of the loss functions of the two are set to be the same; lp and lt represent the length of the predicted syntax tree and the spoken sentence respectively; They are the probability distributions of the three predicted at a certain time step; p (i) ,y l (i) ,y t (j) They represent the true one-hot vectors of the three respectively; <·, ·> represents the inner product; the final training goal is to minimize L.

[0059] Table 1 gives some case studies of sign language translation.

[0060] Table 1

[0061]

[0062] As shown in Table 1, three sign language translation examples are listed for Chinese, English and German respectively. Since Transformer is a widely used baseline model, the present invention is compared with it. In the first and second examples, it can be seen that the prediction results of the present invention are closer to the target text in expression and semantics; in the third example, the prediction results of Transformer obviously miss some content, which is a common situation in the field of neural machine translation, while the present invention can better predict the results, which further illustrates that the present invention can improve the robustness of the translation model.

Claims

1. A multi-task learning sign language translation method based on syntax tree, characterized in that The following steps are involved: 1) Obtain the syntax tree of the spoken sentence and construct the data set; The grammar tree of the spoken sentence is obtained by using the Berkeley grammar analyzer to obtain a pre-order traversal sequence of the grammar tree. Since the pre-order traversal sequence corresponds to many specific structures of the grammar tree, the specific structure of the grammar tree is restored by using depth information; The constructed data set is a data set required for constructing multi-task learning. The input is a sign language vocabulary sequence, and the output includes a pre-order traversal sequence of a syntax tree, the depth of each node of the syntax tree, and a spoken sentence, a total of three parts; 2) Build a neural network; The neural network is based on the open source Transformer model and is divided into two parts: encoder and decoder. The sign language vocabulary sequence is first embedded and positionally encoded, and then input into the encoder. The encoder is composed of N identical layers stacked together, each layer consisting of two sublayers, namely a multi-head attention mechanism and a feedforward neural network, and each sublayer is added with residual connections and regularization. Assuming the input is x, the output of the sublayer is expressed as: sub_layer_out=LayerNorm(x+(SubLayerx(x))) (1) The multi-head attention mechanism is evolved from the attention mechanism, which is expressed in the following form: Among them, Q, K, and V are three vectors involved in the attention mechanism. Q represents a query, and its similarity with each K is calculated as the weight. Then the weight is normalized using the softmax function, and finally the weight and the corresponding V are weighted summed. The multi-head attention mechanism projects Q, K, and V through h different linear transformations, and then splices the results of different attention mechanisms together: MultiHead(Q,K,V)=Concat(head1,…,head h )W O (3) Among them, head i It is expressed as: head i =Attention(QW i Q ,KW i K ,VW i V ) (4) The feedforward neural network provides a nonlinear transformation to convert the output of the previous layer of the network into a suitable dimension; After the encoder obtains the abstract feature representation of the input, it is input into the decoder for decoding; The structure of the decoder is similar to that of the encoder, except that it also includes a masked multi-head attention mechanism. In the multi-head attention mechanism of the decoder, K and V come from the output of the encoder, while Q comes from the previous output of the decoder. 3) predict the pre-order traversal sequence of the syntax tree, the depth of each node of the syntax tree, and the spoken sentence; The specific steps of predicting the pre-order traversal sequence of the syntax tree, the depth of each node of the syntax tree, and the spoken sentence are: First, the output of the decoder is converted to the dimension of the vocabulary size through a linear layer, and then the word probability distribution is calculated through the softmax function; for the pre-order traversal sequence of the syntax tree, the depth of each node of the syntax tree, and the spoken sentence, three branches are required for prediction, and the final loss function is: Among them, α and β are two hyperparameters. Since the nodes and depth of the syntax tree are inseparable, the weights of the loss functions of the two are set to be the same; lp and lt represent the length of the predicted syntax tree and the spoken sentence respectively; are respectively used to predict the pre-order traversal sequence of the syntax tree, the depth of each node in the syntax tree, and the probability distribution of the spoken sentence at a certain time step; p (i) ,y l (i) ,y t (j) They represent the pre-order traversal sequence of the syntax tree, the depth of each node in the syntax tree, and the real one-hot vector of the spoken sentence respectively; <·,·> represents the inner product; the final training goal is to minimize L.

2. A grammar tree-based multi-task learning sign language translation method as claimed in claim 1, characterized in that In step 1), the sign language vocabulary sequence is GLS = {g1, g2, g3, ..., g w }, where g i Represents the i-th word.

Images