A sign language translation method and system based on class annotation attention and knowledge transfer

Abstract

The application discloses a sign language translation method and system based on class annotation attention and knowledge transfer, and belongs to the field of unannotated sign language translation. A sign language video and corresponding spoken language text are combined as a training set, and a sign language video embedded feature vector sequence and a spoken language text embedded feature vector sequence are respectively acquired; an encoder containing a multi-head class annotation attention mechanism is used to encode the sign language video embedded feature vector sequence, and a decoder autoregression is used to generate target spoken language text; a loss function term based on knowledge transfer and a cross entropy loss function term of sign language translation are combined to train the encoder and the decoder; and the trained encoder and the decoder are used to realize an unannotated sign language translation task. The application improves the translation effect in the unannotated sign language translation task by introducing the class annotation attention mechanism and the knowledge transfer.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unannotated sign language translation, and in particular to a sign language translation method and system based on class annotation attention and knowledge transfer. BACKGROUND

[0002] At present, sign language translation tasks have attracted widespread attention and become an important artificial intelligence task, which aims to convert sign language videos into spoken language sentences, so that the deaf or hearing impaired can more easily communicate with normal hearing people.

[0003] Existing technologies can be divided into three categories, 1) two-stage annotated supervised methods, 2) end-to-end annotated supervised methods, and 3) end-to-end unannotated methods.

[0004] The first two methods rely on annotations, i.e., time-ordered annotated sign language words, to assist the model in learning alignment and semantic information. However, the annotation requires sign language experts to spend a lot of time to complete, so it is expensive and tedious. At present, existing technologies have begun to focus on end-to-end unannotated methods, which directly learn to translate sign language videos into spoken language sentences without the help of annotations, making the method more versatile. However, the deficiency is that the position of the semantic boundary in the continuous sign language video is not fully learned, and the semantic of the sign language video is not fully understood globally. If these two problems are solved, the accuracy of sign language translation can be effectively improved. SUMMARY

[0005] In order to overcome the defects that the position of the semantic boundary in the continuous sign language video is not fully learned and the semantic of the sign language video is not fully understood globally in the prior art, the present application provides a sign language translation method and system based on class annotation attention and knowledge transfer to translate sign language videos into spoken language texts.

[0006] The specific technical solutions adopted by the present application are as follows:

[0007] In a first aspect, the present application provides a sign language translation method based on class annotation attention and knowledge transfer, comprising the following steps:

[0008] 1) Obtain a combination of sign language videos and corresponding spoken language texts as a training set, and obtain sign language video embedding feature vector sequences and spoken language text embedding feature vector sequences, respectively;

[0009] 2) encode the sign language video embedding feature vector sequence using an encoder containing a multi-head class annotation attention mechanism, and then generate the target spoken language text using a decoder autoregression;

[0010] 3) Obtain a loss function item based on knowledge transfer according to the one-to-one semantic relationship between the sign language video and the spoken language text in the training set; take the spoken language text embedding feature vector sequence and the sign language video embedding feature vector sequence encoded by the encoder as the input of the decoder, and obtain the cross-entropy loss function item of sign language translation for the process of generating the target spoken language text by the decoder autoregression; combine the loss function item based on knowledge transfer and the cross-entropy loss function item of sign language translation to train the encoder and the decoder;

[0011] 4) Obtain the sign language video embedding feature vector sequence to be translated, and use the trained encoder containing the multi-head class annotation attention mechanism to encode the sign language video embedding feature vector sequence, and then use the trained decoder to autoregressively generate the target spoken language text.

[0012] Further, the method for obtaining the sign language video embedding feature vector sequence and the spoken language text embedding feature vector sequence is:

[0013] Sign language video: extract the feature sequence of the sign language video, and for each element in the feature sequence, sequentially pass through a linear layer, batch normalization and a nonlinear activation function, and add position encoding to obtain the sign language video embedding feature vector sequence;

[0014] Spoken language text: use the subword embedding in the pre-trained subword segmentation model as the initialization value of the embedding layer, and for the word vector obtained after the initialization value is word-embedded, sequentially pass through a linear layer, batch normalization and a nonlinear activation function, and add position encoding to obtain the spoken language text embedding feature vector sequence.

[0015] Further, the multi-head class annotation attention mechanism is specifically:

[0016] 3.1) Transversely splice each feature vector in the sign language video embedding feature vector sequence with a length of T, and then take the transpose to obtain a representation matrix X;

[0017] 3.2) For the X obtained in step 3.1), generate a query matrix Q, an original key matrix K, and an original value matrix V, respectively;

[0018] 3.3) For the query matrix Q, equivalently represent it as a block splicing of several column vectors, and take the transpose of the matrix formed, denoted as Q = [q1; q2; …; qN], where qN is the Nth query vector; for each query vector qN, initialize N attention positions P = (p1, p2, …, pN), where pN is the Nth attention position; T T , where q T is the Nth query vector; for each query vector q t , initialize N attention positions P = (p1, p2, …, p N ), where p N is the Nth attention position;

[0019] ​3.4) Calculate N offsets according to the query vector to dynamically adjust the position of attention;

[0020] 3.5) Represent the original key matrix K as K = [k1; k2;... ; kt;... ; kT], and the original value matrix V as V = [v1; v2;... ; vt;... ; vT], where ktis the t-th original key vector, and vtis the t-th original value vector; use linear interpolation to obtain the final key matrix K' used for calculation T ] T and the value matrix V' used for calculation t T ] T t where vtis the t-th original value vector; use linear interpolation to obtain the final key matrix K' used for calculation and the value matrix V' used for calculation

[0021] Iterate through the position t ∈ [1, T] to obtain T linearly interpolated key matrices and T linearly interpolated value matrices Synthesize the T linearly interpolated key matrices into the key matrix K', and synthesize the T linearly interpolated value matrices into the value matrix V';

[0022] 3.6) Use the key matrix K' and the value matrix V' obtained in step 3.5) to calculate a series of single-head class annotation attention vectors; iterate through the position t ∈ [1, T] to obtain T single-head class annotation attention vectors, and synthesize the single-head class annotation attention matrix Z;

[0023] 3.7) Repeat steps 3.2) to 3.6) to finally produce multiple different single-head class annotation attention matrices by generating different query matrices Q, original key matrices K, and original value matrices V in step 3.2), and aggregate to obtain the multi-head class annotation attention matrix Z

[0024]

[0025] where W represents the projection matrix, concat(·) represents the operation of concatenating matrices in the first dimension, and Zh h represents the h-th single-head class annotation attention matrix.

[0026] Further, in step 3.4), the calculation formula of the offset and the adjustment of the attention position is:

[0027] O = W o q t

[0028]

[0029] where O represents the offset matrix, W o represents the mapping matrix, ​​For the adjusted attention position, % is the integer division remainder operator.

[0030] Further, in step 3.5), the linear interpolation formula of the key and the linear interpolation formula of the value are as follows:

[0031]

[0032]

[0033]

[0034] wherein, is the adjusted i-th attention position, represents the floor function, b i represents the result of the adjusted i-th attention position after floor operation, u i represents the result of the adjusted i-th attention position after ceiling operation; and represent the b i -th original key vector and the original value vector, respectively, and represent the u i -th original key vector and the original value vector, respectively, represents the linear interpolation key of the i-th key vector of the original key matrix at the t-th position, represents the linear interpolation value of the i-th key vector of the original value matrix at the t-th position.

[0035] Further, in step 3.6), the formula for calculating the single-head class annotation attention vector is:

[0036]

[0037] wherein, z t is the t-th single-head class annotation attention vector, <·,·> is the scaled dot product of two vectors, and exp· indicates the exponential function with base natural logarithm e.

[0038] Further, the encoder is composed of a plurality of encoder layers stacked together, and each encoder layer is constructed by layer normalization operation, multi-head class annotation attention mechanism, residual connection and feedforward neural network according to the following formula:

[0039] z=MHGA(LN(x))+x

[0040]

[0041] wherein, x represents the input vector of the encoder layer, and z represents the intermediate result in the encoder layer, denotes the output vector of the encoder layer, LN(·) denotes a layer normalization operation, MHGA(·) denotes a multi-head class-annotation attention mechanism, and MLP(·) denotes a feed-forward neural network.

[0042] Further, the knowledge transfer-based loss function term acquisition method is:

[0043] 3.1) Using the Sentence BERT semantic similarity pre-training model, obtaining pre-training embedding vectors of sentences in the oral text, and calculating a cosine similarity matrix S between all sentences;

[0044] 5.2) Directly splicing each vector in the encoded sign language video embedding feature sequence to obtain an embedding vector e representing the entire sign language video i , traversing all sign language videos in the training set;

[0045] 5.3) Calculating the mean square error of the cosine similarity between the embedding vector e of each video i and the cosine similarity between the oral sentences to obtain a knowledge transfer-based loss function term:

[0046]

[0047] wherein, denotes the knowledge transfer-based loss function term, S[i,j] represents an element located in the ith row and jth column of the cosine similarity matrix S, and e i ·e j denotes the dot product of the ith and jth sign language video embedding vectors, ||·|| denotes the vector norm, and D train denotes the number of sentences in the oral text.

[0048] In a second aspect, the present application provides a sign language translation system based on class-annotation attention and knowledge transfer, comprising:

[0049] a sign language video preprocessing module for obtaining a sign language video embedding feature vector sequence;

[0050] an oral text preprocessing module for obtaining an oral text embedding feature vector sequence;

[0051] an encoder module comprising a multi-head class-annotation attention mechanism for encoding the sign language video embedding feature vector sequence;

[0052] a decoder module for autoregressively generating target oral text based on the oral text embedding feature vector sequence and the sign language video embedding feature vector sequence encoded by the encoder during the training stage, and for autoregressively generating target oral text based on the sign language video embedding feature vector sequence encoded by the encoder during the actual translation stage;

[0053] The training module is used for obtaining a loss function item based on knowledge transfer according to a one-to-one semantic relationship between a sign language video and a spoken language text in a training set; embedding a feature vector sequence of the spoken language text and a feature vector sequence embedded by an encoder of the sign language video as inputs of a decoder, obtaining a cross-entropy loss function item of sign language translation for a process of autoregressive generation of target spoken language text of the decoder; and training the encoder and the decoder by combining the loss function item based on knowledge transfer and the cross-entropy loss function item of sign language translation.

[0054] Compared with the prior art, the present application has the beneficial effects that:

[0055] The present application is a method for solving the unannotated sign language translation task by using the class annotation attention mechanism and knowledge transfer.

[0056] (1) By using the class annotation attention mechanism, the present application enables the model to implicitly learn the position of the semantic boundary in the continuous sign language video and enables it to focus attention on the video segment with the same local semantics, thus replacing the expensive annotation, realizing efficient unannotated sign language translation.

[0057] (2) By transferring the knowledge of sentence and sentence similarity from the natural language model to the sign language translation model, the present application enables the model to know which sign language videos are similar in semantics and which are different in semantics, so as to help the model better understand the semantic features of the sign language video at the sentence level, realizing efficient unannotated sign language translation.

[0058] In summary, by jointly using the class annotation attention mechanism and the knowledge transfer from the spoken language text, the present application can fully learn the local semantic boundary and global semantic features of the sign language video, replace the expensive annotation, provide the model with local and global semantic supervision information, and realize efficient unannotated sign language translation. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is a schematic diagram of a single-head class annotation attention mechanism;

[0060] Figure 2 is a schematic diagram of the overall framework of the sign language translation method based on the class annotation attention and knowledge transfer according to the embodiment of the present application, wherein <bos>and <eos>respectively represent the start symbol and the end symbol;

[0061] Figure 3 is a specific structure of the encoder layer;

[0062] Figure 4 is a specific structure of the decoder layer. DETAILED DESCRIPTION

[0063] The present application will be further described and explained with reference to the accompanying drawings and specific embodiments. (For the sake of convenience, the encoder and the decoder are simplified as single layers, and the complete encoder and the decoder are stacked by multiple layers as shown in Figure 2 )

[0064] The method for solving the unannotated sign language translation task by using the annotation-like attention mechanism and knowledge transfer includes the following steps:

[0065] Step 1: obtaining a sign language video embedding feature vector by using a single-layer feedforward neural network;

[0066] Step 2: obtaining a spoken language text embedding feature vector by using a single-layer feedforward neural network;

[0067] Step 3: constructing a multi-head annotation-like attention mechanism (MHGA);

[0068] Step 4: encoding the sign language video embedding feature vector by using an encoder containing the annotation-like attention mechanism;

[0069] Step 5: obtaining a loss function term based on knowledge transfer by using the one-to-one semantic relationship between the sign language video and the annotated spoken language text;

[0070] Step 6: for the process of generating the target spoken language text by the decoder autoregression, the cross-entropy loss function term of the sign language translation is calculated by using the following formula:

[0071] Step 7: combining the loss function terms obtained in steps 4 and 5 to obtain the final complete loss function, and completing the unannotated sign language translation task by training.

[0072] The steps 1 and 2 above for obtaining the sign language video embedding feature vector and the spoken language text embedding feature vector can be specifically implemented as follows.

[0073] The training set is organized as follows, there are D train sign language videos and D train corresponding spoken language sentences, wherein each sign language video corresponds to a spoken language sentence, and each sign language video and the corresponding spoken language sentence are input into the network at the same time during the training of the network, and the gradient descent learning method is used to optimize the loss function described below At test time, a sign language video is inputted, encoded by the same encoder as in training, and the decoder autoregressively outputs a spoken sentence.

[0074] For each given sign language video, it is extracted as a sequence of features X' = (x'1, x'2,..., x'T) by the pre-trained I3D network. T T is the length of the sequence, and the elements in the sequence are feature vectors. For each element in the sequence X', it is passed through a linear layer, followed by batch normalization and a non-linear ReLU activation function, and position encoding is added to obtain the sign language video embedding feature vector The calculation formula is as follows:

[0075]

[0076] where x r ′ is the rth feature vector in the feature sequence X ′ , W1 and b1 are the weight matrix and bias vector of the linear layer respectively, BN(·) represents the batch normalization operation, relu(·) represents the ReLU activation function, f pos (·) represents the position encoding function, and r represents the rth position of the feature sequence X ′ .

[0077] Finally, the sequence of sign language video embedding feature vectors X is obtained.

[0078] For each given spoken sentence (containing U words) in the training set, first add the special symbol <bos>, as the 0th word of the spoken sentence, to mark the beginning of the sentence (as shown in Figure 2 ), and then initializing the spoken text embedding feature vector using a pre-trained subword segmentation model (BPEmb). Specifically, the subword embedding in the pre-trained subword segmentation model (BPEmb) is used as the initialization value y ′ m of the embedding layer. For the word vector Emb(y ′ m ) obtained after word embedding, the representation of the spoken text is calculated according to the following formula:

[0079]

[0080] wherein, is the word embedding feature vector of the mth word of the spoken sentence, W2 and b2 are the weight matrix and bias vector of the linear layer respectively, BN(·) represents the batch normalization operation, relu(·) represents the ReLU activation function, and f pos (·) represents the position encoding function.

[0081] Finally, the word embedding feature vector sequence corresponding to the spoken sentence is obtained

[0082] wherein, U is the number of words contained in the spoken sentence, is the word embedding feature vector of each word of the spoken sentence (the 0th word is the special symbol <bos>).

[0083] The multi-head annotation attention mechanism (MHGA) is constructed as described in step 3. For the convenience of description, a special case of the multi-head annotation attention mechanism, the single-head annotation attention mechanism, is described first, and the flowchart is as shown in Figure 1 Specifically:

[0084] 3.1) For the feature vector sequence obtained in step 2 , the matrix is recorded as The matrix X is transposed by horizontally splicing each feature vector in the feature vector sequence

[0085] 3.2) For the X obtained in step 3.1), the query matrix Q, the original key matrix K, and the original value matrix V are respectively generated according to the following formula:

[0086] Q = W Q X

[0087] K = W K X

[0088] V = W V X

[0089] Wherein, the query matrix the original key matrix the original value matrix the query mapping matrix the original key mapping matrix the original value mapping matrix

[0090] Q = [q1; q2; …; q T ] T ; wherein each column vector is called a query vector (1≤t≤T), and N attention positions P = (p1, p2, …, p N ) are initialized; wherein, p1 to p N increase by 1, and t represents the tth position of the feature sequence X';

[0091] 3.4) Calculate N offsets according to the input query to dynamically adjust the position of attention; calculate according to the following formula:

[0092]

[0093] Wherein, t is defined as 3.3), represents the tth position of the feature sequence X', q t is the input query, and O is the offset matrix, For the mapping matrix, The adjusted attention position is T, where T is the length of the input feature vector sequence, and % is the integer division modulo operator.

[0094] 3.5) The original bond matrix Represented as K = [k1; k2; ...; k T ] T , where k p For the p-th original key vector, Original value matrix Represented as V = [v1; v2; ...; v T ] T , where v q For the q-th original value vector, Linear interpolation is used to obtain the final key matrix used for calculation. Sum matrix The key and the value after linear interpolation are calculated using the following formula:

[0095]

[0096]

[0097]

[0098] Where t is defined as in 3.3), it represents the t-th position of the feature sequence X′. express The i-th element, Indicates to Round down to the nearest integer, and denote the result as b. i ;u i for The result of rounding up is given by b. i +1 is calculated to obtain u i ; and They represent the bth i One original key vector and one original value vector, and Representing the uth i One original key vector and one original value vector,

[0099] Traverse positions t∈[1,T] to obtain T linearly interpolated key matrices. And the value matrix after T linear interpolations All have dimensions N×D; combine the T linearly interpolated key matrices into a key matrix K' of dimension T×N×D, and combine the T linearly interpolated value matrices into a value matrix V' of dimension T×N×D, such as Figure 1 As shown;

[0100] 3.6) Using the key matrix K' and the value matrix V' obtained in step 3.5), a series of single-head class annotation attention vectors z are calculated according to the following formula: t :

[0101]

[0102] where t is defined as in 3.3), represents the t-th position of the feature sequence X', i and j represent the i-th and j-th positions of the feature sequence X' respectively, q t is the t-th query vector, and are obtained in step 3.5), and represent the i-th key and the i-th value after linear interpolation respectively, z t is the t-th single-head class annotation attention vector, <·,·> is the scaled dot product of two vectors, and exp· represents the exponential function with base e.

[0103] Traverse the position t ∈ [1, T] to obtain T single-head class annotation attention vectors, which are directly combined into a single-head class annotation attention matrix Z of dimension T × D, as shown in Figure 1 ;

[0104] To calculate the multi-head class annotation attention mechanism, the operations of steps 3.1) to 3.6) need to be repeated. Due to the different initializations of the query mapping matrix, the original key mapping matrix, and the original value mapping matrix in step 3.2), multiple different single-head class annotation attention matrices will be generated; assuming that the operation is repeated h times, i.e., the multi-head class annotation attention mechanism with h heads is calculated, then h different single-head class annotation attention matrices Z i will be generated, where i ∈ [1, h], and the final multi-head class annotation attention matrix is calculated according to the following formula:

[0105]

[0106] where the projection matrix concat(·) is an operation that concatenates matrices in the first dimension, and Z1, Z2, …, Z h are h single-head class annotation attention matrices of dimension T × D respectively;

[0107] In step 4, for the sign language video embedding feature vector sequence obtained in step 1, each vector (r is defined as in step 1, representing the r-th position of the feature sequence X ′ ) is input into an encoder containing a class annotation attention mechanism; as Figure 3 , the encoder is composed of several encoder layer stacks, each of which includes a layer normalization operation (LN), the multi-head attention mechanism (MHGA) described in step 3, a residual connection, and a feed-forward neural network (MLP), which is constructed according to the following formula:

[0108]

[0109]

[0110] wherein, is the input vector of the encoder layer, z r is the intermediate result within the encoder layer, is the output vector of the encoder layer;

[0111] The final encoded sign language video embedding feature sequence

[0112] In step 5, the loss function term based on knowledge transfer is obtained using the one-to-one semantic relationship between the sign language videos and the annotated spoken language texts in the training set; specifically:

[0113] 5.1) Use the Sentence BERT semantic similarity pre-training model to obtain the pre-training embedding vectors of the sentences in the spoken language text, and calculate the cosine similarity matrix between all sentences wherein D train is the number of sentences in the spoken language text;

[0114] 5.2) For the encoded sign language video embedding feature sequence These T vectors are directly concatenated to obtain an embedding vector representing the entire sign language video D is the embedding vector dimension of the entire sign language video;

[0115] For the D train sign language videos in the training set, an embedding vector representing each video is obtained For ease of distinction, let e i (where 1≤i≤D train ) be the embedding vector of the i-th video.

[0116] 5.3) By minimizing the mean square error of the cosine similarity between the embedding vectors e i of each video and the cosine similarity between the spoken language sentences, the model learns the degree of similarity between the sign language videos in terms of language, so as to transfer the semantic information of the spoken language into the learning process of the embedding vector of each video (by backpropagation of the gradient during the training process to update the parameters in the encoder, which determine how the embedding vector of each video is encoded); the loss function term based on knowledge transfer is calculated using the following formula​

[0117]

[0118] in, of Let S[i,j] represent the loss function, and the subscript kt is an abbreviation for knowledge transfer. S[i,j] represents the element in the i-th row and j-th column of the cosine similarity matrix S. i ·e j Let |e| represent the dot product of the i-th and j-th sign language video embedding vectors. i ||||e j || represents the product of the moduli of the embedding vectors of the i-th and j-th sign language videos;

[0119] In step 6, the process of the decoder generating the target spoken text through autoregression is essentially modeling a sign language video. Generate a spoken sentence The conditional probability, i.e. Where p(·) represents probability. Indicates that in a given Generate under the conditions During training, sign language videos and corresponding spoken sentences are input into the model in pairs; the input process for each pair of videos and spoken sentences is the same, starting with a sign language video. and its corresponding spoken sentences For example, sign language videos After processing in steps 1, 3, and 4 sequentially, the encoded sign language video embedding feature sequence is obtained. Where T is the sequence length, and the corresponding spoken statement Using the method in step 2, the word embedding feature vector sequence corresponding to the spoken sentence is obtained. Where U is the number of words in the spoken sentence. The word embedding feature vectors for each word in a spoken sentence (the 0th word is a special symbol). <bos>During training, the decoder needs to go through U-step decoding steps. In each decoding step, the encoded sign language video is embedded into the feature sequence. Word embedding feature vector sequence Part of Input decoder together (e.g.) Figure 2 As shown in the figure, u represents the u-th decoding step (1≤u≤U);

[0120] The cross-entropy loss function for sign language translation is calculated using the following formula from the outputs of all U-step decoders:

[0121]

[0122] in, This indicates translation loss; the subscript T is an abbreviation for Translation. Indicates the d-th word w d In decoding step u, the true probability is given by D, where D is the target language vocabulary size, and h is the target language vocabulary size. u This indicates the decoder output at decoding step u. Indicates in h u Generated under conditions The conditional probability, ∏ is the product symbol, and ∑ is the summation symbol;

[0123] In step 7, the loss function terms based on knowledge transfer obtained in steps 5 and 6 are... Loss function term for sign language translation By minimizing the joint loss term To train the encoder-decoder network consisting of the encoder in step 4 and the decoder in step 6 (e.g.) Figure 2 (As shown), joint loss item Migration loss and translation loss Weighted sum:

[0124]

[0125] Where, λ kt , λ T For weight parameters, the subscript kt is an abbreviation for knowledge transfer, and the subscript T is an abbreviation for translation.

[0126] Using this loss function and the gradient descent learning method, all parameters involved in the two factors considered simultaneously in the model are trained to generate spoken text from sign language videos, thus completing the unannotated sign language translation task.

[0127] The above method will be applied to the following examples to embody the technical effects of the present application, and the specific steps in the examples will not be repeated.

[0128] The present application is tested on three real sign language translation datasets PHOENIX14T, CSL-Daily, SP-10, and each dataset tests the unannotated sign language translation task. In order to objectively evaluate the performance of the present application, the present application uses two evaluation standards of BLEU and ROUGE-L to evaluate the effect of the present application in the selected test set, and compares with the following prior art models:

[0129] Comparison 1. Conv2d-RNN model, which is the first proposed unannotated sign language translation model, which uses an encoder-decoder architecture based on gated recurrent unit (GRU) for sequence modeling;

[0130] Comparison 2. Tokenization-SLT model, which reaches the state-of-the-art level on the ROUGE score of PHOENIX14T dataset, which uses adversarial learning, multi-task learning and transfer learning to find a semi-supervised tokenization method to reduce the dependence on vocabulary annotation;

[0131] Comparison 3. Joint-SLT model, which is the first sign language translation model based on Transformer architecture, which jointly learns the tasks of sign language recognition and sign language translation;

[0132] Comparison 4. TSPNet model, which reaches the state-of-the-art level on the BLEU score of PHOENIX14T dataset, which improves the translation performance by learning the hierarchical features of sign language, wherein TSPNet-Sequential refers to the version of applying scale-to-scale and scale-to-scale attention in sequence, and TSPNet-Joint refers to the version of joint modeling of local video and non-local video.

[0133] According to the steps described in the specific embodiment, the experimental results obtained are shown in Tables 1 to 3, and the model of the present application is represented as GASLT.

[0134] Table 1: Test results of unannotated sign language translation task obtained by the present application for PHOENIX14T dataset

[0135] Model ROUGE-L BLEU-I BLEU-2 BLEU-3 BLEU-4 Conv2d-RNN 29.7 27.1 15.61 10.82 8.35 Joint-SLT 31.1 30.88 18.57 13.12 10.19 Tokenization-SLT 36.28 37.22 23.88 17.08 13.25 TSPNet-Sequential 34.77 35.65 22.8 16.6 12.97 TSPNet-Joint 34.96 36.1 23.12 16.88 13.41 GASLT 39.86 39.07 26.74 21.86 15.74

[0136] Table 2: Test results of unannotated sign language translation task obtained by the present application for CSL-Daily dataset

[0137] Model ROUGE-L BLEU-I BLEU-2 BLEU-3 BLEU-4 Joint-SLT 12.23 12.49 6.79 3.99 1.6 TSPNet-Joint 15.18 13.55 7.07 3.77 2.2 GASLT 16.98 21.72 10.92 6.61 4.35

[0138] Table 3: Test results of the unannotated sign language translation task obtained by the present application for the SP-10 dataset

[0139] Model ROUGE-L BLEU-I BLEU-2 BLEU-3 BLEU-4 Joint-SLT 19.61 21.56 8.29 3.68 1.72 TSPNet-Joint 18.38 17.09 8.98 5.07 2.97 GASLT 20.35 19.9 9.94 5.98 4.07

[0140] As can be seen from Table 1, the translation performance of the GASLT method is significantly better than the two previous state-of-the-art unannotated sign language translation models, Tokenization-SLT and TSPNet-Joint, with BLEU4 score improved from 13.41 to 15.74 (17.37%), and ROUGE-L score improved from 36.28 to 39.86 (9.86%).

[0141] As can be seen from Tables 2 and 3, the method of GASLT is better than the existing method on the two datasets. Benefiting from the injection of prior information about semantic time positioning in the annotation-like attention mechanism and the flexible attention span proposed in the present application, the GASLT model can keep the attention in the right place. With the help of knowledge transfer, the GASLT model is better than the previous unannotated sign language translation methods, which shows that the method of using annotation-like attention mechanism and knowledge transfer is effective for the unannotated sign language translation task.

[0142] In this embodiment, a sign language translation system based on annotation-like attention and knowledge transfer is also provided, which is used to implement the above-mentioned embodiments. The terms "module", "unit" and the like used below can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware implementation is also possible.

[0143] A sign language video preprocessing module for obtaining a sequence of sign language video embedding feature vectors;

[0144] A spoken language text preprocessing module for obtaining a sequence of spoken language text embedding feature vectors;

[0145] An encoder module comprising a multi-head annotation-like attention mechanism for encoding the sequence of sign language video embedding feature vectors;

[0146] A decoder module for autoregressively generating the target spoken language text according to the sequence of spoken language text embedding feature vectors and the sequence of sign language video embedding feature vectors encoded by the encoder in the training stage, and autoregressively generating the target spoken language text according to the sequence of sign language video embedding feature vectors encoded by the encoder in the actual translation stage;

[0147] The training module is used for obtaining a loss function item based on knowledge transfer according to a one-to-one semantic relationship between a sign language video and a spoken language text in a training set; embedding a spoken language text feature vector sequence and a sign language video embedding feature vector sequence coded by an encoder as inputs of a decoder, obtaining a cross-entropy loss function item of sign language translation for a process of autoregressive generation of a target spoken language text by the decoder; and training the encoder and the decoder by combining the loss function item based on knowledge transfer and the cross-entropy loss function item of sign language translation.

[0148] For the system embodiment, since it basically corresponds to the method embodiment, the relevant part is described in the method embodiment, and the implementation method of the remaining modules is not described here. The system embodiment described above is only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0149] The system embodiment of the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The system embodiment can be realized by software, hardware or a combination of software and hardware. Taking software realization as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor of the device with data processing capability.

[0150] The above is only a specific embodiment of the present application. Obviously, the present application is not limited to the above embodiment, and there are many variations. All variations that can be directly derived or inferred by those skilled in the art from the disclosure of the present application should be considered as the protection scope of the present application.< / bos> < / bos> < / bos> < / eos> < / bos>

Claims

1. A sign language translation method based on class annotation attention and knowledge transfer, characterized in that, The method comprises the following steps: 1) obtaining a combination of sign language video and corresponding spoken language text as a training set, and obtaining a sign language video embedded feature vector sequence and a spoken language text embedded feature vector sequence respectively; 2) using an encoder comprising a multi-head class annotation attention mechanism to encode the sign language video embedded feature vector sequence, and using a decoder autoregression to generate target spoken language text; The multi-head class annotation attention mechanism is specifically: 2.1) transversely splicing each feature vector in the sign language video embedded feature vector sequence with a length of T, and then taking a transpose to obtain a representation matrix X; 2.2) generating a query matrix Q, an original key matrix K and an original value matrix V respectively for X obtained in step 2.1); 2.3) For the query matrix Q, express its equivalent representation as a concatenation of several column vector blocks, construct the transpose of the resulting matrix, denoted as Q = [q1; q2;... ; qN]T. T ] T where qNis the Nth query vector; for each query vector qN, initialize N attention positions P = (p1, p2,..., pN), where pNis the Nth attention position. T t N N ​​​​ 2.4) calculating N offsets according to the query vector to dynamically adjust the attention position; 2.5) Represent the original bond matrix K as K = [k1; k2; ...; k T ] T , where k t For the t-th original key vector, the original value matrix V is represented as V = [v1; v2; ...; v T ] T , where v t Let t be the t-th original value vector; use linear interpolation to obtain the final key matrix used for computation. Sum matrix T linear interpolated key matrices are obtained by traversing positions t e [1, T] T linear interpolated value matrices are obtained by traversing positions t e [1, T] The T linear interpolated key matrices are synthesized into a key matrix K', and the T linear interpolated value matrices are synthesized into a value matrix V'. 2.6) using the key matrix K' and the value matrix V' obtained in step 2.5) to calculate a series of single-head class annotation attention vectors; traversing the position t∈[1,T], T single-head class annotation attention vectors are obtained, and a single-head class annotation attention matrix Z is synthesized; 2.7) Repeat steps 2.2) to 2.6) by generating different query matrix Q, original key matrix K, original value matrix V in step 2.2) to finally produce multiple different single-head class annotation attention matrices, and aggregate to obtain multi-head class annotation attention matrix where W denotes a projection matrix, concat(·) denotes an operation of concatenating matrices in the first dimension, Z h denotes the h-th single-head class annotation attention matrix; 3) obtaining a loss function term based on knowledge transfer according to the one-to-one semantic relationship between the sign language video and the spoken language text in the training set; taking the spoken language text embedded feature vector sequence and the sign language video embedded feature vector sequence encoded by the encoder as the input of the decoder, obtaining a cross-entropy loss function term of sign language translation for the process of autoregression of the decoder to generate target spoken language text; combining the loss function term based on knowledge transfer and the cross-entropy loss function term of sign language translation to train the encoder and the decoder; 4) obtaining a sign language video embedded feature vector sequence to be translated, and using the trained encoder comprising the multi-head class annotation attention mechanism to encode the sign language video embedded feature vector sequence, and then using the trained decoder to autoregression to generate target spoken language text.

2. The sign language translation method based on class-annotation attention and knowledge transfer according to claim 1, characterized in that, The method for obtaining the sign language video embedded feature vector sequence and the spoken language text embedded feature vector sequence is: Sign language video: extracting a feature sequence of the sign language video, for each element in the feature sequence, sequentially passing through a linear layer, batch normalization and a nonlinear activation function, and adding position encoding to obtain a sign language video embedded feature vector sequence; Spoken language text: using the subword embedding in the pre-trained subword segmentation model as the initialization value of the embedding layer, for the word vector obtained after the word embedding of the initialization value, sequentially passing through a linear layer, batch normalization and a nonlinear activation function, and adding position encoding to obtain a spoken language text embedded feature vector sequence.

3. The sign language translation method based on class-annotation attention and knowledge transfer according to claim 1, characterized in that, In step 2.2), the calculation formulas of the query matrix Q, the original key matrix K and the original value matrix V are as follows: Q = W Q X K = W K X V = W V X where W Q represents the query mapping matrix, W K represents the original key mapping matrix, W V represents the original value mapping matrix.

4. The sign language translation method based on class-annotation attention and knowledge transfer of claim 1, wherein, In step 2.4), the calculation formulas of the offset and the adjusted attention position are as follows: O = W o q t where O denotes an offset matrix, W o denotes a mapping matrix, is the adjusted attention position, % is the integer division remainder operator.

5. The sign language translation method based on class-annotation attention and knowledge transfer of claim 1, wherein, In step 2.5), the calculation formulas of the linearly interpolated key and the linearly interpolated value are as follows: wherein, is the adjusted i-th attention position, denotes the floor function, b i denotes the result of the floor function of the adjusted i-th attention position, u i denotes the result of the ceiling function of the adjusted i-th attention position; and denote the b i -th original key vector and the original value vector, respectively, and denote the u i -th original key vector and the original value vector, respectively, denotes the linearly interpolated key of the i-th key vector of the original key matrix at the t-th position, denotes the linearly interpolated value of the i-th key vector of the original value matrix at the t-th position.

6. The sign language translation method based on class-annotation attention and knowledge transfer of claim 1, wherein, In step 2.6), the calculation formula of the single-head class annotation attention vector is as follows: where z t is the t-th single-head class annotation attention vector, <·, ·> is the scaled dot-product between two vectors, and exp· denotes the exponential function with base of the natural logarithm e.

7. The sign language translation method based on class-annotation attention and knowledge transfer of claim 1, wherein, The encoder is composed of a plurality of encoder layers stacked together, and each encoder layer is constructed according to the following formula by layer normalization operation, multi-head class annotation attention mechanism, residual connection and feedforward neural network: z=MHGA(LN(x))+x where x denotes an input vector of an encoder layer, z denotes an intermediate result within the encoder layer, denotes an output vector of the encoder layer, LN(·) denotes a layer normalization operation, MHGA(·) denotes a multi-head class annotation attention mechanism, and MLP(·) denotes a feedforward neural network.

8. The sign language translation method based on class-annotation attention and knowledge transfer of claim 1, wherein, The knowledge transfer-based loss function item acquisition method is: 5.1) using the Sentence BERT semantic similarity pre-training model, obtaining pre-training embedding vectors of sentences in the oral text, and calculating a cosine similarity matrix S between all sentences; 5.2) Directly concatenate each vector in the sequence of embedded features of the coded sign language video to obtain an embedded vector e representing the whole sign language video i , iterate over all sign language videos in the training set; 5.3) Compute the embedding vector e for each video i The cosine similarity between the embedding vectors e and the cosine similarity between the spoken sentences s are averaged to get the knowledge transfer based loss function term: wherein, represents a loss function term based on knowledge transfer, S[i,j] represents an element of the cosine similarity matrix S located in the ith row and jth column, e i ·e j represents the dot product of the ith and jth sign language video embedding vectors, ||·|| represents the vector norm, D train represents the number of sentences in the spoken text.

9. A sign language translation system based on class annotation attention and knowledge transfer, for implementing the sign language translation method based on class annotation attention and knowledge transfer of claim 1, characterized in that the system It comprises: A sign language video preprocessing module for obtaining a sign language video embedding feature vector sequence; An oral text preprocessing module for obtaining an oral text embedding feature vector sequence; An encoder module comprising a multi-head class annotation attention mechanism for encoding the sign language video embedding feature vector sequence; A decoder module for autoregressively generating target oral text according to the oral text embedding feature vector sequence and the sign language video embedding feature vector sequence encoded by the encoder during the training stage, and for autoregressively generating target oral text according to the sign language video embedding feature vector sequence encoded by the encoder during the actual translation stage; A training module for obtaining a knowledge transfer-based loss function item according to the one-to-one semantic relationship between the sign language video and the oral text in the training set; taking the oral text embedding feature vector sequence and the sign language video embedding feature vector sequence encoded by the encoder as inputs of the decoder, obtaining a sign language translation cross-entropy loss function item for the process of autoregressively generating target oral text by the decoder; and training the encoder and the decoder by combining the knowledge transfer-based loss function item and the sign language translation cross-entropy loss function item.

Images