Sign language video generation method based on improved Transformer model

Abstract

The present invention provides a method and device for generating sign language videos based on an improved Transformer model. The method of the present invention first extracts the skeletal posture sequence in the sign language video and removes redundant information to reduce the amount of calculation. In addition, considering the importance of spatiotemporal information to the accuracy of generating sign language videos, a semantically rich embedding module is designed to encode position and speed information into the same high-dimensional space as the input of the model, thereby improving the coordination of joint movements and improving the accuracy of feature representation. Finally, an encoder-decoder model with a pyramid structure is constructed. The encoder accepts a spoken sentence as input and encodes the information in the sequence into an intermediate representation. The decoder then decodes the intermediate representation into a target sign language posture sequence in a semi-autoregressive manner. The present invention can effectively improve the utilization rate of semantic information and the overall expression ability of movements, thereby significantly improving the accuracy and speed of sign language video generation.

Description

Technical Field

[0001] The present invention relates to a human-computer interaction method, in particular to a sign language video generation method based on an improved Transformer model. Background Art

[0002] Sign language, as a visual gesture language, conveys information through gestures and spatial body movements. It is the most natural way for hearing-impaired people to interact with computers and the outside world. Real-time sign language communication has become a key topic in the fields of computer vision and natural language processing. Accurate automatic sign language generation can significantly improve the quality of communication between deaf and hearing people, helping people with disabilities better integrate into today's society. The goal of sign language video generation is to translate spoken sentences into personalized sign language videos that are understandable to humans.

[0003] Currently, sign language video generation is primarily categorized into two approaches: animation synthesis-based and deep learning-based. For traditional sign language animation video synthesis, most methods build a sign language word motion database based on motion tracking principles and human motion editing methods. Sign language motion segments corresponding to the input text are searched within the database and spliced ​​together for synthesis, using a virtual reality modeling language approach to vividly display the video. Overall, generating sign language videos based on animation synthesis offers the advantages of ease of operation and high efficiency, but it relies on the construction of a large-scale sign language animation database. Furthermore, the animation videos lack realistic details of movement execution, and the comprehensibility of the synthesized animations is still affected by the artificially designed appearance and movements. Consequently, a growing number of studies are exploring more flexible and natural sign language generation solutions.

[0004] In recent years, as traditional methods have been unable to cope with the increasing scale of data, deep learning has continued to achieve remarkable results in the field of sign language video generation. In 2020, the first deep learning-based sign language synthesis method was proposed. This method uses a recurrent neural network-based encoder-decoder structure to encode sign language video features into latent representations. Combined with a generative adversarial network, the generator is conditioned on both the human pose skeleton and appearance. The generated realistic video frames are then evaluated by a discriminator. However, this recurrent neural network-based model suffers from insufficient feature extraction due to the limitations of its local receptive field. Saunders et al. proposed a sign language generation model based on a progressive Transformer. This model, for the first time, converts discrete spoken sentences into continuous sign language gesture sequences in an end-to-end manner. A symbolic Transformer and a progressive Transformer are constructed to perform the translation from spoken sentences to sign language word sequences and the conversion from word sequences to human skeleton videos, respectively. Because the decoder's generation of the current frame depends on the previously generated pose sequence, this leads to error accumulation and high inference latency, resulting in slow network convergence and low sign language generation accuracy.

[0005] Many current deep learning-based models for sign language video generation tasks are based on autoregressive decoding. These models need to consider every previously generated frame when generating each target sign gesture (i.e., decoding). Consequently, the translation generation process is serial, making error accumulation and high inference latency a significant bottleneck for these models. Non-autoregressive models, on the other hand, are machine translation models that use parallel decoding. Unlike autoregressive models, which output word-by-word from left to right, non-autoregressive models can output at all times in parallel, resulting in more efficient decoding speeds but at the expense of accuracy. While some studies have used non-autoregressive decoding to generate sign language videos, these studies have not considered the issue of skeletal sequence duplication or loss due to discarding the dependencies between target sign gesture sequences. Summary of the Invention

[0006] The purpose of the present invention is to provide a sign language video generation method and device based on an improved Transformer model to address the shortcomings of existing methods in exploring posture information.

[0007] The present invention is implemented as follows: a method for generating sign language videos based on an improved Transformer model, comprising the following steps:

[0008] a. Use OpenPose to extract the 2D skeleton sequence of the target sign language posture from the target sign language video. Eight joints of the upper body and 21 joints of each hand are captured for model training. The 2D data representing the sign language posture is upgraded to 3D data. By observing the distribution of the 3D data and cleaning the skeleton information at abnormal and incorrect joints, the target sign language posture sequence is formed.

[0009] b. Input the spoken sentence and the target sign language gesture sequence into the encoder-decoder model, and train the encoder-decoder model to establish a mapping relationship between the spoken sentence and the target sign language gesture sequence; once the mapping relationship is established, a trained sign language video generation network model is formed.

[0010] c. Use the trained sign language video to generate a network model to process the input spoken sentences. The output is the probability distribution of the sign language corresponding to each moment, ultimately achieving end-to-end translation of spoken sentences into personalized sign language videos expressed in human skeleton and graphic formats.

[0011] In step a, data cleaning includes data processing methods such as data discarding and weighted linear interpolation.

[0012] The encoder-decoder model includes a text feature encoder with sign language length prediction and a pyramid semi-autoregressive decoder combined with a semantically rich embedding layer.

[0013] In step b, the encoder-decoder model is trained by inputting the spoken sentence into a text feature encoder to learn semantic features and passing them to a pyramid semi-autoregressive decoder, adding a convolutional neural network and a softmax classifier to the last layer of the encoder to predict the length of the sign language; inputting the target sign language gesture sequence into the pyramid semi-autoregressive decoder to extract spatiotemporal features, and decoding the target sign language sequence in a semi-autoregressive manner by introducing a relaxed masked-attention mechanism; and establishing a mapping relationship between the spoken sentence and the sign language gesture after model training.

[0014] In step b, the extraction of spatiotemporal features is to encode information on the time dimension and spatial displacement into the same space as the input of the model to address the lack of semantic information exploration; the pyramid semi-autoregressive decoder groups the target sign language gesture sequence in a coarse-to-fine manner, maintaining the cascade characteristics between the groups, and generating target frames in parallel within each group.

[0015] A spoken sentence containing N words is represented as: S = (s1,…,s N );

[0016] The target sign language gesture sequence is expressed as: T = (t1,…,t M );

[0017] Among them, s i is the i-th word in the spoken sentence, N is the number of words in the spoken sentence, t i is the sign language posture of the i-th frame, and M is the number of video frames.

[0018] The goal is to fit a parametric model that maximizes the conditional probability P(T|S) for translating text to sign language gesture sequences.

[0019] The joint velocity information between adjacent frames is integrated into the joint set of the skeleton sequence and expressed as:

[0020]

[0021] in, Represents the three-dimensional coordinate information of joint u at the tth frame It represents the velocity information of joint u at the tth frame, which is obtained by subtracting the three-dimensional coordinate information of the tth frame and the t-1th frame.

[0022] The length of the target sign language gesture sequence is L, P(L|S) is modeled separately, and the maximum value of L is set to 100.

[0023] The text feature encoder includes multiple layers with the same structure but different training parameters. Each layer consists of two sublayers, namely a multi-head attention mechanism and a position-by-position fully connected feedforward network. Each sublayer uses residual connection and layer normalization to ensure that the gradient will not be 0, alleviating the occurrence of gradient vanishing. The output of each sublayer is expressed as:

[0024]

[0025] in, It is the feature vector obtained by encoding the source spoken sentence S through the word embedding layer;

[0026] The word embedding layer uses a two-layer fully connected network (FC) and ReLU activation function ReLU. w1 is the weight matrix of the first layer of the fully connected network, b1 is the bias term of the first layer of the fully connected network, w2 is the weight matrix of the second layer of the fully connected network, b2 is the bias term of the second layer of the fully connected network. After multiplying the weight matrix with the input vector and then adding it to the bias, the positional encoding module is introduced to maintain the word order information:

[0027]

[0028] Among them, S n Represents the nth word in the sentence.

[0029] The multi-head attention mechanism projects query (Q), key (K) and value (V) through h different linear transformations, and finally concatenates the different attention results; where Q, K, and V are Multiplying with three trainable parameter matrices; the expression formula of the multi-head attention mechanism is:

[0030] MultiHead(Q,K,V)=Concat(head1,…,head i )W O (4)

[0031] Among them, Q, K, and V are Multiplying with three trainable parameter matrices, head i represents the i-th head; W O It is a weight matrix obtained by a fully connected layer and participates in model training along with other parameters.

[0032] The calculation formula for the i-th head is as follows:

[0033] head i =Attention(QW iQ ,KW i K ,VW i V ) (5)

[0034] Among them, W i Q 、W i K and W i V The corresponding parameters are Q, K, and V, which are three trainable parameter matrices. Attention is the scaled dot product attention. The higher the similarity between two vectors, the larger the dot product result, and the more attention the model pays. The calculation formula is as follows:

[0035]

[0036] Among them, QK E Divide by Keep the variance controlled to 1; the Softmax function is a normalized exponential function, so that the range of each element is between (0,1) and the sum of all elements is 1.

[0037] The semantically rich embedding layer uses two layers of fully connected networks (FC) and ReLU activation functions to map position and velocity information into the same vector space:

[0038]

[0039] in, Represents the position information of joint u after encoding at the tth frame; Represents the velocity information of joint u at the tth frame; w1 is the weight matrix of the first layer fully connected network, b1 is the bias term of the first layer fully connected network, w2 is the weight matrix of the second layer fully connected network, b2 is the bias term of the second layer fully connected network. After multiplying the weight matrix with the input vector and then adding it to the bias, the positional encoding module is introduced to maintain the posture order information.

[0040] The sign language gesture sequence T is obtained through the semantically rich embedding layer Expressed as:

[0041]

[0042] in, is the set of encoded position information and speed information;

[0043] The semi-autoregressive decoder with a pyramid structure maintains the global autoregressive characteristics and local non-autoregressive characteristics through the relaxed masked-attention mechanism. Divided into groups, where each group contains d frames, the conditional probability can be expressed as:

[0044]

[0045] in, It is the feature vector obtained by encoding the source spoken sentence S through the word embedding layer; is the feature vector obtained by encoding the source spoken sentence S through the word embedding layer Generate a set of encoded position information and speed information The conditional probability of G represents the set of encoded position information and speed information The sequence groups are divided into the following categories:

[0046]

[0047] Where M is the total number of video frames, and d is the number of frames in each group.

[0048] Predicting Group using the Relaxedmasked-attention mechanism model k When you are in the pose frame, you can see the Group 1tok All frame information of; given the target number of frames M and the division level d, is defined as follows:

[0049]

[0050] Among them, relaxed_mask is a two-dimensional matrix of length M, and d is the number of frames contained in each group;

[0051] Then the relaxed masked-attention with residual connection is defined as follows:

[0052]

[0053] Among them, QK E Divide by To further maintain variance control, the output of the sign language video generation network model first passes through a Linear layer and then a Softmax layer; logits are obtained after the Linear layer, and this logits value is converted into a probability value after the Softmax layer. The largest Softmax value is the sign language posture corresponding to the current moment.

[0054] In order to better mine semantic information from sign language gesture videos and reduce computational complexity and resource consumption, the present invention is based on a semantically rich pyramid semi-autoregressive Transformer model. Compared with the traditional TransforEMR model, the present invention adopts a coarse-to-fine pyramid semi-autoregressive approach to alleviate the gradient explosion problem caused by error accumulation when training the decoder. In this way, the autoregressive characteristics are maintained globally, while a sign language video generation model for sign language gesture sequences is generated in parallel locally.

[0055] This method encodes position and velocity information into the same high-dimensional space as input to the model decoder, improving the coordination of joint movements and making the generated hand movements more natural. Experiments have shown that the velocity information contained in sign language gesture sequences is a key feature that has been overlooked in previous research.

[0056] To address the diverse rotation angles caused by the complexity of hand joints, which can lead to overlapping and confusion of finger joints, the present invention adds a semantically rich embedding layer to encode position and velocity information into the same high-dimensional space, achieving a good balance between decoding speed and accuracy. The improved Transformer decoder combines the information of sign language movements in the time dimension and spatial displacement, filtering out redundant information in the video sequence, which helps to improve the accuracy of sign language synthesis.

[0057] The present invention uses a coarse-to-fine pyramid semi-autoregressive decoder to generate the target sign language gesture sequence. Unlike the autoregressive Transformer model, which outputs sequentially from left to right, the pyramid semi-autoregressive Transformer model decodes in groups and in parallel, resulting in more efficient decoding speed. This method achieves a better balance between autoregressive and non-autoregressive approaches, enabling local parallel generation of sign language gestures while maintaining the autoregressive properties globally. This method improves inference speed by 3.6 times without compromising accuracy.

[0058] The overall performance of this invention is better than that of mainstream algorithms and is more suitable for human-computer interaction. The beneficial effect of the improved Transformer model is that it can effectively improve the accuracy of generating sign language videos compared with the original network method, while also achieving efficient generation speed, facilitating the real-time deployment of the device. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a flowchart of sign language video generation.

[0060] Figure 2 It is the overall structural framework diagram of the improved Transformer model.

[0061] Figure 3 This is a dynamic information mining framework diagram in the present invention.

[0062] Figure 4 This is the unconstrained mask attention framework diagram of the present invention.

[0063] Figure 5 Graphs showing the relationship between inference delay and predicted sign language length for different models. (a) shows a comparison of autoregressive, non-autoregressive, and semi-autoregressive model configurations, and (b) shows a comparison of semi-autoregressive model grouping sizes of 1, 2, 4, and 8, respectively.

[0064] Figure 6 These are the sign language video results generated by different models on the Chinese Sign Language dataset CSL.

[0065] Figure 7 These are the sign language video results generated by different models on the German sign language dataset RWTH-PHOENIX-Weather 2014T. DETAILED DESCRIPTION

[0066] like Figure 1 As shown in Figure 1, the process of sign language video generation is achieved by the sign language synthesis model and the sign language skeleton extraction module jointly realizing the prediction of sign language actions from text.

[0067] like Figure 2 As shown in the figure, the overall structure of the improved Transformer model consists of two parts: a text feature encoder with sign language length prediction and a pyramid semi-autoregressive decoder combined with a rich semantic embedding layer.

[0068] The sign language video generation method of the present invention includes the following three steps:

[0069] Step 1: First, data preprocessing is performed. Openpose is used to extract the 2D skeleton sequence of the target sign language gesture from the target sign language video. To remove redundant information and reduce computational complexity, 8 joints in the upper body and 21 joints in each of the left and right hands, for a total of 50 joints, are captured for model training. Then, 2D to 3D Inverse Kinematics is applied to elevate the 2D information of the sign language gesture sequence to 3D. By observing the distribution and imbalance of the 3D data, the skeleton information of abnormal and incorrect joints is cleaned and discarded or processed using weighted linear interpolation.

[0070] Step 2: It includes two processes: encoding of spoken sentences and decoding of target sign language gesture sequences.

[0071] The encoding process is to input the spoken sentence into the text feature encoder of the improved Transformer network, learn the semantic features and pass them to the decoder, and add an additional convolutional neural network and softmax classifier in the last layer to predict the length of sign language.

[0072] The decoding process is to input the sign language gesture sequence extracted in step 1 into the pyramid semi-autoregressive decoder of the improved Transformer network, use the semantically rich embedding layer to extract spatiotemporal features from the sign language gesture sequence, and encode the information on the time dimension and spatial displacement into the same space as the input of the model to solve the lack of semantic information exploration; the pyramid-structured semi-autoregressive decoder groups the target sign language gesture sequence in a coarse-to-fine manner, maintaining the cascade characteristics between each group, but generating target frames in parallel within each group.

[0073] Step 3: The model output is the probability distribution of the sign language corresponding to each moment, ultimately achieving end-to-end translation of spoken sentences into personalized sign language videos expressed in human skeleton and graphic formats.

[0074] The text feature encoder in step 2 above mainly consists of four parts: word embedding layer, multi-head self-attention layer, feed-forward layer, and sign language length prediction layer.

[0075] Combine Figure 2 In step 2, the text feature encoder learns semantic features from the input spoken sentence and passes them to the decoder. In the word embedding layer, two layers of fully connected networks (FC) and ReLU activation functions are used to encode the source input S into a feature vector w1 and b1 are the weight matrix and bias term of the first layer of the fully connected network, w2 and b2 are the weight matrix and bias term of the second layer of the fully connected network. The weight matrix is ​​multiplied by the input vector and then added with the bias. Finally, the positional encoding module is introduced to maintain the word order information. The formula is as follows:

[0076]

[0077] Among them, S n Represents the nth word in the sentence.

[0078] The encoder consists of multiple layers with the same structure but different training parameters. Each layer consists of two sublayers: a multi-head attention mechanism and a position-wise feed-forward network. Each sublayer uses residual connections and layer normalization to ensure that the gradient does not reach zero, alleviating the problem of vanishing gradients. Therefore, the output of the sublayer can be expressed as:

[0079]

[0080] in, It is the feature vector obtained by encoding the source spoken sentence S through the word embedding layer;

[0081] Assuming the number of multi-head attention heads is h, multiple heads are equivalent to inputting the original information into different spaces, ensuring that the transformer can capture the features of different subspaces. Multi-head attention projects the query Q, key K, and value V through h different linear transformations, and finally concatenates these h matrices into a final matrix that is passed to the feedforward neural network layer.

[0082] MultiHead(Q,K,V)=Concat(head1,…,head i )W O (4)

[0083] Among them, Q, K, and V are Multiplying with three trainable parameter matrices, head i represents the i-th head; W O It is a weight matrix obtained by a fully connected layer and participates in model training along with other parameters.

[0084] The calculation formula for the i-th head is as follows:

[0085] head i =Attention(QW i Q ,KW i K ,VW i V ) (5)

[0086] Among them, W i Q 、W i K and W i V The corresponding parameters are Q, K, and V, which are three trainable parameter matrices. Attention is the scaled dot product attention. The higher the similarity between two vectors, the larger the dot product result, and the more attention the model pays. The calculation formula is as follows:

[0087]

[0088] Among them, QK E Divide by Keep the variance controlled to 1; the Softmax function is a normalized exponential function, so that the range of each element is between (0,1) and the sum of all elements is 1.

[0089] Assuming x is the output after the multi-head attention layer, Position-wise feed-forward networks provide nonlinear transformations. The reason why it is position-wise is that the transformation parameters for each position i are the same when passing through the linear layer.

[0090]

[0091] The output of the last layer of the encoder is used to predict the sign language length using a single-layer neural network and a softmax classifier. Assuming that the length of the target sign language gesture sequence is L, the translation is split into L / d segments at different decoder layers. The segments are generated from left to right in an autoregressive manner, while within each segment, d-frame target gesture sequences are generated in parallel in a non-autoregressive manner. Therefore, the length of the target sign language sequence is an important latent variable that affects the final translation quality. P(L|S) is modeled separately, and the maximum value of L is set to 100 in the experiment. It is worth noting that the present invention only uses the predicted length to assist in reasoning when testing the sign language synthesis model. During training, the model uses the length of the reference target sequence.

[0092] The pyramid semi-autoregressive decoder in step 2 above mainly consists of four parts: semantically rich embedding layer, relaxed masked-attention, relaxed encoder-decoder attention, and feed-forward layer.

[0093] Combine Figure 3 The present invention mines dynamic information. The dark arrow represents the thumb joint movement information, and the light arrow represents the elbow joint movement information. In step 3, its semantic embedding layer learns features from the sign language gesture sequence extracted in step 1. and speed To represent the dynamic information of joint u at the tth frame, the joint velocity information and position information between adjacent frames are integrated into the joint set of the skeleton sequence, which is expressed as:

[0094]

[0095] In the semantically rich embedding layer, two layers of fully connected networks (FC) and ReLU activation functions are used to map the position and velocity information into the same vector space. The formula is as follows:

[0096]

[0097]

[0098] in, Represents the position information of joint u after encoding at the tth frame; Represents the velocity information of joint u at the tth frame; w1 is the weight matrix of the first layer fully connected network, b1 is the bias term of the first layer fully connected network, w2 is the weight matrix of the second layer fully connected network, b2 is the bias term of the second layer fully connected network. After multiplying the weight matrix with the input vector and then adding it to the bias, the positional encoding module is introduced to maintain the posture order information.

[0099] The sign language gesture sequence T is obtained through the semantically rich embedding layer Expressed as:

[0100]

[0101] in, is the set of encoded position information and speed information;

[0102] The pyramid semi-autoregressive decoder in step 2 above has a pyramid structure with progressively finer division levels d. The model generates the target sign language sequence using a globally serial and locally parallel autoregressive approach. The encoder outputs the sign language gesture and spoken sentence in frame td as input, and outputs the predicted sign language gesture probability. This structure is similar to the encoder, consisting of multiple layers. In addition to the two sublayers in each encoder layer, the decoder also incorporates a third sublayer, Relaxed Encoder-Decoder Attention. In Relaxed Encoder-Decoder Attention, the output (K, V) of each encoder interacts with the input Q of all decoders, allowing the decoder to focus on the appropriate locations in the encoder output. Therefore, unlike self-attention, the intermediate attention layer's K and V come from the encoder, and Q comes from the output of the previous decoder.

[0103] By setting unconstrained masked attention, the present invention implements a pyramid semi-autoregressive mechanism, the purpose of which is to avoid the time delay of the autoregressive model. Moreover, it can solve the problem that the generation quality of the non-autoregressive model is weaker than that of the autoregressive model due to the abandonment of the dependencies between the target sequences. In order to balance the inference speed and generation quality, a pyramid semi-autoregressive model is established in a coarse-to-fine manner, and four different grouping sizes are set in each layer of the decoder: d=8, d=4, d=2, and d=1. It should be noted here that when d=1, it is actually equivalent to an autoregressive model. The long-range semantic information that the bottom layer pays special attention to is fused with the short-range global semantic information captured by the top layer, so that the decoder can maintain long-term dependencies while having a more efficient decoding speed.

[0104] The semi-autoregressive decoder with a pyramid structure maintains the global autoregressive characteristics and local non-autoregressive characteristics through the relaxed masked-attention mechanism. Divided into groups, where each group contains d frames, the conditional probability can be expressed as:

[0105]

[0106] in, It is the feature vector obtained by encoding the source spoken sentence S through the word embedding layer; is the feature vector obtained by encoding the source spoken sentence S through the word embedding layer Generate a set of encoded position information and speed information The conditional probability of G represents the set of encoded position information and speed information The sequence groups are divided into the following categories:

[0107]

[0108] Where M is the total number of video frames, and d is the number of frames in each group.

[0109] Combine Figure 4 , the mask mechanism in the unconstrained mask attention is to prevent the leakage of contextual information and avoid the model from peeking at the sequence to be predicted in advance. Unlike the standard self-attention, the present invention creates a self-attention mechanism with a relaxedmask mechanism. Since the pyramid-type semi-autoregressive decoder is generated in parallel within the group and maintains the autoregressive property globally, it captures the posture frames in the previous group and the current group at the same time. The token of the future frame to be masked is set to the invalid feature -∞, so the output after adding -∞ to the position to be masked and performing softmax is 0. Relaxedmasked-attention enables the model to predict Group k When you are in the pose frame, you can see the Group 1tok Given the target number of frames M and the division level d, is defined as follows:

[0110]

[0111] Among them, relaxed_mask is a two-dimensional matrix of length M, and d is the number of frames contained in each group; therefore, the relaxedmasked-attention with residual connection is defined as follows:

[0112]

[0113] Among them, QK E Divide by To further maintain variance control, the output of the sign language video generation network model first passes through a Linear layer and then a Softmax layer; logits are obtained after the Linear layer, and this logits value is converted into a probability value after the Softmax layer. The largest Softmax value is the sign language posture corresponding to the current moment.

[0114] Figure 4 In the matrix, the darker areas represent the -∞ portion, representing masked information, while the lighter areas represent unmasked information. The ordinate represents the position of the target word, and the abscissa represents the visible position. Applying this matrix to each sequence achieves a semi-autoregressive effect.

[0115] To obtain the optimal pyramid semi-autoregressive model, the present invention sets different grouping levels and compares them with a two-layer (N=2) lightweight autoregressive Transformer. d={2,2} represents two decoder layers, both using the relaxed masked attention mechanism with d=2. Figure 5 (b) depicts the relationship between the coarse-to-fine grouping level and inference latency, showing an inverse relationship between grouping level and inference time. As d increases, the inference latency decreases. Table 1 summarizes the speed and accuracy of the Pyramid Semi-Autoregressive model in generating target sign language under different configurations. The results show that as d increases, the BLEU score of the predicted sign language sequence gradually decreases. When d = {2, 2}, the Pyramid Semi-Autoregressive model decodes 1.90 times faster than the Transformer, with a BLEU score reduction of only 0.09. When d = {8, 8}, the Pyramid Semi-Autoregressive model achieves a 7.32x speedup, while the BLEU score decreases by 4.84.

[0116] Table 1 Ablation experiment of pyramid semi-autoregressive model on RWTH-PHOENIX-Weather 2014T dataset

[0117]

[0118] Table 1 presents ablation results of the proposed pyramid semi-autoregressive model on the RWTH-PHOENIX-Weather 2014T dataset for sign language synthesis. It can be seen that a pyramid-like structure with coarse-to-fine grouping levels further improves performance compared to a single configuration. d = {1, 1, 1, 1} indicates that all layers have a grouping level of 1, effectively equivalent to an autoregressive transformer. d = {8, 4, 2, 1} means that the bottom layer uses relaxed masked attention with d = 8, the second layer uses d = 4, the third layer uses d = 2, and the top layer uses restricted self-attention with autoregressive output. While d = {2, 2, 2, 2} and d = {8, 6, 4, 1} have comparable BLEU scores, the latter achieves a 3.60x speedup, far exceeding the 1.84x speedup of the former. We also experimented with d = {8, 6, 4, 2}, achieving a 4.75× speedup. However, the BLEU score was significantly compromised, resulting in a 2.48 BLEU score difference compared to d = {8, 4, 2, 1}. Therefore, we set d = {8, 4, 2, 1} as the optimal pyramid grouping level.

[0119] In step 3 above, the model is trained to learn the mapping between spoken sentences and sign language movements. During training, the present invention uses the following hyperparameter configuration: Embedding dimension = 512, Hidden dimension = 512, Feed-forward dimension = 2048, Number of Heads = 8, Number of Layers = 4, Batch size = 16, and dropout = 0.1. The present invention uses Adam as the optimizer, with a total number of iterations of 40k. In particular, the cosine annealing algorithm is used to warm up the learning rate lr:

[0120]

[0121] The present invention sets the maximum initial learning rate lr max Set to 1e-3, the minimum learning rate lr min Set to 1e-4. step warmup Indicates the total number of warm-up steps. Here, a progressive warm-up strategy is used for the first 100 steps. total It is equal to the total number of epochs multiplied by the total number of training samples, and finally divided by the set batchsize.

[0122] To accelerate the convergence of the pyramid semi-autoregressive model's gradient descent and obtain a model with low generalization error, this paper transfers knowledge learned from a pretrained autoregressive model to the pyramid semi-autoregressive model for model initialization. This includes all parameters in the text feature encoder, the semantically rich embedding layer, and some parameters in the pyramid semi-autoregressive decoder. Other parameters are randomly initialized. While accelerating convergence, this approach also avoids vanishing or exploding gradients caused by uninitialized or improperly initialized gradients, slightly improving the accuracy of sign language synthesis.

[0123] After the model training is completed in step 3 above, the model output is the probability distribution of the sign language corresponding to each moment, and finally the end-to-end translation of spoken sentences into personalized sign language videos expressed in human skeleton and graphic formats is achieved.

[0124] The present invention verifies the effectiveness of the model in sign language synthesis using two public sign language datasets. RWTH-PHOENIX-Weather 2014T records the sign language video commentary of the daily news and weather forecast of the German public television station PHOENIX. RWTH-PHOENIX-Weather 2014T contains 8,257 video samples, with a total of 2,887 words combined into 5,356 continuous sentences related to weather forecasts. The present invention also trains and scores the model on the Chinese Sign Language dataset (CSL). There are 100 sentences in the dataset, a total of 5,000 continuous sign language videos, and each sentence contains an average of 4 to 8 words.

[0125] This paper uses a transformer-based sign language translation model as a back-translation evaluation method to assess the accuracy of sign language synthesis results. The paper changes the input of the sign language translation model from sign language video frames to sign language gesture sequences to train the back-translation evaluation model. The model score is presented using standard metrics, including BLEU-1 / 4 and ROUGH, which measure the quality of translation based on precision and recall, respectively. DTW processing: measures the similarity between the true value and the predicted sequence.

[0126] To further demonstrate the effectiveness of the proposed method and apparatus for sign language video generation based on the improved Transformer model, we compared it with other deep learning-based sign language video generation algorithms, including autoregressive and non-autoregressive Transformer models, on the RWTH-PHOENIX-Weather 2014T dataset and the Chinese Sign Language (CSL) dataset. Table 2 summarizes the results of sign language generation on the RWTH-PHOENIX-Weather 2014T dataset.

[0127] As shown in Table 2, combining the autoregressive Transformer with the semantically rich embedding layer (SR) of the present invention improves overall accuracy, with BLEU-1 and ROUGE scores increasing by 0.15 and 0.38, respectively. This demonstrates the importance of combining positional and motion features. A non-autoregressive Transformer was also used for comparative experiments to explore the impact of different decoder regression methods on accuracy. Table 2 shows that the non-autoregressive Transformer significantly compromises the accuracy of the generated sign language videos. Furthermore, when comparing the autoregressive Transformer with the pyramid semi-autoregressive Transformer, while the accuracy score on the validation set decreases slightly, the BLEU-1 and ROUGE scores on the test set increase by 0.68 and 0.79, respectively. This demonstrates that the pyramid semi-autoregressive model of the present invention effectively mitigates error accumulation during decoding. Finally, the results demonstrate that the pyramid semi-autoregressive Transformer based on semantically rich embeddings achieves optimal accuracy. Compared to the autoregressive Transformer, the present invention achieves a 0.91 BLEU score and a 1.42 ROUGE score improvement on the test set.

[0128] Table 2 Comparison of accuracy on the RWTH-PHOENIX-Weather 2014T dataset

[0129]

[0130] As shown in Table 3, two factors affect the inference efficiency of the model. One is the time complexity of the decoder regression method, and the other is the length of the target sequence. In the transformer model, only one time step is moved each time during inference. The present invention uses O(ts) to represent the time complexity of this step. For the probability distribution obtained in the inference stage, the present invention uses greedy search to take the possibility with the maximum probability from the probability distribution of each sign language frame. Its time complexity is represented by O(gs). The time complexity of each model is shown in Table 2, where L represents the length of the target sign language sequence, N represents the number of layers of the decoder, and ∑d * Indicates the total size of the pyramid-shaped groups. Figure 5 (a) The inference latency of the autoregressive model increases linearly with the prediction length. The parallel nature of the non-autoregressive Transformer makes inference latency independent of the target length, resulting in an 18.4x speedup. The pyramid semi-autoregressive Transformer of our invention improves inference speed by 3.6x without compromising accuracy.

[0131] Table 3. Inference latency, model speedup, and time complexity on the RWTH-PHOENIX-Weather 2014T dataset

[0132]

[0133] Figure 5 The relationship between inference delay and predicted sign language length under different model configurations. 20 sign language sequences with lengths in the range of [31,163] are randomly selected for prediction. Figure 5 (a) Experimental results of autoregressive Transformer, non-autoregressive Transformer and pyramid semi-autoregressive Transformer; right Figure 5 (b) shows the experimental results of the pyramid semi-autoregressive Transformer under single configurations d = 1, 2, 4, and 8. To simplify the computation, a two-layer decoder is selected for the experiment.

[0134] In order to more intuitively demonstrate the performance of the method of the present invention, the present invention visualizes the sign language gesture sequences generated by different models on the RWTH-PHOENIX-Weather-2014T and Chinese Sign Language datasets CSL. Figure 6 As shown, 10 frames of the sign language sequence of “Our country is prosperous and democratic” are sampled from left to right for comparison, where each column represents the gesture frames generated by different models at a certain moment. The first row of gesture sequences is the result generated by the autoregressive Transformer. It can be seen that the frames t8 and t9 deviate greatly from the true label (GT) or even get distorted. This is due to the accumulation of errors caused by excessive dependence between frames. The second row is the result generated by the non-autoregressive Transformer, in which the t5 frame is lost, and duplicate frames are generated at t7, t8, and t9. This is because it excessively discards the correlation between the sign language gesture frames and outputs them in parallel at each moment. The third row is a more reasonable and natural sign language gesture sequence generated by the pyramid semi-autoregressive Transformer of the present invention.

[0135] Figure 7 Visualization results on the RWTH-PHOENIX-Weather-2014T dataset are shown. At frames t8 and t9, the autoregressive Transformer exhibits unrealistic results, while the non-autoregressive Transformer suffers from frame loss. In contrast, the improved Transformer model of our present invention generates more stable sign language gesture sequences. This demonstrates that our present invention leverages the properties of semi-autoregressive decoding, thereby avoiding the shortcomings of both autoregressive and non-autoregressive Transformers.

Claims

1. A sign language video generation method based on an improved Transformer model, characterized by: The following steps are involved: A. Use OpenPose to extract the 2D skeleton sequence of the target sign language posture from the target sign language video. Eight joints of the upper body and 21 joints of each of the left and right hands are captured for model training. The 2D data representing the sign language posture is converted to 3D data. By observing the distribution of the 3D data and cleaning the skeleton information of abnormal and incorrect joints, the target sign language posture sequence is formed. b. Input the spoken sentence and the target sign language gesture sequence into an encoder-decoder model and train the encoder-decoder model to establish a mapping relationship between the spoken sentence and the target sign language gesture sequence; Once the mapping relationship is established, a trained sign language video generation network model is formed; c. Use the trained sign language video generation network model to process the input spoken sentences. The output is the probability distribution of the sign language corresponding to each moment, ultimately achieving end-to-end translation of spoken sentences into personalized sign language videos expressed in human skeleton and graphic formats. The encoder-decoder model consists of a text feature encoder with sign language length prediction and a pyramid semi-autoregressive decoder combined with a semantically rich embedding layer; In step b, the encoder-decoder model is trained by feeding the spoken sentence into a text feature encoder to learn semantic features and then passing them to a pyramid semi-autoregressive decoder. A convolutional neural network and a softmax classifier are added to the last layer of the encoder to predict the length of the sign language. The target sign language gesture sequence is input into the pyramid semi-autoregressive decoder to extract spatiotemporal features. The target sign language sequence is decoded in a semi-autoregressive manner by introducing the relaxed masked-attention mechanism. After model training, a mapping relationship between spoken sentences and sign language gestures is established. In step b, spatiotemporal feature extraction encodes sign language information in the temporal dimension and spatial displacement into the same space as the model input. The pyramid semi-autoregressive decoder groups the target sign language gesture sequence, maintaining the cascade characteristics between groups, and generates target frames in parallel within each group. The semantically rich embedding layer uses a two-layer fully connected network FC and ReLU activation function to map the position and velocity information into the same vector space.

2. The method for generating a sign language video according to claim 1, wherein: In step b, Represent a spoken sentence containing N words as: ; The target sign language gesture sequence is represented as: ; Among them, s i is the i-th word in the spoken sentence, N is the number of words in the spoken sentence, t i is the sign language posture of the i-th frame, M is the total number of video frames; The goal is to fit the maximum conditional probability Parametric models for text-to-sign language gesture sequence translation; The joint velocity information between adjacent frames is integrated into the joint set of the skeleton sequence and expressed as: （1） in, = ( , , ), = - ; Represents the three-dimensional coordinate information of joint u at the tth frame , , ; represents the velocity information of joint u at the tth frame, which is obtained by subtracting the three-dimensional coordinate information of the tth frame and the t-1th frame; The length of the target sign language gesture sequence is L. Perform separate modeling and set the maximum value of L to 100.

3. The method for generating a sign language video according to claim 2, wherein: The text feature encoder includes several layers with the same structure but different training parameters. Each layer consists of two sublayers: a multi-head attention mechanism and a position-by-position fully connected feedforward network. Each sublayer uses a residual connection and layer normalization to ensure that the gradient does not reach 0, thereby alleviating the occurrence of gradient vanishing. The output of each sublayer is expressed as: （2） in, It is composed of source spoken sentences The feature vector obtained by encoding the word embedding layer; The word embedding layer uses a two-layer fully connected network FC and ReLU activation function ReLU. w1 is the weight matrix of the first layer of the fully connected network, b1 is the bias term of the first layer of the fully connected network, w2 is the weight matrix of the second layer of the fully connected network, b2 is the bias term of the second layer of the fully connected network. After multiplying the weight matrix with the input vector and then adding it to the bias, the positional encoding module is introduced to maintain the word order information as follows: （3） in, Represents the nth word in the sentence.

Images