Sign language translation method based on multi-stage sampling and early attention

Abstract

The present invention relates to a sign language translation method based on multi-order sampling and early attention: the method is characterized in that it comprises two parts, the first part: firstly, CNN is used to extract video features frame by frame, then different sampling rates are set, time series features are downsampled, and time series features of different fields of view are obtained, for the time series features under each sampling rate, non-local modules are used to associate features with larger spans, and finally the time series features are upsampled to the original length and multiple features are fused; the second part, an encoder-decoder network based on early attention is established: the basic structure of the encoder-decoder network is LSTM, the encoder encodes the feature sequence after multi-scale fusion, and the decoder decodes the encoded feature sequence into text information; and the early attention mechanism is used to weight the fusion of visual features.

Description

Technical Field

[0001] The invention relates to the field of sign language translation, and in particular to a sign language translation method based on multi-order sampling and early attention. Background Art

[0002] For the hearing-impaired, sign language is their most important means of communication and plays a very important role in the deaf-mute community. As a visual language, sign language mainly conveys semantic information through hand information, including hand shape and movement trajectory, assisted by facial expressions, and is one of the main ways for hearing-impaired people to communicate in daily life [1]. However, due to the high learning cost and difficulty of sign language itself, it is not widely used among healthy people. At the same time, due to regional differences, communication between hearing-impaired people in different regions is also difficult. Therefore, it is of great significance to propose a method that can automatically translate sign language videos into natural language sentences that can be understood by healthy people. Subsequently, related research on continuous sign language recognition and sign language translation came into being.

[0003] Continuous Sign Language Recognition[2][3][4] is a video understanding task that belongs to the research field of Computer Vision. The input of this task is a sign language video, that is, a continuous frame sequence containing sign language actions, and the output is a recognized sign language sentence, that is, a sequence of words corresponding to the sign language. The Sign Language Translation task[5][6][7][8] goes a step further based on the sign language recognition task and translates the word sequence output by the sign language recognition task into sentences that conform to the grammar of natural language, thus realizing the corresponding translation from sign language video to natural language.

[0004] In recent years, researchers have generally used deep learning-based methods to conduct research related to continuous sign language recognition and sign language translation. The general idea of ​​these studies is feature extraction and sequence alignment. First, after preprocessing the input sign language video, a two-dimensional or three-dimensional convolutional neural network (CNN) is used to extract the visual features of the video; after obtaining the visual features, the feature sequence is processed using a temporal modeling model. For this sequence-to-sequence problem, the commonly used one is the recurrent neural network (RNN) and its variants, such as the encoder-decoder architecture with long short-term memory (LSTM[9] and gated recurrent unit (GRU)

[10] as the main components. In addition, the connectionist temporal classification (CTC) method is used to solve the sequence-to-sequence alignment problem. CTC is usually applied to weakly supervised sequence alignment, for example, it has made significant progress in speech recognition. In recent years, it has been used to align sign language videos and text sequences.

[0005] The above-mentioned methods have achieved some good results in continuous sign language recognition and translation tasks. However, since CNN and RNN networks rely heavily on the feature calculation of convolutional layers, their receptive fields are relatively limited when extracting features, ignoring the local and overall temporal features of the video; in addition, the traditional encoder-decoder model lacks attention to the long-term attention content in the sequence, resulting in duplication or missing information in the translation results.

[0006] References

[0007] [1]Ong SCW,Ranganath S.Automatic sign language analysis:A surveyand the future beyond lexical meaning[J].IEEE Transactions on PatternAnalysis&Machine Intelligence,2005, 27(06):873-891.

[0008] [2]Min Y,Hao A,Chai X,et al.Visual Alignment Constraint forContinuous Sign Language Recognition[C].Proceedings of the IEEE / CVFInternational Conference on Computer Vision.2021:11542-11551.

[0009] [3]Niu Z,Mak B.Stochastic fine-grained labeling of multi-state signglosses for continuous sign language recognition[C].European Conference onComputer Vision.Springer,Cham,2020:172-186.

[0010] [4]Cheng K L,Yang Z,Chen Q,et al.Fully convolutional networks forcontinuous sign language recognition[C].European Conference on ComputerVision.Springer,Cham,2020:697-714.

[0011] [5]Zhou H,Zhou W,Qi W,et al.Improving Sign Language translation withmonolingual data by sign back-translation[C].Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2021:1316-1325.

[0012] [6]Orbay A,Akarun L.Neural sign language translation by learningtokenization[C].2020 15th IEEE International Conference on Automatic Face andGesture Recognition(FG 2020).IEEE,2020:222-228.

[0013] [7]Yin K,Read J.Better sign language translation with STMC-transformer[C].Proceedings of the 28th International Conference onComputational Linguistics.2020:5975-5989.

[0014] [8]Camgoz N C,Hadfield S,Koller O,et al.Neural sign languagetranslation[C].Proceedings of the IEEE conference on computer vision andpattern recognition.2018:7784-7793.

[0015] [9]Hochreiter S,Schmidhuber J.Long Short-Term Memory[J].NeuralComputation,1997, 9(8):1735-1780.

[0016]

[10] Cho K,Van B,Gulcehre C,et al.Learning phraserepresentations using RNN encoder-decoder for statistical machine translation[J].arXiv preprint arXiv:1406.1078,2014.

[0017]

[11] Iandola FN,Han S,Moskewicz MW,et al.SqueezeNet:AlexNet-levelaccuracy with 50x fewer parameters and<0.5MB model size[J].arXiv preprintarXiv:1602.07360,2016.

[0018]

[12] Wang

[0019]

[13] Li D, Xu C, Yu X, et al. Tspnet: Hierarchical feature learning via temporal semantic pyramid for sign language translation [J]. Advances in NeuralInformation Processing Systems, 2020, 33: 12034-12045. Summary of the invention

[0020] The purpose of the present invention is to provide a new sign language translation method to realize multi-scale temporal feature extraction and fusion and avoid word duplication translation, in view of the fact that the existing sign language translation methods lack the ability to explore multi-scale long-term temporal information and cannot solve the problem of repeated word translation. To this end, the present invention adopts the following technical solution: a sign language translation method based on multi-order sampling and early attention, including a new multi-scale temporal feature extraction based on multi-order sampling and an encoder-decoder network based on early attention. The technical solution is as follows:

[0021] A sign language translation method based on multi-order sampling and early attention: characterized by comprising two parts,

[0022] Part I: First, CNN is used to extract video features frame by frame. Then, different sampling rates are set to downsample the temporal features to obtain temporal features of different fields of view. For the temporal features at each sampling rate, the non-local module

[12] is used to associate features with larger spans. Finally, the temporal features are upsampled to the original length and multiple features are fused. The specific process is as follows:

[0023] s11 Frame-level feature extraction: Use the lightweight model SqueezeNet to extract the features of the input video and obtain a frame-by-frame feature sequence X with a length of T and a feature dimension of d;

[0024] s12 Multi-stage sampling of time series features: Use convolutional layers and pooling layers to extract K feature sequences X of different time scales i , the formula is as follows:

[0025] X i =p i (c i (X)),i≤K

[0026] where c i 、p i They represent the point-by-point convolution layer and pooling layer used for sampling in the time dimension respectively; the pooling layer kernel size is set to an integer multiple of 2. The larger the value, the shorter the feature sequence length and the larger the time span, which can better express the overall information of the video, but the more detailed visual information is lost; on the contrary, when the value is smaller, the sampling sequence obtained contains more detailed information;

[0027] s13 Temporal relationship calculation: Use the non-local module to calculate each feature sequence X i The weighted calculation is performed as follows: for each feature sequence X i Use two point-by-point convolutions to get the feature sequence X i The self-attention matrix T i is the feature sequence X i The length of A i After softmax normalization calculation and X i Multiply them together to get the normalized feature sequence X i ';

[0028] s14 Time series multi-scale fusion: Each normalized feature sequence X i 'After extending the inverse average pooling in the time dimension to the original length of the video, and then performing element-by-element addition operations, the multi-scale fused feature sequence I is obtained

[0029] The second part is to establish an encoder-decoder network based on early attention: the basic structure of the encoder-decoder network is LSTM. The encoder encodes the multi-scale fused feature sequence I, and the decoder decodes the encoded feature sequence into text information; the early attention mechanism is used to weight the fusion of visual features. The method is as follows:

[0030] s21 encoding process: The multi-scale fused feature sequence I is input into the LSTM encoder to obtain the encoder hidden state sequence h of length T 1 ,h 2 ,...,h T ;

[0031] s22 calculates attention m ij : Using the encoder hidden state h at encoding time i i and the decoder state s at decoding time j j Calculate attention m ij ;

[0032] Based on s22, s23 adds the attention before decoding time j to obtain the early attention e ij , and then the early attention e ij Add to current attention n ij Calculation of; After that, the current attention n of the early attention will be added ij After softmax normalization, the attention weight a is obtained ij ;

[0033] s24 semantic coding calculation: using attention weight a ij The encoder hidden state sequence h 1 ,h 2 ,...,h T Perform weighted summation to obtain the semantic coding sequence C 1 ,C 2 ,...,C U ;

[0034] s25 loss function L of the attention calculation part a : For encoding time i and decoding time j, take the corresponding attention weight a ij and the softmax normalized early attention softmax (e ij ) is the minimum value of the two, and the sum of the values ​​corresponding to all combinations of i and j is obtained to obtain L a , as shown below;

[0035]

[0036] a ij With softmax(e ij )The greater the difference, the a The smaller the value of;

[0037] s26 decoding process: LSTM decoder recursively predicts a translation sequence y=y of length U 1 y 2 ...yU , for time j, using the obtained semantic code C j , the decoding symbol y at the previous moment of time j j-1 and the decoder state s j Get the decoded symbol y at time j j ;

[0038] During s27 training, for the translation sequence y and the target sequence z, the cross entropy loss L is calculated e ;

[0039] s28 total loss function L: L e and L a Add them together to get the total loss function L. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Flowchart of the sign language translation method based on multi-stage sampling and early attention.

[0041] Figure 2 Schematic diagram of extracting multi-scale feature sequences.

[0042] Figure 3 It is an encoder-decoder network based on early attention.

[0043] Figure 4 A diagram showing the Chinese sign language video dataset constructed for the present invention.

[0044] Figure 5 The translation accuracy of the method of the present invention is compared with that of the existing sign language translation method. DETAILED DESCRIPTION

[0045] The present invention is described below in conjunction with the accompanying drawings and embodiments.

[0046] 1. Multi-scale temporal feature extraction based on multi-order sampling: First, use CNN to extract video features frame by frame, then set different sampling rates, downsample the temporal features, and obtain temporal features of different fields of view. For the temporal features at each sampling rate, use simplified non-local modules to associate features with larger spans, and finally upsample the temporal features to their original length and fuse multiple features. The specific process is divided into the following steps:

[0047] (1) Frame-level feature extraction: The lightweight model SqueezeNet

[11] is used to extract the features of the input video and obtain a frame-by-frame feature sequence X with a length of T and a feature dimension of d.

[0048] (2) Time series feature sampling: The present invention designs a multi-order feature sampling method, using convolutional layers and pooling layers to extract K feature sequences X at different time scales. i , which can be written as the formula:

[0049] X i =p i (c i (X)),i≤K (1)

[0050] where c i 、p i They represent the point-by-point convolution layer and pooling layer for sampling in the time dimension. The pooling layer kernel size is set to an integer multiple of 2. The larger the value, the shorter the feature sequence length and the larger the time span, which can better express the overall information of the video, but the more detailed visual information is lost; on the contrary, when the value is smaller, the sampling sequence obtained contains more detailed information. The sampling sequence is as follows: Figure 2 shown.

[0051] (3) Temporal relationship calculation: Use a simplified non-local module

[12] to calculate each feature sequence X i Perform weighted calculation as follows: Use two point-by-point convolutions f and g to get the feature sequence X i The self-attention matrix T i is the feature sequence X i The length of A i After softmax normalization calculation and X i Multiply them together to associate features that are far apart and obtain a normalized feature sequence X' i For each feature sequence X i The calculation can be written as formula:

[0052] A i =f(X i )·g(X i ) T (2)

[0053] X i =softmax(A i )X i (3)

[0054] Where T stands for transpose.

[0055] (4) Temporal multi-scale fusion: After calculating the temporal relationship of the K feature sequences under different sampling conditions, each feature sequence is expanded to the original length of the video through inverse average pooling and element-by-element addition is performed. The specific process can be written as the formula:

[0056]

[0057] Where r represents the inverse average pooling layer in the time dimension. The obtained multi-scale fused feature sequence I is input into the encoder-decoder network.

[0058] 2. Encoder-decoder network based on early attention: The encoder-decoder network designed by the present invention is as follows Figure 3 As shown, its basic structure is LSTM[9]. The encoder encodes the multi-scale fused feature sequence I, and the decoder decodes the encoded feature sequence into text information. In order to reflect the importance of visual features to each step of decoding, the attention mechanism is used to weight the fused visual features. In each step of decoding, both the text features of the current step and the weighted visual features should be considered, which can help to obtain more accurate translation results. However, the existing methods will cause the translation results to have repeated word translation. In order to solve this problem, the present invention designs a translation method based on early attention information.

[0059] (1) Encoding process: The multi-scale fused feature sequence I is input into the LSTM encoder to obtain the encoder hidden state sequence h of length T 1 ,h 2 ,...,h T .

[0060] (2) Calculate the attention m ij : Using the encoder hidden state h at encoding time i i and the decoder state s at decoding time j j Calculate attention m ij :

[0061] m ij =λtanh(W h h i +W s s j +b) (5)

[0062] Among them, W h , W s , b and λ are learnable parameters.

[0063] (3) Based on (2), add the attention before decoding time j to get the early attention e ij , added to the current attention n ij Calculation:

[0064]

[0065] n ij =λtanh(W h h i +W s s j +W e e ij +b) (7)

[0066] Likewise, W e is a learnable parameter. The current attention n that will be added to the early attention ij After softmax normalization, we get the attention weight a ij :

[0067] a ij =softmax(n ij ) (8)

[0068] (4) Semantic encoding calculation: using attention weight a ij The encoder hidden state sequence h 1 ,h 2 ,...,h T Perform weighted summation to obtain the semantic coding sequence C 1 ,C 2 ,...,C U , as shown below.

[0069]

[0070] (5) Loss function L of the attention calculation part a : For encoding time i and decoding time j, take the corresponding attention weight a ij and the softmax normalized early attention softmax (e ij ) is the minimum value of the two, and the sum of the values ​​corresponding to all combinations of i and j is obtained to obtain L a , as shown below.

[0071]

[0072] a ij With softmax(e ij )The greater the difference, the a The smaller the value of .

[0073] (6) Decoding process: The LSTM decoder recursively predicts a translation sequence y=y of length U 1 y 2 ...y U For time j, the semantic code C obtained by formula (9) is j , the decoding symbol y at the previous moment of time j j-1 (j=1 is the start symbol) and decoder state s j Get the decoded symbol y at time j j .

[0074] (7) During training, for the translation sequence y and the target sequence z, calculate the cross entropy loss L e .

[0075] (8) Total loss function L: L e and L a Add together to get the total loss function L:

[0076] L=L e +L a (11)

Claims

1. A sign language translation method based on multi-order sampling and early attention: characterized in that, It consists of two parts: Part 1: First, CNN is used to extract video features frame by frame. Then, different sampling rates are set to downsample the temporal features to obtain temporal features of different fields of view. For the temporal features at each sampling rate, the non-local module is used to associate features with larger spans. Finally, the temporal features are upsampled to the original length and multiple features are fused. The specific process is as follows: s11 frame-level feature extraction: extract the features of the input video to obtain a frame-by-frame feature sequence X with a length of T and a feature dimension of d; s12 Multi-stage sampling of time series features: Use convolutional layers and pooling layers to extract K feature sequences X of different time scales i , the formula is as follows: X i =p i (c i (X)),i≤K where c i 、p i They represent the point-by-point convolution layer and pooling layer used for sampling in the time dimension respectively; the pooling layer kernel size is set to an integer multiple of 2. The larger the value, the shorter the feature sequence length and the larger the time span, which can better express the overall information of the video, but the more detailed visual information is lost; on the contrary, when the value is smaller, the sampling sequence obtained contains more detailed information; s13 Temporal relationship calculation: Use the non-local module to calculate each feature sequence X i The weighted calculation is performed as follows: for each feature sequence X i Use two point-by-point convolutions to get the feature sequence X i The self-attention matrix T i is the feature sequence X i The length of A i After softmax normalization calculation and X i Multiply them together to get the normalized feature sequence X′ i ; s14 Time series multi-scale fusion: Each normalized feature sequence X′ i After extending the inverse average pooling in the time dimension to the original length of the video, the element-by-element addition operation is performed to obtain the multi-scale fused feature sequence I; The second part is to establish an encoder-decoder network based on early attention: the basic structure of the encoder-decoder network is LSTM. The encoder encodes the multi-scale fused feature sequence I, and the decoder decodes the encoded feature sequence into text information; the early attention mechanism is used to weight the fusion of visual features. The method is as follows: s21 encoding process: The multi-scale fused feature sequence I is input into the LSTM encoder to obtain the encoder hidden state sequence h1,h2,...,h with a length of T T ; s22 calculates attention m ij : Using the encoder hidden state h at encoding time i i and the decoder state s at decoding time j j Calculate attention m ij ; Based on s22, s23 adds the attention before decoding time j to obtain the early attention e ij , and then the early attention e ij Add to current attention n ij Calculation of; After that, the current attention n of the early attention will be added ij After softmax normalization, we get the attention weight a ij ; s24 semantic encoding calculation: using attention weight a ij For the encoder hidden state sequence h1,h2,...,h T Perform weighted summation to obtain the semantic coding sequence C1, C2, ..., C U ; s25 loss function L of the attention calculation part a : For encoding time i and decoding time j, take the corresponding attention weight a ij and the softmax normalized early attention softmax (e ij ) is the minimum value of the two, and the sum of the values ​​corresponding to all combinations of i and j is obtained to obtain L a , as shown below; a ij With softmax(e ij )The greater the difference, the a The smaller the value of; s26 decoding process: LSTM decoder recursively predicts a translation sequence y=y1y2...y of length U U , for time j, using the obtained semantic code C j , the decoding symbol y at the previous moment of time j j-1 and the decoder state s j Get the decoded symbol y at time j j ; During s27 training, for the translation sequence y and the target sequence z, the cross entropy loss L is calculated e ; s28 total loss function L: L e and L a Add them together to get the total loss function L.

2. The method according to claim 1, characterized in that s11 frame-level feature extraction: Use the lightweight model SqueezeNet s11 to extract features of the input video.

Images