A WiFi sign language translation system and method based on deep learning

Abstract

This invention discloses a WiFi sign language translation system and method based on deep learning. The method comprises: 1. collecting WiFi signals and obtaining CSI matrix information containing amplitude and phase information from the original CSI data packet; 2. processing the CSI data and filtering the amplitude values ​​of the extracted CSI subcarriers using PCA (principal component analysis); 3. constructing a local-global sequence feature extractor to model the deep features of sign language movements and establish local and global dependencies of the sign language movement information; 4. using a long short-term memory network as a decoder to convert the sequence features into sign sentences, and post-processing the hand language sentences using a connectionist temporal classification model. This invention can effectively reduce the difficulty of sign language recognition and translation, improve the convenience and efficiency of sign language recognition and translation, and can translate sign language into text in real time.

Description

Technical Field

[0001] The present invention relates to a real-time sign language translation system, and in particular to a WiFi sign language translation system and method based on deep learning. Background Art

[0002] Currently, deaf-mute people who want to communicate normally with others can only do so through sign language. With the continuous development of society, the desire of hearing-impaired people to use technology to achieve direct communication with the outside world is growing stronger. For the deaf-mute, sign language is one of their main means of communication.

[0003] In recent years, the rise of the artificial intelligence industry has made it possible for deaf and mute people to communicate with the hearing community. However, the complex and ever-changing environments in which sign language is used have always made the study of sign language fraught with difficulties.

[0004] While traditional sign language recognition has made significant progress, it requires signers to wear proprietary equipment, which is often very expensive. This has limited its widespread adoption. WiFi signals can detect gestures through walls, enabling non-line-of-sight sensing. Low-light or no-light conditions do not affect WiFi signal characteristics. WiFi signals do not contain personal image information, which facilitates privacy protection.

[0005] While vision-based sign language recognition research has been underway for decades, research on wireless sensing-based sign language recognition is still in its infancy, and research that combines both visual understanding and wireless sensing is even rarer. Existing research primarily focuses on gesture recognition based on wireless sensing.

[0006] Through continuous exploration, WiFi sensing has evolved from a simple signal strength-based approach to one based on channel state information (CSI). This approach fully leverages existing signal characteristics and achieves more granular sensing. Furthermore, because the collected signals are noisy, various signal processing techniques, including filtering, principal component analysis, and wavelet transforms, have been extensively studied. While vision-based sign language recognition research has been underway for decades, wireless sensing-based sign language recognition research is still in its infancy. Current WiFi sensing feature extraction primarily calculates statistical features from the raw signal waveform. Furthermore, discrete wavelet transforms are used to extract wavelet coefficients for each frequency band, or short-time Fourier transforms are used to calculate time-frequency features. However, CSI signals are susceptible to interference from the surrounding environment and the device itself, as well as variations caused by reflections and scattering from the transmitter to the receiver. Therefore, direct feature extraction from collected CSI data is difficult. A signal propagation link exists between any two transmitting and receiving antennas, each containing 30 subcarriers of CSI data. Using all the data from each link and subcarrier would be computationally expensive. Summary of the Invention

[0007] In order to address the deficiencies of the above-mentioned prior art, the present invention proposes a WiFi sign language translation system and method based on deep learning, in order to achieve the translation of continuous sign language sentences through Wi-Fi signals and improve the accuracy of sign language translation, thereby making sign language translation technology more popular and enabling hearing-impaired people to communicate with each other more conveniently.

[0008] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0009] The characteristics of the WiFi sign language translation system based on deep learning of the present invention include: a data acquisition module, a data preprocessing module, a local-global sequence feature extractor and a real-time translation module;

[0010] The data acquisition module collects original CSI data packets, extracts channel state information of the WiFi signal from the original CSI data packets, and then obtains a three-dimensional CSI matrix containing original amplitude and phase information from the channel state information;

[0011] The data preprocessing module performs dimensionality reduction processing on the three-dimensional CSI matrix to obtain a time series vector, then removes abnormal points in the time series vector using the Hampel identifier method, then uses a filtering algorithm to filter quantities unrelated to the principal component, and finally uses a discrete wavelet transform to remove noise in the time series vector, thereby obtaining a preprocessed time series vector;

[0012] The local-global sequence feature extractor uses a local feature extractor containing a residual convolutional network to extract features of each channel of the preprocessed time series vector to obtain a local feature vector of each channel; then uses a feature enhancer containing a gating mechanism to dynamically adjust the weight of the local feature vector of each channel; finally, uses a global feature extractor containing a Transformer encoder to encode the local feature vectors of each channel and their weights to obtain a global layer feature vector;

[0013] The real-time translation module decodes the global layer feature vector to obtain a word vector and converts it into a word sequence number, thereby finding the corresponding word in the corpus according to the word sequence number and forming a word sequence, and then post-processing the word sequence to output a complete natural sign language sentence.

[0014] The real-time sign language translation system of the present invention is also characterized in that a deep neural network model of a local-global sequence feature extractor is constructed, comprising: a local feature extractor, a feature enhancer, and a global feature extractor;

[0015] The residual convolutional network in the local feature extractor comprises an N-layer structure, each layer consisting of a first convolutional layer, a first normalization layer, a second convolutional layer, a second normalization layer and a residual connection layer;

[0016] The feature enhancer consists of a compressed channel attention mechanism layer and an excitation layer;

[0017] The Transformer encoder in the global feature extractor comprises an L-layer structure, each layer of which is sequentially composed of a first normalization layer, a multi-head attention layer, a first residual connection layer, a second normalization layer, a first multi-layer perceptron, and a second residual connection layer.

[0018] The characteristics of the WiFi sign language translation method based on deep learning of the present invention are that it is carried out according to the following steps:

[0019] Step 1: Collect original WiFi signals:

[0020] Obtaining a CSI data packet of an original WiFi signal, extracting channel state information of the WiFi signal from the original CSI data packet, and then obtaining a three-dimensional CSI matrix containing original amplitude and phase information from the channel state information;

[0021] Step 2: CSI data preprocessing:

[0022] The PCA method is used to reduce the dimension of the obtained three-dimensional CSI matrix to obtain a time series vector. The Hampel identifier method is then used to remove abnormal points in the time series vector. The filtering algorithm is then used to filter the quantities in the time series vector that are not related to the principal component. Finally, the discrete wavelet transform method is used to remove the noise in the time series vector, thereby obtaining the preprocessed time series vector, which is recorded as Among them, x t represents the effective signal at time t, T and d represent the time series length and feature vector dimension respectively;

[0023] Step 3: Local-global feature extraction:

[0024] Step 3.1: Timing information embedding:

[0025] Use formula (1) to get the position embedding vector

[0026]

[0027] In formula (1), Linear(·) represents the fully connected layer, f pos (·) represents the feature embedding matrix and a lookup table consisting of a timestamp vector [1,...,T];

[0028] Step 3.2: Construct a deep neural network model of local-global sequence feature extractor, including: local feature extractor, feature enhancer and global feature extractor;

[0029] Step 3.2.1: The local feature extractor uses formula (2) to extract the local features of the n-1 layer feature sign language action. Encode and output the local features of the sign language action at the nth layer

[0030]

[0031]

[0032] In formula (2), ReLU(·) is the ReLU activation function, BN1 and BN2 represent the first and second batch normalization layers, respectively, Conv1 and Conv2 represent the first and second convolutional layers, respectively. Represents the intermediate variable of the local feature extractor. When n=1, let

[0033] Step 3.2.2: The feature enhancer uses equations (3) and (4) to enhance the local features of the Nth layer of the sign language action. Perform feature extraction to obtain channel-enhanced local features of sign language movements

[0034]

[0035]

[0036] In formula (3), Sigmoid(·) is the Sigmoid activation function, ReLU(·) is the ReLU activation function, and W h , b represents the pre-trained weights and bias of the first fully connected layer, W h ′, b′ represent the pre-trained weights and bias of the second fully connected layer; Represents the channel attention feature weight matrix;

[0037] Step 3.2.3: The global feature extractor outputs the lth layer global feature of the sign language using formula (5)

[0038]

[0039]

[0040] In formula (5), MSA(·) is the multi-head attention layer, MLP(·) is the multi-layer perceptron layer, LN(·) is the layer normalization function, L = 6L is the number of layers of the Transformer encoder, Represents the intermediate variable of the global feature. When l=1, let

[0041] Step 4: Translate natural sentences:

[0042] Step 4.1: Use formula (6) to transform the Lth layer containing the global features of sign language actions into Decoding into word vectors

[0043]

[0044] In formula (6), LSTM(·) is a bidirectional long short-term memory network with two hidden layers and serves as a decoder;

[0045] Step 4.2: Word vector Finally, it is converted into a word sequence number, and the corresponding word in the corpus is found according to the word sequence number to form a word sequence. Then, the words with the same meaning and repeated words in the word sequence are merged, and then all words representing transition actions are deleted to obtain the processed word sequence and serve as the final translation output of the natural sign language sentence.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. Simplicity and Convenience: Currently, WiFi devices and signals are ubiquitous in many locations, making WiFi data acquisition convenient. Instead of computing and analyzing large amounts of multimedia data based on images or videos, the system only needs to use wireless router WiFi signals to obtain the CSI data of sign language information and perform simple preprocessing such as noise reduction and redundancy removal. This includes removing local outliers, low-pass filtering, and discrete wavelet transform denoising, which can serve as key information for extracting feature information. The parameters used in feature extraction are relatively simple to calculate, making it more feasible and, to a certain extent, achieving relatively ideal data processing results.

[0048] 2. Universal Applicability: Traditional human perception technologies are mainly used in cameras, radars, and wearable devices. These traditional methods all have their own limitations. WiFi signals can effectively overcome these limitations. They are independent of lighting conditions, do not infringe on user privacy, are low-cost, and have high accuracy. Furthermore, WiFi signals can translate sign language into text in real time when deaf and mute people communicate with others without the need for wearing proprietary equipment, enabling normal people to communicate with deaf and mute people in real time.

[0049] 3. Accuracy: After obtaining the valid signal in the data, features are extracted from the preprocessed CSI data, and then classified and identified using a classification algorithm. A deep neural network model of a local-global sequence feature extractor is constructed by adopting a deep learning method, matching the sign language information database, obtaining its valid sentence information, and processing the sentence information content to obtain a processed sentence sequence as the final translated natural sentence output. Among them, the deep learning method used in implementing the classification algorithm for the recognition and classification of sign language words and the natural language processing of real-time sign language translation makes the output natural sign language sentences have better coherence and higher accuracy, improving the effect of real-time sign language translation sentences, thereby achieving higher recognition accuracy, better translation effect, and more efficient classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a schematic diagram of the structure of the WiFi sign language translation system based on deep learning of the present invention;

[0051] Figure 2 This is a WiFi propagation model diagram of the data acquisition module of the present invention;

[0052] Figure 3 Schematic diagram of the local-global feature extractor of the present invention;

[0053] Figure 4 Schematic diagram of the structure of the real-time translation module of the present invention.

[0054] Figure 5 This is an operational flow chart of the WiFi sign language translation method based on deep learning of the present invention. DETAILED DESCRIPTION

[0055] The specific technical solutions for implementing the present invention will be described in detail below with reference to the accompanying drawings.

[0056] In this embodiment, a WiFi sign language translation system based on deep learning is Figure 1 As shown, it includes a data acquisition module, a data preprocessing module, a local-global sequence feature extractor, and a real-time translation module;

[0057] The data acquisition module uses the acquisition equipment to collect and store the channel state information of the WiFi signal. It obtains the CSI matrix information containing amplitude and phase information from the collected original CSI data packet. The extracted three-dimensional CSI matrix is ​​a 3×3×30 three-dimensional matrix. The WiFi propagation model is shown in the figure below. Figure 2 shown.

[0058] The data preprocessing module reduces the dimension of the collected CSI data set to obtain a time series vector, removes outliers using the Hampel identifier method, filters out quantities unrelated to the main components, and then removes noise from the signal using a discrete wavelet transform to obtain the preprocessed time series vector.

[0059] The structure diagram of the local-global sequence feature extractor is as follows Figure 3 As shown in the figure, a local feature extractor containing a residual convolutional network is used to extract the features of each channel of the preprocessed time series vector to obtain the local feature vector of each channel; a feature enhancer containing a gating mechanism is then used to dynamically adjust the weights of the local feature vectors of each channel; finally, a global feature extractor containing a Transformer encoder is used to encode the local feature vectors of each channel and their weights to obtain the global layer feature vector;

[0060] In the specific implementation, a deep neural network model of local-global sequence feature extractor is constructed, including: local feature extractor, feature enhancer and global feature extractor;

[0061] The residual convolutional network in the local feature extractor contains an N-layer structure, each of which consists of the first convolutional layer, the first normalization layer, the second convolutional layer, the second normalization layer and the residual connection layer;

[0062] The feature enhancer consists of a compressed channel attention mechanism layer and an excitation layer;

[0063] The Transformer encoder in the global feature extractor contains an L-layer structure, each of which consists of a first normalization layer, a multi-head attention layer, a first residual connection layer, a second normalization layer, a first multi-layer perceptron, and a second residual connection layer in sequence.

[0064] The structure diagram of the real-time translation module is as follows Figure 4 As shown in the figure, the global layer feature vector is decoded to obtain the word vector and convert it into a word sequence number, so as to find the corresponding word in the corpus according to the word sequence number and form a word sequence. The word sequence is then post-processed to output a complete natural sign language sentence.

[0065] In this embodiment, a WiFi sign language translation method based on deep learning is to first use the acquisition equipment to realize the acquisition and storage of the channel state information of the WiFi signal, obtain the three-dimensional CSI matrix information of Ntx*Nrx*30 containing amplitude and phase information from the collected original CSI data packet, perform dimensionality reduction processing on the CSI matrix to obtain a time series vector, remove abnormal points in the CSI data by the Hampel identifier method, and then filter out the quantities unrelated to the main components, and then remove the noise in the signal by discrete wavelet transform, and then use the constructed local-global sequence feature extractor deep neural network model to extract features from the processed CSI as key information, and then use the long short-term memory network to decode the sign language features to obtain word vectors, and then convert them into word numbers, find the corresponding words in the corpus according to the word numbers and form a word sequence, and then post-process the word sequence to output a complete natural sign language sentence. Figure 5 Specifically, the following steps are included:

[0066] Step 1: Collect original WiFi signals:

[0067] A sign language information database is pre-stored on a terminal device; the sign language information corpus is used to store sign language phrase information corresponding to the sign language information; an Intel-5300 CSI TOOL software package is used on a host computer to obtain a CSI data packet of an original WiFi signal, extract channel state information of the WiFi signal from the original CSI data packet, and then obtain a three-dimensional CSI matrix containing original amplitude and phase information from the channel state information;

[0068] Step 2: CSI data preprocessing:

[0069] The obtained three-dimensional CSI matrix is ​​processed with PCA to reduce the dimensionality to obtain a time series vector. The Hampel identifier method is used to remove abnormal points in the time series vector. Then, a filtering algorithm is used to filter out the quantities in the time series vector that are not related to the principal components. Finally, discrete wavelet transform is used to remove noise in the time series vector to obtain the preprocessed time series vector.

[0070] The Hampel identifier method treats points that leave the closed interval [μ-γ×σ,μ+γ×σ] as outliers, where μ represents the median of a waveform, σ represents the absolute deviation of the median of a waveform, and γ generally takes the default value of 3.

[0071] Step 3: Local-global feature extraction:

[0072] Step 3.1: Timing information embedding:

[0073] The effective CSI time series vector after wavelet transform preprocessing is recorded as Among them, x t represents the effective signal at time t, T and d represent the time series length and feature vector dimension respectively; a learnable position embedding vector representing the time series information is added to the preprocessed CSI signal

[0074]

[0075] In formula (1), Linear(·) represents the fully connected layer, f pos (·) represents the learnable feature embedding matrix and a lookup table consisting of a timestamp vector [1,...,T];

[0076] Step 3.2: Construct a local-global sequence feature extractor, including a local feature extractor, a feature enhancer, and a global feature extractor;

[0077] Step 3.2.1: The local feature extractor uses formula (2) to extract the local features of the n-1 layer feature sign language action. Encode and output the local features of the sign language action at the nth layer

[0078]

[0079]

[0080] In formula (2), ReLU(·) is the ReLU activation function, BN1 and BN2 represent the first and second batch normalization layers, respectively, Conv1 and Conv2 represent the first and second convolutional layers, respectively. Represents the intermediate variable of the local feature extractor. When n=1, let

[0081] Step 3.2.2: The feature enhancer uses equations (3) and (4) to enhance the local features of the Nth layer of the sign language action. Perform feature extraction to obtain channel-enhanced local features of sign language movements

[0082]

[0083]

[0084] In formula (3), Sigmoid(·) is the Sigmoid activation function, ReLU(·) is the ReLU activation function, and W h , b represents the pre-trained weights and bias of the first fully connected layer, W h ′, b′ represent the pre-trained weights and bias of the second fully connected layer; Represents the channel attention feature weight matrix;

[0085] Step 3.2.3: The global feature extractor outputs the lth layer global feature of the sign language using formula (5)

[0086]

[0087]

[0088] In formula (5), MSA(·) is the multi-head attention layer, MLP(·) is the multi-layer perceptron layer, LN(·) is the layer normalization function, L = 6L is the number of layers of the Transformer encoder, Represents the intermediate variable of the global feature. When l=1, let

[0089] Step 4: Translate natural sentences:

[0090] Step 4.1: Use formula (6) to transform the Lth layer containing the global features of sign language actions into Decoding into word vectors

[0091]

[0092] In formula (6), LSTM(·) is a bidirectional long short-term memory network with two hidden layers and serves as a decoder;

[0093] Step 4.2: Word vector Finally, it is converted into a word sequence number, and the corresponding word in the corpus is found according to the word sequence number to form a word sequence. Then, the words with the same meaning and repeated words in the word sequence are merged, and then all words representing transition actions are deleted to obtain the processed word sequence and serve as the final translation output of the natural sign language sentence.

[0094] In summary, the present invention proposes a WiFi sign language translation system and method based on deep learning, which utilizes ubiquitous WiFi to recognize and translate sign language. It does not require computational analysis of large amounts of multimedia data based on images or videos, but only requires processing of CSI data. This effectively solves the limitations of traditional methods, and users do not need to carry any equipment. The deep learning method is used to construct a deep neural network model of a local-global sequence feature extractor, which can further improve the accuracy of recognition. The real-time sign language translation system and method effectively reduce the cost of sign language recognition and translation, improve the convenience of sign language recognition and translation, and translate sign language into text in real time.

Claims

1. A deep learning-based WiFi sign language translation system, featuring: Data acquisition module, data preprocessing module, local-global sequence feature extractor and real-time translation module; The data acquisition module collects original CSI data packets, extracts channel state information of the WiFi signal from the original CSI data packets, and then obtains a three-dimensional CSI matrix containing original amplitude and phase information from the channel state information; The data preprocessing module performs dimensionality reduction processing on the three-dimensional CSI matrix to obtain a time series vector, then removes abnormal points in the time series vector using the Hampel identifier method, then uses a filtering algorithm to filter quantities unrelated to the principal component, and finally uses a discrete wavelet transform to remove noise in the time series vector, thereby obtaining a preprocessed time series vector; The local-global sequence feature extractor uses a local feature extractor containing a residual convolutional network to extract features of each channel of the preprocessed time series vector to obtain a local feature vector of each channel; then uses a feature enhancer containing a gating mechanism to dynamically adjust the weight of the local feature vector of each channel; finally, uses a global feature extractor containing a Transformer encoder to encode the local feature vectors of each channel and their weights to obtain a global layer feature vector, including: Step 3.1: Timing information embedding: Use formula (1) to get the position embedding vector : (1) In formula (1), represents the fully connected layer, Represented by the feature embedding matrix and timestamp vector A lookup table composed of Represents the preprocessed time series vector; Step 3.2: Construct a deep neural network model of local-global sequence feature extractor, including: local feature extractor, feature enhancer and global feature extractor; Step 3.2.1: The local feature extractor uses formula (2) to extract the local features of the n-1 layer feature sign language action. Encode and output the local features of the sign language action at the nth layer : (2) In formula (2), is the ReLU activation function, BN1 and BN2 represent the first and second batch normalization layers respectively, Conv1 and Conv2 represent the first and second convolutional layers respectively, Represents the intermediate variable of the local feature extractor. When n=1, let ; Step 3.2.2: The feature enhancer uses equations (3) and (4) to enhance the local features of the Nth layer of the sign language action. Perform feature extraction to obtain channel-enhanced local features of sign language movements : (3) (4) In formula (3), is the Sigmoid activation function, is the ReLU activation function, , represents the pre-trained weights and biases of the first fully connected layer, , Represents the pre-trained weights and biases of the second fully connected layer; Represents the channel attention feature weight matrix; Step 3.2.3: The global feature extractor outputs the lth layer global feature of the sign language using formula (5) : (5) In formula (5), is the multi-head attention layer, is a multi-layer perceptron layer, is the layer normalization function, is the number of layers of the Transformer encoder, Represents the intermediate variable of the global feature. When l=1, let ; The real-time translation module decodes the global layer feature vector to obtain a word vector and converts it into a word sequence number, thereby finding the corresponding word in the corpus according to the word sequence number and forming a word sequence, and then post-processing the word sequence to output a complete natural sign language sentence.

2. The deep learning-based WiFi sign language translation system according to claim 1, characterized in that: Construct a deep neural network model of local-global sequence feature extractor, including: local feature extractor, feature enhancer and global feature extractor; The residual convolutional network in the local feature extractor comprises an N-layer structure, each layer consisting of a first convolutional layer, a first normalization layer, a second convolutional layer, a second normalization layer and a residual connection layer; The feature enhancer consists of a compressed channel attention mechanism layer and an excitation layer; The Transformer encoder in the global feature extractor contains Layer structure, each layer is composed of the first normalization layer, multi-head attention layer, the first residual connection layer, the second normalization layer, the first multi-layer perceptron and the second residual connection layer in sequence.

3. A WiFi sign language translation method based on deep learning, characterized in that: The steps are as follows: Step 1: Collect original WiFi signals: Obtaining a CSI data packet of an original WiFi signal, extracting channel state information of the WiFi signal from the CSI data packet, and then obtaining a three-dimensional CSI matrix containing original amplitude and phase information from the channel state information; Step 2: CSI data preprocessing: The PCA method is used to reduce the dimension of the obtained three-dimensional CSI matrix to obtain a time series vector. The Hampel identifier method is then used to remove abnormal points in the time series vector. The filtering algorithm is then used to filter the quantities in the time series vector that are not related to the principal component. Finally, the discrete wavelet transform method is used to remove the noise in the time series vector, thereby obtaining the preprocessed time series vector, which is recorded as ,in, represents the effective signal at time t, T and d represent the time series length and feature vector dimension respectively; Step 3: Local-global feature extraction: Step 3.1: Timing information embedding: Use formula (1) to get the position embedding vector : (1) In formula (1), represents the fully connected layer, Represented by the feature embedding matrix and timestamp vector A lookup table composed of Step 3.2: Construct a deep neural network model of local-global sequence feature extractor, including: local feature extractor, feature enhancer and global feature extractor; Step 3.2.1: The local feature extractor uses formula (2) to extract the local features of the n-1 layer feature sign language action. Encode and output the local features of the sign language action at the nth layer : (2) In formula (2), is the ReLU activation function, BN1 and BN2 represent the first and second batch normalization layers respectively, Conv1 and Conv2 represent the first and second convolutional layers respectively, Represents the intermediate variable of the local feature extractor. When n=1, let ; Step 3.2.2: The feature enhancer uses equations (3) and (4) to enhance the local features of the Nth layer of the sign language action. Perform feature extraction to obtain channel-enhanced local features of sign language movements : (3) (4) In formula (3), is the Sigmoid activation function, is the ReLU activation function, , represents the pre-trained weights and biases of the first fully connected layer, , Represents the pre-trained weights and biases of the second fully connected layer; Represents the channel attention feature weight matrix; Step 3.2.3: The global feature extractor outputs the lth layer global feature of the sign language using formula (5) : (5) In formula (5), is the multi-head attention layer, is a multi-layer perceptron layer, is the layer normalization function, is the number of layers of the Transformer encoder, Represents the intermediate variable of the global feature. When l=1, let ; Step 4: Translate natural sentences: Step 4.1: Use formula (6) to transform the Lth layer containing the global features of sign language actions into Decoding into word vectors : (6) In formula (6), A bidirectional long short-term memory network with two hidden layers as the decoder; Step 4.2: Word vector Finally, it is converted into a word sequence number, and the corresponding word in the corpus is found according to the word sequence number to form a word sequence. Then, the words with the same meaning and repeated words in the word sequence are merged, and then all words representing transition actions are deleted to obtain the processed word sequence and serve as the final translation output of the natural sign language sentence.

Images