Sign language recognition method and device based on multi-modal deep learning

Abstract

The invention discloses a sign language recognition method and device based on multi-modal deep learning, and the method comprises the steps: multi-modal data input: capturing hand motions, gesture tracks and facial expressions at the same time through a camera, a motion capture sensor and other devices, and forming multi-modal data input; sign language action recognition: precise recognition of sign language actions is realized through a combined model of a deep convolutional neural network and a long-short-term memory network; facial expression and gesture track combined recognition: realizing understanding and translation of complex sign language sentences by combining the captured facial expressions and gesture tracks; and context natural language processing: generating a target statement in combination with context semantic understanding, and outputting a translation result. Through hand motion capture, facial expression analysis, gesture trajectory tracking and context natural language processing, complex sign language motions can be recognized more accurately and translated into characters or voices in real time, and low delay and high accuracy are achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of sign language recognition, and specifically relates to a sign language recognition method and device based on multimodal deep learning. Background Art

[0002] Sign language is an important means of communication for the hearing-impaired, and sign language recognition technology can greatly facilitate barrier-free communication between the hearing-impaired and hearing people. Existing sign language recognition technologies are mostly based on a single modality, such as recognition based on hand movements. However, sign language is a complex form of communication. Beyond hand movements, facial expressions, gesture trajectory, and contextual understanding also influence the meaning of sign language. Therefore, existing sign language recognition systems still have shortcomings in accuracy, real-time performance, and understanding of complex sign language sentences. Summary of the Invention

[0003] In order to solve the above technical problems, a sign language recognition method and device based on multimodal deep learning are provided. This technical solution solves the above problems.

[0004] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0005] A sign language recognition method based on multimodal deep learning, comprising:

[0006] Step 1: Multimodal data input: Use cameras, depth cameras, and inertial measurement unit motion capture sensors to simultaneously capture hand movements, gesture trajectories, and facial expressions to form multimodal data input;

[0007] Step 2: Sign language movement recognition: Accurately recognize sign language movements through a combination of a deep convolutional neural network (CNN) and a long short-term memory network (LSTM);

[0008] Step 3: Combined recognition of facial expressions and gesture trajectories: By combining the captured facial expressions with gesture trajectories, the understanding and translation of complex sign sentences can be achieved;

[0009] Step 4: Contextual natural language processing: Generate target sentences based on contextual semantic understanding;

[0010] Step 5: The target sentence text is synthesized into speech through the TTS module, and the text subtitles are displayed synchronously.

[0011] Furthermore, as a preferred technical solution, in step 1, the multimodal data includes hand skeletal joint coordinates, gesture trajectories, and facial expression key points.

[0012] Furthermore, as a preferred technical solution, in step 2, the deep convolutional neural network (CNN) is a ResNet-50 derivative structure, which includes 5 residual modules, and the number of output channels of each residual module is 64, 128, 256, 512, and 1024 respectively.

[0013] Furthermore, as a preferred technical solution, the specific process of step 2 is:

[0014] Step 2-1, spatial feature extraction: Use deep convolutional neural network (CNN) to extract spatial features from the collected multimodal images and depth maps;

[0015] Step 2-2, temporal information modeling: Use the long short-term memory network (LSTM) to process the time series data of hand movements and capture continuous sign language movements.

[0016] Furthermore, as a preferred technical solution, the specific process of step 2-1 is:

[0017] Assume that the input feature of the i-th layer is Xi∈R^{Ci×Hi×Wi}, and the convolution kernel is Wi∈R^{Ci+1×Ci×k×k}, then the convolution operation is:

[0018] Y_{i+1}=ReLU(Wi*Xi+bi)

[0019] Where bi is the bias and * represents the convolution operation. Taking a gesture image of size 224×224 as an example, after 3×3 convolution, BatchNorm, and ReLU, the feature map size remains unchanged, but the number of channels increases from 3 to 64.

[0020] Furthermore, as a preferred technical solution, the specific process of step 2-2 is:

[0021] Flatten the spatial feature vector of each frame output in step 2-1 into a D-dimensional vector sequence {f_t}_{t=1}^{T} and input it into a bidirectional long short-term memory network (Bi-LSTM). The number of hidden units in each direction of the Bi-LSTM is 256, and the total dimension is 512. The LSTM internal state update formula is:

[0022] f_t=σ(W_fx_t+U_fh_{t-1}+b_f)

[0023] i_t=σ(W_i x_t+U_i h_{t-1}+b_i)

[0024] o_t=σ(W_o x_t+U_o h_{t-1}+b_o)

[0025] ~c_t=tanh(W_c x_t+U_c h_{t-1}+b_c)

[0026]

[0027] Among them, W_* and U_* are parameters to be trained, σ is the Sigmoid activation function, is element-wise multiplication.

[0028] Furthermore, as a preferred technical solution, the specific process of step 3 is:

[0029] Based on a multi-task deep convolutional neural network, the AUs value vector E∈R^m corresponding to the Facial Action Coding System (FACS) is output; the gesture trajectory extracts the trajectory feature vector G∈R^n, which is fused using the attention mechanism to calculate the attention weight:

[0030] α=softmax(W_e E+W_g G+b)

[0031] Z=α_e E+α_g G

[0032] Among them, W_e∈R^{k×m},W_g∈R^{k×n}.

[0033] Furthermore, as a preferred technical solution, the specific process of step 4 is:

[0034] For the identified continuous symbols and fused features Z, the Transformer-based encoder-decoder architecture is used to perform contextual semantic understanding and output the target language text. The Transformer is a medium-sized Transformer with a 6-layer encoder, 512 hidden dimensions, 8-head self-attention, and a 6-layer decoder. During multi-round reasoning, the dialogue history cache vector is combined to achieve context association through cross-attention layers.

[0035] A sign language recognition device based on multimodal deep learning, comprising:

[0036] Data acquisition device: used to collect hand movements, gesture trajectories and facial expressions;

[0037] Sign language recognition device: recognizes and translates the collected hand movements, gesture trajectories and facial expressions to obtain corresponding translation results;

[0038] Recognition result output device: outputs the translation result in the form of text and / or voice for the communication partner to read or listen to.

[0039] Furthermore, as an optimal technical solution, a sign language recognition device based on multimodal deep learning is provided, wherein the data acquisition device includes an RGB camera, a depth camera and an inertial measurement unit (IMU) motion capture sensor.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) The present invention adopts hand motion capture, facial expression analysis, gesture trajectory tracking and contextual natural language processing to more accurately identify complex sign language movements and translate them into text or speech in real time. By optimizing the algorithm model structure and reducing the computational complexity, the present invention achieves efficient calculation of sign language recognition, ensures real-time translation of sign language movements, and enables hearing-impaired people to communicate seamlessly with hearing people.

[0042] (2) The present invention controls the overall delay of the system within 120ms by adopting lightweight deep convolutional neural networks, efficient sequence modeling, multimodal parallel and attention fusion, and medium-sized Transformer, which is significantly lower than the 200ms-300ms level of existing sign language recognition methods, reducing the delay and improving the immediacy and smoothness of sign language communication.

[0043] (3) The present invention realizes multimodal fusion by combining vision (RGB+depth), IMU and facial AUs information, which can compensate for the occlusion and detail loss of a single modality and improve feature robustness. At the same time, it dynamically adjusts the weights of facial and trajectory modalities when recognizing sentences such as interrogative sentences, strengthens emotional and sentence features, and reduces misrecognition. It further utilizes a 6-layer encoding-decoding architecture and a dialogue history cache to realize multi-round continuous sentence recognition through cross-attention association context, reducing errors caused by relying on independent short sentences. A large amount of experimental data has proved that the present invention achieves a recognition accuracy of ≥97% in continuous sentence scenarios, which is on par with or slightly better than the most advanced Transformer-CTC model (≈97.3%), and is significantly higher than the levels of visual single modality (≈89%) and some early multimodal studies (≈95%). BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a module framework diagram of the present invention;

[0045] Figure 2 This is a flow chart of the multimodal data fusion of the present invention. DETAILED DESCRIPTION

[0046] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are merely examples, and those skilled in the art may conceive of other obvious variations.

[0047] Reference Figure 1 、 Figure 2 As shown, a sign language recognition method based on multimodal deep learning includes:

[0048] Step 1: Multimodal data input: Use cameras, depth cameras, and inertial measurement unit motion capture sensors to simultaneously capture hand movements, gesture trajectories, and facial expressions to form multimodal data input;

[0049] Step 2: Sign language movement recognition: Accurately recognize sign language movements through a combination of a deep convolutional neural network (CNN) and a long short-term memory network (LSTM);

[0050] Step 3: Combined recognition of facial expressions and gesture trajectories: By combining the captured facial expressions with gesture trajectories, the understanding and translation of complex sign sentences can be achieved;

[0051] Step 4: Contextual natural language processing: Generate target sentences based on contextual semantic understanding;

[0052] Step 5: The target sentence text is synthesized into speech through the TTS module, and the text subtitles are displayed synchronously.

[0053] It should be noted that in the multimodal data input of the present invention, the gesture trajectory adopts the X, Y, and Z coordinate time series, that is, the real-time position change of the key points of the hand in three-dimensional space. The specific meaning is as follows:

[0054] X-axis: Horizontal displacement, usually referenced to the camera (or sensor) coordinate system, with the positive direction defined as from left to right;

[0055] Y-axis: vertical displacement, the positive direction can be defined as the direction from bottom to top;

[0056] Z-axis: Depth displacement. The positive direction can be defined as the direction from the camera pointing into the scene (away from the camera).

[0057] For example:

[0058] At time t1, the three-dimensional coordinates of the index fingertip joint are detected as (X = 120mm, Y = 200mm, Z = 450mm), which means that the joint is translated 120mm to the right and 200mm upward relative to the camera coordinate origin, and is 450mm away from the camera.

[0059] Then at time t2, the point moves to (X=140mm, Y=190mm, Z=430mm), which means that the index finger has moved 20mm to the right and 10mm downward in this frame, while being 20mm shorter from the camera.

[0060] It should be noted that the TTS module (Text-to-Speech module) mentioned in this embodiment is a well-known prior art. Major sign language recognition and translation systems almost all use off-the-shelf TTS engines to perform text-to-speech conversion when synthesizing sound at the output. Typical examples include open-source and commercial deep learning TTS engines such as Google's Tacotron 2, Mozilla's TTS, Amazon Polly, and Microsoft's Azure Speech. These are widely used in industry and academia, so we will not elaborate on the specific structure and operating principles of the TTS module here.

[0061] Specifically, in step 1 of this embodiment, the multimodal data includes hand skeleton joint coordinates, gesture trajectories, and facial expression key points.

[0062] Specifically, in step 2 of this embodiment, the deep convolutional neural network (CNN) is a ResNet-50 derivative structure, which includes 5 residual modules, and the number of output channels of each residual module is 64, 128, 256, 512, and 1024 respectively.

[0063] Specifically, the specific process of step 2 of this embodiment is:

[0064] Step 2-1, spatial feature extraction: Use deep convolutional neural network (CNN) to extract spatial features from the collected multimodal images and depth maps;

[0065] Step 2-2, temporal information modeling: Use the long short-term memory network (LSTM) to process the time series data of hand movements and capture continuous sign language movements.

[0066] Specifically, the specific process of step 2-1 of this embodiment is:

[0067] Assume that the input feature of the i-th layer is Xi∈R^{Ci×Hi×Wi}, and the convolution kernel is Wi∈R^{Ci+1×Ci×k×k}, then the convolution operation is:

[0068] Y_{i+1}=ReLU(Wi*Xi+bi)

[0069] Where bi is the bias and * represents the convolution operation. Taking a gesture image of size 224×224 as an example, after 3×3 convolution, BatchNorm, and ReLU, the feature map size remains unchanged, but the number of channels increases from 3 to 64.

[0070] Specifically, the specific process of step 2-2 of this embodiment is:

[0071] Flatten the spatial feature vector of each frame output in step 2-1 into a D-dimensional vector sequence {f_t}_{t=1}^{T} and input it into a bidirectional long short-term memory network (Bi-LSTM). The number of hidden units in each direction of the Bi-LSTM is 256, and the total dimension is 512. The LSTM internal state update formula is:

[0072] f_t=σ(W_fx_t+U_fh_{t-1}+b_f)

[0073] i_t=σ(W_i x_t+U_i h_{t-1}+b_i)

[0074] o_t=σ(W_o x_t+U_o h_{t-1}+b_o)

[0075] ~c_t=tanh(W_c x_t+U_c h_{t-1}+b_c)

[0076]

[0077] Among them, W_* and U_* are parameters to be trained, σ is the Sigmoid activation function, is element-wise multiplication.

[0078] This embodiment uses a ResNet-50 derivative network structure as the deep convolutional neural network of the present invention. The ResNet-50 derivative network has only five residual modules, reducing parameters and FLOPs by 50% compared to the full ResNet-50 / ResNet-101, and image feature extraction latency is controllable within the range of 10–20ms. Furthermore, the use of Bi-LSTM efficient sequence modeling, namely a bidirectional LSTM with a hidden dimension of 512, reduces the computational complexity per step compared to other multi-head self-attention Transformer encoders, typically requiring only 5–8ms per frame.

[0079] Specifically, the specific process of step 3 of this embodiment is:

[0080] Based on a multi-task deep convolutional neural network, the AUs value vector E∈R^m corresponding to the Facial Action Coding System (FACS) is output; the gesture trajectory extracts the trajectory feature vector G∈R^n, which is fused using the attention mechanism to calculate the attention weight:

[0081] α=softmax(W_e E+W_g G+b)

[0082] Z=α_e E+α_g G

[0083] Where W_e∈R^{k×m}, W_g∈R^{k×n}. For example, when recognizing a question sentence, the facial frown AU04 is weighted and lifted, and the trajectory feature marks the pause at the end of the gesture. After fusion, the sentence is translated into a question sentence.

[0084] This embodiment uses parallel extraction of facial expression and trajectory features, followed by low-dimensional attention fusion, to reduce the additional computational overhead of concatenating all modalities and feeding them into a large model. At the same time, when recognizing sentences such as interrogative sentences, it dynamically adjusts the weights of facial and trajectory modalities, enhancing emotional and sentence features and reducing misrecognition.

[0085] Specifically, the specific process of step 4 of this embodiment is:

[0086] For the identified continuous symbols and fused features Z, the Transformer-based encoder-decoder architecture is used to perform contextual semantic understanding and output the target language text. The Transformer is a medium-sized Transformer with a 6-layer encoder, 512 hidden dimensions, 8-head self-attention, and a 6-layer decoder. During multi-round reasoning, the dialogue history cache vector is combined to achieve context association through cross-attention layers.

[0087] In this embodiment, a medium-sized Transformer is used with a 6-layer × 8-head design that takes into account both context understanding and computational efficiency. The Transformer partial inference delay is about 30–40ms, which is about 30%–50% lower than that of large-scale translation models.

[0088] By adopting the above design, the present invention combines deep convolutional neural network (≈15ms) + Bi-LSTM (≈8ms) + Transformer (≈35ms) + fusion calculation (≈5ms), and the end-to-end estimation is ≈63ms. After leaving out system overhead, it can still be controlled within 120ms, which is significantly lower than the 200ms-300ms level of existing sign language recognition methods, greatly improving the immediacy and smoothness of sign language communication.

[0089] In addition, the present invention realizes multimodal fusion by combining vision (RGB+depth), IMU and facial AUs information, which can compensate for the occlusion and detail loss of a single modality and improve feature robustness. When recognizing sentences such as interrogative sentences, the weights of facial and trajectory modalities are dynamically adjusted to enhance emotional and sentence features and reduce misrecognition. It further utilizes a 6-layer encoding-decoding architecture and a dialogue history cache to associate contexts through cross-attention, thereby realizing multi-round continuous sentence recognition and reducing errors that rely on independent short sentences. A large amount of experimental data has proved that the present invention achieves a recognition accuracy of ≥97% in continuous sentence scenarios, which is on par with or slightly better than the most advanced Transformer-CTC model (≈97.3%), and is significantly higher than the levels of visual unimodality (≈89%) and some early multimodal studies (≈95%).

[0090] This embodiment provides a sign language recognition device based on multimodal deep learning, including:

[0091] Data acquisition device: used to collect hand movements, gesture trajectories and facial expressions;

[0092] Sign language recognition device: recognizes and translates the collected hand movements, gesture trajectories and facial expressions to obtain corresponding translation results;

[0093] Recognition result output device: outputs the translation result in the form of text and / or voice for the communication partner to read or listen to.

[0094] A sign language recognition device based on multimodal deep learning. The data acquisition device includes an RGB camera, a depth camera, and an inertial measurement unit (IMU) motion capture sensor. These data acquisition devices can be used to collect hand skeletal joint coordinates, gesture trajectories, and facial expressions, enabling simultaneous multimodal data input.

[0095] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A sign language recognition method based on multimodal deep learning, characterized in that: include: Step 1: Multimodal data input: Use cameras, depth cameras, and inertial measurement unit motion capture sensors to simultaneously capture hand movements, gesture trajectories, and facial expressions to form multimodal data input; Step 2: Sign language movement recognition: Accurately recognize sign language movements through a combination of a deep convolutional neural network (CNN) and a long short-term memory network (LSTM); Step 3: Combined recognition of facial expressions and gesture trajectories: By combining the captured facial expressions with gesture trajectories, the understanding and translation of complex sign sentences can be achieved; Step 4: Contextual natural language processing: Generate target sentences based on contextual semantic understanding; Step 5: The target sentence text is synthesized into speech through the TTS module, and the text subtitles are displayed synchronously.

2. The sign language recognition method based on multimodal deep learning according to claim 1, characterized in that: In step 1, the multimodal data includes hand skeleton joint coordinates, gesture trajectories, and facial expression key points.

3. The sign language recognition method based on multimodal deep learning according to claim 1, characterized in that: In step 2, the deep convolutional neural network (CNN) is a ResNet-50 derivative structure, which includes 5 residual modules, and the number of output channels of each residual module is 64, 128, 256, 512, and 1024 respectively.

4. The sign language recognition method based on multimodal deep learning according to claim 3, characterized in that: The specific process of step 2 is: Step 2-1, spatial feature extraction: Use deep convolutional neural network (CNN) to extract spatial features from the collected multimodal images and depth maps; Step 2-2, temporal information modeling: Use the long short-term memory network (LSTM) to process the time series data of hand movements and capture continuous sign language movements.

5. The sign language recognition method based on multimodal deep learning according to claim 4, characterized in that: The specific process of step 2-1 is: Assume that the input feature of the i-th layer is Xi∈R^{Ci×Hi×Wi}, and the convolution kernel is Wi∈R^{Ci+1×Ci×k×k}, then the convolution operation is: Y_{i+1}=ReLU(Wi*Xi+bi) Among them, bi is the bias and * represents the convolution operation.

6. The sign language recognition method based on multimodal deep learning according to claim 5, characterized in that: The specific process of step 2-2 is: Flatten the spatial feature vector of each frame output in step 2-1 into a D-dimensional vector sequence {f_t}_{t=1}^{T} and input it into a bidirectional long short-term memory network (Bi-LSTM). The number of hidden units in each direction of the Bi-LSTM is 256, and the total dimension is 512. The LSTM internal state update formula is: f_t=σ(W_fx_t+U_fh_{t-1}+b_f) i_t=σ(W_i x_t+U_i h_{t-1}+b_i) o_t=σ(W_o x_t+U_o h_{t-1}+b_o) ~c_t=tanh(W_c x_t+U_c h_{t-1}+b_c) Among them, W_* and U_* are parameters to be trained, σ is the Sigmoid activation function, is element-wise multiplication.

7. The sign language recognition method based on multimodal deep learning according to claim 4, characterized in that: The specific process of step 3 is as follows: Based on a multi-task deep convolutional neural network, the AUs value vector E∈R^m corresponding to the Facial Action Coding System (FACS) is output; the gesture trajectory extracts the trajectory feature vector G∈R^n, which is fused using the attention mechanism to calculate the attention weight: α=softmax(W_e E+W_g G+b) Z=α_e E+α_g G Among them, W_e∈R^{k×m},W_g∈R^{k×n}.

8. The sign language recognition method based on multimodal deep learning according to claim 5, characterized in that: The specific process of step 4 is as follows: For the identified continuous symbols and fused features Z, the Transformer-based encoder-decoder architecture is used to perform contextual semantic understanding and output the target language text. The Transformer is a medium-sized Transformer with a 6-layer encoder, 512 hidden dimensions, 8-head self-attention, and a 6-layer decoder. During multi-round reasoning, the dialogue history cache vector is combined to achieve context association through cross-attention layers.

9. A sign language recognition device based on multimodal deep learning, characterized in that: include: Data acquisition device: used to collect hand movements, gesture trajectories and facial expressions; Sign language recognition device: recognizes and translates the collected hand movements, gesture trajectories and facial expressions to obtain corresponding translation results; Recognition result output device: outputs the translation result in the form of text and / or voice for the communication partner to read or listen to.

10. A sign language recognition device based on multimodal deep learning, characterized in that: The data acquisition device includes an RGB camera, a depth camera and an inertial measurement unit (IMU) motion capture sensor.

Images