Sign language recognition and translation method based on lightweight neural network

By combining NCNN and Transformer models, combined with Kalman filter and position coding, the problems of high cost and dynamic range limitation in existing sign language recognition technology are solved, efficient and accurate sign language recognition and translation are achieved, and communication among hearing-impaired people is improved. Efficiency and quality of life.

CN119028019AInactive Publication Date: 2024-11-26SICHUAN UNIV

Patent Information

Application Number
CN202411158355.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing sign language recognition technology has problems such as high cost, motion fuzzy, dynamic range limitations and large model size, making it difficult to achieve efficient and accurate sign language recognition and translation.

Method used

Using a sign language recognition and translation method based on lightweight neural networks, the action data is extracted by denoising and standardizing the original video data, and combining NCNN network and Transformer model to perform local feature extraction and semantic translation recognition, combined with Kalman filtering and position coding, optimize the model structure to improve identification accuracy and efficiency.

Benefits of technology

It achieves the improvement of the accuracy and speed of sign language recognition at low cost and high efficiency, reduces the impact of motion blur, is suitable for various ambient lighting conditions, and improves the communication efficiency and quality of life of hearing-impaired people.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028019A_ABST
    Figure CN119028019A_ABST
Patent Text Reader

Abstract

The invention discloses a sign language recognition and translation method based on a lightweight neural network, and the method comprises the steps: carrying out the noise reduction and batch size standardization processing of original video data, and extracting the motion data in a continuous sign language video data set; carrying out filtering and normalization processing on the obtained action data; inputting the filtered and normalized data into an NCNN network, extracting local features of each action sequence to obtain a local feature vector S, and finally outputting a local feature vector sequence set omega = {S1, S2,... Sn}, n being the length of the sequence; adding position coding information to each local feature vector S of the generated local feature data set; the feature vector set added with the position coding information is input into a transformer model to form a CNN-transformer hybrid model, and the CNN-transformer hybrid model is used for carrying out translation recognition on sign language semantics; the invention aims to improve the translation accuracy, reduce the training cost and improve the recognition speed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of dynamic sign language action recognition, and in particular to a sign language recognition and translation method based on a lightweight neural network. Background Art

[0002] The initial gesture recognition method was to record and detect the information of each joint of the hand through wearable devices, such as data gloves. Data gloves generally have sensors for measuring movements. When the wearer makes gestures, the sensors will collect the motion data of the gestures, and then obtain the recognition results by designing algorithms and modeling and classifying the collected data. Grimes invented the first data glove for gesture recognition in 1983. Liang et al. used data gloves to collect and classify data of Chinese Taiwanese sign language, with a classification accuracy of 90.5%. However, due to the high cost of the recognition method of data gloves, dependence on hardware equipment, poor user experience, it is difficult to promote and use on a large scale, and the audience range is small. In the method of using camera recognition, there have been many achievements in the research of gesture recognition using traditional frame cameras. Among them, the gesture recognition method based on image processing mainly realizes gesture recognition through steps such as image preprocessing, feature extraction and classification. Common image processing techniques include edge detection, color segmentation, morphological operations, etc. By extracting the shape, color, texture and other features of the gesture, machine learning algorithms are used for classification and recognition. Fu Yonggang and others used two cameras to shoot hand gesture videos, and matched the image frames with pre-set gestures to perform gesture recognition. Camera-based visual methods are more in line with users' daily lives for gesture recognition, but traditional cameras have a small dynamic range, low data sampling rate, and high power consumption, which hinders the application of gesture recognition. In summary, traditional methods have problems such as high cost, motion blur, dynamic range limitation, and large model size. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the purpose of the present invention is to provide a sign language recognition and translation method based on a lightweight neural network, which aims to improve the translation accuracy while reducing the training cost and increasing the recognition speed.

[0004] To achieve the above object, the technical solution adopted by the present invention is: 1. A sign language recognition and translation method based on a lightweight neural network, comprising the following steps:

[0005] Step 1: Perform noise reduction and batch size normalization on the original video data, and extract the action data in the continuous sign language video dataset;

[0006] Step 2: Filter and normalize the obtained motion data;

[0007] Step 3: Input the filtered and normalized data into the NCNN network to extract the local features of each action sequence, obtain the local feature vector S, and finally output the local feature vector sequence set Ω = {S1, S2, ... S n}, where n is the length of the sequence;

[0008] Step 4: Add position coding information to each local feature vector S of the generated local feature data set;

[0009] Step 5: Input the feature vector set with position encoding information into the transformer model to form an NCNN-Transformer hybrid model for translation and recognition of sign language semantics.

[0010] As a further improvement of the present invention, in step 1, the action data includes 3D coordinate data of the hand and arm joints, specifically including three directional dimensions of x, y, and z, and the tensor is obtained. Among them, B represents the batch size of the video dataset, C in represents the dimension of the input tensor, T represents the length of the video frame, and N represents the number of joint points.

[0011] As a further improvement of the present invention, the step 2 is specifically as follows:

[0012] Filtering processing: Use Kalman filter to filter the collected sign language action sequence data, estimate and correct the parameters of the sign language action sequence data, and remove the noise of the sign language action sequence data;

[0013] Normalization processing uses the maximum and minimum normalization function to limit the value of each data in the filtered sign language action sequence to between [0, 1]. The function formula is: Among them, f is the original sign language action sequence data, max(f) is the maximum value in the original sign language action sequence, min(f) is the minimum value in the original sign language action sequence, and f * It is the normalized sign language action sequence data.

[0014] As a further improvement of the present invention, the parameters for estimating and correcting the sign language motion sequence data include process noise covariance (Q), measurement noise covariance (R), estimation error covariance (P) and initial estimation value (X); wherein, the process noise covariance (Q) is used to estimate the size of uncertainty introduced in the data generation process, the measurement noise covariance (R) is used to estimate the size of noise introduced in the actual measurement process, the estimation error covariance (P) represents the uncertainty of the estimation value, and the initial estimation value (X) represents a rough estimate of the initial state of the filter.

[0015] As a further improvement of the present invention, in step 3, the NCNN network first calculates the common part and the probability branch; if the result of the probability branch exceeds a preset threshold, the bounding box branch is calculated; if the result of the probability branch is lower than the threshold.

[0016] As a further improvement of the present invention, the step 4 is specifically as follows:

[0017] For the local feature vector sequence set Ω = {S1, S2, ..., S n}, the position of each local feature vector S in the sequence is represented by S t , i represents the i-th feature dimension at that position, S t (i) Represents the position encoding function of this dimension. The specific formula is:

[0018] S t (i) =PE (pos,k)

[0019]

[0020] Where k is an integer, d model Represents dimension, i is a positive integer; after obtaining the position code, it is added to the obtained local feature vector to achieve that each local feature vector S is added with the position code information.

[0021] As a further improvement of the present invention, in step 5, the transformer model includes an Encoder layer and a Decoder layer; the Encoder layer is composed of multiple blocks, each block is composed of a multi-head attention function and a feedforward fully connected layer connected in sequence, and the Chinese information is denoted by I∈R n×d , d represents the element dimension, I is divided into m parts and combined with the weight W i Multiply them together to form the input vector W i I, the query feature vector q that forms the multi-head attention function i ∈R n×d , key feature vector k i ∈R n×d and the eigenvalue vector v i ∈R n×d :

[0022] q i =W iq I, k i =W i k I, v i =W i v I, i = 1, ..., m

[0023] Among them, W i q , W i k , W i v They are generated by i , k i 、v i The corresponding linear projection of ;

[0024] Calculate the attention weight matrix a i =Attention(Q, K, V) i , a i ∈R n×n , using the improved self-attention function:

[0025]

[0026] φ(x)=φ(x)=e x ,in, and φ(x) are functions used to calculate self-attention, both defined as e x , which is the number of vectors in the matrix Q and the matrix K, Q i The exponential function of the transpose; finally, each a i Merge to form the output result b of the m-head attention mechanism,

[0027] b∈R n×d is the output of the multi-head attention module, d is the dimension of each element in I, Represents matrix multiplication;

[0028] The output result b is further processed by the feed-forward fully connected layer to form the semantic features of sign language.

[0029] As a further improvement of the present invention, the Decoder layer includes two multi-head attention functions; the first multi-head attention function adopts the Masked operation; the K and V matrices of the second multi-head attention function are calculated using the encoding information matrix of the Encoder, and Q is calculated using the output of the previous Decoder layer; finally, the exponential activation function e is used x Instead of the softmax function, the probability distribution of the sign language translation prediction value is output, and the prediction value with the highest probability is taken as the final sign language translation result.

[0030] As a further improvement of the present invention, the present invention further includes: evaluating the accuracy of sign language translation by the NCNN-Transformer hybrid model, using a multi-classification cross entropy loss function with a class balancing weight factor α as the loss function, comparing the sign language translation prediction value with the true value of the label, calculating the multi-classification cross entropy loss function value with the class balancing weight factor α, and continuously iterating training until the loss function value reaches the optimal value.

[0031] As a further improvement of the present invention, the loss function is as follows:

[0032]

[0033] Where n is the number of training set samples, m is the number of categories, and y tj is the unique hot encoding of the true label of each sign language action sequence, is the probability predicted by the model, α j is the weight factor for category j, α j ∈[0, 1].

[0034] The beneficial effects of the present invention are:

[0035] 1. The present invention solves the problems of high cost, motion blur, dynamic range limitation and large model size in traditional methods. In the present invention, a neuromorphic visual sensor DVS can be specifically used as an acquisition device to record gesture events through pixel-level changes. The unique recording method allows it to quickly capture subtle changes in gestures, thereby reducing the impact of motion blur. At the same time, its high dynamic range reduces the requirements for ambient lighting in application scenarios, allowing it to accurately identify under various ambient lighting conditions. At the same time, the good compatibility of neuromorphic data combined with neuromorphic neural networks SNNs means that it does not require a complex network structure to have extremely high computational efficiency when processing and learning data, making it easy to deploy on mobile devices.

[0036] 2. The present invention can improve the communication efficiency between the hearing-impaired and the hearing-impaired, improve their quality of life, and help them integrate into society. Under the premise of protecting the privacy rights of the hearing-impaired, a way for barrier-free communication with others is provided for them, thereby solving a series of existing problems caused by communication barriers for the hearing-impaired, and helping the hearing-impaired to better express themselves and integrate into society. The sign language of the deaf-mute is efficiently and real-timely converted into text sentences through instant recognition and accurate conversion of sign language. The real-time translation of sign language allows the deaf-mute to quickly explain the situation of themselves or their companions to the doctor in an emergency, such as at a medical site, so that the doctor can make a better decision. In the workplace, they can express their ideas independently and accurately without going through others, weakening employment restrictions while promoting their own career development. It not only facilitates personal life and enhances personal self-confidence, but also can weaken the social prejudice against the deaf-mute population to a certain extent, and reduce the contradictions caused by communication problems between the deaf-mute population and other parts of society. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of an embodiment of the present invention;

[0038] Figure 2 Schematic diagram of the model structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0040] Example 1

[0041] like Figure 1 and Figure 2 As shown, a sign language recognition and translation method based on a lightweight neural network includes:

[0042] 1. Data preprocessing:

[0043] After denoising and batch size normalization of the original video data, the Mediapipe framework tool is used to extract the 3D coordinate data of the hand and arm joints in the continuous sign language video dataset, including the three dimensions of x, y, and z, to obtain the tensor Among them, B represents the batch size of the video dataset, C in represents the dimension of the input tensor, T represents the length of the video frame, and N represents the number of joint points.

[0044] 2. Filtering and normalization processing:

[0045] The obtained motion data is filtered and normalized, including:

[0046] Filtering: Use Kalman filter to filter the collected sign language motion sequence data, estimate and correct the parameters of the sign language motion sequence data, so as to remove the noise of the sign language motion sequence data. The parameters of the sign language motion sequence data are estimated and corrected, including the process noise covariance (Q), measurement noise covariance (R), estimation error covariance (P) and initial estimation value (X0), which are set to 0.2, 10, 10 and 4 respectively.

[0047] Normalization: Use the maximum and minimum normalization function to limit the value of each data in the filtered sign language action sequence to between [0,1]. The function formula is:

[0048]

[0049] f is the original sign language action sequence data, max(f) is the maximum value in the original sign language action sequence, min(f) is the minimum value in the original sign language action sequence, and f * It is the normalized sign language action sequence data.

[0050] 3.NCNN network:

[0051] NCNN (Nvidia Convolutional Neural Network) is an efficient deep learning framework optimized for embedded and mobile devices, designed to provide fast and low-power reasoning capabilities. It is widely used in tasks such as image processing, object detection, and feature extraction, and is particularly suitable for resource-constrained device environments.

[0052] The specific design of the NCNN network structure is as follows:

[0053] Input Layer:

[0054] The input dimension is [H, W, C], where H and W are the height and width of the input feature map, and C is the number of channels. After filtering and normalization, the input data is input to the NCNN network to extract the local features of each action sequence, obtain the vector S, and finally output the local feature vector sequence set Ω = {S1, S2, ..., Sn}.

[0055] First convolutional layer:

[0056] Number of convolution kernels: 32; convolution kernel size: 3×3; activation function: ReLU; The first convolution layer performs convolution operation through 32 3×3 convolution kernels to extract local features in the input feature map. The expression of the ReLU activation function is f(x)=max(0,x), which is used to introduce nonlinear characteristics.

[0057] First pooling layer:

[0058] Pooling method: maximum pooling; the pooling window is 2×2; the maximum pooling operation downsamples through a 2×2 pooling window to reduce the size of the feature map while retaining the most important features.

[0059] Second convolutional layer:

[0060] Number of convolution kernels: 64; Convolution kernel size: 3×3; Activation function: ReLU; The second convolution layer further extracts features through 64 3×3 convolution kernels and applies the ReLU activation function.

[0061] Second pooling layer:

[0062] Pooling method: maximum pooling; Pooling window: 2×2; Similar to the first pooling layer, downsampling is performed through a 2×2 pooling window.

[0063] The third convolutional layer:

[0064] Number of convolution kernels: 128; Convolution kernel size: 3×3; Activation function: ReLU; The third convolution layer uses 128 3×3 convolution kernels to continue to extract more advanced features and applies the ReLU activation function.

[0065] The third pooling layer:

[0066] Pooling method: maximum pooling; Pooling window: 2×2; Downsampling is performed through a 2×2 pooling window to further reduce the size of the feature map.

[0067] Fourth convolutional layer:

[0068] Number of convolution kernels: 256; Convolution kernel size: 3×3; Activation function: ReLU; The fourth convolution layer uses 256 3×3 convolution kernels to perform convolution operations, extract high-level features, and apply the ReLU activation function.

[0069] Fourth pooling layer:

[0070] Pooling method: maximum pooling; pooling window: 2×2; downsampling is performed through a 2×2 pooling window to obtain the final feature map.

[0071] Dropout layer:

[0072] Dropout rate: 0.25; to prevent overfitting, a Dropout layer is added and 25% of the neurons are discarded; the ReLU activation function expression is: f(x) = max(0,x).

[0073] Multi-branch structure and optimization strategy:

[0074] NCNN supports complex network structures with multiple branches and multiple inputs. During the calculation process, the common part and the probability branch (prob branch) can be calculated first according to the needs. If the result of the probability branch exceeds the preset threshold, the bounding box branch (bbox branch) is calculated. If the result of the probability branch is lower than the threshold, the calculation of the bounding box branch is skipped, thereby reducing the amount of calculation and improving the calculation efficiency.

[0075] Optimization strategy implementation steps:

[0076] 1. Determine the network structure: Check the network definition file or model document to clarify the network structure. If you are using a statically defined network structure file (such as .prototxt), you need to add a conditional layer or related logic control layer to the model definition.

[0077] 2. Implement conditional calculation logic: write a custom layer or modify the configuration file, extract the result of the prob branch from the model output, and decide whether to calculate the output of the bbox branch based on its result.

[0078] 3. Modify the inference code: By modifying the inference code, the conditional calculation logic based on the prob branch results is implemented to ensure the performance and accuracy of the optimized model.

[0079] 4. Verification and testing: Verify the effectiveness of the optimization strategy and the accuracy of the model through a series of experiments and tests.

[0080] Through these steps, we can ensure that these branch functions are correctly defined and implemented in the network model, and realize conditional calculation based on the prob branch results, thereby optimizing the reasoning process of the model and improving efficiency.

[0081] The NCNN network, with its efficient convolutional and pooling layer designs, combined with a flexible multi-branch structure, performs well on resource-constrained embedded and mobile devices. Through a reasonable optimization strategy, the amount of computation can be greatly reduced while ensuring model performance, and the reasoning speed can be improved, making it suitable for application scenarios with high real-time requirements.

[0082] 4. Positional encoding:

[0083] Add position encoding information to each local feature vector S of the generated local feature data set. n}, whose length is n, and the position of each local feature vector S in the sequence is represented by S t , i represents the i-th feature dimension at that position, S t (i) Represents the position encoding function of this dimension. The specific formula is:

[0084] S t(i) =PE (pos,k)

[0085]

[0086] Where k is an integer, d model Represents the dimension. After this calculation, the position code can be obtained, and then added to the previously obtained feature vector to achieve that each feature vector S is added with the position code information.

[0087] 5. Overview of the basic architecture of the Transformer model:

[0088] The Transformer model mainly consists of the Encoder layer and the Decoder layer. In this proposal, we focus on the improvement of the Encoder layer. The Encoder layer contains 6 identical blocks, each of which consists of a multi-head attention mechanism (MHA) and a feedforward fully connected layer (FFN).

[0089] Input feature vector:

[0090] Input the feature vector set with position encoding information into the Transformer model. Let the input information be I∈R n×d , n represents the number of elements, d represents the element dimension. Divide I into m parts and add them to the weight matrix W i Multiply to form the input vector W i I, generate the query feature vector Q, key feature vector K and eigenvalue vector V of MHA.

[0091] Q i =W iq I, K i =W ik I, V i =W iv I, (i=1,...,m)

[0092] The improved self-attention function is used to calculate the attention weight matrix:

[0093] a i =Attention(Q,K,V) i

[0094]

[0095] The calculation method of the Attention function has been improved. Specifically, a new self-attention function φ(x)=e x , and uses the exponential function to calculate the attention weight, thereby enhancing the model's ability to capture local features.

[0096] Multi-head attention mechanism output:

[0097] The output b of the multi-head attention mechanism is formed by merging the outputs of multiple attention heads. The output of each head is calculated as follows:

[0098]

[0099] Among them, W iq , W ik , W iv are the linear projection matrices that generate Q, K, and V respectively. b∈R n×d is the output of the multi-head attention module, d is the dimension of each element in I, Represents matrix multiplication. This layer obtains the correlation between local features, namely global features.

[0100] Feedforward Fully Connected Layer (FFN):

[0101] The output result b of the multi-head attention mechanism is processed by the feed-forward fully connected layer. FFN contains two fully connected layers. The activation function of the first layer is ReLU, and the second layer does not use an activation function. The final output dimension is 32.

[0102] Residual connection and layer normalization (Add&Norm):

[0103] After the MHA and FFN layers, there is an Add&Norm layer. The Add part represents the residual connection, which is used to prevent network degradation and speed up network training; the Norm part represents layer normalization, which is used to normalize the activation values ​​of each layer to speed up model convergence. The calculation formula of the Add&Norm layer is as follows:

[0104] LayerNorm(X+SubLayer(X)); where SubLayer represents the MultiHeadAttention or FeedForward layer.

[0105] Improvement of self-attention function: Introducing a new self-attention function φ(x) = e x , enhance the ability to capture local features. Optimized Add&Norm layer: Improved residual connection and layer normalization methods, so that the model maintains stability and convergence speed while increasing depth.

[0106] Through these improvements, the model has stronger capabilities in processing Chinese information and generating sign language semantic features, effectively improving the ability to capture the correlation between features, thereby improving the performance of the overall model.

[0107] 6. Connection and extended description of NCNN and Transformer models:

[0108] In modern deep learning frameworks, combining the advantages of different models can significantly improve the performance and efficiency of the system. Here, we explore the combination of NCNN (Nvidia Convolutional Neural Network) and Transformer models to fully utilize the advantages of both in feature extraction and sequence modeling, thereby optimizing the task of processing Chinese information and generating sign language semantic features.

[0109] How NCNN connects to Transformer:

[0110] First, as an efficient convolutional neural network, NCNN is mainly used to extract local features of input images or feature maps. Its network structure includes multiple convolutional layers and pooling layers, which generate high-dimensional feature vectors through layer-by-layer convolution and downsampling. Specifically, the input feature map is processed by multiple convolutional layers and pooling layers, and the resulting feature vector set Ω = {S1, S2, ..., S n}, these feature vectors contain rich information of the input data.

[0111] Next, these high-dimensional feature vectors extracted by NCNN will be used as input to the Transformer model. The Transformer model performs well in processing sequence data with its excellent sequence modeling ability and attention mechanism. In particular, its Encoder layer effectively captures the global correlation between input feature vectors through the multi-head attention mechanism (MHA) and the feedforward fully connected layer (FFN).

[0112] The specific process is as follows:

[0113] 1. Feature extraction: Use NCNN to extract a set of feature vectors Ω from the input data.

[0114] 2. Feature encoding: Positionally encode the feature vector set Ω extracted by NCNN to generate a feature vector containing position information.

[0115] 3. Input Transformer: Input the position-encoded feature vector into the Encoder layer of Transformer.

[0116] TransformerEncoder layer improvements:

[0117] The Encoder layer of the Transformer model contains 6 identical blocks, each of which consists of a multi-head attention mechanism (MHA) and a feed-forward fully connected layer (FFN). In order to enhance the ability to capture local features, the self-attention function is improved. Specifically, the new self-attention function φ(x) = e x, and uses the exponential function to calculate the attention weight, thereby enhancing the model's ability to capture local features.

[0118] In addition, the residual connection and layer normalization of the Add&Norm layer are optimized to further improve the stability and convergence speed of the model. After each MHA and FFN layer, an improved Add&Norm layer is used, so that the model can maintain efficient training speed and stability while increasing in depth.

[0119] Connection method and model advantages:

[0120] By combining NCNN with the Transformer model, we can take full advantage of the advantages of both in feature extraction and sequence modeling. The convolutional layer of NCNN is good at extracting local features of images or feature maps, while the multi-head attention mechanism of Transformer can effectively capture the global correlation of sequence data.

[0121] This connection method performs particularly well in processing complex language and visual tasks. For example, in processing Chinese information and generating sign language semantic features, the high-dimensional feature vector extracted by NCNN can capture the rich details of the input data, while the Transformer model can model the relationship between these features in a global scope, thereby generating more accurate and natural semantic features.

[0122] By combining NCNN and Transformer models, we can build an efficient and powerful deep learning system that significantly improves the ability to handle complex language and visual tasks. NCNN provides powerful feature extraction capabilities, while the Transformer model enhances the ability to capture the correlation between features through an improved self-attention mechanism and optimized layer normalization method. This combination not only improves the overall performance of the model, but also provides a new solution for processing multimodal data.

[0123] 7.Decoder layer:

[0124] The Decoder layer is similar to the Encoder layer, but there are some differences:

[0125] It contains two MHA layers. The first MHA layer uses the Masked operation, and the K and V matrices of the second MHA layer are calculated using the encoding information matrix of the Encoder, while Q is calculated using the output of the previous Decoder.

[0126] Finally, we use the exponential activation function e x Instead of the softmax function, the probability distribution of the sign language translation prediction value is output, and the prediction value with the highest probability is taken as the final sign language translation result.

[0127] The first Multi-HeadAttention of the decoder block uses the Masked operation, as follows: the mask matrix is ​​an n×n matrix, where n is the dimension of each feature vector of the decoder layer input matrix. For each word in the mask matrix, the words after the current word are blocked, that is, only the word information before the current word can be used in the calculation. The following operation is the same as the previous Self-Attention ion. The Q, K, and V matrices are calculated by inputting the matrix X. Then the product QKT of Q and KT is calculated. After obtaining QKT, the Mask matrix is ​​used to block the information after each word, that is, the QKT matrix is ​​multiplied by the mask matrix bit by bit, and then the self-attention function Attention(Q,K,V) is applied to calculate the attentionscore, and finally the output matrix Z is obtained.

[0128] The second Multi-HeadAttention of the Decoderblock does not change much. The main difference is that the K and V matrices of the Self-Attention are not calculated using the output of the previous Decoder block, but using the Encoder's encoding information matrix C. K and V are calculated based on the Encoder's output C, and Q is calculated based on the output Z of the previous Decoder block (if it is the first Decoder block, the input matrix X is used for calculation). The subsequent calculation method is the same as described above.

[0129] 8. Loss Function:

[0130] In order to evaluate the accuracy of the trained CNN-Transformer hybrid model for sign language translation, the loss function used in this embodiment is a multi-classification cross entropy loss function with a class balance weight factor α, and the formula is as follows:

[0131]

[0132] Where n is the number of training set samples, m is the number of categories, and y tj is the unique hot encoding of the true label of each sign language action sequence, is the probability predicted by the model, α j is the weight factor for category j, α j ∈[0, 1].

[0133] 9. Model Validation:

[0134] The model was further verified using the CSL dataset, and a 3-fold cross-validation without repeated sampling and a grid search strategy were used to find the optimal hyperparameter settings. Finally, a lightweight neural network sign language recognition and translation model with the best fitting effect was obtained.

[0135] This embodiment introduces a lightweight analysis method combining NCNN and Transformer models in the field of sign language recognition and translation, aiming to improve translation accuracy while reducing training costs and increasing recognition speed.

[0136] The LS-HAN continuous sign language recognition framework proposed by USTC has a maximum accuracy of 82.7% when tested on the CSL-500 isolated sign language classification dataset. The Bi-LSTM model published in the Journal of Wuhan University (Science Edition) has a maximum test accuracy of 84.78% on the CSL-500 dataset. Compared with these prior arts, the accuracy of the model of the present invention on the CSL isolated sign language classification dataset is increased to 93%, while the training cost is reduced by more than half and the analysis speed is more than doubled.

[0137] This significant improvement not only demonstrates the efficiency and accuracy advantages of this embodiment in processing continuous sign language action sequences, but also highlights its potential value in actual sign language translation applications.

[0138] Example 2

[0139] Hardware equipment: optical motion capture camera (for collecting continuous sign language motion data), computer (for motion analysis and display of translated text information)

[0140] Software equipment: data processing module (used to filter, reduce noise and standardize the collected data), deep learning model (used to extract sign language movement features and predict text information), and result output module (used to return the results to the display).

[0141] plan:

[0142] 1. In a professional recording room, a sign language teacher demonstrates sign language movements, uses a camera to record sign language movements, and manually labels the collected samples.

[0143] 2. Use the mediapipe framework tool to extract the 3D coordinate data of the hand and arm joints in the continuous sign language video dataset, including the three dimensions of x, y, and z, to obtain the tensor Among them, B represents the batch size of the video dataset, C in represents the dimension of the input tensor, T represents the length of the video frame, and N represents the number of joint points.

[0144] 3. Filter and normalize the obtained motion data, including:

[0145] Filtering: Use Kalman filter to filter the collected sign language motion sequence data, estimate and correct the parameters of the sign language motion sequence data, so as to remove the noise of the sign language motion sequence data. The parameters of the sign language motion sequence data are estimated and corrected, including the process noise covariance (Q), measurement noise covariance (R), estimation error covariance (P) and initial estimate value (X), which are set to 0.2, 10, 10 and 4 respectively. Among them, the process noise covariance (Q) is used to estimate the size of the uncertainty introduced in the data generation process, the measurement noise covariance (R) is used to estimate the size of the noise introduced in the actual measurement process, the estimation error covariance (P) represents the uncertainty of the estimated value, and the initial estimate value (X) represents a rough estimate of the initial state of the filter.

[0146] Normalization processing uses the maximum and minimum normalization function to limit the value of each data in the filtered sign language action sequence to between [0,1]. The function formula is:

[0147]

[0148] Among them, f is the original sign language action sequence data, max(f) is the maximum value in the original sign language action sequence, min(f) is the minimum value of the original sign language action sequence, and f* is the normalized sign language action sequence data.

[0149] 4. The filtered and normalized data is input into the NCNN network to extract the local features of each action sequence, obtain the vector S, and finally output the local feature vector sequence set Ω = {S1, S2, ..., S n}.

[0150] The specific NCNN network design is as follows:

[0151] -The input dimension is [H,W,C], where H and W are the height and width of the input feature map, and C is the number of channels.

[0152] -First convolutional layer: 32 3x3 convolution kernels, using ReLU activation function.

[0153] -First pooling layer: max pooling with a pooling window of 2x2.

[0154] -Second convolutional layer: 64 3x3 convolution kernels, using ReLU activation function.

[0155] -Second pooling layer: max pooling with a pooling window of 2x2.

[0156] -The third convolutional layer: 128 3x3 convolution kernels, using the ReLU activation function.

[0157] -Third pooling layer: max pooling with a pooling window of 2x2.

[0158] -The fourth convolutional layer: 256 3x3 convolution kernels, using the ReLU activation function.

[0159] -Fourth pooling layer: maximum pooling with a pooling window of 2x2.

[0160] -Dropout layer: The dropout rate is 0.25.

[0161] The expression of the ReLU activation function is: f(x)=max(0,x).

[0162] In order to optimize the calculation, NCNN supports complex network structures with multiple branches and multiple inputs. During the calculation process, the common part and the probability branch (prob branch) can be calculated first according to the needs. If the result of the probability branch exceeds the preset threshold, the bounding box branch (bbox branch) is calculated; if the result of the probability branch is lower than the threshold, the calculation of the bounding box branch is skipped, thereby reducing the amount of calculation and improving the calculation efficiency.

[0163] In practical applications, the input data is first processed using the above NCNN network, and then the output is input into the Transformer model for processing. The connection between the two is very important, and it is necessary to ensure that the output of the NCNN network can be seamlessly processed as the input of the Transformer model. This combination makes full use of existing technologies, extracting features through the NCNN network and inputting them into the Transformer model, thereby achieving more efficient processing.

[0164] 5. Add position encoding information to each local feature vector S of the generated local feature data set. n}, whose length is n, and the position of each local feature vector S in the sequence is represented by S t , i represents the i-th feature dimension at that position, S t (i) Represents the position encoding function of this dimension. The specific formula is:

[0165] S t (i) =PE (pos,k)

[0166]

[0167]

[0168] Where k is an integer, dmodel represents the dimension, i = 1, 2, ..., and the position code can be obtained through this calculation, and then added to the feature vector obtained previously, so that each feature vector S is added with the position code information.

[0169] 6. Input the feature vector set with position encoding information into the transformer model. Transformer is divided into Encoder layer and Decoder layer.

[0170] Specifically, the Encoder layer consists of 6 blocks, each of which is composed of a multi-head attention function (MHA) and a feedforward fully connected layer (FFN) connected in sequence. Let the Chinese information be I∈R n×d , n represents the number of elements, d represents the element dimension, I is divided into m parts, and

[0171] Weight W i Multiply them together to form the input vector W i I, query feature vector q that forms MHA i ∈R n×d , key feature vector k i ∈R n×d , and the eigenvalue vector v i ∈R n×d :

[0172] q i =W i q I,k i =W i k I,v i =W i v I,i=1,…,m

[0173] Calculate the attention weight matrix a i =Attention(Q,K,V) i ,a i ∈R n×n , using the improved self-attention function:

[0174]

[0175] in, and φ(x) are functions used to calculate self-attention, both defined as e x , n is the number of elements in the sequence, that is, the number of vectors in the matrix Q and the matrix K, for q i The exponential function of i Merge to form the output result b of the m-head attention mechanism,

[0176] Wi q 、Wi k 、Wi v They are generated by i , k i and v i The corresponding linear projection of b∈R n×d is the output of the multi-head attention module, d is the dimension of each element in I, Represents matrix multiplication. This layer obtains the correlation between local features, namely global features.

[0177] The output result b is further processed by FFN to form the sign language semantic features. FFN includes two fully connected layers. The activation function of the first layer is ReLU, and the second layer does not use an activation function. The final output dimension is 32.

[0178] Both the MHA and FFN layers include an Add&Norm layer. Add stands for residual connection, which is used to prevent network degradation, and Norm stands for Layer Normalization, which is used to normalize the activation values ​​of each layer. The Add&Norm layer consists of two parts, Add and Norm, and its calculation formula is as follows:

[0179] LayerNorm(X+MultiHeadAttention(X))

[0180] LayerNorm(X+FeedForward(X))

[0181] Where X represents the input of MHA or FFN, MultiHeadAttent ion(X) and FeedForward(X) represent the output (the output and input X have the same dimension, so they can be added). Add refers to X+MultiHeadAttent ion(X), which is a residual connection used to solve the problem of multi-layer network training. It allows the network to only focus on the current difference, reduce model complexity, and speed up network training. Norm refers to Layer Normalization. Layer Normalization converts the input of each layer of neurons into vectors with the same mean and variance, which can speed up model convergence.

[0182] 7. The Decoder layer is similar to the Encoder layer, but there are some differences:

[0183] Contains two MHA layers.

[0184] The first MHA layer uses the Masked operation.

[0185] The K, V matrices of the second MHA layer are calculated using the encoded information matrix of the Encoder, while Q is calculated using the output of the previous Decoder.

[0186] Finally, we use the exponential activation function e x Instead of the softmax function, the probability distribution of the sign language translation prediction value is output, and the prediction value with the highest probability is taken as the final sign language translation result.

[0187] The first Multi-Head Attention of the decoder block uses the Masked operation, as follows: the mask matrix is ​​an n×n matrix, where n is the dimension of each feature vector of the decoder layer input matrix. For each word in the mask matrix, the words after the current word are masked, that is, only the word information before the current word can be used in the calculation. The following operation is the same as the previous Self-Attention. The Q, K, and V matrices are calculated by inputting the matrix X. Then Q and K are calculated. T The product of QK T . After getting QK T Then use the Mask matrix to mask the information after each word, that is, use QK T The matrix is ​​bitwise multiplied with the mask matrix, and then the self-attention function Attention(Q,K,V) is applied to calculate the attention score, and finally the output matrix Z is obtained.

[0188] The second Multi-Head Attention of the Decoder block does not change much. The main difference is that the K and V matrices of the Self-Attention are not calculated using the output of the previous Decoder block, but using the Encoder's encoding information matrix C. K and V are calculated based on the Encoder's output C, and Q is calculated based on the output Z of the previous Decoder block (if it is the first Decoder block, the input matrix X is used for calculation). The subsequent calculation method is the same as described above.

[0189] 8. In order to evaluate the accuracy of the trained CNN-Transformer hybrid model for sign language translation, the loss function used in the present invention is a multi-classification cross entropy loss function with a class balance weight factor α. The predicted value of the sign language translation is compared with the true value of the label, and the multi-classification cross entropy loss function value of the class balance weight factor α is calculated. The training is iterated continuously until the loss function value reaches the optimal value. The specific loss function formula is as follows:

[0190]

[0191] Where n is the number of training set samples, m is the number of categories, and y tj is the unique hot encoding of the true label of each sign language action sequence, is the probability predicted by the model, α j is the weight factor for category j, α j ∈[0,1]. Using multi-classification cross entropy with class balance weight factor α as loss function can solve the degradation of model training performance caused by imbalance of positive and negative samples, and can also achieve the acceleration effect of model training and recognition. Finally, the CSL dataset is used to further verify the model, and 3-fold cross validation without repeated sampling and grid search strategy are used to find the best hyperparameter setting. After further training and improvement, the lightweight neural network sign language recognition and translation model with the best fitting effect is finally obtained.

[0192] The above-mentioned embodiments only express the specific implementation of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A sign language recognition and translation method based on lightweight neural network, It is characterized in that The following steps are involved: Step 1: Perform noise reduction and batch size normalization on the original video data, and extract the action data in the continuous sign language video dataset; Step 2: Filter and normalize the obtained motion data; Step 3: Input the filtered and normalized data into the NCNN network to extract the local features of each action sequence, obtain the local feature vector S, and finally output the local feature vector sequence set Ω = {S 1 , S 2 ,…S n }, where n is the length of the sequence; Step 4: Add position coding information to each local feature vector S of the generated local feature data set; Step 5: Input the feature vector set with position encoding information into the transformer model to form an NCNN-Transformer hybrid model for translation and recognition of sign language semantics.

2. The sign language recognition and translation method based on a lightweight neural network according to claim 1, It is characterized in that In step 1, the motion data includes 3D coordinate data of the hand and arm joints, specifically including the three dimensions of x, y, and z, and the tensor Among them, B represents the batch size of the video dataset, C in represents the dimension of the input tensor, T represents the length of the video frame, and N represents the number of joint points.

3. The sign language recognition and translation method based on lightweight neural network according to claim 1, It is characterized in that The step 2 is specifically as follows: Filtering processing: Use Kalman filter to filter the collected sign language action sequence data, estimate and correct the parameters of the sign language action sequence data, and remove the noise of the sign language action sequence data; Normalization processing uses the maximum and minimum normalization function to limit the value of each data in the filtered sign language action sequence to between [0, 1]. The function formula is: Among them, f is the original sign language action sequence data, max(f) is the maximum value in the original sign language action sequence, min(f) is the minimum value in the original sign language action sequence, and f * It is the normalized sign language action sequence data.

4. The sign language recognition and translation method based on lightweight neural network according to claim 3, It is characterized in that The parameters for estimating and correcting the sign language action sequence data include process noise covariance (Q), measurement noise covariance (R), estimation error covariance (P) and initial estimation value (X); wherein, process noise covariance (Q) is used to estimate the size of uncertainty introduced in the data generation process, measurement noise covariance (R) is used to estimate the size of noise introduced in the actual measurement process, estimation error covariance (P) represents the uncertainty of the estimation value, and initial estimation value (X) represents a rough estimate of the initial state of the filter.

5. The sign language recognition and translation method based on lightweight neural network according to claim 1, It is characterized in that In step 3, the NCNN network first calculates the common part and the probability branch; If the result of the probability branch exceeds the preset threshold, the bounding box branch is calculated; if the result of the probability branch is lower than the threshold.

6. The sign language recognition and translation method based on lightweight neural network according to claim 5, It is characterized in that The step 4 is specifically as follows: For the local feature vector sequence set Ω={S 1 , S 2 , …, S n }, the position of each local feature vector S in the sequence is represented by S t , i represents the i-th feature dimension at that position, S t (i) Represents the position encoding function of this dimension. The specific formula is: S t (i) =PE (pos,k) Where k is an integer, d model Represents dimension, i is a positive integer; after obtaining the position code, it is added to the obtained local feature vector to achieve that each local feature vector S is added with the position code information.

7. The sign language recognition and translation method based on a lightweight neural network according to claim 6, It is characterized in that In step 5, the transformer model includes an encoder layer and a decoder layer; the encoder layer is composed of multiple blocks, each of which is composed of a multi-head attention function and a feedforward fully connected layer connected in sequence. Let the Chinese information be I∈R n×d , d represents the element dimension, I is divided into m parts and combined with the weight W i Multiply them together to form the input vector W i I, the query feature vector q that forms the multi-head attention function i ∈R n×d , key feature vector k i ∈R n×d and the eigenvalue vector v i ∈R n×d : q i =W i q I,k i =W i k I,v i =W i v I,i=1,…,m Among them, W i q , W i k , W i v They are generated by i , k i 、v i The corresponding linear projection of ; Calculate the attention weight matrix a i =Attention(Q, K, V) i , a i ∈R n×n , using the improved self-attention function: in, and φ(x) are functions used to calculate self-attention, both defined as e x , which is the number of vectors in the matrix Q and the matrix K, Q i The exponential function of the transpose; finally, each a i Merge to form the output result b of the m-head attention mechanism, b∈R n×d is the output of the multi-head attention module, d is the dimension of each element in I, Represents matrix multiplication; The output result b is further processed by the feed-forward fully connected layer to form the semantic features of sign language.

8. The sign language recognition and translation method based on lightweight neural network according to claim 6, It is characterized in that The Decoder layer includes two multi-head attention functions; the first multi-head attention function uses the Masked operation; the K and V matrices of the second multi-head attention function are calculated using the Encoder's encoding information matrix, while Q is calculated using the output of the previous Decoder layer; finally, the exponential activation function e is used x Instead of the softmax function, the probability distribution of the sign language translation prediction value is output, and the prediction value with the highest probability is taken as the final sign language translation result.

9. The sign language recognition and translation method based on a lightweight neural network according to any one of claims 1 to 8, It is characterized in that Also includes: The accuracy of sign language translation of the NCNN-Transformer hybrid model is evaluated. The multi-class cross entropy loss function with class balancing weight factor α is used as the loss function. The predicted value of sign language translation is compared with the true value of the label. The value of the multi-class cross entropy loss function with class balancing weight factor α is calculated, and the training is iterated continuously until the loss function value reaches the optimal value.

10. The sign language recognition and translation method based on lightweight neural network according to claim 9, It is characterized in that The loss function is as follows: Where n is the number of training set samples, m is the number of categories, and y tj is the unique hot encoding of the true label of each sign language action sequence, is the probability predicted by the model, α j is the weight factor for category j, α j ∈[0, 1].

Citation Information

Patent Citations

  • Continuous sign language recognition method and device

    CN115393949A

  • Transform sign language segmentation method based on human skeleton input

    CN117058750A

  • Intelligent blind guiding and obstacle avoiding cap for visually impaired people

    CN117598857A

  • CNN-Transform hybrid model-based sign language translation method and system

    CN117975573A

Cited By

  • ConvLSTM model-based sign language recognition method and system

    CN119964250A