A sign language recognition method and system

Through the combined use of hierarchical long and short-term memory network and encoder-decoder structure, the problem that fixed-length feature vectors cannot represent variable-length videos in continuous sign language recognition is solved, and efficient spatio-temporal feature extraction and inter-word dependency modeling is achieved. , improves the accuracy and computing efficiency of sign language recognition.

CN111340005BActive Publication Date: 2024-11-08SHENZHEN KANGHONGTAI TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202010301154.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-04-16
Publication Date
2024-11-08
Estimated Expiration
2040-04-16

AI Technical Summary

Technical Problem

When the existing continuous sign language recognition technology processes variable-length sign language videos, fixed-length feature vectors cannot effectively represent video information, and the hard alignment method requires artificial annotation, which is heavy workload, and assumptions are independent between the output words and words cannot be Effectively handles cases where the input is longer than the output.

Method used

The spatial and temporal characteristics of sign language videos are extracted using a hierarchical long and short-term memory network, and the sign language recognition model is optimized to avoid the independent relationship between output words and adapt to sign language. Identified application scenarios.

Benefits of technology

On the premise of ensuring that the recognition accuracy does not decrease, the calculation speed is improved, key gesture information is automatically extracted, and the feature representation of variable-length sign language videos is adapted to solve the problem of independent relationship between output words and improve the recognition effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111340005B_ABST
    Figure CN111340005B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides a sign language recognition method and system. After layering long short-term memory networks, a maximum pooling operation is performed to obtain key information of the current window. The length of the network is gradually changed from long to short, so as to effectively obtain the spatiotemporal characteristics of the sign language video and extract the intrinsic connection of the video from different levels. The key gesture information in the current video can be automatically extracted. Under the premise of not reducing the network accuracy, the network is compressed to improve the calculation speed. A sign language recognition model based on an encoder-decoder structure and a recurrent neural network jointly trained based on a CTC loss function is jointly optimized and trained. The network avoids the independent relationship between output words and output words, which is more in line with the application scenarios of sign language recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a sign language recognition method and a sign language recognition system. Background Art

[0002] Sign language is the main tool for communication between deaf-mute people. Sign language recognition is complex and changeable. Therefore, it is difficult for people who have not systematically learned sign language to learn sign language. In order to promote effective communication between deaf-mute people and society, the invention of visual sign language technology is crucial. According to the characteristics of sign language video recognition, sign language recognition is divided into two types: isolated sign language recognition technology and continuous sign language recognition technology. For isolated sign language recognition, the video contains only one sign language word, so this type of problem can be regarded as a classification problem. In continuous sign language recognition, one video corresponds to one sentence, so it cannot be simply regarded as a recognition problem. This is a mapping process between two continuous sequences, and the process is more complicated in the recognition task. In terms of evaluation indicators, it cannot be evaluated according to the accuracy rate in isolated sign language recognition.

[0003] At present, many scholars have devoted themselves to the research of continuous sign language recognition and have formed many mature methods. These methods are divided into two parts: sign language recognition methods based on soft alignment and sign language recognition methods based on hard alignment. Early sign language recognition methods based on hard alignment mainly use hidden Markov models or conditional random fields. This method requires manual annotation of all frames of the video in advance, which is particularly labor-intensive.

[0004] In actual use, the length of sign language video is not fixed. It is possible that the number of sign language video frames is greater than the number of corresponding text words. The longer video is encoded into a feature vector of fixed length and decoded sequentially. The feature vector cannot represent the video information well. Summary of the invention

[0005] In view of the above problems, embodiments of the present invention are proposed to provide a sign language recognition method and a corresponding sign language recognition system that overcome the above problems or at least partially solve the above problems.

[0006] In order to solve the above problems, an embodiment of the present invention discloses a sign language recognition method, comprising:

[0007] Collecting a sign language video in a target area, and converting the sign language video into a first video frame;

[0008] Extracting feature frames from the first video frame by using a preset convolutional neural network;

[0009] Inputting the feature frame into a preset hierarchical long short-term memory network to extract a valid frame;

[0010] The valid frame is input into a preset sign language recognition model, and a target sentence text aligned with the sign language video is output.

[0011] Furthermore, after collecting the sign language video in the target area and converting the sign language video into a first video frame, the method includes:

[0012] The first video frame is cropped into a second video frame according to a preset target object cropping rule.

[0013] Furthermore, extracting a feature frame from the first video frame by using a preset convolutional neural network includes:

[0014] The preset residual network is used to extract the second video frame frame by frame to obtain the feature frame.

[0015] Furthermore, the preset hierarchical long short-term memory network is set as a three-layer long short-term memory network, and the inputting of the feature frame into the preset hierarchical long short-term memory network to extract the valid frame comprises:

[0016] Inputting the feature frame into the first layer of long short-term memory network, establishing a temporal relationship of the feature frame, and obtaining a first temporal frame;

[0017] Inputting the first time series frame into the second layer of long short-term memory network, and extracting the key frame in the first preset sliding window;

[0018] The key frame is input into the third-layer long short-term memory network to extract the valid frame in the second preset sliding window.

[0019] Furthermore, the step of inputting the valid frame into a preset sign language recognition model and outputting a target sentence text aligned with the sign language video comprises:

[0020] The valid frame is input into a model jointly trained by an encoder-decoder structure and a recurrent neural network based on a CTC loss function, and a target sentence text aligned with the sign language video is output.

[0021] Furthermore, before inputting the valid frame into a sign language recognition model jointly trained by an encoder-decoder structure and a recurrent neural network based on a CTC loss function and outputting a target sentence text aligned with the sign language video, the method further includes:

[0022] Input the valid frame into the encoder-decoder structure and the CTC loss function respectively to obtain two corresponding loss function values, and obtain a total loss function value by linearly weighting the two corresponding loss function values;

[0023] The sign language recognition model is trained using the total loss function value.

[0024] Furthermore, the using the total loss function value to train the sign language recognition model includes:

[0025] The total loss function value and the gradient descent algorithm are used to update the parameters in the sign language recognition model.

[0026] The embodiment of the present invention discloses a sign language recognition system, comprising:

[0027] A video acquisition module, used to acquire sign language videos in the target area;

[0028] A feature extraction module, used to extract feature frames in the sign language video through a preset convolutional neural network;

[0029] A feature screening module, used for inputting the feature frame into a preset hierarchical long short-term memory network to extract a valid frame;

[0030] The sign language recognition module is used to input the valid frame into a preset sign language recognition model and output a target sentence text aligned with the sign language video.

[0031] An embodiment of the present invention discloses an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the steps of the sign language recognition method as described above when executed by the processor.

[0032] An embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the sign language recognition method as described above are implemented.

[0033] The embodiment of the present invention includes the following advantages: through the hierarchical long short-term memory network, the maximum pooling operation is performed after the multi-layer network to obtain the key information of the current window, the length of the network is gradually changed from long to short, through this network, the spatiotemporal characteristics of the sign language video are effectively obtained, and the internal connection of the video is extracted from different levels, which can realize the automatic extraction of key gesture information in the current video, and compress the network and improve the calculation speed without reducing the network accuracy. The sign language recognition model based on the encoder-decoder structure and the recurrent neural network based on the CTC loss function are used to jointly optimize and train the network, avoiding the independent relationship between output words and output words, which is more in line with the application scenario of sign language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flowchart of steps of an embodiment of a sign language recognition method of the present invention;

[0035] Figure 2is a flowchart of steps of another embodiment of a sign language recognition method of the present invention;

[0036] Figure 3 is a flowchart of steps of another embodiment of a sign language recognition method of the present invention;

[0037] Figure 4 is a structural block diagram of an embodiment of a sign language recognition system of the present invention;

[0038] Figure 5 is a structural block diagram of another embodiment of a sign language recognition system of the present invention;

[0039] Figure 6 is a structural block diagram of another embodiment of a sign language recognition system of the present invention;

[0040] Figure 7 It is a structural block diagram of another embodiment of a sign language recognition system of the present invention. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] One of the core concepts of the embodiments of the present invention is to provide a sign language recognition method and system, wherein a sign language recognition method includes: collecting sign language videos in a target area; extracting feature frames in the sign language video through a preset convolutional neural network; inputting the feature frames into a preset hierarchical long short-term memory network to extract valid frames; inputting the valid frames into a preset sign language recognition model to output a target sentence text aligned with the sign language video. Through a hierarchical long short-term memory network, a maximum pooling operation is performed after a multi-layer network to obtain key information of the current window. The length of the network gradually changes from long to short. Through this network, the spatiotemporal features of the sign language video are effectively obtained, and the intrinsic connection of the video is extracted from different levels. The key gesture information in the current video can be automatically extracted, and the network can be compressed to improve the calculation speed without reducing the network accuracy. A sign language recognition model based on an encoder-decoder structure and a recurrent neural network based on a CTC loss function is used to jointly optimize and train the network, thereby avoiding the independent relationship between output words and output words, which is more in line with the application scenario of sign language recognition.

[0043] Reference Figure 1 , shows a step flow chart of a sign language recognition method embodiment of the present invention, which may specifically include the following steps:

[0044] S100, collecting a sign language video in a target area, collecting a sign language video in the target area, and converting the sign language video into a first video frame;

[0045] S200, extracting a feature frame from the first video frame by using a preset convolutional neural network;

[0046] S300, inputting the feature frame into a preset hierarchical long short-term memory network to extract a valid frame;

[0047] S400, inputting the valid frame into a preset sign language recognition model, and outputting the target sentence text aligned with the sign language video.

[0048] Referring to step S100, the sign language video in the target area is collected, and a clear sign language gesture video in the target area is captured through input devices such as camera video. The sign language video has a high imaging resolution. In a specific embodiment, the sign language video in the target area is collected by using a camera module, and the camera module is worn on the user's head, behind the ears, etc., so the camera module is required to be light in weight and small in size.

[0049] Referring to step S200 described above, feature frames in the sign language video are extracted by a preset convolutional neural network. Convolutional Neural Networks (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input information according to their hierarchical structure. Therefore, they are also called "Shift-Invariant Artificial Neural Networks (SIANN). Common architectures of convolutional neural networks include AlexNet, ZFNet, VGGNet, GoogleNet and ResNet models.

[0050] Referring to the above step S300, the feature frame is input into a preset hierarchical long short-term memory network to extract valid frames. Since there are too many invalid frames in the sign language video itself, such as transition frames and still frames. If these frames are calculated together and used for prediction, this will not improve the accuracy of the experimental results while wasting additional computing resources. Therefore, a hierarchical long short-term memory network (BiLSTM) is designed to extract valid frames in the sign language video. According to the characteristics of the sign language itself, multiple frames of continuous images form a word, multiple words form a phrase, and multiple phrases form a complete sign language meaning. However, in the expression of a word, there are often some marker frames to represent the word, that is, the remaining frames may only be transition frames. At the same time, in a phrase or sentence, some words or phrases can be used to fully represent it; therefore, a hierarchical BiLSTM model is designed, which consists of a three-layer long short-term memory network + maximum pooling (BiLSTM+Max pooling), which means that key information is extracted from three levels of words, phrases, and sentences. Finally, after a video passes through the hierarchical BiLSTM model, the feature representation of the key information of the video can be obtained.

[0051] Extracting spatial features of video frames. Based on the wide application of convolutional neural networks in image and video fields, this application uses 2D convolutional neural networks to extract spatial features of video frames. Specifically, the Resnet-152 network pre-trained on the ImageNet dataset is used to extract spatial features of video frames, and the following can be obtained: where f i ∈R 2048 ,i=1,2,3...N, N represents the number of video frames. The first layer of BiLSTM network is designed to obtain the temporal relationship between video frames. Since the spatial features of video frames have been obtained, the temporal modeling between video frames is required next, so that the temporal-spatial features between videos can be obtained at the same time. For each moment output, since it is a bidirectional network, there will be two different hidden states. The output of the current moment is defined as a bidirectional hidden state for connection, that is,

[0052] Since a sign language word contains several frames (16 frames on average), but among these frames, perhaps only a few frames (1-3 frames on average) represent the specific meaning of the video, so in the high-level semantic feature representation of the video, if only these key frames are retained for extraction, the computational complexity of the downstream network will become very small, and the experimental accuracy will also be improved. Based on this, a maximum pooling operation is performed after the first layer of BiLSTM, the window size is set to 8, and the repetition ratio between each adjacent window is 50%. By performing a maximum pooling operation in the current window, the feature vector with the largest response in the window can be obtained, which is the key information of the current window. Specific representation Here l is 8, ffi represents the output of frame-level BiLSTM, f f Indicates the key information of the current sliding window.

[0053] Since a sign language text contains several sign language words (9-10 on average), but only a few of these words represent the main meaning of the text, in the high-level semantic feature representation of the video, only these key words are retained for extraction, which will reduce the computational complexity of the downstream network and improve the network recognition rate. Based on this, the second layer BiLSTM is designed to model the temporal relationship of the key frame information obtained after the first layer maximum pooling operation. Similarly, the output at the current moment is defined as a bidirectional hidden state connection, that is, Similarly, after the second layer of BiLSTM network, the maximum pooling operation is performed, the window size is set to 4, and the repetition rate between two adjacent windows is 50%, and the key words (actions) in the current text can be obtained. The specific expression is

[0054] The key information of the action level is input into the third-layer BiLSTM network, and the output of this layer network at each moment is obtained through the same forward and backward operations as above. Through bidirectional calculation, the temporal relationship between actions can be better captured. At the same time, after such a three-layer structure, higher-dimensional temporal and spatial relationship features in the video can be extracted, providing rich information for downstream semantic recognition.

[0055] Referring to the above step S400, the valid frame is input into the preset sign language recognition model, and the target sentence text aligned with the sign language video is output. The preset sign language recognition model in this embodiment specifically introduces the method of joint optimization of connectionist temporal classification (CTC) and encoder-decoder structure into the sign language recognition task. There are two main methods for seq2seq model, the first is the seq2seq model based on RNN-CTC, and the second is the encoder-decoder structure based on RNN-RNN. Both methods avoid the problem of frame-by-frame labeling of seq2seq solved by traditional methods. However, CTC assumes that the output is independent of each other. For example, in sign language translation, for the label "My father is a teacher", CTC believes that "I" and "Dad" are independent of each other. This assumption is obviously unreasonable. In addition, the seq2seq model based on RNN-CTC cannot solve the situation where the input is longer than the output. There is no big problem in Chinese sign language, but there will be some shortcomings in the German sign language dataset. In the RWTH-WEATHER 2014 dataset, there are some videos with fewer frames than the actual number of label words. In this case, CTC will inevitably make prediction errors. Based on this, there is another structure to solve the above-mentioned problem of CTC, namely the encoder-decoder structure based on RNN-RNN, which can model the input-to-output relationship of any length. At the same time, this model structure assumes that the output words are not independent of each other. This model structure cannot handle longer sequences well, because the input data is always encoded into a feature vector of fixed length. If the input sequence is too long, the input sequence cannot be well represented in a limited space. For noisy data, this structure cannot model the relationship between input and output well. The method of jointly optimizing CTC and encoder-decoder structure proposed in this application can effectively solve the above problems by introducing it into the sign language recognition task.

[0056] In a specific embodiment, after collecting a sign language video in a target area and converting the sign language video into a first video frame S100, the process includes:

[0057] S500: cropping the first video frame into a second video frame according to a preset target object cropping rule.

[0058] Referring to the above steps, the residual network pre-trained on Imagenet is used to extract the 2D features of the sign language video frame by frame, where Imagenet is a large visualization database for visual object recognition software research; and the features are used for downstream tasks. Specifically, the ffmpeg tool is used to convert the time series and video into video frames, and the target person is centered on the cropped image, so that the size of each image becomes 670×630, so that the blank part can be removed and the recognition accuracy of the model can be improved. The number of frames of the sign language video is composed of tens to hundreds of frames, and the relative variation range is large depending on the corresponding text of the sign language video. Observing the sign language video, it can be seen that the sign language video has too many invalid frames, such as transition frames, static frames, etc. In addition, a sentence of text in the sign language video only needs a few phrases to understand the text. In the sign language word video, only a few frames are needed to understand the meaning of the sign language video. Therefore, it is very important to extract the key gesture information of the sign language video. Usually, the resolution of sign language videos is high and the background is relatively empty, but these backgrounds are not very helpful for sign language recognition. Therefore, this application performs cropping with people as the center, which can remove the empty parts in the image.

[0059] Reference Figure 2 In this embodiment, step S200 of extracting feature frames from a sign language video by using a preset convolutional neural network includes:

[0060] S210, use a preset residual network to extract the second video frame frame by frame to obtain a feature frame. The residual network (Residual Network, ResNet) is easy to optimize and can improve accuracy by increasing the depth. The residual block inside it uses a skip connection to alleviate the gradient vanishing problem caused by increasing the depth in the deep neural network. In this embodiment, Resnet-152 pre-trained on Imagenet is used to extract the 2D features of the sign language video frame by frame, and the features are used for downstream tasks. In view of the network model, the second video frame is input into the residual network-152 (the network is pre-trained on the ImageNet dataset in advance) in sequence, and the 2D features of the video frame can be obtained. The above 2D features are used as feature frames. Represents a set of image frames of a video, where N represents the total number of frames of the video. After the residual network-152, the 2D feature representation of the video frame can be obtained. where f i ∈R 2048 ,i=1,2,3...N.

[0061] Reference Figure 3 In this embodiment, the preset hierarchical long short-term memory network is set to a three-layer long short-term memory network, and the feature frame is input into the preset hierarchical long short-term memory network to extract the valid frame step S300, including:

[0062] S310, inputting the feature frame into the first layer of long short-term memory network, establishing a temporal relationship of the feature frame, and obtaining a first temporal frame;

[0063] S320, inputting the first time series frame into the second layer of long short-term memory network, and extracting the key frame in the first preset sliding window;

[0064] S330, input the key frame into the third-layer long short-term memory network, and extract the valid frame in the second preset sliding window.

[0065] In this embodiment, the structure of sign language is taken into consideration, that is, in a sentence of sign language, only a few words may be needed to better express the meaning of the sign language. Similarly, in a sign language video, a word may be composed of several actions, but among these actions, there must be several key actions, and these key actions can also fully express the meaning of the sign language word. In a sign language video, a sign language word may be composed of many consecutive sign language frames, but among these sign language frames, many are redundant frames, such as transition frames and still frames. Therefore, in an action, only fewer frames may be needed to achieve comparable results. Based on this, a hierarchical long short-term memory network is designed to model the sign language video from frames to words, and from words to phrases / sentences.

[0066] Referring to step S310, the feature frame is input into the first layer of the long short-term memory network to establish the temporal relationship of the feature frame and obtain the first temporal frame. The first layer of the network selects a bidirectional long short-term memory network, which is used to model the temporal relationship between frames. The features of the video frame are input into the BiLSTM, and the forward hidden state can be obtained at the same time. and the backward hidden state The forward direction considers the relationship between the current frame and the previous frame, and the backward direction considers the relationship between the current frame and the next frame. The results of the forward and backward directions are concatenated into arrays, that is, The relationship between the current frame and the entire video can be obtained. A frame of length N passes through the network and obtains an output of length N, but these N outputs are related to each other. Because the input of this layer of network is the feature frame of the video, this layer of network is also called frame-level BiLSTM.

[0067] Referring to step S320, the first time series frame is input into the second layer of long short-term memory network to extract the key frame in the first preset sliding window. At the output of the first layer of long short-term memory network (frame-level BiLSTM network), a sliding window is used, the sliding window size is set to 8, the repetition rate is 50%, and a maximum value is selected in each sliding window, i.e. f firepresents the output of frame-level BiLSTM, f f The first time frame (frame-level key information) is input into the second layer of long short-term memory network, and the output f of the network at each moment is obtained by the forward operation and backward operation of step S310. an The key frames in each sliding window can be obtained (which can be understood as the key frames in an instantaneous action). A sign language word includes several actions. For example, the sign language "hello" includes the actions of raising the hand and pressing the thumb down, but the key action is pressing the thumb down. Therefore, the first layer of the long short-term memory network can extract the feature representation of the key action of pressing the thumb down after the first layer of maximum pooling operation.

[0068] Referring to the above step S330, the key frame is input into the third-layer long short-term memory network, and the valid frame in the second preset sliding window is extracted. Although the first-layer long short-term memory network can extract the key action of thumb pressing down after the first-layer maximum pooling operation. However, a sign language text contains multiple sign language words (actions), some of which are not very important. Therefore, the present application designs a window maximization operation similar to step S320. In this step, the window size is set to 4, and the repetition rate between each sliding window is 50%. The output of the second-layer long short-term memory network is subjected to the sliding window maximum value operation to obtain the feature representation of the key actions in the words. The key information at the action level is input into the third-layer long short-term memory network, and the output of this layer of network at each moment is obtained through the above-mentioned forward and backward operations. The specific process can refer to step S310. After bidirectional calculation, the temporal relationship between actions can be better captured. At the same time, after such a three-layer structure, higher-dimensional temporal and spatial relationship features in the video can be extracted, providing rich information for downstream semantic recognition.

[0069] In a specific embodiment, designing a multi-layer long short-term memory network to model the temporal relationship between words in a sign language video and the temporal relationship between words in a label includes the following steps:

[0070] Use the stammering word segmentation to segment the sentence. According to the known dictionary, the sentence is divided into words, that is, y = (y1, y2, ..., y T ), where T means that the sentence consists of T words;

[0071] Because words are isolated and cannot be directly input into the network, the one-hot encoding method is used to extract the encoding vector of the word. Let y t After encoding, it becomes y' t,This code has only one valid bit, and the beginnings of words are independent;

[0072] The word vector encodings of the sentence words are input into the unidirectional long short-term memory network (LSTM) in sequence to model the temporal and dependency relationships between words.

[0073] In this embodiment, step S400 of inputting valid frames into a preset sign language recognition model and outputting a target sentence text aligned with the sign language video includes:

[0074] The valid frames are input into a model jointly trained with an encoder-decoder structure and a recurrent neural network based on a CTC loss function, and the target sentence text aligned with the sign language video is output.

[0075] Referring to the above steps, there are currently two main methods for seq2seq models. The first is a seq2seq model based on RNN-CTC, and the second is an encoder-decoder structure based on RNN-RNN. Both methods avoid the problem of frame-by-frame labeling of seq2seq that is solved by traditional methods. However, CTC assumes that outputs are independent of each other. For example, in sign language translation, for the label "My father is a teacher", CTC believes that "I" and "Dad" are independent of each other. This assumption is obviously unreasonable. In addition, the seq2seq model based on RNN-CTC cannot solve the situation where the input is longer than the output. There is no big problem in Chinese sign language, but there will be some shortcomings in the German sign language dataset. In the RWTH-WEATHER 2014 dataset, there are some videos with fewer frames than the actual number of label words. In this case, CTC will definitely make a wrong prediction. Based on this, there is another structure to solve the above problems of CTC, that is, the encoder-decoder structure based on RNN-RNN, which can model the input-output relationship of any length. At the same time, the model structure assumes that the output words are not independent of each other, but are related to each other. However, this model structure cannot handle longer sequences well, because the input data is always encoded into a feature vector of fixed length. If the input sequence is too long, the input sequence cannot be well represented in a limited space. At the same time, this structure cannot model the input and output well for noisy data. Based on this, the present application proposes a model based on the joint training of an encoder-decoder structure and a recurrent neural network based on a CTC loss function, and introduces it into the sign language recognition task.

[0076] In this embodiment, before the step of inputting the valid frame into the sign language recognition model jointly trained by the encoder-decoder structure and the recurrent neural network based on the CTC loss function and outputting the target sentence text aligned with the sign language video, the step further includes:

[0077] Input the valid frames into the encoder-decoder structure and the CTC loss function respectively to obtain the corresponding two loss function values, and then linearly weight the two corresponding loss function values ​​to obtain the total loss function value;

[0078] Use the total loss function value to train the sign language recognition model.

[0079] In this embodiment, the step of training the sign language recognition model using the total loss function value includes:

[0080] The total loss function value and the gradient descent algorithm are used to update the parameters in the sign language recognition model.

[0081] Referring to the above steps, since both the encoder-decoder structure and the CTC loss function-based sign language recognition model have recurrent neural networks (RNNs), this application uses a hierarchical bidirectional long short-term memory network as a common part, namely the RNN in the CTC-based sign language recognition structure and the encoder in the encoder-decoder-based sign language recognition structure. The high-dimensional temporal and spatial features of a sign language video are extracted through a hierarchical bidirectional long short-term memory network. Where N' represents the length of the video frame with length N after two key information extractions. s At the same time, the following two changes are made, corresponding to the CTC-based sign language recognition model and the encoder-decoder-based sign language recognition model respectively.

[0082] Use a linear mapping layer to transform F s The dimension of F is changed to the total number of dictionaries. In this experiment, the total number of dictionaries is 178. At the same time, the label of 'blank' is introduced, indicating that no real word is predicted at the current moment, so it is marked with 'blank'. s After being sent to the linear mapping layer, softmax changes are performed to obtain P, and the dictionary V is obtained at each moment * The probability distribution on . Multiplying the labels obtained at each position can get the probability of a fitted path, Perform B operation on π, which is defined as removing 'blank' and repeated characters in the path. Therefore, for a given video feature F, the probability of obtaining a given text L is defined as The probability can be calculated efficiently through the dynamic programming algorithm. Taking the logarithm of the probability and taking the negative, we get the loss function of the network, which is defined as L CTC =-ln(p(LF)).

[0083] In the encoder-decoder based sign language recognition model, the attention mechanism is used. In this case, the newly defined dictionary set V'=VY{ <sos> , <eos>},in <sos>Indicates the start of decoding. <eos>Indicates the end of decoding. For the original text L, redefine label: L'=' <sos>'YEAR' <eos>'. The encoder output h at the kth moment k =LSTM(C k ,s k ,h k-1 ), where C k represents the attention input of the encoder at the current moment; s k Indicates the word vector encoding representation of the input word at the current moment; h k-1 Represents the hidden state of the previous moment in the LSTM network. The output h of the encoder at the kth moment k Input into the linear mapping layer, transform its dimension into |V'|, and get z k Based on this, for a given video feature F, the probability of obtaining a given text L is defined as Taking the logarithm of this probability and negating it, we get the loss function of the network, defined as L Attention =-ln(p(L|F)).

[0084] Set the variable parameter λ and define the total network loss function L total =λL CTC +(1-λ)L Attention , and train the entire network by gradient descent.

[0085] In a specific embodiment, the Chinese continuous sign language data set used in the experiment verifies the feasibility of the method of the present invention. The data set includes 100 commonly used expressions, which are recorded by 50 deaf-mute people, and each person records 5 times. Therefore, the data set contains 25K videos in total, and each video has its corresponding text information. Due to the large differences between different sign languages, there are also large differences between different recorders. Therefore, the data set is divided in the following two ways. The first division method is divided according to different numbers of people. The 20K videos recorded by 40 deaf-mute people in the data set are used as training sets, and the 5K videos recorded by 10 deaf-mute people are used as test sets. The second division method is divided according to unused sentences. A total of 23.5K videos of 94 sentences in the data set are used as training sets, and the remaining 6 sentences are used as test sets. In this division method, there is no duplication between 94 and 6 sentences, but the words in the 6 sentences have all appeared in the training set, but the context in which they appear is different from that in the training set.

[0086] For the same data set, the performance of the proposed method is compared with that of the existing sign language recognition methods. For the evaluation index of continuous sign language recognition, the experiment uses the word error rate (WER). Among them words ins , words del , words sub Respectively represent how many times the network's predicted output is inserted, deleted, and replaced to get the text corresponding to the video. word nums It indicates the number of words in the text corresponding to the video. The smaller the WER, the better the network recognition effect.

[0087] In the process of video feature extraction of the present invention, since the video is composed of multiple frames. Therefore, it is very important to obtain the space-time features in the video frame. 2D convolutional neural network has been proved to be able to obtain the spatial features in the video well. Therefore, in the present invention, the present application first tries to use the Resnet-152 network pre-trained on Imagenet to extract the spatial features of each frame after the video is divided into frames. The temporal relationship can be modeled by a long short-term memory network. In view of the characteristics of key frames, key actions, etc. in the sign language video, in the present invention, a layered long short-term memory network is designed, and a maximum pooling operation is performed after the first layer of the network to obtain the key information of the current window. The maximum pooling operation is performed after the second layer of the network to obtain the key information of the current window. The length of the network changes gradually from long to short. Through this network, the space and time features of the video are effectively obtained, and the intrinsic connection of the video is extracted from different levels.

[0088] The encoder-decoder based sign language recognition model can realize the input of sign language videos of arbitrary length, avoiding the independent relationship between output words, and is more suitable for the application scenarios of sign language recognition. However, the encoder-decoder based sign language recognition model cannot handle long sign language video input or input video containing noise well, because in the encoder-decoder structure, the video features are represented as a feature vector of fixed dimension. When the video length is too long or contains a lot of noise, the fixed feature vector cannot contain effective information, so it cannot achieve a better network recognition effect.

[0089] The sign language recognition model based on RNN-CTC can avoid labeling video frames frame by frame, and realize the alignment process between two sequences by maximizing all possible paths. Since the CTC algorithm can only predict one label at a time, the CTC algorithm cannot align the situation where the input is longer than the output. In addition, the CTC algorithm believes that in the output process, each output is independent of each other, but sign language recognition does not hold true in this case.

[0090] It is proposed to linearly combine the RNN-CTC-based sign language recognition model with the encoder-decoder-based sign language recognition model, jointly optimize, and train the network.

[0091] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0092] Reference Figure 4 , shows a structural block diagram of a sign language recognition system embodiment of the present invention, which may specifically include the following modules:

[0093] This video acquisition module 100 is used to acquire the sign language video in the target area;

[0094] A feature extraction module 200 is used to extract feature frames in the sign language video through a preset convolutional neural network;

[0095] The feature screening module 300 is used to input the feature frame into a preset hierarchical long short-term memory network to extract a valid frame;

[0096] The sign language recognition module 400 is used to input valid frames into a preset sign language recognition model and output a target sentence text aligned with the sign language video.

[0097] Reference Figure 5 , in this embodiment, further comprising:

[0098] The video cropping module 500 is used to convert the sign language video into a first video frame, and crop the first video frame into a second video frame according to a preset target object cropping rule.

[0099] Reference Figure 6 In this embodiment, the feature extraction module 200 includes:

[0100] The feature extraction unit 210 is used to extract the second video frame frame by frame using a preset residual network to obtain a feature frame.

[0101] Reference Figure 7 In this embodiment, the preset hierarchical long short-term memory network is set to a three-layer long short-term memory network, and the feature screening module 300 includes:

[0102] A first screening unit 310 is used to input the feature frame into the first layer of the long short-term memory network, establish a time sequence relationship of the feature frame, and obtain a first time sequence frame;

[0103] The second screening unit 320 is used to input the first time series frame into the second layer of the long short-term memory network to extract the key frame in the first preset sliding window;

[0104] The third screening unit 330 is used to input the key frame into the third layer of the long short-term memory network to extract the valid frame in the second preset sliding window.

[0105] In this embodiment, the sign language recognition module 400 includes:

[0106] The sign language recognition unit is used to input valid frames into a model jointly trained with an encoder-decoder structure and a recurrent neural network based on a CTC loss function, and output a target sentence text aligned with the sign language video.

[0107] In this embodiment, the sign language recognition module 400 further includes:

[0108] A parameter weighting unit is used to input the valid frame into the encoder-decoder structure and the CTC loss function respectively, obtain two corresponding loss function values, and obtain the total loss function value by linearly weighting the two corresponding loss function values;

[0109] The model training unit is used to train the sign language recognition model using the total loss function value.

[0110] In this embodiment, the model training unit includes:

[0111] The model training subunit is used to update the parameters in the sign language recognition model using the total loss function value and the gradient descent algorithm.

[0112] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0113] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0114] An embodiment of the present invention discloses an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the steps of the sign language recognition method as described above when executed by the processor.

[0115] An embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the sign language recognition method as described above are implemented.

[0116] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0117] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0118] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0119] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0120] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0121] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0122] The above is a detailed introduction to a sign language recognition method and a sign language recognition system provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.< / eos> < / sos> < / eos> < / sos> < / eos> < / sos>

Claims

1. A sign language recognition method, characterized in that: include: Collecting a sign language video in a target area, and converting the sign language video into a first video frame; Extracting feature frames from the first video frame by using a preset convolutional neural network; Input the feature frame into a preset hierarchical long short-term memory network to extract a valid frame; input the feature frame into a first layer long short-term memory network to establish a time sequence relationship of the feature frame to obtain a first time sequence frame; Input the first time sequence frame into the second layer of long short-term memory network to extract the key frame in the first preset sliding window; input the key frame into the third layer of long short-term memory network to extract the valid frame in the second preset sliding window; design a multi-layer long short-term memory network to model the time sequence relationship between words in the sign language video and the time sequence relationship between words in the label; Inputting the valid frame into a preset sign language recognition model, and outputting a target sentence text aligned with the sign language video; The valid frames are respectively input into the encoder-decoder structure and the CTC loss function to obtain two corresponding loss function values, and the two corresponding loss function values ​​are linearly weighted to obtain a total loss function value; the sign language recognition model is trained using the total loss function value; the valid frames are input into a model jointly trained by a recurrent neural network based on an encoder-decoder structure and a CTC loss function, and a target sentence text aligned with the sign language video is output.

2. The method according to claim 1, characterized in that The step of collecting the sign language video in the target area and converting the sign language video into a first video frame includes: The first video frame is cropped into a second video frame according to a preset target object cropping rule.

3. The method according to claim 2, characterized in that The extracting a feature frame from the first video frame by using a preset convolutional neural network comprises: The preset residual network is used to extract the second video frame frame by frame to obtain the feature frame.

4. The method according to claim 1, characterized in that The using the total loss function value to train the sign language recognition model comprises: The parameters in the sign language recognition model are updated using the total loss function value and a gradient descent algorithm.

5. A sign language recognition system, characterized in that: include: A video acquisition module, used to acquire the sign language video in the target area, acquire the sign language video in the target area, and convert the sign language video into a first video frame; A feature extraction module, used to extract feature frames in the sign language video through a preset convolutional neural network; A feature screening module is used to input the feature frame into a preset layered long short-term memory network to extract a valid frame; input the feature frame into a first layer of long short-term memory network to establish a time sequence relationship of the feature frame to obtain a first time sequence frame; Input the first time sequence frame into the second layer of long short-term memory network to extract the key frame in the first preset sliding window; input the key frame into the third layer of long short-term memory network to extract the valid frame in the second preset sliding window; design a multi-layer long short-term memory network to model the time sequence relationship between words in the sign language video and the time sequence relationship between words in the label; A sign language recognition module, used to input the valid frame into a preset sign language recognition model, and output a target sentence text aligned with the sign language video; The valid frames are respectively input into the encoder-decoder structure and the CTC loss function to obtain two corresponding loss function values, and the two corresponding loss function values ​​are linearly weighted to obtain a total loss function value; the sign language recognition model is trained using the total loss function value; the valid frames are input into a model jointly trained by a recurrent neural network based on an encoder-decoder structure and a CTC loss function, and a target sentence text aligned with the sign language video is output.

6. An electronic device, characterized in that The method comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the sign language recognition method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the sign language recognition method according to any one of claims 1 to 4 are implemented.

Citation Information

Cited By

  • Real-time video stream sign language recognition method based on human body key points

    CN115457654A

  • A Real-time Sign Language Recognition Method for Video Streams Based on Human Key Points

    CN115457654B