Sign language teaching method, device and system based on gesture generation and recognition
Through the T-G pose generation network and dual-stream VTN network, the sign language data set is trained to generate 3D virtual character action videos, and the learner imitation effect is evaluated through the sign language translation model, which solves the problems of poor expression and low imitation efficiency in sign language teaching, and achieves high quality sign language teaching and rich teaching mode.
Patent Information
- Application Number
- CN202210029311.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-11
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-01-11
AI Technical Summary
The existing sign language teaching model is poor in expression, learners are inefficient in imitation, and sign language video generation is difficult to synchronize body posture, hand movements and facial expressions, resulting in unnatural sign language videos and inaccurate details, making it difficult to meet the learning of hearing-impaired people. need.
The T-G pose generation network and the dual-stream VTN network are used to train the sign language data set to generate 3D virtual character action videos, and the sign language movements imitated by learners are recognized and evaluated through the sign language translation model, providing similarity feedback and improving learning effect.
It realizes high-quality sign language teaching, improves learners' independent learning ability and efficiency, enhances the naturalness and detailed accuracy of sign language movements, enriches the teaching mode, and reduces the difficulty of operation and device complexity.
Smart Images

Figure CN114842547B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical fields of computer vision and pattern recognition, and specifically relates to a sign language teaching method, device and system based on gesture generation and recognition. Background Art
[0002] Sign language is a tool for hearing-impaired people to communicate and express their thoughts. Like oral language, it has the function of transmitting information and can express some complex or abstract language concepts. Usually, according to specific grammar, finger movements are combined with body language and facial expressions to convey information. It has developed from simple gestures to a special language that is vivid and has the function of communication and exchange. Sign language recognition and translation technology has facilitated communication between hearing-impaired people and healthy people. Its research should not only enable healthy people to understand sign language, but also enable hearing-impaired people to understand what healthy people say. This interactive process is particularly important. In terms of educational resources, digital educational resources for the hearing-impaired are relatively scarce, and the matching degree between teaching content and student needs is not high.
[0003] Since the sign language video generation task developed relatively late, the research on this task in the sign language field is still in its infancy. Traditional daily action video generation tasks usually consider human periodic movements, while the difficulty of sign language video generation lies in the fact that sign language movements have subtle changes and low overlap. At the same time, sign language generation requires modeling multiple parts such as body posture, hand movements, facial expressions, etc. to ensure the synchronization of each part in time. Furthermore, the generated sign language video must be understandable and acceptable to the hearing-impaired, which puts higher requirements on the naturalness of the generated video, the accuracy of gesture details, and the coherence between sign language movements. For example, the Chinese patent application with application number CN202110778836.4 discloses a sign language translation system, including a sign language translation method and a sign language reverse translation method; the sign language translation method includes receiving gesture action data, converting the gesture action data into a first text information, performing speech synthesis processing on the current first text information, and obtaining sign language translation voice information; the sign language reverse translation method includes receiving voice data, converting the voice data into a second text information, and parsing the current second text information to obtain sign language animation information that matches the second text information after the current word segmentation. However, this method relies on the need to match animations for each sign language video in the early stage, which is a huge workload and is not popular for teaching. Therefore, it is necessary to build a sign language teaching system based on gesture generation, which can enrich the sign language teaching model for the deaf and mute, improve learning interest, and realize the popularization of high-quality sign language teaching. Summary of the invention
[0004] Technical problem: This invention is committed to introducing artificial intelligence into classroom teaching, expanding the existing sign language teaching model, solving the problems of poor expressiveness and low learner imitation efficiency.
[0005] By training the gesture generation model, the sign language text is converted into a 3D virtual character action video for learners to imitate; and the action video imitated by the learner can be recorded for feedback and scoring, which improves the autonomous learning ability of sign language learners and completes learning efficiently.
[0006] Technical solution: The first aspect of the present invention provides a sign language teaching method based on gesture generation and recognition, comprising:
[0007] Use the TG gesture generation network to train the gesture generation model on the sign language dataset video and sign language text labels;
[0008] Use the two-stream VTN network to train the sign language dataset to generate a sign language translation model;
[0009] The text sentence to be recognized is fed into the trained posture generation model to obtain a 3D virtual character action video for learners to imitate;
[0010] The sign language translation model is used to identify the sign language movements imitated by learners, and the similarity between the movement imitation results and the standard teaching movements is evaluated to provide feedback on the learning effect.
[0011] Furthermore, the method of using the TG posture generation network to train the posture generation model for the sign language dataset video and the sign language text label includes:
[0012] Use the forward Transformer network to train the sign language dataset video and sign language text labels to obtain the posture prediction sequence;
[0013] Use the backward GAN network to perform alternating iterative training on the posture prediction sequence and save the trained posture generation model.
[0014] Furthermore, the forward Transformer network is used to train the sign language dataset video and the sign language text label to obtain the posture prediction sequence including:
[0015] Use the Encoder part of Transformer to extract features of sign language text labels and extract text feature information;
[0016] Detect key points of human skeleton posture and extract three-dimensional coordinate points from the sign language dataset video;
[0017] The extracted text features and three-dimensional coordinate points are sent to the Decoder part of the Transformer to obtain the posture prediction sequence.
[0018] Furthermore, when using the Encoder part of the Transformer to extract features from the sign language text label and extract text feature information, the following formula is used to provide the model with position information:
[0019]
[0020]
[0021] Among them, pos is the position of a word in a sentence, i represents the dimension number of the word vector, and d modet Represents the dimension value of the word vector.
[0022] Further, the step of performing alternating iterative training on the posture prediction sequence using the backward GAN network and saving the trained posture generation model comprises:
[0023] Send the forward-generated gesture prediction sequence into the Generator to obtain the virtual sign language gesture;
[0024] Use the Discriminator to perform discrimination training on virtual sign language gestures and real sign language gestures;
[0025] The generator and discriminator are used for alternating iterative training to make the virtual sign language gestures closer to the real sign language gestures, and a gesture generation model that cannot be distinguished by the discriminator is obtained.
[0026] Furthermore, the method of using a dual-stream VTN network to train a sign language dataset video to generate a sign language translation model includes:
[0027] Perform DNN global human feature detection on isolated word sign language videos and extract global features;
[0028] Use the Openpose model to detect palm features on the isolated word sign language dataset and extract palm features;
[0029] The global features are fused with the palm features, and the continuous sentence sign language dataset is trained using the two-stream VTN network to obtain the sign language translation model.
[0030] Furthermore, the step of feeding the text sentence to be recognized into the trained posture generation model to obtain a 3D virtual character action video for the learner to imitate includes:
[0031] Use the gesture generation model to convert the sign language text sentence to be recognized into human skeleton gesture key points;
[0032] The ThreeDPose model is used to virtually reconstruct the 3D posture of key points of the human skeleton and generate 3D virtual character action videos for learners to imitate.
[0033] Furthermore, the sign language translation model is used to identify the sign language movements imitated by the learner, and the similarity between the movement imitation results and the standard teaching movements is evaluated, and the feedback of the learning effect includes:
[0034] Record the learner's imitating sign language action video and perform dual-stream feature extraction on the video;
[0035] The dual-stream features are fed into the trained sign language translation model to generate the learner’s action imitation results.
[0036] Use the real-time evaluation model to evaluate the similarity between the action imitation results and the standard teaching actions and obtain a score value;
[0037] The accuracy of the movements is graded according to the score value.
[0038] In a second aspect, the present invention provides a sign language teaching device based on gesture generation and recognition, and performs sign language teaching according to the sign language teaching method based on gesture generation and recognition, including:
[0039] A first training unit is configured to train a gesture generation model using a TG gesture generation network, a sign language dataset video and a sign language text label;
[0040] A second training unit is configured to generate a sign language translation model by training a sign language data set using a two-stream VTN network;
[0041] A gesture action generation unit is configured to send the text sentence to be recognized into the trained gesture generation model to obtain a 3D virtual character action video for learners to imitate;
[0042] The sign language translation unit is configured to use the sign language translation model to recognize the sign language movements imitated by the learner, evaluate the similarity between the movement imitation results and the standard teaching movements, and provide feedback on the learning effect.
[0043] In a third aspect, the present invention provides a sign language teaching system based on gesture generation and recognition, and performs sign language teaching according to the provided sign language teaching method based on gesture generation and recognition, including a client and a server:
[0044] The client includes a data acquisition module, a data transmission module and a front-end display module;
[0045] The data collection module is used to collect text label data of sign language teaching, and use an online camera to record video data of movements imitated by learners;
[0046] The data transmission module is used to remotely transmit the collected text label data of the sign language teaching or the recorded action video data of the learner imitating to the server to generate a 3D virtual character action video or perform recognition and scoring;
[0047] The front-end display module is used to display the 3D virtual character action video for the learner to imitate, and can also display the score value of the learner's imitation result transmitted back by the server;
[0048] The server includes a result feedback module, an information processing module and an information storage module;
[0049] The result feedback module is used to remotely transmit the 3D virtual character action video generated by the server or the processed learner imitation result to the front-end display module of the client for display;
[0050] The information processing module is used to generate 3D virtual character action videos from the text label data of sign language teaching, and can also identify the sign language actions imitated by the learners and provide feedback and scoring;
[0051] The information storage module is used to store the sign language data set and the trained gesture generation model and sign language translation model.
[0052] Beneficial effects: Compared with the prior art, the present invention has the following advantages:
[0053] (1) Compared with the existing gesture generation methods, the gesture generation method in the present invention adopts an adversarial gesture generation network composed of an improved Transformer generator and a conditional discriminator, and is trained with a sign language data set to generate sign language actions in any scene, which is convenient for deaf-mute students to learn independently and improve the intelligent teaching content of the classroom. The multi-head Attention mechanism of Transformer is used to extract features from the sign language data set and text labels. Compared with the existing feature extraction technology, it plays a role in enhancing the model and helps the network capture richer features or information. The human skeleton posture key point extraction method based on the Openpose network is used to detect and extract dozens of skeleton key points of the human body and palm. Compared with the existing detection method, it is more accurate, more efficient, and the network model is built faster. The GAN network based on the Generator-Discriminator structure is used to perform alternating iterative training on sign language gestures so that the gesture gestures generated from the text labels can achieve the effect of real sign language actions. The ThreeDPose model is used to generate any 3D virtual character action video for the generated gesture action sequence, which improves the learner's learning interest and enriches the existing sign language teaching model.
[0054] (2) Compared with the existing sign language recognition methods, the sign language teaching method based on gesture recognition in the present invention uses a dual-stream VTN network to detect and extract sign language action features, which captures useful feature information more comprehensively and greatly improves the recognition accuracy. The DNN module based on OpenCV is adopted. Compared with the existing cascade classifier and Dlib algorithm using Haar features, the DNN model has higher recognition accuracy and speed, stronger anti-interference ability, can detect faces in real time, accurately frame the sign language action area, has significant robustness for complex scenes, and improves the accuracy of sign language recognition to a certain extent. The feature sequence is predicted using the structure of the codec, and the action video imitated by the learner can be quickly identified and scored. The similarity between the learner's action and the standard library action can be well evaluated, the learning quality can be fed back, and the learner can be further guided to learn, thereby improving the flexibility of teaching activities.
[0055] (3) The sign language teaching system for online gesture generation and recognition in the present invention includes two parts, a client and a server. The data set and the trained network model are placed on a remote server. The client at the school only needs to be equipped with a camera to collect real-time data and transmit it remotely to the server for gesture generation or sign language translation, which reduces the user's operating difficulty and makes the device of the present invention portable and easy to promote. The server of the present invention is placed on a cloud server, which can be managed and used in a mobile manner, and can flexibly utilize resources. One server can accommodate 5,000 people online at the same time. CUDA is used to perform deep training on neural networks, which improves the speed of building network models, satisfies the teacher's full training of sign language movements in multiple scenarios, improves the quality and efficiency of sign language teaching, and reduces the complexity of sign language teaching devices, enriching the diversity of sign language teaching modes. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 is a flow chart of a sign language teaching method based on gesture generation and recognition in an embodiment of the present invention;
[0057] Figure 2 It is a logic flow chart of a sign language teaching method based on gesture generation and recognition in an embodiment of the present invention;
[0058] Figure 3 This is a flow chart of training a gesture action generation network based on a TG gesture generation network in an embodiment of the present invention;
[0059] Figure 4 A diagram showing a human skeleton posture key point network training structure in an embodiment of the present invention;
[0060] Figure 5 A diagram of a TG posture network training structure generated in an embodiment of the present invention;
[0061] Figure 6 It is a flow chart of sign language translation based on a dual-stream VTN network in an embodiment of the present invention;
[0062] Figure 7 A dual-stream VTN network structure diagram in an embodiment of the present invention;
[0063] Figure 8 This is a schematic diagram of a learner imitating the sign language movements of a 3D virtual character to learn in an embodiment of the present invention;
[0064] Fig. 9 A schematic diagram of a client-server system in an embodiment of the present invention; DETAILED DESCRIPTION
[0065] The present invention will be further described below in conjunction with the embodiments and the accompanying drawings.
[0066] The first aspect of the present invention provides a sign language teaching method based on gesture generation and recognition, combined with Figure 1 and Figure 2 As shown, in an example of the present invention, the method includes:
[0067] Step S100: Using the TG gesture generation network to train a gesture generation model with the sign language dataset video and sign language text labels.
[0068] like Figure 3 As shown, the TG gesture generation network used in the embodiment of the present invention is an adversarial gesture generation network composed of an improved Transformer generator and a conditional discriminator. The forward Transformer network includes two structures, an encoder and a decoder; the backward GAN network includes two structures, a generator and a discriminator. Using TG gesture generation network training as a method for gesture generation can improve the robustness of the gesture generation model, enhance the generalization ability of the network, and give learners more options for imitating sign language movements in more environments.
[0069] The core of the posture generation network is the human posture estimation algorithm. In the embodiment of the present invention, the Openpose network is used, and a bottom-up human skeleton key point detection method is used, that is, the model first detects all the human skeleton key points in the image, and then uses the algorithm to determine which part the key points belong to, and finally connects the key points according to the strategy. Figure 4As shown in the figure, the network is divided into two branches, the first branch network at the top inputs image features and outputs a confidence map for predicting key points of human skeleton; the second branch network at the bottom inputs image features and outputs a part affinity vector field for predicting the affinity between key points of human skeleton. Subsequently, the results of these two sets of outputs will be used to find partial connection relationships through even matching in graph theory, and finally connect all key points belonging to the same part. The feature maps input to the two branches are the feature maps obtained after the original image is extracted by the convolutional neural network.
[0070] More specifically, in the first stage, the network outputs a set of confidence maps S l =ρ l (F) and a set of partial affinity vector fields where ρ l (F) and It is the expression of the inference result of the convolutional neural network based on the extracted feature map at this stage. The two branches are iteratively trained respectively, and each iteration can be regarded as completing a stage. During the iteration process, the input of the branch network at the current stage is the output of the network at the previous stage and the feature map F of the original image. This feature repetition method can produce more accurate predictions. The formula is as follows:
[0071]
[0072]
[0073] Among them, ρ t and is the intermediate result of stage t.
[0074] In order to make the model continuously iteratively converge during the training process and improve the accuracy of the final prediction of each branch result, each loss function is set at the end of each stage. The upper and lower branch networks each have a loss function, and the L2 loss function is used between the true value and the predicted value to constrain the result. Among them, in order to solve some practical problems, such as some data sets do not completely mark all targets, the loss function is weighted in space to strengthen the prediction result. The loss function of the upper and lower branch networks in the tth stage is:
[0075]
[0076]
[0077] in, is the predicted keypoint confidence map for each keypoint j at stage t, is the body part position information map of the true label, W(p) is a binary function used to prevent unnecessary penalties for correct predictions during training, such as when the image lacks a label at position p, but the network is still able to predict the part position information map and part affinity vector field for the image at position p. L is the set of vector fields of the part affinity of the limbs between the predicted key points, c is the limb between two human skeleton key points, and the predicted vector field at stage t is The corresponding true value is In order to avoid the problem of gradient disappearance during training, the network will periodically supplement the gradient through a supervised method at each intermediate stage. The final global loss function is:
[0078]
[0079] The Openpose network only needs to input an RGB image of size w×h when estimating human posture. The model first encodes the connection between the key points of the human body through the feedforward network, which is then used to match the skeleton key points. After obtaining the key points of the human body in the image, it is necessary to correctly connect the key points to construct the human skeleton diagram, specifically, cluster the key points. A commonly used clustering method is to connect based on the distance between the key points. The key points of a single person can be connected according to the definition of the joint position of the human body. For the case where multiple people appear in an image, the key points of different people may be connected together when connecting the key points. In order to solve the interference of other people in sign language recognition in complex environments, the part affinity vector field is used to assist in the correct connection of the key points during prediction through the distance and direction of the key points. The Hungarian algorithm is used for clustering to obtain the optimal matching method of the key points.
[0080] More specifically, if Figure 5 As shown in the figure, the GAN network provided by the present invention consists of two parts, namely a generator network and a discriminator network. The generator uses a low-dimensional Gaussian variable as input and generates the required samples by learning the distribution of real data; the discriminator is essentially a binary classifier that discriminates the input sample data to distinguish whether it is real data or samples generated by the generator.
[0081] For example, the generator generates an image x through a gesture text label, and then inputs the generated virtual image x into the discriminator. The discriminator outputs the probability that x is a real gesture image. If the output is 1, it means that x is a real gesture image; if the output is 0, it means that x is a generated virtual gesture image. During the training process, the goal of the generator is to generate as "real" gesture images as possible to deceive the discriminator, so that the output value of the discriminator tends to 1, and the goal of the discriminator is to distinguish the virtual gesture images generated by the generator from the original real gesture images as much as possible. This means that for the generated virtual gesture images, the discriminator should try to output values that tend to 0. The generator and the discriminator conduct adversarial training with opposite purposes. By continuously iteratively optimizing the loss function and updating the model parameters of the generator and the discriminator at the same time, the generation ability of the generator and the discrimination ability of the discriminator are gradually improved until the Nash equilibrium is reached, that is, the discriminator can no longer distinguish whether the input is a real gesture or a virtual gesture generated by the generator, and its output is always The virtual data output by the generator is very close to the real data in distribution, so the generator has the ability to make the fake look real.
[0082] Therefore, during training, in order to learn the distribution P of the generator on the real sign language samples x that obey the original data distribution, g , first define a priori noise distribution P z (Z), and then the generator maps the prior noise to the sample space through the mapping function G(Z). The discriminator outputs a single scalar according to the input real sign language video or the generated sign language video, D(x) indicates that the sample x comes from the training sample instead of P g In the GAN network, the discriminator and the generator are trained alternately and iteratively, so the two can be unified into one target loss function. The training process can be regarded as optimizing this loss function, as shown below:
[0083]
[0084] Among them, V(D, G) is the cross loss commonly used in binary classification problems, x~p data (x) indicates that the sample x comes from the original data distribution p data A random sample of (x), To obey p data (x) The mathematical expectation of the distribution, z~p z (z) means that it obeys the prior noise distribution p z (z), x is the real sign language training data, z is the prior noise of the input generator, G(z) represents the video generated by the generator, D(x) represents the probability of the discriminator to distinguish whether the real sign language video is real, and D(G(z)) represents the probability of the discriminator to distinguish whether the sign language video generated by the generator is real.
[0085] Since the generator and discriminator of the GAN network are optimized alternately and iteratively during training, when exploring the optimal solution of the discriminator, you can choose to fix the generator and train the discriminator first. The purpose is to adjust its parameters to maximize the objective function. The function is as follows:
[0086]
[0087] When the distribution is continuous, the objective function can be rewritten as an integral in the sampling space, that is, integrated over x. In this way, the problem of solving the optimal discriminator is transformed into the problem of solving the maximum integral, and solving the maximum integral is equivalent to solving the maximum integral.
[0088] When the optimal solution is uncertain, the objective function of the GAN network can be rewritten as follows:
[0089]
[0090] When the discriminator reaches the optimal solution, substituting it into the above formula can obtain the objective function only about the generator:
[0091]
[0092] At this point, solving the optimal generator is equivalent to finding the minimum value of the above formula. It is easy to see from the non-negativity of KL divergence that if and only if P g (x) = P data (x), V(G, D * ) obtains the minimum value. At this time, when the generator reaches the optimal solution, the generated sample distribution is completely consistent with the real sample distribution. g (x) = P data Substituting (x) into the expression of the optimal discriminator, we can obtain:
[0093]
[0094] The above formula can be understood as the output of the optimal discriminator for any sample is That is, it is no longer possible to distinguish P data and P g The difference between the two distributions is that the generated gestures are almost the same as those in the standard library. The trained model can be used to generate gestures for any input sign language label for learners to imitate.
[0095] Step S200: Use the dual-stream VTN network to train the sign language data set to generate a sign language translation model.
[0096] like Figure 6As shown, the present invention adopts a bottom-up human posture estimation algorithm and a palm detection model to detect the sign language action video and extract the skeleton posture key point information, and construct the human skeleton posture key point map data; adopts a dual-stream VTN network to extract the global spatiotemporal feature sequence of the video sequence and the palm skeleton key point spatiotemporal feature sequence of the upper body data and palm data respectively, and performs feature fusion to obtain the global-palm spatiotemporal feature sequence; then uses the self-attention mechanism encoding and decoding network to serialize and model the spatiotemporal feature information; obtains the maximum classification probability of the action video imitated by the learner through a softmax classifier, and obtains the final recognition text after mapping; finally, uses natural language processing technology to evaluate the similarity between the recognition result and the standard library sign language, and feeds back the learner's learning effect.
[0097] The two-stream VTN network is a lightweight feature extraction network that uses a lightweight 2D CNN to process each frame of the input sign language dataset video clip into a feature vector, and uses ResNet-34 as a lightweight benchmark framework for the feature extraction model. Lightweight networks can increase the speed of network training and reduce the number of network parameters. In sign language teaching, feedback results of learners' imitation of sign language movements can be quickly obtained, improving learning efficiency.
[0098] like Figure 7 As shown in the figure, the lightweight network layer at the head clips and processes all the spatiotemporal information feature maps of the human body and the palm. The second step is the encoder of the Transformer network, which uses 2D CNN to process each frame of the input sequence separately to obtain frame embedding. The third step is the decoder, which integrates the temporal information within the frame in a feedforward manner with an attention mechanism to generate a classification label for a given Clip. Its function is to train to predict one word at a time and generate the corresponding sentence.
[0099] More specifically, the feature vector sequence in the lightweight feature extraction network is flattened into a one-dimensional feature sequence. A linear embedding layer is used to map the one-dimensional feature sequence to the latent space, and a learnable one-dimensional position embedding is added to the one-dimensional feature. The final input of the Transformer encoder is then obtained and mapped to multiple query, key, and value matrices using different learned affine transformations. key Sum Use the proportional multiplication attention mechanism to convert to the corresponding multi-head output. The formula is as follows:
[0100]
[0101] The multi-head self-attention mechanism consists of H parallel heads, each head h iCorresponding to an independent dot product attention function. Then the linear transformation W is used to summarize the attention results of all heads. The process can be described as follows:
[0102] MultiHead i (Q;K;V)=Concat(h1;h2;...h 11 )·W
[0103] The spatiotemporal feature representation of the sign language action video is learned by the encoder, and <bos>is a special token that indicates the beginning of a sentence. This process will continue until another special token is predicted to indicate the end of the sentence. <eos>until.
[0104] At the decoder end, the output generated by the encoder end is word-embedded, the position code is embedded into the word embedding feature, and the result is fused with the encoder output feature as input. d The masked multi-head self-attention sublayer calculates the input data to obtain the mask attention, which is input into the multi-head cross attention sublayer and the position feedforward sublayer together with the output of the encoder. The probability of the corresponding word is obtained through the softmax classifier.
[0105] More specifically, the output of the encoder is embedded into a word vector as the input of the decoder, and then the attention vector of the decoder input is calculated. In each time step of the decoder, the output from the encoder and the sub-attention from the decoder are processed using the same structural layer as the encoder, and the key, value, and query matrices are calculated as follows:
[0106] K=W k x
[0107] V=W v x
[0108] Q=W q x
[0109] Among them, K, V, and Q represent key, value, and query matrices respectively.
[0110] The output o of the decoder consists of the output b after the input and attention vector are connected through the residual and layer regularization, as well as the weight matrices W1, W2 and the bias terms b1, b2:
[0111] o=layerNorm(ReLU(xW1+b1)W2+b2+b)
[0112] The decoder output is linearly mapped to convert it into a non-normalized probability vector, and the softmax classifier outputs the normalized category probability value; finally, the word corresponding to the maximum probability value is found to complete the decoding:
[0113] y=softmax(linear(o)W+b)
[0114] Step S300: sending the text sentence to be recognized into the trained posture generation model to obtain a 3D virtual character action video for learners to imitate.
[0115] The present invention uses sign language dataset videos and sign language text labels to train a gesture generation network, which can generate gestures for commonly used sentences in various scenarios input by teachers and obtain 3D virtual character action videos; learners can watch the 3D virtual character action videos for imitation learning, which reduces the difficulty of teacher teaching. Figure 8 shown.
[0116] For example, in an example of the present invention, learners can follow the teacher to learn sign language movements, and can also use the present invention to imitate the learning content. Because teachers cannot take care of every student, and there are very few teachers who understand deaf-mute teaching, students can use the present invention to learn. And unlike traditional sign language teaching, learners can learn according to their own interests, and the gesture generation network can generate target images of different styles. The present invention only takes 3D virtual characters as an example.
[0117] Step S400: using the sign language translation model to identify the sign language movements imitated by the learner, evaluating the similarity between the movement imitation results and the standard teaching movements, and providing feedback on the learning effect.
[0118] In an embodiment of the present invention, a real-time evaluation model is designed for sign language teaching. First, a standard library of sign language label data is established; secondly, the recorded action video of the learner is sent to the sign language translation network for recognition and translation to obtain the recognition result; the recognition result is evaluated for similarity using the real-time evaluation model to obtain the probability and score of similarity with the action in the standard sign language library, and the feedback is given to the learner for subsequent learning.
[0119] More specifically, since the same sign language may look very different when performed by different presenters, the feature vector sequence V s The constraints are imposed, and these feature vector sequences are obtained by the head lightweight module. Then the conditional probability of the sign language label sequence is calculated as the sum k of all path probabilities that can be mapped to by the mapping operation B:
[0120]
[0121] Use the connection time series classification algorithm CTC to perform sequence alignment and obtain the recognition loss L R The weighted sum of:
[0122] L R =-ln(p(k|V s ))
[0123] During the training of the Transformer decoder model, we minimize the joint loss term L loss To train the network:
[0124] L loss =λL R +(1-λ)L T
[0125] Among them, λ is a trade-off parameter that determines the importance of each loss function.
[0126] The following formula is used to give feedback and score the recognition results:
[0127]
[0128] Where S, I, and D represent the minimum number of substitution, insertion, and deletion operations required to convert the recognition sentence into a standard library sentence, respectively. N is the number of words in the standard library sentence.
[0129] In an embodiment of the present invention, a real-time evaluation model is used to evaluate the similarity between the action imitation result and the standard teaching action to obtain a score value; then the action accuracy is graded according to the score value. Specifically, a score value higher than 90 is set as grade A, a score value between 70-90 is set as grade B, a score value between 60-70 is set as grade C, and a score value lower than 60 is set as grade D.
[0130] In a second aspect of the present invention, a sign language teaching device based on gesture generation and recognition is provided. The device performs sign language teaching according to any sign language teaching method based on gesture generation and recognition described in any one of the embodiments of the present invention. In an example of the present invention, the device includes:
[0131] A first training unit is configured to train a gesture generation model using a TG gesture generation network, a sign language dataset video and a sign language text label;
[0132] A second training unit is configured to generate a sign language translation model by training a sign language data set using a two-stream VTN network;
[0133] The gesture action generation unit is configured to send the text sentence to be recognized into the trained gesture generation model to obtain a 3D virtual character action video for learners to imitate.
[0134] The sign language translation unit is configured to use the sign language translation model to recognize the sign language movements imitated by the learner, evaluate the similarity between the movement imitation results and the standard teaching movements, and provide feedback on the learning effect.
[0135] The more specific implementation method of each unit is consistent with the above description of the method and will not be repeated here.
[0136] In a third aspect of the present invention, a sign language teaching system based on gesture generation and recognition is provided, and the system also performs sign language teaching according to any of the sign language teaching methods based on gesture generation and recognition described in the embodiments of the present invention. Fig. 9 As shown, the system includes a client and a server, wherein the client refers to collecting sign language teaching texts or learners recording action videos of learners imitating and transmitting them remotely to the server; receiving 3D virtual character action videos or learners imitating learning effects returned by the server and displaying them. The client includes functions such as data collection, information transmission and front-end display; the server refers to receiving sign language teaching texts or learners imitating action videos sent by the client, calling the information processing module to process them, and transmitting the results to the client.
[0137] Therefore, in the example of the present invention, the client includes a data acquisition module, a data transmission module and a front-end display module; wherein the data acquisition module can collect text label data of sign language teaching, and can also use an online camera to record video data of movements imitated by learners. The information transmission module can remotely transmit the collected text label data of sign language teaching or the recorded video data of movements imitated by learners to the server to generate 3D virtual character movement videos or perform recognition and scoring. The front-end display module can display 3D virtual character movement videos for learners to imitate, and can also display the score values of the learners' imitation results transmitted back by the server;
[0138] The server includes a result feedback module, an information processing module and an information storage module; the result feedback module is used to remotely transmit the 3D virtual character action video generated by the server or the processed learner imitation result to the client's front-end display module for display; the information processing module can generate 3D virtual character action video from the text label data of sign language teaching, recognize the sign language movements imitated by the learner, and provide feedback and scoring; the information storage module is used to store sign language data sets and trained posture generation models and sign language translation models.
[0139] The entire system is implemented by the teacher inputting the text to be recognized on the client and transmitting it to the server to generate a 3D virtual character action video for the learner to imitate; the client can record the learner's imitating action video in real time and transmit it to the server for recognition and scoring, realizing a complete sign language teaching system, enriching the sign language teaching model, improving the situation where teachers cannot take care of every student, and improving the quality of sign language teaching.
[0140] The above embodiments are only preferred implementation modes of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and equivalent substitutions without departing from the principles of the present invention. These technical solutions after improvements and equivalent substitutions to the claims of the present invention all fall within the protection scope of the present invention.< / eos> < / bos>
Claims
1. A sign language teaching method based on gesture generation and recognition, It is characterized in that include: The TG posture generation network is used to train the posture generation model with the sign language dataset video and the sign language text label, including: using the forward Transformer network to train the sign language dataset video and the sign language text label to obtain the posture prediction sequence; specifically, using the Encoder part of the Transformer to extract the feature of the sign language text label, extracting the text feature information, and using the following formula to provide the model with the position information: Among them, pos is the position of a word in a sentence, i represents the dimension number of the word vector, and d model Represents the dimension value of the word vector; detects the key points of human skeleton posture in the sign language dataset video and extracts the three-dimensional coordinate points; sends the extracted text features and three-dimensional coordinate points to the Decoder part of the Transformer to obtain the posture prediction sequence; uses the backward GAN network to perform alternating iterative training on the posture prediction sequence and saves the trained posture generation model; A sign language translation model is generated by training a sign language dataset using a two-stream VTN network, including: performing DNN global human feature detection on isolated word sign language videos and extracting global features; using the Openpose model to detect palm features on isolated word sign language datasets and extract palm features; fusing global features with palm features, and using a two-stream VTN network to train a continuous sentence sign language dataset to obtain a sign language translation model; The text sentence to be recognized is sent to the trained posture generation model to obtain a 3D virtual character action video for learners to imitate, including: using the posture generation model to convert the sign language text sentence to be recognized into human skeleton posture key points; using the ThreeDPose model to virtually reconstruct the 3D posture of the human skeleton key points to generate a 3D virtual character action video for learners to imitate; The sign language translation model is used to identify the sign language movements imitated by learners, and the similarity between the movement imitation results and the standard teaching movements is evaluated to provide feedback on the learning effect.
2. The method according to claim 1, It is characterized in that The method of using the backward GAN network to perform alternating iterative training on the posture prediction sequence and saving the trained posture generation model comprises: Send the forward-generated gesture prediction sequence into the Generator to obtain the virtual sign language gesture; Use the Discriminator to perform discrimination training on virtual sign language gestures and real sign language gestures; The generator and discriminator are used for alternating iterative training to make the virtual sign language gestures closer to the real sign language gestures, and a gesture generation model that cannot be distinguished by the discriminator is obtained.
3. The method according to claim 2, It is characterized in that The sign language translation model is used to identify the sign language movements imitated by the learner, and the similarity between the movement imitation results and the standard teaching movements is evaluated. The feedback of the learning effect includes: Record the learner's imitating sign language action video and perform dual-stream feature extraction on the video; The two-stream features are fed into the trained sign language translation model to generate the learner’s action imitation results; Use the real-time evaluation model to evaluate the similarity between the action imitation results and the standard teaching actions and obtain a score value; The accuracy of the movements is graded according to the score value.
4. A sign language teaching device based on gesture generation and recognition, which performs sign language teaching according to the sign language teaching method based on gesture generation and recognition according to any one of claims 1 to 3, It is characterized in that include: The first training unit is configured to use the TG posture generation network to train the posture generation model with the sign language dataset video and the sign language text label, including: using the forward Transformer network to train the sign language dataset video and the sign language text label to obtain the posture prediction sequence; specifically, using the Encoder part of the Transformer to extract features of the sign language text label, extracting text feature information, and using the following formula to provide the model with position information: Among them, pos is the position of a word in a sentence, i represents the dimension number of the word vector, and d model Represents the dimension value of the word vector; detects the key points of human skeleton posture in the sign language dataset video and extracts the three-dimensional coordinate points; sends the extracted text features and three-dimensional coordinate points to the Decoder part of the Transformer to obtain the posture prediction sequence; uses the backward GAN network to perform alternating iterative training on the posture prediction sequence and saves the trained posture generation model; The second training unit is configured to use a two-stream VTN network to train a sign language data set to generate a sign language translation model, including: performing DNN global human feature detection on an isolated word sign language video to extract global features; using an Openpose model to perform palm feature detection on the isolated word sign language data set to extract palm features; fusing information between the global features and the palm features, and using a two-stream VTN network to train a continuous sentence sign language data set to obtain a sign language translation model; The gesture action generation unit is configured to send the text sentence to be recognized into the trained gesture generation model to obtain a 3D virtual character action video for the learner to imitate, including: using the gesture generation model to convert the sign language text sentence to be recognized into the key points of the human skeleton gesture; using the ThreeDPose model to virtually reconstruct the 3D gestures of the key points of the human skeleton to generate a 3D virtual character action video for the learner to imitate; The sign language translation unit is configured to use the sign language translation model to recognize the sign language movements imitated by the learner, evaluate the similarity between the movement imitation results and the standard teaching movements, and provide feedback on the learning effect.
5. A sign language teaching system based on gesture generation and recognition, which performs sign language teaching according to the sign language teaching method based on gesture generation and recognition according to any one of claims 1 to 3, It is characterized in that Including client and server: The client includes a data acquisition module, a data transmission module and a front-end display module; The data collection module is used to collect text label data of sign language teaching, and use an online camera to record video data of movements imitated by learners; The data transmission module is used to remotely transmit the collected text label data of the sign language teaching or the recorded action video data of the learner imitating to the server to generate a 3D virtual character action video or perform recognition and scoring; The front-end display module is used to display the 3D virtual character action video for the learner to imitate, and can also display the score value of the learner's imitation result transmitted back by the server; The server includes a result feedback module, an information processing module and an information storage module; The result feedback module is used to remotely transmit the 3D virtual character action video generated by the server or the processed learner imitation result to the front-end display module of the client for display; The information processing module is used to generate 3D virtual character action videos from the text label data of sign language teaching, and can also identify the sign language actions imitated by the learners and provide feedback and scoring; The information storage module is used to store the sign language data set and the trained gesture generation model and sign language translation model.
