Three-dimensional sign language teaching system, method and equipment based on augmented reality

Through augmented reality technology and SMPL-X model, a three-dimensional sign language teaching system is implemented, solving the problem that two-dimensional teaching is difficult to display sign language movement details and lack of immediate feedback, and improving learning effect and efficiency.

CN119028206APending Publication Date: 2024-11-26BEIJING NORMAL UNIVERSITY
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411138627.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing online sign language teaching methods are mainly two-dimensional displays, which are difficult to display sign language movement details in all aspects, resulting in poor learning results and lack of immediate feedback, making it difficult for students to evaluate their own learning effects.

Method used

Adopting a three-dimensional sign language teaching system based on augmented reality, the sign language movements of teachers and students are collected through Hololens equipment, and the three-dimensional human skeleton motion data on the server side drives three-dimensional virtual teachers for teaching, and provides instant feedback scores, combining SMPL-X model and DTW algorithm Perform motion capture and evaluation.

Benefits of technology

It improves the authenticity and interactivity of sign language teaching, helps students to master sign language movements more clearly, improve learning efficiency, correct mistakes in a timely manner, and enhance learning flexibility and convenience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028206A_ABST
    Figure CN119028206A_ABST
Patent Text Reader

Abstract

The invention discloses a three-dimensional sign language teaching system, method and equipment based on augmented reality, two-dimensional monocular image information of sign language actions of teachers and students is collected through a first terminal and a second terminal, and three-dimensional virtual teacher teaching is driven at a server side according to three-dimensional skeleton motion data extracted from the images. Meanwhile, real-time calculation is carried out, evaluation of sign language actions of students is obtained and fed back to online learning students, the three-dimensional sign language teacher model can more clearly display details and actions of gestures, the students are helped to better master sign languages, learning efficiency is improved, the sign language evaluation is fed back in time so that the students can rapidly position error points, and learning efficiency is improved. The problem that errors are difficult to correct after being accumulated is avoided, students can learn by using the portable equipment anytime and anywhere without being limited by time and places, and the learning flexibility and convenience are effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of digital teaching, and relates to a three-dimensional sign language teaching system, method and equipment based on augmented reality. Background Art

[0002] The communication and learning methods of the hearing-impaired are rather special. They generally communicate through sign language, which is also the main tool for teaching and communication in special education schools. Traditional offline sign language teaching has problems such as a shortage of excellent sign language teachers and students' lack of concentration. This makes online teaching occupy a very important position. It breaks the limitations of resources and regions and improves the effect of traditional teaching to a certain extent. However, the main form of online teaching is currently two-dimensional image display, and the details of sign language movements from multiple angles are difficult to display. At the same time, due to special education reasons, it is more difficult for students with different receptive and learning abilities to understand, and the teaching effect varies greatly. In addition, there is a problem of no immediate feedback on results in traditional online learning, and it is difficult for students to judge the effect of their own sign language learning. Generally speaking, the current mainstream online sign language teaching methods are not accurate, and students have greater difficulty in learning, and the learning effect is not good.

[0003] For example, the patent application document with Chinese patent application number CN201811107389.4 discloses an AR sign language recognition method and teaching method based on electromyographic motion perception. By wearing a SEMG armband, collecting electromyographic signals, wavelet transform, filtering noise, etc., specific sign language movements are identified and then compared with the standard sign language library movements. Although the effect of real-time feedback correction can be achieved, the low accuracy will lead to learning difficulties.

[0004] For example, the patent application document with Chinese patent application number CN202211691075.X discloses a sign language teaching method, device, equipment and storage medium, which performs posture detection on the learner's sign language video sequence frames, and then compares the coordinate vector similarity with the template skeleton point sequence to provide learners with more fine-grained guidance. However, the learning of two-dimensional sign language videos has the problem that sign language movements cannot be fully displayed, which makes it difficult to improve learners' enthusiasm.

[0005] For example, the patent application document with Chinese patent application number CN202210029311.5 discloses a sign language teaching method, device and system based on gesture generation and recognition. The forward Transformer network is used to obtain a posture prediction sequence from a sign language dataset video, and then the backward GAN (Generative Adversarial Nets) network is used to perform alternate iterative training on the posture prediction sequence to obtain a posture generation model. The ThreeDPose posture transfer technology is used to generate 3D (3-Demesion) virtual character action videos for sign language teaching for learners to imitate, and the learners' movements are evaluated for similarity with the standard library sign language teaching movements, and the learning effect is fed back. However, this method has technical problems such as low realism and interactivity.

[0006] Based on the above technical problems existing in the prior art, the present invention proposes a three-dimensional sign language teaching system, method and device based on augmented reality. Summary of the invention

[0007] The purpose of the present invention is to expand online sign language teaching from two dimensions to three dimensions in response to the deficiencies of the prior art. Students can wear Hololens devices, choose the sign language movements they want to learn, and with the help of virtual teachers and virtual teaching aids, they can get instant feedback and scores after finishing learning, which can improve learning effects.

[0008] The present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a three-dimensional sign language teaching system based on augmented reality, comprising:

[0010] The first terminal is used to collect the teacher's sign language teaching movements;

[0011] The second terminal is used to collect students’ sign language movements;

[0012] The server side is used to receive the two-dimensional monocular image information generated by the first terminal and the second terminal, and obtain three-dimensional human skeleton movement data;

[0013] The augmented reality sign language classroom module compiles augmented reality courses based on three-dimensional human skeletal movement data, drives three-dimensional virtual teachers, and performs real-time calculations of courses. It evaluates students' sign language movements after learning and provides targeted feedback to optimize learning outcomes.

[0014] Furthermore, the augmented reality sign language classroom module includes a 3D virtual character driving module, a learning feedback module, a gesture interaction module and a storage module, wherein the 3D virtual character driving module is used to drive the sign language teacher to perform sign language teaching demonstrations; in the learning feedback module and the gesture interaction module, students learn by observing the teacher's hand movements in all directions through gestures of zooming in, zooming out and moving; after completing the learning, students upload the sign language movement video to the storage module for storage, and then the server generates the evaluation feedback results.

[0015] Furthermore, the server side includes a sign language gesture recognition module, which is used to receive two-dimensional monocular image information generated by the first terminal and the second terminal, calculate and extract three-dimensional human skeleton movement data and transmit the data to the augmented reality sign language classroom module, which is used for three-dimensional virtual character driving and learning movement evaluation calculation in the augmented reality sign language classroom module.

[0016] Furthermore, the first terminal includes a real-time classroom two-dimensional monocular image acquisition module, which is used to display classroom teaching blackboard information and extract teacher's three-dimensional skeletal movement data in the augmented reality sign language classroom module.

[0017] Furthermore, the second terminal includes a real-time two-dimensional monocular image acquisition module for obtaining images of students imitating and learning sign language, which is used to extract three-dimensional skeletal movement data and compare it with the teacher's demonstration to evaluate the learning effect.

[0018] Furthermore, the sign language gesture recognition module uses a multi-point rapid gesture reconstruction system based on the SMPL-X model and quaternion-based spherical linear interpolation to perform full-body 3D sign language motion capture on sign language action videos, and outputs smoothed and optimized three-dimensional sign language motion data.

[0019] In a second aspect, the present invention provides a three-dimensional sign language teaching method based on augmented reality, comprising:

[0020] Acquire three-dimensional sign language motion data by performing sign language posture recognition on the server side on the sign language teaching action acquired by the first terminal;

[0021] In the augmented reality sign language classroom module, the 3D sign language motion data is used to drive the 3D virtual teacher to perform sign language teaching in the second terminal, and the second terminal transmits the learning video to the server to obtain the 3D sign language learning motion data;

[0022] The learning action evaluation is carried out in the augmented reality sign language classroom module, and the learning feedback is obtained and displayed on the second terminal.

[0023] Furthermore, the method comprises the step of posture reconstruction:

[0024] S11, use the SMPL-X model to parameterize the hands, face and body of the three-dimensional human body, and construct a computable human body in three-dimensional space:

[0025] V ω =W(φ ω ,θ ω ,β ω ,ψ ω )……(1),

[0026] In the SMPL-X parameter model V, the operator W is the parameter Human body global posture parameters Facial expression parameters and facial cross-feature variable parameters Composition, where R represents a dimension vector;

[0027] S12, hand posture estimation, hand posture capture module M H Built on an end-to-end deep neural network architecture to predict the defined hand model parameters:

[0028] [φ,θ,β,c]=M H (I H )……(2),

[0029] Among them, I H It is the hand cropping area based on the 2D human key points after the monocular RGB sign language video input, and the global direction of the hand grid Hand estimation parameters Hand model shape parameters c = (t, s) is a set of weak perspective camera parameters used to project a fixed-pose 3D hand model onto the input image, t∈R is the 2D translation on the image plane, and s∈R is the scale factor;

[0030] S13, face and body pose estimation, using a 3D face estimation model encoder-decoder VOCA network to estimate facial expression φ ω and facial posture ψ ω ;

[0031] S14, using the optimization algorithm framework to fit the hand and body modules to estimate the global human body parameters, and perform 3D human body posture integration to obtain the complete SMPL-X 3D human body model parameters, where the optimization objective function is composed of three constraints:

[0032]

[0033] in, is the 2D reprojection loss term between the 2D keypoint estimates and the projections of the 3D joints (hands, body, and face), is the 3D distance loss between the hand mesh of the full body model and the mesh vertices output by the hand module located at the wrist joint of the body, It is a prior regularization term that keeps the 3D pose and shape parameters in a reasonable space, satisfying:

[0034]

[0035] Furthermore, a joint adaptive motion evaluation algorithm based on dynamic time warping (DTW) is used for real-time calculation to obtain student learning feedback results, including:

[0036] S21, extract the action features based on cascaded quaternions with adaptive joint weights, let:

[0037]

[0038] is the vector of the relative magnitude of all joint energies in the nth action segment of the reference action sequence P. Let the joint motion of each action segment be Obeying Gaussian distribution, we can calculate The information entropy is used to represent the relative value of the motion energy. The motion energy of the joints is arranged in descending order. The vector T n The joint IDs of the nth segment motion energy are sorted in descending order, and T is taken n The first H joint IDs of T, the joint weights will be calculated based on each joint ID in T H The number of occurrences is calculated, and the proportion of the fragment with the largest energy of joint i to all fragments is defined as:

[0039]

[0040] Among them, the count function calculates T H The number of times the i-th joint appears in the image, press S for the joints with more intense movements i For weight distribution, the proportion of joints to all weights is α, and the remaining weights are evenly distributed to the remaining joints. i Sort in descending order, take S i The first m elements of until the sum of these elements is greater than α, add the joint IDs corresponding to these elements to the set C, and calculate the weight of each joint:

[0041]

[0042] S22, based on DTW to obtain the frame matching relationship, use the time warping function that meets the conditions to describe the corresponding relationship between the teacher's key point action sequence and the student's key point action sequence, solve the regularization path corresponding to the minimum cumulative distance during matching, and convert the obtained teacher's key point coordinate motion sequence and student's key point coordinate motion sequence into key point angle change sequences P and Q, and construct an n*m matrix M, in which the element M in the i-th row and j-th column is i,j , is p i With q j The distance d(p i ,q j ), the path W to be calculated is used to represent the mapping relationship between the time series P and Q, and the kth element is defined as:

[0043] w k =(i,j) k ……(8),

[0044] Create an array MapFrame, record the unique mapping relationship of all frames in P in Q, traverse the path W, if there are frames in P that have a one-to-many relationship with Q, select a pair of matching frames with the smallest distance and store the mapping relationship in MapFrame; if not, directly store the mapping relationship of the matching frames in MapFrame. MapFrame records the matching relationship between action sequences P and Q. Use the frame number of P as the array subscript to obtain the matching frame number in Q and the minimum distance sum:

[0045]

[0046] S23, calculate the action sequence similarity and obtain evaluation:

[0047]

[0048] Among them, θ is the key point angle feature vector, t(θ) is the DTW distance cumulative value of the key point angle, b(θ) is the cardinality of the key point angle feature, f(θ) is the weight of the corresponding key point angle feature, that is, ω in the first step, s is the full score of the action, SCORE(θ) is the final score of the key point angle feature, β is the number of matching frames of the action, and γ is the number of benchmark frames of the key point.

[0049] In a third aspect, the present invention provides an electronic device comprising one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the operations performed by the augmented reality-based three-dimensional sign language teaching method.

[0050] The beneficial effects of the present invention are:

[0051] The three-dimensional sign language teaching system, method and device based on augmented reality described in the present invention collects two-dimensional monocular image information of the sign language movements of teachers and students through the first terminal and the second terminal, and drives the three-dimensional virtual teacher teaching on the server side according to the three-dimensional skeletal motion data extracted from the image, and performs real-time calculation at the same time to obtain the evaluation of the student's sign language movement and feed it back to the students who are learning online. The three-dimensional sign language teacher model can more clearly display the details and movements of the gestures, help students better master sign language, and improve learning efficiency. Timely sign language evaluation feedback allows students to quickly locate error points and avoid problems that are difficult to correct after accumulated errors. Students can use portable devices for learning anytime and anywhere without being restricted by time and place, and the flexibility and convenience of learning are effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a schematic diagram of a three-dimensional sign language teaching system based on augmented reality in an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of an augmented reality sign language classroom module in an embodiment of the present invention;

[0054] Figure 3 Schematic diagram of multi-point rapid posture reconstruction in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0056] Example

[0057] like Figure 1 As shown, the three-dimensional sign language teaching system based on augmented reality includes a first terminal (teacher terminal), a second terminal (student terminal), a server terminal, and an augmented reality sign language classroom module. The sign language teaching action obtained by the first terminal is used to perform sign language posture recognition on the server terminal to obtain three-dimensional sign language motion data, and the three-dimensional sign language motion data is used in the augmented reality sign language classroom module to drive a three-dimensional virtual teacher to be displayed on the second terminal for sign language teaching. At the same time, the second terminal can transmit the learning video to the server terminal to also obtain the three-dimensional sign language learning motion data, and the learning action evaluation is performed in the augmented reality sign language classroom module, and the learning feedback is obtained and displayed on the first terminal and the second terminal;

[0058] The first terminal and the second terminal use the motion acquisition module to obtain two-dimensional monocular image information of sign language movements, and transmit the information to the server side, and the server side obtains three-dimensional sign language movement data according to the sign language posture recognition module;

[0059] Exemplarily, the first terminal uses a professional camera, such as Canon EOS R5; the second terminal uses a webcam or a mobile phone camera;

[0060] The three-dimensional sign language teaching method based on augmented reality comprises:

[0061] Acquire three-dimensional sign language motion data by performing sign language posture recognition on the server side on the sign language teaching action acquired by the first terminal;

[0062] In the augmented reality sign language classroom module, the 3D sign language motion data is used to drive the 3D virtual teacher to perform sign language teaching in the second terminal, and the second terminal transmits the learning video to the server to obtain the 3D sign language learning motion data;

[0063] Conduct learning action evaluation in the augmented reality sign language classroom module, and obtain learning feedback to be displayed on the first terminal and the second terminal;

[0064] As a specific implementation method, the sign language gesture recognition module in the server side uses a multi-point fast gesture reconstruction system based on the SMPL-X model and spherical linear interpolation based on quaternions to perform full-body 3D sign language motion capture on the sign language action video, and outputs smooth and optimized three-dimensional sign language motion data;

[0065] like Figure 3 As shown, the multi-point fast posture reconstruction based on the SMPL-X model includes:

[0066] Use the SMPL-X model to parameterize the human body in three-dimensional space, including hands, face, and body, and construct a computable human body in three-dimensional space:

[0067] V ω =W(φ ω ,θ ω ,β ω ,ψ) ω ,

[0068] In the SMPL-X parameter model V, the operator W is the parameter Human body global posture parameters Facial expression parameters and facial cross-feature variable parameters Composition, where R represents the dimension vector;

[0069] Hand pose estimation, hand pose capture module M H Built on an end-to-end deep neural network architecture that is able to predict the defined hand model parameters:

[0070] [φ,θ,β,c]=MH (I H ),

[0071] Among them, I H It is the hand cropping area based on the 2D human key points after the monocular RGB sign language video input, and the global direction of the hand grid Hand estimation parameters Hand model shape parameters c = (t, s) is a set of weak perspective camera parameters used to project a fixed-pose 3D hand model onto the input image, t∈R is the 2D translation on the image plane, and s∈R is the scale factor;

[0072] It should be noted that the hand module M based on the principle of HMR practice H An encoder-decoder structure is used, where the encoder uses ResNet50 to output features from the input image, and the decoder regresses the hand model parameters from the features;

[0073] Facial and body pose estimation, using a 3D face estimation model encoder-decoder VOCA (Voice Operated Character Animation, voice-controlled 3D character animation) network to estimate facial expression φ ω and facial posture ψ ω ,Since the network is based on the 3D face model FLAME, its prediction is compatible with the face part of SMPL-X, the first 10 expression parameters of the predicted original 50-dimensional expression parameters are retained, which is compatible with the expression space defined in SMPL-X, and the 10 most important principal components are used based on the same PCA (Principal Component Analysis) expression space as FLAME;

[0074] The optimization algorithm framework is used to fit the hand and body modules to estimate the global human body parameters, and the three-dimensional human posture integration is performed to obtain the complete SMPL-X three-dimensional human model parameters. The optimization objective function is composed of three restrictions:

[0075]

[0076] in, is the 2D reprojection loss term between the 2D keypoint estimates and the projections of the 3D joints (hands, body, and face), is the 3D distance loss between the hand mesh of the full body model and the mesh vertices output by the hand module located at the wrist joint of the body, It is a prior regularization term that keeps the 3D pose and shape parameters in a reasonable space, satisfying:

[0077]

[0078] A Transformer module is introduced in the optimization processing of the hand occlusion part, and a multi-layer multi-head self-attention module is used to extract high-dimensional temporal feature information using a multi-layer perceptron and LN (laryernormalization); a Transformer module containing the motion tree relationship between different joints is introduced as a comprehensive consideration of the spatial relationship, and the local motion information between the hand joints and the global motion information between the arm and the hand are fused; finally, the information between single frames is input into the regression head RegressionHead to complete the final prediction;

[0079] The objective function used to guide model training mainly consists of two parts: one is the reconstruction objective function and the other is the adversarial objective function:

[0080]

[0081] in, is a generator, is the discriminator, Represents the generator Minimize the optimization to generate data that is as realistic as possible. Represents the discriminator Maximize optimization to accurately identify the generated data;

[0082] Adversarial Objective Function The main purpose of is to guide the model to output realistic arm and hand movements, where is the basic form of GAN loss, λ is the weight of this loss, It is expressed as:

[0083]

[0084] The reconstruction objective function is used to learn the bone rotation, which mainly consists of three parts: L1loss, FKloss and inter-frame smoothing loss:

[0085]

[0086] The basic target of reconstruction loss is L1loss, Y represents the real data, and K represents the rotation posture generated by the generator:

[0087]

[0088] Considering that when the rotation of a posture is represented in the form of a skeleton tree, the importance of different joints is different. For example, the rotation of the parent node will directly affect the position of the child node. Therefore, when the same rotation error acts on the parent and child nodes, the overall error caused by the parent node is greater. FKloss is introduced, that is, the output rotation is calculated by the forward kinematics function. The function calculates the joint point position and uses it with the groundtruth joint point position to calculate the loss:

[0089]

[0090] Taking into account the continuity between frames, the inter-frame smoothing loss is also introduced:

[0091]

[0092] After the server obtains the three-dimensional motion data of the teacher and the student according to the above method, it will be transmitted to the augmented reality sign language classroom module, and the teacher's three-dimensional motion data will be redirected by the 3D virtual character driving module and driven by the teacher model in augmented reality. The three-dimensional motion data of the student will be calculated in real time with the three-dimensional motion data of the teacher through the learning feedback module to obtain learning feedback;

[0093] As a specific implementation method, in the 3D virtual character driving module, the 3D sign language action model fbx file saved in the Autodesk Filmbox format contained in the teacher's 3D motion data is used as input, and the selected standard skeleton model is used as a bridge to match the input sign language posture model and the 3D virtual teacher character model in the augmented reality environment with the standard model, and the matching result is presented based on the skeleton structure of the standard model, so as to virtually reconstruct the 3D posture in the augmented reality, generate the 3D virtual character sign language action and display it in the second terminal for learners to learn and imitate;

[0094] As a specific implementation method, in the learning feedback module, the motion sequence file of the three-dimensional human joint points contained in the three-dimensional motion data of the teacher and the student is used as input, and the joint adaptive motion evaluation algorithm based on DTW (Dynamic Time Warping) is used for real-time calculation to obtain the student learning feedback result, and it is displayed on the first terminal and the second terminal;

[0095] The joint adaptive motion evaluation algorithm based on DTW is divided into:

[0096] The first step is to extract the action features based on cascaded quaternions with adaptive joint weights, let:

[0097]

[0098] is the vector of the relative magnitude of all joint energies in the nth action segment of the reference action sequence P. Assuming that each joint motion in each action segment Obeying Gaussian distribution, we can calculate The information entropy is used to represent the relative value of the motion energy, so the variance representation function of the joint motion is selected. Arrange the joint motion energy in descending order, vector T n The joint IDs of the nth segment motion energy are sorted in descending order, and T is taken n The first H joint IDs of T, the joint weights will be calculated based on each joint ID in T H The number of occurrences is calculated, and the proportion of the fragment with the largest energy of joint i to all fragments is defined as:

[0099]

[0100] Among them, the count function calculates T H The number of times the i-th joint appears in the , due to slight movement or even stationary joints S i Close to 0, it also needs to occupy a certain weight when evaluating the action similarity. For joints with more intense movements, press S i The weight distribution is performed, and the proportion of these joints to all weights is α. The remaining weights are evenly distributed to the remaining joints. i Sort in descending order, take the first m elements of Sx until the sum of these elements is greater than α, add the joint IDs corresponding to these elements to the set C, and calculate the weight of each joint:

[0101]

[0102] The second step is to obtain the frame matching relationship based on DTW. DTW is based on the idea of ​​dynamic programming. It uses a time-warping function that meets certain conditions to describe the corresponding relationship between the teacher's key point action sequence and the student's key point action sequence. It solves the regularized path corresponding to the minimum cumulative distance when the two are matched. According to the obtained teacher's key point coordinate motion sequence and student's key point coordinate motion sequence, it is converted into the key point angle change sequence P and Q;

[0103] It should be noted that in order to align the sequences P and Q, an n*m matrix M is constructed, in which the element M in the i-th row and j-th column is i,j , is p i With q j The distance d(p i ,q j ), the path W to be calculated is used to represent the mapping relationship between the time series P and Q, and the kth element is defined as:

[0104] wk =(i,j) k ,

[0105] Create an array MapFrame to record the unique mapping relationship of all frames in P in Q, traverse the path W, and if there are frames in P that have a one-to-many relationship with Q, select a pair of matching frames with the smallest distance and store their mapping relationship in MapFrame; if not, directly store the mapping relationship of the matching frame in MapFrame. MapFrame records the matching relationship between action sequences P and Q. Use the frame number of P as the array subscript to obtain the matching frame number in Q and the minimum distance sum:

[0106]

[0107] The third step is to calculate the similarity of the action sequence and obtain an evaluation. The movement degree and movement difference of each joint are studied one by one. A set of sign language learning action evaluation algorithms is designed through experimental data, which can be expressed using mathematical formulas such as:

[0108]

[0109] Where θ is the key point angle feature vector, t(θ) is the DTW distance cumulative value of the key point angle, b(θ) is the cardinality of the key point angle feature, f(θ) is the weight of the corresponding key point angle feature, i.e. ω in the first step, s is the full score of the action, SCORE(θ) is the final score of the key point angle feature, β is the matching frame number of the action, and γ is the benchmark frame number of the key point;

[0110] As a specific implementation method, the method calculates 36 key points of the upper body including both hands, and the final learning evaluation score can be obtained by accumulating SCORE(θ);

[0111] like Figure 2 As shown, the augmented reality sign language classroom module includes a 3D virtual character driving module, a learning feedback module, a gesture interaction module and a storage module, wherein the 3D virtual character driving module is used to drive the sign language teacher to perform sign language teaching demonstrations. In the learning feedback module and the gesture interaction module, students learn by observing the teacher's hand movements in all directions through gestures of zooming in, zooming out and moving. After learning is completed, students upload the sign language movement video to the storage module for storage, and then the server generates evaluation feedback results.

[0112] As a specific implementation method, the gesture interaction module designs single-hand selection interaction, two-hand zoom interaction and two-hand rotation interaction modes for the augmented reality device HoloLens2, which is used for the second terminal to interact with UI components and 3D virtual teachers in the augmented reality environment, select sign language course content, observe sign language movements from multiple angles, and click "start imitation" and "end imitation" buttons to evaluate learning effects;

[0113] As a specific implementation method, the storage module is used to store the sign language teaching movement data conducted by the teacher and the feedback results obtained from the sign language learning evaluation of the students, which will be stored in the augmented reality sign language classroom module, so that students and teachers can view the history through the second terminal and the first terminal, which is convenient for repeated learning and learning improvement.

[0114] The electronic device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the computer program is loaded and executed by the one or more processors to implement the operations performed by the augmented reality-based three-dimensional sign language teaching method.

[0115] The present invention is not limited by the above embodiments. The above embodiments and descriptions are only for explaining the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, which are within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the appended claims.

Claims

1. A three-dimensional sign language teaching system based on augmented reality, characterized in that: include: The first terminal is used to collect the teacher's sign language teaching movements; The second terminal is used to collect students’ sign language movements; The server side is used to receive the two-dimensional monocular image information generated by the first terminal and the second terminal, and obtain three-dimensional human skeleton movement data; The augmented reality sign language classroom module compiles augmented reality courses based on three-dimensional human skeletal movement data, drives three-dimensional virtual teachers and virtual students, performs real-time calculations of courses, and provides evaluation and feedback on students' sign language movements after learning.

2. The three-dimensional sign language teaching system based on augmented reality according to claim 1, characterized in that: The augmented reality sign language classroom module includes a 3D virtual character driving module, a learning feedback module, a gesture interaction module and a storage module. Among them, the 3D virtual character driving module is used to drive the sign language teacher to demonstrate sign language teaching. In the learning feedback module and the gesture interaction module, students learn by observing the teacher's hand movements in all directions through gestures of zooming in, zooming out and moving. After completing the learning, the students upload the sign language movement video to the storage module for storage, and then the server generates the evaluation feedback results.

3. The three-dimensional sign language teaching system based on augmented reality according to claim 2, characterized in that: The server side includes a sign language gesture recognition module, which is used to receive the two-dimensional monocular image information generated by the first terminal and the second terminal, calculate and extract the three-dimensional human skeleton movement data and transmit the data to the augmented reality sign language classroom module for the three-dimensional virtual character driving and learning action evaluation calculation in the augmented reality sign language classroom module, wherein: The first terminal includes a real-time classroom two-dimensional monocular image acquisition module, which is used to display classroom teaching blackboard information and extract teacher's three-dimensional skeletal movement data in the augmented reality sign language classroom module; The second terminal includes a real-time two-dimensional monocular image acquisition module for characters, which obtains images of students imitating and learning sign language, and is used to extract three-dimensional skeletal movement data, which is compared with the teacher's demonstration to evaluate the learning effect.

4. The three-dimensional sign language teaching system based on augmented reality according to claim 3, characterized in that: The sign language gesture recognition module uses a multi-point fast gesture reconstruction system based on the SMPL-X model and quaternion-based spherical linear interpolation to perform full-body 3D sign language motion capture on sign language action videos, and outputs smoothed and optimized three-dimensional sign language motion data.

5. A teaching method applied to the three-dimensional sign language teaching system based on augmented reality as described in any one of claims 1 to 4, characterized in that: include: Acquire three-dimensional sign language motion data by performing sign language posture recognition on the server side on the sign language teaching action acquired by the first terminal; In the augmented reality sign language classroom module, the 3D sign language motion data is used to drive the 3D virtual teacher to perform sign language teaching in the second terminal, and the second terminal transmits the learning video to the server to obtain the 3D sign language learning motion data; The learning action evaluation is performed in the augmented reality sign language classroom module, and the learning feedback is obtained and displayed on the first terminal and the second terminal.

6. The teaching method according to claim 5, characterized in that: The server uses a multi-point rapid posture reconstruction system based on the SMPL-X model and quaternion-based spherical linear interpolation to perform full-body 3D sign language motion capture on sign language action videos, and outputs smooth and optimized 3D sign language motion data.

7. The teaching method according to claim 6, characterized in that: The method comprises the steps of posture reconstruction: S11, use the SMPL-X model to parameterize the hands, face and body of the three-dimensional human body, and construct a computable human body in three-dimensional space: V ω =W(φ ω ,i ω ,b ω ,ψ ω )……(1), In the SMPL-X parameter model V, the operator W is the parameter Human body global posture parameters Facial expression parameters and facial cross-feature variable parameters Composition, where R represents the dimension vector; S12, hand posture estimation, hand posture capture module M H Built on an end-to-end deep neural network architecture to predict the defined hand model parameters: [φ,θ,β,c]=M H (I H )……(2), Among them, I H It is the hand cropping area based on the 2D human key points after the monocular RGB sign language video input, and the global direction of the hand grid Hand estimation parameters Hand model shape parameters c = (t, s) is a set of weak perspective camera parameters used to project a fixed-pose 3D hand model onto the input image, t∈R is the 2D translation on the image plane, and s∈R is the scale factor; S13, face and body pose estimation, using a 3D face estimation model encoder-decoder VOCA network to estimate facial expression φ ω and facial posture ψ ω ; S14, using the optimization algorithm framework to fit the hand and body modules to estimate the global human body parameters, and perform 3D human body posture integration to obtain the complete SMPL-X 3D human body model parameters, where the optimization objective function is composed of three constraints: where is the 2D reprojection loss term between the 2D keypoint estimate and the projection of the 3D joint, is the 3D distance loss between the hand mesh of the full body model and the mesh vertices output by the hand module located at the wrist joint of the body, It is a prior regularization term that keeps the 3D pose and shape parameters in a reasonable space, satisfying:

8. The teaching method according to claim 7, characterized in that: The joint adaptive motion evaluation algorithm based on dynamic time normalization (DTW) is used for real-time calculation to obtain student learning feedback results, including: S21, extract the action features based on cascaded quaternions with adaptive joint weights, let: is the vector of the relative magnitude of all joint energies in the nth action segment of the reference action sequence P. Let the joint motion of each action segment be Obeying Gaussian distribution, we can calculate The information entropy is used to represent the relative value of the motion energy. The motion energy of the joints is arranged in descending order. The vector T n The joint IDs of the nth segment motion energy are sorted in descending order, and T is taken n The first H joint IDs of T, the joint weights will be calculated based on each joint ID in T H The number of occurrences is calculated, and the proportion of the fragment with the largest energy of joint i to all fragments is defined as: Among them, the count function calculates T H The number of times the i-th joint appears in the image, press S for the joints with more intense movements i For weight distribution, the proportion of joints to all weights is α, and the remaining weights are evenly distributed to the remaining joints. i Sort in descending order, take S i The first m elements of until the sum of these elements is greater than α, add the joint IDs corresponding to these elements to the set C, and calculate the weight of each joint: S22, based on DTW to obtain the frame matching relationship, use the time warping function that meets the conditions to describe the corresponding relationship between the teacher's key point action sequence and the student's key point action sequence, solve the regularization path corresponding to the minimum cumulative distance during matching, and convert the obtained teacher's key point coordinate motion sequence and student's key point coordinate motion sequence into key point angle change sequences P and Q, and construct an n*m matrix M. In the matrix M, the element M in the i-th row and j-th column is i,j , is p i With q j The distance d(p i ,q j ), the path W to be calculated is used to represent the mapping relationship between the time series P and Q, and the kth element is defined as: w k =(i,j) k ……(8), Create an array MapFrame, record the unique mapping relationship of all frames in P in Q, traverse the path W, if there are frames in P that have a one-to-many relationship with Q, select a pair of matching frames with the smallest distance and store the mapping relationship in MapFrame; if not, directly store the mapping relationship of the matching frames in MapFrame. MapFrame records the matching relationship between action sequences P and Q. Use the frame number of P as the array subscript to obtain the matching frame number in Q and the minimum distance sum: S23, calculate the action sequence similarity and obtain evaluation: Among them, θ is the key point angle feature vector, t(θ) is the DTW distance cumulative value of the key point angle, b(θ) is the cardinality of the key point angle feature, f(θ) is the weight of the corresponding key point angle feature, that is, ω in the first step, s is the full score of the action, SCORE(θ) is the final score of the key point angle feature, β is the number of matching frames of the action, and γ is the number of benchmark frames of the key point.

9. An electronic device, characterized in that: It includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the one computer program is loaded and executed by the one or more processors to implement the operations performed by the teaching method described in any one of claims 5-8.

Citation Information

Patent Citations

  • Deep neural network AR sign language translation learning method, client and server

    CN108427910A

  • Method for evaluating standard motion of human body

    CN108764120A

  • Gesture language teaching method, device and system based on gesture action generation and recognition

    CN114842547A

  • Gesture language teaching system based on gesture recognition

    CN114842712A