A sign language translation training system based on gesture tracking and a sign language tracking device

Abstract

The present invention relates to the technical field of sign language translation, and specifically to a sign language translation training system and a sign language tracking device based on gesture tracking. The system includes a gesture tracking module, a skeletal structure completion module, a gesture standard comparison module, a gesture dynamic correction module, and a training effect analysis module. The present invention, by screening visible skeletal joints, detecting spatial displacement trends and calculating motion rates and direction change rates, can effectively extract key frame data that meets stability standards, improve the accuracy of gesture trajectory analysis, use skeletal structure constraints to complete missing key points, and avoid interference with recognition results caused by missing key points. By calculating the joint angle change rate and analyzing the deviation range, dynamic posture correction is performed in combination with the finger opening and closing angle and the wrist rotation angle, and adjustment suggestions are provided based on the correction angle, thereby achieving optimized guidance for sign language training and improving the pertinence and refinement of sign language training.

Description

Technical Field

[0001] The present invention relates to the technical field of sign language translation, and in particular to a sign language translation training system and a sign language tracking device based on gesture tracking. Background Art

[0002] The field of sign language translation technology involves computer vision, gesture recognition, natural language processing, and human-computer interaction technologies. Its goal is to convert sign language into text or speech to enable barrier-free communication between hearing-impaired and hearing individuals. This technology relies on core components such as gesture detection, gesture classification, semantic parsing, and language modeling, and is widely used in sign language translation equipment, education and training, intelligent interactive systems, and assisted communication platforms. Sign language translation systems not only support real-time translation but also incorporate contextual understanding to optimize translation accuracy, enabling sign language expressions to be recognized and understood more naturally.

[0003] The gesture-tracking sign language interpretation training system is a specialized assistance system designed to improve individual sign language interpretation skills. Based on gesture tracking technology, the system captures and analyzes the user's gestures in real time, assessing whether their sign language movements meet standards and providing immediate feedback. Through visual gesture comparison, translation accuracy assessment, and real-time correction, it helps users improve their proficiency and accuracy in sign language interpretation. It is suitable for training sign language interpreters, language learning for the hearing-impaired, and teaching support for sign language education institutions.

[0004] Traditional training systems rely solely on basic key point detection and trajectory analysis, failing to fully consider the problem of repairing lost key points in the skeletal topology. This results in missing key points affecting recognition accuracy in the case of complex gesture changes or occlusions. The gesture matching calculation method is limited to static comparison, and fails to perform detailed processing on the time weights of key frames, resulting in a decrease in the accuracy of dynamic gesture matching and difficulty in adapting to the natural and smooth characteristics of sign language expression. Feedback during training mainly relies on overall matching, and no detailed analysis is performed on specific joint angle errors. It is difficult for trainees to accurately adjust gesture movements, making it more difficult to standardize gesture expression. In the training evaluation phase, traditional systems are mainly based on static error comparison, failing to provide targeted suggestions based on the trainee's actual adjustment trends, affecting the efficiency and accuracy of gesture learning. Summary of the Invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a sign language translation training system based on gesture tracking.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a sign language translation training system based on gesture tracking, the system comprising:

[0007] The gesture tracking module obtains the three-dimensional coordinate data of the key points of the hand skeleton, detects the spatial displacement trend of the key points during the gesture movement, calculates the movement rate and direction change rate of the gesture trajectory, selects the key frame data that meets the gesture stability standard, and obtains the skeleton topology feature set;

[0008] The skeletal structure completion module analyzes the key points missing in the current frame based on the skeletal topology feature set, calculates the reasonable positions of the key points under the constraints of the skeletal structure, screens the candidate positions of the completion points, detects the spatial consistency and motion continuity of the candidate points, screens out the completion points that do not meet the constraints, and establishes a coordinate set of the completion key points;

[0009] The gesture standard comparison module calls the joint coordinate data of the current frame based on the completed key point coordinate set, extracts the standard gesture data according to the current sign language training content, calculates the gesture feature deviation value, and calculates the gesture matching confidence based on the feature deviation value of the key frame;

[0010] The gesture dynamic correction module analyzes the key joints where the deviation exceeds the range based on the gesture matching confidence, calculates the target angle correction value of the joint based on the finger opening and closing angles, wrist rotation angles and motion trends of adjacent frames, and generates dynamic posture correction parameters.

[0011] The present invention has improvements in that the skeleton topology feature set includes the wrist spatial coordinates, the palm center point position, the five-finger fingertip coordinates, the relative angles of the finger joints and the ratios of the adjacent joint lengths; the completed key point coordinate set includes the finger joint point spatial coordinates, the wrist key point position, the skeleton connection relationship and the joint rotation completion angle; the gesture matching confidence specifically refers to the gesture feature deviation value, the key frame matching error, the time weighted comparison score and the gesture standard similarity evaluation coefficient; the dynamic posture correction parameters include the corrected joint angle value, the target posture adjustment vector, the gesture adjustment reference benchmark and the correction step sequence.

[0012] The present invention is improved in that the gesture tracking module includes:

[0013] The skeleton key point acquisition submodule obtains the three-dimensional coordinate data of the hand skeleton key points, screens the visible skeleton joint points according to the hand topology structure, determines the visibility according to the coordinate value of each skeleton joint point in space, stores the three-dimensional coordinate data of the visible joint points, and generates a visible skeleton joint point coordinate set;

[0014] The spatial displacement trend calculation submodule calculates the spatial displacement of key points during gesture movement based on the visible skeleton joint point coordinate set, and obtains the displacement vector of the key points in each frame using the formula:

[0015] ;

[0016] and

[0017] ;

[0018] The motion rate change rate is obtained by calculation, and frames whose rate change rate fluctuation is lower than the change threshold are selected according to the gesture stability standard to generate a key frame data set;

[0019] in, Representative The three-dimensional coordinates of the frame joint points, For the Frame rate, For the Frame rate change rate, is the time interval between adjacent frames;

[0020] The skeletal topology feature extraction submodule calculates the joint angle changes between key points based on the key frame data set, obtains the change trend of the frame joint angle, establishes a time series of the joint angle, screens the skeletal topology structure whose stability exceeds the stability threshold, obtains the joint point set in the skeletal topology structure, and obtains the skeletal topology feature set.

[0021] The present invention is improved in that the skeletal structure completion module includes:

[0022] The missing key point analysis submodule analyzes the missing key points of the current frame based on the skeleton topology feature set, determines the existence status of each joint point in the skeleton topology structure, locates the position of the missing joint point in the skeleton chain according to the topological relationship, calls the relative position relationship of the known adjacent key points, obtains the three-dimensional coordinate set of the adjacent key points, and generates the adjacent coordinate set of the missing key points;

[0023] The completion point position calculation submodule calls the bone length ratio of the joint according to the adjacent coordinate set of the missing key points, and calculates the reasonable position of the missing key points under the constraints of the bone structure based on the relative position relationship of the adjacent key points, using the formula:

[0024] ;

[0025] Calculate and obtain the candidate position set of the completion point;

[0026] in, is the three-dimensional coordinate of the completed point, are the three-dimensional coordinates of nearby known key points, is the three-dimensional coordinate of another known key point, is the bone length between the completion point and the adjacent key point, is the ratio of bone lengths between known key points;

[0027] The completion key point screening submodule detects the spatial consistency and motion continuity of the candidate points based on the completion point candidate position set, calculates the spatial position deviation and motion trajectory deviation of the candidate points between adjacent frames, screens out candidate points whose spatial deviation and trajectory deviation exceed the deviation threshold, obtains the completion points that meet the skeletal structure constraints, and obtains the completion key point coordinate set.

[0028] The present invention is improved in that the gesture standard comparison module includes:

[0029] The gesture feature extraction submodule calls the joint coordinate data of the current frame based on the completed key point coordinate set, extracts the finger bending angle, wrist rotation angle and spatial position of the key joints, obtains standard gesture data according to the current sign language training content, extracts standard gesture features corresponding to the joints, including finger bending angle, wrist rotation angle and spatial position, and obtains the current frame gesture feature set and the standard gesture feature set;

[0030] The feature deviation calculation submodule calculates the deviation value of each feature item based on the current frame gesture feature set and the standard gesture feature set, using the formula:

[0031] ;

[0032] Calculate and obtain a set of gesture feature deviation values;

[0033] in, is the total deviation of gesture features, The current frame eigenvalues, The first of the standard gestures eigenvalues, For the The weight of the feature, is the total number of features;

[0034] The gesture matching confidence calculation submodule filters the gesture key frames according to the gesture feature deviation value set and extracts the time weight of the key frames. According to the feature deviation value and time weight of the key frames, the formula is used:

[0035] ;

[0036] Calculate and obtain the gesture matching confidence;

[0037] in, is the gesture matching confidence, For the The total feature deviation of key frames, For the The time weight of each keyframe, is the total number of standard keyframes.

[0038] The present invention is improved in that the gesture dynamic correction module includes:

[0039] The joint angle change calculation submodule extracts the deviation degree of the current gesture posture based on the gesture matching confidence, calculates the angle change rate of each joint, calculates the joint angle change between adjacent frames based on the finger bending angle, wrist rotation angle and the spatial position of the key joints, and organizes the angle change rate of each joint to obtain a joint angle change rate dataset;

[0040] The key joint deviation judgment submodule analyzes the key joints whose deviation exceeds the range based on the joint angle change rate data set, compares the angle change rate of each joint with the set angle change rate threshold, filters the key joints whose change rate exceeds the threshold, and generates a key joint deviation set;

[0041] The dynamic posture correction parameter generation submodule calculates the target angle correction value of each joint based on the key joint deviation set, combined with the finger opening and closing angles, wrist rotation angles and the motion trends of adjacent frames, using the formula:

[0042] ;

[0043] Calculate and obtain the target angle correction value set to generate dynamic posture correction parameters;

[0044] in, is the correction angle of the target joint, is the current joint angle, For adjacent frames The reference angle of the mid-joint, For adjacent frames The time weight, is the total number of adjacent frames, Indicates the deviation sign.

[0045] The present invention is improved in that the system further includes a training effect analysis module;

[0046] The training effect analysis module calls the dynamic posture correction parameters to remind the trainee, detects the trainee's current gesture adjustment trend, analyzes whether the gesture correction path meets the target correction angle, determines whether additional posture adjustment guidance is needed, outputs adjustment suggestions for the training process, and obtains sign language training adjustment suggestions;

[0047] The sign language training adjustment suggestions specifically include gesture correction prompts, adjustment path guidance, posture matching analysis and training progress feedback.

[0048] The present invention is improved in that the training effect analysis module includes:

[0049] The gesture adjustment trend detection submodule calls the dynamic gesture correction parameters, reminds the trainee of the current gesture, detects the changes in the trainee's joint angles after the gesture adjustment, calculates the angle change rate and spatial displacement between adjacent frames, and obtains a gesture adjustment trend data set based on the gesture correction direction and speed;

[0050] The correction path conformity judgment submodule analyzes whether the gesture correction path conforms to the target correction angle based on the gesture adjustment trend data set, compares the current gesture adjustment trend with the target correction path, and calculates the angle deviation and displacement error using the formula:

[0051] ;

[0052] Calculate and obtain the corrected path error value, determine whether the corrected path error value is within the set threshold range, screen the key joints that need additional adjustment, and obtain a set of additional posture adjustment requirements;

[0053] in, To correct the path error value, The current frame joint angles, Correct the angle for the target, For the The weight of the joints, is the total number of joints;

[0054] The training adjustment suggestion generation submodule determines whether additional posture adjustment guidance is needed based on the additional posture adjustment requirement set, outputs the adjustment direction and amplitude of each joint in combination with the current gesture adjustment trend, and generates sign language training adjustment suggestions.

[0055] A sign language tracking device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the sign language translation training system based on gesture tracking as described above is implemented.

[0056] Compared with the prior art, the advantages and positive effects of the present invention are:

[0057] In the present invention, by screening visible skeletal joints, detecting spatial displacement trends and calculating motion rates and direction change rates, key frame data that meet stability standards can be effectively extracted, thereby improving the accuracy of gesture trajectory analysis. The missing key points are supplemented by skeletal structure constraints, and the spatial consistency and motion continuity of the supplemented points are guaranteed by combining the bone length ratio and the relative position of adjacent key points, thereby avoiding interference of key point loss on the recognition results. For the gesture training process, the gesture feature deviation value is calculated, and the key frames are screened in combination with the time weight, so that the matching calculation is more consistent with the semantic features of the gesture, and the accuracy of the matching confidence calculation is optimized. For errors in the training process, dynamic posture correction is performed by calculating the joint angle change rate, analyzing the deviation range, and combining the finger opening and closing angle and the wrist rotation angle. This allows the trainee to accurately adjust the gesture movement and improve the standardization of gesture expression. The correction path is analyzed in combination with the posture adjustment trend, and adjustment suggestions are provided based on the correction angle, thereby achieving optimized guidance for sign language training and improving the pertinence and refinement of sign language training. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a system flow chart of the present invention;

[0059] Figure 2 is a flowchart of the gesture tracking module of the present invention;

[0060] Figure 3 This is a flowchart of the skeleton structure completion module of the present invention;

[0061] Figure 4 This is a flow chart of the gesture standard comparison module of the present invention;

[0062] Figure 5 This is a flow chart of the gesture dynamic correction module of the present invention;

[0063] Figure 6 This is a flow chart of the training effect analysis module of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0065] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the devices or elements referred to must have a specific direction, be constructed and operate in a specific direction, and therefore should not be understood as limiting the present invention. In addition, in the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0066] See also Figure 1 The present invention provides a technical solution: a sign language translation training system based on gesture tracking, the system comprising:

[0067] The gesture tracking module obtains the 3D coordinate data of the key points of the hand skeleton, selects the visible skeletal joints based on the hand topology, detects the spatial displacement trend of the key points during the gesture movement, calculates the movement rate and direction change rate of the gesture trajectory, selects the key frame data that meets the gesture stability standard, extracts the joint angle change trend, and obtains the skeletal topology feature set;

[0068] The skeletal structure completion module analyzes the missing key points in the current frame based on the skeletal topology feature set, calls the bone length ratio of the joint, calculates the reasonable position of the key points under the skeletal structure constraints based on the relative position relationship of the known adjacent key points, screens the candidate positions of the completion points, detects the spatial consistency and motion continuity of the candidate points, screens out the completion points that do not meet the constraints, and establishes a coordinate set of the completion key points;

[0069] The gesture standard comparison module calls the joint coordinate data of the current frame based on the completed key point coordinate set, including extracting the finger bending angle, wrist rotation angle and the spatial position of the key joints. Based on the current sign language training content, it extracts standard gesture data, calculates the gesture feature deviation value, selects gesture key frames and extracts the time weight. Based on the feature deviation value of the key frames, it calculates the gesture matching confidence.

[0070] The gesture dynamic correction module extracts the deviation degree of the current gesture posture based on the gesture matching confidence, calculates the rate of change of each joint angle, analyzes the key joints where the deviation exceeds the range, and calculates the target angle correction value of the joint based on the finger opening and closing angles, wrist rotation angles and the motion trends of adjacent frames to generate dynamic gesture correction parameters;

[0071] The training effect analysis module calls the dynamic posture correction parameters to remind the trainee, detects the trainee's current gesture adjustment trend, analyzes whether the gesture correction path meets the target correction angle, determines whether additional posture adjustment guidance is needed, and outputs adjustment suggestions for the training process to obtain sign language training adjustment suggestions;

[0072] The skeleton topology feature set includes the wrist spatial coordinates, palm center position, five-finger fingertip coordinates, finger joint relative angles, and adjacent joint length ratios. The completed key point coordinate set includes the finger joint spatial coordinates, wrist key point positions, skeleton connection relationships, and joint rotation completion angles. The gesture matching confidence specifically refers to the gesture feature deviation value, key frame matching error, time weighted comparison score, and gesture standard similarity evaluation coefficient. The dynamic gesture correction parameters include the corrected joint angle value, target gesture adjustment vector, gesture adjustment reference benchmark, and correction step sequence. The sign language training adjustment suggestions specifically include gesture correction prompts, adjustment path guidance, gesture matching analysis, and training progress feedback.

[0073] See also Figure 2 , the gesture tracking module includes:

[0074] The skeleton key point acquisition submodule obtains the three-dimensional coordinate data of the hand skeleton key points, screens the visible skeleton joint points according to the hand topology structure, determines the visibility according to the coordinate value of each skeleton joint point in space, stores the three-dimensional coordinate data of the visible joint points, and generates a visible skeleton joint point coordinate set;

[0075] The skeletal key point acquisition submodule obtains the three-dimensional coordinate data of the hand skeletal key points. By calling the depth camera and computer vision algorithm, it first collects multiple frames of hand image data and performs three-dimensional reconstruction on each frame to obtain the spatial coordinates of each skeletal joint point. In the specific execution process, the visible skeletal joint points are screened according to the hand topology. The screening process calculates the angle between each joint point and the camera and the occlusion relationship. If the angle is greater than the set threshold, the skeletal key point acquisition submodule will be used to obtain the three-dimensional coordinate data of the hand skeletal key points. If the joint point is blocked by other parts, it is judged as invisible and removed from the joint point set. The three-dimensional coordinate data of the remaining visible joint points are stored to form a visible skeleton joint point coordinate set, where the threshold The setting is based on the fact that the human hand joints are within the normal range of motion, and the angle between them and the camera is usually arrive When the angle exceeds When the joints are easily blocked by the hands, the experiment found that more than After that, the 3D reconstruction error increases significantly, so we select As a threshold, the visibility and reconstruction accuracy of the joint points in the field of view are guaranteed. For example, when the hand is extended in the direction perpendicular to the camera, the root joints are eliminated because the palm covers the finger roots, while the fingertip joints are retained because they are exposed in the camera field of view, as shown in Table 1.

[0076] Table 1 Example of visible skeleton joint point coordinate set

[0077]

[0078] As shown in Table 1, the three-dimensional coordinates of the fingertip joints are retained after screening, and the invisible joints are eliminated and stored to generate a set of visible skeleton joint coordinates.

[0079] The spatial displacement trend calculation submodule calculates the spatial displacement of key points during gesture movement based on the visible skeleton joint point coordinate set, and obtains the displacement vector of the key points in each frame using the formula:

[0080] ;

[0081] and

[0082] ;

[0083] The motion rate change rate is obtained by calculation, and frames whose rate change rate fluctuation is lower than the change threshold are selected according to the gesture stability standard to generate a key frame data set;

[0084] in, Representative The three-dimensional coordinates of the frame joint points, For the Frame rate, For the Frame rate change rate, is the time interval between adjacent frames;

[0085] The spatial displacement trend calculation submodule calculates the spatial displacement of key points during gesture movement based on the visible skeleton joint point coordinate set. By differentiating the three-dimensional coordinates of the joint points in each frame, the displacement vector of the key points in each frame is obtained. , and combined with the time interval Calculate the movement rate , for example, for the inter-frame time interval seconds (30 frames per second), if the change in the three-dimensional coordinates of a joint point is mm, then its movement speed is mm / s, further calculate the rate of change of motion speed , if the motion rate of the previous frame is mm / s, then mm / s2, where the rate of change threshold The reason for setting mm / s2 is that, through statistical analysis of the rate change rates of smooth hand movements (such as slight movement of the fingertips) and violent movements (such as rapid waving), it was found that the rate change rate of smooth movements is usually less than 100 mm / s2, while that of violent movements exceeds 150 mm / s2. Therefore, 100 mm / s2 is set as the threshold for distinguishing between stable and unstable states to accurately identify key frames and ensure the reliability of the key frame data set. According to the gesture stability standard, frames with rate change fluctuations lower than the set threshold are screened to generate the key frame data set, as shown in Table 2.

[0086] Table 2 Key frame data set example

[0087]

[0088] As shown in Table 2, frames with a rate change rate lower than a threshold are screened to form a key frame data set.

[0089] The skeleton topology feature extraction submodule calculates the joint angle changes between key points based on the key frame data set, obtains the change trend of the frame joint angle, establishes the time series of the joint angle, screens the skeleton topology structure whose stability exceeds the stability threshold, obtains the joint point set in the skeleton topology structure, and obtains the skeleton topology feature set;

[0090] The skeleton topology feature extraction submodule calculates the joint angle changes between key points based on the key frame data set. In the specific process, the joint angle is calculated based on the skeleton vector formed by adjacent joint points using the vector angle formula. , taking the finger joints as an example, joint points 1 and 2, joint points 2 and 3 form the bone vectors and , then the joint angle is , calculate the joint angle change of each frame and establish a time series, filter the stability exceeding the stability threshold The skeletal topology structure, where the stability threshold The setting basis is that it can be found through statistical data that in normal gesture transformation, the fluctuation of joint angle is lower than When the gesture is continuous and stable, the gesture is more continuous and stable, and when the gesture is more There is obvious fluctuation, so set In order to stabilize the threshold and ensure the stability and coherence of the topological structure, a stable joint point set is obtained, and a skeleton topological feature set is obtained, as shown in Table 3.

[0091] Table 3 Examples of skeleton topology feature sets

[0092]

[0093] As shown in Table 3, the bone topology structures whose joint angle stability exceeds the set threshold are screened and stored to form a bone topology feature set.

[0094] See also Figure 3 , the skeleton structure completion module includes:

[0095] The missing key point analysis submodule analyzes the missing key points in the current frame based on the skeleton topology feature set, determines the existence status of each joint point in the skeleton topology structure, locates the position of the missing joint point in the skeleton chain based on the topological relationship, calls the relative position relationship of the known adjacent key points, obtains the three-dimensional coordinate set of the adjacent key points, and generates the adjacent coordinate set of the missing key points;

[0096] The missing key point analysis submodule analyzes the key points missing in the current frame based on the skeleton topology feature set. First, the skeleton joint point coordinate set of the current frame is obtained and compared one by one with all the joint points defined in the topology feature set to determine the existence status of each joint point. By comparing the three-dimensional coordinates of the joint point, if the three-dimensional coordinates of a joint point are empty, the joint point is determined to be missing. Taking the hand skeleton as an example, if the little fingertip joint point is not detected in the current frame, it is determined to be a missing joint point. Further, according to the topological relationship, the position of the missing joint point in the skeleton chain is located, and the predecessor joint point and successor joint point of the missing joint point are determined through the upstream and downstream relationship of the joint point. For example, if the little fingertip joint point is missing, the predecessor joint point is the middle joint point of the little finger, and the successor joint point is empty (there is no successor node at the fingertip). Then, the relative position relationship of the known adjacent key points is called to obtain the three-dimensional coordinates of the middle joint point of the little finger of the predecessor joint point. and the three-dimensional coordinates of the little finger base joint , generate the adjacent coordinate set of missing key points, as shown in Table 4.

[0097] Table 4 Examples of missing keypoint neighboring coordinate sets

[0098]

[0099] As shown in Table 4, the three-dimensional coordinates of the middle segment and root joint points of the little fingertip adjacent to the missing joint point are extracted and stored as the missing key point adjacent coordinate set.

[0100] The completion point position calculation submodule calls the bone length ratio of the joint based on the adjacent coordinate set of the missing key points. Based on the relative position relationship of the adjacent key points, it calculates the reasonable position of the missing key points under the constraints of the bone structure using the formula:

[0101] ;

[0102] Calculate and obtain the candidate position set of the completion point;

[0103] in, is the three-dimensional coordinate of the completed point, are the three-dimensional coordinates of nearby known key points, is the three-dimensional coordinate of another known key point, is the bone length between the completion point and the adjacent key point, is the ratio of bone lengths between known key points;

[0104] The completion point position calculation submodule calls the bone length ratio of the joint based on the adjacent coordinate set of the missing key points , according to the relative position relationship of the adjacent key points, calculate the reasonable position of the missing key points under the constraints of the bone structure. In the specific execution process, first calculate the bone length between the known adjacent key points For example, the distance between the middle joint point of the little finger and the root joint point is mm, call the preset bone length ratio This ratio is set based on the anatomical data of the hand. The length from the tip of the little finger to the middle segment accounts for about 75% of the length from the middle segment to the root. This value comes from the actual measured hand joint ratio data and fluctuates with the size of the hand. Usually, the ratio of adult hands fluctuates between 0.7 and 0.8. Here, 0.75 is taken as the middle value to ensure applicability and rationality. Calculate the distance between the missing key point and the middle segment joint point mm, then, according to the formula , calculate the three-dimensional coordinates of the missing key points, where the adjacent known key points and another known key point , substitute into the calculation to get , and finally get the three-dimensional coordinates of the completed point , generate a set of candidate locations for completion points, as shown in Table 5.

[0105] Table 5 Example of candidate location set for completion points

[0106]

[0107] As shown in Table 5, the three-dimensional coordinates of the completion points are calculated based on the relative positions of the adjacent joint points and the ratio of the bone lengths, and a set of candidate locations of the completion points is generated.

[0108] The completion key point screening submodule detects the spatial consistency and motion continuity of the candidate points based on the set of candidate positions of the completion points, calculates the spatial position deviation and motion trajectory deviation of the candidate points between adjacent frames, and screens out candidate points whose spatial deviation and trajectory deviation exceed the deviation threshold. It obtains the completion points that meet the constraints of the skeletal structure and obtains the coordinate set of the completion key points.

[0109] The completion key point screening submodule detects the spatial consistency and motion continuity of the candidate points based on the completion point candidate position set. First, the spatial position deviation between the candidate point and the corresponding joint point in the previous frame is calculated to determine its spatial consistency. Specifically, the module calculates the three-dimensional coordinate difference. ,in, and the coordinates of the joint point in the previous frame , then the spatial deviation mm, spatial deviation threshold mm is set based on the normal fluctuation range of the joint points during gesture movement. This value refers to the tolerance range for noise interference in the field of gesture recognition and is verified by multiple sets of experimental data. The threshold usually fluctuates between 0.8 and 1.2 mm. This time it is set to 1.0 mm to balance accuracy and stability. Less than the deviation threshold mm, the spatial consistency is determined to be qualified, otherwise it is eliminated. Then, the motion trajectory deviation of the candidate point is calculated by comparing the motion vector differences of the candidate points in adjacent frames. ,in, mm / s and mm / s, then mm / s, motion trajectory deviation threshold The mm / s is set based on the acceleration fluctuation range of normal gesture movement. Through comparative analysis of normal gesture trajectories and abnormal jitter data, it is set to 0.5 mm / s to effectively filter out abnormal motion points. Ultimately, candidate points with spatial deviations and trajectory deviations exceeding the threshold are screened out, and completion points that meet the skeletal structure constraints are obtained. The coordinate set of the completed key points is obtained, and the missing key points are completed.

[0110] See also Figure 4 , the gesture standard comparison module includes:

[0111] The gesture feature extraction submodule calls the joint coordinate data of the current frame based on the completed key point coordinate set, extracts the finger bending angle, wrist rotation angle and spatial position of the key joints, obtains the standard gesture data according to the current sign language training content, and extracts the standard gesture features corresponding to the joints, including finger bending angle, wrist rotation angle and spatial position, to obtain the current frame gesture feature set and the standard gesture feature set;

[0112] The gesture feature extraction submodule completes the key point coordinate set, calls the joint coordinate data of the current frame, analyzes the three-dimensional coordinates of each joint point in each frame, and calculates the bending angle of each finger, the rotation angle of the wrist, and the spatial position of the key joint point. The calculation of the finger bending angle is based on the joint vector formed by adjacent joint points, and the vector angle formula is used. Get the bending angle of each joint, for example, the vector formed between the second joint (proximal segment) and the third joint (middle segment) of the index finger and ,like and , the bending angle is Similarly, the bending angles of each joint of each finger are calculated and stored as a bending angle array. The wrist rotation angle is obtained by calculating the angle between the plane vectors formed by the wrist joint and the arm joint. For example, if the vector between the wrist joint and the elbow joint is , and the vector of the arm joint point is , then the wrist rotation angle is ,The spatial positions of key joints are recorded in the form of three-dimensional coordinates, as shown in Table 6.,Then, according to the current sign language training content, the standard,gesture data is obtained, the standard gesture database is called, and the standard,gesture features corresponding to the joints are extracted, including the finger bending angle,,wrist rotation angle and spatial position. The gesture features extracted in the current frame,are stored in correspondence with the standard gesture features, and the current frame,gesture feature set and the standard gesture feature set are obtained.

[0113] Table 6 Gesture feature set examples

[0114]

[0115] As shown in Table 6, the bending angle and spatial position of each joint point in the current frame are extracted and stored as a gesture feature set, and corresponded to the standard gesture feature set.

[0116] The feature deviation calculation submodule calculates the deviation value of each feature item based on the current frame gesture feature set and the standard gesture feature set, using the formula:

[0117] ;

[0118] Calculate and obtain a set of gesture feature deviation values;

[0119] in, is the total deviation of gesture features, The current frame eigenvalues, The first of the standard gestures eigenvalues, For the The weight of the feature, is the total number of features;

[0120] The feature deviation calculation submodule calculates the deviation value of each feature item based on the current frame gesture feature set and the standard gesture feature set. First, each feature value in the current frame gesture feature set is compared one by one. Corresponding standard gesture feature values , calculate the square of the difference between the two , and then call the corresponding weight according to the importance of each feature item And perform weighted processing to obtain weighted deviation , where the weight The setting basis is based on the importance of each feature item in gesture recognition. Through the analysis of a large number of sign language movements, it is determined that the finger bending angle has a greater impact on gesture classification, so a higher weight is set. , while the wrist rotation angle has relatively little effect on gesture classification, so a lower weight is set. The setting range is fine-tuned according to the recognition accuracy of different sign language letters or words to ensure the rationality of the weight setting. The specific adjustment is based on the error fluctuation range not exceeding 0.05. For example, the bending angle of the second joint of the index finger in the current frame is , while the bending angle of the corresponding joint in the standard gesture is , the bending angle deviation is Similarly, calculate the weighted deviations of all feature items and add them up to get the total deviation of gesture feature. ,As shown in Table 7, the deviation value set of all feature items is calculated and stored.

[0121] Table 7 Example of gesture feature deviation value set

[0122]

[0123] As shown in Table 7, the deviation value of each feature item in the current frame is calculated and weighted and stored, and finally the total deviation value of the gesture feature is obtained.

[0124] The gesture matching confidence calculation submodule filters the gesture key frames according to the gesture feature deviation value set and extracts the time weight of the key frames. Based on the feature deviation value and time weight of the key frames, the formula is used:

[0125] ;

[0126] Calculate and obtain the gesture matching confidence;

[0127] in, is the gesture matching confidence, For the The total feature deviation of key frames, For the The time weight of each keyframe, is the total number of standard key frames;

[0128] The gesture matching confidence calculation submodule selects gesture key frames based on the gesture feature deviation value set and extracts the time weight of the key frames. First, based on the key nodes of the gesture action, the frames with significant changes in motion trajectory or low feature deviation are selected as key frames. For example, if the deviation values ​​of all feature items in a frame are less than the set threshold , then the frame is marked as a key frame, and then the time weight corresponding to each key frame is called , and according to the formula Calculate the gesture matching confidence. Taking the three key frames in the current gesture as an example, the total feature deviation values ​​are 、 、 , and the corresponding time weights are 、 、 , time weight Set as the relative timing weight of the frame in the entire gesture action to ensure the consistency of the gesture action before and after, usually between 0.2 and 0.4. The specific value is set according to the amplitude of the motion change between frames. The gesture matching confidence is ,The calculation results show that the matching confidence between the current gesture and the standard ,gesture is 67%. Finally, the gesture matching confidence ,is obtained and stored.

[0129] See also Figure 5 , the gesture dynamic correction module includes:

[0130] The joint angle change calculation submodule extracts the deviation degree of the current gesture posture based on the gesture matching confidence, calculates the angle change rate of each joint, calculates the joint angle change between adjacent frames based on the finger bending angle, wrist rotation angle and the spatial position of the key joints, and summarizes the angle change rate of each joint to obtain the joint angle change rate dataset;

[0131] The joint angle change calculation submodule extracts the deviation degree of the current gesture posture based on the gesture matching confidence. First, the gesture feature set of the current frame and the adjacent previous frame is obtained, including the bending angle of each joint point, the wrist rotation angle and the three-dimensional spatial position of the key joint points. By comparing the joint angles between adjacent frames one by one, the angle change rate of each joint is calculated. In the specific calculation process, the angle change of each joint is obtained based on the vector change between the joint points. For example, the angle between the second and third joints of the index finger , in the current frame, the angle is , which is , then the angle of the joint changes , the angle change rate is , where the inter-frame time interval is Seconds (30 frames per second), the angle change rate of each joint is calculated similarly and sorted into an angle change rate array, which is then stored as a joint angle change rate dataset, as shown in Table 8. The angle change rate threshold is set to This value is set based on the upper limit of the angle change rate of human fingers and wrists during normal gesture changes. It refers to the maximum angle change rate of normal gesture transformation in multiple gesture experiments, and is finally selected based on the fluctuation range of gesture operation speed in the experiment. As a threshold, it ensures that it can capture faster gesture changes without misjudging natural fluctuations as abnormalities. At the same time, this value fluctuates with the change of gesture type (such as static gesture and dynamic gesture) and operation speed (such as normal gesture and fast gesture), usually in arrive The angle change rate dataset of each joint is finally obtained by adjusting the angle change rate between the two joints to meet the gesture recognition requirements of different scenarios.

[0132] Table 8 Example of joint angle change rate dataset

[0133]

[0134] As shown in Table 8, the angle change and its change rate of each joint point between adjacent frames are calculated and stored as a joint angle change rate dataset.

[0135] The key joint deviation judgment submodule analyzes the key joints whose deviations exceed the range based on the joint angle change rate dataset, compares the angle change rate of each joint with the set angle change rate threshold, filters out the key joints whose change rates exceed the threshold, and generates a key joint deviation set;

[0136] The key joint deviation judgment submodule analyzes the key joints whose deviation exceeds the range based on the joint angle change rate dataset. First, the angle change rate of each joint is obtained and compared with the set angle change rate threshold. The set angle change rate threshold is determined according to the gesture stability requirement. Taking the second joint of the index finger as an example, if its angle change rate is , and the angle change rate threshold is set to ,but , it is judged to exceed the threshold. Similarly, the angle change rates of all joints are compared one by one. If the angle change rate of a joint exceeds the threshold, the joint point is marked as a key joint with deviation out of range, and the joint point number, angle change rate and threshold are stored to generate a key joint deviation set, as shown in Table 9.

[0137] Table 9 Example of key joint deviation set

[0138]

[0139] As shown in Table 9, the angle change rate of the second joint of the index finger exceeds the threshold, so it is marked as a key joint with deviation out of range and stored in the key joint deviation set.

[0140] The dynamic posture correction parameter generation submodule calculates the target angle correction value of each joint based on the key joint deviation set, combined with the finger opening and closing angles, wrist rotation angles and the motion trends of adjacent frames, using the formula:

[0141] ;

[0142] Calculate and obtain the target angle correction value set to generate dynamic posture correction parameters;

[0143] in, is the correction angle of the target joint, is the current joint angle, For adjacent frames The reference angle of the mid-joint, For adjacent frames The time weight, is the total number of adjacent frames, is the deviation sign indication, -1 means decreasing the angle, 1 means increasing the angle;

[0144] The dynamic posture correction parameter generation submodule calculates the target angle correction value of each joint based on the key joint deviation set, combined with the finger opening and closing angles, wrist rotation angles and the motion trends of adjacent frames. First, call the joint points marked as exceeding the threshold in the key joint deviation set and obtain their current angles. and the reference angles of adjacent frames Then, calculate the average deviation of the joint angles in adjacent frames. Taking the second joint of the index finger in the current frame as an example, let the reference angles in the three adjacent frames be 、 、 , and set the time weight 、 、 The time weight is set according to the temporal distance between each frame and the current frame. The closer the distance, the greater the weight, in order to reflect the influence of temporal proximity on gesture correction. The time weight value is adjusted according to the interval between the previous and next frames and the gesture speed, and is usually set to arrive The sum is 1, then according to the formula Calculate the target angle correction value, where and (decrease the angle), then , and finally get the target angle correction value , stored as a target angle correction value set, generating dynamic attitude correction parameters, as shown in Table 10.

[0145] Table 10 Example of dynamic posture correction parameter set

[0146]

[0147] As shown in Table 10, the target angle correction value is calculated based on the reference angles and time weights of adjacent frames, and the dynamic posture correction parameters are generated.

[0148] See also Figure 6 , the training effect analysis module includes:

[0149] The gesture adjustment trend detection submodule calls the dynamic posture correction parameters to remind the trainee of the current gesture, detects the changes in joint angles after the trainee makes posture adjustments, calculates the angle change rate and spatial displacement between adjacent frames, and obtains the gesture adjustment trend dataset based on the gesture correction direction and speed.

[0150] The gesture adjustment trend detection submodule calls the dynamic posture correction parameters to remind the trainee of the current gesture. First, according to the obtained dynamic posture correction parameters, the joint number, target correction angle and correction direction to be adjusted are extracted. Taking the second joint of the index finger as an example, if the dynamic posture correction parameter is , the current angle is , if the correction direction is to decrease, the system prompts the trainee to bend the joint, detects the change in joint angle after the trainee adjusts the posture, calculates the angle change rate of each joint point through the three-dimensional coordinates of consecutive frames, and combines the spatial displacement to obtain the angle change and spatial movement during the gesture adjustment process. For example, in the subsequent frames, if the angle of the second joint of the index finger changes to 、 、 , the corresponding angle change rates are 、 and , and calculate the three-dimensional spatial displacement at the same time, for example, the three-dimensional coordinates of the joint point from Move to , then the spatial displacement is mm, further combined with the correction direction and speed, the angle change rate and spatial displacement between adjacent frames are sorted into a gesture adjustment trend dataset, as shown in Table 11, and finally the gesture adjustment trend dataset is obtained.

[0151] Table 11 Example of gesture adjustment trend dataset

[0152]

[0153] As shown in Table 11, the joint angle, angle change rate, spatial displacement, and correction direction in each frame are recorded to form a gesture adjustment trend dataset.

[0154] The correction path conformity judgment submodule analyzes whether the gesture correction path conforms to the target correction angle based on the gesture adjustment trend dataset, compares the current gesture adjustment trend with the target correction path, and calculates the angle deviation and displacement error using the formula:

[0155] ;

[0156] Calculate and obtain the corrected path error value, determine whether the corrected path error value is within the set threshold range, screen the key joints that need additional adjustment, and obtain a set of additional posture adjustment requirements;

[0157] in, To correct the path error value, The current frame joint angles, Correct the angle for the target, For the The weight of the joints, is the total number of joints;

[0158] The correction path conformity judgment submodule analyzes whether the gesture correction path conforms to the target correction angle based on the gesture adjustment trend dataset. First, the joint angle of each frame in the gesture adjustment trend dataset is obtained and compared with the target correction angle. Compare and calculate the angle deviation , and call the corresponding weight according to the importance of each joint , using the formula Calculate the corrected path error value. Taking the second joint of the index finger as an example, if the angles of the current three frames are 、 、 , the target angle is , the weights are 、 、 , then the corrected path error value is Then, the calculated corrected path error value is compared with the set threshold For comparison, if , then the current correction path is judged to meet the requirements, otherwise it is judged to be unsatisfactory, and then the key joints that need additional adjustment are screened to generate a set of additional posture adjustment requirements, as shown in Table 12.

[0159] About the setting of threshold and weight: threshold The threshold is set to 1.0 based on the accuracy requirement of gesture fine-tuning. It is obtained by measuring the stability of finger bending and rotation in the natural fluctuation range of human joints during gesture training. arrive The specific value depends on the complexity of the gesture and the learning stage. For example, it can be set to 2.0 in the beginner stage and 1.0 or lower in the advanced stage to improve gesture accuracy. The weight is set based on the importance of the joint and the degree of influence in the gesture. For example, the index finger and thumb are more recognizable in sign language, so their weight is larger and is usually set to or , while wrist rotation has less effect on gesture recognition, so it is set to or In this example, the weights of the index finger joints are set to 、 and This is related to its weight and angle change sensitivity in the gesture. The specific weight value is adjusted by the actual recognition effect of different gestures to ensure accuracy and robustness.

[0160] Table 12 Example of corrected path error values ​​and additional adjustment requirements

[0161]

[0162] As shown in Table 12, the corrected path error value of the second joint of the index finger is less than the threshold, so no additional adjustment is required. However, the error value of the wrist joint exceeds the threshold, so it is determined that additional adjustment is required and stored as an additional posture adjustment requirement set.

[0163] The training adjustment suggestion generation submodule determines whether additional posture adjustment guidance is needed based on the additional posture adjustment requirement set. Combined with the current gesture adjustment trend, it outputs the adjustment direction and amplitude of each joint and generates sign language training adjustment suggestions.

[0164] The training adjustment suggestion generation submodule determines whether additional posture adjustment guidance is needed based on the additional posture adjustment requirement set. First, the joint number marked as requiring additional adjustment in the additional posture adjustment requirement set is called, and the angle change rate and spatial displacement of the joint point are extracted based on the current gesture adjustment trend. Taking the wrist joint as an example, if its angle change rate is , the spatial displacement is mm, if the correction direction is decreasing, then the angle change rate is lower than the set correction speed threshold , it is judged that the adjustment speed is too slow, the output prompt "speed up the rotation speed" is given, and the required additional adjustment range is calculated. If the current angle is , the target angle is , the additional adjustment is , and then generate training adjustment suggestions, including adjustment direction and adjustment amplitude, as shown in Table 3.

[0165] A sign language tracking device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the sign language translation training system based on gesture tracking as described above is implemented.

[0166] The above are merely preferred embodiments of the present invention and do not limit the present invention in any other form. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A sign language translation training system based on gesture tracking, characterized in that: The system comprises: The gesture tracking module obtains the three-dimensional coordinate data of the key points of the hand skeleton, detects the spatial displacement trend of the key points during the gesture movement, calculates the movement rate and direction change rate of the gesture trajectory, selects the key frame data that meets the gesture stability standard, and obtains the skeleton topology feature set; The skeleton structure completion module analyzes the key points missing in the current frame based on the skeleton topology feature set, calculates the reasonable position of the key points under the constraints of the skeleton structure, screens the candidate positions of the completion points, detects the spatial consistency and motion continuity of the candidate points, screens out the completion points that do not meet the constraints, and establishes a coordinate set of the completion key points; The gesture standard comparison module calls the joint coordinate data of the current frame based on the completed key point coordinate set, extracts the standard gesture data according to the current sign language training content, calculates the gesture feature deviation value, and calculates the gesture matching confidence based on the feature deviation value of the key frame; The gesture standard comparison module includes: The gesture feature extraction submodule calls the joint coordinate data of the current frame based on the completed key point coordinate set, extracts the finger bending angle, wrist rotation angle and spatial position of the key joints, obtains standard gesture data according to the current sign language training content, extracts standard gesture features corresponding to the joints, including finger bending angle, wrist rotation angle and spatial position, and obtains the current frame gesture feature set and the standard gesture feature set; The feature deviation calculation submodule calculates the deviation value of each feature item based on the current frame gesture feature set and the standard gesture feature set, using the formula: ； Calculate and obtain a set of gesture feature deviation values; in, is the total deviation of gesture features, The current frame eigenvalues, The first of the standard gestures eigenvalues, For the The weight of the feature, is the total number of features; The gesture matching confidence calculation submodule filters the gesture key frames according to the gesture feature deviation value set and extracts the time weight of the key frames. According to the feature deviation value and time weight of the key frames, the formula is used: ； Calculate and obtain the gesture matching confidence; in, is the gesture matching confidence, For the The total feature deviation of key frames, For the The time weight of each keyframe, is the total number of standard key frames; The gesture dynamic correction module analyzes the key joints where the deviation exceeds the range based on the gesture matching confidence, calculates the target angle correction value of the joint based on the finger opening and closing angles, wrist rotation angles and motion trends of adjacent frames, and generates dynamic posture correction parameters.

2. The sign language translation training system based on gesture tracking according to claim 1, characterized in that: The skeleton topology feature set includes the wrist spatial coordinates, the palm center position, the five-finger fingertip coordinates, the relative angles of the finger joints and the ratios of the adjacent joint lengths. The completed key point coordinate set includes the finger joint spatial coordinates, the wrist key point position, the skeleton connection relationship and the joint rotation completion angle. The gesture matching confidence specifically refers to the gesture feature deviation value, the key frame matching error, the time weighted comparison score and the gesture standard similarity evaluation coefficient. The dynamic gesture correction parameters include the corrected joint angle value, the target gesture adjustment vector, the gesture adjustment reference benchmark and the correction step sequence.

3. The sign language translation training system based on gesture tracking according to claim 1, characterized in that: The gesture tracking module includes: The skeleton key point acquisition submodule obtains the three-dimensional coordinate data of the hand skeleton key points, screens the visible skeleton joint points according to the hand topology structure, determines the visibility according to the coordinate value of each skeleton joint point in space, stores the three-dimensional coordinate data of the visible joint points, and generates a visible skeleton joint point coordinate set; The spatial displacement trend calculation submodule calculates the spatial displacement of key points during gesture movement based on the visible skeleton joint point coordinate set, and obtains the displacement vector of the key points in each frame using the formula: ； and ； The motion rate change rate is obtained by calculation, and frames whose rate change rate fluctuation is lower than the change threshold are selected according to the gesture stability standard to generate a key frame data set; in, Representative The three-dimensional coordinates of the frame joint points, For the Frame rate, For the Frame rate change rate, is the time interval between adjacent frames; The skeletal topology feature extraction submodule calculates the joint angle changes between key points based on the key frame data set, obtains the change trend of the frame joint angle, establishes a time series of the joint angle, screens the skeletal topology structure whose stability exceeds the stability threshold, obtains the joint point set in the skeletal topology structure, and obtains the skeletal topology feature set.

4. The sign language translation training system based on gesture tracking according to claim 1, characterized in that: The skeleton structure completion module includes: The missing key point analysis submodule analyzes the missing key points of the current frame based on the skeleton topology feature set, determines the existence status of each joint point in the skeleton topology structure, locates the position of the missing joint point in the skeleton chain according to the topological relationship, calls the relative position relationship of the known adjacent key points, obtains the three-dimensional coordinate set of the adjacent key points, and generates the adjacent coordinate set of the missing key points; The completion point position calculation submodule calls the bone length ratio of the joint according to the adjacent coordinate set of the missing key points, and calculates the reasonable position of the missing key points under the constraints of the bone structure based on the relative position relationship of the adjacent key points, using the formula: ； Calculate and obtain the candidate position set of the completion point; in, is the three-dimensional coordinate of the completed point, are the three-dimensional coordinates of nearby known key points, is the three-dimensional coordinate of another known key point, is the bone length between the completion point and the adjacent key point, is the ratio of bone lengths between known key points; The completion key point screening submodule detects the spatial consistency and motion continuity of the candidate points based on the completion point candidate position set, calculates the spatial position deviation and motion trajectory deviation of the candidate points between adjacent frames, screens out candidate points whose spatial deviation and trajectory deviation exceed the deviation threshold, obtains the completion points that meet the skeletal structure constraints, and obtains the completion key point coordinate set.

5. The sign language translation training system based on gesture tracking according to claim 1, characterized in that: The gesture dynamic correction module includes: The joint angle change calculation submodule extracts the deviation degree of the current gesture posture based on the gesture matching confidence, calculates the angle change rate of each joint, calculates the joint angle change between adjacent frames based on the finger bending angle, wrist rotation angle and the spatial position of the key joints, and organizes the angle change rate of each joint to obtain a joint angle change rate dataset; The key joint deviation judgment submodule analyzes the key joints whose deviation exceeds the range based on the joint angle change rate data set, compares the angle change rate of each joint with the set angle change rate threshold, filters the key joints whose change rate exceeds the threshold, and generates a key joint deviation set; The dynamic posture correction parameter generation submodule calculates the target angle correction value of each joint based on the key joint deviation set, combined with the finger opening and closing angles, wrist rotation angles and the motion trends of adjacent frames, using the formula: ； Calculate and obtain the target angle correction value set to generate dynamic posture correction parameters; in, is the correction angle of the target joint, is the current joint angle, For adjacent frames The reference angle of the mid-joint, For adjacent frames The time weight, is the total number of adjacent frames, Indicates the deviation sign.

6. The sign language translation training system based on gesture tracking according to claim 1, characterized in that: The system also includes a training effect analysis module; The training effect analysis module calls the dynamic posture correction parameters to remind the trainee, detects the trainee's current gesture adjustment trend, analyzes whether the gesture correction path meets the target correction angle, determines whether additional posture adjustment guidance is needed, outputs adjustment suggestions for the training process, and obtains sign language training adjustment suggestions; The sign language training adjustment suggestions specifically include gesture correction prompts, adjustment path guidance, posture matching analysis and training progress feedback.

7. The sign language translation training system based on gesture tracking according to claim 6, characterized in that: The training effect analysis module includes: The gesture adjustment trend detection submodule calls the dynamic gesture correction parameters, reminds the trainee of the current gesture, detects the changes in the trainee's joint angles after the gesture adjustment, calculates the angle change rate and spatial displacement between adjacent frames, and obtains a gesture adjustment trend dataset based on the gesture correction direction and speed; The correction path conformity judgment submodule analyzes whether the gesture correction path conforms to the target correction angle based on the gesture adjustment trend data set, compares the current gesture adjustment trend with the target correction path, and calculates the angle deviation and displacement error using the formula: ； Calculate and obtain the corrected path error value, determine whether the corrected path error value is within the set threshold range, screen the key joints that need additional adjustment, and obtain a set of additional posture adjustment requirements; in, To correct the path error value, The current frame joint angles, Correct the angle for the target, For the The weight of each joint, is the total number of joints; The training adjustment suggestion generation submodule determines whether additional posture adjustment guidance is needed based on the additional posture adjustment requirement set, outputs the adjustment direction and amplitude of each joint in combination with the current gesture adjustment trend, and generates sign language training adjustment suggestions.

8. A sign language tracking device, comprising a memory and a processor, characterized in that: The memory stores a computer program, and when the processor executes the computer program, the sign language translation training system based on gesture tracking according to any one of claims 1 to 7 is implemented.

Images