Sign language recognition method based on Mediape and XGBoost algorithms

Abstract

The invention provides a sign language recognition method based on Mediape and an XGBoost algorithm, relates to the technical field of computer vision and pattern recognition, and is characterized in that accurate collaborative extraction of key points of multiple parts of hands, arms and faces is realized by adopting a Mediape framework, and a backward mode of traditional image processing is replaced; a multi-modal feature fusion system is constructed, and expression types of recognizable sign languages are expanded; in combination with feature screening, dynamic weight optimization and efficient classification capability of an XGBoost algorithm, core pain points in the prior art are solved, and accurate and stable recognition of complex sign language actions is realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision and pattern recognition technology, specifically to a sign language recognition method based on Mediapipe and XGBoost algorithms. Background Technology

[0002] Sign language is a core communication tool for people with hearing impairments, and its accuracy depends on the comprehensive capture of body movements and facial expressions. Current technologies primarily rely on traditional image processing methods combined with classification algorithms to categorize different sign language gestures. These methods are greatly affected by changes in lighting, background interference, and limb occlusion, resulting in low accuracy in key point localization, slow extraction speed, and high sensitivity to noise, thus limiting the improvement of recognition accuracy. To address this issue, wearable devices and human-computer interface systems have emerged; however, these devices are a burden for people, affecting user comfort.

[0003] Mediapipe is a deep learning-based real-time pose estimation framework developed by Google. This open-source framework aims to help developers build high-performance real-time multimedia processing pipelines. It is particularly suitable for processing video and images and is used in scenarios such as facial expression recognition, gesture recognition, and pose estimation.

[0004] Current gesture recognition methods using the Mediapipe algorithm mostly focus on feature extraction from 21 key points on the hand. However, sign language, as a form of body language, involves not only conventional gesture recognition but also, in most cases, physical movements and even facial expressions. Therefore, applying traditional Mediapipe-based gesture recognition methods to sign language recognition lacks feature extraction from key points on the face and limbs, resulting in low accuracy in sign language recognition. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a sign language recognition method based on Mediapipe and XGBoost algorithms. By employing the Mediapipe framework, it achieves accurate collaborative extraction of key points from multiple parts of the hand, arm, and face, replacing the outdated methods of traditional image processing. It constructs a multimodal feature fusion system to expand the types of recognizable sign language expressions. Combining the feature selection, dynamic weight optimization, and efficient classification capabilities of the XGBoost algorithm, it solves the core pain points of existing technologies and achieves accurate and stable recognition of complex sign language movements.

[0006] This invention provides a sign language recognition method based on Mediapipe and XGBoost algorithms, comprising the following steps:

[0007] Step 1: Data acquisition and preprocessing from multiple locations;

[0008] Step 2: Construction of multi-part collaborative feature engineering;

[0009] Step 3: Dynamic optimization of the XGBoost model;

[0010] Step 4: Hierarchical multimodal cross-validation;

[0011] Step 5: Set up a sign language recognition system.

[0012] Furthermore, step 1 involves employing Mediapipe multi-frame collaborative extraction technology to simultaneously acquire 21 key points of the hand, 3 core joints of the arm (shoulder joint, elbow joint, and wrist joint), and 68 core facial expression key points, ensuring the spatiotemporal synchronization of data from multiple body parts; through confidence filtering, interpolation completion, and Gaussian filtering, noise and outliers are removed to ensure the reliability of key point data; compared with traditional image processing methods, the Mediapipe algorithm can significantly reduce key point localization errors and improve extraction speed.

[0013] Furthermore, step 2 includes hand features, retaining the original distance matrix and angle matrix features to ensure fine motion capture capability; arm features, introducing joint angle, movement rate and posture trend features to reflect key information of large-scale limb movement; facial features, selecting core areas of eyes, eyebrows and mouth, extracting features such as opening and closing degree and shape changes to capture emotional auxiliary semantics; after splicing features from multiple parts, XGBoost feature importance selection and PCA dimensionality reduction are used to reduce computational complexity while retaining key information, forming an 80-dimensional fused feature vector.

[0014] Furthermore, step 3 includes constructing an XGBoost classifier, introducing a dynamic weight allocation mechanism for multi-part features, and adjusting the contribution of each part feature according to the sign language type and complexity; adopting a hyperparameter optimization strategy combining grid search and cross-validation to avoid model overfitting; and designing an online incremental learning algorithm to update the decision tree node weights of the XGBoost model through local weighted regression, adapting to the limb characteristics and sign language habits of different users, without retraining the entire model, thus improving practicality.

[0015] Furthermore, step 4 includes constructing three single-part auxiliary classifiers to independently classify the features of each part, and obtaining the final result through hierarchical weighted fusion (limb layer + expression layer); adaptively adjusting the weights for different scenarios (hand occlusion, insufficient light), for example, increasing the weight of arm and face features when the hand is occluded, and decreasing the weight of face features when the light is insufficient, to ensure the stability and reliability of the recognition results.

[0016] Furthermore, step 5 includes building a human-computer interaction system that can be used for real-time sign language recognition and embedding it into a mobile phone.

[0017] The above method extracts the coordinates of 21 key points of the hand, 3 core joints of the arm (shoulder, elbow, and wrist), and 68 core facial expression key points; calculates the joint angles, movement rates, opening and closing degrees, and morphological changes of the eyes, eyebrows, and mouth; determines an 80-dimensional fusion feature vector through XGBoost feature importance screening and PCA dimensionality reduction; constructs an XGBoost classifier, introduces a dynamic weight allocation mechanism for multi-part features, and adjusts the contribution of each part feature according to the type and complexity of sign language; constructs three single-part auxiliary classifiers to independently classify the features of each part, obtains the final result through hierarchical weighted fusion, and adaptively adjusts the weights for different scenarios; and builds a real-time feedback APP that can be embedded in mobile phones to achieve high-precision real-time recognition of sign language in complex backgrounds.

[0018] The present invention has the following advantages over the prior art:

[0019] 1. The sign language recognition method based on Mediapipe and XGBoost algorithms provided by this invention uses the Mediapipe algorithm to replace the traditional image processing method, which significantly improves the accuracy of key point positioning, enhances the ability to resist light interference and occlusion, and provides a reliable data foundation for subsequent feature engineering.

[0020] 2. The sign language recognition method based on Mediapipe and XGBoost algorithms provided in this invention, through the fusion of multimodal features of "hand-arm-face" and the strong classification ability of XGBoost, can recognize complex sign language types that include body movements and emotional expressions, covering more real-world communication scenarios. Multi-part collaborative engineering and dynamic weight allocation algorithms enable the model to adapt to different types and complexities of sign language movements, which can significantly improve the sign language recognition rate. The combination of XGBoost's generalization ability and hierarchical cross-validation mechanism enables the model to maintain stable recognition even in complex scenarios such as hand occlusion, lighting changes, and large body movement amplitudes, improving robustness by more than 50%.

[0021] 3. The sign language recognition method based on Mediapipe and XGBoost algorithms provided by this invention, with Mediapipe's efficient key point extraction and XGBoost's fast inference capability, reduces the response latency of the entire recognition process to less than 90ms, meeting the requirements of real-time interaction; the online incremental learning mechanism allows the model to adapt to the sign language habits of different users without retraining, greatly improving its practicality; it supports the recognition of complex sign language containing body movements and emotional expressions, adapts to users of different ages and body characteristics, and is applicable to various scenarios such as barrier-free communication terminals and public service robots. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart of the sign language recognition system described in this invention;

[0024] Figure 2 This is a flowchart of the multi-site data acquisition and preprocessing process described in this invention;

[0025] Figure 3 This is a flowchart illustrating the construction process of multi-part collaborative feature engineering as described in this invention;

[0026] Figure 4 This is a schematic diagram of the hand features described in this invention;

[0027] Figure 5 This is a schematic diagram of the arm features described in this invention;

[0028] Figure 6 This is a schematic diagram of the facial features described in this invention;

[0029] Figure 7 This is a flowchart of the dynamic optimization process of the XGBoost model described in this invention;

[0030] Figure 8 This is a flowchart of the hierarchical multimodal cross-validation process described in this invention. Detailed Implementation

[0031] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Example 1:

[0033] This embodiment provides a sign language recognition method based on Mediapipe and XGBoost algorithms;

[0034] Figure 1 The flowchart of the sign language recognition system of the present invention includes the following steps: S1: multi-part data acquisition and preprocessing; S2: multi-part coordinated feature engineering construction; S3: dynamic optimization of XGBoost model; S4: hierarchical multimodal cross-validation; S5: construction of sign language recognition system.

[0035] Step S1 involves multi-site data acquisition and preprocessing. Figure 2 The flowchart shown is for S1. The multi-part data acquisition and preprocessing steps include: S11: extraction of 21 key points of the hand, 3 core joint points of the arm (shoulder joint, elbow joint, wrist joint) and 68 core facial expression key points; S12: spatiotemporal synchronization calibration; S13: confidence filtering, interpolation completion and Gaussian filtering; S14: outputting the denoised multi-part key point dataset.

[0036] In step S11: the MediaPipe Hands algorithm is used to extract 21 key points of the hand; the MediaPipe Pose algorithm is used to extract 3 core joints of the arm (shoulder joint, elbow joint, and wrist joint); and the MediaPipe Face Mesh algorithm is used to extract 68 core facial expression key points.

[0037] In step S12: spatiotemporal synchronization calibration refers to ensuring that the extracted key point data is synchronized in time and space; the time synchronization means that the key point data of the hand, arm and face are extracted based on the same frame image / video stream at the same time; the spatial synchronization means that the key point data of the hand, arm and face are obtained based on the same coordinate system.

[0038] In step S13, the functions of confidence screening, interpolation completion, and Gaussian filtering are to remove noise and outliers, ensuring the reliability of key data.

[0039] Step S14 is to output a dataset of key points for multiple parts after denoising.

[0040] Figure 3 The above is a flowchart of the multi-part coordinated feature engineering construction process in step S2; the multi-part coordinated feature engineering construction includes: S21: extraction of hand, arm and face features; S22: splicing and fusion of multi-part features; S23: XGBoost feature importance screening and feature dimensionality reduction; S24: output of 80-dimensional fused feature vector.

[0041] The geometric features of the hand features in step S21 include: the degree of finger bending and the angle between two fingers;

[0042] The formula for calculating the degree of finger bending is as follows: Figure 4 As shown, taking the calculation of the degree of bending of the thumb as an example, let the wrist joint coordinate be B. hands (x hands-0 , y hands-0 The coordinates of the thumb metacarpophalangeal joint are A. hands (x hands-2 , y hands-2The coordinates of the thumb tip are C. hands (x hands-4 , y hands-4 At this point, the two vectors representing the degree of thumb bending are:

[0043]

[0044]

[0045] The degree of finger bending can be expressed as the angle θ between the two vectors mentioned above. hands The calculation formula is as follows:

[0046]

[0047] The formula for calculating the angle between the two fingers is as follows: Figure 4 As shown, taking the angle between the little finger and the ring finger as an example, let the coordinate of the metacarpophalangeal joint of the little finger be D. hands (x hands-17 , y hands-17 ), ring fingertip E hands (x hands-16 , y hands-16 ), little fingertip F hands (x hands-20 , y hands-20 At this point, the two vectors representing the angle between the little finger and the ring finger are:

[0048]

[0049]

[0050] The angle between the little finger and the ring finger can be represented by the angle α between the two vectors mentioned above. hands The calculation formula is as follows:

[0051]

[0052] In summary, the degree of bending of each finger and the angle between them can be calculated using the above methods.

[0053] The geometric features of the arm in step S21 include: the angle between the arm and the vertical direction, the angle between the line connecting the right shoulder joint and the wrist joint and the vertical direction, and the angle between the upper arm and the forearm.

[0054] The formula for calculating the angle between the arm and the vertical direction is as follows: Figure 5 As shown, taking the right arm and the vertical direction as an example, let the coordinate of the right elbow joint be B. pose (x pose-14 , y pose-14 The key coordinate for the right shoulder is C. pose(x pose-12 , y pose-12 At this point, the two vectors representing the arm and the vertical direction are:

[0055]

[0056]

[0057] The angle between the arm and the vertical direction can be represented by the angle θ between the two vectors mentioned above. pose The calculation formula is as follows:

[0058]

[0059] The formula for calculating the angle between the line connecting the right shoulder joint and wrist joint and the vertical direction is as follows: Figure 5 As shown, let the coordinates of the right wrist joint be A. pose (x pose-16 , y pose-16 At this point, the two vectors representing the line connecting the right shoulder joint and wrist joint and the perpendicular direction are:

[0060]

[0061]

[0062] The angle between the line connecting the right shoulder joint and wrist joint and the vertical direction can be expressed as the angle α between the two vectors mentioned above. pose The calculation formula is as follows:

[0063]

[0064] The formula for calculating the angle between the upper arm and forearm is as follows: Figure 5 As shown, the two vectors representing the line connecting the right shoulder joint and wrist joint and the vertical direction are:

[0065]

[0066]

[0067] The angle between the upper arm and forearm can be represented by the angle β between the two vectors mentioned above. pose The calculation formula is as follows:

[0068]

[0069] In summary, the angle between the left arm and the vertical direction, the angle between the line connecting the right shoulder joint and wrist joint and the vertical direction, and the angle between the upper arm and forearm can also be calculated using the above method.

[0070] The geometric features of the face in step S21 include: the width-to-height ratio of the lips, the vertical distance between the key points of the upper and lower eyelids and the opening and closing of the eyes, and the height and spacing of the key points of the eyebrows.

[0071] The formula for calculating the width-to-height ratio of the lips is as follows: Figure 5 As shown, let key point A be located at the upper left corner of the lip. face Coordinates are (x face-49 , y face-49 Key point B on the upper right corner of the lip face Coordinates are (x face-55 , y face-55 ), Key point C of the upper middle vertex of the lip face Coordinates are (x face-52 , y face-52 Key point D at the lower lip midpoint face Coordinates are (x face-58 , y face-58 At this point, the width W of the lips mouth and height H mouth It can be expressed by the following formula:

[0072]

[0073]

[0074] The width-to-height ratio of the lips to Radio mouth The calculation formula is as follows:

[0075]

[0076] The formulas for calculating the vertical distance between the midpoints of the upper and lower eyelids and the degree of eye opening are as follows: Figure 5 As shown, taking the right eye as an example, let E be the midpoint of the upper eyelid of the right eye. face Coordinates are (x face-45 , y face-45 ), midpoint of the right lower eyelid F face Coordinates are (x face-47 ,y face-47 At this point, the formula for calculating the vertical distance to the right eye is as follows:

[0077]

[0078] Let H0 be the vertical distance between the midpoints of the upper and lower eyelids when the right eye is fully open. At this time, the degree of opening / closing of the right eye is:

[0079]

[0080] The formulas for calculating the height and spacing of key eyebrow points are as follows: Figure 6 As shown, let the coordinates of the leftmost point of the left eyebrow be (x face-18, y face-18 The coordinates of the left eyebrow and left center are (x face-19 , y face-19 The coordinates of the midpoint of the left eyebrow are (x face-20 , y face-20 The coordinates of the left eyebrow and the right middle are (x face-21 , y face-21 The coordinates of the rightmost point of the left eyebrow are (x face-22 , y face-22 The coordinates of the leftmost point of the right eyebrow are (x face-23 , y face-23 The coordinates of the right eyebrow and the left center position are (x face-24 , y face-24 The coordinates of the midpoint of the right eyebrow are (x face-25 ,y face-25 The coordinates of the right eyebrow and the middle right position are (x face-26 , y face-26 The rightmost coordinate of the right eyebrow is (x face-27 , y face-27 Define the vertical distance between the midpoint of the eyebrow and the corresponding upper eyelid as the eyebrow height; define the distance between the midpoints of the left and right eyebrows as the eyebrow key point spacing;

[0081] The height of the left and right brow key points is the average height of the brow key points minus the average height of the corresponding upper eyelid key points. The calculation formula is as follows:

[0082]

[0083]

[0084] The formula for calculating the spacing between key eyebrow points is as follows:

[0085]

[0086] In summary, the geometric features selected for the sign language recognition system in this invention are: the degree of finger bending, the angle between two fingers, the angle between the arm and the vertical direction, the angle between the line connecting the right shoulder joint and wrist joint and the vertical direction, the angle between the upper arm and forearm, the width-to-height ratio of the lips, the vertical distance between the key points of the upper and lower eyelids, the degree of eye opening and closing, the height of the key points of the eyebrows, and the distance between the eyebrows. The features mentioned above are only some of the features, and the calculation formulas for the remaining features are as shown above.

[0087] The multi-part feature splicing and fusion S22 refers to splicing the feature vectors of the hand (189-dimensional), arm (19-dimensional), and face (24-dimensional) to form an initial 232-dimensional feature set;

[0088] XGBoost Feature Importance Screening and Feature Dimension Reduction S23 refers to using XGBoost's feature importance evaluation mechanism to screen out the top 80% of core features by weight, and then using principal component analysis (PCA) to reduce the feature dimension to 80 dimensions.

[0089] The output of the 80-dimensional fused feature vector S24 refers to the output of the 80-dimensional fused sign language feature vector from multiple parts.

[0090] Figure 7 The flowchart of the dynamic optimization process of the XGBoost model described in this invention includes: S31: Constructing an XGBoost classifier; S32: Introducing a dynamic weight allocation mechanism for multi-part features; S33: Grid search, cross-validation, and hyperparameter optimization; S34: Designing an online incremental learning algorithm; and S35: Outputting an optimized XGBoost model adapted to user habits.

[0091] The step S31 involves constructing an XGBoost classifier using a CART tree structure. The initial number of trees is set to 150-250, the learning rate is 0.1-0.3, and the maximum tree depth is 3-7.

[0092] The multi-part feature dynamic weight allocation mechanism in step S32 is as follows: feature weights are allocated according to the type (static / dynamic) and complexity of sign language movements: in dynamic sign language, the weight of arm features is increased to 0.2 (hand features 0.6, face features 0.2), and in static sign language, the weight of face features is increased to 0.3 (hand features 0.6, arm features 0.1); sample weights are allocated according to the complexity of the movement (determined by the combined variance of the dynamic features of the hands and arms), with the weight coefficient of high-complexity samples being 1.2-1.5 times and low-complexity samples being 0.7-0.9 times;

[0093] Step S33, grid search, cross-validation, and hyperparameter optimization, involves dividing the training set into a training subset and a validation subset in an 8:2 ratio. The combination of the number of trees, learning rate, and maximum tree depth is traversed through grid search, and the hyperparameter combination with the highest accuracy on the validation set is selected by cross-validation.

[0094] The S34 design of the online incremental learning algorithm: collects user feedback on recognition error samples in real time, updates the decision tree node weights of the XGBoost model through a local weighted regression algorithm, and only updates nodes with feature similarity higher than 0.6 with the error samples, without retraining the entire model, and adapts to the sign language habits and body characteristics of specific users.

[0095] Step S35 outputs an optimized XGBoost model adapted to user habits: the output is a dynamically optimized model that can automatically adapt to the sign language expression habits and body feature differences of different users in actual use, and can continuously optimize the recognition effect without retraining the entire model.

[0096] Figure 8 The flowchart of the hierarchical multimodal cross-validation described in this invention specifically includes the following steps: S41: constructing three single-part auxiliary classifiers; S42: independent classification of single-part features; S43: hierarchical weighted fusion; S44: scene-adaptive weight adjustment; and S45: outputting stable and reliable sign language recognition results.

[0097] Step S41 describes the construction of three single-part auxiliary classifiers: a hand feature classifier, an arm feature classifier, and a face feature classifier based on a lightweight CNN.

[0098] Step S42, independent classification of single-part features: The preprocessed single-part features of "hand, arm, and face" are input into the corresponding auxiliary classifiers to complete the classification calculation independently, resulting in 3 sets of non-interfering "single-part recognition results + confidence scores", providing raw data support for subsequent fusion;

[0099] The step S43, hierarchical weighted fusion, adopts a "two-layer progressive fusion" strategy. First, it fuses limb-related features (hands + arms), then it fuses facial expression features (face). Combining the weights of each layer with the reliability of single-part positions, it finally obtains a comprehensive recognition result, avoiding feature conflicts caused by simple splicing.

[0100] The aforementioned step S44 scene adaptive weight adjustment: For complex environmental scenes (insufficient light, hand occlusion), the weights of features of each part in the fusion process are dynamically adjusted to reduce the impact of environmental interference on weak part features and ensure recognition stability.

[0101] Step S45 outputs stable and reliable sign language recognition results: after the entire process of "independent classification of single parts → hierarchical fusion → scene adaptive adjustment", the final sign language category, comprehensive recognition confidence (0-1 interval) and feature contribution of each part (such as 60% for the hand, 25% for the arm, and 15% for the face) are output, ensuring the accuracy and interpretability of the results.

[0102] The aforementioned step S5, building a sign language recognition system, involves creating a sign language recognition app that can be embedded in a mobile phone.

[0103] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A sign language recognition method based on Mediapipe and XGBoost algorithms, characterized in that, Includes the following steps: S1: Multi-site data acquisition and preprocessing; S2: Construction of multi-part coordination feature engineering; S3: Dynamic optimization of the XGBoost model; S4: Hierarchical multimodal cross-validation; S5: Sign language recognition system setup.

2. The sign language recognition method based on Mediapipe and XGBoost algorithms according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11: Extraction of key points of the hand, core joints of the arm, and key points of the core facial expressions; S12: Spatiotemporal synchronization calibration; S13: Remove noise and outliers through confidence screening, interpolation completion, and Gaussian filtering; S14: Output the denoised dataset of key points for multiple parts.

3. The sign language recognition method based on Mediapipe and XGBoost algorithms according to claim 2, characterized in that, Step S2 specifically includes the following steps: S21: Extraction of hand, arm, and facial features; S22: Multi-part feature splicing and fusion; S23: XGBoost feature importance selection and feature dimensionality reduction reduce computational complexity while retaining key information, forming an 80-dimensional fused feature vector; S24: Output of the 80-dimensional fused feature vector.

4. The sign language recognition method based on Mediapipe and XGBoost algorithms according to claim 3, characterized in that, Step S3 specifically includes the following steps: S31: Construct the XGBoost classifier; S32: Introduces a dynamic weight allocation mechanism for multi-part features; S33: Employs a hyperparameter optimization strategy that combines grid search and cross-validation to avoid model overfitting; S34: Design an online incremental learning algorithm to update the decision tree node weights of the XGBoost model through local weighted regression; S35: Output an optimized XGBoost model adapted to user habits.

5. The sign language recognition method based on Mediapipe and XGBoost algorithms according to claim 4, characterized in that, Step S4 includes the following steps: S41: Construct 3 single-site auxiliary classifiers; S42: Independent classification of single-site features; S43: Hierarchical weighted fusion; S44: Scene-adaptive weight adjustment; S45: Outputs stable and reliable sign language recognition results.

6. The sign language recognition method based on Mediapipe and XGBoost algorithms according to claim 5, characterized in that, Step S5 specifically involves building a human-computer interaction system that can be used for real-time sign language recognition and embedding it into a mobile phone.

7. The sign language recognition method based on Mediapipe and XGBoost algorithms according to claim 6, characterized in that, In step S11, the extracted key points include 21 key points of the hand, 3 core joint points of the arm shoulder joint, elbow joint, and wrist joint, and 68 core facial expression key points.

8. The sign language recognition method based on Mediapipe and XGBoost algorithms according to claim 7, characterized in that, The geometric features of the hand in step S21 include: the degree of finger curvature and the angle between two fingers.

9. The sign language recognition method based on Mediapipe and XGBoost algorithms according to claim 8, characterized in that, In step S21, the geometric features of the arm include: the angle between the arm and the vertical direction, the angle between the line connecting the right shoulder joint and the wrist joint and the vertical direction, and the angle between the upper arm and the forearm.

10. The sign language recognition method based on Mediapipe and XGBoost algorithms according to claim 9, characterized in that, The geometric features of the face in step S21 include: the width-to-height ratio of the lips, the vertical distance between the key points of the upper and lower eyelids and the opening and closing of the eyes, and the height and spacing of the key points of the eyebrows.

Images