A basic sign language hand shape recognition method

Abstract

This invention provides a method for recognizing basic hand shapes in sign language. The method includes: acquiring multiple frames of basic hand shape images in sign language; extracting hand joint point data from the images using MediaPipe; calculating finger joint angles and inter-finger angles based on the extracted joint point dataset; establishing a classification model using the SVM method; reading the sign language video to be recognized; extracting hand joints from multiple consecutive frames of sign language images using MediaPipe; calculating finger joint angles, inter-finger angles, and palm direction data frame by frame; and inputting the angle data into a pre-trained SVM model to obtain the corresponding basic hand shape and hand shape direction. This invention enables more accurate recognition of basic hand shapes and understanding of sign language movements, improving the accuracy of basic hand shape recognition.

Description

Technical Field

[0001] This invention relates to the field of intelligent recognition technology, and more specifically, to a method for recognizing basic hand shapes in sign language. Background Technology

[0002] Sign language is a special form of communication and an important tool for hearing-impaired individuals to communicate with other groups in society. An efficient and accurate sign language recognition method is needed to help hearing-impaired individuals communicate more easily, improving their social skills and quality of life.

[0003] The inventors of this application discovered in their research that existing sign language recognition methods primarily treat words as the smallest semantic units in sign language, classifying input sign language words by collecting a large number of hand movement images. The development of image processing technology and deep learning has brought increasingly efficient and accurate methods for sign language recognition. However, this method, which uses words as the smallest semantic units, does not conform to the phonological structure of sign language. Furthermore, the large vocabulary and high complexity of sign language exacerbate the difficulty of sign language recognition. Therefore, based on the phonological structure of sign language, the elements constituting sign language can be divided into five categories: hand shape, position, movement, palm or finger orientation, and non-hand control information. Summary of the Invention

[0004] To address the aforementioned problems, the purpose of this invention is to provide a method for recognizing basic hand shapes in sign language, which identifies hand shape elements and palm orientation elements in sign language, thereby aiding in the recognition of sign language actions.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for recognizing basic sign language hand shapes includes the following steps:

[0007] Step 1: Acquire multiple frames of basic sign language hand shapes and extract hand joint data from the images using MediaPipe.

[0008] Step 2: Calculate the finger joint angles and inter-finger angles based on the extracted joint point dataset.

[0009] Where: (x i-1 ,y i-1 (x) is the coordinate of the previous joint point. i ,y i (x) represents the coordinates of the current joint. i+1 ,y i+1 ) is the coordinate of the next joint.

[0010] Angle between fingers

[0011] Wherein: P1 and P2 are the metacarpophalangeal joints and distal interphalangeal joints of the preceding finger; Q1 and Q2 are the metacarpophalangeal joints and distal interphalangeal joints of the following finger.

[0012] Step 3: Model using the SVM method by calling the sklearn package in Python. During modeling, a training set and a test machine are selected. First, the training set is trained using the `svm` function. The built SVM model is saved to `svm_model`. The `predict` function is then used to test the test set, calculating the accuracy (precision) of the SVM model on the test set. Based on the precision, the penalty parameters, kernel coefficients, and maximum iteration coefficients in the SVM model are adjusted.

[0013] Step 4: Read the sign language video to be recognized and extract the hand joints from multiple consecutive frames of sign language images using MediaPipe.

[0014] Step 5: Calculate finger joint angles, inter-finger angles, and palm orientation data frame by frame. Taking the left hand as an example, the palm orientation classification rules are as follows:

[0015] n = u × v = (u y v z -u z v y ,u z v x -u x v z ,u x v y -u y v x ,)=(n x ,n y ,n z )

[0016] max_component = max(|n x |,|n y |,|n z |)

[0017] When max_component = |n x |, and n x When the value is >0, the palm faces inward;

[0018] When max_component = |n x |, and n x When ≤0, the palm faces outward;

[0019] When max_component = |n y |, and n yWhen the value is >0, the palm faces upward;

[0020] When max_component = |n y |, and n y When ≤0, palm down;

[0021] When max_component = |n z |, and n z When the value is >0, the palm faces forward;

[0022] When max_component = |n z |, and n z When ≤0, the palm faces backward;

[0023] Where: u = p1 - p o v = p2 - p o p o p1 is the metacarpophalangeal joint of the index finger, and p2 is the metacarpophalangeal joint of the little finger.

[0024] Step 6: Input the hand joint angle data to be identified into the trained classification model svm_model, and output the corresponding basic sign language hand shape and hand shape direction. The recognition result is text.

[0025] The beneficial effects of this invention are: starting from the phonological structure of sign language, it identifies hand shape elements and palm direction elements in sign language, which helps to more accurately identify sign language movements. Attached Figure Description

[0026] Figure 1 This is a basic sign language hand shape diagram in an embodiment of the present invention.

[0027] Figure 2 This is a diagram showing the orientation of the palm in an embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram of the overall technical process of the basic hand shape recognition method for sign language described in this embodiment of the invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0030] Step 1: A method for recognizing basic hand shapes in sign language according to an embodiment of the present invention includes acquiring images of basic hand shapes. These images are taken by nine people, with ten images taken for each hand shape from each person. The coordinates of 21 joint points of the hand in the images are extracted using MediaPipe, including: wrist joint coordinates, thumb carpal joint coordinates, thumb metacarpophalangeal joint coordinates, thumb interphalangeal joint coordinates, thumb fingertip joint coordinates, index finger metacarpophalangeal joint coordinates, index finger proximal interphalangeal joint coordinates, index finger distal interphalangeal joint coordinates, index fingertip joint coordinates, middle finger metacarpophalangeal joint coordinates, middle finger proximal interphalangeal joint coordinates, middle finger distal interphalangeal joint coordinates, middle fingertip joint coordinates, ring finger metacarpophalangeal joint coordinates, ring finger proximal interphalangeal joint coordinates, ring finger distal interphalangeal joint coordinates, ring fingertip joint coordinates, little finger metacarpophalangeal joint coordinates, little finger proximal interphalangeal joint coordinates, little finger distal interphalangeal joint coordinates, and little fingertip joint coordinates.

[0031] Step 2: Calculate the finger joint angles and inter-finger angles based on the extracted joint point dataset.

[0032] Where: (x i-1 ,y i-1 (x) is the coordinate of the previous joint point. i ,y i (x) represents the coordinates of the current joint. i+1 ,y i+1 ) is the coordinate of the next joint.

[0033] Angle between fingers

[0034] Where: P1 and P2 are the metacarpophalangeal joints and distal interphalangeal joints of the preceding finger; Q1 and Q2 are the metacarpophalangeal joints and distal interphalangeal joints of the following finger.

[0035] Based on the above calculations, the hand joint angle θ is obtained. i ={θ i |i=12,3,…,14} and the angle β between fingers i ={β i |i=12,3,4}

[0036] Step 3: Build a classification model. The SVM method is used to build the classification model. This requires using the sklearn package in Python to complete the entire modeling process. The first step is to select a training set and a test set. Here, all data is shuffled, and 80% of the data is selected as the training set, and the remaining 20% ​​as the test set. In the training set, a `train_data` variable is needed to represent the explanatory variables, including hand joint angles and finger angles. Another variable, `train_label`, is needed to represent the dependent variable, which is the basic hand shape variable in sign language. Note that `train_label` can only contain one column of data, while `train_data` can contain multiple columns, each representing an angle. The number of rows in `train_data` and `train_label` should be the same. During training, the SVC function is called. The training statement is: `clf = svm.SVC(C = 39, gamma = 1e-04, max_iter = 12)`. The values ​​of `C`, `gamma`, and `max_iter` can be adjusted according to the actual situation. After training, the built SVM model is saved in `svm_model`. Next, we will use the established SVM model `svm_model` to test the test set. During testing, we will call the `predict` function with the statement: `train_result = clf.predict(train_data)`. After the test is complete, we will print out the test results, i.e., the classification accuracy.

[0037] Step 4: Then read the sign language video to be recognized and extract the hand joints from multiple consecutive frames of sign language images using MediaPipe.

[0038] Step 5: Calculate the finger joint angles, inter-finger angles, and palm orientation data based on the joint point data. Taking the left hand as an example, the palm orientation classification rules are as follows:

[0039] n = u × v = (y y v z -u z v y ,y y v z -u x v z ,u x v y -u y v x ,)=(n x ,n y ,n z )

[0040] max_component = max(|n x |,|ny |,|n z |)

[0041] When max_component = |n x |, and n x When the value is >0, the palm faces inward;

[0042] When max_component = |n x |, and n x When ≤0, the palm faces outward;

[0043] When max_component = |n y |, and n y When the value is >0, the palm faces upward;

[0044] When max_component = |n y |, and n y When ≤0, palm down;

[0045] When max_component = |n z |, and n z When the value is >0, the palm faces forward;

[0046] When max_component = |n z |, and n z When ≤0, the palm faces backward;

[0047] Where: u = p1 - p o v = p2 - p o p o p1 is the metacarpophalangeal joint of the index finger, and p2 is the metacarpophalangeal joint of the little finger.

[0048] Step 6: Input the obtained finger joint angle data and inter-finger angle data into the trained classification model svm_model, output the corresponding basic sign language hand shape and hand shape direction, and finally recognize the results as text.

Claims

1. A method for recognizing basic hand shapes in sign language, characterized in that, Includes the following steps: Step 1: Acquire multiple frames of basic sign language hand shape images, and extract hand joint data from the images using MediaPipe; Step 2: Calculate the finger joint angles and inter-finger angles based on the extracted joint point dataset. The calculation formula is as follows: Joint angle Where: (x i-1 ,y i-1 (x) is the coordinate of the previous joint point. i ,y i (x) represents the coordinates of the current joint. i+1 ,y i+1 () represents the coordinates of the next joint; the angle between the fingers. Where: P1 and P2 are the metacarpophalangeal joints and distal interphalangeal joints of the preceding finger; Q1 and Q2 are the metacarpophalangeal joints and distal interphalangeal joints of the following finger. Step 3: Build a classification model using the SVM method. When modeling, select a training set and a test set. First, train the training set by calling the svm function. Save the built SVM model to svm_model. Then, test the test set by calling the predict function and calculate the accuracy of the SVM model on the test set. Step 4: Read the sign language video to be recognized, and extract the hand joints from multiple consecutive frames of sign language images using MediaPipe; Step 5: Calculate finger joint angles, inter-finger angles, and palm orientation data frame by frame. Taking the left hand as an example, the palm orientation classification rules are as follows: n=u×v=(u y v z -u z v y ,u z v x -u x v z ,u x v y -u y v x ,)=(n x ,n y ,n z ) max_component=max(|n x |,|n y |,|n z |) When max_component = |n x |, and n x When the value is >0, the palm faces inward; When max_component = |n x |, and n x When ≤0, the palm faces outward; When max_component = |n y |, and n y When the value is >0, the palm faces upward; When max_component = |n y |, and n y When ≤0, palm down; When max_component = |n z |, and n z When the value is >0, the palm faces forward; When max_component = |n z |, and n z When ≤0, the palm faces backward; Where: u = p1 - p0, v = p2 - p0, p0 is the wrist joint, p1 is the metacarpophalangeal joint of the index finger, and p2 is the metacarpophalangeal joint of the little finger. Step 6: Input the obtained finger joint angle data and inter-finger angle data into the trained classification model svm_model, output the corresponding basic sign language hand shape and hand shape direction, and finally recognize the results as text.

2. The sign language basic hand shape recognition method according to claim 1, characterized in that, There are 61 basic sign language hand shapes collected in step 1.

3. The sign language basic hand shape recognition method according to claim 1, characterized in that, The hand joint data collected in step 1 includes the joint data of the collector's two hands, w = {w left ,w right },in: This data is collected using the multi_hand_landmarks function in Mediapipe, and the results are saved in featureCode.txt.

4. The sign language basic hand shape recognition method according to claim 1, characterized in that, The joint angle data obtained in step 2 are used to remove outliers using a local anomaly factor algorithm.

5. The sign language basic hand shape recognition method according to claim 1, characterized in that, In step 4, the training set, the train_data variable represents the explanatory variables, including the angles of each joint of the hand and the angles between the fingers, and the train_label variable represents the dependent variable, namely the basic hand shape variables of sign language.

6. The sign language basic hand shape recognition method according to claim 1, characterized in that, In step 5, the video is read using the VideoCapture function of cv2 to obtain frame-by-frame hand joint data.

7. The sign language basic hand shape recognition method according to claim 1, characterized in that, Step S7 calls the pre-trained svm_model to obtain the corresponding basic sign language hand shapes, and converts the basic sign language hand shapes into text using dictionary.txt.

Images