A gesture recognition method, apparatus, device and medium

Abstract

The application provides a gesture recognition method, device, equipment and medium, the method comprises the following steps: S1, acquiring a single frame image, detecting hand three-dimensional key points of a hand in the single frame image, and obtaining a hand three-dimensional key point set P; S2, rotating the hand three-dimensional key points to a standard space angle, and obtaining a standard space angle point set Q; S3, calculating hand joint three-dimensional vectors; S4, calculating cosine similarity between the hand joint three-dimensional vectors; S5, matching the same gesture template according to the joint cosine similarity, and outputting a matching result. The application can ensure the reliability and stability of the algorithm for gesture recognition under the condition of only single gesture template picture data set, thereby providing a technical basis for user-defined gestures and gesture update.

Description

Technical Field

[0001] This invention relates to the field of gesture recognition technology, and in particular to a method, apparatus, device, and medium for recognizing hand gestures. Background Technology

[0002] The current process of gesture recognition technology is as follows: Figure 1 As shown, the process generally includes the following steps: First, perform hand target detection on the input image to obtain the hand ROI (Region of Interest); then, input the hand ROI image into the gesture classification and discrimination model to obtain the gesture classification result.

[0003] In current mainstream gesture recognition algorithms, gesture classification is typically based on deep learning models. Training these models usually requires tens of thousands of images of different gestures from various scenarios. The method involves training the model on a large dataset of gesture images and then using that model to classify the images. However, this approach demands a high quantity and quality of data and cannot update gestures online.

[0004] In engineering practice, there are problems such as high cost of acquiring datasets, long production cycle of data machines, and the need for a large amount of new gesture data to add new gestures. If there is an algorithm that can guarantee the reliability and stability of gesture recognition when there is only a single gesture template image dataset, it would provide a technical basis for user-defined gestures and gesture updates, which would be very meaningful. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a gesture recognition method, device, equipment and medium that can ensure the reliability and stability of the algorithm for gesture recognition with only a single gesture template image dataset, thereby providing a technical basis for user-defined gestures and gesture updates.

[0006] In a first aspect, the present invention provides a gesture recognition method, comprising the following steps:

[0007] S1. Acquire a single-frame image, detect the three-dimensional key points of the hand in the single-frame image, and obtain the set of three-dimensional key points of the hand P;

[0008] S2. Rotate the three-dimensional key points of the hand to a standard space angle to obtain the standard space angle point set Q;

[0009] S3. Calculate the three-dimensional vector of the hand joints;

[0010] S4. Calculate the cosine similarity between the three-dimensional vectors of hand joints;

[0011] S5. Match identical gesture templates based on the joint cosine similarity and output the matching results.

[0012] In a second aspect, the present invention provides a gesture recognition device, comprising:

[0013] The key point detection module is used to detect the three-dimensional key points of the hand in a single frame image, and obtain the set of three-dimensional key points of the hand P;

[0014] The conversion module is used to rotate the three-dimensional key points of the hand to a standard spatial angle to obtain a standard spatial angle point set Q;

[0015] The vector calculation module is used to calculate the three-dimensional vectors of hand joints;

[0016] The similarity calculation module is used to calculate the cosine similarity between the three-dimensional vectors of hand joints;

[0017] The matching module is used to match identical gesture templates based on the joint cosine similarity and output the matching results.

[0018] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0019] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0020] The one or more technical solutions provided in the embodiments of this invention have at least the following technical effects or advantages: by detecting three-dimensional key points of the hand, calculating three-dimensional vectors based on the three-dimensional key points of the hand, and matching gestures by calculating the cosine similarity between the three-dimensional vectors of the hand joints. This method overcomes the shortcomings of gesture recognition based on classification models, which require a large dataset to train the model, and has advantages such as strong ability to distinguish approximate gestures, simple addition of new gestures, simple algorithm implementation, and fast running speed. In particular, by correcting the three-dimensional key points of the hand to a standard spatial angle through Rogrigues rotation, the success rate of the algorithm in recognizing the same gesture under different camera perspectives is greatly improved.

[0021] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] Figure 1This is a flowchart of the method in Embodiment 1 of the present invention;

[0024] Figure 2 This is a schematic diagram of the three-dimensional key point set P of the hand in an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram of the initial set of rotation control points and the target set of points in standard spatial angles in an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of the three-dimensional vector of the hand joint in an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram illustrating the cosine similarity calculation state between the three-dimensional vectors of hand joints in an embodiment of the present invention.

[0028] Figure 6 This is a schematic diagram illustrating the state of matching gesture templates based on joint cosine similarity in an embodiment of the present invention;

[0029] Figure 7 This is a schematic diagram of the device in Embodiment 2 of the present invention;

[0030] Figure 8 This is a schematic diagram of the electronic device in Embodiment 3 of the present invention;

[0031] Figure 9 This is a schematic diagram of the structure of the medium in Embodiment 4 of the present invention. Detailed Implementation

[0032] This application provides a gesture recognition method, apparatus, device, and medium that can ensure the reliability and stability of the algorithm for gesture recognition with only a single gesture template image dataset, thereby providing a technical basis for user-defined gestures and gesture updates.

[0033] The technical solution in this application embodiment has the following general idea: It detects the three-dimensional key points of the hand in a single frame image (using a neural network). mediapipe hands The method involves detecting and obtaining a set of 3D keypoints P for the hand. Based on these keypoints, 3D vectors of the hand joints are calculated, and gestures are matched by calculating the cosine similarity between these vectors. This method overcomes the limitation of classification-based gesture recognition requiring large datasets for training, while offering advantages such as strong ability to distinguish approximate gestures, ease of adding new gestures, simple algorithm implementation, and fast execution speed. Specifically, the Rogrigues rotation correction of the 3D keypoints to a standard spatial angle significantly improves the algorithm's success rate in recognizing the same gesture from different camera perspectives.

[0034] Example 1

[0035] like Figure 1 As shown, this embodiment provides a gesture recognition method, including the following steps:

[0036] S1. Acquire a single-frame image, detect the three-dimensional key points of the hand in the single-frame image, and obtain the set of three-dimensional key points of the hand P;

[0037] S2. Rotate the three-dimensional key points of the hand to a standard space angle to obtain the standard space angle point set Q;

[0038] S3. Calculate the three-dimensional vector of the hand joints;

[0039] S4. Calculate the cosine similarity between the three-dimensional vectors of hand joints;

[0040] S5. Match the same gesture template based on the joint cosine similarity and output the matching result.

[0041] The detection of the three-dimensional key points of the hand is performed by the neural network Mediapipe hands; the neural network Mediapipe hands can infer the x-axis, y-axis coordinates and z-axis depth coordinates of the key points of the hand from a single frame, thereby obtaining the initial coordinates of the key points of the hand, which are represented by 21 three-dimensional coordinate points (x, y, z).

[0042] like Figure 2 As shown, the three-dimensional key points of the hand include 21 nodes, namely the metacarpophalangeal joint, metacarpophalangeal joint, interphalangeal joint, and fingertips, denoted as {A,B,C,...,U}. The set of three-dimensional key points of the hand, P, is represented as P={A,B,C,...,U}.

[0043] The three-dimensional vector of the hand joint corresponds to any joint of the metacarpal or phalangeal bones.

[0044] Specifically, S2 is:

[0045] like Figure 3 As shown, three rotation control points are selected from the three-dimensional keypoint set P = {A, B, C, ..., U} of the hand to obtain the initial point set P of the rotation control points. tri ={A,J,R}, such as Figure 3 As shown in (a), the initial point set P tri ={A,J,R} corresponds to the target point set Q in standard space angles. tri ={A std J std ,R std},like Figure 3 As shown in (b); where A and A std Corresponding to the carpal joint, J and J std Corresponding to the metacarpophalangeal joint of the middle finger, R and Rstd This corresponds to the metacarpophalangeal joint of the little finger;

[0046] Therefore, the standard spatial angle of the three-dimensional key points of the hand is defined as follows:

[0047] Point A std Point J is the origin of the coordinate system. std Point R lies on the positive Y-axis. std Located on the XY-axis plane, that is:

[0048] A std = (0, 0, 0)

[0049] J std = (0, y j ,0)

[0050] R std =(x r y r ,0)

[0051] Calculate point A in standard spatial angles std Point L std Point R std After finding the coordinates, solve for the relationship between points A, J, and R and point A. std Point L std Point R std The homography matrix H between them:

[0052]

[0053]

[0054]

[0055] Perform SVD decomposition on the homography matrix H to obtain P. tri To Q tri The rotation transformation matrix R and the translation transformation vector T:

[0056] [U, S, V] = SVD(H)

[0057] R = VU T

[0058]

[0059] Applying the Rodrigues rotation transformation to the set of three-dimensional key points P = {A, B, C, ..., U} of the hand, we obtain the standard spatial angle point set Q = {A, B, C, ..., U}, that is:

[0060] Q = RP + T.

[0061] Specifically, S3 is:

[0062] For the standard spatial angle point set Q, the 21 points of the hand's three-dimensional key point set P = {A, B, C, ..., U} are divided into five groups of finger points according to the fingers, as follows:

[0063] Thumb = {A, B, C, D, E}

[0064] Indexfinger={A, F, G, H, I}

[0065] Middlefinger={A, J, K, L, M}

[0066] Ringfinger={A, N, O, P, Q}

[0067] Pinky={A, R, S, T, U};

[0068] like Figure 4 As shown, the three-dimensional vector of the hand joint for each group of finger points is calculated from the point coordinates, resulting in:

[0069]

[0070] Specifically, S4 is:

[0071] Calculate the cosine similarity S between the 3D vector of the hand joints of the current gesture and the 3D vector of the hand joints at the corresponding positions in the gesture template in the template library. i ;

[0072] like Figure 5 As shown, the cosine similarity S of the three-dimensional vectors of the hand joints is calculated respectively. i The calculation formula is as follows:

[0073]

[0074] The cosine similarity calculation result S of the three-dimensional vectors of a single hand joint i The range is [-1, 1].

[0075] S i The larger the value, the higher the similarity between the joint's 3D vectors.

[0076] S i =1 indicates that vectors A and B have the same direction.

[0077] S i =-1 indicates that vectors A and B are in completely opposite directions;

[0078] The set S of cosine similarity of joint vectors between gestures is defined as:

[0079]

[0080] Specifically, S5 is:

[0081] When the set S of cosine similarity of joint vectors between gestures is S, the similarity S of all three-dimensional vectors of hand joints. i If all values ​​are greater than the threshold t, then the overall similarity M of the entire gesture is calculated using the following formula:

[0082]

[0083] N=20, and the corresponding gesture templates are used as candidate gestures;

[0084] If, within the entire gesture, there is a similarity S between the three-dimensional vectors of a certain hand joint... i If the similarity is below the threshold t (the size can be set according to the accuracy requirements), the overall similarity of the entire gesture is directly judged to be 0, and the corresponding gesture template is filtered out.

[0085] Once all gesture templates in the template library have been matched, the gesture template with the highest similarity M value among all candidate gestures is selected as the matching result and output.

[0086] like Figure 6 As shown, if the current gesture is the FIVE gesture, the vector diagram of the FIVE gesture is shown in (a). When it is matched with the OK gesture template in the template library, the vector diagram of the OK gesture template is shown in (b). The similarity S of the three-dimensional vectors of all hand joints is determined. i When considering the threshold value t, it was found that not all hand joint 3D vectors have a similarity S. i If all similarities are greater than the threshold t, the overall similarity M of the entire gesture is determined to be 0, and the OK gesture template is filtered out. The overall similarity M with the next gesture template is then determined. After all gesture templates in the template library have been matched, the gesture template with the highest similarity M value among all candidate gestures is selected as the matching result and output.

[0087] Based on the same inventive concept, this application also provides an apparatus corresponding to the method in Embodiment 1, as detailed in Embodiment 2.

[0088] Example 2

[0089] like Figure 7 As shown, this embodiment provides a gesture recognition device, including:

[0090] The key point detection module is used to detect the three-dimensional key points of the hand in a single frame image, and obtain the set of three-dimensional key points of the hand P;

[0091] The conversion module is used to rotate the three-dimensional key points of the hand to a standard spatial angle to obtain a standard spatial angle point set Q;

[0092] The vector calculation module is used to calculate the three-dimensional vectors of hand joints;

[0093] The similarity calculation module is used to calculate the cosine similarity between the three-dimensional vectors of hand joints;

[0094] The matching module is used to match identical gesture templates based on the joint cosine similarity and output the matching results.

[0095] like Figure 2 As shown, the detection of the three-dimensional key points of the hand is performed through the neural network mediapipe hands;

[0096] The three-dimensional key points of the hand include 21 nodes, including the metacarpophalangeal joint, metacarpophalangeal joint, interphalangeal joint, and fingertips, denoted as {A,B,C,...,U}. The set of three-dimensional key points of the hand, P, is represented as P={A,B,C,...,U}.

[0097] The three-dimensional vector of the hand joint corresponds to any joint of the metacarpal or phalangeal bones.

[0098] like Figure 3 As shown, the conversion module is specifically used for:

[0099] Three rotation control points are selected from the three-dimensional keypoint set P = {A, B, C, ..., U} of the hand to obtain the initial point set P of the rotation control points. tri ={A,J,R}, the initial point set P tri ={A,J,R} corresponds to the target point set Q in standard space angles. tri ={A std J std ,R std}; where A and A std Corresponding to the carpal joint, J and J std Corresponding to the metacarpophalangeal joint of the middle finger, R and R std This corresponds to the metacarpophalangeal joint of the little finger;

[0100] Therefore, the standard spatial angle of the three-dimensional key points of the hand is defined as follows:

[0101] Point A std Point J is the origin of the coordinate system. std Point R lies on the positive Y-axis. std Located on the XY-axis plane, that is:

[0102] A std = (0, 0, 0)

[0103] J std = (0, y j ,0)

[0104] R std =(x r g r ,0)

[0105] Calculate point A in standard spatial angles std Point L std Point R std After finding the coordinates, solve for the relationship between points A, J, and R and point A. std Point L std Point R std The homography matrix H between them:

[0106]

[0107]

[0108]

[0109] Perform SVD decomposition on the homography matrix H to obtain P. tri To Q tri The rotation transformation matrix R and the translation transformation vector T:

[0110] [U, S, V] = SVD(H)

[0111] R = VU T

[0112]

[0113] Applying the Rodrigues rotation transformation to the set of three-dimensional key points P = {A, B, C, ..., U} of the hand, we obtain the standard spatial angle point set Q = {A, B, C, ..., U}, that is:

[0114] Q = RP + T.

[0115] The vector calculation module is specifically used for:

[0116] For the standard spatial angle point set Q, the 21 points of the hand's three-dimensional key point set P = {A, B, C, ..., U} are divided into five groups of finger points according to the fingers, as follows:

[0117] Thumb = {A, B, C, D, E}

[0118] Indexfinger={A, F, G, H, I}

[0119] Middlefinger={A, J, K, L, M}

[0120] Ringfinger={A, N, O, P, Q}

[0121] Pinky={A, R, S, T, U};

[0122] like Figure 4 As shown, the three-dimensional vector of the hand joint for each group of finger points is calculated from the point coordinates, resulting in:

[0123]

[0124] The similarity calculation module is specifically used for:

[0125] Calculate the cosine similarity M between the three-dimensional vector of the hand joints of the current gesture and the three-dimensional vector of the hand joints at the corresponding positions in the gesture template in the template library;

[0126] like Figure 5 As shown, the cosine similarity S of the three-dimensional vectors of the hand joints is calculated respectively. i The calculation formula is as follows:

[0127]

[0128] The cosine similarity calculation result S of the three-dimensional vectors of a single hand joint i The range is [-1, 1]. S i The larger the value, the higher the similarity between the joint's three-dimensional vectors. i =1 indicates that vectors A and B have the same direction, S i =-1 indicates that vectors A and B are in completely opposite directions;

[0129] The set S of cosine similarity of joint vectors between gestures is defined as:

[0130]

[0131] The matching module is specifically used for:

[0132] When the set S of cosine similarity of joint vectors between gestures is S, the similarity S of all three-dimensional vectors of hand joints. i If all values ​​are greater than the threshold t, then the overall similarity M of the entire gesture is calculated using the following formula:

[0133]

[0134] N=20, and the corresponding gesture templates are used as candidate gestures;

[0135] If, within the entire gesture, there is a similarity S between the three-dimensional vectors of a certain hand joint... iIf the similarity is below the threshold t, the overall similarity of the entire gesture is directly determined to be 0, and the corresponding gesture template is filtered out.

[0136] Once all gesture templates in the template library have been matched, the gesture template with the highest similarity M value among all candidate gestures is selected as the matching result and output.

[0137] like Figure 6 As shown, if the current gesture is the FIVE gesture, the vector diagram of the FIVE gesture is shown in (a). When it is matched with the OK gesture template in the template library, the vector diagram of the OK gesture template is shown in (b). The similarity S of the three-dimensional vectors of all hand joints is determined. i When considering the threshold value t, it was found that not all hand joint 3D vectors have a similarity S. i If all similarities are greater than the threshold t, the overall similarity M of the entire gesture is determined to be 0, and the OK gesture template is filtered out. The overall similarity M with the next gesture template is then determined. After all gesture templates in the template library have been matched, the gesture template with the highest similarity M value among all candidate gestures is selected as the matching result and output.

[0138] Since the apparatus described in Embodiment 2 of the present invention is an apparatus used to implement the method of Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in Embodiment 1 of the present invention, and therefore will not be described again here. All apparatuses used in the method of Embodiment 1 of the present invention fall within the scope of protection of the present invention.

[0139] Based on the same inventive concept, this application provides an electronic device embodiment corresponding to Embodiment 1, as detailed in Embodiment 3.

[0140] Example 3

[0141] This embodiment provides an electronic device, such as... Figure 8 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement any of the embodiments in Example 1.

[0142] Since the electronic device described in this embodiment is the device used to implement the method in Embodiment 1 of this application, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the method described in Embodiment 1 of this application. Therefore, how the electronic device implements the method in the embodiment of this application will not be described in detail here. Any device used by those skilled in the art to implement the method in the embodiment of this application falls within the scope of protection of this application.

[0143] Based on the same inventive concept, this application provides a storage medium corresponding to Embodiment 1, as detailed in Embodiment 4.

[0144] Example 4

[0145] This embodiment provides a computer-readable storage medium, such as... Figure 9 As shown, a computer program is stored thereon, which, when executed by a processor, can implement any of the embodiments in Example 1.

[0146] The methods, apparatus, systems, devices, and media provided in this application have at least the following technical effects or advantages: By detecting three-dimensional key points of the hand, calculating three-dimensional vectors based on these key points, and matching gestures by calculating the cosine similarity between the three-dimensional vectors of hand joints, this method overcomes the shortcomings of classification-based gesture recognition, which requires large datasets to train the model. It also has advantages such as strong ability to distinguish approximate gestures, ease of adding new gestures, simple algorithm implementation, and fast running speed. In particular, by correcting the three-dimensional key points of the hand to a standard spatial angle using Rogrigues rotation, the algorithm significantly improves the recognition success rate of the same gesture under different camera perspectives.

[0147] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or systems, or as computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A gesture recognition method, characterized in that: Includes the following steps: S1. Acquire a single-frame image and detect the three-dimensional key points of the hand in the single-frame image to obtain a set of three-dimensional key points P; the three-dimensional key points of the hand include 21 nodes, including the metacarpophalangeal joint, metacarpophalangeal joint, interphalangeal joint, and fingertips, denoted as P. Then the set of three-dimensional key points P of the hand is represented as ; S2. Rotate the three-dimensional key points of the hand to a standard spatial angle to obtain the standard spatial angle point set Q; specifically: In the set of 3D key points of the hand Select three rotation control points to obtain the initial set of rotation control points. The initial point set The target point set corresponding to the standard spatial angle Among them, A and Corresponding to the carpal joint, J and Corresponding to the metacarpophalangeal joint of the middle finger, R and This corresponds to the metacarpophalangeal joint of the little finger; Therefore, the standard spatial angle of the three-dimensional key points of the hand is defined as follows: point The origin of the coordinate system; point Point on the positive Y-axis; Located on the XY-axis plane, that is: ; Calculate points in standard spatial angles ,point ,point After finding the coordinates, solve for the relationship between points A, J, R, and point J. ,point ,point The homography matrix H between them: ; Perform SVD decomposition on the homography matrix H to obtain arrive The rotation transformation matrix R and the translation transformation vector T: ; For the set of three-dimensional key points of the hand Apply the Rodrigues rotation transformation to obtain the standard spatial angle point set. ,Right now: ; S3. Calculate the three-dimensional vector of the hand joints; S4. Calculate the cosine similarity between the three-dimensional vector of the hand joint and the three-dimensional vector of the hand joint at the corresponding position in the gesture template in the template library; S5. Calculate the overall similarity of the entire gesture based on the cosine similarity, and match the same gesture template based on the overall similarity to output the matching result.

2. The method according to claim 1, characterized in that: The detection of the three-dimensional key points of the hand is performed using a neural network called mediapipe hands; The three-dimensional vector of the hand joint corresponds to any joint of the metacarpal or phalangeal bones.

3. The method according to claim 2, characterized in that: Specifically, S3 is: For the standard spatial angle point set Q, the set of three-dimensional key points of the hand is... The 21 points are divided into five groups based on the fingers, as follows: ; Calculate the three-dimensional vector of the hand joint for each group of finger points from the point coordinates, and obtain: ; Specifically, S4 is: Calculate the cosine similarity between the 3D vector of the hand joints of the current gesture and the 3D vector of the hand joints at the corresponding positions in the gesture template in the template library. ; Cosine similarity calculation results of 3D vectors of a single hand joint The range is i= , , … ; The larger the value, the higher the similarity between the joint's 3D vectors. This indicates that vectors A and B have the same direction. This indicates that vectors A and B have completely opposite directions; The set S of cosine similarity of joint vectors between gestures is defined as: ; Specifically, S5 is: The similarity of all three-dimensional vectors of hand joints in the set S of cosine similarity of joint vectors between gestures. If all values ​​are greater than the threshold t, then the overall similarity of the entire gesture is calculated using the following formula. : ; N=20, and the corresponding gesture templates are used as alternative gestures; If, within the entire gesture, there is a similarity score between the three-dimensional vectors of a certain hand joint... If the similarity is below the threshold t, the overall similarity of the gesture is directly determined to be 0, and the corresponding gesture template is filtered out. Once all gesture templates in the template library have been matched, the gesture template with the highest similarity M value among all candidate gestures is selected as the matching result and output.

4. A gesture recognition device, characterized in that: include: The keypoint detection module is used to detect the three-dimensional keypoints of the hand in a single frame image, obtaining a set P of three-dimensional keypoints of the hand; the three-dimensional keypoints of the hand include 21 nodes, including the wrist joint, metacarpophalangeal joint, interphalangeal joint, and fingertips, denoted as P. Then the set of three-dimensional key points P of the hand is represented as ; The conversion module is used to rotate the three-dimensional key points of the hand to a standard spatial angle to obtain a standard spatial angle point set Q; the body is used for: In the set of 3D key points of the hand Select three rotation control points to obtain the initial set of rotation control points. The initial point set The target point set corresponding to the standard spatial angle Among them, A and Corresponding to the carpal joint, J and Corresponding to the metacarpophalangeal joint of the middle finger, R and This corresponds to the metacarpophalangeal joint of the little finger; Therefore, the standard spatial angle of the three-dimensional key points of the hand is defined as follows: point The origin of the coordinate system; point Point on the positive Y-axis; Located on the XY-axis plane, that is: ; Calculate points in standard spatial angles ,point ,point After finding the coordinates, solve for the relationship between points A, J, R, and point J. ,point ,point The homography matrix H between them: ; Perform SVD decomposition on the homography matrix H to obtain arrive The rotation transformation matrix R and the translation transformation vector T: ; For the set of three-dimensional key points of the hand Apply the Rodrigues rotation transformation to obtain the standard spatial angle point set. ,Right now: ; The vector calculation module is used to calculate the three-dimensional vectors of the hand joints; The similarity calculation module is used to calculate the cosine similarity between the three-dimensional vector of the hand joint and the three-dimensional vector of the hand joint at the corresponding position in the gesture template in the template library; The matching module is used to calculate the overall similarity of the entire gesture based on the cosine similarity, match the same gesture template based on the overall similarity, and output the matching result.

5. The apparatus according to claim 4, characterized in that: The detection of the three-dimensional key points of the hand is performed using a neural network called mediapipe hands; The three-dimensional vector of the hand joint corresponds to any joint of the metacarpal or phalangeal bones.

6. The apparatus according to claim 5, characterized in that: The vector calculation module is specifically used for: For the standard spatial angle point set Q, the set of three-dimensional key points of the hand is... The 21 points are divided into five groups based on the fingers, as follows: ; Calculate the three-dimensional vector of the hand joint for each group of finger points from the point coordinates, and obtain: ; The similarity calculation module is specifically used for: Calculate the cosine similarity M between the three-dimensional vector of the hand joints of the current gesture and the three-dimensional vector of the hand joints at the corresponding positions in the gesture template in the template library; Cosine similarity calculation results of 3D vectors of a single hand joint The range is i= , , … ; The larger the value, the higher the similarity between the joint's 3D vectors. This indicates that vectors A and B have the same direction. This indicates that vectors A and B have completely opposite directions; The set S of cosine similarity of joint vectors between gestures is defined as: ; The matching module is specifically used for: The similarity of all three-dimensional vectors of hand joints in the set S of cosine similarity of joint vectors between gestures. If all values ​​are greater than the threshold t, then the overall similarity of the entire gesture is calculated using the following formula. : ; N=20, and the corresponding gesture templates are used as alternative gestures; If, within the entire gesture, there is a similarity score between the three-dimensional vectors of a certain hand joint... If the similarity is below the threshold t, the overall similarity of the gesture is directly determined to be 0, and the corresponding gesture template is filtered out. Once all gesture templates in the template library have been matched, the gesture template with the highest similarity M value among all candidate gestures is selected as the matching result and output.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 3.

Images