Gesture recognition method and device, computer device

Abstract

The present disclosure relates to a gesture recognition method, device and computer equipment. The method comprises: in the case of triggering gesture recognition, acquiring the spatial coordinates of each joint in the hand to be recognized; calculating the first included angle between the tip joint and the palm joint of each finger in the hand to be recognized and the projection distance according to the pre-established projection coordinate system and the spatial coordinates, and determining the bending degree of each finger according to the first included angle and the projection distance; wherein the projection coordinate system is established according to the palm joint and the wrist joint in the hand to be recognized; and determining the gesture recognition result of the hand to be recognized according to the bending degree of each finger. The present method can recognize a variety of complex gestures without a large amount of calculation.

Description

Technical Field

[0001] This disclosure relates to the field of gesture data processing, and in particular to a gesture recognition method, apparatus, and computer device. Background Technology

[0002] With the development of the times and the iteration of hardware devices, human-computer interaction has evolved from mouse clicks, represented by Windows desktop applications, to taps and swipes, represented by mobile device applications, and now to stereoscopic gestures and voice, represented by MR / AR smart glasses devices such as HoloLens. The way human-computer interaction is becoming more and more like the natural behavior of human users in reality. Among them, gesture recognition is one of the most important and popular research directions in interaction behavior.

[0003] Currently, the most commonly used recognition methods in the field of gesture recognition are sensor tracking schemes based on wearable devices and deep learning model recognition schemes based on image recognition.

[0004] However, sensor tracking solutions based on wearable devices can only recognize basic gestures, and the types of gestures they can recognize are limited. They cannot recognize complex gestures, while deep learning model recognition solutions based on image recognition are computationally intensive and highly complex. Summary of the Invention

[0005] Therefore, it is necessary to provide a gesture recognition method, device, or computer equipment that can recognize a variety of complex gestures without requiring a large amount of computation, in order to address the aforementioned technical problems.

[0006] Firstly, this disclosure provides a gesture recognition method. The method includes:

[0007] When gesture recognition is triggered, obtain the spatial coordinates of each joint in the hand to be recognized;

[0008] Based on the pre-established projection coordinate system and the spatial coordinates, the first included angle and projection distance between the fingertip joint and the palm joint of each finger in the hand to be identified are calculated, and the curvature of each finger is determined based on the first included angle and the projection distance; wherein, the projection coordinate system is established based on the palm joint and wrist joint of the hand to be identified;

[0009] The gesture recognition result of the hand to be recognized is determined based on the curvature of each finger.

[0010] In one embodiment, the process of establishing the projected coordinate system includes:

[0011] The origin of the coordinate system is determined based on the palm joint of the hand to be identified;

[0012] The first coordinate axis is determined based on the first direction from the wrist joint to the palm joint in the hand to be identified;

[0013] The second coordinate axis is determined based on a straight line perpendicular to the first coordinate axis;

[0014] A projected coordinate system is established based on the origin of the coordinate system, the first coordinate axis, and the second coordinate axis. In one embodiment, the projection distance includes: a first projection distance and a second projection distance; calculating the first included angle and projection distance between the fingertip joint and the palm joint of each finger in the hand to be identified includes:

[0015] Based on the spatial coordinates of the fingertip joints and the palm joints of each finger, the first projection distance, the second projection distance, and the first included angle are obtained.

[0016] Wherein, the first projection distance includes: the distance between the fingertip joints of each finger and the first coordinate axis in the projection coordinate system;

[0017] The second projection distance includes the distance between the fingertip joints of each finger and the second coordinate axis in the projection coordinate system;

[0018] The first included angle includes: determining the angle between the first vector and the second coordinate axis, wherein the first vector is the vector formed by the fingertip joints of each finger and the palm joints.

[0019] In one embodiment, after obtaining the spatial coordinates of each joint in the hand to be identified, the method further includes:

[0020] Based on the spatial coordinates of each joint, determine the flexion and extension state of each finger in the hand to be identified and / or the opening and closing state between each finger;

[0021] The gesture recognition result of the hand to be recognized is determined based on the flexion and extension state of each finger and / or the opening and closing state between each finger, or the gesture recognition result of the hand to be recognized is determined based on the flexion and extension state of each finger and / or the opening and closing state between each finger, as well as the curvature.

[0022] The flexion-extension state includes: finger extension state, finger curling state, intermediate state between extension and curling, and unrecognizable state; the opening-closing state includes: closed state, open state, and unrecognizable state.

[0023] In one embodiment, determining the flexion / extension state of each finger and / or the opening / closing state between each finger in the hand to be identified includes:

[0024] Based on the spatial coordinates of each joint in each finger of the hand to be identified, the dot product between each joint in each finger is calculated, and the flexion and extension state of each finger is determined based on the dot product.

[0025] Based on the spatial coordinates of each joint in each finger of the hand to be identified, the second included angle between each finger is calculated, and the opening and closing state between each finger is determined based on the second included angle.

[0026] In one embodiment, calculating the dot product between the joints of each finger and determining the flexion / extension state of each finger based on the dot product includes:

[0027] Determine the joint vector based on the spatial coordinates of adjacent joints in each finger;

[0028] Calculate the sum of the dot products among the joint vectors in each finger;

[0029] The flexion and extension states of each finger are determined based on the sum of the dot products and a pre-set range.

[0030] In one embodiment, calculating the second angle between each finger and determining the opening / closing state of each finger based on the second angle includes:

[0031] The recognition vector for each finger is determined based on the spatial coordinates of the wrist joint in the hand to be identified and the spatial coordinates of the middle joints in each finger.

[0032] Calculate the second angle between the recognition vectors of adjacent fingers;

[0033] The opening and closing state between each finger is determined based on the second included angle and a preset included angle threshold.

[0034] In one embodiment, before obtaining the spatial coordinates of each joint in the hand to be identified, the method further includes:

[0035] The parameters of the hand to be identified are calibrated according to the preset gesture calibration action to determine the extreme states of the hand to be identified. The extreme states include at least: the maximum and minimum extension length of the hand to be identified, the maximum and minimum closing angle of each finger in the hand to be identified, and the maximum and minimum bending degree of each finger in the hand to be identified.

[0036] In one embodiment, the trigger gesture recognition includes:

[0037] The first position vector is determined based on the spatial coordinates of the eye and the spatial coordinates of the wrist joint in the hand to be identified.

[0038] The second position vector is determined based on the spatial coordinates of the wrist joint and the spatial coordinates of the palm joint in the hand to be identified.

[0039] Calculate the third angle between the first position vector and the second position vector;

[0040] Gesture recognition is triggered when the third included angle is within a preset included angle range threshold.

[0041] In one embodiment, after calculating the third angle between the first position vector and the second position vector, the method further includes:

[0042] If the third included angle is not within the preset included angle threshold, or if the hand to be recognized is not within the preset hand recognition area, the gesture recognition is terminated.

[0043] Secondly, this disclosure also provides a gesture recognition device. The device includes:

[0044] The coordinate acquisition module is used to acquire the spatial coordinates of each joint in the hand to be recognized when gesture recognition is triggered.

[0045] The curvature determination module is used to calculate the first included angle and projection distance between the fingertip joint and the palm joint of each finger in the hand to be identified, based on a pre-established projection coordinate system and the spatial coordinates, and to determine the curvature of each finger based on the first included angle and the projection distance; wherein, the projection coordinate system is established based on the palm joint and wrist joint in the hand to be identified;

[0046] The gesture recognition module is used to determine the gesture recognition result of the hand to be recognized based on the curvature of each finger.

[0047] Thirdly, this disclosure also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of any of the methods described above.

[0048] Fourthly, this disclosure also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0049] Fifthly, this disclosure also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of any of the methods described above.

[0050] In the above embodiments, the spatial coordinate information of each joint in the hand to be recognized obtained by the AR device can be fully utilized. Based on this spatial coordinate information and the projected coordinate system, the curvature can be determined. The curvature can be obtained through simple mathematical calculations, without the need for deep learning models or massive computational demands. Since the curvature represents the positional relationship between each finger and the palm joint, the posture of each finger in the hand to be recognized can be accurately determined, thus enabling the recognition of various gestures. Furthermore, this solution can be applied to devices with limited computing power and in scenarios lacking network interaction capabilities and unable to utilize high-performance neural network recognition capabilities in the background, enabling rapid gesture recognition. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0052] Figure 1 This is a schematic diagram illustrating the application environment of the gesture recognition method in one embodiment;

[0053] Figure 2 This is a flowchart illustrating a gesture recognition method in one embodiment;

[0054] Figure 3 This is a schematic diagram of the joints of the hand in one embodiment;

[0055] Figure 4 This is a flowchart illustrating the process of establishing a projected coordinate system in one embodiment;

[0056] Figure 5 This is a schematic diagram of the projected coordinate system in one embodiment;

[0057] Figure 6 This is a schematic diagram of the process after step S202 in one embodiment;

[0058] Figure 7 This is a schematic diagram of the flow of step S301 in one embodiment;

[0059] Figure 8 This is a schematic diagram of the flow of step S402 in one embodiment;

[0060] Figure 9 This is a schematic diagram of the joint vector of the thumb in one embodiment;

[0061] Figure 10 This is a schematic diagram of the flow of step S404 in one embodiment;

[0062] Figure 11 This is a schematic diagram of the recognition vectors for the thumb and index finger in one embodiment;

[0063] Figure 12 This is a flowchart illustrating the steps for triggering gesture recognition in one embodiment;

[0064] Figure 13 This is a schematic diagram of the start / stop angle position of the gesture service in one embodiment;

[0065] Figure 14 This is a flowchart illustrating the gesture recognition method in another embodiment;

[0066] Figure 15 This is a schematic block diagram of the structure of a gesture recognition device in one embodiment;

[0067] Figure 16 This is a schematic diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this disclosure.

[0069] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0070] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0071] As mentioned in the background section, current methods for gesture recognition mainly fall into several categories. First, there's the high-accuracy approach, aiming for accurate gesture recognition results. This typically utilizes image recognition, machine learning, and neural networks. However, this approach suffers from drawbacks: it's computationally intensive and complex, relying heavily on backend server resources for computation, leading to significant resource dependence during deployment. Second, there's algorithm-based gesture recognition, which defines basic rules to determine whether a gesture conforms to or not. This approach is easy to understand, write, maintain, and debug, but it limits the types of gestures it can recognize. Other existing algorithms are overly simplistic and abstract, lacking quantitative and qualitative criteria for gesture recognition, a unified and achievable standard process, and generally offering low accuracy.

[0072] Therefore, to solve the above problems, this disclosure provides a gesture recognition method that can be applied to, for example... Figure 1 In the application environment shown, the AR (Augmented Reality) device 102 communicates with the server 104 via a network. A data storage system can store the data that the server 104 needs to process. The data storage system can be integrated onto the server 104 or placed in the cloud or on another network server. When gesture recognition is triggered by the AR device 102, the AR device 102 acquires the spatial coordinates of each joint in the hand to be recognized. The server 104 acquires the spatial coordinates of each joint in the hand to be recognized from the AR device. Based on a pre-established projection coordinate system and spatial coordinates, the server 104 calculates the first angle and projection distance between the fingertip joint and the palm joint of each finger in the hand to be recognized, and determines the curvature of each finger based on the first angle and projection distance. The projection coordinate system and the server 104 are established based on the palm joint and wrist joint of the hand to be recognized. The server 104 can determine the gesture recognition result of the hand to be recognized based on the curvature of each finger. The AR device 102 can be an augmented reality device, including binocular AR devices (such as HoloLens and Magic Leap), monocular AR devices (such as Google Glass and the AR cycling glasses Solos), plug-in AR devices (such as the Lenovo Star Wars AR headset), contact lens AR devices (such as Mojo Vision), static holographic display AR devices (such as Sony's static desktop display), and in-vehicle AR devices (such as Garmin's head-up display (HUD)). The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers. It is understood that this solution can also be applied independently to AR devices and implemented through them.

[0073] In one embodiment, such as Figure 2 As shown, a gesture recognition method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0074] S202, when gesture recognition is triggered, obtain the spatial coordinates of each joint in the hand to be recognized.

[0075] Gesture recognition can be used to identify the gestures of the hand being recognized. AR devices such as HoloLens can be used to obtain the spatial coordinate data of each joint in the hand being recognized. This spatial coordinate data can be represented using (X, Y, Z). The joints in the hand being recognized can be referenced as follows: Figure 3 As shown, the following variables can be used to represent the joints in the hand to be identified: Thumb: Fingertip joint A1, Distal joint A2, Proximal joint A3, Metacarpal joint A4. Index finger: Fingertip joint B1, Distal joint B2, Middle joint B3, Proximal joint B4, Metacarpal joint B5. Middle finger: Fingertip joint C1, Distal joint C2, Middle joint C3, Proximal joint C4, Metacarpal joint C5. Ring finger: Fingertip joint D1, Distal joint D2, Middle joint D3, Proximal joint D4, Metacarpal joint D5. Little finger: Fingertip joint E1, Distal joint E2, Middle joint E3, Proximal joint E4, Metacarpal joint E5. Palmar joint: P. Wrist joint: W.

[0076] Specifically, gesture recognition on the AR device can be triggered by the interaction of the hand to be recognized. Once gesture recognition is triggered, the AR device acquires the spatial coordinates of each joint in the hand. The server then retrieves these spatial coordinates from the AR device.

[0077] S204, based on the pre-established projection coordinate system and the spatial coordinates, calculate the first included angle and projection distance between the fingertip joint and the palm joint of each finger in the hand to be identified, and determine the curvature of each finger based on the first included angle and the projection distance.

[0078] The projection coordinate system is established based on the palmar and wrist joints of the hand to be identified. It is used to determine the relative positional relationship between the fingertip joints and palmar joints of each finger. The curvature can typically be data representing the finger's posture. The projection distance can typically be the distance between the projection coordinate system axes and is used to represent the positional relationship between the fingertip joints and palmar joints of each finger. The first included angle can typically represent the angular relationship between the fingertip joints and palmar joints of each finger.

[0079] Specifically, a projection coordinate system can be established based on the palm joint P and wrist joint W in the hand to be identified. Based on the projection coordinate system and the spatial coordinates of the fingertip joints and the palm joints of each finger in the hand to be identified, the first angle and projection distance between the fingertip joints and the palm joints of each finger in the hand to be identified can be calculated. The curvature of each finger can be determined based on the first angle and the projection distance.

[0080] S206, determine the gesture recognition result of the hand to be recognized based on the curvature of each finger.

[0081] Specifically, after determining the curvature of each finger, the positional relationship of each finger relative to the palm joint is determined. This positional relationship can be matched with the pre-set positional relationships corresponding to various gestures, and the gesture recognition result of the hand to be recognized is determined based on the matching result. The pre-set positional relationships corresponding to various gestures can usually be adjusted and determined by the user during gesture calibration.

[0082] The aforementioned gesture recognition method fully utilizes the spatial coordinate information of each joint in the hand to be recognized obtained from AR devices. Based on this spatial coordinate information and the projected coordinate system, the curvature can be determined through simple mathematical calculations, eliminating the need for deep learning models and massive computational demands. Since the curvature represents the positional relationship between each finger and the palm joint, it accurately determines the posture of each finger in the hand to be recognized, thus enabling the recognition of various gestures. Furthermore, this solution can be applied to devices with limited computing power and in scenarios lacking network interaction capabilities and unable to leverage high-performance neural network recognition capabilities, enabling rapid gesture recognition.

[0083] In one embodiment, such as Figure 4 As shown, the process of establishing the projected coordinate system includes:

[0084] S302, determine the origin of the coordinate system based on the palm joint of the hand to be identified;

[0085] S304, determine the first coordinate axis based on the first direction from the wrist joint to the palm joint in the hand to be identified;

[0086] S306, Determine the second coordinate axis based on a straight line perpendicular to the first coordinate axis;

[0087] S308, Establish a projected coordinate system based on the origin of the coordinate system, the first coordinate axis, and the second coordinate axis.

[0088] Specifically, such as Figure 5As shown, the origin of the projection coordinate system can be the palm joint of the hand to be identified. The first coordinate axis can be determined by a first direction from the wrist joint W to the palm joint P of the hand to be identified. The second coordinate axis can be determined by a straight line perpendicular to the first coordinate axis. A projection coordinate system is established based on the origin, the first coordinate axis, and the second coordinate axis. The first coordinate axis can be either the X-axis or the Y-axis. The second coordinate axis can also be either the X-axis or the Y-axis. When the first coordinate axis is the Y-axis, the second coordinate axis is the X-axis. When the first coordinate axis is the X-axis, the second coordinate axis is the Y-axis. The positive directions of the first and second coordinate axes are not limited in this embodiment.

[0089] In this embodiment, a projection coordinate system is established by the wrist joint and the palm joint, with the palm joint as the origin of the coordinate system. This allows for the accurate determination of the relative positional relationship between each finger and between the palm joint, thereby enabling the accurate determination of the curvature of each finger and improving the accuracy of gesture recognition.

[0090] In one embodiment, the projection distance includes: a first projection distance and a second projection distance; calculating the first angle and projection distance between the fingertip joint and the palmar joint of each finger in the hand to be identified includes:

[0091] Based on the spatial coordinates of the fingertip joints and the palm joints of each finger, the first projection distance, the second projection distance, and the first included angle are obtained.

[0092] Wherein, the first projection distance includes: the distance between the fingertip joints of each finger and the first coordinate axis in the projection coordinate system;

[0093] The second projection distance includes the distance between the fingertip joints of each finger and the second coordinate axis in the projection coordinate system;

[0094] The first included angle includes: determining the angle between the first vector and the second coordinate axis, wherein the first vector is the vector formed by the fingertip joints of each finger and the palm joints.

[0095] Specifically, based on the spatial coordinates of each finger and the spatial coordinates of the palm joint, the first projection distance, the second projection distance of each finger, and the angle between each finger and the palm joint are calculated.

[0096] In some exemplary embodiments, reference continues to be made to... Figure 5 ,set up Figure 5Let A be the fingertip joint and O be the palm joint. The coordinates of A can be (X1, Y1), and the first projection distance can be X1. The second projection distance can be Y1. The first angle can be determined by the angle between the vector OX formed by the palm joint O and the positive X direction, and the vector OA formed by the fingertip joint A and the palm joint O. Alternatively, the first angle can be determined by the angle between the vector XO formed by the palm joint O and the negative X direction, and the vector OA formed by the fingertip joint A and the palm joint O.

[0097] In this embodiment, the first projection distance, the second projection distance, and the first included angle are used to refine the determination of the finger state, which can accurately determine the state of the finger.

[0098] In one embodiment, such as Figure 6 As shown, after obtaining the spatial coordinates of each joint in the hand to be identified, the method further includes:

[0099] S301, Based on the spatial coordinates of each joint, determine the flexion and extension state of each finger in the hand to be identified and / or the opening and closing state between each finger;

[0100] S303, determine the gesture recognition result of the hand to be recognized based on the flexion and extension state of each finger and / or the opening and closing state between each finger, or,

[0101] S305, determine the gesture recognition result of the hand to be recognized based on the flexion and extension state of each finger and / or the opening and closing state between each finger, as well as the degree of curvature.

[0102] The flexion-extension state includes: finger extension state, finger curling state, intermediate state between extension and curling, and unidentifiable state; the flexion-extension state indicates whether each finger in the hand to be identified is bent.

[0103] The opening / closing states include: a closed state, a non-closed state, and an unrecognizable state. The opening / closing state refers to the state between two fingers. An unrecognizable state can be determined if the finger to be recognized in the hand is obscured, or if the finger is not within the corresponding recognition area; in this case, the finger's state can be considered unrecognizable.

[0104] Specifically, the flexion and extension state of each finger in the hand to be recognized is determined based on the spatial coordinates of each finger. The opening and closing state of the fingers in the hand to be recognized is determined based on the spatial coordinates between each finger. The gesture recognition result of the hand to be recognized can be determined based on the flexion and extension state of each finger and / or the opening and closing state of the fingers.

[0105] Some gestures can be identified simply by the flexion / extension or opening / closing state of each finger. Only one feature is needed to identify each gesture; not all features need to be used. For example, for each gesture a user needs to identify, the user can freely combine all the aforementioned feature values ​​to uniquely represent the gesture. For instance, a gesture with fingers spread can be defined as all fingers being in a non-closed state, ignoring all other features, thus eliminating the need to use flexion / extension states.

[0106] Therefore, gesture recognition can be performed using only the flexion / extension or opening / closing state of the hand to be identified. If some gestures cannot be recognized using only the flexion / extension or opening / closing state, then the flexion / extension and opening / closing states can be combined to perform gesture recognition on the hand to be identified.

[0107] If gesture recognition cannot be performed based on the combined flexion / extension and opening / closing states, it may be because the flexion / extension state or the opening / closing state is unrecognizable, or the gesture is too complex to be recognized even after combining the flexion / extension and opening / closing states. For example, several gestures may have similar states, making it impossible to determine the gesture based solely on a limited set of states (flexion / extension and / or opening / closing states). In such cases, the gesture recognition result of the hand to be recognized can be determined based on the flexion / extension state of each finger and / or the opening / closing state between each finger, as well as the degree of curvature.

[0108] In this embodiment, either a flexion / extension state or an opening / closing state can be determined. Recognizing simple gestures using these states improves recognition efficiency and reduces computational load. For more complex gestures, a degree of curvature can be introduced for recognition, further enhancing the accuracy of gesture recognition.

[0109] In one embodiment, such as Figure 7 As shown, determining the flexion / extension state of each finger and / or the opening / closing state between each finger in the hand to be identified includes:

[0110] S402, based on the spatial coordinates of each joint in each finger of the hand to be identified, calculate the dot product between each joint in each finger, and determine the flexion and extension state of each finger based on the dot product.

[0111] S404, calculate the second included angle between each finger based on the spatial coordinates of each joint in each finger of the hand to be identified, and determine the opening and closing state between each finger based on the second included angle.

[0112] Specifically, based on the spatial coordinates of each joint in each finger of the hand to be identified, a vector is determined between each joint. The dot product between the joints of each finger is then calculated based on this vector, and the flexion / extension state of each finger is determined based on the dot product. Furthermore, based on the spatial coordinates of each joint, a vector is determined between each finger. This vector is then used to determine the second angle between the fingers, and the opening / closing state of each finger is determined based on this second angle.

[0113] In this embodiment, the flexion-extension state and the opening-closing state can be determined by the spatial coordinates of different joints. The opening-closing state determination, such as the five fingers together and the five fingers spread apart, is relatively quick and effective. The flexion-extension state determination, such as whether each finger is bent, is relatively effective. It can quickly identify the state between the fingers, and thus recognize simple gestures.

[0114] In one embodiment, such as Figure 8 As shown, the calculation of the dot product between each joint in each finger, and the determination of the flexion and extension state of each finger based on the dot product, includes:

[0115] S502, determine the joint vector based on the spatial coordinates of adjacent joints in each finger;

[0116] S504, calculate the sum of the dot products between the joint vectors in each finger;

[0117] S506, determine the flexion and extension state of each finger based on the sum of the dot products and the preset range.

[0118] Here, the joint vector can be a vector formed by the relationship between two adjacent joints. Adjacent joints can be referenced. Figure 3 As shown, A1 and A2 are adjacent joints. The preset range is usually determined based on the results after the user completes the final calibration guide, but currently the final value is usually between 0.95 and 0.98, 0.5 and 0.6, and 0 and 0.1, with the thumb being about half of that range.

[0119] Specifically, the spatial coordinates of adjacent joints in each finger can be determined, and joint vectors can be determined based on these coordinates. Typically, the thumb can be determined with two joint vectors. Other fingers can typically be determined with three joint vectors. Each joint vector is normalized, and the sum of the dot products between the normalized finger joint vectors is calculated. Normalization means keeping the vector direction unchanged but changing its length to 1, which helps control the range of the final calculated value. The flexion / extension state of each finger is determined based on the sum of the dot products and a pre-set range. If the sum of the dot products is between 0.95 and 0.98, the finger is in an extended state. If the sum of the dot products is between 0.5 and 0.6, the finger is in an intermediate state between extension and flexion. If the sum of the dot products is between 0 and 0.1, the finger is in a flexed state.

[0120] In some exemplary embodiments, such as Figure 9 As shown, taking the thumb as an example, the three finger joints correspond to three different spatial coordinates, such as the fingertip joint C (X1, Y1, Z1), the distal phalanx joint B (X2, Y2, X2), and the proximal phalanx joint A (X3, Y3, Z3). Figure 4 There are two angles between adjacent joints: the angle between the distal phalanx and distal phalanx, and the angle between the distal phalanx and proximal phalanx. These two angles are represented mathematically using the concept of a dot product. Vectors AB and BC are the joint vectors, respectively. The normalized dot product of vectors AB and BC is calculated. The dot product (also called dot multiplication, inner product, etc.) represents the angle between two vectors. If the dot product value is greater than 0, it indicates that the two vectors are in the same direction; the closer the value is to 0, the closer the angle is to perpendicularity; and 0 indicates that the two vectors are perpendicular. In this embodiment, only three joints of the thumb are used for calculation, while the other fingers have four joints each. Therefore, the value for the thumb is a single dot product, while the values ​​for the other fingers are the sum of multiple dot products. The sum of the dot products represents the flexion and extension state of that finger.

[0121] In one embodiment, such as Figure 10 As shown, calculating the second angle between each finger and determining the opening / closing state of each finger based on the second angle includes:

[0122] S602, determine the recognition vector corresponding to each finger based on the spatial coordinates of the wrist joint in the hand to be identified and the spatial coordinates of the middle joints in each finger.

[0123] S604, calculate the second angle between the recognition vectors of adjacent fingers.

[0124] S606, determine the opening and closing state between each finger based on the second included angle and the preset included angle threshold.

[0125] The pre-set angle threshold can usually be determined based on gesture calibration. Since hand sizes vary significantly among individuals, the threshold needs to be determined according to gesture calibration parameters. That is, after prompting the user to spread their five fingers for calibration, the user needs to complete the specified action and click "OK," and the range is determined based on the recorded user values. Those skilled in the art can also manually set the threshold. For example, a value within 5° generally indicates that two fingers are together, while a value greater than 5° indicates that two fingers are not together.

[0126] Specifically, the spatial coordinates of the wrist joint (where flexibility is relatively poor) and the middle joints of each finger (the thumb is the distal joint) can be selected to determine the recognition vector for each finger. The vector formed by each middle joint and the wrist joint can serve as a recognition vector. This recognition vector typically represents the positional relationship of the fingers. A second angle is calculated between the recognition vectors of adjacent fingers. Based on this second angle and a pre-set angle threshold, the opening / closing state of each finger is determined. For example, if the little finger is close to the adjacent ring finger, we mark the little finger as being in a closed state. Other fingers are marked in the same direction, and each finger (except the thumb) also has a corresponding opening / closing state.

[0127] In some exemplary embodiments, such as Figure 11 As shown, B is the distal phalanx of the thumb, C is the middle phalanx of the index finger, and A is the wrist joint. Therefore, AB can be the recognition vector for the thumb, and AC can be the recognition vector for the index finger. The second angle can be the angle between AC and AB, and the second angle can also be the angle used to determine when the thumb and index finger are together.

[0128] In one embodiment, before obtaining the spatial coordinates of each joint in the hand to be identified, the method further includes:

[0129] The parameters of the hand to be identified are calibrated according to the preset gesture calibration action to determine the extreme states of the hand to be identified. The extreme states include at least: the maximum and minimum extension length of the hand to be identified, the maximum and minimum closing angle of each finger in the hand to be identified, and the maximum and minimum bending degree of each finger in the hand to be identified.

[0130] Among them, the extreme state can be the limit state of the hand.

[0131] Specifically, before recognizing the hand to be recognized, parameter calibration of the hand is required to determine the various parameters and extreme states of the hand. Preset basic gestures can be provided for parameter calibration. These basic gestures can include clenching the fist, opening and closing the fingers, and widening the fingers, etc. By recording these gestures, the relative extreme values ​​of various finger states in the user's hand to be recognized can be calculated, such as the maximum and minimum extension length, the maximum and minimum closing angle, and the maximum and minimum bending degree, facilitating automatic parameter adjustment and calibration during subsequent calculations.

[0132] You can also set the gestures to be recognized and use those gestures to calibrate the parameters of the hand to be recognized. Subsequent recognition of the parameters corresponding to that gesture can then be performed on the hand to be recognized based on those parameters.

[0133] In one embodiment, such as Figure 12 As shown, the trigger gesture recognition includes:

[0134] S702, determine the first position vector based on the spatial coordinates of the eye and the spatial coordinates of the wrist joint in the hand to be identified;

[0135] S704, determine the second position vector based on the spatial coordinates of the wrist joint and the spatial coordinates of the palm joint in the hand to be identified;

[0136] S706, Calculate the third angle between the first position vector and the second position vector;

[0137] S708, if the third included angle is within a preset included angle range threshold, gesture recognition is triggered.

[0138] The preset angle range threshold is typically determined based on a person's natural hand-raising habit and the optimal recognition area of ​​the AR device. This angle allows the human eye to comfortably see their own gesture, and the AR device's camera can fully detect a complete hand. The preset angle range threshold is usually a range, such as 60 degrees to 120 degrees. A preferred angle is 90 degrees, which is generally the midpoint of most optimal ranges. The eye area is typically the location of the AR device.

[0139] Specifically, such as Figure 13 As shown, the first position vector is determined based on the spatial coordinates of the eye and the spatial coordinates of the wrist joint in the hand to be identified. The second position vector is determined based on the spatial coordinates of the wrist joint and the spatial coordinates of the palm joint in the hand to be identified. Calculate the user's eye ray and wrist position vector Vectors of wrist position and palm position The third included angle. This third included angle can be used as the start-up judgment angle. If the third included angle is within a preset included angle range threshold, gesture recognition is triggered.

[0140] In this embodiment, determining whether gesture recognition can be triggered by using the third included angle reduces the probability of misrecognition. Subsequent gesture recognition steps are automatically triggered only at specific angles, minimizing the impact on daily operations. This guides the user's hands to the optimal angle and position for gesture recognition, improving the accuracy of subsequent recognition.

[0141] In one embodiment, after calculating the third angle between the first position vector and the second position vector, the method further includes:

[0142] If the third angle is not included in the preset angle threshold, or if the hand to be recognized is not in the preset hand recognition area, the gesture recognition will be exited.

[0143] Specifically, if the third included angle is not within the preset included angle threshold, it can be proven that gesture recognition is not required through the AR device, and therefore gesture recognition is exited.

[0144] AR devices typically have some hardware limitations. Like the human eye, they have a field of view (FOV). For example, the human eye cannot see anything beyond 180 degrees (i.e., it cannot see things behind its head). The angle of view of AR devices is generally much smaller than that of the human eye. It is common for the human eye to see something but the AR device cannot recognize it. Therefore, when the AR device fails to detect the wrist or palm joint of the hand being recognized, it cannot calculate the third angle and thus exits gesture recognition.

[0145] It is understood that in some embodiments of this disclosure, the hand to be identified can be either the left or right hand. Depending on the hand, the corresponding implementation may have slight differences, such as the direction of the thumb. When the hand to be identified is determined to be either the left or right hand, those skilled in the art can adjust accordingly to meet the recognition requirements of different hands.

[0146] In one embodiment, this disclosure also provides another gesture recognition method, such as Figure 14 As shown, it includes the following steps:

[0147] S802 calibrates the parameters of the hand to be recognized according to the preset gesture calibration action, and determines the extreme state of the hand to be recognized.

[0148] S804, determine the first position vector based on the spatial coordinates of the eye and the spatial coordinates of the wrist joint in the hand to be identified.

[0149] S806, determine the second position vector based on the spatial coordinates of the wrist joint and the spatial coordinates of the palm joint in the hand to be identified.

[0150] S808, calculate the third angle between the first position vector and the second position vector.

[0151] S810, if the third included angle is within a preset included angle range threshold, gesture recognition is triggered.

[0152] S812, when gesture recognition is triggered, obtains the spatial coordinates of each joint in the hand to be recognized.

[0153] S814, based on the pre-established projection coordinate system and the spatial coordinates, calculate the first included angle and projection distance between the fingertip joint and the palm joint of each finger in the hand to be identified, and determine the curvature of each finger based on the first included angle and the projection distance; wherein, the projection coordinate system is established based on the palm joint and wrist joint in the hand to be identified.

[0154] S816 determines the gesture recognition result of the hand to be recognized based on the curvature of each finger.

[0155] S818, based on the spatial coordinates of each joint, determine the flexion and extension state of each finger in the hand to be identified and / or the opening and closing state between each finger.

[0156] S820 determines the gesture recognition result of the hand to be recognized based on the flexion and extension state of each finger and / or the opening and closing state between each finger.

[0157] Alternatively, S822, the gesture recognition result of the hand to be recognized is determined based on the flexion and extension state of each finger and / or the opening and closing state between each finger, as well as the degree of curvature.

[0158] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0159] Based on the same inventive concept, this disclosure also provides a gesture recognition device for implementing the gesture recognition method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more gesture recognition device embodiments provided below can be found in the limitations of the gesture recognition method described above, and will not be repeated here.

[0160] In one embodiment, such as Figure 15As shown, a gesture recognition device 1400 is provided, including: a coordinate acquisition module 1402, a curvature determination module 1404, and a gesture recognition module 1406, wherein:

[0161] The coordinate acquisition module 1402 is used to acquire the spatial coordinates of each joint in the hand to be recognized when gesture recognition is triggered.

[0162] The curvature determination module 1404 is used to calculate the first included angle and projection distance between the fingertip joint and the palm joint of each finger in the hand to be identified according to a pre-established projection coordinate system and the spatial coordinates, and to determine the curvature of each finger according to the first included angle and the projection distance; wherein, the projection coordinate system is established based on the palm joint and wrist joint in the hand to be identified;

[0163] The gesture recognition module 1406 is used to determine the gesture recognition result of the hand to be recognized based on the curvature of each finger.

[0164] In one embodiment of the device, the device further includes: a coordinate system establishment module, configured to determine the origin of the coordinate system based on the palm joint in the hand to be identified; determine a first coordinate axis based on a first direction from the wrist joint to the palm joint in the hand to be identified; determine a second coordinate axis based on a straight line perpendicular to the first coordinate axis; and establish a projected coordinate system based on the origin of the coordinate system, the first coordinate axis, and the second coordinate axis.

[0165] In one embodiment of the device, the projection distance includes: a first projection distance and a second projection distance.

[0166] In one embodiment of the device, the curvature determination module 1404 is further configured to obtain a first projection distance, a second projection distance, and a first included angle based on the spatial coordinates of the fingertip joints and the palm joints of each finger; wherein, the first projection distance includes: the distance between the fingertip joints of each finger and the first coordinate axis in the projection coordinate system; the second projection distance includes: the distance between the fingertip joints of each finger and the second coordinate axis in the projection coordinate system; the first included angle includes: determining the included angle between a first vector and the second coordinate axis, wherein the first vector is a vector formed by the fingertip joints of each finger and the palm joints.

[0167] In one embodiment of the device, the device further includes: a state determination module, used to determine the flexion and extension state of each finger in the hand to be identified and / or the opening and closing state between each finger based on the spatial coordinates of each joint.

[0168] The gesture recognition module 1406 is further configured to determine the gesture recognition result of the hand to be recognized based on the flexion and extension state of each finger and / or the opening and closing state between each finger, or to determine the gesture recognition result of the hand to be recognized based on the flexion and extension state of each finger and / or the opening and closing state between each finger, as well as the curvature.

[0169] The flexion-extension state includes: finger extension state, finger curling state, intermediate state between extension and curling, and unrecognizable state; the opening-closing state includes: closed state, open state, and unrecognizable state.

[0170] In one embodiment of the device, the state determination module includes a flexion-extension state determination module, used to calculate the dot product between the joints of each finger according to the spatial coordinates of each joint in each finger of the hand to be identified, and determine the flexion-extension state of each finger according to the dot product.

[0171] The opening / closing state determination module is used to calculate the second included angle between each finger based on the spatial coordinates of each joint in each finger of the hand to be identified, and to determine the opening / closing state between each finger based on the second included angle.

[0172] In one embodiment of the device, the flexion-extension state determination module includes: a joint vector determination module, used to determine joint vectors based on the spatial coordinates of adjacent joints in each finger.

[0173] The dot product calculation module is used to calculate the sum of the dot products between the joint vectors in each finger.

[0174] The range determination module is used to determine the flexion and extension state of each finger based on the sum of the dot products and a pre-set range interval.

[0175] In one embodiment of the device, the opening / closing state determination module includes: a recognition vector determination module, used to determine the recognition vector corresponding to each finger based on the spatial coordinates of the wrist joint in the hand to be identified and the spatial coordinates of the middle joints in each finger.

[0176] The included angle calculation module is used to calculate the second included angle between the recognition vectors of adjacent fingers.

[0177] The threshold determination module is used to determine the opening and closing state between each finger based on the second included angle and a preset included angle threshold.

[0178] In one embodiment of the device, the device further includes: a parameter calibration module, used to perform parameter calibration on the hand to be identified according to a preset gesture calibration action, and determine the extreme states in the hand to be identified, wherein the extreme states include at least: the maximum and minimum extension length of the hand to be identified, the maximum and minimum closing angle of each finger in the hand to be identified, and the maximum and minimum bending degree of each finger in the hand to be identified.

[0179] In one embodiment of the device, the device further includes: a gesture recognition triggering module, configured to determine a first position vector based on the spatial coordinates of the eyes and the spatial coordinates of the wrist joint in the hand to be recognized; determine a second position vector based on the spatial coordinates of the wrist joint and the spatial coordinates of the palm joint in the hand to be recognized; calculate a third angle between the first position vector and the second position vector; and trigger gesture recognition if the third angle is within a preset angle range threshold.

[0180] In one embodiment of the device, the device further includes a gesture recognition exit module, used to exit gesture recognition when the third included angle is not within a preset included angle threshold, or when the hand to be recognized is not within a preset hand recognition area.

[0181] Each module in the aforementioned gesture recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0182] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 16 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores spatial coordinate data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a gesture recognition method.

[0183] Those skilled in the art will understand that Figure 16The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the computer device to which the present disclosure is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0184] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0185] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0186] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0187] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this disclosure may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this disclosure may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0188] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0189] The embodiments described above are merely illustrative of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the appended claims.

Claims

1. A gesture recognition method, characterized in that, The method includes: When gesture recognition is triggered, obtain the spatial coordinates of each joint in the hand to be recognized; Based on the pre-established projection coordinate system and the spatial coordinates, the first included angle and projection distance between the fingertip joint and the palm joint of each finger in the hand to be identified are calculated, and the curvature of each finger is determined based on the first included angle and the projection distance; wherein, the projection coordinate system is established based on the palm joint and wrist joint in the hand to be identified, and is used to determine the relative positional relationship between the fingertip joint and the palm joint of each finger. The gesture recognition result of the hand to be recognized is determined based on the curvature of each finger; The process of establishing the projection coordinate system includes: determining the origin of the coordinate system based on the palm joint of the hand to be identified; determining the first coordinate axis based on the first direction from the wrist joint to the palm joint of the hand to be identified; determining the second coordinate axis based on the straight line perpendicular to the first coordinate axis; and establishing the projection coordinate system based on the origin of the coordinate system, the first coordinate axis, and the second coordinate axis.

2. The method according to claim 1, characterized in that, The projection distance includes: a first projection distance and a second projection distance; the calculation of the first angle and projection distance between the fingertip joint and the palm joint of each finger in the hand to be identified includes: Based on the spatial coordinates of the fingertip joints and the palm joints of each finger, the first projection distance, the second projection distance, and the first included angle are obtained. Wherein, the first projection distance includes: the distance between the fingertip joints of each finger and the first coordinate axis in the projection coordinate system; The second projection distance includes the distance between the fingertip joints of each finger and the second coordinate axis in the projection coordinate system; The first included angle includes: determining the angle between the first vector and the second coordinate axis, wherein the first vector is the vector formed by the fingertip joints of each finger and the palm joints.

3. The method according to claim 1, characterized in that, After obtaining the spatial coordinates of each joint in the hand to be identified, the method further includes: Based on the spatial coordinates of each joint, determine the flexion and extension state of each finger in the hand to be identified and / or the opening and closing state between each finger; The gesture recognition result of the hand to be recognized is determined based on the flexion and extension state of each finger and / or the opening and closing state between each finger, or the gesture recognition result of the hand to be recognized is determined based on the flexion and extension state of each finger and / or the opening and closing state between each finger, as well as the curvature. The flexion-extension state includes: finger extension state, finger curling state, intermediate state between extension and curling, and unrecognizable state; the opening-closing state includes: closed state, open state, and unrecognizable state.

4. The method according to claim 3, characterized in that, Determining the flexion and extension state of each finger in the hand to be identified includes: Determine the joint vector based on the spatial coordinates of adjacent joints in each finger; Calculate the sum of the dot products among the joint vectors in each finger; The flexion and extension states of each finger are determined based on the sum of the dot products and a pre-set range.

5. The method according to claim 3, characterized in that, Determine the open / closed state of each finger, including: Based on the spatial coordinates of the wrist joint in the hand to be identified and the spatial coordinates of the middle joints in each finger, determine the identification vector corresponding to each finger; Calculate the second angle between the recognition vectors of adjacent fingers; The opening and closing state between each finger is determined based on the second included angle and a preset included angle threshold.

6. The method according to claim 1, characterized in that, Before obtaining the spatial coordinates of each joint in the hand to be identified, the method further includes: The parameters of the hand to be identified are calibrated according to the preset gesture calibration action to determine the extreme states of the hand to be identified. The extreme states include at least: the maximum and minimum extension length of the hand to be identified, the maximum and minimum closing angle of each finger in the hand to be identified, and the maximum and minimum bending degree of each finger in the hand to be identified.

7. The method according to claim 1, characterized in that, The trigger gesture recognition includes: The first position vector is determined based on the spatial coordinates of the eye and the spatial coordinates of the wrist joint in the hand to be identified. The second position vector is determined based on the spatial coordinates of the wrist joint and the spatial coordinates of the palm joint in the hand to be identified. Calculate the third angle between the first position vector and the second position vector; If the third included angle is within a preset included angle range threshold, gesture recognition is triggered; After calculating the third angle between the first position vector and the second position vector, the method further includes: If the third included angle is not within the preset included angle threshold, or if the hand to be recognized is not within the preset hand recognition area, the gesture recognition is terminated.

8. The method according to claim 1, characterized in that, The method is applied to the interaction of AR devices.

9. A gesture recognition device, characterized in that, The device includes: The coordinate acquisition module is used to acquire the spatial coordinates of each joint in the hand to be recognized when gesture recognition is triggered. A curvature determination module is used to calculate the first included angle and projected distance between the fingertip joint and palm joint of each finger in the hand to be identified, based on a pre-established projection coordinate system and the spatial coordinates, and to determine the curvature of each finger based on the first included angle and the projected distance. The projection coordinate system is established based on the palm joint and wrist joint of the hand to be identified, and is used to determine the relative positional relationship between the fingertip joint and palm joint of each finger. The process of establishing the projection coordinate system includes: determining the origin of the coordinate system based on the palm joint of the hand to be identified; determining a first coordinate axis based on a first direction from the wrist joint to the palm joint of the hand to be identified; determining a second coordinate axis based on a straight line perpendicular to the first coordinate axis; and establishing a projection coordinate system based on the origin, the first coordinate axis, and the second coordinate axis. The gesture recognition module is used to determine the gesture recognition result of the hand to be recognized based on the curvature of each finger.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images